vDataFrame[].drop_outliers

In [ ]:
vDataFrame[].drop_outliers(threshold: float = 4.0,
                           use_threshold: bool = True, 
                           alpha: float = 0.05)

Drops the vcolumn outliers.

Parameters

Name Type Optional Description
threshold
float
✓
Uses the Gaussian distribution to define the outliers. After normalizing the data (Z-Score), if the absolute value of the record is greater than the threshold it will be considered as an outlier.
use_threshold
bool
✓
Uses the threshold instead of the 'alpha' parameter.
alpha
float
✓
Number representing the outliers threshold. Values lesser than quantile(alpha) or greater than quantile(1-alpha) will be dropped.

Returns

vDataFrame : self.parent

Example

In [58]:
from verticapy.datasets import load_titanic
titanic = load_titanic()
display(titanic)
123
fare
Numeric(10,5)
123
survived
Int
Abc
sex
Varchar(20)
Abc
boat
Varchar(100)
123
pclass
Int
123
age
Numeric(6,3)
Abc
ticket
Varchar(36)
Abc
Varchar(164)
Abc
embarked
Varchar(20)
Abc
cabin
Varchar(30)
123
body
Int
123
parch
Int
Abc
home.dest
Varchar(100)
123
sibsp
Int
1151.550000female[null]12.000113781SC22 C26[null]2Montreal, PQ / Chesterville, ON1
2151.550000male[null]130.000113781SC22 C261352Montreal, PQ / Chesterville, ON1
3151.550000female[null]125.000113781SC22 C26[null]2Montreal, PQ / Chesterville, ON1
40.000000male[null]139.000112050SA36[null]0Belfast, NI0
549.504200male[null]171.000PC 17609C[null]220Montevideo, Uruguay0
6227.525000male[null]147.000PC 17757CC62 C641240New York, NY1
725.925000male[null]1[null]PC 17318S[null][null]0New York, NY0
8247.520800male[null]124.000PC 17558CB58 B60[null]1Montreal, PQ0
975.241700maleA136.00013050CC6[null]0Winnipeg, MN0
1026.000000male[null]125.00013905C[null]1480San Francisco, CA0
1135.500000male[null]145.000113784ST[null]0Trenton, NJ0
1226.550000male[null]142.000110489SD22[null]0London / Winnipeg, MB0
1330.500000male[null]141.000113054SA21[null]0Pomeroy, WA0
1450.495800male[null]148.000PC 17591CB102080Omaha, NE0
1539.600000male[null]1[null]112379C[null][null]0Philadelphia, PA0
1626.550000male[null]145.000113050SB38[null]0Washington, DC0
1731.000000male[null]1[null]113798S[null][null]0[null]0
185.000000male[null]133.000695SB51 B53 B55[null]0New York, NY0
1947.100000male[null]128.000113059S[null][null]0Montevideo, Uruguay0
2047.100000male[null]117.000113059S[null][null]0Montevideo, Uruguay0
2126.000000male[null]149.00019924S[null][null]0Ascot, Berkshire / Rochester, NY0
2278.850000male[null]136.00019877SC461720Little Onn Hall, Staffs1
2361.175000male[null]146.000W.E.P. 5734SE31[null]0Amenia, ND1
240.000000male[null]1[null]112051S[null][null]0Liverpool, England / Belfast0
25136.779200male[null]127.00013508CC89[null]0Los Angeles, CA1
2652.000000male[null]1[null]110465SA14[null]0Stoughton, MA0
2725.587500male[null]147.0005727SE58[null]0Victoria, BC0
2883.158300male[null]137.000PC 17756CE52[null]1Lakewood, NJ1
2926.550000male[null]1[null]113791S[null][null]0Roachdale, IN0
3071.000000male[null]170.000WE/P 5735SB222691Milwaukee, WI1
3171.283300male[null]139.000PC 17599CC85[null]0New York, NY1
3252.000000male[null]131.000F.C. 12750SB71[null]0Montreal, PQ1
33106.425000male[null]150.000PC 17761CC86620Deephaven, MN / Cedar Rapids, IA1
3429.700000male[null]139.000PC 17580CA181330Philadelphia, PA0
3531.679200female[null]136.000PC 17531CA29[null]0New York, NY0
36221.779200male[null]1[null]PC 17483SC95[null]0[null]0
3727.750000male[null]130.000113051CC111[null]0New York, NY0
38263.000000male[null]119.00019950SC23 C25 C27[null]2Winnipeg, MB3
39263.000000male[null]164.00019950SC23 C25 C27[null]4Winnipeg, MB1
4026.550000male[null]1[null]113778SD34[null]0Westcliff-on-Sea, Essex0
410.000000male[null]1[null]112058SB102[null]0[null]0
4253.100000male[null]137.000113803SC123[null]0Scituate, MA1
4338.500000male[null]147.000111320SE632750St Anne's-on-Sea, Lancashire0
4479.200000male[null]124.000PC 17593CB86[null]0[null]0
4534.654200male[null]171.000PC 17754CA5[null]0New York, NY0
46153.462500male[null]138.000PC 17582SC911471Winnipeg, MB0
4779.200000male[null]146.000PC 17593CB82 B84[null]0New York, NY0
4842.400000male[null]1[null]113796S[null][null]0[null]0
4983.475000male[null]145.00036973SC83[null]0New York, NY1
500.000000male[null]140.000112059SB941100[null]0
5193.500000male[null]155.00012749SB693071Montreal, PQ1
5242.500000male[null]142.000113038SB11[null]0London / Middlesex0
5351.862500male[null]1[null]17463SE46[null]0Brighton, MA0
5450.000000male[null]155.000680SC39[null]0London / Birmingham0
5552.000000male[null]142.000113789S[null]380New York, NY1
5630.695800male141[null]PC 17600C[null][null]0New York, NY0
5728.712500female[null]150.000PC 17595CC49[null]0Paris, France New York, NY0
5826.000000male[null]146.000694S[null]800Bennington, VT0
5926.000000male[null]150.000113044SE60[null]0London0
60211.500000male[null]132.500113503CC132450[null]0
6129.700000male[null]158.00011771CB372580Buffalo, NY0
6251.862500male[null]141.00017464SD21[null]0Southington / Noank, CT1
6326.550000male[null]1[null]113028SC124[null]0Portland, OR0
6427.720800male[null]1[null]PC 17612C[null][null]0Chicago, IL0
6530.000000male[null]129.000113501SD61260Springfield, MA0
6645.500000male[null]130.000113801S[null][null]0London / New York, NY0
6726.000000male[null]130.000110469SC106[null]0Brockton, MA0
6853.100000male[null]119.000113773SD30[null]0New York, NY1
6975.241700male[null]146.00013050CC62920Vancouver, BC0
7051.862500male[null]154.00017463SE461750Dorchester, MA0
7182.170800male[null]128.000PC 17604C[null][null]0New York, NY1
7226.550000male[null]165.00013509SE382490East Bridgewater, MA0
7390.000000male[null]144.00019928QC782300Fond du Lac, WI2
7430.500000male[null]155.000113787SC30[null]0Montreal, PQ0
7542.400000male[null]147.000113796S[null][null]0Washington, DC0
7629.700000male[null]137.000PC 17596CC118[null]1Brooklyn, NY0
77113.275000male[null]158.00035273CD481222Lexington, MA0
7826.000000male[null]164.000693S[null]2630Isle of Wight, England0
7961.979200male[null]165.000113509CB302341Providence, RI0
8027.720800male[null]128.500PC 17562CD431890?Havana, Cuba0
810.000000male[null]1[null]112052S[null][null]0Belfast0
8228.500000male[null]145.500113043SC1241660Surbiton Hill, Surrey0
8393.500000male[null]123.00012749SB24[null]0Montreal, PQ0
8466.600000male[null]129.000113776SC2[null]0Isleworth, England1
85108.900000male[null]118.000PC 17758CC65[null]0Madrid, Spain1
8652.000000male[null]147.000110465SC1102070Worcester, MA0
870.000000male[null]138.00019972S[null][null]0Rotterdam, Netherlands0
88135.633300male[null]122.000PC 17760C[null]2320[null]0
89227.525000male[null]1[null]PC 17757C[null][null]0[null]0
9050.495800male[null]131.000PC 17590SA24[null]0Trenton, NJ0
9150.000000male[null]1[null]113767SA32[null]0Seattle, WA0
9240.125000male[null]136.00013049CA10[null]0Winnipeg, MB0
9359.400000male[null]155.000PC 17603C[null][null]0New York, NY1
9426.550000male[null]133.000113790S[null]1090London0
95262.375000male[null]161.000PC 17608CB57 B59 B63 B66[null]3Haverford, PA / Cooperstown, NY1
9655.900000male[null]150.00013507SE44[null]0Duluth, MN1
9726.550000male[null]156.000113792S[null][null]0New York, NY0
9830.695800male[null]156.00017764CA7[null]0St James, Long Island, NY0
9960.000000male[null]124.00013695SC31[null]0Huntington, WV1
10026.000000male[null]1[null]113056SA19[null]0Streatham, Surrey0
Rows: 1-100 of 1234 | Columns: 14
In [59]:
# Filtering the 'fare' having a Zscore greater than 3.0
titanic["fare"].drop_outliers(threshold = 3.0)
38 element(s) was/were filtered
123
fare
Numeric(10,5)
123
survived
Int
Abc
sex
Varchar(20)
Abc
boat
Varchar(100)
123
pclass
Int
123
age
Numeric(6,3)
Abc
ticket
Varchar(36)
Abc
Varchar(164)
Abc
embarked
Varchar(20)
Abc
cabin
Varchar(30)
123
body
Int
123
parch
Int
Abc
Varchar(100)
123
sibsp
Int
1151.550000female[null]12.000113781SC22 C26[null]21
2151.550000male[null]130.000113781SC22 C2613521
3151.550000female[null]125.000113781SC22 C26[null]21
40.000000male[null]139.000112050SA36[null]00
549.504200male[null]171.000PC 17609C[null]2200
625.925000male[null]1[null]PC 17318S[null][null]00
775.241700maleA136.00013050CC6[null]00
826.000000male[null]125.00013905C[null]14800
935.500000male[null]145.000113784ST[null]00
1026.550000male[null]142.000110489SD22[null]00
1130.500000male[null]141.000113054SA21[null]00
1250.495800male[null]148.000PC 17591CB1020800
1339.600000male[null]1[null]112379C[null][null]00
1426.550000male[null]145.000113050SB38[null]00
1531.000000male[null]1[null]113798S[null][null]00
165.000000male[null]133.000695SB51 B53 B55[null]00
1747.100000male[null]128.000113059S[null][null]00
1847.100000male[null]117.000113059S[null][null]00
1926.000000male[null]149.00019924S[null][null]00
2078.850000male[null]136.00019877SC4617201
2161.175000male[null]146.000W.E.P. 5734SE31[null]01
220.000000male[null]1[null]112051S[null][null]00
23136.779200male[null]127.00013508CC89[null]01
2452.000000male[null]1[null]110465SA14[null]00
2525.587500male[null]147.0005727SE58[null]00
2683.158300male[null]137.000PC 17756CE52[null]11
2726.550000male[null]1[null]113791S[null][null]00
2871.000000male[null]170.000WE/P 5735SB2226911
2971.283300male[null]139.000PC 17599CC85[null]01
3052.000000male[null]131.000F.C. 12750SB71[null]01
31106.425000male[null]150.000PC 17761CC866201
3229.700000male[null]139.000PC 17580CA1813300
3331.679200female[null]136.000PC 17531CA29[null]00
3427.750000male[null]130.000113051CC111[null]00
3526.550000male[null]1[null]113778SD34[null]00
360.000000male[null]1[null]112058SB102[null]00
3753.100000male[null]137.000113803SC123[null]01
3838.500000male[null]147.000111320SE6327500
3979.200000male[null]124.000PC 17593CB86[null]00
4034.654200male[null]171.000PC 17754CA5[null]00
41153.462500male[null]138.000PC 17582SC9114710
4279.200000male[null]146.000PC 17593CB82 B84[null]00
4342.400000male[null]1[null]113796S[null][null]00
4483.475000male[null]145.00036973SC83[null]01
450.000000male[null]140.000112059SB9411000
4693.500000male[null]155.00012749SB6930711
4742.500000male[null]142.000113038SB11[null]00
4851.862500male[null]1[null]17463SE46[null]00
4950.000000male[null]155.000680SC39[null]00
5052.000000male[null]142.000113789S[null]3801
5130.695800male141[null]PC 17600C[null][null]00
5228.712500female[null]150.000PC 17595CC49[null]00
5326.000000male[null]146.000694S[null]8000
5426.000000male[null]150.000113044SE60[null]00
5529.700000male[null]158.00011771CB3725800
5651.862500male[null]141.00017464SD21[null]01
5726.550000male[null]1[null]113028SC124[null]00
5827.720800male[null]1[null]PC 17612C[null][null]00
5930.000000male[null]129.000113501SD612600
6045.500000male[null]130.000113801S[null][null]00
6126.000000male[null]130.000110469SC106[null]00
6253.100000male[null]119.000113773SD30[null]01
6375.241700male[null]146.00013050CC629200
6451.862500male[null]154.00017463SE4617500
6582.170800male[null]128.000PC 17604C[null][null]01
6626.550000male[null]165.00013509SE3824900
6790.000000male[null]144.00019928QC7823002
6830.500000male[null]155.000113787SC30[null]00
6942.400000male[null]147.000113796S[null][null]00
7029.700000male[null]137.000PC 17596CC118[null]10
71113.275000male[null]158.00035273CD4812220
7226.000000male[null]164.000693S[null]26300
7361.979200male[null]165.000113509CB3023410
7427.720800male[null]128.500PC 17562CD4318900
750.000000male[null]1[null]112052S[null][null]00
7628.500000male[null]145.500113043SC12416600
7793.500000male[null]123.00012749SB24[null]00
7866.600000male[null]129.000113776SC2[null]01
79108.900000male[null]118.000PC 17758CC65[null]01
8052.000000male[null]147.000110465SC11020700
810.000000male[null]138.00019972S[null][null]00
82135.633300male[null]122.000PC 17760C[null]23200
8350.495800male[null]131.000PC 17590SA24[null]00
8450.000000male[null]1[null]113767SA32[null]00
8540.125000male[null]136.00013049CA10[null]00
8659.400000male[null]155.000PC 17603C[null][null]01
8726.550000male[null]133.000113790S[null]10900
8855.900000male[null]150.00013507SE44[null]01
8926.550000male[null]156.000113792S[null][null]00
9030.695800male[null]156.00017764CA7[null]00
9160.000000male[null]124.00013695SC31[null]01
9226.000000male[null]1[null]113056SA19[null]00
93146.520800male[null]157.000PC 17569CB78[null]01
9426.550000male[null]162.000113514SC87[null]00
9527.720800male[null]1[null]PC 17605C[null][null]00
9632.320800male[null]161.00036963SD504600
9779.650000male[null]152.000110413SE67[null]11
98110.883300male[null]149.00017421CC68[null]11
9927.720800male[null]140.000PC 17601C[null][null]00
10033.500000male[null]161.000111240SB1924500
Out[59]:
Rows: 1-100 of 1196 | Columns: 14
In [60]:
# Filtering the 'age' greater than the last decile 
# or lesser than the first decile
titanic["age"].drop_outliers(use_threshold = False, alpha = 0.1)
413 element(s) was/were filtered
123
fare
Numeric(10,5)
123
survived
Int
Abc
sex
Varchar(20)
Abc
boat
Varchar(100)
123
pclass
Int
123
age
Numeric(6,3)
Abc
ticket
Varchar(36)
Abc
Varchar(164)
Abc
embarked
Varchar(20)
Abc
cabin
Varchar(30)
123
body
Int
123
parch
Int
Abc
Varchar(100)
123
sibsp
Int
1151.550000male[null]130.000113781SC22 C2613521
2151.550000female[null]125.000113781SC22 C26[null]21
30.000000male[null]139.000112050SA36[null]00
475.241700maleA136.00013050CC6[null]00
526.000000male[null]125.00013905C[null]14800
635.500000male[null]145.000113784ST[null]00
726.550000male[null]142.000110489SD22[null]00
830.500000male[null]141.000113054SA21[null]00
950.495800male[null]148.000PC 17591CB1020800
1026.550000male[null]145.000113050SB38[null]00
115.000000male[null]133.000695SB51 B53 B55[null]00
1247.100000male[null]128.000113059S[null][null]00
1347.100000male[null]117.000113059S[null][null]00
1426.000000male[null]149.00019924S[null][null]00
1578.850000male[null]136.00019877SC4617201
1661.175000male[null]146.000W.E.P. 5734SE31[null]01
17136.779200male[null]127.00013508CC89[null]01
1825.587500male[null]147.0005727SE58[null]00
1983.158300male[null]137.000PC 17756CE52[null]11
2071.283300male[null]139.000PC 17599CC85[null]01
2152.000000male[null]131.000F.C. 12750SB71[null]01
22106.425000male[null]150.000PC 17761CC866201
2329.700000male[null]139.000PC 17580CA1813300
2431.679200female[null]136.000PC 17531CA29[null]00
2527.750000male[null]130.000113051CC111[null]00
2653.100000male[null]137.000113803SC123[null]01
2738.500000male[null]147.000111320SE6327500
2879.200000male[null]124.000PC 17593CB86[null]00
29153.462500male[null]138.000PC 17582SC9114710
3079.200000male[null]146.000PC 17593CB82 B84[null]00
3183.475000male[null]145.00036973SC83[null]01
320.000000male[null]140.000112059SB9411000
3342.500000male[null]142.000113038SB11[null]00
3452.000000male[null]142.000113789S[null]3801
3528.712500female[null]150.000PC 17595CC49[null]00
3626.000000male[null]146.000694S[null]8000
3726.000000male[null]150.000113044SE60[null]00
3851.862500male[null]141.00017464SD21[null]01
3930.000000male[null]129.000113501SD612600
4045.500000male[null]130.000113801S[null][null]00
4126.000000male[null]130.000110469SC106[null]00
4253.100000male[null]119.000113773SD30[null]01
4375.241700male[null]146.00013050CC629200
4482.170800male[null]128.000PC 17604C[null][null]01
4590.000000male[null]144.00019928QC7823002
4642.400000male[null]147.000113796S[null][null]00
4729.700000male[null]137.000PC 17596CC118[null]10
4827.720800male[null]128.500PC 17562CD4318900
4928.500000male[null]145.500113043SC12416600
5093.500000male[null]123.00012749SB24[null]00
5166.600000male[null]129.000113776SC2[null]01
52108.900000male[null]118.000PC 17758CC65[null]01
5352.000000male[null]147.000110465SC11020700
540.000000male[null]138.00019972S[null][null]00
55135.633300male[null]122.000PC 17760C[null]23200
5650.495800male[null]131.000PC 17590SA24[null]00
5740.125000male[null]136.00013049CA10[null]00
5826.550000male[null]133.000113790S[null]10900
5955.900000male[null]150.00013507SE44[null]01
6060.000000male[null]124.00013695SC31[null]01
61110.883300male[null]149.00017421CC68[null]11
6227.720800male[null]140.000PC 17601C[null][null]00
6334.020800male[null]147.00036967SD46[null]00
6477.287500male[null]121.00035281SD2616910
6526.550001male3148.00019952SE12[null]00
6669.300001female9124.000PC 17477CB35[null]00
6776.291701female8132.00011813CD15[null]00
6852.554201male5137.00011751SD35[null]11
6952.554201female5147.00011751SD35[null]11
7030.000001male5126.000111369CC148[null]00
7191.079201male7125.00011967CB49[null]01
7291.079201female7119.00011967CB49[null]01
73135.633301female8135.000PC 17760SC99[null]00
7426.550001maleD128.000110564SC52[null]00
7531.000001male7140.000112277CA31[null]00
76164.866701female8130.00036928SC7[null]00
7755.000001female6122.000113505SE33[null]10
7827.720801female6144.000PC 17610CB4[null]00
79134.500001female3141.00016966CE40[null]00
8026.287501male5142.000PC 17476SE24[null]00
81120.000001female4114.000113760SB96 B98[null]21
82120.000001maleC136.000113760SB96 B98[null]21
83120.000001female4136.000113760SB96 B98[null]21
8461.175001female4147.000W.E.P. 5734SE31[null]01
8553.100001male5127.000113806SE8[null]01
8653.100001female5133.000113806SE8[null]01
8786.500001female8130.000110152SB77[null]00
8829.700001male7145.000PC 17594CA9[null]00
89136.779201female4126.00013508CC89[null]01
90151.550001female11122.000113781S[null][null]00
9183.158301female14139.000PC 17756CE49[null]11
9271.000001female7136.000WE/P 5735SB22[null]20
9371.283301female4138.000PC 17599CC85[null]01
9430.500001male3127.000113804S[null][null]00
95151.550001female8133.000113781S[null][null]00
9652.000001female3127.000F.C. 12750SB71[null]21
9757.000001male3131.00017474SB20[null]01
9857.000001female3117.00017474SB20[null]01
99106.425001female2148.000PC 17761CC86[null]01
10083.158301female7123.00011767CC54[null]10
Out[60]:
Rows: 1-100 of 783 | Columns: 14

See Also

vDataFrame[].fill_outliers Fills the vcolumns outliers.
vDataFrame.outliers Adds a new vcolumn labeled with 0 and 1 (1 meaning global outlier).