vDataFrame[].fill_outliers

In [ ]:
vDataFrame[].fill_outliers(method: str = "winsorize",
                           threshold: float = 4.0, 
                           use_threshold: bool = True,
                           alpha: float = 0.05)

Fills the vcolumns outliers using the input method.

Parameters

Name Type Optional Description
method
str
✓
 Method to use to fill the vcolumn outliers.
  • mean : Replaces the upper and lower outliers by their respective average.
  • null : Replaces the outliers by the NULL value.
  • winsorize : Clips the vcolumn using as lower bound quantile(alpha) and as upper bound quantile(1-alpha) if 'use_threshold' is set to False else the lower and upper ZScores.
threshold
float
✓
 Uses the Gaussian distribution to define the outliers. After normalizing the data (Z-Score), if the absolute value of the record is greater than the threshold it will be considered as an outlier.
use_threshold
bool
✓
 Uses the threshold instead of the 'alpha' parameter.
alpha
float
✓
 Number representing the outliers threshold. Values lesser than quantile(alpha) or greater than quantile(1-alpha) will be filled.

Returns

vDataFrame : self.parent

Example

In [70]:
from verticapy.datasets import load_market
market = load_market().filter("Price < 0.7")
display(market.head(20))

294 element(s) was/were filtered
123
Price
Float
Abc
Form
Varchar(32)
Abc
Name
Varchar(32)
10.5104657455FrozenApples
20.537867915537FrozenApples
30.6311325278Ready to drinkApples
40.5494172928FreshBananas
50.566983414531FreshBananas
60.579208394258Fresh green cabbageCabbage
70.6238712291Fresh green cabbageCabbage
80.520793672FreshCantaloupe
90.535873776106FreshCantaloupe
100.674839678618FrozenGrapefruit
110.658986796FrozenOranges
120.689934119435FrozenOranges
130.627661945936FreshPineapple
140.6527945474FreshPineapple
150.590752357725FrozenPineapple
160.6203795725FrozenPineapple
170.564319813929FreshPotatoes
180.6037360906FreshPotatoes
190.3166387792FreshWatermelon
200.333412035323FreshWatermelon
Rows: 20 | Columns: 3
In [66]:
# All the outliers (abs(ZSCORE) > 1.5) will be replaced by the NULL values
market["Price"].fill_outliers(method = "null",
                              threshold = 1.5,
                              use_threshold = True)
123
Price
Float
Abc
Form
Varchar(32)
Abc
Name
Varchar(32)
10.5104657455FrozenApples
20.537867915537FrozenApples
30.6311325278Ready to drinkApples
40.5494172928FreshBananas
50.566983414531FreshBananas
60.579208394258Fresh green cabbageCabbage
70.6238712291Fresh green cabbageCabbage
80.520793672FreshCantaloupe
90.535873776106FreshCantaloupe
100.674839678618FrozenGrapefruit
110.658986796FrozenOranges
120.689934119435FrozenOranges
130.627661945936FreshPineapple
140.6527945474FreshPineapple
150.590752357725FrozenPineapple
160.6203795725FrozenPineapple
170.564319813929FreshPotatoes
180.6037360906FreshPotatoes
19[null]FreshWatermelon
20[null]FreshWatermelon
Out[66]:
Rows: 20 | Columns: 3
In [69]:
# All the outliers (abs(ZSCORE) > 1.5) will be replaced by the lower and 
# upper bound having a ZSCORE = 1.5 and -1.5
market["Price"].fill_outliers(method = "winsorize",
                              threshold = 1.5,
                              use_threshold = True)
123
Price
Float
Abc
Form
Varchar(32)
Abc
Name
Varchar(32)
10.5104657455FrozenApples
20.537867915537FrozenApples
30.6311325278Ready to drinkApples
40.5494172928FreshBananas
50.566983414531FreshBananas
60.579208394258Fresh green cabbageCabbage
70.6238712291Fresh green cabbageCabbage
80.520793672FreshCantaloupe
90.535873776106FreshCantaloupe
100.674839678618FrozenGrapefruit
110.658986796FrozenOranges
120.689934119435FrozenOranges
130.627661945936FreshPineapple
140.6527945474FreshPineapple
150.590752357725FrozenPineapple
160.6203795725FrozenPineapple
170.564319813929FreshPotatoes
180.6037360906FreshPotatoes
190.422205378756987FreshWatermelon
200.422205378756987FreshWatermelon
Out[69]:
Rows: 20 | Columns: 3
In [71]:
# All the outliers (values > quantile(0.8) or < quantile(0.8)) will be 
# replaced by the nearest of the two quantiles
market["Price"].fill_outliers(method = "winsorize",
                              alpha = 0.2,
                              use_threshold = False)
123
Price
Float
Abc
Form
Varchar(32)
Abc
Name
Varchar(32)
10.5328577552848FrozenApples
20.537867915537FrozenApples
30.6311325278Ready to drinkApples
40.5494172928FreshBananas
50.566983414531FreshBananas
60.579208394258Fresh green cabbageCabbage
70.6238712291Fresh green cabbageCabbage
80.5328577552848FreshCantaloupe
90.535873776106FreshCantaloupe
100.63546493172FrozenGrapefruit
110.63546493172FrozenOranges
120.63546493172FrozenOranges
130.627661945936FreshPineapple
140.63546493172FreshPineapple
150.590752357725FrozenPineapple
160.6203795725FrozenPineapple
170.564319813929FreshPotatoes
180.6037360906FreshPotatoes
190.5328577552848FreshWatermelon
200.5328577552848FreshWatermelon
Out[71]:
Rows: 20 | Columns: 3

See Also

vDataFrame[].drop_outliers Drops the vcolumn outliers.
vDataFrame.outliers Computes the vDataFrame Global Outliers.