Outliers¶

Outliers are data points that differ significantly from the rest of the data. While some outliers can reveal some important information (machine failure, systems fraud…), they can also be simple errors.

Some machine learning algorithms are sensitive to outliers. In fact, they can destroy the final predictions because of how much bias they add to the data, and handling outliers in our data is one of the most important parts of the data preparation.

Outliers consist of three main types:

Global Outliers: Values far outside the entirety of their source dataset.

Contextual Outliers: Values deviate significantly from the rest of the data points in the same context.

Collective Outliers: Values that aren’t global or contextual outliers, but as a collection deviate significantly from the entire dataset.

Global outliers are often the most critical type and can add a significant amount of bias into the data. Fortunately, we can easily identify these outliers by computing the Z-Score.

Let’s look at some examples using the Heart Disease dataset. This dataset contains information on patients who are likely to have heart-related complications.

import verticapy as vp

heart = vp.read_csv("heart.csv")
heart.head(100)

	123 age Int 100%	...	123 thal Int 100%	123 target Int 100%
1	29	...	2	1
2	29	...	2	1
3	29	...	2	1
4	29	...	2	1
5	35	...	2	1
6	35	...	2	1
7	35	...	2	1
8	35	...	2	1
9	35	...	3	0
10	35	...	3	0
11	35	...	3	0
12	39	...	2	1
13	39	...	2	1
14	39	...	2	1
15	42	...	3	1
16	42	...	3	1
17	42	...	3	1
18	42	...	2	1
19	42	...	2	1
20	42	...	2	1
21	43	...	2	1
22	43	...	2	1
23	43	...	2	1
24	43	...	2	1
25	43	...	2	1
26	43	...	2	1
27	44	...	2	0
28	44	...	2	0
29	44	...	2	0
30	44	...	2	1
31	44	...	2	1
32	44	...	2	1
33	44	...	2	1
34	44	...	2	1
35	44	...	2	1
36	45	...	2	1
37	45	...	2	1
38	45	...	2	1
39	45	...	2	1
40	45	...	2	1
41	45	...	2	1
42	48	...	2	1
43	48	...	2	1
44	48	...	2	1
45	51	...	3	0
46	51	...	3	0
47	51	...	3	0
48	51	...	3	0
49	51	...	2	1
50	51	...	2	1
51	51	...	2	1
52	51	...	2	1
53	51	...	2	1
54	51	...	2	1
55	51	...	2	1
56	52	...	2	1
57	52	...	2	1
58	52	...	2	1
59	52	...	3	1
60	52	...	3	1
61	52	...	3	1
62	52	...	0	0
63	52	...	0	0
64	52	...	0	0
65	52	...	0	0
66	52	...	2	1
67	52	...	2	1
68	52	...	2	1
69	52	...	2	1
70	52	...	2	1
71	52	...	2	1
72	52	...	2	1
73	52	...	2	1
74	52	...	2	1
75	53	...	2	1
76	53	...	2	1
77	53	...	2	1
78	54	...	2	1
79	54	...	2	1
80	54	...	2	1
81	54	...	2	0
82	54	...	2	0
83	54	...	2	0
84	54	...	2	0
85	54	...	3	1
86	54	...	3	1
87	54	...	3	1
88	55	...	3	0
89	55	...	3	0
90	55	...	3	0
91	55	...	3	0
92	55	...	2	0
93	55	...	2	0
94	55	...	2	0
95	55	...	2	0
96	55	...	2	1
97	55	...	2	1
98	55	...	2	1
99	56	...	2	1
100	56	...	2	1

Let’s focus on a patient’s maximum heart rate (thalach) and the cholesterol (chol) to identify some outliers.

heart.scatter(["thalach", "chol"])

We can see some outliers of the distribution: people with high cholesterol and others with a very low heart rate. Let’s compute the global outliers using the outliers() method.

heart.outliers(["thalach", "chol"], "global_outliers")
heart.scatter(["thalach", "chol"], by = "global_outliers")

It is also possible to draw an outlier plot using the outliers_plot() method.

heart.outliers_plot(["thalach", "chol"],)

We’ve detected some global outliers in the distribution and we can impute these with the fill_outliers() method.

Generally, you can identify global outliers with the Z-Score. We’ll consider a Z-Score greater than 3 indicates that the datapoint is an outlier. Some less precise techniques consider the data points belonging in the first and last alpha-quantile as outliers. You’re free to choose either of these strategies when filling outliers.

heart["thalach"].fill_outliers(
    use_threshold = True,
    threshold = 3.0,
    method = "winsorize",
)
heart["chol"].fill_outliers(
    use_threshold = True,
    threshold = 3.0,
    method = "winsorize",
)
heart.scatter(
    ["thalach", "chol"],
    by = "global_outliers",
)

Other techniques like DBSCAN or local outlier factor (LOF) can be to used to check other data points for outliers.

from verticapy.machine_learning.vertica import DBSCAN

model = DBSCAN(eps = 20, min_samples = 10)
model.fit(heart, ["thalach", "chol"])
model.plot()

heart_dbscan = model.predict()
heart_dbscan["outliers_dbscan"] = "(dbscan_cluster = -1)::int"
heart_dbscan.scatter(
    ["thalach", "chol"],
    by = "outliers_dbscan",
)

While DBSCAN identifies outliers when computing the clusters, LOF computes an outlier score. Generally, a LOF Score greater than 1.5 indicates an outlier.

from verticapy.machine_learning.vertica import LocalOutlierFactor

model = LocalOutlierFactor()
model.fit(heart, ["thalach", "chol",])
model.plot()

heart_lof = model.predict()
heart_lof["outliers"] = "(CASE WHEN lof_score > 1.5 THEN 1 ELSE 0 END)"
heart_lof.scatter(
    ["thalach", "chol"],
    by = "outliers",
)

We have many other techniques like the KMeans clustering for finding outliers, but the most important method is using the Z-Score. After identifying outliers, we just have to decide how to impute the missing values. We’ll focus on missing values in the next lesson.