{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Outliers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Outliers are data points that differ significantly from the rest of the data. While some outliers can reveal some important information (machine failure, systems fraud...), they can also be simple errors.\n", "\n", "Some machine learning algorithms are sensitive to outliers. In fact, they can destroy the final predictions because of how much bias they add to the data, and handling outliers in our data is one of the most important parts of the data preparation.\n", "\n", "Outliers consist of three main types:\n", "\n", "\n", "\n", "Global outliers are often the most critical type and can add a significant amount of bias into the data. Fortunately, we can easily identify these outliers by computing the Z-Score.\n", "\n", "Let's look at some examples using the 'Heart Disease' dataset. This dataset contains information on patients who are likely to have heart-related complications." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The table \"public\".\"heart\" has been successfully created.\n" ] }, { "data": { "text/html": [ "
123
age
Integer
123
sex
Integer
123
cp
Integer
123
trestbps
Integer
123
chol
Integer
123
fbs
Integer
123
restecg
Integer
123
thalach
Integer
123
exang
Integer
123
oldpeak
Numeric(7)
123
slope
Integer
123
ca
Integer
123
thal
Integer
123
target
Integer
129111302040020200.02021
229111302040020200.02021
329111302040020200.02021
429111302040020200.02021
534011182100119200.72021
634011182100119200.72021
734011182100119200.72021
834131181820017400.02021
934131181820017400.02021
1034131181820017400.02021
1135001381830118201.42021
1235001381830118201.42021
1335001381830118201.42021
1435001381830118201.42021
1535101201980113011.61030
1635101201980113011.61030
1735101201980113011.61030
1835101201980113011.61030
1935101262820015610.02030
2035101262820015610.02030
2135101262820015610.02030
2235111221920117400.02021
2335111221920117400.02021
2435111221920117400.02021
2535111221920117400.02021
2637021202150117000.02021
2737021202150117000.02021
2837021202150117000.02021
2937121302500118703.50021
3037121302500118703.50021
3137121302500118703.50021
3238121381750117300.02421
3338121381750117300.02421
3438121381750117300.02421
3538121381750117300.02421
3638121381750117300.02421
3738121381750117300.02421
3838121381750117300.02421
3938121381750117300.02421
4038131202310118213.81030
4138131202310118213.81030
4238131202310118213.81030
4338131202310118213.81030
443902941990117900.02021
453902941990117900.02021
463902941990117900.02021
4739021382200115200.01021
4839021382200115200.01021
4939021382200115200.01021
5039021382200115200.01021
5139101182190114001.21030
5239101182190114001.21030
5339101182190114001.21030
5439101182190114001.21030
5539121403210018200.02021
5639121403210018200.02021
5739121403210018200.02021
5840101101670011412.01030
5940101101670011412.01030
6040101101670011412.01030
6140101101670011412.01030
6240101522230118100.02030
6340101522230118100.02030
6440101522230118100.02030
6540101522230118100.02030
6640131401990117811.42031
6740131401990117811.42031
6840131401990117811.42031
6941011051980116800.02121
7041011051980116800.02121
7141011051980116800.02121
7241011263060116300.02021
7341011263060116300.02021
7441011263060116300.02021
7541011302040017201.42021
7641011302040017201.42021
7741011302040017201.42021
7841021122680017210.02021
7941021122680017210.02021
8041021122680017210.02021
8141101101720015800.02030
8241101101720015800.02030
8341101101720015800.02030
8441111102350115300.02021
8541111102350115300.02021
8641111102350115300.02021
8741111201570118200.02021
8841111201570118200.02021
8941111201570118200.02021
9041111201570118200.02021
9141111352030113200.01011
9241111352030113200.01011
9341111352030113200.01011
9441111352030113200.01011
9541121122500117900.02021
9641121122500117900.02021
9741121122500117900.02021
9841121302140016802.01021
9941121302140016802.01021
10041121302140016802.01021
Rows: 1-100 | Columns: 14
" ], "text/plain": [ "None age sex cp trestbps chol fbs \\\\\n", "1 29 1 1 130 204 0 \\\\\n", "2 29 1 1 130 204 0 \\\\\n", "3 29 1 1 130 204 0 \\\\\n", "4 29 1 1 130 204 0 \\\\\n", "5 34 0 1 118 210 0 \\\\\n", "6 34 0 1 118 210 0 \\\\\n", "7 34 0 1 118 210 0 \\\\\n", "8 34 1 3 118 182 0 \\\\\n", "9 34 1 3 118 182 0 \\\\\n", "10 34 1 3 118 182 0 \\\\\n", "11 35 0 0 138 183 0 \\\\\n", "12 35 0 0 138 183 0 \\\\\n", "13 35 0 0 138 183 0 \\\\\n", "14 35 0 0 138 183 0 \\\\\n", "15 35 1 0 120 198 0 \\\\\n", "16 35 1 0 120 198 0 \\\\\n", "17 35 1 0 120 198 0 \\\\\n", "18 35 1 0 120 198 0 \\\\\n", "19 35 1 0 126 282 0 \\\\\n", "20 35 1 0 126 282 0 \\\\\n", "21 35 1 0 126 282 0 \\\\\n", "22 35 1 1 122 192 0 \\\\\n", "23 35 1 1 122 192 0 \\\\\n", "24 35 1 1 122 192 0 \\\\\n", "25 35 1 1 122 192 0 \\\\\n", "26 37 0 2 120 215 0 \\\\\n", "27 37 0 2 120 215 0 \\\\\n", "28 37 0 2 120 215 0 \\\\\n", "29 37 1 2 130 250 0 \\\\\n", "30 37 1 2 130 250 0 \\\\\n", "31 37 1 2 130 250 0 \\\\\n", "32 38 1 2 138 175 0 \\\\\n", "33 38 1 2 138 175 0 \\\\\n", "34 38 1 2 138 175 0 \\\\\n", "35 38 1 2 138 175 0 \\\\\n", "36 38 1 2 138 175 0 \\\\\n", "37 38 1 2 138 175 0 \\\\\n", "38 38 1 2 138 175 0 \\\\\n", "39 38 1 2 138 175 0 \\\\\n", "40 38 1 3 120 231 0 \\\\\n", "41 38 1 3 120 231 0 \\\\\n", "42 38 1 3 120 231 0 \\\\\n", "43 38 1 3 120 231 0 \\\\\n", "44 39 0 2 94 199 0 \\\\\n", "45 39 0 2 94 199 0 \\\\\n", "46 39 0 2 94 199 0 \\\\\n", "47 39 0 2 138 220 0 \\\\\n", "48 39 0 2 138 220 0 \\\\\n", "49 39 0 2 138 220 0 \\\\\n", "50 39 0 2 138 220 0 \\\\\n", "51 39 1 0 118 219 0 \\\\\n", "52 39 1 0 118 219 0 \\\\\n", "53 39 1 0 118 219 0 \\\\\n", "54 39 1 0 118 219 0 \\\\\n", "55 39 1 2 140 321 0 \\\\\n", "56 39 1 2 140 321 0 \\\\\n", "57 39 1 2 140 321 0 \\\\\n", "58 40 1 0 110 167 0 \\\\\n", "59 40 1 0 110 167 0 \\\\\n", "60 40 1 0 110 167 0 \\\\\n", "61 40 1 0 110 167 0 \\\\\n", "62 40 1 0 152 223 0 \\\\\n", "63 40 1 0 152 223 0 \\\\\n", "64 40 1 0 152 223 0 \\\\\n", "65 40 1 0 152 223 0 \\\\\n", "66 40 1 3 140 199 0 \\\\\n", "67 40 1 3 140 199 0 \\\\\n", "68 40 1 3 140 199 0 \\\\\n", "69 41 0 1 105 198 0 \\\\\n", "70 41 0 1 105 198 0 \\\\\n", "71 41 0 1 105 198 0 \\\\\n", "72 41 0 1 126 306 0 \\\\\n", "73 41 0 1 126 306 0 \\\\\n", "74 41 0 1 126 306 0 \\\\\n", "75 41 0 1 130 204 0 \\\\\n", "76 41 0 1 130 204 0 \\\\\n", "77 41 0 1 130 204 0 \\\\\n", "78 41 0 2 112 268 0 \\\\\n", "79 41 0 2 112 268 0 \\\\\n", "80 41 0 2 112 268 0 \\\\\n", "81 41 1 0 110 172 0 \\\\\n", "82 41 1 0 110 172 0 \\\\\n", "83 41 1 0 110 172 0 \\\\\n", "84 41 1 1 110 235 0 \\\\\n", "85 41 1 1 110 235 0 \\\\\n", "86 41 1 1 110 235 0 \\\\\n", "87 41 1 1 120 157 0 \\\\\n", "88 41 1 1 120 157 0 \\\\\n", "89 41 1 1 120 157 0 \\\\\n", "90 41 1 1 120 157 0 \\\\\n", "91 41 1 1 135 203 0 \\\\\n", "92 41 1 1 135 203 0 \\\\\n", "93 41 1 1 135 203 0 \\\\\n", "94 41 1 1 135 203 0 \\\\\n", "95 41 1 2 112 250 0 \\\\\n", "96 41 1 2 112 250 0 \\\\\n", "97 41 1 2 112 250 0 \\\\\n", "98 41 1 2 130 214 0 \\\\\n", "99 41 1 2 130 214 0 \\\\\n", "100 41 1 2 130 214 0 \\\\\n", "None restecg thalach exang oldpeak slope \\\\\n", "1 0 202 0 0.0 2 \\\\\n", "2 0 202 0 0.0 2 \\\\\n", "3 0 202 0 0.0 2 \\\\\n", "4 0 202 0 0.0 2 \\\\\n", "5 1 192 0 0.7 2 \\\\\n", "6 1 192 0 0.7 2 \\\\\n", "7 1 192 0 0.7 2 \\\\\n", "8 0 174 0 0.0 2 \\\\\n", "9 0 174 0 0.0 2 \\\\\n", "10 0 174 0 0.0 2 \\\\\n", "11 1 182 0 1.4 2 \\\\\n", "12 1 182 0 1.4 2 \\\\\n", "13 1 182 0 1.4 2 \\\\\n", "14 1 182 0 1.4 2 \\\\\n", "15 1 130 1 1.6 1 \\\\\n", "16 1 130 1 1.6 1 \\\\\n", "17 1 130 1 1.6 1 \\\\\n", "18 1 130 1 1.6 1 \\\\\n", "19 0 156 1 0.0 2 \\\\\n", "20 0 156 1 0.0 2 \\\\\n", "21 0 156 1 0.0 2 \\\\\n", "22 1 174 0 0.0 2 \\\\\n", "23 1 174 0 0.0 2 \\\\\n", "24 1 174 0 0.0 2 \\\\\n", "25 1 174 0 0.0 2 \\\\\n", "26 1 170 0 0.0 2 \\\\\n", "27 1 170 0 0.0 2 \\\\\n", "28 1 170 0 0.0 2 \\\\\n", "29 1 187 0 3.5 0 \\\\\n", "30 1 187 0 3.5 0 \\\\\n", "31 1 187 0 3.5 0 \\\\\n", "32 1 173 0 0.0 2 \\\\\n", "33 1 173 0 0.0 2 \\\\\n", "34 1 173 0 0.0 2 \\\\\n", "35 1 173 0 0.0 2 \\\\\n", "36 1 173 0 0.0 2 \\\\\n", "37 1 173 0 0.0 2 \\\\\n", "38 1 173 0 0.0 2 \\\\\n", "39 1 173 0 0.0 2 \\\\\n", "40 1 182 1 3.8 1 \\\\\n", "41 1 182 1 3.8 1 \\\\\n", "42 1 182 1 3.8 1 \\\\\n", "43 1 182 1 3.8 1 \\\\\n", "44 1 179 0 0.0 2 \\\\\n", "45 1 179 0 0.0 2 \\\\\n", "46 1 179 0 0.0 2 \\\\\n", "47 1 152 0 0.0 1 \\\\\n", "48 1 152 0 0.0 1 \\\\\n", "49 1 152 0 0.0 1 \\\\\n", "50 1 152 0 0.0 1 \\\\\n", "51 1 140 0 1.2 1 \\\\\n", "52 1 140 0 1.2 1 \\\\\n", "53 1 140 0 1.2 1 \\\\\n", "54 1 140 0 1.2 1 \\\\\n", "55 0 182 0 0.0 2 \\\\\n", "56 0 182 0 0.0 2 \\\\\n", "57 0 182 0 0.0 2 \\\\\n", "58 0 114 1 2.0 1 \\\\\n", "59 0 114 1 2.0 1 \\\\\n", "60 0 114 1 2.0 1 \\\\\n", "61 0 114 1 2.0 1 \\\\\n", "62 1 181 0 0.0 2 \\\\\n", "63 1 181 0 0.0 2 \\\\\n", "64 1 181 0 0.0 2 \\\\\n", "65 1 181 0 0.0 2 \\\\\n", "66 1 178 1 1.4 2 \\\\\n", "67 1 178 1 1.4 2 \\\\\n", "68 1 178 1 1.4 2 \\\\\n", "69 1 168 0 0.0 2 \\\\\n", "70 1 168 0 0.0 2 \\\\\n", "71 1 168 0 0.0 2 \\\\\n", "72 1 163 0 0.0 2 \\\\\n", "73 1 163 0 0.0 2 \\\\\n", "74 1 163 0 0.0 2 \\\\\n", "75 0 172 0 1.4 2 \\\\\n", "76 0 172 0 1.4 2 \\\\\n", "77 0 172 0 1.4 2 \\\\\n", "78 0 172 1 0.0 2 \\\\\n", "79 0 172 1 0.0 2 \\\\\n", "80 0 172 1 0.0 2 \\\\\n", "81 0 158 0 0.0 2 \\\\\n", "82 0 158 0 0.0 2 \\\\\n", "83 0 158 0 0.0 2 \\\\\n", "84 1 153 0 0.0 2 \\\\\n", "85 1 153 0 0.0 2 \\\\\n", "86 1 153 0 0.0 2 \\\\\n", "87 1 182 0 0.0 2 \\\\\n", "88 1 182 0 0.0 2 \\\\\n", "89 1 182 0 0.0 2 \\\\\n", "90 1 182 0 0.0 2 \\\\\n", "91 1 132 0 0.0 1 \\\\\n", "92 1 132 0 0.0 1 \\\\\n", "93 1 132 0 0.0 1 \\\\\n", "94 1 132 0 0.0 1 \\\\\n", "95 1 179 0 0.0 2 \\\\\n", "96 1 179 0 0.0 2 \\\\\n", "97 1 179 0 0.0 2 \\\\\n", "98 0 168 0 2.0 1 \\\\\n", "99 0 168 0 2.0 1 \\\\\n", "100 0 168 0 2.0 1 \\\\\n", "None ca thal target \n", "1 0 2 1 \n", "2 0 2 1 \n", "3 0 2 1 \n", "4 0 2 1 \n", "5 0 2 1 \n", "6 0 2 1 \n", "7 0 2 1 \n", "8 0 2 1 \n", "9 0 2 1 \n", "10 0 2 1 \n", "11 0 2 1 \n", "12 0 2 1 \n", "13 0 2 1 \n", "14 0 2 1 \n", "15 0 3 0 \n", "16 0 3 0 \n", "17 0 3 0 \n", "18 0 3 0 \n", "19 0 3 0 \n", "20 0 3 0 \n", "21 0 3 0 \n", "22 0 2 1 \n", "23 0 2 1 \n", "24 0 2 1 \n", "25 0 2 1 \n", "26 0 2 1 \n", "27 0 2 1 \n", "28 0 2 1 \n", "29 0 2 1 \n", "30 0 2 1 \n", "31 0 2 1 \n", "32 4 2 1 \n", "33 4 2 1 \n", "34 4 2 1 \n", "35 4 2 1 \n", "36 4 2 1 \n", "37 4 2 1 \n", "38 4 2 1 \n", "39 4 2 1 \n", "40 0 3 0 \n", "41 0 3 0 \n", "42 0 3 0 \n", "43 0 3 0 \n", "44 0 2 1 \n", "45 0 2 1 \n", "46 0 2 1 \n", "47 0 2 1 \n", "48 0 2 1 \n", "49 0 2 1 \n", "50 0 2 1 \n", "51 0 3 0 \n", "52 0 3 0 \n", "53 0 3 0 \n", "54 0 3 0 \n", "55 0 2 1 \n", "56 0 2 1 \n", "57 0 2 1 \n", "58 0 3 0 \n", "59 0 3 0 \n", "60 0 3 0 \n", "61 0 3 0 \n", "62 0 3 0 \n", "63 0 3 0 \n", "64 0 3 0 \n", "65 0 3 0 \n", "66 0 3 1 \n", "67 0 3 1 \n", "68 0 3 1 \n", "69 1 2 1 \n", "70 1 2 1 \n", "71 1 2 1 \n", "72 0 2 1 \n", "73 0 2 1 \n", "74 0 2 1 \n", "75 0 2 1 \n", "76 0 2 1 \n", "77 0 2 1 \n", "78 0 2 1 \n", "79 0 2 1 \n", "80 0 2 1 \n", "81 0 3 0 \n", "82 0 3 0 \n", "83 0 3 0 \n", "84 0 2 1 \n", "85 0 2 1 \n", "86 0 2 1 \n", "87 0 2 1 \n", "88 0 2 1 \n", "89 0 2 1 \n", "90 0 2 1 \n", "91 0 1 1 \n", "92 0 1 1 \n", "93 0 1 1 \n", "94 0 1 1 \n", "95 0 2 1 \n", "96 0 2 1 \n", "97 0 2 1 \n", "98 0 2 1 \n", "99 0 2 1 \n", "100 0 2 1 \n", "Rows: 1-100 | Columns: 14" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import verticapy as vp\n", "\n", "vp.set_option(\"plotting_lib\",\"highcharts\")\n", "vp.drop(\"public.heart\") # To make sure there is no other table with that name\n", "vdf = vp.read_csv(\"data/heart.csv\", schema = \"public\", table_name = \"heart\")\n", "display(vdf)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's focus on a patient's maximum heart rate (thalach) and the cholesterol (chol) to identify some outliers." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%matplotlib inline\n", "vdf.scatter([\"thalach\", \"chol\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see some outliers of the distribution: people with high cholesterol and others with a very low heart rate. Let's compute the global outliers using the 'outliers' method." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vdf.outliers([\"thalach\", \"chol\"], \"global_outliers\")\n", "vdf.scatter([\"thalach\", \"chol\"], by = \"global_outliers\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is also possible to draw an outlier plot using the 'outliers_plot' method." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vdf.outliers_plot([\"thalach\", \"chol\"],)" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "We've detected some global outliers in the distribution and we can impute these with the 'fill_outliers' method.\n", "\n", "Generally, you can identify global outliers with the Z-Score. We'll consider a Z-Score greater than 3 indicates that the datapoint is an outlier. Some less precise techniques consider the data points belonging in the first and last alpha-quantile as outliers. You're free to choose either of these strategies when filling outliers." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vdf[\"thalach\"].fill_outliers(use_threshold = True,\n", " threshold = 3.0,\n", " method = \"winsorize\")\n", "vdf[\"chol\"].fill_outliers(use_threshold = True,\n", " threshold = 3.0,\n", " method = \"winsorize\")\n", "vdf.scatter([\"thalach\", \"chol\"], by = \"global_outliers\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Other techniques like DBSCAN or local outlier factor (LOF) can be to used to check other data points for outliers." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
123
age
Integer
123
sex
Integer
123
cp
Integer
123
trestbps
Integer
123
chol
Numeric(34)
123
fbs
Integer
123
restecg
Integer
123
thalach
Numeric(33)
123
exang
Integer
123
oldpeak
Numeric(7)
123
slope
Integer
123
ca
Integer
123
thal
Integer
123
target
Integer
123
global_outliers
Integer
12911130204.000202.000.020210
22911130204.000202.000.020210
32911130204.000202.000.020210
42911130204.000202.000.020210
53401118210.001192.000.720210
63401118210.001192.000.720210
73401118210.001192.000.720210
83413118182.000174.000.020210
93413118182.000174.000.020210
103413118182.000174.000.020210
113500138183.001182.001.420210
123500138183.001182.001.420210
133500138183.001182.001.420210
143500138183.001182.001.420210
153510120198.001130.011.610300
163510120198.001130.011.610300
173510120198.001130.011.610300
183510120198.001130.011.610300
193510126282.000156.010.020300
203510126282.000156.010.020300
213510126282.000156.010.020300
223511122192.001174.000.020210
233511122192.001174.000.020210
243511122192.001174.000.020210
253511122192.001174.000.020210
263702120215.001170.000.020210
273702120215.001170.000.020210
283702120215.001170.000.020210
293712130250.001187.003.500210
303712130250.001187.003.500210
313712130250.001187.003.500210
323812138175.001173.000.024210
333812138175.001173.000.024210
343812138175.001173.000.024210
353812138175.001173.000.024210
363812138175.001173.000.024210
373812138175.001173.000.024210
383812138175.001173.000.024210
393812138175.001173.000.024210
403813120231.001182.013.810300
413813120231.001182.013.810300
423813120231.001182.013.810300
433813120231.001182.013.810300
44390294199.001179.000.020210
45390294199.001179.000.020210
46390294199.001179.000.020210
473902138220.001152.000.010210
483902138220.001152.000.010210
493902138220.001152.000.010210
503902138220.001152.000.010210
513910118219.001140.001.210300
523910118219.001140.001.210300
533910118219.001140.001.210300
543910118219.001140.001.210300
553912140321.000182.000.020210
563912140321.000182.000.020210
573912140321.000182.000.020210
584010110167.000114.012.010300
594010110167.000114.012.010300
604010110167.000114.012.010300
614010110167.000114.012.010300
624010152223.001181.000.020300
634010152223.001181.000.020300
644010152223.001181.000.020300
654010152223.001181.000.020300
664013140199.001178.011.420310
674013140199.001178.011.420310
684013140199.001178.011.420310
694101105198.001168.000.021210
704101105198.001168.000.021210
714101105198.001168.000.021210
724101126306.001163.000.020210
734101126306.001163.000.020210
744101126306.001163.000.020210
754101130204.000172.001.420210
764101130204.000172.001.420210
774101130204.000172.001.420210
784102112268.000172.010.020210
794102112268.000172.010.020210
804102112268.000172.010.020210
814110110172.000158.000.020300
824110110172.000158.000.020300
834110110172.000158.000.020300
844111110235.001153.000.020210
854111110235.001153.000.020210
864111110235.001153.000.020210
874111120157.001182.000.020210
884111120157.001182.000.020210
894111120157.001182.000.020210
904111120157.001182.000.020210
914111135203.001132.000.010110
924111135203.001132.000.010110
934111135203.001132.000.010110
944111135203.001132.000.010110
954112112250.001179.000.020210
964112112250.001179.000.020210
974112112250.001179.000.020210
984112130214.000168.002.010210
994112130214.000168.002.010210
1004112130214.000168.002.010210
Rows: 1-100 | Columns: 15
" ], "text/plain": [ "None age sex cp trestbps chol fbs \\\\\n", "1 29 1 1 130 204.0 0 \\\\\n", "2 29 1 1 130 204.0 0 \\\\\n", "3 29 1 1 130 204.0 0 \\\\\n", "4 29 1 1 130 204.0 0 \\\\\n", "5 34 0 1 118 210.0 0 \\\\\n", "6 34 0 1 118 210.0 0 \\\\\n", "7 34 0 1 118 210.0 0 \\\\\n", "8 34 1 3 118 182.0 0 \\\\\n", "9 34 1 3 118 182.0 0 \\\\\n", "10 34 1 3 118 182.0 0 \\\\\n", "11 35 0 0 138 183.0 0 \\\\\n", "12 35 0 0 138 183.0 0 \\\\\n", "13 35 0 0 138 183.0 0 \\\\\n", "14 35 0 0 138 183.0 0 \\\\\n", "15 35 1 0 120 198.0 0 \\\\\n", "16 35 1 0 120 198.0 0 \\\\\n", "17 35 1 0 120 198.0 0 \\\\\n", "18 35 1 0 120 198.0 0 \\\\\n", "19 35 1 0 126 282.0 0 \\\\\n", "20 35 1 0 126 282.0 0 \\\\\n", "21 35 1 0 126 282.0 0 \\\\\n", "22 35 1 1 122 192.0 0 \\\\\n", "23 35 1 1 122 192.0 0 \\\\\n", "24 35 1 1 122 192.0 0 \\\\\n", "25 35 1 1 122 192.0 0 \\\\\n", "26 37 0 2 120 215.0 0 \\\\\n", "27 37 0 2 120 215.0 0 \\\\\n", "28 37 0 2 120 215.0 0 \\\\\n", "29 37 1 2 130 250.0 0 \\\\\n", "30 37 1 2 130 250.0 0 \\\\\n", "31 37 1 2 130 250.0 0 \\\\\n", "32 38 1 2 138 175.0 0 \\\\\n", "33 38 1 2 138 175.0 0 \\\\\n", "34 38 1 2 138 175.0 0 \\\\\n", "35 38 1 2 138 175.0 0 \\\\\n", "36 38 1 2 138 175.0 0 \\\\\n", "37 38 1 2 138 175.0 0 \\\\\n", "38 38 1 2 138 175.0 0 \\\\\n", "39 38 1 2 138 175.0 0 \\\\\n", "40 38 1 3 120 231.0 0 \\\\\n", "41 38 1 3 120 231.0 0 \\\\\n", "42 38 1 3 120 231.0 0 \\\\\n", "43 38 1 3 120 231.0 0 \\\\\n", "44 39 0 2 94 199.0 0 \\\\\n", "45 39 0 2 94 199.0 0 \\\\\n", "46 39 0 2 94 199.0 0 \\\\\n", "47 39 0 2 138 220.0 0 \\\\\n", "48 39 0 2 138 220.0 0 \\\\\n", "49 39 0 2 138 220.0 0 \\\\\n", "50 39 0 2 138 220.0 0 \\\\\n", "51 39 1 0 118 219.0 0 \\\\\n", "52 39 1 0 118 219.0 0 \\\\\n", "53 39 1 0 118 219.0 0 \\\\\n", "54 39 1 0 118 219.0 0 \\\\\n", "55 39 1 2 140 321.0 0 \\\\\n", "56 39 1 2 140 321.0 0 \\\\\n", "57 39 1 2 140 321.0 0 \\\\\n", "58 40 1 0 110 167.0 0 \\\\\n", "59 40 1 0 110 167.0 0 \\\\\n", "60 40 1 0 110 167.0 0 \\\\\n", "61 40 1 0 110 167.0 0 \\\\\n", "62 40 1 0 152 223.0 0 \\\\\n", "63 40 1 0 152 223.0 0 \\\\\n", "64 40 1 0 152 223.0 0 \\\\\n", "65 40 1 0 152 223.0 0 \\\\\n", "66 40 1 3 140 199.0 0 \\\\\n", "67 40 1 3 140 199.0 0 \\\\\n", "68 40 1 3 140 199.0 0 \\\\\n", "69 41 0 1 105 198.0 0 \\\\\n", "70 41 0 1 105 198.0 0 \\\\\n", "71 41 0 1 105 198.0 0 \\\\\n", "72 41 0 1 126 306.0 0 \\\\\n", "73 41 0 1 126 306.0 0 \\\\\n", "74 41 0 1 126 306.0 0 \\\\\n", "75 41 0 1 130 204.0 0 \\\\\n", "76 41 0 1 130 204.0 0 \\\\\n", "77 41 0 1 130 204.0 0 \\\\\n", "78 41 0 2 112 268.0 0 \\\\\n", "79 41 0 2 112 268.0 0 \\\\\n", "80 41 0 2 112 268.0 0 \\\\\n", "81 41 1 0 110 172.0 0 \\\\\n", "82 41 1 0 110 172.0 0 \\\\\n", "83 41 1 0 110 172.0 0 \\\\\n", "84 41 1 1 110 235.0 0 \\\\\n", "85 41 1 1 110 235.0 0 \\\\\n", "86 41 1 1 110 235.0 0 \\\\\n", "87 41 1 1 120 157.0 0 \\\\\n", "88 41 1 1 120 157.0 0 \\\\\n", "89 41 1 1 120 157.0 0 \\\\\n", "90 41 1 1 120 157.0 0 \\\\\n", "91 41 1 1 135 203.0 0 \\\\\n", "92 41 1 1 135 203.0 0 \\\\\n", "93 41 1 1 135 203.0 0 \\\\\n", "94 41 1 1 135 203.0 0 \\\\\n", "95 41 1 2 112 250.0 0 \\\\\n", "96 41 1 2 112 250.0 0 \\\\\n", "97 41 1 2 112 250.0 0 \\\\\n", "98 41 1 2 130 214.0 0 \\\\\n", "99 41 1 2 130 214.0 0 \\\\\n", "100 41 1 2 130 214.0 0 \\\\\n", "None restecg thalach exang oldpeak slope \\\\\n", "1 0 202.0 0 0.0 2 \\\\\n", "2 0 202.0 0 0.0 2 \\\\\n", "3 0 202.0 0 0.0 2 \\\\\n", "4 0 202.0 0 0.0 2 \\\\\n", "5 1 192.0 0 0.7 2 \\\\\n", "6 1 192.0 0 0.7 2 \\\\\n", "7 1 192.0 0 0.7 2 \\\\\n", "8 0 174.0 0 0.0 2 \\\\\n", "9 0 174.0 0 0.0 2 \\\\\n", "10 0 174.0 0 0.0 2 \\\\\n", "11 1 182.0 0 1.4 2 \\\\\n", "12 1 182.0 0 1.4 2 \\\\\n", "13 1 182.0 0 1.4 2 \\\\\n", "14 1 182.0 0 1.4 2 \\\\\n", "15 1 130.0 1 1.6 1 \\\\\n", "16 1 130.0 1 1.6 1 \\\\\n", "17 1 130.0 1 1.6 1 \\\\\n", "18 1 130.0 1 1.6 1 \\\\\n", "19 0 156.0 1 0.0 2 \\\\\n", "20 0 156.0 1 0.0 2 \\\\\n", "21 0 156.0 1 0.0 2 \\\\\n", "22 1 174.0 0 0.0 2 \\\\\n", "23 1 174.0 0 0.0 2 \\\\\n", "24 1 174.0 0 0.0 2 \\\\\n", "25 1 174.0 0 0.0 2 \\\\\n", "26 1 170.0 0 0.0 2 \\\\\n", "27 1 170.0 0 0.0 2 \\\\\n", "28 1 170.0 0 0.0 2 \\\\\n", "29 1 187.0 0 3.5 0 \\\\\n", "30 1 187.0 0 3.5 0 \\\\\n", "31 1 187.0 0 3.5 0 \\\\\n", "32 1 173.0 0 0.0 2 \\\\\n", "33 1 173.0 0 0.0 2 \\\\\n", "34 1 173.0 0 0.0 2 \\\\\n", "35 1 173.0 0 0.0 2 \\\\\n", "36 1 173.0 0 0.0 2 \\\\\n", "37 1 173.0 0 0.0 2 \\\\\n", "38 1 173.0 0 0.0 2 \\\\\n", "39 1 173.0 0 0.0 2 \\\\\n", "40 1 182.0 1 3.8 1 \\\\\n", "41 1 182.0 1 3.8 1 \\\\\n", "42 1 182.0 1 3.8 1 \\\\\n", "43 1 182.0 1 3.8 1 \\\\\n", "44 1 179.0 0 0.0 2 \\\\\n", "45 1 179.0 0 0.0 2 \\\\\n", "46 1 179.0 0 0.0 2 \\\\\n", "47 1 152.0 0 0.0 1 \\\\\n", "48 1 152.0 0 0.0 1 \\\\\n", "49 1 152.0 0 0.0 1 \\\\\n", "50 1 152.0 0 0.0 1 \\\\\n", "51 1 140.0 0 1.2 1 \\\\\n", "52 1 140.0 0 1.2 1 \\\\\n", "53 1 140.0 0 1.2 1 \\\\\n", "54 1 140.0 0 1.2 1 \\\\\n", "55 0 182.0 0 0.0 2 \\\\\n", "56 0 182.0 0 0.0 2 \\\\\n", "57 0 182.0 0 0.0 2 \\\\\n", "58 0 114.0 1 2.0 1 \\\\\n", "59 0 114.0 1 2.0 1 \\\\\n", "60 0 114.0 1 2.0 1 \\\\\n", "61 0 114.0 1 2.0 1 \\\\\n", "62 1 181.0 0 0.0 2 \\\\\n", "63 1 181.0 0 0.0 2 \\\\\n", "64 1 181.0 0 0.0 2 \\\\\n", "65 1 181.0 0 0.0 2 \\\\\n", "66 1 178.0 1 1.4 2 \\\\\n", "67 1 178.0 1 1.4 2 \\\\\n", "68 1 178.0 1 1.4 2 \\\\\n", "69 1 168.0 0 0.0 2 \\\\\n", "70 1 168.0 0 0.0 2 \\\\\n", "71 1 168.0 0 0.0 2 \\\\\n", "72 1 163.0 0 0.0 2 \\\\\n", "73 1 163.0 0 0.0 2 \\\\\n", "74 1 163.0 0 0.0 2 \\\\\n", "75 0 172.0 0 1.4 2 \\\\\n", "76 0 172.0 0 1.4 2 \\\\\n", "77 0 172.0 0 1.4 2 \\\\\n", "78 0 172.0 1 0.0 2 \\\\\n", "79 0 172.0 1 0.0 2 \\\\\n", "80 0 172.0 1 0.0 2 \\\\\n", "81 0 158.0 0 0.0 2 \\\\\n", "82 0 158.0 0 0.0 2 \\\\\n", "83 0 158.0 0 0.0 2 \\\\\n", "84 1 153.0 0 0.0 2 \\\\\n", "85 1 153.0 0 0.0 2 \\\\\n", "86 1 153.0 0 0.0 2 \\\\\n", "87 1 182.0 0 0.0 2 \\\\\n", "88 1 182.0 0 0.0 2 \\\\\n", "89 1 182.0 0 0.0 2 \\\\\n", "90 1 182.0 0 0.0 2 \\\\\n", "91 1 132.0 0 0.0 1 \\\\\n", "92 1 132.0 0 0.0 1 \\\\\n", "93 1 132.0 0 0.0 1 \\\\\n", "94 1 132.0 0 0.0 1 \\\\\n", "95 1 179.0 0 0.0 2 \\\\\n", "96 1 179.0 0 0.0 2 \\\\\n", "97 1 179.0 0 0.0 2 \\\\\n", "98 0 168.0 0 2.0 1 \\\\\n", "99 0 168.0 0 2.0 1 \\\\\n", "100 0 168.0 0 2.0 1 \\\\\n", "None ca thal target global_outliers \n", "1 0 2 1 0 \n", "2 0 2 1 0 \n", "3 0 2 1 0 \n", "4 0 2 1 0 \n", "5 0 2 1 0 \n", "6 0 2 1 0 \n", "7 0 2 1 0 \n", "8 0 2 1 0 \n", "9 0 2 1 0 \n", "10 0 2 1 0 \n", "11 0 2 1 0 \n", "12 0 2 1 0 \n", "13 0 2 1 0 \n", "14 0 2 1 0 \n", "15 0 3 0 0 \n", "16 0 3 0 0 \n", "17 0 3 0 0 \n", "18 0 3 0 0 \n", "19 0 3 0 0 \n", "20 0 3 0 0 \n", "21 0 3 0 0 \n", "22 0 2 1 0 \n", "23 0 2 1 0 \n", "24 0 2 1 0 \n", "25 0 2 1 0 \n", "26 0 2 1 0 \n", "27 0 2 1 0 \n", "28 0 2 1 0 \n", "29 0 2 1 0 \n", "30 0 2 1 0 \n", "31 0 2 1 0 \n", "32 4 2 1 0 \n", "33 4 2 1 0 \n", "34 4 2 1 0 \n", "35 4 2 1 0 \n", "36 4 2 1 0 \n", "37 4 2 1 0 \n", "38 4 2 1 0 \n", "39 4 2 1 0 \n", "40 0 3 0 0 \n", "41 0 3 0 0 \n", "42 0 3 0 0 \n", "43 0 3 0 0 \n", "44 0 2 1 0 \n", "45 0 2 1 0 \n", "46 0 2 1 0 \n", "47 0 2 1 0 \n", "48 0 2 1 0 \n", "49 0 2 1 0 \n", "50 0 2 1 0 \n", "51 0 3 0 0 \n", "52 0 3 0 0 \n", "53 0 3 0 0 \n", "54 0 3 0 0 \n", "55 0 2 1 0 \n", "56 0 2 1 0 \n", "57 0 2 1 0 \n", "58 0 3 0 0 \n", "59 0 3 0 0 \n", "60 0 3 0 0 \n", "61 0 3 0 0 \n", "62 0 3 0 0 \n", "63 0 3 0 0 \n", "64 0 3 0 0 \n", "65 0 3 0 0 \n", "66 0 3 1 0 \n", "67 0 3 1 0 \n", "68 0 3 1 0 \n", "69 1 2 1 0 \n", "70 1 2 1 0 \n", "71 1 2 1 0 \n", "72 0 2 1 0 \n", "73 0 2 1 0 \n", "74 0 2 1 0 \n", "75 0 2 1 0 \n", "76 0 2 1 0 \n", "77 0 2 1 0 \n", "78 0 2 1 0 \n", "79 0 2 1 0 \n", "80 0 2 1 0 \n", "81 0 3 0 0 \n", "82 0 3 0 0 \n", "83 0 3 0 0 \n", "84 0 2 1 0 \n", "85 0 2 1 0 \n", "86 0 2 1 0 \n", "87 0 2 1 0 \n", "88 0 2 1 0 \n", "89 0 2 1 0 \n", "90 0 2 1 0 \n", "91 0 1 1 0 \n", "92 0 1 1 0 \n", "93 0 1 1 0 \n", "94 0 1 1 0 \n", "95 0 2 1 0 \n", "96 0 2 1 0 \n", "97 0 2 1 0 \n", "98 0 2 1 0 \n", "99 0 2 1 0 \n", "100 0 2 1 0 \n", "Rows: 1-100 | Columns: 15" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vdf" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/opt/venv/lib/python3.10/site-packages/vertica_python/vertica/connection.py:659: UserWarning: [INFO] Cannot commit; no transaction in progress\n", " warnings.warn(notice)\n" ] }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from verticapy.learn.cluster import DBSCAN\n", "\n", "vp.drop(\"dbscan_heart\")\n", "model = DBSCAN(\"dbscan_heart\", eps = 20, min_samples = 10)\n", "model.fit(\"public.heart\", [\"thalach\", \"chol\"])\n", "model.plot()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vdf_tmp = model.predict()\n", "vdf_tmp[\"outliers_dbscan\"] = \"(dbscan_cluster = -1)::int\"\n", "vdf_tmp.scatter([\"thalach\", \"chol\"], by = \"outliers_dbscan\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "While DBSCAN identifies outliers when computing the clusters, LOF computes an outlier score. Generally, a LOF Score greater than 1.5 indicates an outlier." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from verticapy.learn.neighbors import LocalOutlierFactor\n", "\n", "vp.drop(\"lof_heart\")\n", "model = LocalOutlierFactor(\"lof_heart\")\n", "model.fit(\"heart\", [\"thalach\", \"chol\",])\n", "model.plot()" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lof_heart = model.predict()\n", "lof_heart[\"outliers\"] = \"(CASE WHEN lof_score > 1.5 THEN 1 ELSE 0 END)\"\n", "lof_heart.scatter([\"thalach\", \"chol\"], by = \"outliers\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We have many other techniques like the k-means clustering for finding outliers, but the most important method is using the Z-Score. After identifying outliers, we just have to decide how to impute the missing values. We'll focus on missing values in the next lesson." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }