Expand Menu
-
Home / Workshop / Full Stack / Linear Regression / Index

VerticaPy
Python API for Vertica Data Science at Scale
Linear Regression¶
Linear regression is one of the most popular regression algorithms and produces good predictions for well-prepared data. Its optimization function computes coefficients to express a response column as a linear relationship of its predictors.
You must verify the Gauss-Markov assumptions when using linear regression algorithms:
- Linearity : the parameters we are estimating using the OLS method must be linear.
- Non-Collinearity : the regressors being calculated aren’t perfectly correlated with each other.
- Exogeneity : the regressors aren’t correlated with the error term.
- Homoscedasticity : no matter what the values of our regressors might be, the error of the variance is constant.
To create a good linear regression model, it's important to:
- Impute missing values
- Encode categorical features (linear regression only accepts numerical variables)
- Compute the correlation matrix to retrieve highly-correlated predictors
- Decompose the data (optional)
- Normalize the data (optional, but recommended)
Example without decomposition¶
Let's use the 'africa_education' dataset to compute a linear regression model of students' performance in school.
In [46]:
import verticapy as vp
africa = vp.read_csv("data/africa_education.csv")
africa = africa.select(["(zralocp + zmalocp) / 2 AS student_score",
"(zraloct + zmaloct) / 2 AS teacher_score",
"XNUMYRS AS teacher_year_teaching",
"numstu AS number_students_school",
"PENGLISH AS english_at_home",
"PTRAVEL AS travel_distance",
"PTRAVEL2 AS means_of_travel",
"PMOTHER AS m_education",
"PFATHER AS f_education",
"PLIGHT AS source_of_lighting",
"PABSENT AS days_absent",
"PREPEAT AS repeated_grades",
"zpsit AS sitting_place",
"PAGE AS age",
"zpses AS socio_eco_statut",
"country_long AS country"])
display(africa)
123 student_scoreFloat | 123 teacher_scoreFloat | 123 teacher_year_teachingNumeric(7,3) | 123 number_students_schoolInteger | Abc english_at_homeVarchar(32) | Abc travel_distanceVarchar(22) | Abc means_of_travelVarchar(26) | Abc Varchar(68) | Abc Varchar(68) | Abc source_of_lightingVarchar(24) | 123 days_absentInteger | Abc repeated_gradesVarchar(20) | Abc sitting_placeVarchar(54) | 123 ageInteger | 123 socio_eco_statutNumeric(7,3) | Abc countryVarchar(24) | |
1 | 681.138508424325 | [null] | 26.0 | 24 | ALL THE TIME | >0.5-1KM | CAR | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 15.0 | South Africa | ||
2 | 425.993367323877 | 537.289572911762 | 10.0 | 23 | SOMETIMES | >4.5-5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 14 | 5.0 | Namibia | ||
3 | 534.329515370892 | 537.289572911762 | 10.0 | 23 | SOMETIMES | >0.5-1KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 13 | 7.0 | Namibia | ||
4 | 536.690743411639 | 537.289572911762 | 10.0 | 23 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 8.0 | Namibia | ||
5 | 569.392927563969 | 537.289572911762 | 10.0 | 23 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 13 | 9.0 | Namibia | ||
6 | 542.037992351316 | 537.289572911762 | 10.0 | 23 | MOST OF THE TIME | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 11.0 | Namibia | ||
7 | 573.771789981159 | 537.289572911762 | 10.0 | 23 | MOST OF THE TIME | >1.5-2KM | CAR | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 11.0 | Namibia | ||
8 | 589.279157441376 | 537.289572911762 | 10.0 | 23 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 12.0 | Namibia | ||
9 | 496.740841813343 | 537.289572911762 | 10.0 | 23 | SOMETIMES | >2.5-3KM | CAR | ELECTRIC | 0 | NEVER | I have my own sitting place | 14 | 12.0 | Namibia | ||
10 | 535.805274812767 | 549.40293941491 | 1.0 | 22 | ALL THE TIME | >1-1.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 13 | 10.0 | Mozambique | ||
11 | 420.509155247374 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | >5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 13 | 3.0 | Zanzibar | ||
12 | 504.298378016207 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 13 | 3.0 | Zanzibar | ||
13 | 450.768711398358 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 14 | 4.0 | Zanzibar | ||
14 | 441.927245450751 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | >1-1.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 14 | 5.0 | Zanzibar | ||
15 | 469.987298066451 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 14 | 5.0 | Zanzibar | ||
16 | 445.259642792635 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | >4KM-4.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 13 | 6.0 | Zanzibar | ||
17 | 551.071920864503 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 13 | 6.0 | Zanzibar | ||
18 | 473.825234391116 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 12 | 7.0 | Zanzibar | ||
19 | 544.095820873929 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 8.0 | Zanzibar | ||
20 | 564.874824604798 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 14 | 8.0 | Zanzibar | ||
21 | 386.940445160128 | 569.993686161283 | 11.0 | 23 | SOMETIMES | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 14 | 3.0 | Mozambique | ||
22 | 458.501728934408 | 593.122754483926 | 3.0 | 23 | SOMETIMES | >2-2.5KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 14 | 5.0 | Mozambique | ||
23 | 460.360232184133 | 569.993686161283 | 11.0 | 23 | ALL THE TIME | >1.5-2KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 15 | 7.0 | Mozambique | ||
24 | 457.035551231015 | 593.122754483926 | 3.0 | 23 | SOMETIMES | >1.5-2KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 7.0 | Mozambique | ||
25 | 504.282977573042 | 593.122754483926 | 3.0 | 23 | ALL THE TIME | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 8.0 | Mozambique | ||
26 | 475.872247772718 | 569.993686161283 | 11.0 | 23 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 16 | 10.0 | Mozambique | ||
27 | 503.899980617811 | 584.886538196464 | 3.0 | 7 | ALL THE TIME | >0.5-1KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 15 | 2.0 | Mozambique | ||
28 | 480.484906064866 | 584.886538196464 | 3.0 | 7 | ALL THE TIME | >2.5-3KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 15 | 3.0 | Mozambique | ||
29 | 516.744491629541 | 641.876995222798 | 20.0 | 25 | SOMETIMES | >0.5-1KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
30 | 394.65259093503 | 641.876995222798 | 20.0 | 25 | NEVER | >5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
31 | 506.128325450387 | 641.876995222798 | 20.0 | 25 | SOMETIMES | UP TO 0.5KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 11 | 7.0 | Namibia | ||
32 | 440.401203829463 | 641.876995222798 | 20.0 | 25 | SOMETIMES | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 12 | 8.0 | Namibia | ||
33 | 453.010211603314 | 641.876995222798 | 20.0 | 25 | SOMETIMES | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 11 | 8.0 | Namibia | ||
34 | 498.219068060275 | 641.876995222798 | 20.0 | 25 | SOMETIMES | >1-1.5KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 11 | 11.0 | Namibia | ||
35 | 359.385615553614 | 610.956816556605 | 30.0 | 16 | ALL THE TIME | UP TO 0.5KM | WALK | GAS | 0 | NEVER | I have my own sitting place | 14 | [null] | Mozambique | ||
36 | 525.275821925396 | 610.956816556605 | 30.0 | 16 | ALL THE TIME | UP TO 0.5KM | WALK | GAS | 0 | NEVER | I have my own sitting place | 17 | [null] | Mozambique | ||
37 | 457.035551231015 | 610.956816556605 | 30.0 | 16 | ALL THE TIME | UP TO 0.5KM | WALK | GAS | 0 | NEVER | I have my own sitting place | 12 | [null] | Mozambique | ||
38 | 509.337990443557 | 610.956816556605 | 30.0 | 16 | ALL THE TIME | UP TO 0.5KM | WALK | GAS | 0 | NEVER | I have my own sitting place | 12 | [null] | Mozambique | ||
39 | 434.970625831997 | 610.956816556605 | 30.0 | 16 | SOMETIMES | UP TO 0.5KM | WALK | GAS | 0 | NEVER | I have my own sitting place | 17 | [null] | Mozambique | ||
40 | [null] | 641.477648982361 | 2.0 | 16 | SOMETIMES | >5KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 17 | 4.0 | Mozambique | ||
41 | 428.988211440412 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >5KM | WALK | FIRE | 0 | NEVER | I have my own sitting place | 12 | 2.0 | Namibia | ||
42 | 532.255684949772 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >0.5-1KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 14 | 2.0 | Namibia | ||
43 | 392.664714363554 | 676.11173539576 | 8.0 | 20 | ALL THE TIME | >3-3.5KM | WALK | CANDLE | 0 | NEVER | No place/share | 13 | 4.0 | Zambia | ||
44 | 415.95588691273 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >5KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 13 | 4.0 | Namibia | ||
45 | 479.525184773189 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >0.5-1KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 12 | 4.0 | Namibia | ||
46 | 410.328608434232 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >1-1.5KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 12 | 4.0 | Namibia | ||
47 | 425.993367323877 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >3-3.5KM | WALK | FIRE | 0 | NEVER | I have my own sitting place | 12 | 4.0 | Namibia | ||
48 | 409.430742983857 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >1.5-2KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 5.0 | Namibia | ||
49 | 460.605886281468 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
50 | 560.188989035577 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
51 | 378.486087509513 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 14 | 6.0 | Namibia | ||
52 | 516.831361986859 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 6.0 | Namibia | ||
53 | 557.155342162198 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
54 | 631.282997954796 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
55 | 501.677328837489 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >2.5-3KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 18 | 6.0 | Namibia | ||
56 | 466.248533790025 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
57 | 594.667991916549 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
58 | 619.020904331767 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
59 | 422.517903579997 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
60 | 514.338009258817 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 6.0 | Namibia | ||
61 | 512.766872215611 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 7.0 | Namibia | ||
62 | 456.266205420931 | 676.11173539576 | 8.0 | 20 | SOMETIMES | UP TO 0.5KM | WALK | CANDLE | 0 | NEVER | No place/share | 11 | 7.0 | Zambia | ||
63 | 380.388263378723 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >5KM | CAR | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 13 | 8.0 | Namibia | ||
64 | 421.344721387902 | 676.11173539576 | 8.0 | 20 | SOMETIMES | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | No place/share | 11 | 8.0 | Zambia | ||
65 | 486.501284763764 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 13 | 9.0 | Namibia | ||
66 | 453.357504553945 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 9.0 | Namibia | ||
67 | 498.219068060275 | 638.008383664231 | 6.0 | 24 | SOMETIMES | >0.5-1KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 9.0 | Namibia | ||
68 | 514.807687171462 | 638.008383664231 | 6.0 | 24 | NEVER | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 10.0 | Namibia | ||
69 | 565.065579807459 | 683.217366694882 | 12.0 | 21 | MOST OF THE TIME | >0.5-1KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 13.0 | South Africa | ||
70 | 626.714865963592 | 638.008383664231 | 8.0 | 21 | ALL THE TIME | UP TO 0.5KM | CAR | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 14.0 | South Africa | ||
71 | 457.134154158281 | 683.217366694882 | 12.0 | 21 | SOMETIMES | >5KM | BUS/TRUCK/VAN | ELECTRIC | 0 | NEVER | I have my own sitting place | 13 | 14.0 | South Africa | ||
72 | 510.499756960558 | 634.222848016075 | 9.0 | 20 | MOST OF THE TIME | >1-1.5KM | WALK | FIRE | 0 | NEVER | I have my own sitting place | 16 | 2.0 | Uganda | ||
73 | 494.235377715545 | 634.222848016075 | 9.0 | 20 | SOMETIMES | >1.5-2KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 17 | 2.0 | Uganda | ||
74 | 365.266644239282 | 617.7225229859 | 1.0 | 13 | NEVER | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 13 | 2.0 | Zambia | ||
75 | 541.173490674172 | 645.715848976398 | 6.0 | 49 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 3.0 | Namibia | ||
76 | 455.36049792936 | 645.715848976398 | 8.0 | 20 | SOMETIMES | >5KM | WALK | PARAFFIN/OIL | 0 | NEVER | No place/share | 15 | 3.0 | Malawi | ||
77 | 422.517903579997 | 645.715848976398 | 8.0 | 20 | NEVER | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | No place/share | 12 | 5.0 | Malawi | ||
78 | 398.31103066659 | 651.629260258888 | 1.0 | 21 | MOST OF THE TIME | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 14 | 5.0 | Malawi | ||
79 | 395.120560563494 | 639.897815295239 | 1.0 | 22 | SOMETIMES | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
80 | 503.716941803917 | 645.715848976398 | 6.0 | 49 | SOMETIMES | >5KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
81 | 427.462485792718 | 623.159046261738 | 8.0 | 18 | SOMETIMES | UP TO 0.5KM | WALK | CANDLE | 0 | NEVER | No place/share | 13 | 6.0 | Zambia | ||
82 | 600.91272658553 | 645.715848976398 | 6.0 | 49 | SOMETIMES | >2-2.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 7.0 | Namibia | ||
83 | 565.065579807459 | 645.715848976398 | 6.0 | 49 | SOMETIMES | >5KM | CAR | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 7.0 | Namibia | ||
84 | 540.204883338675 | 645.715848976398 | 6.0 | 49 | MOST OF THE TIME | >2.5-3KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 7.0 | Namibia | ||
85 | 558.474690429794 | 645.715848976398 | 6.0 | 49 | SOMETIMES | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 12 | 7.0 | Namibia | ||
86 | 575.637662232951 | 645.715848976398 | 6.0 | 49 | NEVER | >0.5-1KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 13 | 7.0 | Namibia | ||
87 | 439.680053114802 | 645.715848976398 | 8.0 | 20 | SOMETIMES | UP TO 0.5KM | WALK | CANDLE | 0 | NEVER | No place/share | 11 | 7.0 | Malawi | ||
88 | 353.773643278958 | 617.7225229859 | 1.0 | 13 | SOMETIMES | >0.5-1KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 13 | 7.0 | Zambia | ||
89 | 488.376137298698 | 645.715848976398 | 6.0 | 49 | SOMETIMES | >1-1.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 8.0 | Namibia | ||
90 | 480.908033418792 | 645.715848976398 | 6.0 | 49 | MOST OF THE TIME | >1-1.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 8.0 | Namibia | ||
91 | 535.286927844308 | 645.715848976398 | 6.0 | 49 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 8.0 | Namibia | ||
92 | 408.529715955022 | 651.629260258888 | 1.0 | 21 | SOMETIMES | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 15 | 9.0 | Malawi | ||
93 | 385.53865385057 | 623.159046261738 | 8.0 | 18 | SOMETIMES | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 12 | 9.0 | Zambia | ||
94 | 528.310827853733 | 645.715848976398 | 6.0 | 49 | SOMETIMES | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 12 | 10.0 | Namibia | ||
95 | 548.67653911654 | 645.715848976398 | 6.0 | 49 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 10.0 | Namibia | ||
96 | 588.423257247566 | 645.715848976398 | 6.0 | 49 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 13 | 10.0 | Namibia | ||
97 | 583.034422772394 | 645.715848976398 | 6.0 | 49 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 13 | 10.0 | Namibia | ||
98 | 455.36049792936 | 645.715848976398 | 8.0 | 20 | NEVER | >3.5-4KM | WALK | ELECTRIC | 0 | NEVER | No place/share | 14 | 10.0 | Malawi | ||
99 | 459.986311593889 | 645.715848976398 | 6.0 | 49 | SOMETIMES | >2.5-3KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 13 | 11.0 | Namibia | ||
100 | 594.667991916549 | 645.715848976398 | 6.0 | 49 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 12.0 | Namibia |
Rows: 1-100 | Columns: 16
First, let's look for missing values.
In [2]:
africa.count_percent()
Out[2]:
count | percent | |
"number_students_school" | 60890.0 | 100.0 |
"english_at_home" | 60890.0 | 100.0 |
"travel_distance" | 60890.0 | 100.0 |
"means_of_travel" | 60890.0 | 100.0 |
"m_education" | 60890.0 | 100.0 |
"source_of_lighting" | 60890.0 | 100.0 |
"days_absent" | 60890.0 | 100.0 |
"repeated_grades" | 60890.0 | 100.0 |
"sitting_place" | 60890.0 | 100.0 |
"age" | 60890.0 | 100.0 |
"country" | 60890.0 | 100.0 |
"socio_eco_statut" | 60832.0 | 99.905 |
"student_score" | 60809.0 | 99.867 |
"teacher_year_teaching" | 60708.0 | 99.701 |
"f_education" | 60599.0 | 99.522 |
"teacher_score" | 52122.0 | 85.6 |
Rows: 1-16 | Columns: 3
We'll simply drop the missing values to avoid adding bias to the data.
In [47]:
africa.dropna()
8988 elements were filtered.
Out[47]:
123 student_scoreFloat | 123 teacher_scoreFloat | 123 teacher_year_teachingNumeric(7,3) | 123 number_students_schoolInteger | Abc english_at_homeVarchar(32) | Abc travel_distanceVarchar(22) | Abc means_of_travelVarchar(26) | Abc Varchar(68) | Abc Varchar(68) | Abc source_of_lightingVarchar(24) | 123 days_absentInteger | Abc repeated_gradesVarchar(20) | Abc sitting_placeVarchar(54) | 123 ageInteger | 123 socio_eco_statutNumeric(7,3) | Abc countryVarchar(24) | |
1 | 425.993367323877 | 537.289572911762 | 10.0 | 23 | SOMETIMES | >4.5-5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 14 | 5.0 | Namibia | ||
2 | 534.329515370892 | 537.289572911762 | 10.0 | 23 | SOMETIMES | >0.5-1KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 13 | 7.0 | Namibia | ||
3 | 536.690743411639 | 537.289572911762 | 10.0 | 23 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 8.0 | Namibia | ||
4 | 569.392927563969 | 537.289572911762 | 10.0 | 23 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 13 | 9.0 | Namibia | ||
5 | 542.037992351316 | 537.289572911762 | 10.0 | 23 | MOST OF THE TIME | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 11.0 | Namibia | ||
6 | 573.771789981159 | 537.289572911762 | 10.0 | 23 | MOST OF THE TIME | >1.5-2KM | CAR | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 11.0 | Namibia | ||
7 | 589.279157441376 | 537.289572911762 | 10.0 | 23 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 12.0 | Namibia | ||
8 | 496.740841813343 | 537.289572911762 | 10.0 | 23 | SOMETIMES | >2.5-3KM | CAR | ELECTRIC | 0 | NEVER | I have my own sitting place | 14 | 12.0 | Namibia | ||
9 | 535.805274812767 | 549.40293941491 | 1.0 | 22 | ALL THE TIME | >1-1.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 13 | 10.0 | Mozambique | ||
10 | 420.509155247374 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | >5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 13 | 3.0 | Zanzibar | ||
11 | 504.298378016207 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 13 | 3.0 | Zanzibar | ||
12 | 450.768711398358 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 14 | 4.0 | Zanzibar | ||
13 | 441.927245450751 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | >1-1.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 14 | 5.0 | Zanzibar | ||
14 | 469.987298066451 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 14 | 5.0 | Zanzibar | ||
15 | 445.259642792635 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | >4KM-4.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 13 | 6.0 | Zanzibar | ||
16 | 551.071920864503 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 13 | 6.0 | Zanzibar | ||
17 | 473.825234391116 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 12 | 7.0 | Zanzibar | ||
18 | 544.095820873929 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 8.0 | Zanzibar | ||
19 | 564.874824604798 | 624.124902231691 | 1.0 | 25 | ALL THE TIME | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 14 | 8.0 | Zanzibar | ||
20 | 386.940445160128 | 569.993686161283 | 11.0 | 23 | SOMETIMES | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 14 | 3.0 | Mozambique | ||
21 | 458.501728934408 | 593.122754483926 | 3.0 | 23 | SOMETIMES | >2-2.5KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 14 | 5.0 | Mozambique | ||
22 | 460.360232184133 | 569.993686161283 | 11.0 | 23 | ALL THE TIME | >1.5-2KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 15 | 7.0 | Mozambique | ||
23 | 457.035551231015 | 593.122754483926 | 3.0 | 23 | SOMETIMES | >1.5-2KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 7.0 | Mozambique | ||
24 | 504.282977573042 | 593.122754483926 | 3.0 | 23 | ALL THE TIME | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 8.0 | Mozambique | ||
25 | 475.872247772718 | 569.993686161283 | 11.0 | 23 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 16 | 10.0 | Mozambique | ||
26 | 503.899980617811 | 584.886538196464 | 3.0 | 7 | ALL THE TIME | >0.5-1KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 15 | 2.0 | Mozambique | ||
27 | 480.484906064866 | 584.886538196464 | 3.0 | 7 | ALL THE TIME | >2.5-3KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 15 | 3.0 | Mozambique | ||
28 | 516.744491629541 | 641.876995222798 | 20.0 | 25 | SOMETIMES | >0.5-1KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
29 | 394.65259093503 | 641.876995222798 | 20.0 | 25 | NEVER | >5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
30 | 506.128325450387 | 641.876995222798 | 20.0 | 25 | SOMETIMES | UP TO 0.5KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 11 | 7.0 | Namibia | ||
31 | 440.401203829463 | 641.876995222798 | 20.0 | 25 | SOMETIMES | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 12 | 8.0 | Namibia | ||
32 | 453.010211603314 | 641.876995222798 | 20.0 | 25 | SOMETIMES | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 11 | 8.0 | Namibia | ||
33 | 498.219068060275 | 641.876995222798 | 20.0 | 25 | SOMETIMES | >1-1.5KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 11 | 11.0 | Namibia | ||
34 | 428.988211440412 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >5KM | WALK | FIRE | 0 | NEVER | I have my own sitting place | 12 | 2.0 | Namibia | ||
35 | 532.255684949772 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >0.5-1KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 14 | 2.0 | Namibia | ||
36 | 392.664714363554 | 676.11173539576 | 8.0 | 20 | ALL THE TIME | >3-3.5KM | WALK | CANDLE | 0 | NEVER | No place/share | 13 | 4.0 | Zambia | ||
37 | 415.95588691273 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >5KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 13 | 4.0 | Namibia | ||
38 | 479.525184773189 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >0.5-1KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 12 | 4.0 | Namibia | ||
39 | 410.328608434232 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >1-1.5KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 12 | 4.0 | Namibia | ||
40 | 425.993367323877 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >3-3.5KM | WALK | FIRE | 0 | NEVER | I have my own sitting place | 12 | 4.0 | Namibia | ||
41 | 409.430742983857 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >1.5-2KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 5.0 | Namibia | ||
42 | 460.605886281468 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
43 | 560.188989035577 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
44 | 378.486087509513 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 14 | 6.0 | Namibia | ||
45 | 516.831361986859 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 6.0 | Namibia | ||
46 | 557.155342162198 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
47 | 631.282997954796 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
48 | 501.677328837489 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >2.5-3KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 18 | 6.0 | Namibia | ||
49 | 466.248533790025 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
50 | 594.667991916549 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
51 | 619.020904331767 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
52 | 422.517903579997 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
53 | 514.338009258817 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 6.0 | Namibia | ||
54 | 512.766872215611 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 7.0 | Namibia | ||
55 | 456.266205420931 | 676.11173539576 | 8.0 | 20 | SOMETIMES | UP TO 0.5KM | WALK | CANDLE | 0 | NEVER | No place/share | 11 | 7.0 | Zambia | ||
56 | 380.388263378723 | 656.225505518354 | 8.0 | 23 | SOMETIMES | >5KM | CAR | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 13 | 8.0 | Namibia | ||
57 | 421.344721387902 | 676.11173539576 | 8.0 | 20 | SOMETIMES | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | No place/share | 11 | 8.0 | Zambia | ||
58 | 486.501284763764 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 13 | 9.0 | Namibia | ||
59 | 453.357504553945 | 638.008383664231 | 6.0 | 24 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 9.0 | Namibia | ||
60 | 498.219068060275 | 638.008383664231 | 6.0 | 24 | SOMETIMES | >0.5-1KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 9.0 | Namibia | ||
61 | 514.807687171462 | 638.008383664231 | 6.0 | 24 | NEVER | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 10.0 | Namibia | ||
62 | 565.065579807459 | 683.217366694882 | 12.0 | 21 | MOST OF THE TIME | >0.5-1KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 13.0 | South Africa | ||
63 | 626.714865963592 | 638.008383664231 | 8.0 | 21 | ALL THE TIME | UP TO 0.5KM | CAR | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 14.0 | South Africa | ||
64 | 457.134154158281 | 683.217366694882 | 12.0 | 21 | SOMETIMES | >5KM | BUS/TRUCK/VAN | ELECTRIC | 0 | NEVER | I have my own sitting place | 13 | 14.0 | South Africa | ||
65 | 510.499756960558 | 634.222848016075 | 9.0 | 20 | MOST OF THE TIME | >1-1.5KM | WALK | FIRE | 0 | NEVER | I have my own sitting place | 16 | 2.0 | Uganda | ||
66 | 494.235377715545 | 634.222848016075 | 9.0 | 20 | SOMETIMES | >1.5-2KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 17 | 2.0 | Uganda | ||
67 | 365.266644239282 | 617.7225229859 | 1.0 | 13 | NEVER | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 13 | 2.0 | Zambia | ||
68 | 541.173490674172 | 645.715848976398 | 6.0 | 49 | SOMETIMES | UP TO 0.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 3.0 | Namibia | ||
69 | 455.36049792936 | 645.715848976398 | 8.0 | 20 | SOMETIMES | >5KM | WALK | PARAFFIN/OIL | 0 | NEVER | No place/share | 15 | 3.0 | Malawi | ||
70 | 422.517903579997 | 645.715848976398 | 8.0 | 20 | NEVER | >0.5-1KM | WALK | PARAFFIN/OIL | 0 | NEVER | No place/share | 12 | 5.0 | Malawi | ||
71 | 398.31103066659 | 651.629260258888 | 1.0 | 21 | MOST OF THE TIME | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 14 | 5.0 | Malawi | ||
72 | 395.120560563494 | 639.897815295239 | 1.0 | 22 | SOMETIMES | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
73 | 503.716941803917 | 645.715848976398 | 6.0 | 49 | SOMETIMES | >5KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 12 | 6.0 | Namibia | ||
74 | 427.462485792718 | 623.159046261738 | 8.0 | 18 | SOMETIMES | UP TO 0.5KM | WALK | CANDLE | 0 | NEVER | No place/share | 13 | 6.0 | Zambia | ||
75 | 600.91272658553 | 645.715848976398 | 6.0 | 49 | SOMETIMES | >2-2.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 7.0 | Namibia | ||
76 | 565.065579807459 | 645.715848976398 | 6.0 | 49 | SOMETIMES | >5KM | CAR | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 7.0 | Namibia | ||
77 | 540.204883338675 | 645.715848976398 | 6.0 | 49 | MOST OF THE TIME | >2.5-3KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 7.0 | Namibia | ||
78 | 558.474690429794 | 645.715848976398 | 6.0 | 49 | SOMETIMES | UP TO 0.5KM | WALK | PARAFFIN/OIL | 0 | NEVER | I have my own sitting place | 12 | 7.0 | Namibia | ||
79 | 575.637662232951 | 645.715848976398 | 6.0 | 49 | NEVER | >0.5-1KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 13 | 7.0 | Namibia | ||
80 | 439.680053114802 | 645.715848976398 | 8.0 | 20 | SOMETIMES | UP TO 0.5KM | WALK | CANDLE | 0 | NEVER | No place/share | 11 | 7.0 | Malawi | ||
81 | 353.773643278958 | 617.7225229859 | 1.0 | 13 | SOMETIMES | >0.5-1KM | WALK | CANDLE | 0 | NEVER | I have my own sitting place | 13 | 7.0 | Zambia | ||
82 | 488.376137298698 | 645.715848976398 | 6.0 | 49 | SOMETIMES | >1-1.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 11 | 8.0 | Namibia | ||
83 | 480.908033418792 | 645.715848976398 | 6.0 | 49 | MOST OF THE TIME | >1-1.5KM | WALK | ELECTRIC | 0 | NEVER | I have my own sitting place | 12 | 8.0 | Namibia | ||
84 | 535.286927844308 | 645.715848976398 | 6.0 | 49 | SOMETIMES | UP TO 0.5KM | WALK |