Feedback

×

• Feedback
• Was this page helpful?*
  • Yes
  • No
• Do you have any additional feedback?
  • Yes
  • No
• Additional Feedback

Linear Regression

Linear regression is one of the most popular regression algorithms and produces good predictions for well-prepared data. Its optimization function computes coefficients to express a response column as a linear relationship of its predictors.

You must verify the Gauss-Markov assumptions when using linear regression algorithms:

• Linearity: the parameters we are estimating using the OLS method must be linear.

• Non-Collinearity: the regressors being calculated aren’t perfectly correlated with each other.

• Exogeneity: the regressors aren’t correlated with the error term.

• Homoscedasticity: no matter what the values of our regressors might be, the error of the variance is constant.

To create a good linear regression model, it’s important to:

• Impute missing values.

• Encode categorical features (linear regression only accepts numerical variables).

• Compute the correlation matrix to retrieve highly-correlated predictors.

• Decompose the data (optional).

• Normalize the data (optional, but recommended).

Example without decomposition

Let’s use the africa_education dataset to compute a linear regression model of students’ performance in school.

from verticapy.datasets import load_africa_education

africa = load_africa_education()
africa = africa.select(
    [
        "(zralocp + zmalocp) / 2 AS student_score",
        "zraloct AS teacher_score",
        "XNUMYRS AS teacher_year_teaching",
        "numstu AS number_students_school",
        "PENGLISH AS english_at_home",
        "PTRAVEL AS travel_distance",
        "PTRAVEL2 AS means_of_travel",
        "PMOTHER AS m_education",
        "PFATHER AS f_education",
        "PLIGHT AS source_of_lighting",
        "PABSENT AS days_absent",
        "PREPEAT AS repeated_grades",
        "zpsit AS sitting_place",
        "PAGE AS age",
        "zpses AS socio_eco_statut",
        "country_long AS country",
    ],
)
africa.head(100)

	123 student_score Float(22) 99%	...	123 socio_eco_statut Numeric(9) 99%	Abc country Varchar(24) 100%
1	357.1760554	...	[null]	Mozambique
2	548.0440936	...	1.0	Mozambique
3	427.97953655	...	1.0	Mozambique
4	431.55012475	...	1.0	Malawi
5	422.5179036	...	1.0	Zambia
6	367.82150545	...	1.0	Mozambique
7	390.59366675	...	1.0	Malawi
8	421.3447214	...	1.0	Zambia
9	430.6369854	...	1.0	Tanzania
10	536.5504598	...	2.0	Tanzania
11	442.96103005	...	2.0	Malawi
12	532.25568495	...	2.0	Uganda
13	439.3294711	...	2.0	Mozambique
14	415.27831365	...	2.0	Tanzania
15	423.0274273	...	2.0	Namibia
16	610.003642	...	2.0	Tanzania
17	443.1010291	...	2.0	Tanzania
18	501.08052265	...	2.0	Namibia
19	448.05639505	...	2.0	Mozambique
20	589.0925764	...	2.0	Tanzania
21	525.2758219	...	2.0	Mozambique
22	392.00570405	...	2.0	Mozambique
23	321.1299002	...	2.0	Mozambique
24	466.7373761	...	2.0	Malawi
25	494.23537775	...	2.0	Uganda
26	604.09734565	...	2.0	Tanzania
27	519.2366424	...	2.0	Namibia
28	502.06867415	...	2.0	Uganda
29	378.4860875	...	2.0	Zambia
30	384.113366	...	2.0	Zambia
31	531.6862021	...	2.0	Tanzania
32	387.62298345	...	2.0	Tanzania
33	465.58674165	...	2.0	Malawi
34	428.54585685	...	2.0	Zimbabwe
35	459.917909	...	3.0	Tanzania
36	448.1354862	...	3.0	Mozambique
37	543.3015243	...	3.0	Tanzania
38	468.2764465	...	3.0	Mozambique
39	503.88363135	...	3.0	Tanzania
40	561.2185357	...	3.0	Mozambique
41	485.2000255	...	3.0	Kenya
42	552.0016312	...	3.0	Tanzania
43	519.69556795	...	3.0	Tanzania
44	364.77445005	...	3.0	Mozambique
45	444.7735204	...	3.0	Malawi
46	417.89208995	...	3.0	Namibia
47	422.38353385	...	3.0	Mozambique
48	464.89787835	...	3.0	Uganda
49	418.54468005	...	3.0	Uganda
50	528.9471953	...	3.0	Tanzania
51	610.5671158	...	3.0	Tanzania
52	445.80907615	...	3.0	Malawi
53	535.07574435	...	3.0	Tanzania
54	501.44606355	...	3.0	Mozambique
55	448.38439795	...	3.0	Namibia
56	518.9185695	...	3.0	Mozambique
57	389.59055735	...	3.0	Tanzania
58	699.58499885	...	3.0	Tanzania
59	433.75709155	...	3.0	Malawi
60	540.68676775	...	3.0	Uganda
61	473.4822422	...	3.0	Uganda
62	641.08794825	...	3.0	Tanzania
63	592.3426887	...	3.0	Tanzania
64	417.34281455	...	3.0	Malawi
65	564.5413827	...	3.0	Tanzania
66	576.01604155	...	3.0	Tanzania
67	424.1239538	...	3.0	Zambia
68	595.9993573	...	3.0	Kenya
69	530.22097325	...	3.0	Uganda
70	517.0661695	...	3.0	Zimbabwe
71	416.3602296	...	4.0	Namibia
72	573.10215485	...	4.0	Tanzania
73	436.4143824	...	4.0	Malawi
74	499.7551349	...	4.0	Malawi
75	473.6045948	...	4.0	Tanzania
76	383.93699955	...	4.0	Tanzania
77	595.70664635	...	4.0	Tanzania
78	441.1832625	...	4.0	Uganda
79	565.39124245	...	4.0	Tanzania
80	536.68808725	...	4.0	Mozambique
81	439.3294711	...	4.0	Mozambique
82	496.7408418	...	4.0	Namibia
83	447.4437773	...	4.0	Namibia
84	667.8671708	...	4.0	Tanzania
85	697.6814944	...	4.0	Tanzania
86	592.3426887	...	4.0	Tanzania
87	442.48809535	...	4.0	Mozambique
88	451.69440555	...	4.0	Mozambique
89	429.86792415	...	4.0	Mozambique
90	577.2088944	...	4.0	Kenya
91	431.55012475	...	4.0	Namibia
92	471.31550095	...	4.0	Uganda
93	440.6723473	...	4.0	Uganda
94	399.7076038	...	4.0	Namibia
95	713.3214605	...	4.0	Tanzania
96	471.31550095	...	4.0	Kenya
97	431.55012475	...	4.0	Kenya
98	[null]	...	4.0	Mozambique
99	423.00807355	...	4.0	Tanzania
100	584.8568118	...	4.0	Tanzania

First, let’s look for missing values.

africa.count_percent()

	...	count	percent
"number_students_school"	...	19290.0	100.0
"english_at_home"	...	19290.0	100.0
"travel_distance"	...	19290.0	100.0
"means_of_travel"	...	19290.0	100.0
"m_education"	...	19290.0	100.0
"source_of_lighting"	...	19290.0	100.0
"days_absent"	...	19290.0	100.0
"repeated_grades"	...	19290.0	100.0
"sitting_place"	...	19290.0	100.0
"age"	...	19290.0	100.0
"country"	...	19290.0	100.0
"socio_eco_statut"	...	19271.0	99.902
"student_score"	...	19263.0	99.86
"teacher_year_teaching"	...	19233.0	99.705
"f_education"	...	19188.0	99.471
"teacher_score"	...	17214.0	89.238

We’ll simply drop the missing values to avoid adding bias to the data.

africa.dropna()

	123 student_score Float(22) 100%	...	123 teacher_score Float(22) 100%	Abc country Varchar(24) 100%
1	657.83952525	...	753.8647201	Kenya
2	387.88641055	...	763.3071026	Mozambique
3	392.73400295	...	763.3071026	Mozambique
4	422.06612525	...	777.9952532	Mozambique
5	436.79013695	...	696.5427817	Tanzania
6	531.6862021	...	734.979955	Tanzania
7	530.3308348	...	735.456843	Mozambique
8	607.95710255	...	753.8647201	Uganda
9	492.4671919	...	590.2921338	Mozambique
10	417.34281455	...	613.9457789	Namibia
11	458.2617984	...	648.0909602	Mozambique
12	515.30886255	...	709.1326251	Mozambique
13	469.0471837	...	727.8266349	Malawi
14	469.0409539	...	735.456843	Mozambique
15	535.42243675	...	740.6072335	Kenya
16	358.63862805	...	740.6072335	Kenya
17	421.37283465	...	763.3071026	Mozambique
18	400.2268045	...	846.0948606	Zambia
19	749.46855975	...	604.3126412	Tanzania
20	730.67809685	...	721.8178461	Tanzania

We need to encode the categorical columns to dummies to retain linearity.

africa.one_hot_encode(max_cardinality = 20)

	123 student_score Float(22) 100%	...	123 country_Zambia Bool 100%	123 country_Zanzibar Bool 100%
1	657.83952525	...	0	0
2	387.88641055	...	0	0
3	392.73400295	...	0	0
4	422.06612525	...	0	0
5	436.79013695	...	0	0
6	531.6862021	...	0	0
7	530.3308348	...	0	0
8	607.95710255	...	0	0
9	492.4671919	...	0	0
10	417.34281455	...	0	0
11	458.2617984	...	0	0
12	515.30886255	...	0	0
13	469.0471837	...	0	0
14	469.0409539	...	0	0
15	535.42243675	...	0	0
16	358.63862805	...	0	0
17	421.37283465	...	0	0
18	400.2268045	...	1	0
19	749.46855975	...	0	0
20	730.67809685	...	0	0

Linear regression can only handle numerical columns, so we’ll drop the categorical columns.

africa.drop(
    columns = [
        "english_at_home",
        "travel_distance",
        "means_of_travel",
        "m_education",
        "f_education",
        "source_of_lighting",
        "repeated_grades",
        "sitting_place",
        "country",
    ],
)

	123 student_score Float(22) 100%	...	123 teacher_score Float(22) 100%	123 country_Zanzibar Bool 100%
1	657.83952525	...	753.8647201	0
2	387.88641055	...	763.3071026	0
3	392.73400295	...	763.3071026	0
4	422.06612525	...	777.9952532	0
5	436.79013695	...	696.5427817	0
6	531.6862021	...	734.979955	0
7	530.3308348	...	735.456843	0
8	607.95710255	...	753.8647201	0
9	492.4671919	...	590.2921338	0
10	417.34281455	...	613.9457789	0
11	458.2617984	...	648.0909602	0
12	515.30886255	...	709.1326251	0
13	469.0471837	...	727.8266349	0
14	469.0409539	...	735.456843	0
15	535.42243675	...	740.6072335	0
16	358.63862805	...	740.6072335	0
17	421.37283465	...	763.3071026	0
18	400.2268045	...	846.0948606	0
19	749.46855975	...	604.3126412	0
20	730.67809685	...	721.8178461	0

Let’s look at the correlation between the response column and the predictors. We’ll look to keep columns with correlations coefficients greater than 20% (the top 10 features).

x = africa.corr(focus = "student_score", show = False)
africa = africa.select(columns = x["index"][0:12])
africa.head(100)

	123 student_score Float(22) 100%	...	123 source_of_lighting_CANDLE Integer 100%	123 country_Zambia Integer 100%
1	548.0440936	...	1	0
2	427.97953655	...	0	0
3	431.55012475	...	0	0
4	422.5179036	...	1	1
5	367.82150545	...	0	0
6	390.59366675	...	0	0
7	421.3447214	...	1	1
8	536.5504598	...	0	0
9	442.96103005	...	1	0
10	532.25568495	...	1	0
11	439.3294711	...	0	0
12	415.27831365	...	0	0
13	423.0274273	...	1	0
14	610.003642	...	0	0
15	443.1010291	...	0	0
16	501.08052265	...	0	0
17	448.05639505	...	0	0
18	589.0925764	...	0	0
19	525.2758219	...	1	0
20	392.00570405	...	0	0
21	321.1299002	...	0	0
22	466.7373761	...	0	0
23	494.23537775	...	0	0
24	604.09734565	...	0	0
25	519.2366424	...	0	0
26	502.06867415	...	0	0
27	378.4860875	...	0	1
28	384.113366	...	1	1
29	531.6862021	...	0	0
30	387.62298345	...	0	0
31	465.58674165	...	1	0
32	428.54585685	...	0	0
33	459.917909	...	0	0
34	448.1354862	...	1	0
35	543.3015243	...	0	0
36	468.2764465	...	1	0
37	503.88363135	...	0	0
38	561.2185357	...	0	0
39	485.2000255	...	0	0
40	552.0016312	...	0	0
41	519.69556795	...	0	0
42	364.77445005	...	1	0
43	444.7735204	...	0	0
44	417.89208995	...	1	0
45	422.38353385	...	0	0
46	464.89787835	...	1	0
47	418.54468005	...	0	0
48	528.9471953	...	0	0
49	610.5671158	...	0	0
50	445.80907615	...	0	0
51	535.07574435	...	0	0
52	501.44606355	...	0	0
53	448.38439795	...	1	0
54	518.9185695	...	1	0
55	389.59055735	...	0	0
56	699.58499885	...	0	0
57	433.75709155	...	0	0
58	540.68676775	...	0	0
59	473.4822422	...	0	0
60	641.08794825	...	0	0
61	592.3426887	...	0	0
62	417.34281455	...	0	0
63	564.5413827	...	0	0
64	576.01604155	...	0	0
65	424.1239538	...	0	1
66	595.9993573	...	0	0
67	530.22097325	...	0	0
68	517.0661695	...	0	0
69	416.3602296	...	1	0
70	573.10215485	...	0	0
71	436.4143824	...	0	0
72	499.7551349	...	0	0
73	473.6045948	...	0	0
74	383.93699955	...	0	0
75	595.70664635	...	0	0
76	441.1832625	...	0	0
77	565.39124245	...	0	0
78	536.68808725	...	0	0
79	439.3294711	...	1	0
80	496.7408418	...	1	0
81	447.4437773	...	1	0
82	667.8671708	...	0	0
83	697.6814944	...	0	0
84	592.3426887	...	0	0
85	442.48809535	...	0	0
86	451.69440555	...	0	0
87	429.86792415	...	1	0
88	577.2088944	...	0	0
89	431.55012475	...	0	0
90	471.31550095	...	1	0
91	440.6723473	...	0	0
92	399.7076038	...	1	0
93	713.3214605	...	0	0
94	471.31550095	...	0	0
95	431.55012475	...	0	0
96	423.00807355	...	0	0
97	584.8568118	...	0	0
98	456.26620545	...	0	0
99	477.77385455	...	1	0
100	681.9499137	...	0	0

Let’s examine the correlation matrix to see if we have any independent predictors.

africa.corr()

Some of these features are highly-correlated, like socioeconomic status and having an electric lighting. We’ll drop the lighting column to avoid unexpected results while computing the linear regression.

africa["source_of_lighting_ELECTRIC"].drop()

	123 student_score Float(22) 100%	...	123 source_of_lighting_CANDLE Integer 100%	123 country_Zambia Integer 100%
1	657.83952525	...	0	0
2	387.88641055	...	0	0
3	392.73400295	...	0	0
4	422.06612525	...	0	0
5	436.79013695	...	0	0
6	531.6862021	...	0	0
7	530.3308348	...	0	0
8	607.95710255	...	1	0
9	492.4671919	...	1	0
10	417.34281455	...	1	0
11	458.2617984	...	0	0
12	515.30886255	...	0	0
13	469.0471837	...	0	0
14	469.0409539	...	1	0
15	535.42243675	...	0	0
16	358.63862805	...	0	0
17	421.37283465	...	0	0
18	400.2268045	...	0	1
19	749.46855975	...	0	0
20	730.67809685	...	0	0

Let’s normalize the dataset to follow the Gaussian-Markov assumptions.

africa.normalize(columns = africa.get_columns(exclude_columns = ["student_score"]))

	123 student_score Float(22) 100%	...	123 source_of_lighting_CANDLE Float 100%	123 country_Zambia Float 100%
1	657.83952525	...	-0.5181275027104616	-0.23208691930827755
2	387.88641055	...	-0.5181275027104616	-0.23208691930827755
3	392.73400295	...	-0.5181275027104616	-0.23208691930827755
4	422.06612525	...	-0.5181275027104616	-0.23208691930827755
5	436.79013695	...	-0.5181275027104616	-0.23208691930827755
6	531.6862021	...	-0.5181275027104616	-0.23208691930827755
7	530.3308348	...	-0.5181275027104616	-0.23208691930827755
8	607.95710255	...	1.9299139918588077	-0.23208691930827755
9	492.4671919	...	1.9299139918588077	-0.23208691930827755
10	417.34281455	...	1.9299139918588077	-0.23208691930827755
11	458.2617984	...	-0.5181275027104616	-0.23208691930827755
12	515.30886255	...	-0.5181275027104616	-0.23208691930827755
13	469.0471837	...	-0.5181275027104616	-0.23208691930827755
14	469.0409539	...	1.9299139918588077	-0.23208691930827755
15	535.42243675	...	-0.5181275027104616	-0.23208691930827755
16	358.63862805	...	-0.5181275027104616	-0.23208691930827755
17	421.37283465	...	-0.5181275027104616	-0.23208691930827755
18	400.2268045	...	-0.5181275027104616	4.308478564962018
19	749.46855975	...	-0.5181275027104616	-0.23208691930827755
20	730.67809685	...	-0.5181275027104616	-0.23208691930827755

We can use a cross-validation to test our model.

from verticapy.machine_learning.vertica import LinearRegression
from verticapy.machine_learning.model_selection import cross_validate

cross_validate(
    LinearRegression(solver = "BFGS"),
    input_relation = africa,
    X = africa.get_columns(exclude_columns = ["student_score"]),
    y = "student_score",
)

	...	bic	time
1-fold	...	48632.0282141365	6.193042278289795
2-fold	...	48707.6694481753	5.871992588043213
3-fold	...	48986.3689253337	5.665011644363403
avg	...	48775.355529215165	5.910015503565471
std	...	152.3710144741786	0.21723780237301593

The model isn’t bad. We’re just using a few variables to get a median absolute error of 47; that is, our score has a distance of 47 from the true value. This seems high, but if we keep in mind that the final score is over 1000, our predictions are quite good.

Let’s compare the importance of our features.

model = LinearRegression(solver = "BFGS")

model.fit(
    input_relation = africa,
    X = africa.get_columns(exclude_columns = ["student_score"]),
    y = "student_score",
)



=======
details
=======
        predictor        |coefficient|std_err | t_value |p_value 
-------------------------+-----------+--------+---------+--------
        Intercept        | 504.47464 | 0.54939|918.25139| 0.00000
    socio_eco_statut     | 19.93079  | 0.69586|28.64183 | 0.00000
   means_of_travel_car   |  8.02692  | 0.52140|15.39500 | 0.00000
   socio_eco_statut_14   |  7.16913  | 0.54330|13.19556 | 0.00000
  english_at_home_never  | -12.60354 | 0.53635|-23.49890| 0.00000
  repeated_grades_never  |  8.79099  | 0.58041|15.14606 | 0.00000
           age           | -5.33563  | 0.61692|-8.64887 | 0.00000
    country_tanzania     | 20.79941  | 0.57771|36.00338 | 0.00000
      teacher_score      | 11.65665  | 0.56041|20.80013 | 0.00000
source_of_lighting_candle| -3.87087  | 0.56637|-6.83449 | 0.00000
     country_zambia      | -12.10108 | 0.54174|-22.33732| 0.00000


==============
regularization
==============
type| lambda 
----+--------
none| 1.00000


===========
call_string
===========
linear_reg('"public"."_verticapy_tmp_linearregression_v_mldb_3e89952097bd11efa8720242ac120002_"', '"public"."_verticapy_tmp_view_v_mldb_3ef1d13097bd11efa8720242ac120002_"', '"student_score"', '"socio_eco_statut", "means_of_travel_CAR", "socio_eco_statut_14", "english_at_home_NEVER", "repeated_grades_NEVER", "age", "country_Tanzania", "teacher_score", "source_of_lighting_CANDLE", "country_Zambia"'
USING PARAMETERS optimizer='bfgs', epsilon=1e-06, max_iterations=100, regularization='none', lambda=1, alpha=0.5, fit_intercept=true)

===============
Additional Info
===============
       Name       |Value
------------------+-----
 iteration_count  |  6  
rejected_row_count|  0  
accepted_row_count|17099

model.features_importance()

The following factors seem to have the greatest influence on a student’s performance:

• Having a good teacher.

• Being of good socio-economic status.

• Tanzanian teachers tend to overrate their students.

• Age (younger students tend to perform better).

• Being able to get to school by car.

Let’s add the prediction to the vDataFrame to see how our model performs its estimations.

model.predict(africa, name = "estimated_student_score")
africa.boxplot(["estimated_student_score", "student_score"])

africa.describe(columns = ["student_score", "estimated_student_score"])

	...	approx_75%	max
"student_score"	...	558.464704565385	918.5515125
"estimated_student_score"	...	534.304974060098	675.48182789268

Our model has trouble catching outliers: exceptionally well-performing and struggling students.

Let’s draw a residual plot.

africa["residual"] = africa["student_score"] - africa["estimated_student_score"]
africa.scatter(["residual", "student_score"])

We see a high heteroscedasticity, indicating that we can’t trust the p-value of the coefficients.

model.coef_
Out[3]: 
array([ 19.93079185,   8.02692135,   7.16912611, -12.60354222,
         8.7909853 ,  -5.33563056,  20.7994124 ,  11.65665235,
        -3.87086824, -12.10107544])

Let’s look at the model’s analysis of variance (ANOVA) table.

model.report("anova")

	...	F	p_value
Regression	...	761.4282673968138	0.0
Residual	...
Total	...

According to the ANOVA table, at least one of our variables is influencing the prediction.

We can also see that a student’s estimated score and true score skew heavily from a normal distribution.

africa["estimated_student_score"].hist()

from verticapy.machine_learning.model_selection.statistical_tests import jarque_bera

jarque_bera(africa, "estimated_student_score")
Out[5]: (566.447140204892, 9.944120070619405e-124)

Our model doesn’t verify the basic hypothesis and therefore isn’t stable enough to be put into production. Let’s look at a second technique.

Example with decomposition

Let’s look at the same dataset, but use decomposition techniques to filter out unimportant information. We don’t have to normalize our data or look at correlations with these types of methods.

We’ll begin by repeating the data preparation process of the previous section and export the resulting vDataFrame to Vertica.

africa = load_africa_education()
africa = africa.select(
    [
        "(zralocp + zmalocp) / 2 AS student_score",
        "zraloct AS teacher_score",
        "XNUMYRS AS teacher_year_teaching",
        "numstu AS number_students_school",
        "PENGLISH AS english_at_home",
        "PTRAVEL AS travel_distance",
        "PTRAVEL2 AS means_of_travel",
        "PMOTHER AS m_education",
        "PFATHER AS f_education",
        "PLIGHT AS source_of_lighting",
        "PABSENT AS days_absent",
        "PREPEAT AS repeated_grades",
        "zpsit AS sitting_place",
        "PAGE AS age",
        "zpses AS socio_eco_statut",
        "country_long AS country",
    ],
)
africa.dropna()
africa.one_hot_encode(max_cardinality = 20)
africa.drop(
    columns = [
        "english_at_home",
        "travel_distance",
        "means_of_travel",
        "m_education",
        "f_education",
        "source_of_lighting",
        "repeated_grades",
        "sitting_place",
        "country",
    ],
)

	123 student_score Float(22) 100%	...	123 teacher_score Float(22) 100%	123 country_Zanzibar Bool 100%
1	657.83952525	...	753.8647201	0
2	387.88641055	...	763.3071026	0
3	392.73400295	...	763.3071026	0
4	422.06612525	...	777.9952532	0
5	436.79013695	...	696.5427817	0
6	531.6862021	...	734.979955	0
7	530.3308348	...	735.456843	0
8	607.95710255	...	753.8647201	0
9	492.4671919	...	590.2921338	0
10	417.34281455	...	613.9457789	0
11	458.2617984	...	648.0909602	0
12	515.30886255	...	709.1326251	0
13	469.0471837	...	727.8266349	0
14	469.0409539	...	735.456843	0
15	535.42243675	...	740.6072335	0
16	358.63862805	...	740.6072335	0
17	421.37283465	...	763.3071026	0
18	400.2268045	...	846.0948606	0
19	749.46855975	...	604.3126412	0
20	730.67809685	...	721.8178461	0

Let’s create our principal component analysis (PCA) model.

from verticapy.machine_learning.vertica import PCA

model = PCA()
model.fit(
    africa,
    africa.get_columns(exclude_columns = ["student_score"]),
)
africa_pca = model.transform()
africa_pca.head(100)

	123 student_score Float(22) 100%	...	123 col85 Float(22) 100%	123 col86 Float(22) 100%
1	657.83952525	...	0.000359577699693903	-0.00027997200419123
2	387.88641055	...	0.00143586986929819	0.00023401543263306
3	392.73400295	...	-0.00026893486013583	0.000597456574135668
4	422.06612525	...	-0.00759699519924349	0.00321536138185285
5	436.79013695	...	0.000477878254887722	0.000586078032193242
6	531.6862021	...	0.00032537305415783	0.000223271620745539
7	530.3308348	...	-0.00790908915800277	0.00276579178468772
8	607.95710255	...	-0.000448267332631759	0.00016682149011722
9	492.4671919	...	-0.00154424872816067	0.000298794345840233
10	417.34281455	...	-0.000354076250603067	0.000278375154693887
11	458.2617984	...	0.000898368377604158	0.000369241593603549
12	515.30886255	...	-0.000466662576917495	0.000116772027345443
13	469.0471837	...	-0.000851027752715577	0.000578910440615927
14	469.0409539	...	0.000224037202007239	0.000266757730610997
15	535.42243675	...	-0.000551107549323452	0.000535499961478971
16	358.63862805	...	-0.00164738654101032	0.00131674529425853
17	421.37283465	...	-0.000180061726714492	-0.000186331793102462
18	400.2268045	...	0.000411553330455122	0.000171698360315153
19	749.46855975	...	-0.000212948445670731	0.000414280898441772
20	730.67809685	...	0.000591773188009201	-8.19261160070404e-05
21	539.2019957	...	-0.000895242543321454	0.00089279759559188
22	425.99336735	...	-0.000135657957848094	0.000242428842493865
23	397.4796701	...	-0.000562747555528514	2.55345437112765e-05
24	453.0202996	...	0.000342909926303206	0.000779725960077904
25	449.99250385	...	0.0001339925919444	0.000196258018422879
26	482.2092927	...	-4.48518535405074e-05	0.000496011381165726
27	510.49975695	...	-1.20109828432714e-05	0.000319115309786115
28	641.9622534	...	-0.000862109828396851	0.000851658003102353
29	510.65265825	...	0.00041537029501207	0.000514312256077254
30	535.58574205	...	-0.00057193680860548	0.000724992415398366
31	399.7076038	...	-0.000322066573663471	0.00045881152904497
32	602.52488005	...	0.000293346147550579	0.000866651942232944
33	565.60944755	...	-0.00138729668233833	0.000944194840685255
34	451.69440555	...	-0.00730603354950347	0.00264216392643703
35	495.5545671	...	0.00139738582608655	-0.00058474861860522
36	456.9573452	...	0.00160913149549304	-0.000187013743951973
37	476.34012315	...	-0.00120009189876403	0.00018986107134941
38	585.9983071	...	0.000394209879192926	-0.000384525192063908
39	596.11136875	...	0.000358976415015503	-0.00037771761997554
40	452.3792203	...	0.000681432549534532	-0.000298043681504545
41	614.97165705	...	0.00051821569540126	-0.000392878537166623
42	473.4818625	...	9.10721168901016e-05	-0.000263104033196432
43	508.95161015	...	-0.000481136699261928	-0.000655703155920966
44	588.9083679	...	1.72963967945152e-05	-9.91417439334259e-05
45	551.2711833	...	-8.35803913525755e-05	-0.000353382184719902
46	365.06159545	...	0.000353926892988085	-0.000111189897703668
47	386.94044515	...	0.00018103598217592	-2.22823827304551e-05
48	427.07559925	...	0.000118900354521405	0.000234190891084192
49	540.20488335	...	-0.000488178036155922	0.000190949515654138
50	470.4509993	...	0.000750881080153201	-0.000114125196752734
51	489.53970635	...	0.000440533442536709	8.85555946701309e-05
52	556.7909405	...	0.000321757095271491	-0.00012833989827228
53	528.38596655	...	0.000465826553677736	1.45062973675688e-05
54	555.0513114	...	0.00042550461529055	-0.000282153342778823
55	519.13142985	...	0.000724116158647454	-0.000290774440477193
56	482.43679495	...	0.000389245226583548	-0.000177547632580087
57	459.9863116	...	0.00063650082674112	6.36564302649487e-05
58	599.1407463	...	0.000209610510831788	5.46683853875884e-05
59	476.61604135	...	0.000314993507149266	-0.00017378746156667
60	404.85261815	...	0.000459262665982364	-8.61116170375874e-05
61	413.12320985	...	0.00070301860311775	-0.000201018517687249
62	630.4264972	...	0.000443377852281037	-0.000134958310202261
63	743.28447985	...	0.000914103785246775	-0.000513648968669834
64	238.94653645	...	0.00153100952542612	-0.00048676700709966
65	498.99230345	...	-6.7976154939581e-05	-0.000178571290525526
66	497.0200491	...	0.000566639922096787	0.000710924129080911
67	413.91855045	...	0.000706203264469199	-8.6191784619231e-05
68	457.58833645	...	-8.55283352749521e-06	4.28884424216678e-05
69	514.33800925	...	0.000232765954098187	7.82518454783603e-06
70	419.0689409	...	0.000121074311387918	3.14852520442872e-05
71	440.6723473	...	0.000121074311387918	3.14852520442872e-05
72	433.6962473	...	0.000121074311387918	3.14852520442872e-05
73	519.13142985	...	0.000121074311387918	3.14852520442872e-05
74	758.1966219	...	0.00122982804388162	-0.000519809682233395
75	525.65015365	...	0.000103274538788054	2.68969994162426e-05
76	640.024659	...	-0.000177887631074497	-0.000126896049547868
77	433.75709155	...	0.000714080760975406	-0.00010092901016448
78	602.12015765	...	0.000672741865594057	8.63662531016798e-05
79	500.19151275	...	0.000749725492647564	-0.000408676209367053
80	579.51566615	...	-0.000343642707958085	-0.000335689878077163
81	495.5545671	...	0.000711419986526615	-0.000606594531118205
82	428.54585685	...	0.000587911000092106	0.000134760309745109
83	519.20701115	...	0.000617238788763242	-0.000497245873567768
84	406.15482625	...	-0.000395387825102419	7.98429174994424e-05
85	590.5249487	...	0.000333769474608915	-0.000366131716646391
86	541.55778385	...	0.000633900800826524	-0.000356379111243875
87	544.01198645	...	0.000497228805909419	-0.000315148555179469
88	490.89222805	...	-0.000457282358303608	-0.000112962872181551
89	548.6765391	...	-7.20068130863998e-05	-0.000416001493545991
90	455.36049795	...	-0.000709146622833584	-0.000277594076985878
91	519.147368	...	-0.000207478220038972	-1.14918718115311e-06
92	574.3841869	...	0.00050064296009617	-0.000113392631338879
93	490.7928971	...	0.000184485476359688	-8.25380891488927e-05
94	497.5443413	...	-0.000609485113200837	-0.000346131463047032
95	611.75199915	...	0.000633539284887123	9.95676846338726e-05
96	532.5956098	...	0.000522265108608628	-0.000322639249720833
97	525.8445453	...	-0.000101606260283591	0.000763504677147397
98	501.42749965	...	0.000412844275820035	-0.000465228143146301
99	512.3963035	...	0.00050218862950323	-0.000329715367076172
100	482.2092927	...	-0.000464696833843353	-0.000408899606932095

We can verify the Gauss-Markov assumptions with our PCA model.

africa_pca.corr(
    columns =
        [
            "student_score",
            "col1",
            "col2",
            "col3",
            "col4",
            "col5",
            "col6",
            "col7",
        ],
)

Let’s use a cross-validation to test our linear regression model.

cross_validate(
    LinearRegression(solver = "BFGS"),
    input_relation = africa_pca,
    X = africa_pca.get_columns(exclude_columns = ["student_score"]),
    y = "student_score",
)

	...	bic	time
1-fold	...	48515.488694506	25.473784923553467
2-fold	...	48586.0284943099	24.47300410270691
3-fold	...	48744.759681479	18.33084535598755
avg	...	48615.42562343163	22.75921146074931
std	...	95.87992409725639	3.157869569946982

As you can see, we’ve created a much more accurate model here than in our first attempt. This example emphasizes the importance of filtering noise from the data.

Conclusion

We’ve seen two techniques that can help us create powerful linear regression models. While the first method normalized the data and looked for correlations, the second method applied a PCA model. The second one allows us to confirm the Gauss-Markov assumptions - an essential part of using linear models.