verticapy.machine_learning.model_selection.statistical_tests.tsa.durbin_watson¶

verticapy.machine_learning.model_selection.statistical_tests.tsa.durbin_watson(input_relation: Annotated[str | vDataFrame, ''], eps: str, ts: str, by: Annotated[str | list[str], 'STRING representing one column or a list of columns'] | None = None) → float¶

Durbin Watson test (residuals autocorrelation).

Parameters¶

input_relation: SQLRelation: Input relation.
eps: str: Input residual vDataColumn.
ts: str: vDataColumn used as timeline to order the data. It can be a numerical or date-like type (date, datetime, timestamp…) vDataColumn.
by: SQLColumns, optional: vDataColumns used in the partition.

Returns¶

float: Durbin Watson statistic.

Examples¶

Initialization¶

Let’s try this test on a dummy dataset that has the following elements:

A value of interest that has noise related to time
Time-stamp data

Before we begin we can import the necessary libraries:

import verticapy as vp

import numpy as np

Data¶

Now we can create the dummy dataset:

# Initialization
N = 50 # Number of Rows

days = list(range(N))

y_val = [2 * x + np.random.normal(scale = 4 * x * x) for x in days]

# vDataFrame
vdf = vp.vDataFrame(
    {
        "day": days,
        "y1": y_val,
    }
)

Model Fitting¶

Next, we can fit a Linear Model. To do that we need to first import the model and intialize:

from verticapy.machine_learning.vertica.linear_model import LinearRegression

model = LinearRegression()

Next we can fit the model:

model.fit(vdf, X = "day", y = "y1")


=======
details
=======
predictor|coefficient| std_err |t_value |p_value 
---------+-----------+---------+--------+--------
Intercept|-638.30828 |880.82131|-0.72467| 0.47217
   day   | 43.03305  |30.97760 | 1.38917| 0.17119


==============
regularization
==============
type| lambda 
----+--------
none| 1.00000


===========
call_string
===========
linear_reg('"public"."_verticapy_tmp_linearregression_v_mldb_a30bde9a979511efa8720242ac120002_"', '"public"."_verticapy_tmp_view_v_mldb_a34c2fd6979511efa8720242ac120002_"', '"y1"', '"day"'
USING PARAMETERS optimizer='newton', epsilon=1e-06, max_iterations=100, regularization='none', lambda=1, alpha=0.5, fit_intercept=true)

===============
Additional Info
===============
       Name       |Value
------------------+-----
 iteration_count  |  1  
rejected_row_count|  0  
accepted_row_count| 50  

We can create a column in the vDataFrame that has the predictions:

model.predict(vdf, X = "day", name = "y_pred")

	123 day Integer	123 y1 Numeric(23)	123 y_pred Float(22)
1	0	0.0	-638.308276681683
2	1	1.4718141238459528	-595.275229236085
3	2	9.563767147206292	-552.242181790486
4	3	-80.59173869539234	-509.209134344888
5	4	-59.011448637908856	-466.176086899289
6	5	-85.4664615740646	-423.143039453691
7	6	-98.16387107457986	-380.109992008092
8	7	128.19569314466747	-337.076944562494
9	8	-296.77209056013197	-294.043897116895
10	9	-240.93297393180575	-251.010849671297
11	10	24.24745121101933	-207.977802225698
12	11	186.63950560486157	-164.9447547801
13	12	-287.06175497856964	-121.911707334501
14	13	-2004.4320601125016	-78.8786598889025
15	14	3.193073644676552	-35.845612443304
16	15	-296.0328966088682	7.18743500229448
17	16	672.4955280456309	50.220482447893
18	17	897.2707148382281	93.2535298934915
19	18	1032.9040446063786	136.28657733909
20	19	-4.8059967885965875	179.319624784688
21	20	-1279.0420313279094	222.352672230287
22	21	-1111.8508309081021	265.385719675886
23	22	3983.0165848275396	308.418767121484
24	23	-1315.7761168096056	351.451814567083
25	24	-459.9123496010786	394.484862012681
26	25	72.0841069518651	437.51790945828
27	26	-386.85885049056213	480.550956903878
28	27	-2417.42276003794	523.584004349477
29	28	-65.36336128519035	566.617051795075
30	29	-6360.333842282093	609.650099240674
31	30	-714.2076595577226	652.683146686272
32	31	2118.740010485263	695.716194131871
33	32	-3685.4304092921357	738.749241577469
34	33	2274.592916047279	781.782289023068
35	34	1239.603942295978	824.815336468666
36	35	8113.194942855282	867.848383914265
37	36	8460.654677364753	910.881431359863
38	37	2439.636266448762	953.914478805462
39	38	-5836.372157175366	996.94752625106
40	39	668.4808967930048	1039.98057369666
41	40	2016.0019102507429	1083.01362114226
42	41	2800.8314533785333	1126.04666858786
43	42	-1703.5334499388593	1169.07971603345
44	43	387.26603072462984	1212.11276347905
45	44	3430.791946623575	1255.14581092465
46	45	-1441.42657777703	1298.17885837025
47	46	4586.501794698723	1341.21190581585
48	47	12457.699345574638	1384.24495326145
49	48	-4247.561202373672	1427.27800070705
50	49	-2726.646239093369	1470.31104815264

Rows: 1-50 | Columns: 3

Then we can calculate the residuals i.e. eps:

vdf["eps"] = vdf["y1"] - vdf["y_pred"]

We can plot the residuals to see the trend:

vdf.scatter(["day", "eps"])

Test¶

Now we can apply the Durbin Watson Test:

from verticapy.machine_learning.model_selection.statistical_tests import durbin_watson

durbin_watson(input_relation = vdf, ts = "day", eps = "eps")
Out[12]: 1.79438889005769

We can see that the Durbin-Watson statistic is not equal to 2. This shows the presence of autocorrelation.

Note

The Durbin-Watson statistic values can be interpretted as such:

Approximately 2: No significant autocorrelation.

Less than 2: Positive autocorrelation (residuals are correlated positively with their lagged values).

Greater than 2: Negative autocorrelation (residuals are correlated negatively with their lagged values).