Scaling Techniques¶

Normalizing data is crucial when using machine learning algorithms because of how sensitive most of them are to un-scaled data. For example, the neighbors-based and KMeans algorithms use the p-distance in their learning phase.

Normalization is the first step before using a linear regression due to Gauss-Markov assumptions.

Unscaled data can also create complications for the convergence of some ML algorithms. Normalization is also a way to encode the data and to retain the global distribution. When we know the estimators to use to scale the data, we can easily un-scaled the data and come back to the original distribution.

There are three main scaling techniques:

Z-Score: We reduce and center the feature values using the average and standard deviation. This scaling technique is sensitive to outliers.
Robust Z-Score: We reduce and center the feature values using the median and the median absolute deviation. This scaling technique is robust to outliers.
Min-Max: We reduce the feature values by using a bijection to [0,1]. The max will reach 1 and the min will reach 0. This scaling technique is robust to outliers.

To demonstrate data scaling in VerticaPy, we will use the well-known titanic dataset.

from verticapy.datasets import load_titanic

titanic = load_titanic()
titanic.head(100)

	123 pclass Int 100%	...	123 body Int 9%	Abc Varchar(100) 57%
1	1	...	22
2	1	...	[null]
3	1	...	[null]
4	1	...	[null]
5	1	...	[null]
6	1	...	[null]
7	1	...	[null]
8	1	...	62
9	1	...	[null]
10	1	...	[null]
11	1	...	[null]
12	1	...	[null]
13	1	...	110
14	1	...	[null]
15	1	...	[null]
16	1	...	38
17	1	...	[null]
18	1	...	126
19	1	...	292
20	1	...	175
21	1	...	[null]
22	1	...	122
23	1	...	166
24	1	...	[null]
25	1	...	207
26	1	...	[null]
27	1	...	232
28	1	...	[null]
29	1	...	[null]
30	1	...	[null]
31	1	...	[null]
32	1	...	[null]
33	1	...	46
34	1	...	169
35	1	...	[null]
36	1	...	[null]
37	1	...	[null]
38	1	...	[null]
39	1	...	[null]
40	1	...	[null]
41	1	...	[null]
42	1	...	[null]
43	1	...	[null]
44	1	...	[null]
45	1	...	[null]
46	1	...	[null]
47	1	...	[null]
48	1	...	[null]
49	1	...	[null]
50	1	...	[null]
51	1	...	[null]
52	1	...	[null]
53	1	...	[null]
54	1	...	[null]
55	1	...	[null]
56	1	...	[null]
57	1	...	[null]
58	1	...	[null]
59	1	...	[null]
60	1	...	[null]
61	1	...	[null]
62	1	...	[null]
63	1	...	[null]
64	1	...	[null]
65	1	...	[null]
66	1	...	[null]
67	1	...	[null]
68	1	...	[null]
69	1	...	[null]
70	1	...	[null]
71	1	...	[null]
72	1	...	[null]
73	1	...	[null]
74	2	...	[null]
75	2	...	[null]
76	2	...	[null]
77	2	...	[null]
78	2	...	[null]
79	2	...	[null]
80	2	...	[null]
81	2	...	[null]
82	2	...	[null]
83	2	...	236
84	2	...	[null]
85	2	...	[null]
86	2	...	[null]
87	2	...	155
88	2	...	[null]
89	2	...	75
90	2	...	35
91	2	...	[null]
92	2	...	[null]
93	2	...	[null]
94	2	...	[null]
95	2	...	[null]
96	2	...	[null]
97	2	...	[null]
98	2	...	165
99	2	...	[null]
100	2	...	[null]

Let’s look at the fare and age of the passengers.

titanic.select(["age", "fare"])

	123 age Numeric(8) 80%	123 fare Numeric(12) 99%
1	71.0	49.5042
2	45.0	35.5
3	[null]	31.0
4	17.0	47.1
5	27.0	136.7792
6	37.0	83.1583
7	31.0	52.0
8	50.0	106.425
9	36.0	31.6792
10	37.0	53.1
11	24.0	79.2
12	45.0	83.475
13	40.0	0.0
14	42.0	42.5
15	[null]	51.8625
16	42.0	52.0
17	[null]	30.6958
18	29.0	30.0
19	46.0	75.2417
20	54.0	51.8625

These lie in different numerical intervals so it’s probably a good idea to scale them. To scale data in VerticaPy, we can use the scale() method.

help(titanic["age"].scale)
Help on function scale in module verticapy.core.vdataframe._scaler:

scale(method: Literal['zscore', 'robust_zscore', 'minmax'] = 'zscore', by: Optional[Annotated[Union[str, list[str]], 'STRING representing one column or a list of columns']] = None, return_trans: bool = False) -> 'vDataFrame'

Scales the input vDataColumns using the input method.

Parameters
----------
method: str, optional
    Method used to scale the data.
     - zscore:
        Normalization using the Z-Score.

        .. math::

            Z_{score}(x) = (x - x_{avg}) / x_{std}

     - robust_zscore:
        Normalization using the Robust Z-Score.

        .. math::

            Z_{rscore}(x) = (x - x_{med}) / (1.4826 * x_{mad})

     - minmax:
        Normalization using the MinMax.

        .. math::

            Z_{minmax}(x) = (x - x_{min}) / (x_{max} - x_{min})
by: SQLColumns, optional
    vDataColumns used in the partition.
return_trans: bool, optimal
    If  set to True,  the method  returns the  transformation
    used instead of the parent vDataFrame. This parameter is used
    for testing purposes.

Returns
-------
vDataFrame
    self._parent

The three main scaling techniques are available. Let’s scale the fare and the age using the MinMax method.

titanic["age"].scale(method = "minmax")
titanic["fare"].scale(method = "minmax")
titanic.select(["age", "fare"])

	123 age Numeric(22) 80%	123 fare Numeric(26) 99%
1	0.887034015313167	0.096625763278767
2	0.560687837328982	0.069291385304605
3	[null]	0.060507970265993
4	0.209238107192168	0.091933077404138
5	0.334755867955316	0.266975218277623
6	0.460273628718464	0.162314191734533
7	0.384962972260575	0.101497240446182
8	0.623446717710556	0.207727765663171
9	0.447721852642149	0.061833680375821
10	0.460273628718464	0.10364429745562
11	0.297100539726371	0.154588104679569
12	0.560687837328982	0.162932348966251
13	0.497928956947408	0.0
14	0.523032509100038	0.082954475364668
15	[null]	0.101228858320002
16	0.523032509100038	0.101497240446182
17	[null]	0.059914211409383
18	0.359859420107945	0.058556100257413
19	0.573239613405297	0.146862017624605
20	0.673653822015815	0.101228858320002

Both of the features now scale in [0,1]. It is also possible to scale by a specific partition with the by parameter.