Normalization

Normalizing data is crucial when using machine learning algorithms because of how sensitive most of them are to un-normalized data. For example, the neighbors-based and k-means algorithms use the p-distance in their learning phase. Normalization is the first step before using a linear regression due to Gauss-Markov assumptions.

Unnormalized data can also create complications for the convergence of some ML algorithms. Normalization is also a way to encode the data and to retain the global distribution. When we know the estimators to use to normalize the data, we can easily un-normalize the data and come back to the original distribution.

There are three main normalization techniques:

• Z-Score : We reduce and center the feature values using the average and standard deviation. This normalization is sensitive to outliers.

• Robust Z-Score : We reduce and center the feature values using the median and the median absolute deviation. This normalization is robust to outliers.

• Min-Max : We reduce the feature values by using a bijection to [0,1]. The max will reach 1 and the min will reach 0. This normalization is robust to outliers.

To demonstrate data normalization in VerticaPy, we will use the well-known ‘Titanic’ dataset.

from verticapy.datasets import load_titanic
vdf = load_titanic()
display(vdf)

	123 pclass Int	123 survived Int	Abc Varchar(164)	Abc sex Varchar(20)	123 age Numeric(6,3)	123 sibsp Int	123 parch Int	Abc ticket Varchar(36)	123 fare Numeric(10,5)	Abc cabin Varchar(30)	Abc embarked Varchar(20)	Abc boat Varchar(100)	123 body Int	Abc home.dest Varchar(100)
1	1	0		female	2.0	1	2	113781	151.55	C22 C26	S	[null]	[null]	Montreal, PQ / Chesterville, ON
2	1	0		male	30.0	1	2	113781	151.55	C22 C26	S	[null]	135	Montreal, PQ / Chesterville, ON
3	1	0		female	25.0	1	2	113781	151.55	C22 C26	S	[null]	[null]	Montreal, PQ / Chesterville, ON
4	1	0		male	39.0	0	0	112050	0.0	A36	S	[null]	[null]	Belfast, NI
5	1	0		male	71.0	0	0	PC 17609	49.5042	[null]	C	[null]	22	Montevideo, Uruguay
6	1	0		male	47.0	1	0	PC 17757	227.525	C62 C64	C	[null]	124	New York, NY
7	1	0		male	[null]	0	0	PC 17318	25.925	[null]	S	[null]	[null]	New York, NY
8	1	0		male	24.0	0	1	PC 17558	247.5208	B58 B60	C	[null]	[null]	Montreal, PQ
9	1	0		male	36.0	0	0	13050	75.2417	C6	C	A	[null]	Winnipeg, MN
10	1	0		male	25.0	0	0	13905	26.0	[null]	C	[null]	148	San Francisco, CA
11	1	0		male	45.0	0	0	113784	35.5	T	S	[null]	[null]	Trenton, NJ
12	1	0		male	42.0	0	0	110489	26.55	D22	S	[null]	[null]	London / Winnipeg, MB
13	1	0		male	41.0	0	0	113054	30.5	A21	S	[null]	[null]	Pomeroy, WA
14	1	0		male	48.0	0	0	PC 17591	50.4958	B10	C	[null]	208	Omaha, NE
15	1	0		male	[null]	0	0	112379	39.6	[null]	C	[null]	[null]	Philadelphia, PA
16	1	0		male	45.0	0	0	113050	26.55	B38	S	[null]	[null]	Washington, DC
17	1	0		male	[null]	0	0	113798	31.0	[null]	S	[null]	[null]	[null]
18	1	0		male	33.0	0	0	695	5.0	B51 B53 B55	S	[null]	[null]	New York, NY
19	1	0		male	28.0	0	0	113059	47.1	[null]	S	[null]	[null]	Montevideo, Uruguay
20	1	0		male	17.0	0	0	113059	47.1	[null]	S	[null]	[null]	Montevideo, Uruguay
21	1	0		male	49.0	0	0	19924	26.0	[null]	S	[null]	[null]	Ascot, Berkshire / Rochester, NY
22	1	0		male	36.0	1	0	19877	78.85	C46	S	[null]	172	Little Onn Hall, Staffs
23	1	0		male	46.0	1	0	W.E.P. 5734	61.175	E31	S	[null]	[null]	Amenia, ND
24	1	0		male	[null]	0	0	112051	0.0	[null]	S	[null]	[null]	Liverpool, England / Belfast
25	1	0		male	27.0	1	0	13508	136.7792	C89	C	[null]	[null]	Los Angeles, CA
26	1	0		male	[null]	0	0	110465	52.0	A14	S	[null]	[null]	Stoughton, MA
27	1	0		male	47.0	0	0	5727	25.5875	E58	S	[null]	[null]	Victoria, BC
28	1	0		male	37.0	1	1	PC 17756	83.1583	E52	C	[null]	[null]	Lakewood, NJ
29	1	0		male	[null]	0	0	113791	26.55	[null]	S	[null]	[null]	Roachdale, IN
30	1	0		male	70.0	1	1	WE/P 5735	71.0	B22	S	[null]	269	Milwaukee, WI
31	1	0		male	39.0	1	0	PC 17599	71.2833	C85	C	[null]	[null]	New York, NY
32	1	0		male	31.0	1	0	F.C. 12750	52.0	B71	S	[null]	[null]	Montreal, PQ
33	1	0		male	50.0	1	0	PC 17761	106.425	C86	C	[null]	62	Deephaven, MN / Cedar Rapids, IA
34	1	0		male	39.0	0	0	PC 17580	29.7	A18	C	[null]	133	Philadelphia, PA
35	1	0		female	36.0	0	0	PC 17531	31.6792	A29	C	[null]	[null]	New York, NY
36	1	0		male	[null]	0	0	PC 17483	221.7792	C95	S	[null]	[null]	[null]
37	1	0		male	30.0	0	0	113051	27.75	C111	C	[null]	[null]	New York, NY
38	1	0		male	19.0	3	2	19950	263.0	C23 C25 C27	S	[null]	[null]	Winnipeg, MB
39	1	0		male	64.0	1	4	19950	263.0	C23 C25 C27	S	[null]	[null]	Winnipeg, MB
40	1	0		male	[null]	0	0	113778	26.55	D34	S	[null]	[null]	Westcliff-on-Sea, Essex
41	1	0		male	[null]	0	0	112058	0.0	B102	S	[null]	[null]	[null]
42	1	0		male	37.0	1	0	113803	53.1	C123	S	[null]	[null]	Scituate, MA
43	1	0		male	47.0	0	0	111320	38.5	E63	S	[null]	275	St Anne's-on-Sea, Lancashire
44	1	0		male	24.0	0	0	PC 17593	79.2	B86	C	[null]	[null]	[null]
45	1	0		male	71.0	0	0	PC 17754	34.6542	A5	C	[null]	[null]	New York, NY
46	1	0		male	38.0	0	1	PC 17582	153.4625	C91	S	[null]	147	Winnipeg, MB
47	1	0		male	46.0	0	0	PC 17593	79.2	B82 B84	C	[null]	[null]	New York, NY
48	1	0		male	[null]	0	0	113796	42.4	[null]	S	[null]	[null]	[null]
49	1	0		male	45.0	1	0	36973	83.475	C83	S	[null]	[null]	New York, NY
50	1	0		male	40.0	0	0	112059	0.0	B94	S	[null]	110	[null]
51	1	0		male	55.0	1	1	12749	93.5	B69	S	[null]	307	Montreal, PQ
52	1	0		male	42.0	0	0	113038	42.5	B11	S	[null]	[null]	London / Middlesex
53	1	0		male	[null]	0	0	17463	51.8625	E46	S	[null]	[null]	Brighton, MA
54	1	0		male	55.0	0	0	680	50.0	C39	S	[null]	[null]	London / Birmingham
55	1	0		male	42.0	1	0	113789	52.0	[null]	S	[null]	38	New York, NY
56	1	0		male	[null]	0	0	PC 17600	30.6958	[null]	C	14	[null]	New York, NY
57	1	0		female	50.0	0	0	PC 17595	28.7125	C49	C	[null]	[null]	Paris, France New York, NY
58	1	0		male	46.0	0	0	694	26.0	[null]	S	[null]	80	Bennington, VT
59	1	0		male	50.0	0	0	113044	26.0	E60	S	[null]	[null]	London
60	1	0		male	32.5	0	0	113503	211.5	C132	C	[null]	45	[null]
61	1	0		male	58.0	0	0	11771	29.7	B37	C	[null]	258	Buffalo, NY
62	1	0		male	41.0	1	0	17464	51.8625	D21	S	[null]	[null]	Southington / Noank, CT
63	1	0		male	[null]	0	0	113028	26.55	C124	S	[null]	[null]	Portland, OR
64	1	0		male	[null]	0	0	PC 17612	27.7208	[null]	C	[null]	[null]	Chicago, IL
65	1	0		male	29.0	0	0	113501	30.0	D6	S	[null]	126	Springfield, MA
66	1	0		male	30.0	0	0	113801	45.5	[null]	S	[null]	[null]	London / New York, NY
67	1	0		male	30.0	0	0	110469	26.0	C106	S	[null]	[null]	Brockton, MA
68	1	0		male	19.0	1	0	113773	53.1	D30	S	[null]	[null]	New York, NY
69	1	0		male	46.0	0	0	13050	75.2417	C6	C	[null]	292	Vancouver, BC
70	1	0		male	54.0	0	0	17463	51.8625	E46	S	[null]	175	Dorchester, MA
71	1	0		male	28.0	1	0	PC 17604	82.1708	[null]	C	[null]	[null]	New York, NY
72	1	0		male	65.0	0	0	13509	26.55	E38	S	[null]	249	East Bridgewater, MA
73	1	0		male	44.0	2	0	19928	90.0	C78	Q	[null]	230	Fond du Lac, WI
74	1	0		male	55.0	0	0	113787	30.5	C30	S	[null]	[null]	Montreal, PQ
75	1	0		male	47.0	0	0	113796	42.4	[null]	S	[null]	[null]	Washington, DC
76	1	0		male	37.0	0	1	PC 17596	29.7	C118	C	[null]	[null]	Brooklyn, NY
77	1	0		male	58.0	0	2	35273	113.275	D48	C	[null]	122	Lexington, MA
78	1	0		male	64.0	0	0	693	26.0	[null]	S	[null]	263	Isle of Wight, England
79	1	0		male	65.0	0	1	113509	61.9792	B30	C	[null]	234	Providence, RI
80	1	0		male	28.5	0	0	PC 17562	27.7208	D43	C	[null]	189	?Havana, Cuba
81	1	0		male	[null]	0	0	112052	0.0	[null]	S	[null]	[null]	Belfast
82	1	0		male	45.5	0	0	113043	28.5	C124	S	[null]	166	Surbiton Hill, Surrey
83	1	0		male	23.0	0	0	12749	93.5	B24	S	[null]	[null]	Montreal, PQ
84	1	0		male	29.0	1	0	113776	66.6	C2	S	[null]	[null]	Isleworth, England
85	1	0		male	18.0	1	0	PC 17758	108.9	C65	C	[null]	[null]	Madrid, Spain
86	1	0		male	47.0	0	0	110465	52.0	C110	S	[null]	207	Worcester, MA
87	1	0		male	38.0	0	0	19972	0.0	[null]	S	[null]	[null]	Rotterdam, Netherlands
88	1	0		male	22.0	0	0	PC 17760	135.6333	[null]	C	[null]	232	[null]
89	1	0		male	[null]	0	0	PC 17757	227.525	[null]	C	[null]	[null]	[null]
90	1	0		male	31.0	0	0	PC 17590	50.4958	A24	S	[null]	[null]	Trenton, NJ
91	1	0		male	[null]	0	0	113767	50.0	A32	S	[null]	[null]	Seattle, WA
92	1	0		male	36.0	0	0	13049	40.125	A10	C	[null]	[null]	Winnipeg, MB
93	1	0		male	55.0	1	0	PC 17603	59.4	[null]	C	[null]	[null]	New York, NY
94	1	0		male	33.0	0	0	113790	26.55	[null]	S	[null]	109	London
95	1	0		male	61.0	1	3	PC 17608	262.375	B57 B59 B63 B66	C	[null]	[null]	Haverford, PA / Cooperstown, NY
96	1	0		male	50.0	1	0	13507	55.9	E44	S	[null]	[null]	Duluth, MN
97	1	0		male	56.0	0	0	113792	26.55	[null]	S	[null]	[null]	New York, NY
98	1	0		male	56.0	0	0	17764	30.6958	A7	C	[null]	[null]	St James, Long Island, NY
99	1	0		male	24.0	1	0	13695	60.0	C31	S	[null]	[null]	Huntington, WV
100	1	0		male	[null]	0	0	113056	26.0	A19	S	[null]	[null]	Streatham, Surrey

Rows: 1-100 | Columns: 14

Let’s look at the ‘fare’ and ‘age’ of the passengers.

vdf.select(["age", "fare"])

	123 age Numeric(6,3)	123 fare Numeric(10,5)
1	2.0	151.55
2	30.0	151.55
3	25.0	151.55
4	39.0	0.0
5	71.0	49.5042
6	47.0	227.525
7	[null]	25.925
8	24.0	247.5208
9	36.0	75.2417
10	25.0	26.0
11	45.0	35.5
12	42.0	26.55
13	41.0	30.5
14	48.0	50.4958
15	[null]	39.6
16	45.0	26.55
17	[null]	31.0
18	33.0	5.0
19	28.0	47.1
20	17.0	47.1
21	49.0	26.0
22	36.0	78.85
23	46.0	61.175
24	[null]	0.0
25	27.0	136.7792
26	[null]	52.0
27	47.0	25.5875
28	37.0	83.1583
29	[null]	26.55
30	70.0	71.0
31	39.0	71.2833
32	31.0	52.0
33	50.0	106.425
34	39.0	29.7
35	36.0	31.6792
36	[null]	221.7792
37	30.0	27.75
38	19.0	263.0
39	64.0	263.0
40	[null]	26.55
41	[null]	0.0
42	37.0	53.1
43	47.0	38.5
44	24.0	79.2
45	71.0	34.6542
46	38.0	153.4625
47	46.0	79.2
48	[null]	42.4
49	45.0	83.475
50	40.0	0.0
51	55.0	93.5
52	42.0	42.5
53	[null]	51.8625
54	55.0	50.0
55	42.0	52.0
56	[null]	30.6958
57	50.0	28.7125
58	46.0	26.0
59	50.0	26.0
60	32.5	211.5
61	58.0	29.7
62	41.0	51.8625
63	[null]	26.55
64	[null]	27.7208
65	29.0	30.0
66	30.0	45.5
67	30.0	26.0
68	19.0	53.1
69	46.0	75.2417
70	54.0	51.8625
71	28.0	82.1708
72	65.0	26.55
73	44.0	90.0
74	55.0	30.5
75	47.0	42.4
76	37.0	29.7
77	58.0	113.275
78	64.0	26.0
79	65.0	61.9792
80	28.5	27.7208
81	[null]	0.0
82	45.5	28.5
83	23.0	93.5
84	29.0	66.6
85	18.0	108.9
86	47.0	52.0
87	38.0	0.0
88	22.0	135.6333
89	[null]	227.525
90	31.0	50.4958
91	[null]	50.0
92	36.0	40.125
93	55.0	59.4
94	33.0	26.55
95	61.0	262.375
96	50.0	55.9
97	56.0	26.55
98	56.0	30.6958
99	24.0	60.0
100	[null]	26.0

Rows: 1-100 | Columns: 2

These lie in different numerical intervals so it’s probably a good idea to normalize them. To normalize data in VerticaPy, we can use the ‘normalize’ method.

help(vdf["age"].normalize)

Help on method normalize in module verticapy.vcolumn:

normalize(method:str='zscore', by:list=[], return_trans:bool=False) method of verticapy.vcolumn.vColumn instance
    ---------------------------------------------------------------------------
    Normalizes the input vColumns using the input method.

    Parameters
    ----------
    method: str, optional
            Method to use to normalize.
                    zscore        : Normalization using the Z-Score (avg and std).
                            (x - avg) / std
                    robust_zscore : Normalization using the Robust Z-Score (median and mad).
                            (x - median) / (1.4826 * mad)
                    minmax        : Normalization using the MinMax (min and max).
                            (x - min) / (max - min)
    by: list, optional
            vColumns used in the partition.
    return_trans: bool, optimal
            If set to True, the method will return the transformation used instead of
            the parent vDataFrame. This parameter is used for testing purpose.

    Returns
    -------
    vDataFrame
            self.parent

    See Also
    --------
    vDataFrame.outliers : Computes the vDataFrame Global Outliers.

The three main normalization techniques are available. Let’s normalize the ‘fare’ and the ‘age’ using the ‘MinMax’ method.

vdf["age"].normalize(method = "minmax")
vdf["fare"].normalize(method = "minmax")
vdf.select(["age", "fare"])

	123 age Numeric(20,15)	123 fare Numeric(24,15)
1	0.020961466047446	0.295805899800363
2	0.37241119618426	0.295805899800363
3	0.309652315802686	0.295805899800363
4	0.485377180871093	0.0
5	0.887034015313167	0.096625763278767
6	0.585791389481612	0.44409922370226
7	[null]	0.050602229972447
8	0.297100539726371	0.483128426019833
9	0.447721852642149	0.146862017624605
10	0.309652315802686	0.050748620223091
11	0.560687837328982	0.069291385304605
12	0.523032509100038	0.05182214872781
13	0.510480733023723	0.059532035261703
14	0.598343165557926	0.098561237579275
15	[null]	0.077294052339785
16	0.560687837328982	0.05182214872781
17	[null]	0.060507970265993
18	0.410066524413204	0.009759350042902
19	0.34730764403163	0.091933077404138
20	0.209238107192168	0.091933077404138
21	0.610894941634241	0.050748620223091
22	0.447721852642149	0.153904950176566
23	0.573239613405297	0.119405647774907
24	[null]	0.0
25	0.334755867955316	0.266975218277623
26	[null]	0.101497240446182
27	0.585791389481612	0.049943473844552
28	0.460273628718464	0.162314191734533
29	[null]	0.05182214872781
30	0.874482239236852	0.13858277060921
31	0.485377180871093	0.139135735382641
32	0.384962972260575	0.101497240446182
33	0.623446717710556	0.207727765663171
34	0.485377180871093	0.057970539254838
35	0.447721852642149	0.061833680375821
36	[null]	0.432884169006959
37	0.37241119618426	0.054164392738107
38	0.234341659344797	0.513341812256651
39	0.799171582778963	0.513341812256651
40	[null]	0.05182214872781
41	[null]	0.0
42	0.460273628718464	0.10364429745562
43	0.585791389481612	0.075146995330346
44	0.297100539726371	0.154588104679569
45	0.887034015313167	0.067640493651348
46	0.472825404794778	0.299538851191773
47	0.573239613405297	0.154588104679569
48	[null]	0.08275928836381
49	0.560687837328982	0.162932348966251
50	0.497928956947408	0.0
51	0.68620559809213	0.182499845802269
52	0.523032509100038	0.082954475364668
53	[null]	0.101228858320002
54	0.68620559809213	0.097593500429021
55	0.523032509100038	0.101497240446182
56	[null]	0.059914211409383
57	0.623446717710556	0.056043067621365
58	0.573239613405297	0.050748620223091
59	0.623446717710556	0.050748620223091
60	0.403790636375047	0.412820506814759
61	0.723860926321074	0.057970539254838
62	0.510480733023723	0.101228858320002
63	[null]	0.05182214872781
64	[null]	0.054107398133856
65	0.359859420107945	0.058556100257413
66	0.37241119618426	0.088810085390409
67	0.37241119618426	0.050748620223091
68	0.234341659344797	0.10364429745562
69	0.573239613405297	0.146862017624605
70	0.673653822015815	0.101228858320002
71	0.34730764403163	0.16038672010106
72	0.811723358855278	0.05182214872781
73	0.548136061252667	0.175668300772238
74	0.68620559809213	0.059532035261703
75	0.585791389481612	0.08275928836381
76	0.460273628718464	0.057970539254838
77	0.723860926321074	0.221098075221947
78	0.799171582778963	0.050748620223091
79	0.811723358855278	0.120975341635808
80	0.353583532069788	0.054107398133856
81	[null]	0.0
82	0.566963725367139	0.055628295244542
83	0.284548763650056	0.182499845802269
84	0.359859420107945	0.129994542571456
85	0.221789883268482	0.212558643934408
86	0.585791389481612	0.101497240446182
87	0.472825404794778	0.0
88	0.271996987573742	0.264738570434791
89	[null]	0.44409922370226
90	0.384962972260575	0.098561237579275
91	[null]	0.097593500429021
92	0.447721852642149	0.078318784094289
93	0.68620559809213	0.115941078509677
94	0.410066524413204	0.05182214872781
95	0.761516254550019	0.512121893501288
96	0.623446717710556	0.109109533479646
97	0.698757374168445	0.05182214872781
98	0.698757374168445	0.059914211409383
99	0.297100539726371	0.117112200514825
100	[null]	0.050748620223091

Rows: 1-100 | Columns: 2

Both of the features now scale in [0,1]. It is also possible to normalize by a specific partition with the ‘by’ parameter.