VerticaPy

Python API for Vertica Data Science at Scale

Quick Start¶

First, install the API using the pip command.

root@ubuntu:~$ pip3 install verticapy[all]

If you don't want to install all the dependencies, which include geopandas and some other geospatial dependencies, you can use the regular 'pip install' command.

root@ubuntu:~$ pip3 install verticapy

You can set up a new connection.

In [ ]:

import verticapy as vp

# Creating a new connection
vp.new_connection({"host": "10.211.55.14", 
                   "port": "5433", 
                   "database": "testdb", 
                   "password": "XxX", 
                   "user": "dbadmin"},
                   name = "MyVerticaConnection")
# Connecting to the Database
vp.connect("MyVerticaConnection")

Create a VerticaPy schema for native VerticaPy models (that is, models available in VerticaPy, but not Vertica itself):

In [ ]:

vp.create_verticapy_schema()

After creating a connection and the verticapy schema, you can start importing your own files (click here to download the iris dataset).

In [1]:

iris = vp.read_csv("iris.csv")
iris

Out[1]:

	123 Id Int	123 SepalLengthCm Numeric(5,2)	123 SepalWidthCm Numeric(5,2)	123 PetalLengthCm Numeric(5,2)	123 PetalWidthCm Numeric(5,2)	Abc Species Varchar(30)
1	1	5.1	3.5	1.4	0.2	Iris-setosa
2	2	4.9	3.0	1.4	0.2	Iris-setosa
3	3	4.7	3.2	1.3	0.2	Iris-setosa
4	4	4.6	3.1	1.5	0.2	Iris-setosa
5	5	5.0	3.6	1.4	0.2	Iris-setosa
6	6	5.4	3.9	1.7	0.4	Iris-setosa
7	7	4.6	3.4	1.4	0.3	Iris-setosa
8	8	5.0	3.4	1.5	0.2	Iris-setosa
9	9	4.4	2.9	1.4	0.2	Iris-setosa
10	10	4.9	3.1	1.5	0.1	Iris-setosa
11	11	5.4	3.7	1.5	0.2	Iris-setosa
12	12	4.8	3.4	1.6	0.2	Iris-setosa
13	13	4.8	3.0	1.4	0.1	Iris-setosa
14	14	4.3	3.0	1.1	0.1	Iris-setosa
15	15	5.8	4.0	1.2	0.2	Iris-setosa
16	16	5.7	4.4	1.5	0.4	Iris-setosa
17	17	5.4	3.9	1.3	0.4	Iris-setosa
18	18	5.1	3.5	1.4	0.3	Iris-setosa
19	19	5.7	3.8	1.7	0.3	Iris-setosa
20	20	5.1	3.8	1.5	0.3	Iris-setosa
21	21	5.4	3.4	1.7	0.2	Iris-setosa
22	22	5.1	3.7	1.5	0.4	Iris-setosa
23	23	4.6	3.6	1.0	0.2	Iris-setosa
24	24	5.1	3.3	1.7	0.5	Iris-setosa
25	25	4.8	3.4	1.9	0.2	Iris-setosa
26	26	5.0	3.0	1.6	0.2	Iris-setosa
27	27	5.0	3.4	1.6	0.4	Iris-setosa
28	28	5.2	3.5	1.5	0.2	Iris-setosa
29	29	5.2	3.4	1.4	0.2	Iris-setosa
30	30	4.7	3.2	1.6	0.2	Iris-setosa
31	31	4.8	3.1	1.6	0.2	Iris-setosa
32	32	5.4	3.4	1.5	0.4	Iris-setosa
33	33	5.2	4.1	1.5	0.1	Iris-setosa
34	34	5.5	4.2	1.4	0.2	Iris-setosa
35	35	4.9	3.1	1.5	0.1	Iris-setosa
36	36	5.0	3.2	1.2	0.2	Iris-setosa
37	37	5.5	3.5	1.3	0.2	Iris-setosa
38	38	4.9	3.1	1.5	0.1	Iris-setosa
39	39	4.4	3.0	1.3	0.2	Iris-setosa
40	40	5.1	3.4	1.5	0.2	Iris-setosa
41	41	5.0	3.5	1.3	0.3	Iris-setosa
42	42	4.5	2.3	1.3	0.3	Iris-setosa
43	43	4.4	3.2	1.3	0.2	Iris-setosa
44	44	5.0	3.5	1.6	0.6	Iris-setosa
45	45	5.1	3.8	1.9	0.4	Iris-setosa
46	46	4.8	3.0	1.4	0.3	Iris-setosa
47	47	5.1	3.8	1.6	0.2	Iris-setosa
48	48	4.6	3.2	1.4	0.2	Iris-setosa
49	49	5.3	3.7	1.5	0.2	Iris-setosa
50	50	5.0	3.3	1.4	0.2	Iris-setosa
51	51	7.0	3.2	4.7	1.4	Iris-versicolor
52	52	6.4	3.2	4.5	1.5	Iris-versicolor
53	53	6.9	3.1	4.9	1.5	Iris-versicolor
54	54	5.5	2.3	4.0	1.3	Iris-versicolor
55	55	6.5	2.8	4.6	1.5	Iris-versicolor
56	56	5.7	2.8	4.5	1.3	Iris-versicolor
57	57	6.3	3.3	4.7	1.6	Iris-versicolor
58	58	4.9	2.4	3.3	1.0	Iris-versicolor
59	59	6.6	2.9	4.6	1.3	Iris-versicolor
60	60	5.2	2.7	3.9	1.4	Iris-versicolor
61	61	5.0	2.0	3.5	1.0	Iris-versicolor
62	62	5.9	3.0	4.2	1.5	Iris-versicolor
63	63	6.0	2.2	4.0	1.0	Iris-versicolor
64	64	6.1	2.9	4.7	1.4	Iris-versicolor
65	65	5.6	2.9	3.6	1.3	Iris-versicolor
66	66	6.7	3.1	4.4	1.4	Iris-versicolor
67	67	5.6	3.0	4.5	1.5	Iris-versicolor
68	68	5.8	2.7	4.1	1.0	Iris-versicolor
69	69	6.2	2.2	4.5	1.5	Iris-versicolor
70	70	5.6	2.5	3.9	1.1	Iris-versicolor
71	71	5.9	3.2	4.8	1.8	Iris-versicolor
72	72	6.1	2.8	4.0	1.3	Iris-versicolor
73	73	6.3	2.5	4.9	1.5	Iris-versicolor
74	74	6.1	2.8	4.7	1.2	Iris-versicolor
75	75	6.4	2.9	4.3	1.3	Iris-versicolor
76	76	6.6	3.0	4.4	1.4	Iris-versicolor
77	77	6.8	2.8	4.8	1.4	Iris-versicolor
78	78	6.7	3.0	5.0	1.7	Iris-versicolor
79	79	6.0	2.9	4.5	1.5	Iris-versicolor
80	80	5.7	2.6	3.5	1.0	Iris-versicolor
81	81	5.5	2.4	3.8	1.1	Iris-versicolor
82	82	5.5	2.4	3.7	1.0	Iris-versicolor
83	83	5.8	2.7	3.9	1.2	Iris-versicolor
84	84	6.0	2.7	5.1	1.6	Iris-versicolor
85	85	5.4	3.0	4.5	1.5	Iris-versicolor
86	86	6.0	3.4	4.5	1.6	Iris-versicolor
87	87	6.7	3.1	4.7	1.5	Iris-versicolor
88	88	6.3	2.3	4.4	1.3	Iris-versicolor
89	89	5.6	3.0	4.1	1.3	Iris-versicolor
90	90	5.5	2.5	4.0	1.3	Iris-versicolor
91	91	5.5	2.6	4.4	1.2	Iris-versicolor
92	92	6.1	3.0	4.6	1.4	Iris-versicolor
93	93	5.8	2.6	4.0	1.2	Iris-versicolor
94	94	5.0	2.3	3.3	1.0	Iris-versicolor
95	95	5.6	2.7	4.2	1.3	Iris-versicolor
96	96	5.7	3.0	4.2	1.2	Iris-versicolor
97	97	5.7	2.9	4.2	1.3	Iris-versicolor
98	98	6.2	2.9	4.3	1.3	Iris-versicolor
99	99	5.1	2.5	3.0	1.1	Iris-versicolor
100	100	5.7	2.8	4.1	1.3	Iris-versicolor

Rows: 1-100 | Columns: 6

If you don't have data on hand, you can load some well-known datasets. We start by creating a vDataFrame using a dataset. In this example, we use the 'titanic' dataset.

In [1]:

from verticapy.datasets import load_titanic
vdf = load_titanic()

We can now explore our imported dataset through the vDataFrame.

In [3]:

vdf

Out[3]:

	123 pclass Int	123 survived Int	Abc Varchar(164)	Abc sex Varchar(20)	123 age Numeric(6,3)	123 sibsp Int	123 parch Int	Abc ticket Varchar(36)	123 fare Numeric(10,5)	Abc cabin Varchar(30)	Abc embarked Varchar(20)	Abc boat Varchar(100)	123 body Int	Abc home.dest Varchar(100)
1	1	0		female	2.0	1	2	113781	151.55	C22 C26	S	[null]	[null]	Montreal, PQ / Chesterville, ON
2	1	0		male	30.0	1	2	113781	151.55	C22 C26	S	[null]	135	Montreal, PQ / Chesterville, ON
3	1	0		female	25.0	1	2	113781	151.55	C22 C26	S	[null]	[null]	Montreal, PQ / Chesterville, ON
4	1	0		male	39.0	0	0	112050	0.0	A36	S	[null]	[null]	Belfast, NI
5	1	0		male	71.0	0	0	PC 17609	49.5042	[null]	C	[null]	22	Montevideo, Uruguay
6	1	0		male	47.0	1	0	PC 17757	227.525	C62 C64	C	[null]	124	New York, NY
7	1	0		male	[null]	0	0	PC 17318	25.925	[null]	S	[null]	[null]	New York, NY
8	1	0		male	24.0	0	1	PC 17558	247.5208	B58 B60	C	[null]	[null]	Montreal, PQ
9	1	0		male	36.0	0	0	13050	75.2417	C6	C	A	[null]	Winnipeg, MN
10	1	0		male	25.0	0	0	13905	26.0	[null]	C	[null]	148	San Francisco, CA
11	1	0		male	45.0	0	0	113784	35.5	T	S	[null]	[null]	Trenton, NJ
12	1	0		male	42.0	0	0	110489	26.55	D22	S	[null]	[null]	London / Winnipeg, MB
13	1	0		male	41.0	0	0	113054	30.5	A21	S	[null]	[null]	Pomeroy, WA
14	1	0		male	48.0	0	0	PC 17591	50.4958	B10	C	[null]	208	Omaha, NE
15	1	0		male	[null]	0	0	112379	39.6	[null]	C	[null]	[null]	Philadelphia, PA
16	1	0		male	45.0	0	0	113050	26.55	B38	S	[null]	[null]	Washington, DC
17	1	0		male	[null]	0	0	113798	31.0	[null]	S	[null]	[null]	[null]
18	1	0		male	33.0	0	0	695	5.0	B51 B53 B55	S	[null]	[null]	New York, NY
19	1	0		male	28.0	0	0	113059	47.1	[null]	S	[null]	[null]	Montevideo, Uruguay
20	1	0		male	17.0	0	0	113059	47.1	[null]	S	[null]	[null]	Montevideo, Uruguay
21	1	0		male	49.0	0	0	19924	26.0	[null]	S	[null]	[null]	Ascot, Berkshire / Rochester, NY
22	1	0		male	36.0	1	0	19877	78.85	C46	S	[null]	172	Little Onn Hall, Staffs
23	1	0		male	46.0	1	0	W.E.P. 5734	61.175	E31	S	[null]	[null]	Amenia, ND
24	1	0		male	[null]	0	0	112051	0.0	[null]	S	[null]	[null]	Liverpool, England / Belfast
25	1	0		male	27.0	1	0	13508	136.7792	C89	C	[null]	[null]	Los Angeles, CA
26	1	0		male	[null]	0	0	110465	52.0	A14	S	[null]	[null]	Stoughton, MA
27	1	0		male	47.0	0	0	5727	25.5875	E58	S	[null]	[null]	Victoria, BC
28	1	0		male	37.0	1	1	PC 17756	83.1583	E52	C	[null]	[null]	Lakewood, NJ
29	1	0		male	[null]	0	0	113791	26.55	[null]	S	[null]	[null]	Roachdale, IN
30	1	0		male	70.0	1	1	WE/P 5735	71.0	B22	S	[null]	269	Milwaukee, WI
31	1	0		male	39.0	1	0	PC 17599	71.2833	C85	C	[null]	[null]	New York, NY
32	1	0		male	31.0	1	0	F.C. 12750	52.0	B71	S	[null]	[null]	Montreal, PQ
33	1	0		male	50.0	1	0	PC 17761	106.425	C86	C	[null]	62	Deephaven, MN / Cedar Rapids, IA
34	1	0		male	39.0	0	0	PC 17580	29.7	A18	C	[null]	133	Philadelphia, PA
35	1	0		female	36.0	0	0	PC 17531	31.6792	A29	C	[null]	[null]	New York, NY
36	1	0		male	[null]	0	0	PC 17483	221.7792	C95	S	[null]	[null]	[null]
37	1	0		male	30.0	0	0	113051	27.75	C111	C	[null]	[null]	New York, NY
38	1	0		male	19.0	3	2	19950	263.0	C23 C25 C27	S	[null]	[null]	Winnipeg, MB
39	1	0		male	64.0	1	4	19950	263.0	C23 C25 C27	S	[null]	[null]	Winnipeg, MB
40	1	0		male	[null]	0	0	113778	26.55	D34	S	[null]	[null]	Westcliff-on-Sea, Essex
41	1	0		male	[null]	0	0	112058	0.0	B102	S	[null]	[null]	[null]
42	1	0		male	37.0	1	0	113803	53.1	C123	S	[null]	[null]	Scituate, MA
43	1	0		male	47.0	0	0	111320	38.5	E63	S	[null]	275	St Anne's-on-Sea, Lancashire
44	1	0		male	24.0	0	0	PC 17593	79.2	B86	C	[null]	[null]	[null]
45	1	0		male	71.0	0	0	PC 17754	34.6542	A5	C	[null]	[null]	New York, NY
46	1	0		male	38.0	0	1	PC 17582	153.4625	C91	S	[null]	147	Winnipeg, MB
47	1	0		male	46.0	0	0	PC 17593	79.2	B82 B84	C	[null]	[null]	New York, NY
48	1	0		male	[null]	0	0	113796	42.4	[null]	S	[null]	[null]	[null]
49	1	0		male	45.0	1	0	36973	83.475	C83	S	[null]	[null]	New York, NY
50	1	0		male	40.0	0	0	112059	0.0	B94	S	[null]	110	[null]
51	1	0		male	55.0	1	1	12749	93.5	B69	S	[null]	307	Montreal, PQ
52	1	0		male	42.0	0	0	113038	42.5	B11	S	[null]	[null]	London / Middlesex
53	1	0		male	[null]	0	0	17463	51.8625	E46	S	[null]	[null]	Brighton, MA
54	1	0		male	55.0	0	0	680	50.0	C39	S	[null]	[null]	London / Birmingham
55	1	0		male	42.0	1	0	113789	52.0	[null]	S	[null]	38	New York, NY
56	1	0		male	[null]	0	0	PC 17600	30.6958	[null]	C	14	[null]	New York, NY
57	1	0		female	50.0	0	0	PC 17595	28.7125	C49	C	[null]	[null]	Paris, France New York, NY
58	1	0		male	46.0	0	0	694	26.0	[null]	S	[null]	80	Bennington, VT
59	1	0		male	50.0	0	0	113044	26.0	E60	S	[null]	[null]	London
60	1	0		male	32.5	0	0	113503	211.5	C132	C	[null]	45	[null]
61	1	0		male	58.0	0	0	11771	29.7	B37	C	[null]	258	Buffalo, NY
62	1	0		male	41.0	1	0	17464	51.8625	D21	S	[null]	[null]	Southington / Noank, CT
63	1	0		male	[null]	0	0	113028	26.55	C124	S	[null]	[null]	Portland, OR
64	1	0		male	[null]	0	0	PC 17612	27.7208	[null]	C	[null]	[null]	Chicago, IL
65	1	0		male	29.0	0	0	113501	30.0	D6	S	[null]	126	Springfield, MA
66	1	0		male	30.0	0	0	113801	45.5	[null]	S	[null]	[null]	London / New York, NY
67	1	0		male	30.0	0	0	110469	26.0	C106	S	[null]	[null]	Brockton, MA
68	1	0		male	19.0	1	0	113773	53.1	D30	S	[null]	[null]	New York, NY
69	1	0		male	46.0	0	0	13050	75.2417	C6	C	[null]	292	Vancouver, BC
70	1	0		male	54.0	0	0	17463	51.8625	E46	S	[null]	175	Dorchester, MA
71	1	0		male	28.0	1	0	PC 17604	82.1708	[null]	C	[null]	[null]	New York, NY
72	1	0		male	65.0	0	0	13509	26.55	E38	S	[null]	249	East Bridgewater, MA
73	1	0		male	44.0	2	0	19928	90.0	C78	Q	[null]	230	Fond du Lac, WI
74	1	0		male	55.0	0	0	113787	30.5	C30	S	[null]	[null]	Montreal, PQ
75	1	0		male	47.0	0	0	113796	42.4	[null]	S	[null]	[null]	Washington, DC
76	1	0		male	37.0	0	1	PC 17596	29.7	C118	C	[null]	[null]	Brooklyn, NY
77	1	0		male	58.0	0	2	35273	113.275	D48	C	[null]	122	Lexington, MA
78	1	0		male	64.0	0	0	693	26.0	[null]	S	[null]	263	Isle of Wight, England
79	1	0		male	65.0	0	1	113509	61.9792	B30	C	[null]	234	Providence, RI
80	1	0		male	28.5	0	0	PC 17562	27.7208	D43	C	[null]	189	?Havana, Cuba
81	1	0		male	[null]	0	0	112052	0.0	[null]	S	[null]	[null]	Belfast
82	1	0		male	45.5	0	0	113043	28.5	C124	S	[null]	166	Surbiton Hill, Surrey
83	1	0		male	23.0	0	0	12749	93.5	B24	S	[null]	[null]	Montreal, PQ
84	1	0		male	29.0	1	0	113776	66.6	C2	S	[null]	[null]	Isleworth, England
85	1	0		male	18.0	1	0	PC 17758	108.9	C65	C	[null]	[null]	Madrid, Spain
86	1	0		male	47.0	0	0	110465	52.0	C110	S	[null]	207	Worcester, MA
87	1	0		male	38.0	0	0	19972	0.0	[null]	S	[null]	[null]	Rotterdam, Netherlands
88	1	0		male	22.0	0	0	PC 17760	135.6333	[null]	C	[null]	232	[null]
89	1	0		male	[null]	0	0	PC 17757	227.525	[null]	C	[null]	[null]	[null]
90	1	0		male	31.0	0	0	PC 17590	50.4958	A24	S	[null]	[null]	Trenton, NJ
91	1	0		male	[null]	0	0	113767	50.0	A32	S	[null]	[null]	Seattle, WA
92	1	0		male	36.0	0	0	13049	40.125	A10	C	[null]	[null]	Winnipeg, MB
93	1	0		male	55.0	1	0	PC 17603	59.4	[null]	C	[null]	[null]	New York, NY
94	1	0		male	33.0	0	0	113790	26.55	[null]	S	[null]	109	London
95	1	0		male	61.0	1	3	PC 17608	262.375	B57 B59 B63 B66	C	[null]	[null]	Haverford, PA / Cooperstown, NY
96	1	0		male	50.0	1	0	13507	55.9	E44	S	[null]	[null]	Duluth, MN
97	1	0		male	56.0	0	0	113792	26.55	[null]	S	[null]	[null]	New York, NY
98	1	0		male	56.0	0	0	17764	30.6958	A7	C	[null]	[null]	St James, Long Island, NY
99	1	0		male	24.0	1	0	13695	60.0	C31	S	[null]	[null]	Huntington, WV
100	1	0		male	[null]	0	0	113056	26.0	A19	S	[null]	[null]	Streatham, Surrey

Rows: 1-100 | Columns: 14

You filter your data with a pandas-like syntax.

In [4]:

vdf[["name", "age"]][(vdf["age"] > 30) & (vdf["age"] < 60)]

Out[4]:

	Abc Varchar(164)	123 age Numeric(6,3)
1		39.0
2		47.0
3		36.0
4		45.0
5		42.0
6		41.0
7		48.0
8		45.0
9		33.0
10		49.0
11		36.0
12		46.0
13		47.0
14		37.0
15		39.0
16		31.0
17		50.0
18		39.0
19		36.0
20		37.0
21		47.0
22		38.0
23		46.0
24		45.0
25		40.0
26		55.0
27		42.0
28		55.0
29		42.0
30		50.0
31		46.0
32		50.0
33		32.5
34		58.0
35		41.0
36		46.0
37		54.0
38		44.0
39		55.0
40		47.0
41		37.0
42		58.0
43		45.5
44		47.0
45		38.0
46		31.0
47		36.0
48		55.0
49		33.0
50		50.0
51		56.0
52		56.0
53		57.0
54		52.0
55		49.0
56		40.0
57		47.0
58		54.0
59		57.0
60		50.0
61		51.0
62		48.0
63		53.0
64		50.0
65		32.0
66		37.0
67		47.0
68		42.0
69		35.0
70		40.0
71		58.0
72		45.0
73		44.0
74		59.0
75		41.0
76		42.0
77		53.0
78		36.0
79		58.0
80		36.0
81		36.0
82		47.0
83		33.0
84		36.0
85		45.0
86		39.0
87		55.0
88		36.0
89		38.0
90		51.0
91		33.0
92		31.0
93		53.0
94		54.0
95		48.0
96		38.0
97		54.0
98		50.0
99		43.0
100		48.0

Rows: 1-100 | Columns: 2

With the exception of aggregations, everything is happening in the database and nothing is loaded in memory. You can view the generated SQL code using the set_option function.

In [3]:

vp.set_option("sql_on", True)
vdf.describe()

Getting the version.

SELECT
version()

Computing the descriptive statistics of all numerical columns using SUMMARIZE_NUMCOL.

SELECT
SUMMARIZE_NUMCOL("pclass", "survived", "age", "sibsp", "parch", "fare", "body") OVER ()
FROM
"public"."titanic"

Out[3]:

	count	mean	std	min	approx_25%	approx_50%	approx_75%	max
"pclass"	1234	2.28444084278768	0.842485636190292	1.0	1.0	3.0	3.0	3.0
"survived"	1234	0.364667747163696	0.481532018641288	0.0	0.0	0.0	1.0	1.0
"age"	997	30.1524573721163	14.4353046299159	0.33	21.0	28.0	39.0	80.0
"sibsp"	1234	0.504051863857374	1.04111727241629	0.0	0.0	0.0	1.0	8.0
"parch"	1234	0.378444084278768	0.868604707790393	0.0	0.0	0.0	0.0	9.0
"fare"	1233	33.963793673966	52.6460729831293	0.0	7.8958	14.4542	31.3875	512.3292
"body"	118	164.14406779661	96.5760207557808	1.0	79.25	160.5	257.5	328.0

Rows: 1-7 | Columns: 9

VerticaPy is smart enough to not recompute the same aggregation twice. Each virtual column has its own catalog which will be updated as the user modifies it.

In [4]:

vdf["age"].catalog

Out[4]:

{'approx_25%': 21,
 'approx_50%': 28,
 'approx_75%': 39,
 'avg': 30.1524573721163,
 'biserial': {},
 'count': 997,
 'cov': {},
 'cramer': {},
 'kendall': {},
 'max': 80,
 'min': 0.33,
 'pearson': {},
 'regr_avgx': {},
 'regr_avgy': {},
 'regr_count': {},
 'regr_intercept': {},
 'regr_r2': {},
 'regr_slope': {},
 'regr_sxx': {},
 'regr_sxy': {},
 'regr_syy': {},
 'spearman': {},
 'spearmand': {},
 'std': 14.4353046299159}

Draw the dataset Correlation Matrix.

In [5]:

vdf.corr(method = "spearman")

Computing the spearman Corr Matrix.

SELECT
CORR_MATRIX("pclass", "survived", "age", "sibsp", "parch", "fare", "body") OVER ()
FROM
(
SELECT
RANK() OVER (ORDER BY "pclass") AS "pclass",
RANK() OVER (ORDER BY "survived") AS "survived",
RANK() OVER (ORDER BY "age") AS "age",
RANK() OVER (ORDER BY "sibsp") AS "sibsp",
RANK() OVER (ORDER BY "parch") AS "parch",
RANK() OVER (ORDER BY "fare") AS "fare",
RANK() OVER (ORDER BY "body") AS "body"
FROM
"public"."titanic") spearman_table

Out[5]:

	"pclass"	"survived"	"age"	"sibsp"	"parch"	"fare"	"body"
"pclass"	1.0	-0.335634606444129	-0.167006853613121	-0.0691851178284916	-0.0429882472107741	-0.722522164791555	0.0306003583215355
"survived"	-0.335634606444129	1.0	-0.0909763969172584	0.0934303518392484	0.171691455579124	0.322409613111481	0.211060721441923
"age"	-0.167006853613121	-0.0909763969172584	1.0	-0.158699476099267	-0.235344620947684	0.00451935857538285	-0.0206617211295797
"sibsp"	-0.0691851178284916	0.0934303518392484	-0.158699476099267	1.0	0.436524985907755	0.443942923967283	0.0332791733089436
"parch"	-0.0429882472107741	0.171691455579124	-0.235344620947684	0.436524985907755	1.0	0.402373369515066	0.0606076641278389
"fare"	-0.722522164791555	0.322409613111481	0.00451935857538285	0.443942923967283	0.402373369515066	1.0	-0.000856180961294435
"body"	0.0306003583215355	0.211060721441923	-0.0206617211295797	0.0332791733089436	0.0606076641278389	-0.000856180961294435	1.0

Rows: 1-7 | Columns: 8

We can turn off the SQL code generation.

In [7]:

vp.set_option("sql_on", False)

Use statistical tests.

In [8]:

from verticapy.stats import jarque_bera
jarque_bera(vdf, "age")

Out[8]:

	value
Jarque Bera Test Statistic	28.5338631758186
p_value	6.367227064092739e-07
# Observations Used	997.0
Kurtosis - 3	0.15689691331997
Skewness	0.408876460779437
Distribution Normality	❌

Rows: 1-6 | Columns: 2

Two lines code is enough to prepare your data and train/test your ML models.

In [10]:

# Modules
from verticapy.learn.model_selection import cross_validate
from verticapy.learn.ensemble import RandomForestClassifier

# Data Preparation
vdf["sex"].label_encode()["boat"].fillna(method = "0ifnull")["name"].str_extract(
    ' ([A-Za-z]+)\.').eval("family_size", expr = "parch + sibsp + 1").drop(
    columns = ["cabin", "body", "ticket", "home.dest"])["fare"].fill_outliers().fillna()

# Model Evaluation
cross_validate(RandomForestClassifier("rf_titanic", max_leaf_nodes = 100, n_estimators = 30), 
               vdf, 
               ["age", "family_size", "sex", "pclass", "fare", "boat"], 
               "survived", 
               cutoff = 0.35,
               cv = 3)

795 elements were filled.

Out[10]:

	auc	prc_auc	accuracy	log_loss	precision	recall	f1_score	mcc	informedness	markedness	csi	time
1-fold	0.9969699334165633	0.9937975211674981	0.975669099756691	0.031868369689969	0.971830985915493	0.9583333333333334	0.965034965034965	0.9464340719646614	0.9433520599250937	0.9495261532017385	0.9324324324324325	0.4893820285797119
2-fold	0.9936147523928432	0.9901599304516124	0.9732360097323601	0.0367951656894819	0.965034965034965	0.9583333333333334	0.9616724738675957	0.9411255958554565	0.9396067415730336	0.9426469053334725	0.9261744966442953	0.37854981422424316
3-fold	0.9947665056360722	0.9934248596508658	0.975669099756691	0.037334477995368	0.9699248120300752	0.9555555555555556	0.9626865671641791	0.9446954288644421	0.9410628019323672	0.9483420782171255	0.9280575539568345	0.3722829818725586
avg	0.9951170638151595	0.9924607704233255	0.9748580697485807	0.03533267112493963	0.9689302543268444	0.9574074074074075	0.9631313353555799	0.9440850322281867	0.9413405344768315	0.9468383789174455	0.9288881610111874	0.4134049415588379
std	0.0017048396414843588	0.002001278986815406	0.0014047451805100366	0.003012267047436987	0.0035054719683567464	0.0016037507477489547	0.0017248045601518517	0.002706366141279453	0.0018880423496999755	0.003677886145077642	0.003210587351430818	0.06587265450142515

Rows: 1-5 | Columns: 13

VerticaPy lets you to bring the logic to the data with just a few lines of code.

Enjoy!