Titanic¶

This example uses the 'titanic' dataset to predict the survival of passengers on the Titanic. You can download the Jupyter Notebook of the study here.

Initialization¶

This example uses the following version of VerticaPy:

In [1]:

import verticapy as vp
vp.__version__

Out[1]:

'0.9.0'

Connect to Vertica. This example uses an existing connection called "VerticaDSN." For details on how to create a connection, use see the connection tutorial.

In [2]:

vp.connect("VerticaDSN")

Use the following command to allow Matplotlib to display graphics.

In [3]:

%matplotlib inline

Let's load the dataset.

In [4]:

from verticapy.datasets import load_titanic
titanic = load_titanic()
display(titanic)

	123 pclass Int	123 survived Int	Abc Varchar(164)	Abc sex Varchar(20)	123 age Numeric(6,3)	123 sibsp Int	123 parch Int	Abc ticket Varchar(36)	123 fare Numeric(10,5)	Abc cabin Varchar(30)	Abc embarked Varchar(20)	Abc boat Varchar(100)	123 body Int	Abc home.dest Varchar(100)
1	1	0		female	2.0	1	2	113781	151.55	C22 C26	S	[null]	[null]	Montreal, PQ / Chesterville, ON
2	1	0		male	30.0	1	2	113781	151.55	C22 C26	S	[null]	135	Montreal, PQ / Chesterville, ON
3	1	0		female	25.0	1	2	113781	151.55	C22 C26	S	[null]	[null]	Montreal, PQ / Chesterville, ON
4	1	0		male	39.0	0	0	112050	0.0	A36	S	[null]	[null]	Belfast, NI
5	1	0		male	71.0	0	0	PC 17609	49.5042	[null]	C	[null]	22	Montevideo, Uruguay
6	1	0		male	47.0	1	0	PC 17757	227.525	C62 C64	C	[null]	124	New York, NY
7	1	0		male	[null]	0	0	PC 17318	25.925	[null]	S	[null]	[null]	New York, NY
8	1	0		male	24.0	0	1	PC 17558	247.5208	B58 B60	C	[null]	[null]	Montreal, PQ
9	1	0		male	36.0	0	0	13050	75.2417	C6	C	A	[null]	Winnipeg, MN
10	1	0		male	25.0	0	0	13905	26.0	[null]	C	[null]	148	San Francisco, CA
11	1	0		male	45.0	0	0	113784	35.5	T	S	[null]	[null]	Trenton, NJ
12	1	0		male	42.0	0	0	110489	26.55	D22	S	[null]	[null]	London / Winnipeg, MB
13	1	0		male	41.0	0	0	113054	30.5	A21	S	[null]	[null]	Pomeroy, WA
14	1	0		male	48.0	0	0	PC 17591	50.4958	B10	C	[null]	208	Omaha, NE
15	1	0		male	[null]	0	0	112379	39.6	[null]	C	[null]	[null]	Philadelphia, PA
16	1	0		male	45.0	0	0	113050	26.55	B38	S	[null]	[null]	Washington, DC
17	1	0		male	[null]	0	0	113798	31.0	[null]	S	[null]	[null]	[null]
18	1	0		male	33.0	0	0	695	5.0	B51 B53 B55	S	[null]	[null]	New York, NY
19	1	0		male	28.0	0	0	113059	47.1	[null]	S	[null]	[null]	Montevideo, Uruguay
20	1	0		male	17.0	0	0	113059	47.1	[null]	S	[null]	[null]	Montevideo, Uruguay
21	1	0		male	49.0	0	0	19924	26.0	[null]	S	[null]	[null]	Ascot, Berkshire / Rochester, NY
22	1	0		male	36.0	1	0	19877	78.85	C46	S	[null]	172	Little Onn Hall, Staffs
23	1	0		male	46.0	1	0	W.E.P. 5734	61.175	E31	S	[null]	[null]	Amenia, ND
24	1	0		male	[null]	0	0	112051	0.0	[null]	S	[null]	[null]	Liverpool, England / Belfast
25	1	0		male	27.0	1	0	13508	136.7792	C89	C	[null]	[null]	Los Angeles, CA
26	1	0		male	[null]	0	0	110465	52.0	A14	S	[null]	[null]	Stoughton, MA
27	1	0		male	47.0	0	0	5727	25.5875	E58	S	[null]	[null]	Victoria, BC
28	1	0		male	37.0	1	1	PC 17756	83.1583	E52	C	[null]	[null]	Lakewood, NJ
29	1	0		male	[null]	0	0	113791	26.55	[null]	S	[null]	[null]	Roachdale, IN
30	1	0		male	70.0	1	1	WE/P 5735	71.0	B22	S	[null]	269	Milwaukee, WI
31	1	0		male	39.0	1	0	PC 17599	71.2833	C85	C	[null]	[null]	New York, NY
32	1	0		male	31.0	1	0	F.C. 12750	52.0	B71	S	[null]	[null]	Montreal, PQ
33	1	0		male	50.0	1	0	PC 17761	106.425	C86	C	[null]	62	Deephaven, MN / Cedar Rapids, IA
34	1	0		male	39.0	0	0	PC 17580	29.7	A18	C	[null]	133	Philadelphia, PA
35	1	0		female	36.0	0	0	PC 17531	31.6792	A29	C	[null]	[null]	New York, NY
36	1	0		male	[null]	0	0	PC 17483	221.7792	C95	S	[null]	[null]	[null]
37	1	0		male	30.0	0	0	113051	27.75	C111	C	[null]	[null]	New York, NY
38	1	0		male	19.0	3	2	19950	263.0	C23 C25 C27	S	[null]	[null]	Winnipeg, MB
39	1	0		male	64.0	1	4	19950	263.0	C23 C25 C27	S	[null]	[null]	Winnipeg, MB
40	1	0		male	[null]	0	0	113778	26.55	D34	S	[null]	[null]	Westcliff-on-Sea, Essex
41	1	0		male	[null]	0	0	112058	0.0	B102	S	[null]	[null]	[null]
42	1	0		male	37.0	1	0	113803	53.1	C123	S	[null]	[null]	Scituate, MA
43	1	0		male	47.0	0	0	111320	38.5	E63	S	[null]	275	St Anne's-on-Sea, Lancashire
44	1	0		male	24.0	0	0	PC 17593	79.2	B86	C	[null]	[null]	[null]
45	1	0		male	71.0	0	0	PC 17754	34.6542	A5	C	[null]	[null]	New York, NY
46	1	0		male	38.0	0	1	PC 17582	153.4625	C91	S	[null]	147	Winnipeg, MB
47	1	0		male	46.0	0	0	PC 17593	79.2	B82 B84	C	[null]	[null]	New York, NY
48	1	0		male	[null]	0	0	113796	42.4	[null]	S	[null]	[null]	[null]
49	1	0		male	45.0	1	0	36973	83.475	C83	S	[null]	[null]	New York, NY
50	1	0		male	40.0	0	0	112059	0.0	B94	S	[null]	110	[null]
51	1	0		male	55.0	1	1	12749	93.5	B69	S	[null]	307	Montreal, PQ
52	1	0		male	42.0	0	0	113038	42.5	B11	S	[null]	[null]	London / Middlesex
53	1	0		male	[null]	0	0	17463	51.8625	E46	S	[null]	[null]	Brighton, MA
54	1	0		male	55.0	0	0	680	50.0	C39	S	[null]	[null]	London / Birmingham
55	1	0		male	42.0	1	0	113789	52.0	[null]	S	[null]	38	New York, NY
56	1	0		male	[null]	0	0	PC 17600	30.6958	[null]	C	14	[null]	New York, NY
57	1	0		female	50.0	0	0	PC 17595	28.7125	C49	C	[null]	[null]	Paris, France New York, NY
58	1	0		male	46.0	0	0	694	26.0	[null]	S	[null]	80	Bennington, VT
59	1	0		male	50.0	0	0	113044	26.0	E60	S	[null]	[null]	London
60	1	0		male	32.5	0	0	113503	211.5	C132	C	[null]	45	[null]
61	1	0		male	58.0	0	0	11771	29.7	B37	C	[null]	258	Buffalo, NY
62	1	0		male	41.0	1	0	17464	51.8625	D21	S	[null]	[null]	Southington / Noank, CT
63	1	0		male	[null]	0	0	113028	26.55	C124	S	[null]	[null]	Portland, OR
64	1	0		male	[null]	0	0	PC 17612	27.7208	[null]	C	[null]	[null]	Chicago, IL
65	1	0		male	29.0	0	0	113501	30.0	D6	S	[null]	126	Springfield, MA
66	1	0		male	30.0	0	0	113801	45.5	[null]	S	[null]	[null]	London / New York, NY
67	1	0		male	30.0	0	0	110469	26.0	C106	S	[null]	[null]	Brockton, MA
68	1	0		male	19.0	1	0	113773	53.1	D30	S	[null]	[null]	New York, NY
69	1	0		male	46.0	0	0	13050	75.2417	C6	C	[null]	292	Vancouver, BC
70	1	0		male	54.0	0	0	17463	51.8625	E46	S	[null]	175	Dorchester, MA
71	1	0		male	28.0	1	0	PC 17604	82.1708	[null]	C	[null]	[null]	New York, NY
72	1	0		male	65.0	0	0	13509	26.55	E38	S	[null]	249	East Bridgewater, MA
73	1	0		male	44.0	2	0	19928	90.0	C78	Q	[null]	230	Fond du Lac, WI
74	1	0		male	55.0	0	0	113787	30.5	C30	S	[null]	[null]	Montreal, PQ
75	1	0		male	47.0	0	0	113796	42.4	[null]	S	[null]	[null]	Washington, DC
76	1	0		male	37.0	0	1	PC 17596	29.7	C118	C	[null]	[null]	Brooklyn, NY
77	1	0		male	58.0	0	2	35273	113.275	D48	C	[null]	122	Lexington, MA
78	1	0		male	64.0	0	0	693	26.0	[null]	S	[null]	263	Isle of Wight, England
79	1	0		male	65.0	0	1	113509	61.9792	B30	C	[null]	234	Providence, RI
80	1	0		male	28.5	0	0	PC 17562	27.7208	D43	C	[null]	189	?Havana, Cuba
81	1	0		male	[null]	0	0	112052	0.0	[null]	S	[null]	[null]	Belfast
82	1	0		male	45.5	0	0	113043	28.5	C124	S	[null]	166	Surbiton Hill, Surrey
83	1	0		male	23.0	0	0	12749	93.5	B24	S	[null]	[null]	Montreal, PQ
84	1	0		male	29.0	1	0	113776	66.6	C2	S	[null]	[null]	Isleworth, England
85	1	0		male	18.0	1	0	PC 17758	108.9	C65	C	[null]	[null]	Madrid, Spain
86	1	0		male	47.0	0	0	110465	52.0	C110	S	[null]	207	Worcester, MA
87	1	0		male	38.0	0	0	19972	0.0	[null]	S	[null]	[null]	Rotterdam, Netherlands
88	1	0		male	22.0	0	0	PC 17760	135.6333	[null]	C	[null]	232	[null]
89	1	0		male	[null]	0	0	PC 17757	227.525	[null]	C	[null]	[null]	[null]
90	1	0		male	31.0	0	0	PC 17590	50.4958	A24	S	[null]	[null]	Trenton, NJ
91	1	0		male	[null]	0	0	113767	50.0	A32	S	[null]	[null]	Seattle, WA
92	1	0		male	36.0	0	0	13049	40.125	A10	C	[null]	[null]	Winnipeg, MB
93	1	0		male	55.0	1	0	PC 17603	59.4	[null]	C	[null]	[null]	New York, NY
94	1	0		male	33.0	0	0	113790	26.55	[null]	S	[null]	109	London
95	1	0		male	61.0	1	3	PC 17608	262.375	B57 B59 B63 B66	C	[null]	[null]	Haverford, PA / Cooperstown, NY
96	1	0		male	50.0	1	0	13507	55.9	E44	S	[null]	[null]	Duluth, MN
97	1	0		male	56.0	0	0	113792	26.55	[null]	S	[null]	[null]	New York, NY
98	1	0		male	56.0	0	0	17764	30.6958	A7	C	[null]	[null]	St James, Long Island, NY
99	1	0		male	24.0	1	0	13695	60.0	C31	S	[null]	[null]	Huntington, WV
100	1	0		male	[null]	0	0	113056	26.0	A19	S	[null]	[null]	Streatham, Surrey

Rows: 1-100 | Columns: 14

Data Exploration and Preparation¶

Let's explore the data by displaying descriptive statistics of all the columns.

In [5]:

titanic.describe(method = "categorical", unique = True)

Out[5]:

	dtype	count	top	top_percent	unique
"pclass"	int	1234	3	53.728	3.0
"survived"	int	1234	0	63.533	2.0
"name"	varchar(164)	1234	Kelly, Mr. James	0.162	1232.0
"sex"	varchar(20)	1234	male	65.964	2.0
"age"	numeric(6,3)	997	[null]	19.206	96.0
"sibsp"	int	1234	0	67.747	7.0
"parch"	int	1234	0	76.904	8.0
"ticket"	varchar(36)	1234	CA. 2343	0.81	887.0
"fare"	numeric(10,5)	1233	8.05	4.7	277.0
"cabin"	varchar(30)	286	[null]	76.823	182.0
"embarked"	varchar(20)	1232	S	70.746	3.0
"boat"	varchar(100)	439	[null]	64.425	26.0
"body"	int	118	[null]	90.438	118.0
"home.dest"	varchar(100)	706	[null]	42.788	359.0

Rows: 1-14 | Columns: 6

The columns "body" (passenger ID), "home.dest" (passenger origin/destination), "embarked" (origin port) and "ticket" (ticket ID) shouldn't influence survival, so we can ignore these.

Let's focus our analysis on the columns "name" and "cabin." We'll begin with the passengers' names.

In [6]:

from verticapy.learn.preprocessing import CountVectorizer
model = CountVectorizer("name_voc")
model.fit(titanic, ["Name"]).transform()

Out[6]:

	Abc token Varchar(128)	123 df Numeric(36,18)	123 cnt Integer	123 rnk Integer
1	mr	0.14816310052482842	734	1
2	miss	0.046023415421881306	228	2
3	mrs	0.03734356075898264	185	3
4	william	0.016148566814695196	80	4
5	john	0.013726281792490917	68	5
6	master	0.011303996770286637	56	6
7	henry	0.008881711748082359	44	7
8	james	0.007468712151796528	37	8
9	charles	0.007468712151796528	37	8
10	thomas	0.006863140896245458	34	10
11	joseph	0.006257569640694388	31	11
12	george	0.006257569640694388	31	11
13	edward	0.005450141299959629	27	13
14	mary	0.005248284214775939	26	14
15	johan	0.004440855874041179	22	15
16	elizabeth	0.004440855874041179	22	15
17	frederick	0.004037141703673799	20	17
18	samuel	0.003835284618490109	19	18
19	robert	0.003633427533306419	18	19
20	alexander	0.003633427533306419	18	19
21	arthur	0.003431570448122729	17	21
22	alfred	0.003431570448122729	17	21
23	richard	0.003229713362939039	16	23
24	maria	0.003229713362939039	16	23
25	anna	0.003229713362939039	16	23
26	karl	0.002825999192571659	14	26
27	jr	0.002825999192571659	14	26
28	albert	0.002825999192571659	14	26
29	peter	0.002624142107387969	13	29
30	j	0.002624142107387969	13	29
31	patrick	0.002422285022204279	12	31
32	frank	0.002422285022204279	12	31
33	ernest	0.002422285022204279	12	31
34	alice	0.002422285022204279	12	31
35	margaret	0.002220427937020589	11	35
36	h	0.002220427937020589	11	35
37	andersson	0.002220427937020589	11	35
38	walter	0.002018570851836899	10	38
39	sage	0.002018570851836899	10	38
40	martin	0.002018570851836899	10	38
41	leonard	0.002018570851836899	10	38
42	edith	0.00181671376665321	9	42
43	e	0.00181671376665321	9	42
44	y	0.00161485668146952	8	44
45	victor	0.00161485668146952	8	44
46	van	0.00161485668146952	8	44
47	rev	0.00161485668146952	8	44
48	harry	0.00161485668146952	8	44
49	goodwin	0.00161485668146952	8	44
50	francis	0.00161485668146952	8	44
51	elias	0.00161485668146952	8	44
52	dr	0.00161485668146952	8	44
53	david	0.00161485668146952	8	44
54	daniel	0.00161485668146952	8	44
55	asplund	0.00161485668146952	8	44
56	annie	0.00161485668146952	8	44
57	nils	0.00141299959628583	7	57
58	marie	0.00141299959628583	7	57
59	louise	0.00141299959628583	7	57
60	l	0.00141299959628583	7	57
61	helen	0.00141299959628583	7	57
62	ernst	0.00141299959628583	7	57
63	emil	0.00141299959628583	7	57
64	de	0.00141299959628583	7	57
65	davies	0.00141299959628583	7	57
66	carl	0.00141299959628583	7	57
67	brown	0.00141299959628583	7	57
68	bertram	0.00141299959628583	7	57
69	bertha	0.00141299959628583	7	57
70	benjamin	0.00141299959628583	7	57
71	august	0.00141299959628583	7	57
72	a	0.00141299959628583	7	57
73	sofia	0.00121114251110214	6	73
74	smith	0.00121114251110214	6	73
75	skoog	0.00121114251110214	6	73
76	rice	0.00121114251110214	6	73
77	panula	0.00121114251110214	6	73
78	olsen	0.00121114251110214	6	73
79	kate	0.00121114251110214	6	73
80	johnson	0.00121114251110214	6	73
81	johansson	0.00121114251110214	6	73
82	ivan	0.00121114251110214	6	73
83	ida	0.00121114251110214	6	73
84	hansen	0.00121114251110214	6	73
85	gustafsson	0.00121114251110214	6	73
86	fortune	0.00121114251110214	6	73
87	emily	0.00121114251110214	6	73
88	ellen	0.00121114251110214	6	73
89	douglas	0.00121114251110214	6	73
90	delia	0.00121114251110214	6	73
91	catherine	0.00121114251110214	6	73
92	carter	0.00121114251110214	6	73
93	bridget	0.00121114251110214	6	73
94	augusta	0.00121114251110214	6	73
95	ada	0.00121114251110214	6	73
96	williams	0.00100928542591845	5	96
97	white	0.00100928542591845	5	96
98	vander	0.00100928542591845	5	96
99	sidney	0.00100928542591845	5	96
100	philip	0.00100928542591845	5	96

Rows: 1-100 | Columns: 4

Passengers' titles might come in handy. We can extract these from their names.

Let's move on to the cabins.

In [7]:

model = CountVectorizer("cabin_voc")
model.fit("titanic", ["cabin"]).transform()

Out[7]:

	Abc token Varchar(128)	123 df Numeric(36,18)	123 cnt Integer	123 rnk Integer
1	[null]	0.7337461300309598	948	1
2	f	0.006191950464396285	8	2
3	c27	0.004643962848297214	6	3
4	c25	0.004643962848297214	6	3
5	c23	0.004643962848297214	6	3
6	g6	0.003869969040247678	5	6
7	f4	0.003095975232198142	4	7
8	d	0.003095975232198142	4	7
9	c78	0.003095975232198142	4	7
10	c26	0.003095975232198142	4	7
11	c22	0.003095975232198142	4	7
12	b98	0.003095975232198142	4	7
13	b96	0.003095975232198142	4	7
14	b66	0.003095975232198142	4	7
15	b63	0.003095975232198142	4	7
16	b59	0.003095975232198142	4	7
17	b57	0.003095975232198142	4	7
18	f33	0.002321981424148607	3	18
19	f2	0.002321981424148607	3	18
20	e46	0.002321981424148607	3	18
21	e34	0.002321981424148607	3	18
22	c101	0.002321981424148607	3	18
23	b60	0.002321981424148607	3	18
24	b58	0.002321981424148607	3	18
25	b55	0.002321981424148607	3	18
26	b53	0.002321981424148607	3	18
27	b51	0.002321981424148607	3	18
28	a34	0.002321981424148607	3	18
29	g73	0.001547987616099071	2	29
30	g63	0.001547987616099071	2	29
31	e8	0.001547987616099071	2	29
32	e67	0.001547987616099071	2	29
33	e50	0.001547987616099071	2	29
34	e44	0.001547987616099071	2	29
35	e33	0.001547987616099071	2	29
36	e31	0.001547987616099071	2	29
37	e24	0.001547987616099071	2	29
38	e121	0.001547987616099071	2	29
39	e101	0.001547987616099071	2	29
40	d37	0.001547987616099071	2	29
41	d36	0.001547987616099071	2	29
42	d35	0.001547987616099071	2	29
43	d33	0.001547987616099071	2	29
44	d30	0.001547987616099071	2	29
45	d28	0.001547987616099071	2	29
46	d26	0.001547987616099071	2	29
47	d21	0.001547987616099071	2	29
48	d20	0.001547987616099071	2	29
49	d19	0.001547987616099071	2	29
50	d17	0.001547987616099071	2	29
51	d15	0.001547987616099071	2	29
52	d12	0.001547987616099071	2	29
53	d10	0.001547987616099071	2	29
54	c93	0.001547987616099071	2	29
55	c92	0.001547987616099071	2	29
56	c89	0.001547987616099071	2	29
57	c86	0.001547987616099071	2	29
58	c85	0.001547987616099071	2	29
59	c83	0.001547987616099071	2	29
60	c80	0.001547987616099071	2	29
61	c7	0.001547987616099071	2	29
62	c68	0.001547987616099071	2	29
63	c65	0.001547987616099071	2	29
64	c64	0.001547987616099071	2	29
65	c62	0.001547987616099071	2	29
66	c6	0.001547987616099071	2	29
67	c57	0.001547987616099071	2	29
68	c55	0.001547987616099071	2	29
69	c54	0.001547987616099071	2	29
70	c52	0.001547987616099071	2	29
71	c46	0.001547987616099071	2	29
72	c32	0.001547987616099071	2	29
73	c31	0.001547987616099071	2	29
74	c2	0.001547987616099071	2	29
75	c126	0.001547987616099071	2	29
76	c125	0.001547987616099071	2	29
77	c124	0.001547987616099071	2	29
78	c123	0.001547987616099071	2	29
79	c116	0.001547987616099071	2	29
80	c106	0.001547987616099071	2	29
81	b78	0.001547987616099071	2	29
82	b77	0.001547987616099071	2	29
83	b71	0.001547987616099071	2	29
84	b69	0.001547987616099071	2	29
85	b5	0.001547987616099071	2	29
86	b49	0.001547987616099071	2	29
87	b45	0.001547987616099071	2	29
88	b41	0.001547987616099071	2	29
89	b35	0.001547987616099071	2	29
90	b28	0.001547987616099071	2	29
91	b22	0.001547987616099071	2	29
92	b20	0.001547987616099071	2	29
93	b18	0.001547987616099071	2	29
94	t	0.000773993808049536	1	94
95	f38	0.000773993808049536	1	94
96	e77	0.000773993808049536	1	94
97	e69	0.000773993808049536	1	94
98	e68	0.000773993808049536	1	94
99	e63	0.000773993808049536	1	94
100	e60	0.000773993808049536	1	94

Rows: 1-100 | Columns: 4

Here, we have the cabin IDs, the letter of which represents a certain position on the boat. Let's see how often each cabin occurs in the dataset.

In [8]:

CountVectorizer("cabin_voc").fit("titanic", ["cabin"]).transform(
                )["token"].str_slice(1, 1).groupby(
                columns = ["token"], expr = ["SUM(cnt)"]).head(30)

Out[8]:

	Abc token Varchar(4)	123 SUM Integer
1	[null]	948
2	a	20
3	b	92
4	c	113
5	d	47
6	e	43
7	f	19
8	g	9
9	t	1

Rows: 1-9 | Columns: 2

While NULL values for "boat" clearly represent passengers who have a dedicated "lifeboat," we can't be so sure about NULL values for "cabin". We can guess that these might represent passengers without a cabin. If this is the case, then these are missing values not at random (MNAR).

We'll revisit this problem later. For now, let's drop the columns that don't affect survival and then encode the rest.

In [9]:

titanic.drop(["body", "home.dest", "embarked", "ticket"])

Out[9]:

	123 pclass Int	123 survived Int	Abc Varchar(164)	Abc sex Varchar(20)	123 age Numeric(6,3)	123 sibsp Int	123 parch Int	123 fare Numeric(10,5)	Abc cabin Varchar(30)	Abc boat Varchar(100)
1	1	0		female	2.0	1	2	151.55	C22 C26	[null]
2	1	0		male	30.0	1	2	151.55	C22 C26	[null]
3	1	0		female	25.0	1	2	151.55	C22 C26	[null]
4	1	0		male	39.0	0	0	0.0	A36	[null]
5	1	0		male	71.0	0	0	49.5042	[null]	[null]
6	1	0		male	47.0	1	0	227.525	C62 C64	[null]
7	1	0		male	[null]	0	0	25.925	[null]	[null]
8	1	0		male	24.0	0	1	247.5208	B58 B60	[null]
9	1	0		male	36.0	0	0	75.2417	C6	A
10	1	0		male	25.0	0	0	26.0	[null]	[null]
11	1	0		male	45.0	0	0	35.5	T	[null]
12	1	0		male	42.0	0	0	26.55	D22	[null]
13	1	0		male	41.0	0	0	30.5	A21	[null]
14	1	0		male	48.0	0	0	50.4958	B10	[null]
15	1	0		male	[null]	0	0	39.6	[null]	[null]
16	1	0		male	45.0	0	0	26.55	B38	[null]
17	1	0		male	[null]	0	0	31.0	[null]	[null]
18	1	0		male	33.0	0	0	5.0	B51 B53 B55	[null]
19	1	0		male	28.0	0	0	47.1	[null]	[null]
20	1	0		male	17.0	0	0	47.1	[null]	[null]
21	1	0		male	49.0	0	0	26.0	[null]	[null]
22	1	0		male	36.0	1	0	78.85	C46	[null]
23	1	0		male	46.0	1	0	61.175	E31	[null]
24	1	0		male	[null]	0	0	0.0	[null]	[null]
25	1	0		male	27.0	1	0	136.7792	C89	[null]
26	1	0		male	[null]	0	0	52.0	A14	[null]
27	1	0		male	47.0	0	0	25.5875	E58	[null]
28	1	0		male	37.0	1	1	83.1583	E52	[null]
29	1	0		male	[null]	0	0	26.55	[null]	[null]
30	1	0		male	70.0	1	1	71.0	B22	[null]
31	1	0		male	39.0	1	0	71.2833	C85	[null]
32	1	0		male	31.0	1	0	52.0	B71	[null]
33	1	0		male	50.0	1	0	106.425	C86	[null]
34	1	0		male	39.0	0	0	29.7	A18	[null]
35	1	0		female	36.0	0	0	31.6792	A29	[null]
36	1	0		male	[null]	0	0	221.7792	C95	[null]
37	1	0		male	30.0	0	0	27.75	C111	[null]
38	1	0		male	19.0	3	2	263.0	C23 C25 C27	[null]
39	1	0		male	64.0	1	4	263.0	C23 C25 C27	[null]
40	1	0		male	[null]	0	0	26.55	D34	[null]
41	1	0		male	[null]	0	0	0.0	B102	[null]
42	1	0		male	37.0	1	0	53.1	C123	[null]
43	1	0		male	47.0	0	0	38.5	E63	[null]
44	1	0		male	24.0	0	0	79.2	B86	[null]
45	1	0		male	71.0	0	0	34.6542	A5	[null]
46	1	0		male	38.0	0	1	153.4625	C91	[null]
47	1	0		male	46.0	0	0	79.2	B82 B84	[null]
48	1	0		male	[null]	0	0	42.4	[null]	[null]
49	1	0		male	45.0	1	0	83.475	C83	[null]
50	1	0		male	40.0	0	0	0.0	B94	[null]
51	1	0		male	55.0	1	1	93.5	B69	[null]
52	1	0		male	42.0	0	0	42.5	B11	[null]
53	1	0		male	[null]	0	0	51.8625	E46	[null]
54	1	0		male	55.0	0	0	50.0	C39	[null]
55	1	0		male	42.0	1	0	52.0	[null]	[null]
56	1	0		male	[null]	0	0	30.6958	[null]	14
57	1	0		female	50.0	0	0	28.7125	C49	[null]
58	1	0		male	46.0	0	0	26.0	[null]	[null]
59	1	0		male	50.0	0	0	26.0	E60	[null]
60	1	0		male	32.5	0	0	211.5	C132	[null]
61	1	0		male	58.0	0	0	29.7	B37	[null]
62	1	0		male	41.0	1	0	51.8625	D21	[null]
63	1	0		male	[null]	0	0	26.55	C124	[null]
64	1	0		male	[null]	0	0	27.7208	[null]	[null]
65	1	0		male	29.0	0	0	30.0	D6	[null]
66	1	0		male	30.0	0	0	45.5	[null]	[null]
67	1	0		male	30.0	0	0	26.0	C106	[null]
68	1	0		male	19.0	1	0	53.1	D30	[null]
69	1	0		male	46.0	0	0	75.2417	C6	[null]
70	1	0		male	54.0	0	0	51.8625	E46	[null]
71	1	0		male	28.0	1	0	82.1708	[null]	[null]
72	1	0		male	65.0	0	0	26.55	E38	[null]
73	1	0		male	44.0	2	0	90.0	C78	[null]
74	1	0		male	55.0	0	0	30.5	C30	[null]
75	1	0		male	47.0	0	0	42.4	[null]	[null]
76	1	0		male	37.0	0	1	29.7	C118	[null]
77	1	0		male	58.0	0	2	113.275	D48	[null]
78	1	0		male	64.0	0	0	26.0	[null]	[null]
79	1	0		male	65.0	0	1	61.9792	B30	[null]
80	1	0		male	28.5	0	0	27.7208	D43	[null]
81	1	0		male	[null]	0	0	0.0	[null]	[null]
82	1	0		male	45.5	0	0	28.5	C124	[null]
83	1	0		male	23.0	0	0	93.5	B24	[null]
84	1	0		male	29.0	1	0	66.6	C2	[null]
85	1	0		male	18.0	1	0	108.9	C65	[null]
86	1	0		male	47.0	0	0	52.0	C110	[null]
87	1	0		male	38.0	0	0	0.0	[null]	[null]
88	1	0		male	22.0	0	0	135.6333	[null]	[null]
89	1	0		male	[null]	0	0	227.525	[null]	[null]
90	1	0		male	31.0	0	0	50.4958	A24	[null]
91	1	0		male	[null]	0	0	50.0	A32	[null]
92	1	0		male	36.0	0	0	40.125	A10	[null]
93	1	0		male	55.0	1	0	59.4	[null]	[null]
94	1	0		male	33.0	0	0	26.55	[null]	[null]
95	1	0		male	61.0	1	3	262.375	B57 B59 B63 B66	[null]
96	1	0		male	50.0	1	0	55.9	E44	[null]
97	1	0		male	56.0	0	0	26.55	[null]	[null]
98	1	0		male	56.0	0	0	30.6958	A7	[null]
99	1	0		male	24.0	1	0	60.0	C31	[null]
100	1	0		male	[null]	0	0	26.0	A19	[null]

Rows: 1-100 of 1234 | Columns: 10

In [10]:

titanic["cabin"].str_slice(1, 1)["name"].str_extract(
        ' ([A-Za-z]+)\.')["boat"].fillna(
        method = "0ifnull")["cabin"].fillna("No Cabin")

795 elements were filled.
948 elements were filled.

Out[10]:

	123 pclass Int	123 survived Int	Abc name Varchar(164)	Abc sex Varchar(20)	123 age Numeric(6,3)	123 sibsp Int	123 parch Int	123 fare Numeric(10,5)	Abc cabin Varchar(4)	123 boat Bool
1	1	0	Miss.	female	2.0	1	2	151.55	C	0
2	1	0	Mr.	male	30.0	1	2	151.55	C	0
3	1	0	Mrs.	female	25.0	1	2	151.55	C	0
4	1	0	Mr.	male	39.0	0	0	0.0	A	0
5	1	0	Mr.	male	71.0	0	0	49.5042	No Cabin	0
6	1	0	Col.	male	47.0	1	0	227.525	C	0
7	1	0	Mr.	male	[null]	0	0	25.925	No Cabin	0
8	1	0	Mr.	male	24.0	0	1	247.5208	B	0
9	1	0	Mr.	male	36.0	0	0	75.2417	C	1
10	1	0	Mr.	male	25.0	0	0	26.0	No Cabin	0
11	1	0	Mr.	male	45.0	0	0	35.5	T	0
12	1	0	Mr.	male	42.0	0	0	26.55	D	0
13	1	0	Mr.	male	41.0	0	0	30.5	A	0
14	1	0	Mr.	male	48.0	0	0	50.4958	B	0
15	1	0	Dr.	male	[null]	0	0	39.6	No Cabin	0
16	1	0	Major.	male	45.0	0	0	26.55	B	0
17	1	0	Mr.	male	[null]	0	0	31.0	No Cabin	0
18	1	0	Mr.	male	33.0	0	0	5.0	B	0
19	1	0	Mr.	male	28.0	0	0	47.1	No Cabin	0
20	1	0	Mr.	male	17.0	0	0	47.1	No Cabin	0
21	1	0	Mr.	male	49.0	0	0	26.0	No Cabin	0
22	1	0	Mr.	male	36.0	1	0	78.85	C	0
23	1	0	Mr.	male	46.0	1	0	61.175	E	0
24	1	0	Mr.	male	[null]	0	0	0.0	No Cabin	0
25	1	0	Mr.	male	27.0	1	0	136.7792	C	0
26	1	0	Mr.	male	[null]	0	0	52.0	A	0
27	1	0	Mr.	male	47.0	0	0	25.5875	E	0
28	1	0	Mr.	male	37.0	1	1	83.1583	E	0
29	1	0	Mr.	male	[null]	0	0	26.55	No Cabin	0
30	1	0	Capt.	male	70.0	1	1	71.0	B	0
31	1	0	Mr.	male	39.0	1	0	71.2833	C	0
32	1	0	Mr.	male	31.0	1	0	52.0	B	0
33	1	0	Mr.	male	50.0	1	0	106.425	C	0
34	1	0	Mr.	male	39.0	0	0	29.7	A	0
35	1	0	Miss.	female	36.0	0	0	31.6792	A	0
36	1	0	Mr.	male	[null]	0	0	221.7792	C	0
37	1	0	Mr.	male	30.0	0	0	27.75	C	0
38	1	0	Mr.	male	19.0	3	2	263.0	C	0
39	1	0	Mr.	male	64.0	1	4	263.0	C	0
40	1	0	Mr.	male	[null]	0	0	26.55	D	0
41	1	0	Mr.	male	[null]	0	0	0.0	B	0
42	1	0	Mr.	male	37.0	1	0	53.1	C	0
43	1	0	Mr.	male	47.0	0	0	38.5	E	0
44	1	0	Mr.	male	24.0	0	0	79.2	B	0
45	1	0	Mr.	male	71.0	0	0	34.6542	A	0
46	1	0	Mr.	male	38.0	0	1	153.4625	C	0
47	1	0	Mr.	male	46.0	0	0	79.2	B	0
48	1	0	Mr.	male	[null]	0	0	42.4	No Cabin	0
49	1	0	Mr.	male	45.0	1	0	83.475	C	0
50	1	0	Mr.	male	40.0	0	0	0.0	B	0
51	1	0	Mr.	male	55.0	1	1	93.5	B	0
52	1	0	Mr.	male	42.0	0	0	42.5	B	0
53	1	0	Mr.	male	[null]	0	0	51.8625	E	0
54	1	0	Mr.	male	55.0	0	0	50.0	C	0
55	1	0	Mr.	male	42.0	1	0	52.0	No Cabin	0
56	1	0	Mr.	male	[null]	0	0	30.6958	No Cabin	1
57	1	0	Miss.	female	50.0	0	0	28.7125	C	0
58	1	0	Mr.	male	46.0	0	0	26.0	No Cabin	0
59	1	0	Mr.	male	50.0	0	0	26.0	E	0
60	1	0	Mr.	male	32.5	0	0	211.5	C	0
61	1	0	Mr.	male	58.0	0	0	29.7	B	0
62	1	0	Mr.	male	41.0	1	0	51.8625	D	0
63	1	0	Mr.	male	[null]	0	0	26.55	C	0
64	1	0	Mr.	male	[null]	0	0	27.7208	No Cabin	0
65	1	0	Mr.	male	29.0	0	0	30.0	D	0
66	1	0	Mr.	male	30.0	0	0	45.5	No Cabin	0
67	1	0	Mr.	male	30.0	0	0	26.0	C	0
68	1	0	Mr.	male	19.0	1	0	53.1	D	0
69	1	0	Mr.	male	46.0	0	0	75.2417	C	0
70	1	0	Mr.	male	54.0	0	0	51.8625	E	0
71	1	0	Mr.	male	28.0	1	0	82.1708	No Cabin	0
72	1	0	Mr.	male	65.0	0	0	26.55	E	0
73	1	0	Dr.	male	44.0	2	0	90.0	C	0
74	1	0	Mr.	male	55.0	0	0	30.5	C	0
75	1	0	Mr.	male	47.0	0	0	42.4	No Cabin	0
76	1	0	Mr.	male	37.0	0	1	29.7	C	0
77	1	0	Mr.	male	58.0	0	2	113.275	D	0
78	1	0	Mr.	male	64.0	0	0	26.0	No Cabin	0
79	1	0	Mr.	male	65.0	0	1	61.9792	B	0
80	1	0	Mr.	male	28.5	0	0	27.7208	D	0
81	1	0	Mr.	male	[null]	0	0	0.0	No Cabin	0
82	1	0	Mr.	male	45.5	0	0	28.5	C	0
83	1	0	Mr.	male	23.0	0	0	93.5	B	0
84	1	0	Mr.	male	29.0	1	0	66.6	C	0
85	1	0	Mr.	male	18.0	1	0	108.9	C	0
86	1	0	Mr.	male	47.0	0	0	52.0	C	0
87	1	0	Jonkheer.	male	38.0	0	0	0.0	No Cabin	0
88	1	0	Mr.	male	22.0	0	0	135.6333	No Cabin	0
89	1	0	Mr.	male	[null]	0	0	227.525	No Cabin	0
90	1	0	Mr.	male	31.0	0	0	50.4958	A	0
91	1	0	Mr.	male	[null]	0	0	50.0	A	0
92	1	0	Mr.	male	36.0	0	0	40.125	A	0
93	1	0	Mr.	male	55.0	1	0	59.4	No Cabin	0
94	1	0	Mr.	male	33.0	0	0	26.55	No Cabin	0
95	1	0	Mr.	male	61.0	1	3	262.375	B	0
96	1	0	Mr.	male	50.0	1	0	55.9	E	0
97	1	0	Mr.	male	56.0	0	0	26.55	No Cabin	0
98	1	0	Mr.	male	56.0	0	0	30.6958	A	0
99	1	0	Mr.	male	24.0	1	0	60.0	C	0
100	1	0	Mr.	male	[null]	0	0	26.0	A	0

Rows: 1-100 of 1234 | Columns: 10

Looking at our data now, we can see that some first class passengers have a NULL value for their cabin, so we can safely say that our assumption about the meaning of a NULL value of "cabin" turned out to be incorrect. This means that the "cabin" column has far too many missing values at random (MAR). We'll have to drop it.

In [11]:

titanic["cabin"].drop()

Out[11]:

	123 pclass Int	123 survived Int	Abc name Varchar(164)	Abc sex Varchar(20)	123 age Numeric(6,3)	123 sibsp Int	123 parch Int	123 fare Numeric(10,5)	123 boat Bool
1	1	0	Miss.	female	2.0	1	2	151.55	0
2	1	0	Mr.	male	30.0	1	2	151.55	0
3	1	0	Mrs.	female	25.0	1	2	151.55	0
4	1	0	Mr.	male	39.0	0	0	0.0	0
5	1	0	Mr.	male	71.0	0	0	49.5042	0
6	1	0	Col.	male	47.0	1	0	227.525	0
7	1	0	Mr.	male	[null]	0	0	25.925	0
8	1	0	Mr.	male	24.0	0	1	247.5208	0
9	1	0	Mr.	male	36.0	0	0	75.2417	1
10	1	0	Mr.	male	25.0	0	0	26.0	0
11	1	0	Mr.	male	45.0	0	0	35.5	0
12	1	0	Mr.	male	42.0	0	0	26.55	0
13	1	0	Mr.	male	41.0	0	0	30.5	0
14	1	0	Mr.	male	48.0	0	0	50.4958	0
15	1	0	Dr.	male	[null]	0	0	39.6	0
16	1	0	Major.	male	45.0	0	0	26.55	0
17	1	0	Mr.	male	[null]	0	0	31.0	0
18	1	0	Mr.	male	33.0	0	0	5.0	0
19	1	0	Mr.	male	28.0	0	0	47.1	0
20	1	0	Mr.	male	17.0	0	0	47.1	0
21	1	0	Mr.	male	49.0	0	0	26.0	0
22	1	0	Mr.	male	36.0	1	0	78.85	0
23	1	0	Mr.	male	46.0	1	0	61.175	0
24	1	0	Mr.	male	[null]	0	0	0.0	0
25	1	0	Mr.	male	27.0	1	0	136.7792	0
26	1	0	Mr.	male	[null]	0	0	52.0	0
27	1	0	Mr.	male	47.0	0	0	25.5875	0
28	1	0	Mr.	male	37.0	1	1	83.1583	0
29	1	0	Mr.	male	[null]	0	0	26.55	0
30	1	0	Capt.	male	70.0	1	1	71.0	0
31	1	0	Mr.	male	39.0	1	0	71.2833	0
32	1	0	Mr.	male	31.0	1	0	52.0	0
33	1	0	Mr.	male	50.0	1	0	106.425	0
34	1	0	Mr.	male	39.0	0	0	29.7	0
35	1	0	Miss.	female	36.0	0	0	31.6792	0
36	1	0	Mr.	male	[null]	0	0	221.7792	0
37	1	0	Mr.	male	30.0	0	0	27.75	0
38	1	0	Mr.	male	19.0	3	2	263.0	0
39	1	0	Mr.	male	64.0	1	4	263.0	0
40	1	0	Mr.	male	[null]	0	0	26.55	0
41	1	0	Mr.	male	[null]	0	0	0.0	0
42	1	0	Mr.	male	37.0	1	0	53.1	0
43	1	0	Mr.	male	47.0	0	0	38.5	0
44	1	0	Mr.	male	24.0	0	0	79.2	0
45	1	0	Mr.	male	71.0	0	0	34.6542	0
46	1	0	Mr.	male	38.0	0	1	153.4625	0
47	1	0	Mr.	male	46.0	0	0	79.2	0
48	1	0	Mr.	male	[null]	0	0	42.4	0
49	1	0	Mr.	male	45.0	1	0	83.475	0
50	1	0	Mr.	male	40.0	0	0	0.0	0
51	1	0	Mr.	male	55.0	1	1	93.5	0
52	1	0	Mr.	male	42.0	0	0	42.5	0
53	1	0	Mr.	male	[null]	0	0	51.8625	0
54	1	0	Mr.	male	55.0	0	0	50.0	0
55	1	0	Mr.	male	42.0	1	0	52.0	0
56	1	0	Mr.	male	[null]	0	0	30.6958	1
57	1	0	Miss.	female	50.0	0	0	28.7125	0
58	1	0	Mr.	male	46.0	0	0	26.0	0
59	1	0	Mr.	male	50.0	0	0	26.0	0
60	1	0	Mr.	male	32.5	0	0	211.5	0
61	1	0	Mr.	male	58.0	0	0	29.7	0
62	1	0	Mr.	male	41.0	1	0	51.8625	0
63	1	0	Mr.	male	[null]	0	0	26.55	0
64	1	0	Mr.	male	[null]	0	0	27.7208	0
65	1	0	Mr.	male	29.0	0	0	30.0	0
66	1	0	Mr.	male	30.0	0	0	45.5	0
67	1	0	Mr.	male	30.0	0	0	26.0	0
68	1	0	Mr.	male	19.0	1	0	53.1	0
69	1	0	Mr.	male	46.0	0	0	75.2417	0
70	1	0	Mr.	male	54.0	0	0	51.8625	0
71	1	0	Mr.	male	28.0	1	0	82.1708	0
72	1	0	Mr.	male	65.0	0	0	26.55	0
73	1	0	Dr.	male	44.0	2	0	90.0	0
74	1	0	Mr.	male	55.0	0	0	30.5	0
75	1	0	Mr.	male	47.0	0	0	42.4	0
76	1	0	Mr.	male	37.0	0	1	29.7	0
77	1	0	Mr.	male	58.0	0	2	113.275	0
78	1	0	Mr.	male	64.0	0	0	26.0	0
79	1	0	Mr.	male	65.0	0	1	61.9792	0
80	1	0	Mr.	male	28.5	0	0	27.7208	0
81	1	0	Mr.	male	[null]	0	0	0.0	0
82	1	0	Mr.	male	45.5	0	0	28.5	0
83	1	0	Mr.	male	23.0	0	0	93.5	0
84	1	0	Mr.	male	29.0	1	0	66.6	0
85	1	0	Mr.	male	18.0	1	0	108.9	0
86	1	0	Mr.	male	47.0	0	0	52.0	0
87	1	0	Jonkheer.	male	38.0	0	0	0.0	0
88	1	0	Mr.	male	22.0	0	0	135.6333	0
89	1	0	Mr.	male	[null]	0	0	227.525	0
90	1	0	Mr.	male	31.0	0	0	50.4958	0
91	1	0	Mr.	male	[null]	0	0	50.0	0
92	1	0	Mr.	male	36.0	0	0	40.125	0
93	1	0	Mr.	male	55.0	1	0	59.4	0
94	1	0	Mr.	male	33.0	0	0	26.55	0
95	1	0	Mr.	male	61.0	1	3	262.375	0
96	1	0	Mr.	male	50.0	1	0	55.9	0
97	1	0	Mr.	male	56.0	0	0	26.55	0
98	1	0	Mr.	male	56.0	0	0	30.6958	0
99	1	0	Mr.	male	24.0	1	0	60.0	0
100	1	0	Mr.	male	[null]	0	0	26.0	0

Rows: 1-100 of 1234 | Columns: 9

Let's look at descriptive statistics of the entire Virtual Dataframe.

In [12]:

titanic.describe(method = "all")

Out[12]:

	123 "pclass" Int 100%	123 "survived" Int 100%	123 "age" Numeric(6,3) 80%	123 "sibsp" Int 100%	123 "parch" Int 100%	123 "fare" Numeric(10,5) 99%	123 "boat" Bool 100%	Abc "name" Varchar(164) 100%	Abc "sex" Varchar(20) 100%
dtype	int	int	numeric(6,3)	int	int	numeric(10,5)	bool	varchar(164)	varchar(20)
percent	100.0	100.0	80.794	100.0	100.0	99.919	100.0	100.0	100.0
count	1234	1234	997	1234	1234	1233	1234.0	1234	1234
top	3	0	[null]	0	0	8.05	0	Mr.	male
top_percent	53.728	63.533	19.206	67.747	76.904	4.7	64.425	59.481	65.964
avg	2.28444084278768	0.364667747163695	30.1524573721163	0.504051863857374	0.378444084278768	33.9637936739659	0.355753646677472	4.73176661264181	4.6807131280389
stddev	0.842485636190292	0.481532018641288	14.4353046299159	1.04111727241629	0.868604707790392	52.6460729831293	0.478935143777661	1.08376455904966	0.948042321667023
min	1	0	0.33	0	0	0.0	0	4	4
approx_25%	1.0	0.0	21.0	0.0	0.0	7.8958	0.0	4	4
approx_50%	3.0	0.0	28.0	0.0	0.0	14.4542	0.0	4	4
approx_75%	3.0	1.0	39.0	1.0	0.0	31.3875	1.0	5	6
max	3	1	80.0	8	9	512.3292	1	10	6
range	2	1	79.67	8	9	512.3292	1	6	2
empty	[null]	[null]	[null]	[null]	[null]	[null]	[null]	0	0

Rows: 1-14 | Columns: 10

Descriptive statistics can give us valuable insights into our data. Notice, for example, that the column "fare" has many outliers (The maximum of 512.33 is much greater than the 9th decile of 79.13). Most passengers traveled in 3rd class (median of pclass = 3).

The "sibsp" column represents the number of siblings for each passenger, while the "parch" column represents the number of parents and children. We can use these to create a new feature: "family_size".

In [13]:

titanic["family_size"] = titanic["parch"] + titanic["sibsp"] + 1

Let's move on to outliers. We have several tools for locating outliers (LocalOutlier Factor, DBSCAN, k-means...), but we'll just use winsorization in this example. Again, "fare" has many outliers, so we'll start there.

In [14]:

titanic["fare"].fill_outliers(method = "winsorize", 
                              alpha = 0.03)

Out[14]:

	123 pclass Int	123 survived Int	Abc name Varchar(164)	Abc sex Varchar(20)	123 age Numeric(6,3)	123 sibsp Int	123 parch Int	123 fare Numeric(18,13)	123 boat Bool	123 family_size Integer
1	1	0	Miss.	female	2.0	1	2	151.55	0	4
2	1	0	Mr.	male	30.0	1	2	151.55	0	4
3	1	0	Mrs.	female	25.0	1	2	151.55	0	4
4	1	0	Mr.	male	39.0	0	0	0.0	0	1
5	1	0	Mr.	male	71.0	0	0	49.5042	0	1
6	1	0	Col.	male	47.0	1	0	227.525	0	2
7	1	0	Mr.	male	[null]	0	0	25.925	0	1
8	1	0	Mr.	male	24.0	0	1	244.5480856064831	0	2
9	1	0	Mr.	male	36.0	0	0	75.2417	1	1
10	1	0	Mr.	male	25.0	0	0	26.0	0	1
11	1	0	Mr.	male	45.0	0	0	35.5	0	1
12	1	0	Mr.	male	42.0	0	0	26.55	0	1
13	1	0	Mr.	male	41.0	0	0	30.5	0	1
14	1	0	Mr.	male	48.0	0	0	50.4958	0	1
15	1	0	Dr.	male	[null]	0	0	39.6	0	1
16	1	0	Major.	male	45.0	0	0	26.55	0	1
17	1	0	Mr.	male	[null]	0	0	31.0	0	1
18	1	0	Mr.	male	33.0	0	0	5.0	0	1
19	1	0	Mr.	male	28.0	0	0	47.1	0	1
20	1	0	Mr.	male	17.0	0	0	47.1	0	1
21	1	0	Mr.	male	49.0	0	0	26.0	0	1
22	1	0	Mr.	male	36.0	1	0	78.85	0	2
23	1	0	Mr.	male	46.0	1	0	61.175	0	2
24	1	0	Mr.	male	[null]	0	0	0.0	0	1
25	1	0	Mr.	male	27.0	1	0	136.7792	0	2
26	1	0	Mr.	male	[null]	0	0	52.0	0	1
27	1	0	Mr.	male	47.0	0	0	25.5875	0	1
28	1	0	Mr.	male	37.0	1	1	83.1583	0	3
29	1	0	Mr.	male	[null]	0	0	26.55	0	1
30	1	0	Capt.	male	70.0	1	1	71.0	0	3
31	1	0	Mr.	male	39.0	1	0	71.2833	0	2
32	1	0	Mr.	male	31.0	1	0	52.0	0	2
33	1	0	Mr.	male	50.0	1	0	106.425	0	2
34	1	0	Mr.	male	39.0	0	0	29.7	0	1
35	1	0	Miss.	female	36.0	0	0	31.6792	0	1
36	1	0	Mr.	male	[null]	0	0	221.7792	0	1
37	1	0	Mr.	male	30.0	0	0	27.75	0	1
38	1	0	Mr.	male	19.0	3	2	244.5480856064831	0	6
39	1	0	Mr.	male	64.0	1	4	244.5480856064831	0	6
40	1	0	Mr.	male	[null]	0	0	26.55	0	1
41	1	0	Mr.	male	[null]	0	0	0.0	0	1
42	1	0	Mr.	male	37.0	1	0	53.1	0	2
43	1	0	Mr.	male	47.0	0	0	38.5	0	1
44	1	0	Mr.	male	24.0	0	0	79.2	0	1
45	1	0	Mr.	male	71.0	0	0	34.6542	0	1
46	1	0	Mr.	male	38.0	0	1	153.4625	0	2
47	1	0	Mr.	male	46.0	0	0	79.2	0	1
48	1	0	Mr.	male	[null]	0	0	42.4	0	1
49	1	0	Mr.	male	45.0	1	0	83.475	0	2
50	1	0	Mr.	male	40.0	0	0	0.0	0	1
51	1	0	Mr.	male	55.0	1	1	93.5	0	3
52	1	0	Mr.	male	42.0	0	0	42.5	0	1
53	1	0	Mr.	male	[null]	0	0	51.8625	0	1
54	1	0	Mr.	male	55.0	0	0	50.0	0	1
55	1	0	Mr.	male	42.0	1	0	52.0	0	2
56	1	0	Mr.	male	[null]	0	0	30.6958	1	1
57	1	0	Miss.	female	50.0	0	0	28.7125	0	1
58	1	0	Mr.	male	46.0	0	0	26.0	0	1
59	1	0	Mr.	male	50.0	0	0	26.0	0	1
60	1	0	Mr.	male	32.5	0	0	211.5	0	1
61	1	0	Mr.	male	58.0	0	0	29.7	0	1
62	1	0	Mr.	male	41.0	1	0	51.8625	0	2
63	1	0	Mr.	male	[null]	0	0	26.55	0	1
64	1	0	Mr.	male	[null]	0	0	27.7208	0	1
65	1	0	Mr.	male	29.0	0	0	30.0	0	1
66	1	0	Mr.	male	30.0	0	0	45.5	0	1
67	1	0	Mr.	male	30.0	0	0	26.0	0	1
68	1	0	Mr.	male	19.0	1	0	53.1	0	2
69	1	0	Mr.	male	46.0	0	0	75.2417	0	1
70	1	0	Mr.	male	54.0	0	0	51.8625	0	1
71	1	0	Mr.	male	28.0	1	0	82.1708	0	2
72	1	0	Mr.	male	65.0	0	0	26.55	0	1
73	1	0	Dr.	male	44.0	2	0	90.0	0	3
74	1	0	Mr.	male	55.0	0	0	30.5	0	1
75	1	0	Mr.	male	47.0	0	0	42.4	0	1
76	1	0	Mr.	male	37.0	0	1	29.7	0	2
77	1	0	Mr.	male	58.0	0	2	113.275	0	3
78	1	0	Mr.	male	64.0	0	0	26.0	0	1
79	1	0	Mr.	male	65.0	0	1	61.9792	0	2
80	1	0	Mr.	male	28.5	0	0	27.7208	0	1
81	1	0	Mr.	male	[null]	0	0	0.0	0	1
82	1	0	Mr.	male	45.5	0	0	28.5	0	1
83	1	0	Mr.	male	23.0	0	0	93.5	0	1
84	1	0	Mr.	male	29.0	1	0	66.6	0	2
85	1	0	Mr.	male	18.0	1	0	108.9	0	2
86	1	0	Mr.	male	47.0	0	0	52.0	0	1
87	1	0	Jonkheer.	male	38.0	0	0	0.0	0	1
88	1	0	Mr.	male	22.0	0	0	135.6333	0	1
89	1	0	Mr.	male	[null]	0	0	227.525	0	1
90	1	0	Mr.	male	31.0	0	0	50.4958	0	1
91	1	0	Mr.	male	[null]	0	0	50.0	0	1
92	1	0	Mr.	male	36.0	0	0	40.125	0	1
93	1	0	Mr.	male	55.0	1	0	59.4	0	2
94	1	0	Mr.	male	33.0	0	0	26.55	0	1
95	1	0	Mr.	male	61.0	1	3	244.5480856064831	0	5
96	1	0	Mr.	male	50.0	1	0	55.9	0	2
97	1	0	Mr.	male	56.0	0	0	26.55	0	1
98	1	0	Mr.	male	56.0	0	0	30.6958	0	1
99	1	0	Mr.	male	24.0	1	0	60.0	0	2
100	1	0	Mr.	male	[null]	0	0	26.0	0	1

Rows: 1-100 | Columns: 10

Let's encode the column "sex" so we can use it with numerical methods.

In [15]:

titanic["sex"].label_encode()

Out[15]:

	123 pclass Int	123 survived Int	Abc name Varchar(164)	123 sex Int	123 age Numeric(6,3)	123 sibsp Int	123 parch Int	123 fare Numeric(18,13)	123 boat Bool	123 family_size Integer
1	1	0	Miss.	0	2.0	1	2	151.55	0	4
2	1	0	Mr.	1	30.0	1	2	151.55	0	4
3	1	0	Mrs.	0	25.0	1	2	151.55	0	4
4	1	0	Mr.	1	39.0	0	0	0.0	0	1
5	1	0	Mr.	1	71.0	0	0	49.5042	0	1
6	1	0	Col.	1	47.0	1	0	227.525	0	2
7	1	0	Mr.	1	[null]	0	0	25.925	0	1
8	1	0	Mr.	1	24.0	0	1	244.5480856064831	0	2
9	1	0	Mr.	1	36.0	0	0	75.2417	1	1
10	1	0	Mr.	1	25.0	0	0	26.0	0	1
11	1	0	Mr.	1	45.0	0	0	35.5	0	1
12	1	0	Mr.	1	42.0	0	0	26.55	0	1
13	1	0	Mr.	1	41.0	0	0	30.5	0	1
14	1	0	Mr.	1	48.0	0	0	50.4958	0	1
15	1	0	Dr.	1	[null]	0	0	39.6	0	1
16	1	0	Major.	1	45.0	0	0	26.55	0	1
17	1	0	Mr.	1	[null]	0	0	31.0	0	1
18	1	0	Mr.	1	33.0	0	0	5.0	0	1
19	1	0	Mr.	1	28.0	0	0	47.1	0	1
20	1	0	Mr.	1	17.0	0	0	47.1	0	1
21	1	0	Mr.	1	49.0	0	0	26.0	0	1
22	1	0	Mr.	1	36.0	1	0	78.85	0	2
23	1	0	Mr.	1	46.0	1	0	61.175	0	2
24	1	0	Mr.	1	[null]	0	0	0.0	0	1
25	1	0	Mr.	1	27.0	1	0	136.7792	0	2
26	1	0	Mr.	1	[null]	0	0	52.0	0	1
27	1	0	Mr.	1	47.0	0	0	25.5875	0	1
28	1	0	Mr.	1	37.0	1	1	83.1583	0	3
29	1	0	Mr.	1	[null]	0	0	26.55	0	1
30	1	0	Capt.	1	70.0	1	1	71.0	0	3
31	1	0	Mr.	1	39.0	1	0	71.2833	0	2
32	1	0	Mr.	1	31.0	1	0	52.0	0	2
33	1	0	Mr.	1	50.0	1	0	106.425	0	2
34	1	0	Mr.	1	39.0	0	0	29.7	0	1
35	1	0	Miss.	0	36.0	0	0	31.6792	0	1
36	1	0	Mr.	1	[null]	0	0	221.7792	0	1
37	1	0	Mr.	1	30.0	0	0	27.75	0	1
38	1	0	Mr.	1	19.0	3	2	244.5480856064831	0	6
39	1	0	Mr.	1	64.0	1	4	244.5480856064831	0	6
40	1	0	Mr.	1	[null]	0	0	26.55	0	1
41	1	0	Mr.	1	[null]	0	0	0.0	0	1
42	1	0	Mr.	1	37.0	1	0	53.1	0	2
43	1	0	Mr.	1	47.0	0	0	38.5	0	1
44	1	0	Mr.	1	24.0	0	0	79.2	0	1
45	1	0	Mr.	1	71.0	0	0	34.6542	0	1
46	1	0	Mr.	1	38.0	0	1	153.4625	0	2
47	1	0	Mr.	1	46.0	0	0	79.2	0	1
48	1	0	Mr.	1	[null]	0	0	42.4	0	1
49	1	0	Mr.	1	45.0	1	0	83.475	0	2
50	1	0	Mr.	1	40.0	0	0	0.0	0	1
51	1	0	Mr.	1	55.0	1	1	93.5	0	3
52	1	0	Mr.	1	42.0	0	0	42.5	0	1
53	1	0	Mr.	1	[null]	0	0	51.8625	0	1
54	1	0	Mr.	1	55.0	0	0	50.0	0	1
55	1	0	Mr.	1	42.0	1	0	52.0	0	2
56	1	0	Mr.	1	[null]	0	0	30.6958	1	1
57	1	0	Miss.	0	50.0	0	0	28.7125	0	1
58	1	0	Mr.	1	46.0	0	0	26.0	0	1
59	1	0	Mr.	1	50.0	0	0	26.0	0	1
60	1	0	Mr.	1	32.5	0	0	211.5	0	1
61	1	0	Mr.	1	58.0	0	0	29.7	0	1
62	1	0	Mr.	1	41.0	1	0	51.8625	0	2
63	1	0	Mr.	1	[null]	0	0	26.55	0	1
64	1	0	Mr.	1	[null]	0	0	27.7208	0	1
65	1	0	Mr.	1	29.0	0	0	30.0	0	1
66	1	0	Mr.	1	30.0	0	0	45.5	0	1
67	1	0	Mr.	1	30.0	0	0	26.0	0	1
68	1	0	Mr.	1	19.0	1	0	53.1	0	2
69	1	0	Mr.	1	46.0	0	0	75.2417	0	1
70	1	0	Mr.	1	54.0	0	0	51.8625	0	1
71	1	0	Mr.	1	28.0	1	0	82.1708	0	2
72	1	0	Mr.	1	65.0	0	0	26.55	0	1
73	1	0	Dr.	1	44.0	2	0	90.0	0	3
74	1	0	Mr.	1	55.0	0	0	30.5	0	1
75	1	0	Mr.	1	47.0	0	0	42.4	0	1
76	1	0	Mr.	1	37.0	0	1	29.7	0	2
77	1	0	Mr.	1	58.0	0	2	113.275	0	3
78	1	0	Mr.	1	64.0	0	0	26.0	0	1
79	1	0	Mr.	1	65.0	0	1	61.9792	0	2
80	1	0	Mr.	1	28.5	0	0	27.7208	0	1
81	1	0	Mr.	1	[null]	0	0	0.0	0	1
82	1	0	Mr.	1	45.5	0	0	28.5	0	1
83	1	0	Mr.	1	23.0	0	0	93.5	0	1
84	1	0	Mr.	1	29.0	1	0	66.6	0	2
85	1	0	Mr.	1	18.0	1	0	108.9	0	2
86	1	0	Mr.	1	47.0	0	0	52.0	0	1
87	1	0	Jonkheer.	1	38.0	0	0	0.0	0	1
88	1	0	Mr.	1	22.0	0	0	135.6333	0	1
89	1	0	Mr.	1	[null]	0	0	227.525	0	1
90	1	0	Mr.	1	31.0	0	0	50.4958	0	1
91	1	0	Mr.	1	[null]	0	0	50.0	0	1
92	1	0	Mr.	1	36.0	0	0	40.125	0	1
93	1	0	Mr.	1	55.0	1	0	59.4	0	2
94	1	0	Mr.	1	33.0	0	0	26.55	0	1
95	1	0	Mr.	1	61.0	1	3	244.5480856064831	0	5
96	1	0	Mr.	1	50.0	1	0	55.9	0	2
97	1	0	Mr.	1	56.0	0	0	26.55	0	1
98	1	0	Mr.	1	56.0	0	0	30.6958	0	1
99	1	0	Mr.	1	24.0	1	0	60.0	0	2
100	1	0	Mr.	1	[null]	0	0	26.0	0	1

Rows: 1-100 of 1234 | Columns: 10

The column "age" has too many missing values and since most machine learning algorithms can't handle missing values, we need to impute our data. Let's fill the missing values using the average "age" of the passengers that have the same "pclass" and "sex".

In [16]:

titanic["age"].fillna(method = "mean", by = ["pclass", "sex"])

237 elements were filled.

Out[16]:

	123 pclass Int	123 survived Int	Abc name Varchar(164)	123 sex Int	123 age Float	123 sibsp Int	123 parch Int	123 fare Numeric(18,13)	123 boat Bool	123 family_size Integer
1	1	1	Miss.	0	36.0	0	0	135.6333	1	1
2	1	1	Miss.	0	31.0	0	0	134.5	1	1
3	1	1	Miss.	0	21.0	0	0	26.55	1	1
4	1	1	Mrs.	0	50.0	1	1	211.5	1	3
5	1	1	Mrs.	0	45.0	1	1	164.8667	1	3
6	1	1	Miss.	0	31.0	0	2	164.8667	1	3
7	1	1	Mrs.	0	55.0	0	0	135.6333	1	1
8	1	1	Mrs.	0	60.0	1	0	75.25	1	2
9	1	1	Miss.	0	35.0	0	0	244.5480856064831	1	1
10	1	1	Mrs.	0	37.2635658914729	0	0	79.2	1	1
11	1	1	Mrs.	0	39.0	1	1	110.8833	1	3
12	1	1	Mrs.	0	37.2635658914729	1	0	52.0	1	2
13	1	1	Mrs.	0	39.0	1	1	79.65	1	3
14	1	1	Miss.	0	18.0	0	2	79.65	1	3
15	1	1	Mrs.	0	48.0	0	0	25.9292	1	1
16	1	1	Mrs.	0	62.0	0	0	80.0	1	1
17	1	1	Mrs.	0	52.0	1	0	78.2667	1	2
18	1	1	Mrs.	0	43.0	1	0	55.4417	1	2
19	1	1	Mrs.	0	37.2635658914729	1	0	146.5208	1	2
20	1	1	Mrs.	0	40.0	1	1	134.5	1	3
21	1	1	Mrs.	0	23.0	1	0	82.2667	1	2
22	1	1	Mrs.	0	18.0	1	0	60.0	1	2
23	1	1	Mrs.	0	39.0	1	0	55.9	1	2
24	1	1	Miss.	0	40.0	0	0	153.4625	1	1
25	1	1	Miss.	0	30.0	0	0	31.0	1	1
26	1	1	Mrs.	0	35.0	1	0	57.75	1	2
27	1	1	Mlle.	0	24.0	0	0	69.3	1	1
28	1	1	Mrs.	0	48.0	1	3	244.5480856064831	1	5
29	1	1	Miss.	0	18.0	2	2	244.5480856064831	1	5
30	1	1	Mrs.	0	54.0	1	0	59.4	1	2
31	1	1	Countess.	0	33.0	0	0	86.5	1	1
32	1	1	Miss.	0	33.0	0	0	27.7208	1	1
33	1	1	Mrs.	0	43.0	0	1	211.3375	1	2
34	1	1	Mrs.	0	56.0	0	1	83.1583	1	2
35	1	1	Miss.	0	30.0	0	0	93.5	1	1
36	1	1	Mrs.	0	17.0	1	0	108.9	1	2
37	1	1	Mrs.	0	22.0	1	0	66.6	1	2
38	1	1	Miss.	0	22.0	0	1	61.9792	1	2
39	1	1	Dona.	0	39.0	0	0	108.9	1	1
40	1	1	Miss.	0	19.0	0	2	26.2833	1	3
41	1	1	Miss.	0	23.0	1	0	113.275	1	2
42	1	1	Miss.	0	31.0	1	0	113.275	1	2
43	1	1	Mrs.	0	37.0	1	0	90.0	1	2
44	1	1	Miss.	0	33.0	1	0	90.0	1	2
45	1	1	Mrs.	0	37.2635658914729	1	0	82.1708	1	2
46	1	1	Mrs.	0	18.0	1	0	53.1	1	2
47	1	1	Miss.	0	16.0	0	0	86.5	1	1
48	1	1	Miss.	0	15.0	0	1	211.3375	1	2
49	1	1	Miss.	0	58.0	0	0	146.5208	0	1
50	1	1	Miss.	0	21.0	0	0	77.9583	1	1
51	1	1	Mrs.	0	51.0	0	1	39.4	1	2
52	1	1	Miss.	0	16.0	0	1	39.4	1	2
53	1	1	Mrs.	0	55.0	0	0	27.7208	1	1
54	1	1	Dr.	0	49.0	0	0	25.9292	1	1
55	1	1	Miss.	0	30.0	0	0	106.425	1	1
56	1	1	Miss.	0	39.0	0	0	211.3375	1	1
57	1	1	Mrs.	0	45.0	1	0	52.5542	1	2
58	1	1	Mrs.	0	37.2635658914729	1	0	51.8625	1	2
59	1	1	Miss.	0	38.0	0	0	80.0	1	1
60	1	1	Mrs.	0	35.0	1	0	90.0	1	2
61	1	1	Mrs.	0	35.0	1	0	52.0	1	2
62	1	1	Mrs.	0	51.0	1	0	77.9583	1	2
63	1	1	Mrs.	0	44.0	0	1	57.9792	1	2
64	1	1	Miss.	0	16.0	0	1	57.9792	1	2
65	1	1	Mrs.	0	52.0	1	1	93.5	1	3
66	1	1	Miss.	0	24.0	0	0	83.1583	1	1
67	1	1	Mrs.	0	35.0	1	0	83.475	1	2
68	1	1	Mrs.	0	49.0	1	0	76.7292	1	2
69	1	1	Mrs.	0	25.0	1	0	55.4417	1	2
70	1	1	Mrs.	0	45.0	0	1	63.3583	1	2
71	1	1	Mrs.	0	58.0	0	1	153.4625	1	2
72	1	1	Miss.	0	19.0	0	0	30.0	1	1
73	1	1	Mrs.	0	37.2635658914729	1	0	89.1042	1	2
74	1	1	Mrs.	0	45.0	0	1	59.4	1	2
75	1	1	Miss.	0	22.0	0	1	59.4	1	2
76	1	1	Miss.	0	35.0	0	0	211.5	1	1
77	1	1	Mrs.	0	35.0	1	0	53.1	1	2
78	1	1	Mrs.	0	48.0	1	1	79.2	1	3
79	1	1	Miss.	0	22.0	0	2	49.5	1	3
80	1	1	Mrs.	0	37.2635658914729	1	0	133.65	1	2
81	1	1	Miss.	0	30.0	0	0	56.9292	1	1
82	1	1	Mrs.	0	60.0	1	4	244.5480856064831	1	6
83	1	1	Miss.	0	23.0	3	2	244.5480856064831	1	6
84	1	1	Miss.	0	28.0	3	2	244.5480856064831	1	6
85	1	1	Miss.	0	24.0	3	2	244.5480856064831	1	6
86	1	1	Miss.	0	37.2635658914729	0	0	110.8833	1	1
87	1	1	Mrs.	0	37.2635658914729	0	0	31.6833	1	1
88	1	1	Miss.	0	54.0	1	0	78.2667	1	2
89	1	1	Miss.	0	38.0	0	0	227.525	1	1
90	1	1	Mrs.	0	23.0	0	1	83.1583	1	2
91	1	1	Mrs.	0	48.0	1	0	106.425	1	2
92	1	1	Mrs.	0	27.0	1	1	244.5480856064831	1	3
93	1	1	Mrs.	0	54.0	1	1	81.8583	1	3
94	1	1	Mrs.	0	17.0	1	0	57.0	1	2
95	1	1	Mrs.	0	27.0	1	2	52.0	1	4
96	1	1	Miss.	0	33.0	0	0	151.55	1	1
97	1	1	Mrs.	0	38.0	1	0	71.2833	1	2
98	1	1	Mrs.	0	64.0	1	1	26.55	1	3
99	1	1	Miss.	0	36.0	0	2	71.0	1	3
100	1	1	Mrs.	0	55.0	2	0	25.7	1	3

Rows: 1-100 of 1234 | Columns: 10

Let's draw the correlation matrix to see the links between variables.

In [17]:

titanic.corr(method = "spearman")

Out[17]:

	"pclass"	"survived"	"sex"	"age"	"sibsp"	"parch"	"fare"	"boat"	"family_size"
"pclass"	1.0	-0.335634606444129	0.150967389146603	-0.447455588651876	-0.0691851178284916	-0.0429882472107741	-0.722423468046863	-0.344146454498079	-0.109035609183308
"survived"	-0.335634606444129	1.0	-0.528849820180154	-0.00336824465604963	0.0934303518392484	0.171691455579124	0.322278285934243	0.949194918072845	0.185599315104602
"sex"	0.150967389146603	-0.528849820180154	1.0	0.0659428426025525	-0.184712709381236	-0.246951353519755	-0.264115871968389	-0.484361066873315	-0.284892754418329
"age"	-0.447455588651876	-0.00336824465604963	0.0659428426025525	1.0	-0.116480956583223	-0.174615632035404	0.228529791943121	0.00421157947473071	-0.133316437156597
"sibsp"	-0.0691851178284916	0.0934303518392484	-0.184712709381236	-0.116480956583223	1.0	0.436524985907755	0.444010325371998	0.100652268255772	0.854550816071704
"parch"	-0.0429882472107741	0.171691455579124	-0.246951353519755	-0.174615632035404	0.436524985907755	1.0	0.402198981036797	0.175393711669708	0.776055413048162
"fare"	-0.722423468046863	0.322278285934243	-0.264115871968389	0.228529791943121	0.444010325371998	0.402198981036797	1.0	0.334910192561693	0.52665819103252
"boat"	-0.344146454498079	0.949194918072845	-0.484361066873315	0.00421157947473071	0.100652268255772	0.175393711669708	0.334910192561693	1.0	0.190320484343677
"family_size"	-0.109035609183308	0.185599315104602	-0.284892754418329	-0.133316437156597	0.854550816071704	0.776055413048162	0.52665819103252	0.190320484343677	1.0

Rows: 1-9 | Columns: 10

Fare correlates strongly with family size. This is about what you would expect: a larger family means more tickets, and more tickets means a greater fare.

Survival correlates strongly with whether or not a passenger has a lifeboat (the "boat" variable). Still, to increase the generality of our model, we should avoid predictions based on just one variable. Let's split the study into two use cases:

Passengers with a lifeboat
Passengers without a lifeboat

Before we move on: we did a lot of work to clean up this data, but we haven't saved anything to our Vertica database! Let's look at the modifications we've made to the vDataFrame.

In [18]:

print(titanic.current_relation())

(
   SELECT
     "pclass",
     "survived",
     "name",
     "sex",
     COALESCE("age", AVG("age") OVER (PARTITION BY "pclass", "sex")) AS "age",
     "sibsp",
     "parch",
     "fare",
     "boat",
     "family_size" 
   FROM
 (
   SELECT
     "pclass",
     "survived",
     REGEXP_SUBSTR("name", ' ([A-Za-z]+)\.') AS "name",
     DECODE("sex", 'female', 0, 'male', 1, 2) AS "sex",
     "age",
     "sibsp",
     "parch",
     (CASE WHEN "fare" < -176.6204982585513 THEN -176.6204982585513 WHEN "fare" > 244.5480856064831 THEN 244.5480856064831 ELSE "fare" END) AS "fare",
     DECODE("boat", NULL, 0, 1) AS "boat",
     (("parch") + ("sibsp")) + (1) AS "family_size" 
   FROM
 (
   SELECT
     "pclass",
     "survived",
     "name",
     "sex",
     "age",
     "sibsp",
     "parch",
     "fare",
     "boat" 
   FROM
 "public"."titanic") 
VERTICAPY_SUBTABLE) 
VERTICAPY_SUBTABLE) 
VERTICAPY_SUBTABLE

Let see what's happening when we aggregate and turn on SQL generation.

In [19]:

vp.set_option("sql_on", True)
titanic.avg()

Computing the different aggregations.

SELECT
2.28444084278768,
0.364667747163695,
AVG("sex"),
AVG("age"),
0.504051863857374,
0.378444084278768,
AVG("fare"),
0.355753646677472,
AVG("family_size")
FROM
(
SELECT
"pclass",
"survived",
"name",
"sex",
COALESCE("age", AVG("age") OVER (PARTITION BY "pclass", "sex")) AS "age",
"sibsp",
"parch",
"fare",
"boat",
"family_size"
FROM
(
SELECT
"pclass",
"survived",
REGEXP_SUBSTR("name", ' ([A-Za-z]+)\.') AS "name",
DECODE("sex", 'female', 0, 'male', 1, 2) AS "sex",
"age",
"sibsp",
"parch",
(CASE WHEN "fare" < -176.6204982585513 THEN -176.6204982585513 WHEN "fare" > 244.5480856064831 THEN 244.5480856064831 ELSE "fare" END) AS "fare",
DECODE("boat", NULL, 0, 1) AS "boat",
(("parch") + ("sibsp")) + (1) AS "family_size"
FROM
(
SELECT
"pclass",
"survived",
"name",
"sex",
"age",
"sibsp",
"parch",
"fare",
"boat"
FROM
"public"."titanic")
VERTICAPY_SUBTABLE)
VERTICAPY_SUBTABLE)
VERTICAPY_SUBTABLE LIMIT 1

Out[19]:

	avg
"pclass"	2.28444084278768
"survived"	0.364667747163695
"sex"	0.659643435980551
"age"	29.717623352014
"sibsp"	0.504051863857374
"parch"	0.378444084278768
"fare"	32.9113074018842
"boat"	0.355753646677472
"family_size"	1.88249594813614

Rows: 1-9 | Columns: 2

VerticaPy dynamically generates SQL code whenever you make modifications to your data. To avoid recomputation, it also stores previous aggregations. If we filter anything in our data, it will update the catalog with our modifications.

In [20]:

vp.set_option("sql_on", False)
print(titanic.info())

The vDataFrame was modified many times: 
 * {Tue May  3 10:20:59 2022} [Drop]: vColumn "body" was deleted from the vDataFrame.
 * {Tue May  3 10:20:59 2022} [Drop]: vColumn "home.dest" was deleted from the vDataFrame.
 * {Tue May  3 10:20:59 2022} [Drop]: vColumn "embarked" was deleted from the vDataFrame.
 * {Tue May  3 10:20:59 2022} [Drop]: vColumn "ticket" was deleted from the vDataFrame.
 * {Tue May  3 10:21:01 2022} [Apply]: The vColumn 'cabin' was transformed with the func 'x -> SUBSTR(x, 1, 1)'.
 * {Tue May  3 10:21:01 2022} [Apply]: The vColumn 'name' was transformed with the func 'x -> REGEXP_SUBSTR(x, ' ([A-Za-z]+)\.')'.
 * {Tue May  3 10:21:01 2022} [Fillna]: 795 "boat" missing values were  filled.
 * {Tue May  3 10:21:02 2022} [Fillna]: 948 "cabin" missing values were  filled.
 * {Tue May  3 10:21:04 2022} [Drop]: vColumn "cabin" was deleted from the vDataFrame.
 * {Tue May  3 10:21:09 2022} [Eval]: A new vColumn "family_size" was added to the vDataFrame.
 * {Tue May  3 10:21:12 2022} [Apply]: The vColumn 'fare' was transformed with the func 'x -> (CASE WHEN x < -176.6204982585513 THEN -176.6204982585513 WHEN x > 244.5480856064831 THEN 244.5480856064831 ELSE x END)'.
 * {Tue May  3 10:21:14 2022} [Label Encoding]: Label Encoding was applied to the vColumn "sex" using the following mapping:
	female => 0	male => 1
 * {Tue May  3 10:21:16 2022} [Fillna]: 237 "age" missing values were  filled.

Let's move on to modeling our data. Save the vDataframe to your Vertica database.

In [37]:

from verticapy import drop
drop("titanic_boat", method = "view")
titanic_boat = titanic.search(titanic["boat"] == 1).to_db("titanic_boat", relation_type = "view")
drop("titanic_no_boat", method = "view")
titanic_no_boat = titanic.search(titanic["boat"] == 0).to_db("titanic_no_boat", relation_type = "view")

Machine Learning¶

Passengers with a lifeboat¶

First, let's look at the number of survivors.

In [24]:

titanic_boat["survived"].describe()

Out[24]:

	value
name	"survived"
dtype	integer
unique	2.0
count	439.0
1	430
0	9

Rows: 1-6 | Columns: 2

We have nine deaths. Let's try to understand why these passengers died.

In [40]:

titanic_boat.search(titanic_boat["survived"] == 0).head(10)

Out[40]:

	123 pclass Integer	123 survived Integer	Abc name Varchar(164)	123 sex Integer	123 age Float	123 sibsp Integer	123 parch Integer	123 fare Numeric(18,13)	123 boat Integer	123 family_size Integer
1	1	0	Mr.	1	40.9822068965517	0	0	30.6958	1	1
2	1	0	Mr.	1	36.0	0	0	75.2417	1	1
3	2	0	Mr.	1	34.0	1	0	21.0	1	2
4	3	0	Mrs.	0	30.0	1	0	15.55	1	2
5	3	0	Mr.	1	27.0	1	0	14.4542	1	2
6	3	0	Mr.	1	36.0	1	0	15.55	1	2
7	3	0	Mr.	1	26.2142058823529	0	0	7.25	1	1
8	3	0	Mr.	1	25.0	0	0	7.25	1	1
9	3	0	Mr.	1	32.0	1	0	15.85	1	2

Rows: 1-9 | Columns: 10

Apart from a decent amount of these passengers being third-class passengers, it doesn't seem like there are any clear predictors here for their deaths. Making a model from this would be unhelpful.

Passengers without a lifeboat¶

Let's move on to passengers without a lifeboat.

In [38]:

titanic_no_boat["survived"].describe()

Out[38]:

	value
name	"survived"
dtype	integer
unique	2.0
count	795.0
0	775
1	20

Rows: 1-6 | Columns: 2

Only 20 survived. Let's find out why.

In [39]:

titanic_no_boat.search(titanic_boat["survived"] == 1).head(20)

Out[39]:

	123 pclass Integer	123 survived Integer	Abc name Varchar(164)	123 sex Integer	123 age Float	123 sibsp Integer	123 parch Integer	123 fare Numeric(18,13)	123 boat Integer	123 family_size Integer
1	1	1	Miss.	0	58.0	0	0	146.5208	0	1
2	2	1	Mrs.	0	28.0	0	0	12.65	0	1
3	2	1	Mrs.	0	30.0	3	0	21.0	0	4
4	2	1	Mrs.	0	14.0	1	0	30.0708	0	2
5	2	1	Mrs.	0	42.0	1	0	26.0	0	2
6	2	1	Miss.	0	17.0	0	0	10.5	0	1
7	2	1	Mrs.	0	34.0	0	1	23.0	0	2
8	2	1	Miss.	0	18.0	0	1	23.0	0	2
9	2	1	Mrs.	0	42.0	0	0	13.0	0	1
10	3	1	Mrs.	0	15.0	1	0	14.4542	0	2
11	3	1	Mrs.	0	47.0	1	0	7.0	0	2
12	3	1	Miss.	0	22.5766423357664	0	0	7.7792	0	1
13	3	1	Mrs.	0	31.0	0	0	8.6833	0	1
14	3	1	Mrs.	0	22.5766423357664	0	0	7.2292	0	1
15	3	1	Miss.	0	27.0	0	0	7.925	0	1
16	3	1	Miss.	0	26.0	0	0	7.925	0	1
17	3	1	Miss.	0	23.0	0	0	8.05	0	1
18	3	1	Mrs.	0	33.0	3	0	15.85	0	4
19	3	1	Mr.	1	26.2142058823529	0	0	7.75	0	1
20	3	1	Mr.	1	26.2142058823529	0	0	7.75	0	1

Rows: 1-20 | Columns: 10

Most survivors seem to be women. Let's build a model with this in mind.

One of our predictors is categorical: the passenger title. Some of these predictors are corrleated, so it'd be best to work with a non-linear classifier that can handle that. In this case, a random forest classifier seems to be perfect. Let's evaluate it with a cross-validation.

In [41]:

from verticapy.learn.ensemble import RandomForestClassifier
from verticapy.learn.model_selection import cross_validate

predictors = titanic.get_columns(exclude_columns = ['survived'])
response = "survived"
model = RandomForestClassifier("rf_titanic", 
                               n_estimators = 40, 
                               max_depth = 4)
cross_validate(model, titanic_no_boat, predictors, response)

Out[41]:

	auc	prc_auc	accuracy	log_loss	precision	recall	f1_score	mcc	informedness	markedness	csi	time
1-fold	0.9533769063180827	0.35730138533283	0.9659090909090909	0.0389919854372209	0	0.0	0	0	0.0	-0.03409090909090906	0.0	0.4354102611541748
2-fold	0.9291251384274639	0.37062390075308016	0.9735849056603774	0.0349201186235419	0	0.0	0	0	0.0	-0.026415094339622636	0.0	0.464846134185791
3-fold	0.9770114942528736	0.22139249639249636	0.9849056603773585	0.0214770027802224	0	0.0	0	0	0.0	-0.015094339622641506	0.0	0.4200739860534668
avg	0.9531711796661401	0.3164392608261355	0.9747998856489423	0.031796368946995064	0.0	0.0	0.0	0.0	0.0	-0.025200114351057734	0.0	0.4401101271311442
std	0.02394384077692897	0.08258200727917958	0.009556387663240679	0.009165806751183139	0.0	0.0	0.0	0.0	0.0	0.009556387663240679	0.0	0.022753084785294538

Rows: 1-5 | Columns: 13

This dataset is pretty unbalanced so we'll use an AUC to evaluate it. Looking at our table, our model has an average AUC of more than 0.9, so our model is quite good.

We can now build a model with the entire dataset.

In [42]:

model.fit(titanic_no_boat, predictors, response)

Out[42]:


===========
call_string
===========
SELECT rf_classifier('public.rf_titanic', '"public"."_verticapy_tmp_view_dbadmin_43247_5454950161_"', '"survived"', '"pclass", "name", "sex", "age", "sibsp", "parch", "fare", "boat", "family_size"' USING PARAMETERS exclude_columns='', ntree=40, mtry=4, sampling_size=0.632, max_depth=4, max_breadth=1000000000, min_leaf_size=1, min_info_gain=0, nbins=32);

=======
details
=======
 predictor |      type      
-----------+----------------
  pclass   |      int       
   name    |char or varchar 
    sex    |      int       
    age    |float or numeric
   sibsp   |      int       
   parch   |      int       
   fare    |float or numeric
   boat    |      int       
family_size|      int       


===============
Additional Info
===============
       Name       |Value
------------------+-----
    tree_count    | 40  
rejected_row_count|  1  
accepted_row_count| 794

Let's look at the importance of each feature.

In [43]:

model.features_importance()

Out[43]:

	importance	sign
name	26.04	1
age	13.79	1
sex	13.64	1
fare	12.89	1
pclass	9.73	1
parch	9.48	1
sibsp	7.47	1
family_size	6.95	1
boat	0.0	0

Rows: 1-9 | Columns: 3

As expected, a passenger's title and the sex are the most important predictors of survival.

Conclusion¶

We've solved our problem in a Pandas-like way, all without ever loading data into memory!

VerticaPy

Python API for Vertica Data Science at Scale

About the Author

Badr Ouali
Head of Data Science

Badr Ouali works as a Lead Data Scientist for Vertica worldwide. He can embrace data projects end to end through a clear understanding of the “big picture” as well as attention to details, resulting in achieving great business outcomes – a distinctive differentiator in his role. Badr enjoys sharing knowledge and insights related to data analytics with colleagues & peers and has a sweet spot for Python. He loves helping customers finding the best value from their data and empower them to solve their use-cases.