Telco Churn¶

This example uses the Telco Churn dataset to predict which Telco user is likely to churn; that is, customers that will likely stop using Telco. You can download the Jupyter Notebook of the study here.

Churn: customers that left within the last month.
Services: services of each customer (phone, multiple lines, internet, online security, online backup, device protection, tech support, and streaming TV and movies).
Customer account information: how long they’ve been a customer, contract, payment method, paperless billing, monthly charges, and total charges.
Customer demographics: gender, age range, and if they have partners and dependents.

We will follow the data science cycle (Data Exploration - Data Preparation - Data Modeling - Model Evaluation - Model Deployment) to solve this problem.

Initialization¶

This example uses the following version of VerticaPy:

import verticapy as vp

vp.__version__
Out[2]: '1.1.0'

Connect to Vertica. This example uses an existing connection called VerticaDSN . For details on how to create a connection, see the Connection tutorial. You can skip the below cell if you already have an established connection.

vp.connect("VerticaDSN")

Let’s create a Virtual DataFrame of the dataset. The dataset is available here.

churn = vp.read_csv("customers.csv")

Let’s take a look at the first few entries in the dataset.

churn.head(10)

	Abc customerID Varchar(20) 100%	...	123 TotalCharges Numeric(9,3) 99%	010 Churn Boolean 100%
1	0002-ORFBO	...	593.3	❌
2	0003-MKNFE	...	542.4	❌
3	0004-TLHLJ	...	280.85	✅
4	0017-DINOC	...	2460.55	❌
5	0020-JDNXP	...	1993.2	❌
6	0022-TCJCI	...	2791.5	✅
7	0023-XUOPT	...	1215.6	✅
8	0040-HALCW	...	1090.6	❌
9	0042-JVWOJ	...	471.85	❌
10	0064-SUDOG	...	224.5	❌

Data Exploration and Preparation¶

Let’s examine our data.

churn.describe(method = "categorical", unique = True)

	...	top_percent	unique
"customerID"	...	0.014	7043.0
"gender"	...	50.476	2.0
"SeniorCitizen"	...	83.785	2.0
"Partner"	...	51.697	2.0
"Dependents"	...	70.041	2.0
"tenure"	...	8.704	73.0
"PhoneService"	...	90.317	2.0
"MultipleLines"	...	48.133	3.0
"InternetService"	...	43.959	3.0
"OnlineSecurity"	...	49.666	3.0
"OnlineBackup"	...	43.845	3.0
"DeviceProtection"	...	43.944	3.0
"TechSupport"	...	49.311	3.0
"StreamingTV"	...	39.898	3.0
"StreamingMovies"	...	39.543	3.0
"Contract"	...	55.019	3.0
"PaperlessBilling"	...	59.222	2.0
"PaymentMethod"	...	33.579	4.0
"MonthlyCharges"	...	0.866	1585.0
"TotalCharges"	...	0.156	6530.0
"Churn"	...	73.463	2.0

Several variables are categorical, and since they all have low cardinalities, we can compute their dummies. We can also convert all booleans to numeric.

for column in [
    "DeviceProtection",
    "MultipleLines",
    "PaperlessBilling",
    "Churn",
    "TechSupport",
    "Partner",
    "StreamingTV",
    "OnlineBackup",
    "Dependents",
    "OnlineSecurity",
    "PhoneService",
    "StreamingMovies",
]:
    churn[column].decode("Yes", 1, 0)
churn.one_hot_encode().drop(
    [
        "customerID",
        "gender",
        "Contract",
        "PaymentMethod",
        "InternetService",
    ],
)

	123 SeniorCitizen Int 100%	...	123 Partner Integer 100%	123 PaymentMethod_Electronic_check Bool 100%
1	0	...	1	0
2	0	...	0	0
3	0	...	0	1
4	0	...	0	0
5	0	...	1	0
6	1	...	0	0
7	0	...	1	1
8	0	...	1	0
9	0	...	0	0
10	0	...	1	0
11	0	...	0	0
12	0	...	0	1
13	0	...	1	0
14	1	...	0	1
15	0	...	0	0
16	0	...	1	0
17	1	...	1	0
18	0	...	1	1
19	0	...	1	0
20	0	...	0	1

Let’s compute the correlations between the different variables and the response column.

churn.corr(focus = "Churn")

Many features have a strong correlation with the Churn variable. For example, the customers that have a Month to Month contract are more likely to churn. Having this type of contract gives customers a lot of flexibility and allows them to leave at any time. New customers are also likely to churn.

# No lock-in = Churn
churn.barh(["Contract_Month-to-month", "tenure"], method = "avg", of = "Churn", height = 500)

The following scatter plot shows that providing better tariff plans can prevent churning. Indeed, customers having high total charges are more likely to churn even if they’ve been with the company for a long time.

churn.scatter(["TotalCharges", "tenure"], by = "Churn")

Let’s move on to machine learning.

Machine Learning¶

LogisticRegression is a very powerful algorithm and we can use it to detect churns. Let’s split our vDataFrame into training and testing set to evaluate our model.

train, test = churn.train_test_split(
    test_size = 0.2,
    random_state = 0,
)

Let’s train and evaluate our model.

from verticapy.machine_learning.vertica import LogisticRegression

model = LogisticRegression(
    penalty = "L2",
    tol = 1e-6,
    max_iter = 1000,
    solver = "BFGS",
)
model.fit(
    train,
    churn.get_columns(exclude_columns = ["churn"]),
    "churn",
    test,
)
model.classification_report()

	value
auc	0.8419292283921654
prc_auc	0.6496481202189804
accuracy	0.8022678951098512
log_loss	0.178354728672812
precision	0.6447368421052632
recall	0.5340599455040872
f1_score	0.5842026825633383
mcc	0.45947139005227444
informedness	0.43061166964201814
markedness	0.49026529738981606
csi	0.4126315789473684

The model is excellent! Let’s run some machine learning on the entire dataset and compute the importance of each feature.

model.drop()
model.fit(
    churn,
    churn.get_columns(exclude_columns = ["churn"]),
    "churn",
)
model.features_importance()

Based on our model, most churning customers are at least one of the following:

Paying higher bills
New Telco customers
Have a monthly contract

Notice that customers have a Fiber Optic option are also likely to churn. Let’s check if this relationship is causal by computing some aggregations.

import verticapy.sql.functions as fun

# Is Fiber optic a Bad Option? - VerticaPy
churn.groupby(
    ["InternetService_Fiber_optic"],
    [
        fun.avg(churn["tenure"])._as("tenure"),
        fun.avg(churn["totalcharges"])._as("totalcharges"),
        fun.avg(churn["contract_month-to-month"])._as("contract_month_to_month"),
        fun.avg(churn["monthlycharges"])._as("monthlycharges"),
    ]
)

	123 InternetService_Fiber_optic Integer 100%	...	123 contract_month_to_month Float(22) 100%	123 monthlycharges Float(22) 100%
1	0	...	0.442614644033443	43.7882442361287
2	1	...	0.68733850129199	91.5001291989664

It seems like the Fiber Optic option in and of itself doesn’t lead to churning, but customers that have this option tend to churn because their contract puts them into one of the three categories we listed before: they’re paying more.

To retain these customers, we’ll need to make some changes to what types of contracts we offer.

We’ll use a lift chart to help us identify which of our customers are likely to churn.

model.lift_chart()

By targeting less than 30% of the entire distribution, our predictions will be more than three times more accurate than the other 70%.

Conclusion¶

We’ve solved our problem in a Pandas-like way, all without ever loading data into memory!