Predicting Popularity on Spotify¶

This example uses the publicly-available Spotify from Kaggle to predict the popularity of Polish songs and artists on Spotify. We’ll also use a model to group artists together based on how similar their songs are.

You can download the Jupyter notebook of this study here.

Note

We are only using polish artists and a subset of the tracks dataset filtered by a handful of artists.

The tracks dataset (tracks.csv) have the following features:

Id represents the Id of the track generated by Spotify

Numerical:

acousticness (range: [0,1])
danceability (range: [0,1])
energy (range: [0,1])
duration_ms (range: [200000,300000])
instrumentalness (range: [0,1])
valence (range: [0,1])
popularity (range: [0,100])
tempo (range: [50,150])
liveness (range: [0,1])
loudness (range: [-60,0])
speechiness (range: [0,1])

Dummy:

mode: (0 = Minor, 1 = Major)
explicit: (0 = No explicit content and 1 = Explicit content)

Categorical:

key: keys on an octave encoded as integers in range [0,11] (C = 0, C = 1, etc.)
timesignature: predicted time signature.
artists: list of contributing artists.
artists: list of IDs of contributing artists.
release_date: date of release (yyyy-mm-dd).
name: track name.

The artists dataset (artists.csv) has the following features:

id: ID of the artist.
name: artist name.
followers: how many followers the artist has.
popularity: popularity of the artists based on their tracks.
genres: list of genres covered by the artist’s tracks.

We will follow the data science cycle (Data Exploration - Data Preparation - Data Modeling - Model Evaluation - Model Deployment) to solve this problem.

Initialization¶

This example uses the following version of VerticaPy:

import verticapy as vp

vp.__version__
Out[2]: '1.1.0'

Start by importing VerticaPy and loading the SQL extension, which allows you to query the Vertica database with SQL.

%load_ext verticapy.sql
The verticapy.sql extension is already loaded. To reload it, use:
  %reload_ext verticapy.sql

Connect to Vertica. This example uses an existing connection called VerticaDSN. For details on how to create a connection, see the Connection tutorial. You can skip the below cell if you already have an established connection.

vp.connect("VerticaDSN")

Create a new schema, “spotify”.

vp.drop("spotify", method = "schema")
Out[4]: True

vp.create_schema("spotify")
Out[5]: True

Data Loading¶

Load the datasets into the vDataFrame with read_csv() and then view them with head().

# load datasets as vDataFrame objects
artists = vp.read_csv("artists.csv", schema = "spotify", parse_nrows = 100)
tracks  = vp.read_csv("tracks.csv" , schema = "spotify", parse_nrows = 100)

# Display
artists.head(100)

tracks.head(100)

	Abc id Varchar(44) 100%	...	Abc Varchar(56) 100%	123 popularity Int 100%
1	004reCzVFOidvBuYrYia9Y	...		27
2	00ekfPE5ZS3NwF8H8o8GBk	...		47
3	01TgMAgIALWvVXlKjUwpfn	...		9
4	02Cq85QmaYHDi4dW7AxTRZ	...		9
5	02ESuuto8Jwyo4PeiJ1Xim	...		1
6	03FgbE2vKKVEFBFHi8IfJG	...		55
7	03KLzHVK6la8dVop1iVI5x	...		50
8	03jLJnyfZXs1ssrIALfGRm	...		22
9	04Lio76CKJCMPbK5hV6J4w	...		5
10	05AVHcWP9DF6y6LEU845uz	...		15
11	05Fgqq7GfWeNol1TR5H3og	...		35
12	05UsyksBcAUVdfyREMxbDm	...		11
13	070tdNOiP3pIsGlqNfVkG3	...		51
14	0CgCy79P84g1meaXcwwFqZ	...		0
15	0CsrftI3Zs3nvfSW6MRglc	...		0
16	0EPzUAW8kwuPedmmVP6n9S	...		53
17	0EYfWGAHPugeWUKKvoMU79	...		4
18	0EvkY8O19trlgsfrVOTQgg	...		31
19	0FbccBQBb69lfv4arbt6kX	...		36
20	0GPfyyiTlLdG6rQthueRBM	...		26
21	0Gk98lHv6LlqbWPwdMiga2	...		50
22	0GsCeqHAG63k8CRj1NH8e4	...		0
23	0HTub0NhKSRgggtmJBP9aR	...		0
24	0HZL4dV60t13CHasIHwaLP	...		4
25	0HhejlCvg1WCO9nXNZGEkc	...		1
26	0Id5ZU9SxHcgE32nfJMTbh	...		19
27	0KTn3DOb57GcGjPoA09ABL	...		0
28	0KZLEvrZHdqVDKdclXRVK0	...		0
29	0KirHnU7pIfeMYWSJ6xm8I	...		12
30	0LcUNEKY8mVqNmYfrgZrxl	...		24
31	0M5UiR76X2ybfo6N9iVNWr	...		36
32	0MQyjuZSqwUlAmWo7bryry	...		8
33	0MXVKY88dOygNxQSjYAiCn	...		9
34	0MzWXIO3Z73PfIwg0UUGHm	...		36
35	0NMfHNHHyEUp2DZxIXyA0c	...		6
36	0NiIlOoQCQPrri3Mnzb41D	...		43
37	0OWXK55YvtWja5pKp8vqXL	...		17
38	0PbRHFtbXsxQfOHl6m86dd	...		32
39	0PtPpPyhP8KRgSwYDvhPlE	...		2
40	0Qo6PIs38oznVHFxs9WU0N	...		37
41	0RLhLvcRdN4LsXUvRsL9rM	...		31
42	0T0rADxhl1CLxhSS3t4FJr	...		40
43	0T0yIgJd9ZIeVJqZw276iH	...		6
44	0TdjP78ddOnKTEVuF3LBrT	...		30
45	0UCbjuZR3UYm9XtycKimCz	...		20
46	0UvDCk7VwDQ0JWGeTTaGpB	...		0
47	0VJDqh2nHbXaafoaDbecop	...		6
48	0VeG3URJkBzh0CHJqZqmPL	...		18
49	0VhKsa9J4dkGjkSHwthPUl	...		4
50	0W0udbffr9z2chRB5eiq9W	...		2
51	0WDJa0qnagyOnMaiD26wht	...		65
52	0WKQQ0JVwqVfaCUeocL81k	...		0
53	0WRMNsx6J1XPOIYhxs2ZQy	...		0
54	0WX7MXOUx7elCFdxdgvdBU	...		43
55	0XqtseX3XzxM4MwPlVLDBJ	...		3
56	0XuTvNiI3kFV0Jpt5MTaMf	...		0
57	0Y0MpkBrGD02Cx3Mmhfa9I	...		42
58	0aquWYdumRf3dKccHKaQ25	...		11
59	0bbSbmpfnelM4nQfogVk9B	...		0
60	0bfBH61NEvZOmVTUyUL1yO	...		40
61	0buOvM3DxsknoPmgGufB5B	...		0
62	0c7soAA3Nv5aaDBMtCy7v4	...		27
63	0cFQLs28WprLqNslPA1vBH	...		8
64	0fgRKFA5ecwbbU6u6nSh2J	...		9
65	0g8uDsDCthOato1OArQG4d	...		16
66	0hmW4jgeM8oo7PrTja7KzY	...		9
67	0jEJGHxA3gkLdjviT1H0wk	...		40
68	0kh9Gvy9lGZsq84x7I37DC	...		64
69	0lKCO7SCRiTCS4ZEU6l1zx	...		50
70	0lTe15Ofpyg39nXvLNAfcm	...		29
71	0lUa6o7QrvyJKszdaBnoAH	...		45
72	0mGMdkeDynbGXSVd0PY8Oq	...		49
73	0mHLX59DLqrZHIo8mOhiBG	...		23
74	0mneo6UHjcOtZBm1Tw8t67	...		6
75	0pUbw5HdKXcSz6luKNyOiR	...		6
76	0q4Dul29Bz5se2iYcKPQXA	...		1
77	0qI3BHXdNAjGHu3NqRaacs	...		2
78	0sgX7YgncTMJs40ANv08V2	...		3
79	0sijpMmVF9ui4micUVn9d6	...		30
80	0tLpQEO7WioDR2cjo3SgWj	...		0
81	0tMBk4ZobHzjogZ2911v6y	...		5
82	0tmTvcGSgfSn6dLFNxC3Sd	...		36
83	0xBUHtBX1kujjjbayPiYOq	...		2
84	0z0ey55umWNi8s2HsNaGxB	...		1
85	0zrHcdKNA1olsruclYjy7h	...		19
86	10HoQ9uLWCt4YoiaYkmQVU	...		27
87	10ki5CunLQjvMB093EClw3	...		1
88	11mE3U9BBkGQcOty6rOXUw	...		6
89	12C8rZ4SY6WxvuxqykeKw6	...		4
90	12fqDmzvnOz7oQE5J2XDfL	...		0
91	138bq70hDEiUOtwGFRKB9M	...		23
92	13BySi42Trub84QyO83Q6c	...		0
93	154o9Wi0JpSDbYQPMtmpd5	...		57
94	15kkqvIcypRQGUiE17Shej	...		17
95	161n5VNTH8MQ8hh5jgwewr	...		2
96	16E01HKHSbeo1s2GB0zam5	...		13
97	16IE8lpWA2U3bfB4kumGzW	...		45
98	16TsNPlesuA1R9kPLS6nta	...		58
99	177K31OvglpG0Epy24mWyd	...		8
100	17CHv5PPFzmVshxqUpGEdZ	...		17

	Abc id Varchar(44) 100%	...	123 tempo Numeric(10,5) 100%	123 time_signature Int 100%
1	00DK20f7WgF2t1eTwlveGu	...	132.38	4
2	00LQ1LG1WCfyYEgyg17ZeK	...	107.91	4
3	00MAWNbKDCBUMjqmyx7ipK	...	72.491	4
4	00PPopDfcYt0QOu8vxMqm5	...	116.389	4
5	00PVvXOTonYJRSItustZhr	...	80.566	4
6	00a8mZIrYshErBYRiY6J7d	...	50.688	4
7	00iXlZ8jMVxSrxXvQomYU1	...	92.217	4
8	00kTY2h5rdJwP600uhCDkT	...	109.494	4
9	00qTlNBM62qVriou9qedWC	...	109.924	1
10	012eyTyyRCHaWTOhVGVMPR	...	106.776	4
11	017tyGFxZEvwmaZ65Gjt3v	...	209.841	4
12	01AaVwdRjbSIvolvCifrv5	...	67.533	4
13	01JPQ87UHeGysPVwTqMJHK	...	99.967	4
14	01gpCBvSLYcpMjbLPykc1B	...	133.716	5
15	01ie0dwhOQIvsSi6BKAbv8	...	120.709	4
16	01kKvDxBNQ6WZNQSpSWPyV	...	165.826	4
17	021jmGJUrMmS92WnBWvoAC	...	84.94	5
18	02cC87YyLKhCQjU8IH1Frm	...	79.983	4
19	02lI7Q5IBgWpl4CNiB5k8h	...	147.319	3
20	036VdTP0ggdePwbvbFuT8w	...	172.872	3
21	03AMfxhiuPDtqVBUGdzFOU	...	79.284	4
22	03HF18PyO2aob3htRSW0js	...	80.982	4
23	03L76Pappow8L5V2fK71wk	...	84.146	4
24	03MB4uIPXZJ63F1HQk0Bns	...	71.56	3
25	03TIKoaX56q27pRyYZ9zPe	...	137.511	4
26	03YZ3SmELE9LlojJG3GgEl	...	118.841	4
27	03w0JAy22hxUDRKLNPradE	...	116.326	4
28	04N18CfIbOJPnVLGOKgJNB	...	89.128	4
29	04RedUeDN5QjRSJveIz6vp	...	137.664	5
30	04wUuScZcskGGpnSEt5Vmb	...	96.085	4
31	058A8NFVMcfRphgOpxXH9M	...	130.923	4
32	05CBEqQ8CWilFUNW7vYPi9	...	72.753	4
33	05CM8z8uYj064zvn6YOAHj	...	88.292	4
34	05E60GLnBS8eMhcAsIbfpM	...	122.762	4
35	05T5cnFmAXiiBWV5wtM3tX	...	113.17	5
36	05UEdT06NxdH4JTgltDnaw	...	76.438	4
37	05r5QhWMq0T0EQDhYZouLO	...	80.554	4
38	064WS0WvMzTXKl0xx45ryF	...	102.066	4
39	06FoaUqbBSpO4crPQ1GXrF	...	86.615	4
40	06GtF3borSXnkXfCyeUUmX	...	118.773	4
41	06LtO6ogcuQktXe0pQghdh	...	125.296	3
42	06NEuecnoNSQxDC3lfHpyy	...	185.93	4
43	06XszKvYdMfzwpoxHDnpun	...	109.241	1
44	06uDJ8qAO6btD89f4aMlEk	...	99.884	4
45	07KD6sXOexgCnzd2ux4uTe	...	120.644	4
46	07UmbTUVteqSGrFZkMdBvG	...	188.35	4
47	07XHoD2XHmzgxGHhWw7fLx	...	116.461	4
48	07c11BxeE28llfHSPSRqFv	...	85.052	4
49	07kKdEH4zfobUFfahc7RcW	...	66.65	4
50	07rAh606hDOjnTaHlJoT0C	...	131.926	1
51	07ubKGIpVTG49naAmpe8gt	...	62.269	3
52	0874YaGRpvsVw92BDQZr2A	...	165.564	3
53	08CVQP4qjws8a6DtNG3aVK	...	162.407	3
54	08SpaYq4TuWsuqHoxJg3xM	...	116.661	4
55	08kHpj5FOW7LjNyVBz5Zvs	...	142.154	5
56	08lHFwOBB8FnLwzuGwYMEh	...	145.995	4
57	08rO8bafPdrmQssvsAMxVl	...	85.357	4
58	08tC0ditym84D3NBG0jQqA	...	78.073	4
59	093IYCNJHNy1cvAH0Mvkch	...	76.064	4
60	09Qxk2wRRsicvUtcLvtSTo	...	109.458	3
61	09aFzJT56dpsZfN8O2oR6U	...	183.615	5
62	09bZ7ATMlxA7LNjCCoCzdA	...	120.021	4
63	09dn63xIPf3n72HNizEElm	...	207.672	4
64	0A2BKRDiiNpiRYtiOSajYX	...	81.208	4
65	0ABNJ6silC7NHkocdiUxTF	...	70.806	4
66	0ACKq3I2568VVrgcpwFkOI	...	77.069	4
67	0AgnrqcD0dr22vaZeroGOX	...	89.372	4
68	0AjhUemO2FH31c4Bda00tO	...	65.767	3
69	0AwtRpubA0XprHmhMkeFQf	...	111.255	4
70	0B4HWkQHhAzxW43V7Z0fQO	...	120.624	4
71	0B6RBLFwXxp4CqOJ02SqdT	...	85.416	4
72	0BNYfaz6wi8hwz2yWfGciG	...	118.967	4
73	0BZK5owhrgOW72f0CAjDYB	...	150.348	3
74	0BqR2YNnnicAYkkNFkHLvR	...	143.733	3
75	0CGFSEHPCoxIql2t8D5Kjc	...	173.413	4
76	0CK9b2dEDscskyQ0Xd7L4p	...	169.937	4
77	0Cbra96lST7uHGZjSpfR46	...	152.692	4
78	0Cmq2tmC0QGFVgliknr0MN	...	128.516	5
79	0CrCsWktPpP2yO8hSKnCsy	...	91.397	4
80	0D12JQoW6ljmJM86pgrJ1z	...	90.68	4
81	0D1pEisM3QkiacGXJe5dmd	...	122.909	4
82	0DA0LWjbdBTmrlKr2CmQ62	...	117.513	4
83	0DXiJ6NXLtD5SRzsOHPQPI	...	120.306	4
84	0DplGIOdpwkzdwKbOClMlJ	...	172.761	3
85	0DtqOgGu29ZUPyhlRdB54g	...	100.47	4
86	0Dw6qWZblJ3cUKBobHHGlK	...	132.823	4
87	0E0nqXEw6HtFEqsQTTOMcJ	...	100.011	4
88	0E2UM5021iM76L4zvuIsGK	...	96.603	4
89	0EAJ0xn42jBzzD9BRJA0I8	...	114.783	4
90	0EDwfMkuOLWqZmEiYrAVIi	...	108.523	5
91	0EOYB5Dhk3YeLrjvajYiIj	...	86.964	4
92	0EPBPqrmLxVl3UXzcJuqVj	...	80.772	4
93	0EQufZ7uB8mcnISIKfQtyJ	...	118.425	4
94	0ESg82SK2vuKfd8xfZ0f17	...	102.436	4
95	0EcQqsrUkVhPZsPc7InZwS	...	117.78	3
96	0Eh0NJcwxUrtIsyjsNpv1e	...	87.271	3
97	0EzFMcpJw2BaOpq4klnCbf	...	97.285	3
98	0F24HA5PftFkfETekL0cwm	...	123.291	4
99	0FA6uDYBkJ0yqiZwQZHwgg	...	139.878	4
100	0FVH9hZ1Xn1jgl0ehI5wjh	...	125.484	4

Warning

This example uses a sample dataset. For the full analysis, you should consider using the complete dataset.

Since we are only focusing on Polish artists in this subset of data, let us save it in the database with the proper name.

polish_artists = artists

# save it to the database
polish_artists.to_db('"spotify"."polish_artists"', relation_type = "table")

Data Exploration¶

We can visualize the top 60 most-followed Polish artists with a bar chart.

# make a highchart of the top 50 most-followed Polish artists
polish_artists.bar(
    ["name"],
    method = "mean",
    of = "followers",
    max_cardinality = 50,
    width = 800,
)

We can do the same with the most popular tracks. For example, we can graph Monika Brodka’s most popular tracks like so:

# find Monika Brodka's songs
brodka_tracks = tracks.search("artists ilike '%brodka%'")

# plot Brodka's tracks ordered by popularity
brodka_tracks.bar(
    ["name"],
    method = "mean",
    of = "popularity",
    max_cardinality = 25,
    width = 800,
)

To get an idea of what makes Monika Brodka’s songs popular, let’s create a boxplot of the numerical feature distribution of her tracks.

## list of the relevant numerical features
numerical_features = [
    "danceability",
    "energy",
    "speechiness",
    "acousticness",
    "instrumentalness",
    "valence",
    "liveness",
]

# create a boxplot of the above features
brodka_tracks.boxplot(columns = numerical_features)

Timing is a classic factor for success, so let’s look at the popularity of Monika’s songs over time with a smooth curve.

# extract year from the date
brodka_tracks["release_year"] = "year(release_date::date)"

# smooth the popularity using rolling mean
brodka_tracks.rolling(
    func = "mean",
    columns = "popularity",
    window = (-3, 3),
    order_by = "release_year",
    name = "smoothed_popularity",
)

# plot the smoothed curve for popularity of her songs
brodka_tracks.plot(ts = "release_date", columns=["smoothed_popularity"])

Numerical-feature Analysis¶

Bringing it all together, let’s try to get an idea of how these numerical features change and correlate with each other in Monika’s most popular songs.

# extract year from date
tracks["release_year"] = "year(release_date::date)"

# get the average of numerical features during the year
yearly_aggs = tracks.groupby(
    "release_year", [
        "AVG(danceability) as danceability",
        "AVG(energy) as energy",
        "AVG(speechiness) AS speechiness",
        "AVG(acousticness) AS acousticness",
        "AVG(instrumentalness) AS instrumentalness",
        "AVG(valence) AS valence",
        "AVG(liveness) AS liveness",
    ]
)


# plot the cures for numerical features along the different years
yearly_aggs.plot(
    ts = "release_year",
    columns = numerical_features,
)

# correlation of numerical features
tracks[tracks[numerical_features]].corr()

Feature Engineering¶

To expand our analysis, let’s take into account some descriptive features. Since our goal is to predict popularity, some useful features might be:

number of followers
popularity for the artist of the track
the number of artists per track

Additionally, we manipulate our data a bit to make things easier later on:

converting the duration unit from ms to minute
extracting the year from the date.

%%sql
DROP TABLE IF EXISTS spotify.polish_tracks;
CREATE TABLE spotify.polish_tracks AS
SELECT * FROM spotify.tracks
WHERE id_artists IN (SELECT t.id_artists FROM spotify.tracks t JOIN spotify.polish_artists p
                    ON t.id_artists LIKE '%' || p.id || '%');
CREATE TABLE spotify.polish_tracks_clean AS
SELECT
    x.*,
    x.duration_ms / 60000 AS duration_minute,
    x.release_date::date AS release_year,
    y.followers AS artists_followers,
    y.popularity AS artist_popularity
FROM spotify.polish_tracks AS x LEFT JOIN spotify.artists AS y
ON x.id_artists LIKE '%' || y.id || '%';

polish_tracks = vp.vDataFrame("spotify.polish_tracks_clean")

# count the number of artists per track
polish_tracks.regexp(
    column = "artists",
    pattern = ",",
    method = "count",
    name = "nb_singers",
)
polish_tracks["nb_singers"].add(1)

	Abc id Varchar(44) 100%	...	123 artist_popularity Int 100%	123 nb_singers Integer 100%
1	00K3yuFVb33hBNoEyWcdnj	...	53	1
2	04PxH7CFGAaAvo6j0zZAOr	...	52	3
3	04PxH7CFGAaAvo6j0zZAOr	...	50	3
4	04PxH7CFGAaAvo6j0zZAOr	...	45	3
5	06kPUkFJv1RfEVjgEF3oRz	...	47	1
6	07FeXj3ssNHhy86oX5IIKI	...	53	1
7	0ESg82SK2vuKfd8xfZ0f17	...	47	1
8	0H9SRjTU6FJ1wJ5DMtR6wV	...	67	2
9	0H9SRjTU6FJ1wJ5DMtR6wV	...	68	2
10	0KfikiN9pPQ80nIOgvGGkd	...	53	1
11	0SwAC0yvUxGq2frHlNCfvE	...	53	1
12	0YZFbFNA1r8Q649MAaAy1C	...	53	1
13	0byNwHtvPr2vIO9cbwyZjH	...	47	1
14	0geug0mKXXEuE0VEvoZ8kp	...	47	1
15	0jbwghDbZd9dHrUa4oHakc	...	53	1
16	0kXr0uVZTAzjWywvWBkFXf	...	53	1
17	0kzWBjzgUuzRAquOp5L9BR	...	47	1
18	15LfHBNBrTKHGce4SNtOR6	...	52	2
19	15LfHBNBrTKHGce4SNtOR6	...	45	2
20	1OWseQo2xtPDNxFAiAVyDu	...	47	1

Define a list of predictors and the response, and then save the normalized version of the final dataset to the database.

# define predictors and response
predictors = [
    "duration_minute",
    # "release_year",
    "danceability",
    "energy",
    "loudness",
    "speechiness",
    "acousticness",
    "instrumentalness",
    "liveness",
    "valence",
    "artists_followers",
    "artist_popularity",
    "nb_singers",
]
response = "popularity"
polish_tracks.normalize(
    method = "minmax",
    columns = predictors,
)
# save the final dataset to the database
vp.drop("spotify.polish_tracks_data_final")
polish_tracks.to_db('"spotify"."polish_tracks_data_final"', relation_type = "table")

Machine Learning¶

We can use AutoML to easily get a well-performing model.

# define a random seed so models tested by AutoML produce consistent results
vp.set_option("random_state", 2)

AutoML automatically tests several machine learning models and picks the best performing one.

from verticapy.machine_learning.vertica.automl import AutoML

# define the model
auto_model = AutoML(
    "spotify.automl_spotify_polish",
    estimator = "fast",
    preprocess_data = True,
    stepwise = False,
    cv = 2,
)

Train the model.

auto_model.fit(
    "spotify.polish_tracks_data_final",
    predictors,
    response
)

[1m[4mStarting AutoML[0m[0m


[1m[4mTesting Model - LinearRegression[0m[0m

Model: LinearRegression; Parameters: {'tol': 1e-06, 'max_iter': 100, 'solver': 'newton'}; [91mTest_score: 8.03653061563296[0m; [92mTrain_score: 7.6086948476505[0m; [94mTime: 2.5748002529144287[0m;
Model: LinearRegression; Parameters: {'tol': 1e-06, 'max_iter': 100, 'solver': 'bfgs'}; [91mTest_score: 8.03652451786366[0m; [92mTrain_score: 7.60869496994112[0m; [94mTime: 7.14464271068573[0m;
[1mGrid Search Selected Model[0m
LinearRegression; Parameters: {'tol': 1e-06, 'max_iter': 100, 'solver': 'bfgs', 'fit_intercept': True}; [91mTest_score: 8.03652451786366[0m; [92mTrain_score: 7.60869496994112[0m; [94mTime: 7.14464271068573[0m;

[1m[4mTesting Model - ElasticNet[0m[0m

Model: ElasticNet; Parameters: {'tol': 1e-06, 'max_iter': 100, 'solver': 'cgd', 'C': 1.0, 'l1_ratio': 0.5}; [91mTest_score: 8.400499108916339[0m; [92mTrain_score: 8.31347605013778[0m; [94mTime: 3.1259127855300903[0m;
[1mGrid Search Selected Model[0m
ElasticNet; Parameters: {'tol': 1e-06, 'C': 1.0, 'max_iter': 100, 'solver': 'cgd', 'l1_ratio': 0.5, 'fit_intercept': True}; [91mTest_score: 8.400499108916339[0m; [92mTrain_score: 8.31347605013778[0m; [94mTime: 3.1259127855300903[0m;

[1m[4mTesting Model - Ridge[0m[0m

Model: Ridge; Parameters: {'tol': 1e-06, 'max_iter': 100, 'C': 1.0}; [91mTest_score: 8.02461535485319[0m; [92mTrain_score: 7.60920752917329[0m; [94mTime: 2.7998028993606567[0m;
[1mGrid Search Selected Model[0m
Ridge; Parameters: {'tol': 1e-06, 'C': 1.0, 'max_iter': 100, 'solver': 'newton', 'fit_intercept': True}; [91mTest_score: 8.02461535485319[0m; [92mTrain_score: 7.60920752917329[0m; [94mTime: 2.7998028993606567[0m;

[1m[4mTesting Model - Lasso[0m[0m

Model: Lasso; Parameters: {'tol': 1e-06, 'max_iter': 100, 'solver': 'cgd', 'C': 1.0}; [91mTest_score: 8.375940164726595[0m; [92mTrain_score: 8.060900854713466[0m; [94mTime: 2.9285014867782593[0m;
[1mGrid Search Selected Model[0m
Lasso; Parameters: {'tol': 1e-06, 'C': 1.0, 'max_iter': 100, 'solver': 'cgd', 'fit_intercept': True}; [91mTest_score: 8.375940164726595[0m; [92mTrain_score: 8.060900854713466[0m; [94mTime: 2.9285014867782593[0m;
[1m[4mFinal Model[0m[0m

Ridge; Best_Parameters: {'tol': 1e-06, 'C': 1.0, 'max_iter': 100, 'solver': 'newton', 'fit_intercept': True}; [91mBest_Test_score: 8.02461535485319[0m; [92mTrain_score: 7.60920752917329[0m; [94mTime: 2.7998028993606567[0m;

auto_model.plot()

Extract the best model according to AutoML. From here, we can look at the model type and its hyperparameters.

# extract the model type and hyperparameters
best_model = auto_model.best_model_

bm_type = best_model._model_type

hyperparams = best_model.get_params()

print(bm_type)
Ridge

print(hyperparams)
{'tol': 1e-06, 'C': 1.0, 'max_iter': 100, 'solver': 'newton', 'fit_intercept': True}

Thanks to AutoML, we know best model type and its hyperparameters. Let’s create a new model with this information in mind.

from verticapy.machine_learning.vertica import LinearRegression

# define the model
rf_model = LinearRegression("spotify.linear_regression_spotify", **hyperparams)

# train the model
rf_model.fit(polish_tracks, predictors, response)

# use the model to predict
rf_model.predict(
    polish_tracks,
    name = "estimated_popularity",
)

	Abc id Varchar(44) 100%	...	Abc Varchar(196) 100%	123 estimated_popularity Float(22) 100%
1	00K3yuFVb33hBNoEyWcdnj	...		32.687515815542
2	04PxH7CFGAaAvo6j0zZAOr	...		42.3585385142475
3	04PxH7CFGAaAvo6j0zZAOr	...		42.0822912902126
4	04PxH7CFGAaAvo6j0zZAOr	...		38.7309512422115
5	06kPUkFJv1RfEVjgEF3oRz	...		28.1637982132495
6	07FeXj3ssNHhy86oX5IIKI	...		23.7486119838123
7	0ESg82SK2vuKfd8xfZ0f17	...		23.7995214149152
8	0H9SRjTU6FJ1wJ5DMtR6wV	...		56.0540426695094
9	0H9SRjTU6FJ1wJ5DMtR6wV	...		58.8944789378883
10	0KfikiN9pPQ80nIOgvGGkd	...		34.2927702242878
11	0SwAC0yvUxGq2frHlNCfvE	...		42.0160017131988
12	0YZFbFNA1r8Q649MAaAy1C	...		28.5560009504502
13	0byNwHtvPr2vIO9cbwyZjH	...		31.3197072121662
14	0geug0mKXXEuE0VEvoZ8kp	...		31.5709660952455
15	0jbwghDbZd9dHrUa4oHakc	...		38.8080040064003
16	0kXr0uVZTAzjWywvWBkFXf	...		35.11050094137
17	0kzWBjzgUuzRAquOp5L9BR	...		29.1632205931634
18	15LfHBNBrTKHGce4SNtOR6	...		43.2455014418643
19	15LfHBNBrTKHGce4SNtOR6	...		39.6179141698283
20	1OWseQo2xtPDNxFAiAVyDu	...		30.3224018170893

View the regression report and the importance of each feature.

rf_model.regression_report()

	value
explained_variance	0.686560783508525
max_error	23.2314079669432
median_absolute_error	5.26887801846472
mean_absolute_error	6.14061095243189
mean_squared_error	58.2519664104863
root_mean_squared_error	7.63229758398389
r2	0.686560783508526
r2_adj	0.675432763988118
aic	1454.82011140199
bic	1502.92724625364

rf_model.features_importance()

To see how our model performs, let’s plot the popularity and estimated popularity of songs by other Polish artists like Brodka and Akcent.

# results for Brodka
polish_tracks.search(
        "LOWER(artists) LIKE '%brodka%'",
        usecols = [
            "popularity",
            "name",
            "estimated_popularity",
        ],
    ).plot(
    ts = "name",
    columns = ["popularity", "estimated_popularity"],
)

# results for Brodka
polish_tracks.search(
        "LOWER(artists) LIKE '%akcent%'",
        usecols = [
            "popularity",
            "name",
            "estimated_popularity",
        ],
    ).plot(
    ts = "name",
    columns = [
        "popularity",
        "estimated_popularity",
    ],
)

Group Artists using Track Features¶

While our tracks don’t have an explicit “genre” feature, we can approximate the effect by grouping artists based on their tracks’ numerical features.

Let’s start by taking the averages of these numerical features for each artist.

# group by artist
artists_features = polish_tracks.groupby(
    [
        "id_artists",
        "artists",
    ],
    expr=[
        "AVG(danceability) AS danceability",
        "AVG(energy) AS energy",
        "AVG(speechiness) AS speechiness",
        "AVG(acousticness) AS acousticness",
        "AVG(instrumentalness) AS instrumentalness",
        "AVG(valence) AS valence",
        "AVG(liveness) AS liveness",
    ]
)

# save relation to the database as "artists_features"
artists_features.to_db('"spotify"."artists_features"')

	Abc Varchar(364) 100%	...	123 valence Float(22) 100%	123 liveness Float(22) 100%
1		...	0.602496714848883	0.214027245636441
2		...	0.4437144108629	0.157088122605364
3		...	0.687910643889619	0.038527032779906
4		...	0.791940429259746	0.063005534269902
5		...	0.325571358635438	0.12835875090777
6		...	0.676960140166448	0.22585598735024
7		...	0.495181778361805	0.113452532992763
8		...	0.576215505913272	0.291187739463602
9		...	0.253723171265878	0.121434653043848
10		...	0.430573806395094	0.35823754789272
11		...	0.545554095488392	0.323116219667944
12		...	0.153526062198861	0.112388250319285
13		...	0.370346035917652	0.053533418475947
14		...	0.614542268944372	0.090570455512984
15		...	0.471528690319755	0.251947637292465
16		...	0.417323696890057	0.154640272456364
17		...	0.311760840998686	0.081364410387399
18		...	0.681340341655716	0.097488292890592
19		...	0.421265878230399	0.243827160493827
20		...	0.300262812089356	0.154959557258408

Grouping means clustering, so we use an elbow() curve to find a suitable number of clusters.

from verticapy.machine_learning.model_selection import elbow

# define numerical features
preds = [
    "danceability",
    "energy",
    "speechiness",
    "acousticness",
    "instrumentalness",
    "liveness",
    "valence",
]


# elbow curve
elbow_curve = elbow(
    '"spotify"."artists_features"',
    preds,
    n_cluster = (1, 20),
    show = True,
)

elbow_curve

Let’s define and use the Vertica KMeans algorithm to create a model that can group artists together.

from verticapy.machine_learning.vertica.cluster import KMeans

# define k-means
model = KMeans(
    '"spotify"."KMeans_spotify"',
    n_cluster = 7,
)

We can train our new model on the artists_features relation we saved earlier.

# train the model
model.fit(
    '"spotify"."artists_features"',
    X = preds,
)



=======
centers
=======
danceability| energy |speechiness|acousticness|instrumentalness|liveness|valence 
------------+--------+-----------+------------+----------------+--------+--------
   0.74755  | 0.54774|  0.23642  |   0.20312  |     0.00549    | 0.09440| 0.35149
   0.45502  | 0.44870|  0.02991  |   0.22027  |     0.14763    | 0.15310| 0.52708
   0.76389  | 0.55595|  0.43462  |   0.26255  |     0.00187    | 0.11664| 0.68244
   0.52136  | 0.49604|  0.27668  |   0.38858  |     0.00547    | 0.15539| 0.26164
   0.60670  | 0.72327|  0.10514  |   0.10965  |     0.03246    | 0.26350| 0.51603
   0.75753  | 0.61068|  0.35731  |   0.67534  |     0.00002    | 0.14746| 0.68852
   0.50817  | 0.37596|  0.18257  |   0.88200  |     0.00000    | 0.08365| 0.00000


=======
metrics
=======
Evaluation metrics:
     Total Sum of Squares: 13.70887
     Within-Cluster Sum of Squares: 
         Cluster 0: 0.42703059
         Cluster 1: 0.79885754
         Cluster 2: 0.99103343
         Cluster 3: 1.6490998
         Cluster 4: 1.3303828
         Cluster 5: 1.008654
         Cluster 6: 0
     Total Within-Cluster Sum of Squares: 6.2050582
     Between-Cluster Sum of Squares: 7.5038122
     Between-Cluster SS / Total SS: 54.74%
 Number of iterations performed: 4
 Converged: True
 Call:
kmeans('"spotify"."KMeans_spotify"', '"spotify"."artists_features"', '"danceability", "energy", "speechiness", "acousticness", "instrumentalness", "liveness", "valence"', 7
USING PARAMETERS max_iterations=300, epsilon=0.0001, init_method='kmeanspp', distance_method='euclidean')

Plot the result of the k-means algoritm:

model.plot()

# predict the genres
pred_genres = model.predict(
    '"spotify"."artists_features"',
    X = [
        "danceability",
        "energy",
        "speechiness",
        "acousticness",
        "instrumentalness",
        "liveness",
        "valence",
    ],
    name = "pred_genres",
)

Let’s see how our model groups these artists together:

# observe the results
pred_genres["artists", "pred_genres"].sort({"pred_genres": "desc"})

	Abc Varchar(344) 100%	123 pred_genres Integer 100%
1		6
2		5
3		5
4		5
5		5
6		5
7		5
8		5
9		5
10		5
11		4
12		4
13		4
14		4
15		4
16		4
17		4
18		4
19		4
20		4

Conclusion¶

We were able to predict the popularity Polish songs with a RandomForestRegressor model suggested by AutoML. We then created a KMeans model to group artists into genres (clusters) based on the feature-commonalities in their tracks.