Feedback

×

• Feedback
• Was this page helpful?*
  • Yes
  • No
• Do you have any additional feedback?
  • Yes
  • No
• Additional Feedback

Movies Scoring and Clustering

This example uses the filmtv_movies dataset to evaluate the quality of the movies and create clusters of similar movies. You can download the Jupyter notebook here.

The columns provided include:

• year: Movie’s release year.

• filmtv_id: Movie ID.

• title: Movie title.

• genre: Movie genre.

• country: Movie’s country of origin.

• description: Movie description.

• notes: Information about the movie.

• duration: Movie duration.

• votes: Number of votes.

• avg_vote: Average score.

• director: Movie director.

• actors: Actors in the movie.

We will follow the data science cycle (Data Exploration - Data Preparation - Data Modeling - Model Evaluation - Model Deployment) to solve this problem.

Initialization

This example uses the following version of VerticaPy:

import verticapy as vp

vp.__version__
Out[2]: '1.1.0'

Connect to Vertica. This example uses an existing connection called VerticaDSN. For details on how to create a connection, see the Connection tutorial. You can skip the below cell if you already have an established connection.

vp.connect("VerticaDSN")

Let’s create a new schema and assign the data to a vDataFrame object.

vp.drop("movies", method="schema")
vp.create_schema("movies")
filmtv_movies = vp.read_csv("movies.csv", schema = "movies")
filmtv_movies.head(5)

Let’s take a look at the first few entries in the dataset.

	123 filmtv_id Int 100%	...	Abc Varchar(2232) 0%	Abc Varchar(1052) 0%
1	24	...
2	30	...
3	64	...
4	65	...
5	115	...

Data Exploration and Preparation

One of the biggest challenges for any streaming platform is to find a good catalog of movies.

First, let’s explore the dataset.

filmtv_movies.describe(method = "categorical", unique = True)

	...	top_percent	unique
"filmtv_id"	...	0.002	53327.0
"title"	...	0.019	50524.0
"year"	...	3.096	111.0
"genre"	...	30.137	27.0
"duration"	...	11.808	279.0
"country"	...	41.142	2394.0
"director"	...	0.137	19146.0
"actors"	...	5.663	50056.0
"avg_vote"	...	15.019	89.0
"votes"	...	22.986	588.0
"description"	...	99.458	284.0
"notes"	...	99.932	35.0

We can drop the description and notes columns since these fields are empty for most of our dataset.

filmtv_movies.drop(["description", "notes"])

	123 filmtv_id Int 100%	...	Abc title Varchar(486) 99%	123 votes Int 100%
1	24	...	A... come assassino	3
2	30	...	A Ghentar si muore facile	2
3	64	...	A tutte le volanti	1
4	65	...	Gas	1
5	115	...	Goodbye, My Lady	3
6	116	...	Ring of Bright Water	1
7	128	...	Lady Possessed	1
8	180	...	Rhino!	2
9	181	...	Die Flusspiraten vom Mississippi	2
10	188	...	Aida	7
11	201	...	Al Capone	11
12	205	...	Die Nackt und der Satan	5
13	207	...	Al di là della legge	21
14	208	...	All the Way Home	7
15	211	...	Checking Out	2
16	223	...	The Hanging Tree	64
17	224	...	Raintree County	21
18	231	...	Alfred the Great	6
19	232	...	Q Planes	4
20	234	...	The Wizard of Baghdad	1

We have access to more than 50000 movies in 27 different genres. Let’s organize our list by their average rating.

filmtv_movies.sort({"avg_vote" : "desc"})

	123 filmtv_id Int 100%	...	Abc Varchar(486) 99%	123 votes Int 100%
1	76646	...		1
2	77950	...		1
3	86071	...		1
4	80205	...		1
5	79888	...		1
6	86266	...		1
7	122810	...		1
8	128665	...		1
9	86042	...		1
10	84535	...		1
11	139242	...		1
12	121261	...		2
13	86945	...		1
14	141940	...		3
15	138920	...		1
16	134090	...		1
17	145657	...		1
18	129161	...		1
19	146073	...		1
20	139825	...		1

Since we want properly averaged scores, let’s just consider the top 10 movies that have at least 10 votes.

filmtv_movies.search(
    conditions = [filmtv_movies["votes"] > 10],
    order_by = {"avg_vote" : "desc" },
)

	123 filmtv_id Integer 100%	...	Abc Varchar(486) 100%	123 votes Integer 100%
1	25980	...		33
2	16567	...		58
3	29136	...		24
4	7042	...		427
5	7753	...		535
6	3831	...		477
7	5648	...		588
8	23395	...		325
9	27908	...		68
10	16584	...		30
11	12135	...		72
12	10423	...		29
13	5580	...		789
14	26035	...		31
15	23112	...		125
16	18173	...		148
17	27984	...		18
18	23023	...		50
19	12730	...		514
20	28382	...		17

We can see classic movies like The Godfather and Greed. Let’s smooth the avg_vote using a linear regression to make it more representative.

To create our model we could use the votes, the category, the duration, etc. but let’s go with the director and main actors.

We can extract the five main actors for each movie with regular expressions.

for i in range(1, 5):
    filmtv_movies2 = vp.read_csv("movies.csv")
    filmtv_movies2.regexp(
        column = "actors",
        method = "substr",
        pattern = '[^,]+',
        occurrence = i,
        name = "actor",
    )
    if i == 1:
        filmtv_movies = filmtv_movies2.copy()
    else:
        filmtv_movies = filmtv_movies.append(filmtv_movies2)
filmtv_movies["actor"].describe()

By aggregating the data, we can find the number of actors and the number of votes by actor. We can then normalize the data using the min-max method and quantify the notoriety of the actors.

import verticapy.sql.functions as fun

actors_stats = filmtv_movies.groupby(
    columns = ["actor"],
    expr = [
        fun.sum(filmtv_movies["votes"])._as("notoriety_actors"),
        fun.count(filmtv_movies["actors"])._as("castings_actors"),
    ],
)
actors_stats["actor"].dropna()
actors_stats["notoriety_actors"].normalize(method = "minmax")

	Abc actor Varchar(2218) 100%	...	123 notoriety_actors Float 100%	123 castings_actors Integer 100%
1	Aladin Reibel	...	0.000813256074006	4
2	Ann Miller	...	0.081630578428383	4
3	Gwendolyn Laster	...	0.000711599064756	1
4	Andrew Scott	...	0.001423198129511	3
5	Wu Nien-chen	...	0.0	1
6	Salvo Piparo	...	0.001016570092508	1
7	Jay Rodan	...	0.005286164481041	3
8	Maria Conchita Alonso	...	0.039442919589306	19
9	Arsinée Khanjian	...	0.003151367286774	2
10	Emily Bett Rickards	...	0.011792213073091	1
11	Joan Gardner	...	0.000813256074006	4
12	Nick Hanauer	...	0.000203314018502	1
13	Danielle De Metz	...	0.001829826166514	3
14	Lena Endre	...	0.001626512148013	2
15	Leigh J. McCloskey	...	0.0	1
16	Clark Fyans	...	0.000101657009251	1
17	Helgi Skúlason	...	0.000101657009251	1
18	Dulcie Gray	...	0.000101657009251	2
19	Ravi Menon	...	0.0	1
20	Roderick Cook	...	0.0	1

Let’s look at the top ten actors by notoriety.

actors_stats.search(
    order_by = {
        "notoriety_actors" : "desc",
        "castings_actors" : "desc",
    },
).head(10)

	Abc actor Varchar(2218) 100%	...	123 notoriety_actors Numeric(36) 100%	123 castings_actors Integer 100%
1	Robert De Niro	...	1.0	57
2	Morgan Freeman	...	0.861441496391176	52
3	Clint Eastwood	...	0.856460302937888	43
4	Tom Cruise	...	0.820372064653858	34
5	Johnny Depp	...	0.814882586154315	34
6	Tom Hanks	...	0.771881671241232	37
7	Samuel L. Jackson	...	0.742502795567754	49
8	Brad Pitt	...	0.72776252922639	26
9	Leonardo DiCaprio	...	0.715462031107045	20
10	Al Pacino	...	0.648368405001525	40

As expected, we get a list of very popular actors like Robert De Niro, Morgan Freeman, and Clint Eastwood.

Let’s do the same for the directors.

director_stats = filmtv_movies.groupby(
    columns = ["director"],
    expr = [
        fun.sum(filmtv_movies["votes"])._as("notoriety_director"),
        fun.count(filmtv_movies["director"])._as("castings_director"),
    ],
)
director_stats["notoriety_director"].normalize(method = "minmax")

	Abc Varchar(1066) 99%	...	123 notoriety_director Float 100%	123 castings_director Integer 100%
1		...	0.0	4
2		...	0.000346620450607	4
3		...	8.6655112652e-05	4
4		...	0.001473136915078	4
5		...	0.0	4
6		...	0.002599653379549	4
7		...	0.002859618717504	8
8		...	0.0	4
9		...	0.000259965337955	4
10		...	0.003379549393414	8
11		...	0.0	4
12		...	0.005459272097054	16
13		...	0.000346620450607	4
14		...	0.0	4
15		...	0.000779896013865	24
16		...	0.00103986135182	4
17		...	0.000173310225303	4
18		...	8.6655112652e-05	4
19		...	0.000433275563258	12
20		...	0.000606585788562	4

Now let’s look at the top 10 movie directors.

director_stats.search(
    order_by = {
        "notoriety_director" : "desc",
        "castings_director" : "desc",
    },
).head(10)

	Abc director Varchar(1066) 99%	...	123 notoriety_director Numeric(36) 100%	123 castings_director Integer 100%
1	Steven Spielberg	...	1.0	132
2	Woody Allen	...	0.962045060658579	192
3	Clint Eastwood	...	0.893067590987868	152
4	Martin Scorsese	...	0.829289428076256	140
5	Alfred Hitchcock	...	0.753379549393414	208
6	Quentin Tarantino	...	0.686741767764298	40
7	Ridley Scott	...	0.655459272097054	108
8	Stanley Kubrick	...	0.649913344887348	48
9	Tim Burton	...	0.588214904679376	72
10	David Cronenberg	...	0.513431542461005	84

Again, we get a list of popular directors like Steven Spielberg, Woody Allen, and Clint Eastwood.

Let’s join our notoriety metrics for actors and directors with the main dataset.

filmtv_movies_director = filmtv_movies.join(
    director_stats,
    on = {"director": "director"},
    how = "left",
    expr1 = ["*"],
    expr2 = [
        "notoriety_director",
        "castings_director",
    ],
)


filmtv_movies_director_actors = filmtv_movies_director.join(
    actors_stats,
    on = {"actor": "actor"},
    how = "left",
    expr1 = ["*"],
    expr2 = [
        "notoriety_actors",
        "castings_actors",
    ],
)

As we did many operation, it can be nice to save the vDataFrame as a table in the Vertica database.

vp.drop("filmtv_movies_director_actors", method = "table")
filmtv_movies_director_actors.to_db(
    name = "filmtv_movies_director_actors",
    relation_type = "table",
    inplace = True,
)

We can aggregate the data to get metrics on each movie.

filmtv_movies_complete = filmtv_movies_director_actors.groupby(
    columns = [
        "filmtv_id",
        "title",
        "year",
        "genre",
        "country",
        "avg_vote",
        "votes",
        "duration",
        "director",
        "notoriety_director",
        "castings_director",
    ],
    expr = [
        fun.sum(filmtv_movies_director_actors["notoriety_actors"])._as("notoriety_actors"),
        fun.sum(filmtv_movies_director_actors["castings_actors"])._as("castings_actors"),
    ],
)

Let’s compute some statistics on our dataset.

filmtv_movies_complete.describe(method = "all")

	...	Abc "country" Varchar(208) 99%	Abc "director" Varchar(1066) 99%
dtype	...	varchar(208)	varchar(1066)
percent	...	99.904	99.884
count	...	53276	53265
top	...	United States	Mario Mattòli
top_percent	...	41.142	0.137
avg	...	11.4996058262632	14.9825213554867
stddev	...	6.36585063280328	7.95519700230993
min	...	4	3
approx_25%	...	6	12
approx_50%	...	13	14
approx_75%	...	13	16
max	...	104	532
range	...	100	529
empty	...	0	0

We can use the movie’s release year to get create three categories.

filmtv_movies_complete.case_when(
    "period",
    filmtv_movies_complete["year"] < 1990, "Old",
    filmtv_movies_complete["year"] >= 2000, "Recent", "90s",
)

	123 filmtv_id Integer 100%	...	Abc Varchar(486) 99%	Abc period Varchar(6) 100%
1	35840	...		Old
2	46124	...		Recent
3	20158	...		Old
4	16557	...		90s
5	25485	...		Old
6	51429	...		Recent
7	52538	...		Recent
8	30731	...		Recent
9	155325	...		Recent
10	36664	...		Recent
11	37288	...		Recent
12	15595	...		Old
13	32656	...		Recent
14	34832	...		Recent
15	74068	...		Recent
16	6529	...		Old
17	20148	...		90s
18	1126	...		Old
19	40519	...		90s
20	142551	...		Recent

Now, let’s look at the countries that made the most movies.

filmtv_movies_complete.groupby(
    columns = ["country"],
    expr = ["COUNT(*)"]
).sort({"count" : "desc"}).head(10)

	Abc country Varchar(208) 99%	123 COUNT Integer 100%
1	United States	21940
2	Italy	9056
3	France	3040
4	Great Britain	2516
5	Germany	1697
6	Japan	1131
7	Canada	1053
8	Spain	562
9	Italy, France	403
10	Hong Kong	380

We can use this variable to create language groups.

# Language Discretization
Arabic_Middle_Est = [
    "Arab", "Iran", "Turkey", "Egypt", "Tunisia",
    "Lebanon", "Palestine", "Morocco", "Iraq",
    "Sudan", "Algeria", "Yemen", "Afghanistan",
    "Azerbaijan", "Kazakhstan", "Kyrgyzstan",
    "Kurdistan", "Syria", "Uzbekistan",
]


Chinese_Japan_Asian = [
    "Japan", "Hong Kong", "China", "South Korea",
    "Thailand", "Philippines", "Taiwan", "Indonesia",
    "Singapore", "Malaysia", "Vietnam", "Laos", "Cambodia",
    "Bhutan",
]


Indian = ["India", "Pakistan", "Nepal", "Sri Lanka", "Bangladesh",]

Hebrew = ["Israel",]

Spanish_Portuguese = [
    "Spain", "Portugal", "Mexico", "Brasil", "Chile",
    "Argentina", "Colombia", "Cuba", "Venezuela", "Peru",
    "Uruguay", "Dominican Republic", "Ecuador", "Guatemala",
    "Costa Rica", "Paraguay", "Bolivia",
]


English = [
    "United States", "England", "Great Britain", "Ireland",
    "Australia", "New Zealand", "South Africa",
]


French = ["France", "Canada", "Belgium", "Switzerland", "Luxembourg",]

Italian = ["Italy",]

German_North_Europe = [
    "German", "Austria", "Holland", "Netherlands", "Denmark",
    "Norway", "Iceland", "Finland", "Sweden", "Greenland",
]


Russian_Est_Europe = ["Russia", "Soviet Union", "Yugoslavia", "Czechoslovakia", "Poland", "Bulgaria", "Croatia", "Czech Republic", "Serbia", "Ukraine", "Slovenia", "Lithuania", "Latvia", "Estonia", "Bosnia and Herzegovina", "Georgia"]

Grec_Balkan = [
    "Greece", "Macedonia", "Cyprus", "Romania", "Armenia", "Hungary",
    "Albania", "Malta",
]

# Creation of the new feature
filmtv_movies_complete.case_when('language_area',
        vp.StringSQL("REGEXP_LIKE(Country, '{}')".format("|".join(Arabic_Middle_Est))), 'Arabic_Middle_Est',
        vp.StringSQL("REGEXP_LIKE(Country, '{}')".format("|".join(Chinese_Japan_Asian))), 'Chinese_Japan_Asian',
        vp.StringSQL("REGEXP_LIKE(Country, '{}')".format("|".join(Indian))), 'Indian',
        vp.StringSQL("REGEXP_LIKE(Country, '{}')".format("|".join(Hebrew))), 'Hebrew',
        vp.StringSQL("REGEXP_LIKE(Country, '{}')".format("|".join(Spanish_Portuguese))), 'Spanish_Portuguese',
        vp.StringSQL("REGEXP_LIKE(Country, '{}')".format("|".join(English))), 'English',
        vp.StringSQL("REGEXP_LIKE(Country, '{}')".format("|".join(French))), 'French',
        vp.StringSQL("REGEXP_LIKE(Country, '{}')".format("|".join(Italian))), 'Italian',
        vp.StringSQL("REGEXP_LIKE(Country, '{}')".format("|".join(German_North_Europe))), 'German_North_Europe',
        vp.StringSQL("REGEXP_LIKE(Country, '{}')".format("|".join(Russian_Est_Europe))), 'Russian_Est_Europe',
        vp.StringSQL("REGEXP_LIKE(Country, '{}')".format("|".join(Grec_Balkan))), 'Grec_Balkan',
        'Others')

	123 filmtv_id Integer 100%	...	Abc period Varchar(6) 100%	Abc language_area Varchar(19) 100%
1	59122	...	Old	French
2	161519	...	Recent	English
3	144169	...	Recent	Indian
4	40108	...	Recent	Italian
5	34236	...	Old	English
6	36000	...	Old	Italian
7	28139	...	Old	English
8	76652	...	Old	French
9	72927	...	Recent	French
10	57435	...	Old	German_North_Europe
11	16451	...	90s	French
12	13626	...	90s	English
13	4142	...	Old	English
14	36589	...	Recent	Italian
15	46277	...	Recent	Italian
16	32982	...	Old	German_North_Europe
17	17698	...	Old	English
18	31352	...	Old	Italian
19	173623	...	Recent	English
20	37082	...	90s	English

We can do the same for the genres.

filmtv_movies_complete.case_when(
        'Category',
        vp.StringSQL("REGEXP_LIKE(Genre, 'Drama|Noir')"), 'Drama',
        vp.StringSQL("REGEXP_LIKE(Genre, 'Comedy|Grotesque')"), 'Comedy',
        vp.StringSQL("REGEXP_LIKE(Genre, 'Fantasy|Super-hero')"), 'Fantasy',
        vp.StringSQL("REGEXP_LIKE(Genre, 'Romantic|Sperimental|Mélo')"), 'Romantic',
        vp.StringSQL("REGEXP_LIKE(Genre, 'Thriller|Crime|Gangster')"), 'Thriller',
        vp.StringSQL("REGEXP_LIKE(Genre, 'Action|Western|War|Spy')"), 'Action',
        vp.StringSQL("REGEXP_LIKE(Genre, 'Adventure')"), 'Adventure',
        vp.StringSQL("REGEXP_LIKE(Genre, 'Animation')"), 'Animation',
        vp.StringSQL("REGEXP_LIKE(Genre, 'Horror')"), 'Horror',
        'Others'
)

	123 filmtv_id Integer 100%	...	Abc Varchar(486) 99%	Abc Category Varchar(9) 100%
1	12170	...		Comedy
2	4116	...		Comedy
3	2425	...		Comedy
4	76988	...		Drama
5	40681	...		Romantic
6	7304	...		Comedy
7	83180	...		Drama
8	18604	...		Drama
9	20985	...		Horror
10	35916	...		Drama
11	6334	...		Comedy
12	51477	...		Thriller
13	79558	...		Others
14	12213	...		Drama
15	14673	...		Comedy
16	18262	...		Drama
17	5539	...		Thriller
18	37969	...		Drama
19	16214	...		Drama
20	70767	...		Drama

Since we’re more concerned with the Category at this point, we can drop genre.

filmtv_movies_complete.drop(columns = ["genre"])

Let’s look at the missing values.

filmtv_movies_complete.count_percent()

	...	count	percent
"filmtv_id"	...	53327.0	100.0
"avg_vote"	...	53327.0	100.0
"votes"	...	53327.0	100.0
"duration"	...	53327.0	100.0
"period"	...	53327.0	100.0
"language_area"	...	53327.0	100.0
"Category"	...	53327.0	100.0
"title"	...	53325.0	99.996
"year"	...	53317.0	99.981
"country"	...	53276.0	99.904
"director"	...	53265.0	99.884
"notoriety_director"	...	53265.0	99.884
"castings_director"	...	53265.0	99.884
"notoriety_actors"	...	50307.0	94.337
"castings_actors"	...	50307.0	94.337

Let’s impute the missing values for notoriety_actors and castings_actors using different techniques.

We can then drop the few remaining missing values.

filmtv_movies_complete["notoriety_actors"].fillna(
    method = "median",
    by = [
        "director",
        "Category",
    ],
)
filmtv_movies_complete["castings_actors"].fillna(
    method = "median",
    by = [
        "director",
        "Category",
    ],
)
filmtv_movies_complete.dropna()

	123 filmtv_id Integer 100%	...	Abc language_area Varchar(19) 100%	Abc Category Varchar(9) 100%
1	13022	...	English	Action
2	29250	...	English	Action
3	41127	...	Chinese_Japan_Asian	Action
4	14741	...	English	Action
5	16763	...	English	Action
6	157253	...	Chinese_Japan_Asian	Action
7	132848	...	Chinese_Japan_Asian	Action
8	12439	...	English	Action
9	4294	...	English	Action
10	62803	...	English	Action
11	47005	...	English	Action
12	76088	...	Others	Action
13	66736	...	English	Action
14	157811	...	English	Action
15	9703	...	English	Action
16	121219	...	English	Action
17	28616	...	English	Action
18	14515	...	Spanish_Portuguese	Action
19	7397	...	Italian	Action
20	27217	...	Italian	Action

Before we export the data, we should normalize the numerical columns to get the dummies of the different categories.

filmtv_movies_complete.normalize(
    method = "minmax",
    columns = [
        "votes",
        "duration",
        "notoriety_director",
        "castings_director",
        "notoriety_actors",
        "castings_actors",
    ],
)

Out[17]: 
None  filmtv_id    ...          language_area    Category  
1         13134    ...                Italian      Action  
2         43946    ...     Spanish_Portuguese      Action  
3         57083    ...     Spanish_Portuguese      Action  
4          3760    ...                Italian      Action  
5         58321    ...                 French      Action  
6         17252    ...                Italian      Action  
7          5058    ...                Italian      Action  
8          7345    ...                Italian      Action  
9         24945    ...                English      Action  
10         4783    ...                English      Action  
11         3427    ...                English      Action  
12        37026    ...                English      Action  
13       170419    ...                English      Action  
14       149103    ...    Chinese_Japan_Asian      Action  
15        56496    ...                English      Action  
16        26778    ...                English      Action  
17        34351    ...                English      Action  
18        35451    ...                English      Action  
19        11606    ...                English      Action  
20         5323    ...                English      Action  
...         ...    ...                    ...         ...  
Rows: 1-20 of 50831 | Columns: 4

for elem in ["category", "period", "language_area"]:
    filmtv_movies_complete[elem].get_dummies(drop_first = True)

We can export the results to our Vertica database.

filmtv_movies_complete.to_db(
    name = "filmtv_movies_complete",
    relation_type = "table",
    inplace = True,
)
filmtv_movies_complete.to_db(
    name = "filmtv_movies_mco",
    relation_type = "view",
    db_filter = "votes > 0.02",
)

Machine Learning : Adjusting the Films Rates

Let’s create a model to evaluate an unbiased score for each different movie.

from verticapy.machine_learning.vertica.linear_model import LinearRegression

predictors = filmtv_movies_complete.get_columns(
    exclude_columns = [
        "avg_vote",
        "period",
        "director",
        "language_area",
        "title",
        "year",
        "country",
        "Category",
    ],
)


vp.drop("filmtv_movies_lr") # If model name already exists
Out[21]: True

model = LinearRegression(
    "filmtv_movies_lr",
    max_iter = 1000,
    solver = "BFGS",
)


model.fit("filmtv_movies_mco", predictors, "avg_vote")


=======
details
=======
            predictor            |coefficient|std_err | t_value |p_value 
---------------------------------+-----------+--------+---------+--------
            Intercept            |  5.51752  | 0.06811|81.01187 | 0.00000
            filmtv_id            |  0.00000  | 0.00000| 2.12608 | 0.03352
              votes              |  5.51964  | 0.13995|39.43918 | 0.00000
            duration             | 14.38180  | 2.01767| 7.12792 | 0.00000
       notoriety_director        | -0.00375  | 0.08850|-0.04239 | 0.96619
        castings_director        |  0.19043  | 0.06989| 2.72457 | 0.00645
        notoriety_actors         | -1.12558  | 0.10928|-10.29957| 0.00000
         castings_actors         |  0.06521  | 0.11143| 0.58525 | 0.55840
         category_action         | -0.21055  | 0.04275|-4.92542 | 0.00000
       category_adventure        | -0.28171  | 0.06659|-4.23056 | 0.00002
       category_animation        |  0.71959  | 0.23813| 3.02187 | 0.00252
         category_comedy         | -0.15443  | 0.03590|-4.30178 | 0.00002
         category_drama          |  0.51295  | 0.03578|14.33749 | 0.00000
        category_fantasy         | -0.41598  | 0.04860|-8.55911 | 0.00000
         category_horror         | -0.40973  | 0.04744|-8.63595 | 0.00000
         category_others         |  0.56428  | 0.05551|10.16458 | 0.00000
        category_romantic        |  0.17602  | 0.08426| 2.08914 | 0.03672
           period_90s            |  0.51808  | 0.03137|16.51406 | 0.00000
           period_old            |  1.26392  | 0.02943|42.94980 | 0.00000
 language_area_arabic_middle_est |  0.23921  | 0.13128| 1.82214 | 0.06846
language_area_chinese_japan_asian|  0.58808  | 0.07456| 7.88786 | 0.00000
      language_area_english      | -0.22049  | 0.05702|-3.86668 | 0.00011
      language_area_french       |  0.09483  | 0.06345| 1.49469 | 0.13503
language_area_german_north_europe|  0.24511  | 0.08050| 3.04491 | 0.00233
    language_area_grec_balkan    |  0.61652  | 0.36109| 1.70739 | 0.08778
      language_area_hebrew       |  0.43149  | 0.49329| 0.87472 | 0.38175
      language_area_indian       |  0.11994  | 0.75991| 0.15784 | 0.87459
      language_area_italian      | -0.78387  | 0.05999|-13.06757| 0.00000
      language_area_others       |  0.00000  | 0.91351| 0.00000 | 1.00000
language_area_russian_est_europe |  0.58483  | 0.12646| 4.62450 | 0.00000


==============
regularization
==============
type| lambda 
----+--------
none| 1.00000


===========
call_string
===========
linear_reg('public.filmtv_movies_lr', 'filmtv_movies_mco', '"avg_vote"', '"filmtv_id", "votes", "duration", "notoriety_director", "castings_director", "notoriety_actors", "castings_actors", "Category_Action", "Category_Adventure", "Category_Animation", "Category_Comedy", "Category_Drama", "Category_Fantasy", "Category_Horror", "Category_Others", "Category_Romantic", "period_90s", "period_Old", "language_area_Arabic_Middle_Est", "language_area_Chinese_Japan_Asian", "language_area_English", "language_area_French", "language_area_German_North_Europe", "language_area_Grec_Balkan", "language_area_Hebrew", "language_area_Indian", "language_area_Italian", "language_area_Others", "language_area_Russian_Est_Europe"'
USING PARAMETERS optimizer='bfgs', epsilon=1e-06, max_iterations=1000, regularization='none', lambda=1, alpha=0.5, fit_intercept=true)

===============
Additional Info
===============
       Name       |Value
------------------+-----
 iteration_count  | 28  
rejected_row_count|  0  
accepted_row_count|10307

model.report()

	value
explained_variance	0.464651819821462
max_error	4.90691407293243
median_absolute_error	0.593429116536077
mean_absolute_error	0.715710573929926
mean_squared_error	0.832067099504979
root_mean_squared_error	0.912177120687084
r2	0.464651819812942
r2_adj	0.463141155492087
aic	-1834.5053132193
bic	-1617.64412628404

The model is good. Let’s add it in our vDataFrame.

model.predict(
    filmtv_movies_complete,
    name = "unbiased_vote",
)

	123 filmtv_id Int 100%	...	Abc title Varchar(486) 100%	123 unbiased_vote Float(22) 100%
1	18	...	Diner	6.52335043317847
2	61	...	The Appaloosa	6.46961868237301
3	116	...	Ring of Bright Water	6.4620963698666
4	172	...	The Hostage Tower	6.51076490336608
5	173	...	The Pawnshop	6.36489185260189
6	187	...	Ai vostri ordini, signora...	6.06268700597543
7	188	...	Aida	6.69087133291048
8	196	...	It Nearly Wasn't Christmas	6.45844725319076
9	210	...	Above Suspicion	6.6391005059926
10	218	...	Dallas: The Early Years	6.85725736623595
11	224	...	Raintree County	7.42478326783083
12	226	...	Alcune signore perbene	5.88526099157838
13	233	...	Nightwing	6.39474949339561
14	234	...	The Wizard of Baghdad	6.36906694852514
15	235	...	Alibi	7.70518977148171
16	238	...	Her Alibi	6.78324730504166
17	241	...	Alice Doesn't Live Here Anymore	7.55842968143407
18	245	...	Alien terror	6.24929132432935
19	250	...	Run for Cover	6.65040062858399
20	255	...	All'ultimo sangue	5.9667196793943

Since a score can’t be greater than 10 or less than 0, we need to adjust the unbiased_vote.

filmtv_movies_complete["unbiased_vote"] = fun.case_when(
    filmtv_movies_complete["unbiased_vote"] > 10, 10,
    filmtv_movies_complete["unbiased_vote"] < 0, 0,
    filmtv_movies_complete["unbiased_vote"],
)

Let’s look at the top movies.

filmtv_movies_complete.search(
    usecols = [
        "filmtv_id",
        "title",
        "year",
        "country",
        "avg_vote",
        "unbiased_vote",
        "votes",
        "duration",
        "director",
        "notoriety_director",
        "castings_director",
        "notoriety_actors",
        "castings_actors",
        "period",
        "language_area",
    ],
    order_by = {
        "unbiased_vote" : "desc",
        "avg_vote" : "desc",
    },
).head(10)

	123 filmtv_id Integer 100%	...	Abc period Varchar(6) 100%	Abc language_area Varchar(19) 100%
1	5580	...	Old	English
2	27984	...	90s	German_North_Europe
3	17065	...	Old	English
4	1963	...	Old	English
5	6476	...	Old	English
6	4991	...	Old	English
7	7025	...	Old	English
8	12849	...	90s	English
9	27983	...	Old	German_North_Europe
10	1152	...	Old	English

Great, our results are more consistent. Psycho, Pulp Fiction, and The Godfather are among the top movies.

Machine Learning : Creating Movie Clusters

Since KMeans clustering is sensitive to unnormalized data, let’s normalize our new predictors.

filmtv_movies_complete["unbiased_vote"].normalize(method = "minmax")

	123 filmtv_id Int 100%	...	Abc Varchar(486) 100%	123 unbiased_vote Float 100%
1	30	...		0.29204883188871
2	41	...		0.567574841850368
3	58	...		0.445403875628512
4	91	...		0.403384664509852
5	125	...		0.45102875011258
6	136	...		0.384611253670232
7	180	...		0.379759309324951
8	181	...		0.471966137135568
9	186	...		0.436677749985097
10	193	...		0.415990552650199
11	194	...		0.283165231465175
12	198	...		0.31023899390153
13	202	...		0.505342966160913
14	203	...		0.38045294583808
15	207	...		0.306872195545759
16	208	...		0.523419969586664
17	214	...		0.436874173972948
18	217	...		0.356571540814665
19	223	...		0.424933002280405
20	229	...		0.528059407347737

Let’s compute the elbow() curve to find a suitable number of clusters.

predictors = filmtv_movies_complete.get_columns(
    exclude_columns = [
        "avg_vote",
        "period",
        "director",
        "language_area",
        "title",
        "year",
        "country",
        "Category",
        "filmtv_id",
    ],
)


from verticapy.machine_learning.model_selection import elbow

import verticapy

verticapy.set_option("plotting_lib", "plotly") # to switch plotting graphics to plotly

elbow_chart = elbow(
    filmtv_movies_complete,
    predictors,
    n_cluster = (1, 60),
    show = True
)

elbow_chart

By looking at the elbow curve, we can choose 15 clusters. Let’s create a KMeans model.

from verticapy.machine_learning.vertica.cluster import KMeans

model_kmeans = KMeans(n_cluster = 15)

model_kmeans.fit(filmtv_movies_complete, predictors)


=======
centers
=======
 votes  |duration|notoriety_director|castings_director|notoriety_actors|castings_actors|category_action|category_adventure|category_animation|category_comedy|category_drama|category_fantasy|category_horror|category_others|category_romantic|period_90s|period_old|language_area_arabic_middle_est|language_area_chinese_japan_asian|language_area_english|language_area_french|language_area_german_north_europe|language_area_grec_balkan|language_area_hebrew|language_area_indian|language_area_italian|language_area_others|language_area_russian_est_europe|unbiased_vote
--------+--------+------------------+-----------------+----------------+---------------+---------------+------------------+------------------+---------------+--------------+----------------+---------------+---------------+-----------------+----------+----------+-------------------------------+---------------------------------+---------------------+--------------------+---------------------------------+-------------------------+--------------------+--------------------+---------------------+--------------------+--------------------------------+-------------
 0.02401| 0.01194|      0.03978     |     0.15038     |     0.09444    |    0.16525    |    0.00000    |      0.00000     |      0.00000     |    0.00000    |    1.00000   |     0.00000    |    0.00000    |    0.00000    |     0.00000     |  0.00000 |  0.50763 |            0.00000            |             0.00000             |       1.00000       |       0.00000      |             0.00000             |         0.00000         |       0.00000      |       0.00000      |       0.00000       |       0.00000      |             0.00000            |   0.43309   
 0.00858| 0.01166|      0.01794     |     0.11109     |     0.03326    |    0.10935    |    0.03584    |      0.05341     |      0.00000     |    0.17569    |    0.40759   |     0.03022    |    0.01546    |    0.07379    |     0.02389     |  1.00000 |  0.00000 |            0.03584            |             0.00000             |       0.00000       |       0.52565      |             0.24666             |         0.00984         |       0.00773      |       0.00703      |       0.00000       |       0.00281      |             0.03162            |   0.41116   
 0.01289| 0.01199|      0.01909     |     0.09990     |     0.02979    |    0.08323    |    0.00000    |      0.00000     |      0.00000     |    0.00000    |    1.00000   |     0.00000    |    0.00000    |    0.00000    |     0.00000     |  0.00000 |  0.35719 |            0.05760            |             0.00000             |       0.00000       |       0.41001      |             0.20949             |         0.02053         |       0.01734      |       0.02193      |       0.00000       |       0.00319      |             0.07634            |   0.46971   
 0.01239| 0.01270|      0.02101     |     0.10500     |     0.01925    |    0.05126    |    0.25757    |      0.01101     |      0.02006     |    0.08769    |    0.40936   |     0.03539    |    0.06567    |    0.04090    |     0.01573     |  0.11797 |  0.26268 |            0.00000            |             1.00000             |       0.00000       |       0.00000      |             0.00000             |         0.00000         |       0.00000      |       0.00000      |       0.00000       |       0.00000      |             0.00000            |   0.47079   
 0.02216| 0.01126|      0.02191     |     0.09544     |     0.07413    |    0.14450    |    0.56585    |      0.01392     |      0.00611     |    0.00000    |    0.00000   |     0.03938    |    0.03530    |    0.00000    |     0.06042     |  0.00000 |  0.20536 |            0.00950            |             0.00000             |       0.37237       |       0.41650      |             0.02885             |         0.00373         |       0.00407      |       0.00849      |       0.00000       |       0.00068      |             0.01969            |   0.29080   
 0.01272| 0.01213|      0.02960     |     0.11964     |     0.04669    |    0.08920    |    0.00000    |      0.00000     |      0.00000     |    0.00000    |    0.00000   |     0.00000    |    0.00000    |    1.00000    |     0.00000     |  0.06007 |  0.33815 |            0.01001            |             0.00000             |       0.69855       |       0.12977      |             0.06155             |         0.00260         |       0.00445      |       0.00742      |       0.00000       |       0.00000      |             0.02262            |   0.42967   
 0.01807| 0.01166|      0.02911     |     0.15185     |     0.06133    |    0.15909    |    0.00000    |      0.00000     |      0.00000     |    0.00000    |    1.00000   |     0.00000    |    0.00000    |    0.00000    |     0.00000     |  0.13502 |  0.46180 |            0.00000            |             0.00000             |       0.00000       |       0.00000      |             0.00000             |         0.00000         |       0.00000      |       0.00000      |       1.00000       |       0.00000      |             0.00000            |   0.33768   
 0.02620| 0.01079|      0.03110     |     0.08541     |     0.11019    |    0.14938    |    0.00000    |      0.00000     |      0.00000     |    1.00000    |    0.00000   |     0.00000    |    0.00000    |    0.00000    |     0.00000     |  0.32602 |  0.00000 |            0.00000            |             0.00000             |       1.00000       |       0.00000      |             0.00000             |         0.00000         |       0.00000      |       0.00000      |       0.00000       |       0.00000      |             0.00000            |   0.23033   
 0.01905| 0.01062|      0.04448     |     0.19751     |     0.07223    |    0.18512    |    0.26673    |      0.00000     |      0.00105     |    0.41475    |    0.00000   |     0.08200    |    0.00000    |    0.00000    |     0.05057     |  0.00000 |  1.00000 |            0.00035            |             0.00000             |       0.98068       |       0.00000      |             0.00000             |         0.00035         |       0.00035      |       0.00053      |       0.00000       |       0.00000      |             0.00439            |   0.42304   
 0.01284| 0.01123|      0.02515     |     0.17570     |     0.05849    |    0.16641    |    0.00000    |      1.00000     |      0.00000     |    0.00000    |    0.00000   |     0.00000    |    0.00000    |    0.00000    |     0.00000     |  0.00000 |  0.80716 |            0.00635            |             0.01386             |       0.61663       |       0.06755      |             0.05196             |         0.00000         |       0.00000      |       0.00173      |       0.16917       |       0.00000      |             0.00866            |   0.35211   
 0.01071| 0.01076|      0.01753     |     0.10238     |     0.03244    |    0.11043    |    0.00000    |      0.00000     |      0.00000     |    0.81455    |    0.00000   |     0.01130    |    0.00000    |    0.00000    |     0.08654     |  0.02649 |  0.32427 |            0.01307            |             0.00000             |       0.00000       |       0.42035      |             0.38114             |         0.00954         |       0.00530      |       0.01519      |       0.00000       |       0.00106      |             0.03568            |   0.34972   
 0.02610| 0.00974|      0.02624     |     0.08535     |     0.03899    |    0.08871    |    0.00000    |      0.00000     |      0.00000     |    0.00000    |    0.00000   |     0.00000    |    1.00000    |    0.00000    |     0.00000     |  0.09292 |  0.33923 |            0.00295            |             0.00000             |       0.86382       |       0.02852      |             0.02606             |         0.00049         |       0.00147      |       0.00049      |       0.00000       |       0.00000      |             0.00590            |   0.25918   
 0.02558| 0.01076|      0.05363     |     0.23229     |     0.09521    |    0.22780    |    0.11786    |      0.00000     |      0.00120     |    0.73644    |    0.00000   |     0.01608    |    0.03409    |    0.00000    |     0.02472     |  0.07945 |  0.69731 |            0.00000            |             0.00000             |       0.00000       |       0.00000      |             0.00000             |         0.00000         |       0.00000      |       0.00000      |       1.00000       |       0.00000      |             0.00000            |   0.26919   
 0.02767| 0.01131|      0.03195     |     0.11679     |     0.09455    |    0.15817    |    0.09468    |      0.04117     |      0.01346     |    0.00000    |    0.25222   |     0.12967    |    0.00000    |    0.00000    |     0.05272     |  0.63949 |  0.00000 |            0.00000            |             0.00000             |       1.00000       |       0.00000      |             0.00000             |         0.00000         |       0.00000      |       0.00000      |       0.00000       |       0.00000      |             0.00000            |   0.30163   
 0.00767| 0.01014|      0.02353     |     0.14004     |     0.03620    |    0.10252    |    0.03605    |      0.02300     |      0.00186     |    0.00000    |    0.00000   |     0.03170    |    0.06339    |    0.65817    |     0.04164     |  0.14419 |  0.23928 |            0.00000            |             0.00000             |       0.00000       |       0.00000      |             0.00000             |         0.00000         |       0.00000      |       0.00000      |       1.00000       |       0.00000      |             0.00000            |   0.25045   


=======
metrics
=======
Evaluation metrics:
     Total Sum of Squares: 90981.961
     Within-Cluster Sum of Squares: 
         Cluster 0: 1804.1096
         Cluster 1: 1683.224
         Cluster 2: 4389.6354
         Cluster 3: 2641.5329
         Cluster 4: 3456.3598
         Cluster 5: 2094.1254
         Cluster 6: 1137.5742
         Cluster 7: 1023.545
         Cluster 8: 3776.4604
         Cluster 9: 1292.4825
         Cluster 10: 3107.2247
         Cluster 11: 1094.8659
         Cluster 12: 3243.3793
         Cluster 13: 4990.8449
         Cluster 14: 1270.5356
     Total Within-Cluster Sum of Squares: 37005.9
     Between-Cluster Sum of Squares: 53976.062
     Between-Cluster SS / Total SS: 59.33%
 Number of iterations performed: 13
 Converged: True
 Call:
kmeans('"public"."_verticapy_tmp_kmeans_v_mldb_28c2847697b411efa8720242ac120002_"', '"public"."_verticapy_tmp_view_v_mldb_290e88ee97b411efa8720242ac120002_"', '"votes", "duration", "notoriety_director", "castings_director", "notoriety_actors", "castings_actors", "Category_Action", "Category_Adventure", "Category_Animation", "Category_Comedy", "Category_Drama", "Category_Fantasy", "Category_Horror", "Category_Others", "Category_Romantic", "period_90s", "period_Old", "language_area_Arabic_Middle_Est", "language_area_Chinese_Japan_Asian", "language_area_English", "language_area_French", "language_area_German_North_Europe", "language_area_Grec_Balkan", "language_area_Hebrew", "language_area_Indian", "language_area_Italian", "language_area_Others", "language_area_Russian_Est_Europe", "unbiased_vote"', 15
USING PARAMETERS max_iterations=300, epsilon=0.0001, init_method='kmeanspp', distance_method='euclidean')

model_kmeans.clusters_
Out[33]: 
array([[2.40097190e-02, 1.19415685e-02, 3.97783770e-02, 1.50382525e-01,
        9.44354082e-02, 1.65254559e-01, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 1.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        5.07630079e-01, 0.00000000e+00, 0.00000000e+00, 1.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        4.33085941e-01],
       [8.57562347e-03, 1.16567782e-02, 1.79420537e-02, 1.11091591e-01,
        3.32620188e-02, 1.09354113e-01, 3.58397751e-02, 5.34082923e-02,
        0.00000000e+00, 1.75685172e-01, 4.07589599e-01, 3.02178496e-02,
        1.54602952e-02, 7.37877723e-02, 2.38931834e-02, 1.00000000e+00,
        0.00000000e+00, 3.58397751e-02, 0.00000000e+00, 0.00000000e+00,
        5.25650035e-01, 2.46661982e-01, 9.83836964e-03, 7.73014758e-03,
        7.02740689e-03, 0.00000000e+00, 2.81096275e-03, 3.16233310e-02,
        4.11156837e-01],
       [1.28913047e-02, 1.19905299e-02, 1.90860433e-02, 9.98964631e-02,
        2.97902715e-02, 8.32320486e-02, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 1.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        3.57185569e-01, 5.76041459e-02, 0.00000000e+00, 0.00000000e+00,
        4.10005980e-01, 2.09487742e-01, 2.05301973e-02, 1.73410405e-02,
        2.19254535e-02, 0.00000000e+00, 3.18915687e-03, 7.63404425e-02,
        4.69714107e-01],
       [1.23909703e-02, 1.26960553e-02, 2.10070326e-02, 1.04999563e-01,
        1.92463846e-02, 5.12589476e-02, 2.57569799e-01, 1.10106174e-02,
        2.00550531e-02, 8.76917027e-02, 4.09359025e-01, 3.53912702e-02,
        6.56704680e-02, 4.08965788e-02, 1.57294534e-02, 1.17970901e-01,
        2.62681872e-01, 0.00000000e+00, 1.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        4.70789177e-01],
       [2.21597268e-02, 1.12584407e-02, 2.19139779e-02, 9.54401448e-02,
        7.41257382e-02, 1.44497573e-01, 5.65852003e-01, 1.39171758e-02,
        6.10997963e-03, 0.00000000e+00, 0.00000000e+00, 3.93754243e-02,
        3.53021045e-02, 0.00000000e+00, 6.04209097e-02, 0.00000000e+00,
        2.05363204e-01, 9.50441276e-03, 0.00000000e+00, 3.72369314e-01,
        4.16496945e-01, 2.88526816e-02, 3.73387644e-03, 4.07331976e-03,
        8.48608282e-03, 0.00000000e+00, 6.78886626e-04, 1.96877122e-02,
        2.90803425e-01],
       [1.27204766e-02, 1.21343477e-02, 2.96001270e-02, 1.19644255e-01,
        4.66885433e-02, 8.92031246e-02, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 1.00000000e+00, 0.00000000e+00, 6.00667408e-02,
        3.38153504e-01, 1.00111235e-02, 0.00000000e+00, 6.98553949e-01,
        1.29773823e-01, 6.15498702e-02, 2.59547646e-03, 4.44938821e-03,
        7.41564702e-03, 0.00000000e+00, 0.00000000e+00, 2.26177234e-02,
        4.29669456e-01],
       [1.80666501e-02, 1.16551518e-02, 2.91064822e-02, 1.51853253e-01,
        6.13294181e-02, 1.59094787e-01, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 1.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.35022693e-01,
        4.61800303e-01, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 1.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        3.37678646e-01],
       [2.62007978e-02, 1.07855817e-02, 3.10964618e-02, 8.54097947e-02,
        1.10186296e-01, 1.49376517e-01, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 1.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 3.26023001e-01,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        2.30328328e-01],
       [1.90504336e-02, 1.06163167e-02, 4.44819029e-02, 1.97512438e-01,
        7.22325313e-02, 1.85121496e-01, 2.66725198e-01, 0.00000000e+00,
        1.05355575e-03, 4.14749781e-01, 0.00000000e+00, 8.20017559e-02,
        0.00000000e+00, 0.00000000e+00, 5.05706760e-02, 0.00000000e+00,
        1.00000000e+00, 3.51185250e-04, 0.00000000e+00, 9.80684811e-01,
        0.00000000e+00, 0.00000000e+00, 3.51185250e-04, 3.51185250e-04,
        5.26777875e-04, 0.00000000e+00, 0.00000000e+00, 4.38981563e-03,
        4.23044105e-01],
       [1.28448835e-02, 1.12284699e-02, 2.51507959e-02, 1.75696048e-01,
        5.84885263e-02, 1.66409818e-01, 0.00000000e+00, 1.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        8.07159353e-01, 6.35103926e-03, 1.38568129e-02, 6.16628176e-01,
        6.75519630e-02, 5.19630485e-02, 0.00000000e+00, 0.00000000e+00,
        1.73210162e-03, 1.69168591e-01, 0.00000000e+00, 8.66050808e-03,
        3.52106400e-01],
       [1.07132597e-02, 1.07617814e-02, 1.75266164e-02, 1.02383335e-01,
        3.24447787e-02, 1.10430173e-01, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 8.14553161e-01, 0.00000000e+00, 1.13034264e-02,
        0.00000000e+00, 0.00000000e+00, 8.65418580e-02, 2.64924055e-02,
        3.24267043e-01, 1.30695867e-02, 0.00000000e+00, 0.00000000e+00,
        4.20346167e-01, 3.81137407e-01, 9.53726598e-03, 5.29848110e-03,
        1.51889792e-02, 0.00000000e+00, 1.05969622e-03, 3.56764394e-02,
        3.49718470e-01],
       [2.61017252e-02, 9.73760949e-03, 2.62411619e-02, 8.53545286e-02,
        3.89904358e-02, 8.87136721e-02, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        1.00000000e+00, 0.00000000e+00, 0.00000000e+00, 9.29203540e-02,
        3.39233038e-01, 2.94985251e-03, 0.00000000e+00, 8.63815143e-01,
        2.85152409e-02, 2.60570305e-02, 4.91642085e-04, 1.47492625e-03,
        4.91642085e-04, 0.00000000e+00, 0.00000000e+00, 5.89970501e-03,
        2.59177459e-01],
       [2.55755909e-02, 1.07599964e-02, 5.36283448e-02, 2.32287166e-01,
        9.52092857e-02, 2.27804118e-01, 1.17858857e-01, 0.00000000e+00,
        1.20019203e-03, 7.36437830e-01, 0.00000000e+00, 1.60825732e-02,
        3.40854537e-02, 0.00000000e+00, 2.47239558e-02, 7.94527124e-02,
        6.97311570e-01, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 1.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        2.69189068e-01],
       [2.76742140e-02, 1.13052879e-02, 3.19522069e-02, 1.16793329e-01,
        9.45470724e-02, 1.58170323e-01, 9.46801773e-02, 4.11652945e-02,
        1.34578847e-02, 0.00000000e+00, 2.52216593e-01, 1.29670678e-01,
        0.00000000e+00, 0.00000000e+00, 5.27232426e-02, 6.39487017e-01,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        3.01628264e-01],
       [7.67285168e-03, 1.01415464e-02, 2.35289366e-02, 1.40036945e-01,
        3.61956948e-02, 1.02518961e-01, 3.60472343e-02, 2.29956495e-02,
        1.86451212e-03, 0.00000000e+00, 0.00000000e+00, 3.16967060e-02,
        6.33934121e-02, 6.58172778e-01, 4.16407707e-02, 1.44188937e-01,
        2.39279055e-01, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 1.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        2.50446768e-01]])

Let’s add the clusters in the vDataFrame.

model_kmeans.predict(
    filmtv_movies_complete,
    name = "movies_cluster",
)

	123 filmtv_id Int 100%	...	123 unbiased_vote Float 100%	123 movies_cluster Integer 100%
1	18	...	0.39884855000278	8
2	61	...	0.389557731561989	8
3	116	...	0.388257039909394	9
4	172	...	0.39667237165915	8
5	173	...	0.371449295735919	8
6	187	...	0.31919471032144	12
7	188	...	0.427814780231961	14
8	196	...	0.387626066974468	8
9	210	...	0.418863027381675	8
10	218	...	0.45658477941377	8
11	224	...	0.554716331673336	0
12	226	...	0.288515775395156	6
13	233	...	0.376612014529303	11
14	234	...	0.372171216343648	8
15	235	...	0.603201740690985	2
16	238	...	0.443787615147943	8
17	241	...	0.57782528579147	0
18	245	...	0.351460676274621	8
19	250	...	0.420816944493652	8
20	255	...	0.302600890203025	12

Let’s look at the different clusters.

filmtv_movies_complete.search(
    filmtv_movies_complete["movies_cluster"] == 0,
    usecols=[
        "avg_vote",
        "period",
        "director",
        "language_area",
        "title",
        "year",
        "country",
        "Category",
    ]
)

	123 avg_vote Numeric(8) 100%	...	Abc period Varchar(6) 100%	Abc Category Varchar(9) 100%
1	6.3	...	Old	Drama
2	7.8	...	Old	Drama
3	7.5	...	Old	Drama
4	7.4	...	Old	Drama
5	6.0	...	Old	Drama
6	4.0	...	Old	Drama
7	5.1	...	Old	Drama
8	8.0	...	Old	Drama
9	7.1	...	Old	Drama
10	6.0	...	Old	Drama
11	7.9	...	Old	Drama
12	7.0	...	Old	Drama
13	5.3	...	Old	Drama
14	7.5	...	Old	Drama
15	6.9	...	Old	Drama
16	6.9	...	Old	Drama
17	3.5	...	Old	Drama
18	7.0	...	Old	Drama
19	5.8	...	Old	Drama
20	7.6	...	Old	Drama

filmtv_movies_complete.search(
    filmtv_movies_complete["movies_cluster"] == 1,
    usecols=[
        "avg_vote",
        "period",
        "director",
        "language_area",
        "title",
        "year",
        "country",
        "Category",
    ],
)

	123 avg_vote Numeric(8) 100%	...	Abc period Varchar(6) 100%	Abc Category Varchar(9) 100%
1	8.0	...	90s	Drama
2	6.5	...	90s	Drama
3	5.7	...	90s	Adventure
4	4.5	...	90s	Comedy
5	5.9	...	90s	Comedy
6	6.0	...	90s	Drama
7	4.8	...	90s	Comedy
8	8.3	...	90s	Comedy
9	6.3	...	90s	Drama
10	6.0	...	90s	Thriller
11	7.8	...	90s	Drama
12	8.0	...	90s	Drama
13	7.1	...	90s	Drama
14	8.4	...	90s	Drama
15	5.4	...	90s	Drama
16	8.0	...	90s	Drama
17	4.0	...	90s	Comedy
18	6.0	...	90s	Fantasy
19	8.0	...	90s	Adventure
20	5.8	...	90s	Drama

filmtv_movies_complete.search(
    filmtv_movies_complete["movies_cluster"] == 2,
    usecols=[
        "avg_vote",
        "period",
        "director",
        "language_area",
        "title",
        "year",
        "country",
        "Category",
    ],
)

	123 avg_vote Numeric(8) 100%	...	Abc period Varchar(6) 100%	Abc Category Varchar(9) 100%
1	4.0	...	Old	Drama
2	4.0	...	Old	Drama
3	8.1	...	Old	Drama
4	4.8	...	Old	Drama
5	7.1	...	Old	Drama
6	9.0	...	Old	Drama
7	3.5	...	Old	Drama
8	7.7	...	Old	Drama
9	7.1	...	Old	Drama
10	6.2	...	Old	Drama
11	7.7	...	Old	Drama
12	6.0	...	Old	Drama
13	6.9	...	Old	Drama
14	6.0	...	Old	Drama
15	5.2	...	Old	Drama
16	5.8	...	Old	Drama
17	6.0	...	Old	Drama
18	7.1	...	Old	Drama
19	8.0	...	Old	Drama
20	6.0	...	Old	Drama

filmtv_movies_complete.search(
    filmtv_movies_complete["movies_cluster"] == 3,
    usecols=[
        "avg_vote",
        "period",
        "director",
        "language_area",
        "title",
        "year",
        "country",
        "Category",
    ],
)

	123 avg_vote Numeric(8) 100%	...	Abc period Varchar(6) 100%	Abc Category Varchar(9) 100%
1	7.0	...	90s	Action
2	7.4	...	Old	Drama
3	4.8	...	Old	Action
4	6.5	...	90s	Action
5	7.8	...	Old	Fantasy
6	3.0	...	Old	Action
7	7.9	...	90s	Action
8	7.4	...	90s	Drama
9	4.8	...	Old	Action
10	6.0	...	90s	Action
11	7.7	...	90s	Drama
12	7.6	...	Old	Others
13	6.9	...	90s	Drama
14	4.0	...	90s	Adventure
15	7.4	...	Old	Action
16	4.7	...	90s	Action
17	8.0	...	90s	Action
18	6.0	...	Old	Comedy
19	8.3	...	Old	Drama
20	7.4	...	90s	Drama

Each cluster consists of similar movies. These clusters can be used to give movie recommendations or help streaming platforms group movies together.

Conclusion

We’ve solved our problem in a Pandas-like way, all without ever loading data into memory!