Feedback

×

• Feedback
• Was this page helpful?*
  • Yes
  • No
• Do you have any additional feedback?
  • Yes
  • No
• Additional Feedback

verticapy.machine_learning.vertica.feature_extraction.text.TfidfVectorizer

class verticapy.machine_learning.vertica.feature_extraction.text.TfidfVectorizer(name: str | None = None, overwrite_model: bool = False, lowercase: bool = True, vocabulary: Annotated[list | ndarray, 'Array Like Structure'] | None = None, max_df: Annotated[int | float | Decimal, 'Python Numbers'] | None = None, min_df: Annotated[int | float | Decimal, 'Python Numbers'] | None = None, norm: Literal['l1', 'l2', None] = 'l2', smooth_idf: bool = True, compute_vocabulary: bool = True)

[Beta Version] Create tfidf representation of documents.

The formula that is used to compute the tf-idf for a term t of a document d in a document set is

\[tf-idf(t, d) = tf(t, d) * idf(t),\]

and if smooth_idf = False, the idf is computed as

\[idf(t) = log [ n / df(t) ] + 1,\]

where n is the total number of documents in the document set and df(t) is the document frequency of t; the document frequency is the number of documents in the document set that contain the term t. The effect of adding “1” to the idf in the equation above is that terms with zero idf, i.e., terms that occur in all documents in a training set, will not be entirely ignored.

If smooth_idf=True (the default), the constant “1” is added to the numerator and denominator of the idf as if an extra document was seen containing every term in the collection exactly once, which prevents zero divisions:

\[idf(t) = log [ (1 + n) / (1 + df(t)) ] + 1.\]

Parameters

name: str, optional

Name of the model.

overwrite_model: bool, optional

If set to True, training a model with the same name as an existing model overwrites the existing model.

lowercase: bool, optional

Converts all the elements to lowercase before processing.

vocabulary: list, optional

A list of string elements to be regarded as the primary vocabulary.

max_df: PythonNumber, optional

While constructing the vocabulary, exclude terms with a document frequency surpassing the specified threshold, essentially treating them as corpus-specific stop words. If the value is a float within the range [0.0, 1.0], it denotes a proportion of documents; if an integer, it signifies absolute counts. Note that this parameter is disregarded if a custom vocabulary is provided.

min_df: PythonNumber, optional

When constructing the vocabulary, omit terms with a document frequency below the specified threshold, often referred to as the cut-off in literature. If the value is a float within the range [0.0, 1.0], it denotes a proportion of documents; if an integer, it signifies absolute counts. It’s important to note that this parameter is disregarded if a custom vocabulary is provided.

norm: str, optional

The tfidf values of each document will have unit norm, either:

• l2:

  Sum of squares of vector elements is 1.

• l1:

  Sum of absolute values of vector elements is 1.

• None:

  No normalization.

smooth_idf: bool, optional

Smooth idf weights by adding one to document frequencies, as if an extra document was seen containing every term in the collection exactly once. Prevents zero divisions.

compute_vocabulary: bool, optional

If set to true, the vocabulary is computed, making the operation more resource-intensive.

Attributes

Many attributes are created during the fitting phase.

vocabulary_: ArrayLike

The ultimate vocabulary. If empty, it implies that all words are utilized, and the user opted not to compute a specific vocabulary.

fixed_vocabulary_: bool

Boolean indicating whether a vocabulary was supplied by the user.

idf_: vDataFrame

The IDF table which is computed based on the relation used for the fitting process.

tf_: vDataFrame

The TF table which is computed based on the relation used for the fitting process.

stop_words_: ArrayLike

Terms are excluded under the following conditions:

• They appear in an excessive number of documents

(controlled by max_df).

• They appear in an insufficient number of documents

(controlled by min_df).

This functionality is only applicable when no specific vocabulary is provided and compute_vocabulary is set to True.

n_document_: int

Total number of document. This functionality is only applicable when no specific vocabulary is provided and compute_vocabulary is set to True.

Note

All attributes can be accessed using the get_attributes() method.

Examples

We import verticapy:

import verticapy as vp

Hint

By assigning an alias to verticapy, we mitigate the risk of code collisions with other libraries. This precaution is necessary because verticapy uses commonly known function names like “average” and “median”, which can potentially lead to naming conflicts. The use of an alias ensures that the functions from verticapy are used as intended without interfering with functions from other libraries.

For this example, let’s generate some text.

documents = [
    "Natural language processing is a field of study in artificial intelligence.",
    "TF-IDF stands for Term Frequency-Inverse Document Frequency.",
    "Machine learning algorithms can be applied to text data for classification.",
    "The 20 Newsgroups dataset is a collection of text documents used for text classification.",
    "Clustering is a technique used to group similar documents together.",
    "Python is a popular programming language for natural language processing tasks.",
    "TF-IDF is a technique widely used in information retrieval.",
    "An algorithm is a set of instructions designed to perform a specific task.",
    "Data preprocessing is an important step in preparing data for machine learning.",
]

Next, we can insert this text into a vDataFrame:

data = vp.vDataFrame(
    {
        "id": (list(range(1,len(documents)+1))),
        "values": documents,
    }
)

Then we can initialize the object and fit the model, to learn the idf weigths.

from verticapy.machine_learning.vertica.feature_extraction.text import TfidfVectorizer

model = TfidfVectorizer(name = "test_idf")
model.fit(
    input_relation = data,
    index = "id",
    x = "values",
)

We apply the transform function to obtain the idf representation.

model.transform(
    vdf = data,
    index = "id",
    x = "values",
)

	123 row_id Integer	Abc word Varchar(18320)	123 tfidf Float(22)
1	2	term	0.40660486945441
2	2	document	0.40660486945441
3	2	for	0.235418153692232
4	2	frequency	0.40660486945441
5	2	stands	0.40660486945441
6	2	tfidf	0.34342494608277
7	2	frequencyinverse	0.40660486945441
8	5	to	0.277657271947304
9	5	used	0.277657271947304
10	5	together	0.378089503868613
11	5	similar	0.378089503868613
12	5	documents	0.319340414330917
13	5	technique	0.319340414330917
14	5	is	0.177225040025995
15	5	group	0.378089503868613
16	5	clustering	0.378089503868613
17	5	a	0.196572815172405
18	8	to	0.231156487059915
19	8	task	0.314768782735436
20	8	set	0.314768782735436
21	8	of	0.231156487059915
22	8	specific	0.314768782735436
23	8	an	0.265858725166046
24	8	is	0.147544191384394
25	8	instructions	0.314768782735436
26	8	perform	0.314768782735436
27	8	designed	0.314768782735436
28	8	algorithm	0.314768782735436
29	8	a	0.327303377203496
30	3	to	0.252759296711247
31	3	be	0.344185608471555
32	3	for	0.199278332752225
33	3	data	0.290704644512533
34	3	text	0.290704644512533
35	3	machine	0.290704644512533
36	3	learning	0.290704644512533
37	3	classification	0.290704644512533
38	3	can	0.344185608471555
39	3	algorithms	0.344185608471555
40	3	applied	0.344185608471555
41	6	tasks	0.331396587157975
42	6	python	0.331396587157975
43	6	processing	0.279902833503824
44	6	popular	0.331396587157975
45	6	natural	0.279902833503824
46	6	language	0.559805667007647
47	6	for	0.191873680197981
48	6	programming	0.331396587157975
49	6	is	0.15533828054629
50	6	a	0.172296663646098
51	7	widely	0.408405860386832
52	7	used	0.299920669264876
53	7	tfidf	0.34494609169692
54	7	retrieval	0.408405860386832
55	7	information	0.408405860386832
56	7	technique	0.34494609169692
57	7	is	0.19143547814292
58	7	in	0.299920669264876
59	7	a	0.212334616242205
60	1	processing	0.304205711077076
61	1	of	0.264498084357478
62	1	natural	0.304205711077076
63	1	language	0.304205711077076
64	1	field	0.360170467668849
65	1	study	0.360170467668849
66	1	is	0.168825701046106
67	1	in	0.264498084357478
68	1	intelligence	0.360170467668849
69	1	artificial	0.360170467668849
70	1	a	0.18725651478606
71	4	of	0.219590693010443
72	4	20	0.299019491159736
73	4	used	0.219590693010443
74	4	newsgroups	0.299019491159736
75	4	for	0.173127824615757
76	4	text	0.5051132455301
77	4	documents	0.25255662276505
78	4	dataset	0.299019491159736
79	4	is	0.14016189486115
80	4	the	0.299019491159736
81	4	classification	0.25255662276505
82	4	collection	0.299019491159736
83	4	a	0.155463461871492
84	9	preprocessing	0.314847512398901
85	9	preparing	0.314847512398901
86	9	for	0.182292012791185
87	9	an	0.265925221493223
88	9	step	0.314847512398901
89	9	machine	0.265925221493223
90	9	learning	0.265925221493223
91	9	is	0.147581094994826
92	9	in	0.231214303696863
93	9	data	0.531850442986445
94	9	important	0.314847512398901

Rows: 1-94 | Columns: 3

Notice how we can get the idf weight/score of each word in each row. We can also get the results in a more convient form by switching the pivot parameter to True. But for large datasets this is not ideal.

Advanced Analysis

In the above result, we can observe some less informative words such as “is” and “a”, which may not provide meaningful insights.

To address this issue, we can make use of the max_df parameter to exclude words that occur too frequently and might be irrelevant. Similarly, we can leverage the min_df parameter to eliminate words with low frequency that may not contribute significantly.

Let’s apply these parameters to remove common words like “is” and “a.”

model = TfidfVectorizer(max_df = 4, min_df = 1,)
model.fit(
    input_relation = data,
    index = "id",
    x = "values",
)
model.transform(
    vdf = data,
    index = "id",
    x = "values",
)

	123 row_id Integer	Abc word Varchar(18320)	123 tfidf Float(22)
1	2	tfidf	0.353356315856649
2	2	stands	0.418363314367839
3	2	frequencyinverse	0.418363314367839
4	2	term	0.418363314367839
5	2	document	0.418363314367839
6	2	frequency	0.418363314367839
7	5	group	0.392071004311936
8	5	similar	0.392071004311936
9	5	together	0.392071004311936
10	5	clustering	0.392071004311936
11	5	technique	0.331149412197439
12	5	documents	0.331149412197439
13	5	to	0.287924854705095
14	5	used	0.287924854705095
15	8	set	0.337253796525324
16	8	algorithm	0.337253796525324
17	8	perform	0.337253796525324
18	8	specific	0.337253796525324
19	8	designed	0.337253796525324
20	8	instructions	0.337253796525324
21	8	task	0.337253796525324
22	8	an	0.284849925785026
23	8	to	0.247668787784263
24	8	of	0.247668787784263
25	3	be	0.351230258342344
26	3	algorithms	0.351230258342344
27	3	applied	0.351230258342344
28	3	can	0.351230258342344
29	3	data	0.296654667947555
30	3	learning	0.296654667947555
31	3	text	0.296654667947555
32	3	machine	0.296654667947555
33	3	classification	0.296654667947555
34	3	to	0.25793267033028
35	6	python	0.347518644513298
36	6	programming	0.347518644513298
37	6	popular	0.347518644513298
38	6	tasks	0.347518644513298
39	6	language	0.587039559633795
40	6	processing	0.293519779816897
41	6	natural	0.293519779816897
42	7	retrieval	0.426194265971802
43	7	widely	0.426194265971802
44	7	information	0.426194265971802
45	7	tfidf	0.359970462253805
46	7	technique	0.359970462253805
47	7	in	0.312983925759643
48	7	used	0.312983925759643
49	1	field	0.372194346950245
50	1	study	0.372194346950245
51	1	intelligence	0.372194346950245
52	1	artificial	0.372194346950245
53	1	language	0.314361270943981
54	1	processing	0.314361270943981
55	1	natural	0.314361270943981
56	1	of	0.273328050503949
57	1	in	0.273328050503949
58	4	20	0.310702091311095
59	4	the	0.310702091311095
60	4	dataset	0.310702091311095
61	4	collection	0.310702091311095
62	4	newsgroups	0.310702091311095
63	4	text	0.5248478656908
64	4	classification	0.2624239328454
65	4	documents	0.2624239328454
66	4	of	0.228170034288334
67	4	used	0.228170034288334
68	9	important	0.323881981810178
69	9	step	0.323881981810178
70	9	preprocessing	0.323881981810178
71	9	preparing	0.323881981810178
72	9	data	0.547111756382014
73	9	learning	0.273555878191007
74	9	machine	0.273555878191007
75	9	an	0.273555878191007
76	9	in	0.237848939423482

Rows: 1-76 | Columns: 3

Notice how we have removed the unnecessary words.

We can also see which words were omitted using the stop_words_ attribute:

model.stop_words_
Out[4]: array(['a', 'for', 'is'], dtype='<U3')

See also

vDataColumn.pivot() : pivot vDataFrame.

__init__(name: str | None = None, overwrite_model: bool = False, lowercase: bool = True, vocabulary: Annotated[list | ndarray, 'Array Like Structure'] | None = None, max_df: Annotated[int | float | Decimal, 'Python Numbers'] | None = None, min_df: Annotated[int | float | Decimal, 'Python Numbers'] | None = None, norm: Literal['l1', 'l2', None] = 'l2', smooth_idf: bool = True, compute_vocabulary: bool = True) → None

Must be overridden in the child class

Methods

__init__([name, overwrite_model, lowercase, ...])	Must be overridden in the child class
contour([nbins, chart])	Draws the model's contour plot.
deploySQL([X])	Returns the SQL code needed to deploy the model.
does_model_exists(name[, raise_error, ...])	Checks whether the model is stored in the Vertica database.
drop()	Drops the model from the Vertica database.
export_models(name, path[, kind])	Exports machine learning models.
fit(input_relation, index, x[, return_report])	Applies basic pre-processing.
get_attributes([attr_name])	Returns the model attributes.
get_match_index(x, col_list[, str_check])	Returns the matching index.
get_params()	Returns the parameters of the model.
get_plotting_lib([class_name, chart, ...])	Returns the first available library (Plotly, Matplotlib, or Highcharts) to draw a specific graphic.
get_vertica_attributes([attr_name])	Returns the model Vertica attributes.
import_models(path[, schema, kind])	Imports machine learning models.
register(registered_name[, raise_error])	Registers the model and adds it to in-DB Model versioning environment with a status of 'under_review'.
set_params([parameters])	Sets the parameters of the model.
summarize()	Summarizes the model.
to_binary(path)	Exports the model to the Vertica Binary format.
to_pmml(path)	Exports the model to PMML.
to_python([return_proba, ...])	Returns the Python function needed for in-memory scoring without using built-in Vertica functions.
to_sql([X, return_proba, ...])	Returns the SQL code needed to deploy the model without using built-in Vertica functions.
to_tf(path)	Exports the model to the Frozen Graph format (TensorFlow).
transform(vdf, index, x[, pivot])	Transforms input data to tf-idf representation.

Attributes

object_type