Loading...

Getting started with Query Profiler

This starter notebook will help you get up and running with the Query Profiler (QueryProfiler) tool in VerticaPyLab and demonstrate functionality through various examples and use cases.

This tool is a work in progress and the VerticaPy team is continuously adding new features.

See also QueryProfiler , QueryProfilerInterface , QueryProfilerComparison .

VerticaPyLab

The easiest way to use the QueryProfiler tool is through VerticaPyLab. For installation instructions, see Getting Started .

Before using the QueryProfiler tool, confirm that you are connected to a Vertica database. If not, follow the connection instructions in a Jupyter notebook or connect using the Connect option on the VerticaPyLab homepage.

QueryProfiler

The QueryProfiler object is a python object that includes many built-in methods for analyzing queries and their performance. There are a few different ways to create a QueryProfiler object.

Create and save a QueryProfiler object

First, import the verticapy package and load the datasets:

import verticapy as vp

from verticapy.datasets import load_titanic, load_amazon

# load datasets
titanic = load_titanic()

amazon = load_amazon()

Create QueryProfiler object from transaction id and statement id

Before creating the QueryProfiler object, take a look at the data used by the query:

amazon.head(100)
📅
date
Date
100%
...
Abc
state
Varchar(32)
100%
123
number
Int
100%
11998-01-01...AMAPÁ0
21998-01-01...AMAZONAS0
31998-01-01...DISTRITO FEDERAL0
41998-01-01...ESPÍRITO SANTO0
51998-01-01...MARANHÃO0
61998-01-01...PARANÁ0
71998-01-01...PIAUÍ0
81998-01-01...RORAIMA0
91998-01-01...SERGIPE0
101998-01-01...SÃO PAULO0
111998-02-01...GOIÁS0
121998-02-01...MATO GROSSO DO SUL0
131998-02-01...MINAS GERAIS0
141998-02-01...PARAÍBA0
151998-02-01...SANTA CATARINA0
161998-02-01...SÃO PAULO0
171998-03-01...AMAPÁ0
181998-03-01...BAHIA0
191998-03-01...MATO GROSSO DO SUL0
201998-03-01...PARÁ0
211998-03-01...PERNAMBUCO0
221998-03-01...RIO GRANDE DO SUL0
231998-04-01...CEARÁ0
241998-04-01...PARANÁ0
251998-04-01...PARAÍBA0
261998-04-01...PARÁ0
271998-05-01...ALAGOAS0
281998-05-01...AMAPÁ0
291998-05-01...MARANHÃO0
301998-05-01...MATO GROSSO DO SUL0
311998-05-01...RIO GRANDE DO SUL0
321998-05-01...TOCANTINS0
331998-06-01...ESPÍRITO SANTO6
341998-06-01...RIO DE JANEIRO3
351998-06-01...RIO GRANDE DO NORTE1
361998-06-01...SÃO PAULO451
371998-07-01...ESPÍRITO SANTO37
381998-07-01...MARANHÃO274
391998-07-01...MATO GROSSO360
401998-07-01...MATO GROSSO DO SUL3712
411998-07-01...PARAÍBA0
421998-07-01...PARÁ638
431998-07-01...RONDÔNIA365
441998-07-01...SÃO PAULO596
451998-08-01...ALAGOAS1
461998-08-01...BAHIA815
471998-08-01...DISTRITO FEDERAL48
481998-08-01...ESPÍRITO SANTO38
491998-08-01...MARANHÃO1176
501998-08-01...MATO GROSSO228
511998-08-01...MINAS GERAIS875
521998-08-01...PIAUÍ711
531998-08-01...RIO GRANDE DO SUL9
541998-08-01...RORAIMA0
551998-08-01...SERGIPE0
561998-09-01...AMAPÁ20
571998-09-01...DISTRITO FEDERAL33
581998-09-01...PIAUÍ1991
591998-09-01...RORAIMA2
601998-10-01...AMAZONAS83
611998-10-01...GOIÁS1034
621998-10-01...MATO GROSSO576
631998-10-01...PARAÍBA179
641998-10-01...PARÁ3665
651998-10-01...PIAUÍ2586
661998-10-01...SERGIPE0
671998-11-01...ALAGOAS19
681998-11-01...AMAPÁ131
691998-11-01...CEARÁ575
701998-11-01...DISTRITO FEDERAL0
711998-11-01...MARANHÃO2237
721998-11-01...RIO DE JANEIRO6
731998-11-01...RIO GRANDE DO SUL28
741998-11-01...SÃO PAULO488
751998-12-01...BAHIA82
761998-12-01...MARANHÃO1399
771998-12-01...MATO GROSSO100
781998-12-01...PARAÍBA51
791998-12-01...PERNAMBUCO59
801998-12-01...RIO DE JANEIRO1
811998-12-01...RONDÔNIA33
821998-12-01...TOCANTINS9
831999-01-01...ALAGOAS58
841999-01-01...GOIÁS14
851999-01-01...MATO GROSSO239
861999-01-01...MINAS GERAIS36
871999-01-01...PARÁ87
881999-01-01...PERNAMBUCO102
891999-01-01...RONDÔNIA1
901999-01-01...SÃO PAULO7
911999-01-01...TOCANTINS36
921999-02-01...ACRE0
931999-02-01...CEARÁ16
941999-02-01...MATO GROSSO69
951999-02-01...MATO GROSSO DO SUL28
961999-02-01...PERNAMBUCO13
971999-02-01...RONDÔNIA1
981999-02-01...SANTA CATARINA2
991999-02-01...TOCANTINS1
1001999-03-01...AMAPÁ2
titanic.head(100)
123
pclass
Int
100%
...
123
body
Int
9%
Abc
Varchar(100)
57%
11...22
21...[null]
31...[null]
41...[null]
51...[null]
61...[null]
71...[null]
81...62
91...[null]
101...[null]
111...[null]
121...[null]
131...110
141...[null]
151...[null]
161...38
171...[null]
181...126
191...292
201...175
211...[null]
221...122
231...166
241...[null]
251...207
261...[null]
271...232
281...[null]
291...[null]
301...[null]
311...[null]
321...[null]
331...46
341...169
351...[null]
361...[null]
371...[null]
381...[null]
391...[null]
401...[null]
411...[null]
421...[null]
431...[null]
441...[null]
451...[null]
461...[null]
471...[null]
481...[null]
491...[null]
501...[null]
511...[null]
521...[null]
531...[null]
541...[null]
551...[null]
561...[null]
571...[null]
581...[null]
591...[null]
601...[null]
611...[null]
621...[null]
631...[null]
641...[null]
651...[null]
661...[null]
671...[null]
681...[null]
691...[null]
701...[null]
711...[null]
721...[null]
731...[null]
742...[null]
752...[null]
762...[null]
772...[null]
782...[null]
792...[null]
802...[null]
812...[null]
822...[null]
832...236
842...[null]
852...[null]
862...[null]
872...155
882...[null]
892...75
902...35
912...[null]
922...[null]
932...[null]
942...[null]
952...[null]
962...[null]
972...[null]
982...165
992...[null]
1002...[null]

We can now run some queries to create a QueryProfiler object. One way to do so is by using the queries statement_id and transaction_id.

To allow for SQL execution in Jupyter cells, load the sql extension:

%load_ext verticapy.sql
The verticapy.sql extension is already loaded. To reload it, use:
  %reload_ext verticapy.sql

Next, let us run the queries:

%%sql
SELECT
    date,
    MONTH(date) as month,
    AVG(number) as avg_number_test
FROM public.amazon
GROUP BY date
ORDER BY avg_number_test DESC;
📅
date
Date
100%
...
123
month
Integer
100%
123
avg_number_test
Float(22)
100%
12007-09-01...93500.96296296296
22010-09-01...93163.25925925926
32003-09-01...92826.85185185185
42015-09-01...92669.74074074074
52002-08-01...82116.7037037037
62005-08-01...81925.22222222222
72002-10-01...101767.48148148148
82003-10-01...101603.51851851852
92004-10-01...101493.74074074074
102008-09-01...91460.92592592593
112012-10-01...101267.14814814815
122013-09-01...91169.81481481481
132016-10-01...101118.88888888889
142009-09-01...91090.0
152002-11-01...111043.66666666667
162015-11-01...111019.59259259259
172000-10-01...101012.44444444444
181998-10-01...10870.185185185185
192000-09-01...9862.62962962963
202017-07-01...7848.555555555556
%%sql
SELECT
    a.date,
    MONTH(a.date) AS month,
    AVG(a.number) AS avg_number_test,
    b.max_number
FROM
    public.amazon AS a
JOIN (
    SELECT
        date,
        MAX(number) AS max_number
    FROM
        public.amazon
    GROUP BY
        date
) AS b
ON
    a.date = b.date
GROUP BY
    a.date, b.max_number
ORDER BY
    avg_number_test DESC;
📅
date
Date
100%
...
123
month
Integer
100%
123
max_number
Integer
100%
12017-09-01...925004
22002-09-01...911034
32011-09-01...97086
42015-10-01...107781
52005-10-01...108405
62003-08-01...811132
72014-08-01...88555
81998-09-01...910363
91999-08-01...818566
102015-08-01...87913
112008-10-01...107691
121999-09-01...911729
131998-08-01...815406
142001-10-01...106661
152009-10-01...106342
162004-07-01...711517
172003-11-01...116195
182003-12-01...124787
192004-06-01...69813
202013-11-01...113688

In order to create a QueryProfiler object from a query, we need the queries statement_id and transaction_id, both of which are found in the QUERY_REQUESTS system table:

from verticapy.performance.vertica import QueryProfiler, QueryProfilerInterface

qprof = QueryProfiler((45035996273780927,76))

To create a QueryProfiler object w/ multiple queries, provide a list of tuples

qprof = QueryProfilerInterface([(45035996273780927,74), (45035996273780075,6)])

Once the QueryProfiler object is created, you can run the get_queries() method to view the queries contained in the QueryProfiler object:

qprof.get_queries()
...
📅
start_timestamp
Timestamp(29)
100%
📅
end_timestamp
Timestamp(29)
100%
0...2024-10-31 15:34:08.6802962024-10-31 15:34:08.744459
1...2024-10-31 15:34:14.0534102024-10-31 15:34:14.206118

To visualize the query plan, run get_qplan_tree(), which is customizable, allowing you to specify certain metrics or focus on a specified tree path:

_images/user_guide_performance_qprof_get_qplan_tree.png

Create a QueryProfiler object directly from a query

You can also create the QueryProfiler Object directly from an SQL Command:

qprof = QueryProfiler("""
    select transaction_id, statement_id, request, request_duration
    from query_requests where start_timestamp > (now() - interval'1 hour')
    order by request_duration desc limit 10;
    """
)

Save the QueryProfiler object in a target schema

After you create a QueryProfiler object, you can save it to a target schema.

In this example, we will save the object to the sc_demo schema:

vp.create_schema("sc_demo")
Out[6]: False

To save the QueryProfiler object, specify the target_schema and, optionally, a key_id (it is a unique key which is used to search for the stored Qprof object) when creating the QueryProfiler object:

# Save it to your schema
qprof = QueryProfiler(
    (45035996273780927, 76),
    target_schema = "sc_demo",
    key_id = "unique_xx1",
    overwrite = True,
)

Load a QueryProfiler object

To load a previously saved QueryProfiler, simply provide its target_schema and key_id:

from verticapy.performance.vertica import QueryProfiler, QueryProfilerInterface

# Someone else can now connect to my DB and use the object.
qprof = QueryProfiler(
    target_schema = "sc_demo",
    key_id = "unique_xx1",
)

Export and import

You can export and import QueryProfiler objects as .tar files.

Export

To export a QueryProfiler object, use the export_profile() method:

qprof.export_profile(filename="test_export_1.tar")

Note

There is also a shell script which helps you export qprof data without python. See qprof_export .

Import

To import a QueryProfiler object, use the import_profile() method and provide the target_schema and key_id.

Make sure the key_id is unique/unused. Let us create a new schema to load this into:

vp.create_schema("sc_demo_1")

qprof = QueryProfiler.import_profile(
    target_schema = "sc_demo_1",
    key_id = "unique_load_xx1",
    filename = "test_export_1.tar",
    auto_initialize = True,
)

Methods & attributes

The QueryProfiler object includes many useful methods and attributes to aid in the analysis of query performence.

Access performance tables

With the QueryProfiler object, you can access any of the following tables:

qprof.get_table()
Out[7]: 
['dc_requests_issued',
 'dc_explain_plans',
 'dc_query_executions',
 'dc_plan_activities',
 'dc_scan_events',
 'dc_slow_events',
 'execution_engine_profiles',
 'host_resources',
 'query_events',
 'query_plan_profiles',
 'query_profiles',
 'projection_storage',
 'projection_usage',
 'resource_pool_status',
 'storage_containers',
 'dc_lock_attempts',
 'dc_plan_resources',
 'configuration_parameters',
 'query_consumption',
 'projections',
 'projection_columns',
 'resource_acquisitions',
 'resource_pools']

For example, view the QUERY_EVENTS table:

qprof.get_table("query_events")
📅
event_timestamp
Timestamptz(35)
100%
...
Abc
event_severity
Varchar(13)
100%
Abc
Varchar(64000)
100%
12024-08-26 12:02:37.531048-04:00...Informational
22024-08-26 12:02:37.531966-04:00...Informational
32024-08-26 12:02:37.538079-04:00...Informational
42024-10-31 15:24:16.875429-04:00...Informational
52024-10-31 15:24:16.875502-04:00...Informational
62024-10-31 15:24:22.142267-04:00...Informational
72024-10-31 15:24:22.142323-04:00...Informational
82024-10-31 15:24:22.142794-04:00...Critical
92024-10-31 15:24:22.142864-04:00...Critical
102024-10-31 15:24:22.142910-04:00...Critical
112024-10-31 15:24:22.143009-04:00...Critical
122024-10-31 15:24:22.143103-04:00...Critical
132024-10-31 15:24:22.143658-04:00...Informational
142024-10-31 15:24:22.143732-04:00...Warning
152024-10-31 15:24:22.144575-04:00...Informational
162024-10-31 15:24:24.418582-04:00...Informational
172024-10-31 15:24:24.418645-04:00...Informational
182024-10-31 15:24:24.419569-04:00...Critical
192024-10-31 15:24:24.419636-04:00...Critical
202024-10-31 15:24:24.419673-04:00...Critical

Or the DC_EXPLAIN_PLANS table:

qprof.get_table('dc_explain_plans')
📅
time
Timestamptz(35)
100%
...
123
path_line_index
Integer
100%
Abc
Varchar(64000)
100%
12024-10-31 15:34:18.682896-04:00...3
22024-10-31 15:34:18.682897-04:00...4
32024-10-31 15:34:18.682898-04:00...5
42024-10-31 15:34:18.682899-04:00...6
52024-10-31 15:34:18.682900-04:00...7
62024-10-31 15:34:18.682954-04:00...1
72024-10-31 15:34:18.682957-04:00...2
82024-10-31 15:34:18.682958-04:00...3
92024-10-31 15:34:18.682960-04:00...4
102024-10-31 15:34:18.682961-04:00...5
112024-10-31 15:34:18.682962-04:00...6
122024-10-31 15:34:18.683022-04:00...1
132024-10-31 15:34:18.683028-04:00...2
142024-10-31 15:34:18.683029-04:00...3
152024-10-31 15:34:18.683031-04:00...4
162024-10-31 15:34:18.683032-04:00...5
172024-10-31 15:34:18.683033-04:00...6
182024-10-31 15:34:18.683034-04:00...7
192024-10-31 15:34:18.683074-04:00...1
202024-10-31 15:34:18.683077-04:00...2

Or the QUERY_CONSUMPTION table:

qprof.get_table("query_consumption")
📅
start_time
Timestamptz(35)
100%
...
010
is_retry
Boolean
59%
010
success
Boolean
82%
12024-10-30 23:59:28.093856-04:00...[null]
✅
22024-10-30 23:59:28.465125-04:00...[null]
✅
32024-10-30 23:59:30.273588-04:00...[null]
✅
42024-10-30 23:59:35.198877-04:00...[null]
✅
52024-10-30 23:59:35.948951-04:00...[null][null]
62024-10-30 23:59:36.062933-04:00...[null]
✅
72024-10-30 23:59:41.358634-04:00...[null]
✅
82024-10-30 23:59:56.879715-04:00...[null][null]
92024-10-31 00:00:00.683415-04:00...[null]
✅
102024-10-31 00:00:20.915550-04:00...[null]
✅
112024-10-31 00:00:21.274346-04:00...[null]
✅
122024-10-31 00:00:40.441579-04:00...[null]
✅
132024-10-31 00:00:45.360625-04:00...[null][null]
142024-10-31 00:01:02.417764-04:00...[null]
✅
152024-10-31 00:01:03.187370-04:00...[null]
✅
162024-10-31 00:01:03.299935-04:00...[null][null]
172024-10-31 00:01:03.615058-04:00...[null]
✅
182024-10-31 00:01:04.748955-04:00...[null]
✅
192024-10-31 00:01:06.115382-04:00...[null][null]
202024-10-31 00:01:09.571069-04:00...[null]
✅

Get query information

You can retrieve the query information, such as transaction_id and statement_id, from the QueryProfiler object:

qprof = QueryProfiler("""
    select transaction_id, statement_id, request, request_duration
    from query_requests where start_timestamp > (now() - interval'1 hour')
    order by request_duration desc limit 10;
    """
)

Searching the performance tables...
Setting the requests and queries durations...
Checking all the tables consistency using a single SQL query...
Checking all the tables data types...

View the transaction_id and statement_id:

tid = qprof.transaction_id

sid = qprof.statement_id

print(tid, sid)
45035996277028508 78

Display the query request:

# Pretty, Formatted, Results
qprof.get_request()
<IPython.core.display.Markdown object>
Out[12]: "PROFILE \n   SELECT\n     transaction_id,\n     statement_id,\n     request,\n     request_duration \n   FROM\n query_requests \n   WHERE start_timestamp > (now() - INTERVAL'1 hour') \n   ORDER BY request_duration DESC \n   LIMIT 10"

View the number of query steps in a bar graph:

qprof.get_qsteps(kind="bar")
Loading....
qprof.get_qplan()
+-SELECT  LIMIT 10 [Cost: 62K, Rows: 10 (NO STATISTICS)] (PATH ID: 0)
|  Output Only: 10 tuples
|  Execute on: Query Initiator
| +---> SORT [TOPK] [Cost: 62K, Rows: 10K (NO STATISTICS)] (PATH ID: 1)
| |      Order: query_requests.request_duration DESC
| |      Output Only: 10 tuples
| |      Execute on: All Nodes
| |      Execute on: All Nodes
| | +---> JOIN HASH [LeftOuter] [Cost: 2K, Rows: 10K (NO STATISTICS)] (PATH ID: 3)
| | |      Join Cond: (ri.node_name = dc_requests_completed.node_name) AND (ri.session_id = dc_requests_completed.session_id) AND (ri.request_id = dc_requests_completed.request_id)
| | |      Materialize at Output: ri."time", ri.transaction_id, ri.statement_id, ri.request
| | |      Execute on: All Nodes
| | | +-- Outer -> STORAGE ACCESS for ri [Cost: 649, Rows: 10K (NO STATISTICS)] (PATH ID: 4)
| | | |      Projection: v_internal.dc_requests_issued_p
| | | |      Materialize: ri.node_name, ri.session_id, ri.request_id
| | | |      Filter: (ri."time" > '2024-10-31 14:34:55.082575-04'::timestamptz)
| | | |      Execute on: All Nodes
| | | +-- Inner -> STORAGE ACCESS for dc_requests_completed [Cost: 965, Rows: 10K (NO STATISTICS)] (PATH ID: 5)
| | | |      Projection: v_internal.dc_requests_completed_p
| | | |      Materialize: dc_requests_completed.request_id, dc_requests_completed."time", dc_requests_completed.node_name, dc_requests_completed.session_id
| | | |      Filter: (dc_requests_completed.node_name IS NOT NULL)
| | | |      Filter: (dc_requests_completed.session_id IS NOT NULL)
| | | |      Filter: (dc_requests_completed.request_id IS NOT NULL)
| | | |      Execute on: All Nodes
Out[13]: '+-SELECT  LIMIT 10 [Cost: 62K, Rows: 10 (NO STATISTICS)] (PATH ID: 0)\n|  Output Only: 10 tuples\n|  Execute on: Query Initiator\n| +---> SORT [TOPK] [Cost: 62K, Rows: 10K (NO STATISTICS)] (PATH ID: 1)\n| |      Order: query_requests.request_duration DESC\n| |      Output Only: 10 tuples\n| |      Execute on: All Nodes\n| |      Execute on: All Nodes\n| | +---> JOIN HASH [LeftOuter] [Cost: 2K, Rows: 10K (NO STATISTICS)] (PATH ID: 3)\n| | |      Join Cond: (ri.node_name = dc_requests_completed.node_name) AND (ri.session_id = dc_requests_completed.session_id) AND (ri.request_id = dc_requests_completed.request_id)\n| | |      Materialize at Output: ri."time", ri.transaction_id, ri.statement_id, ri.request\n| | |      Execute on: All Nodes\n| | | +-- Outer -> STORAGE ACCESS for ri [Cost: 649, Rows: 10K (NO STATISTICS)] (PATH ID: 4)\n| | | |      Projection: v_internal.dc_requests_issued_p\n| | | |      Materialize: ri.node_name, ri.session_id, ri.request_id\n| | | |      Filter: (ri."time" > \'2024-10-31 14:34:55.082575-04\'::timestamptz)\n| | | |      Execute on: All Nodes\n| | | +-- Inner -> STORAGE ACCESS for dc_requests_completed [Cost: 965, Rows: 10K (NO STATISTICS)] (PATH ID: 5)\n| | | |      Projection: v_internal.dc_requests_completed_p\n| | | |      Materialize: dc_requests_completed.request_id, dc_requests_completed."time", dc_requests_completed.node_name, dc_requests_completed.session_id\n| | | |      Filter: (dc_requests_completed.node_name IS NOT NULL)\n| | | |      Filter: (dc_requests_completed.session_id IS NOT NULL)\n| | | |      Filter: (dc_requests_completed.request_id IS NOT NULL)\n| | | |      Execute on: All Nodes'
qprof.get_qplan_tree()

To view the query plan profile in a pie chart:

qprof.get_qplan_profile(kind = "pie")

To view the cpu time of the query in a bar graph:

qprof.get_cpu_time(kind="bar")

QueryProfiler execution report

The QueryProfiler object can also generate a report that includes various performence metrics, including which operation took the most amount of time:

qprof.get_qexecution_report().sort({"exec_time_us": "desc"})
Abc
node_name
Varchar(128)
100%
...
123
path_id
Integer
100%
123
total_rows_read_sort
Integer
0%
1v_mldb_node0001...3[null]
2v_mldb_node0001...5[null]
3v_mldb_node0001...5[null]
4v_mldb_node0001...3[null]
5v_mldb_node0001...2[null]
6v_mldb_node0001...1[null]
7v_mldb_node0001...1[null]
8v_mldb_node0001...4[null]
9v_mldb_node0001...1[null]
10v_mldb_node0001...-1[null]
11v_mldb_node0003...3[null]
12v_mldb_node0003...5[null]
13v_mldb_node0003...3[null]
14v_mldb_node0001...-1[null]
15v_mldb_node0002...3[null]
16v_mldb_node0004...3[null]
17v_mldb_node0003...1[null]
18v_mldb_node0001...1[null]
19v_mldb_node0002...5[null]
20v_mldb_node0001...0[null]

To view the query execution details:

qprof.get_qexecution()

QueryProfiler Summary Report Export

You can also easily export the entire report in an HTML format. This report can be read without having any connection to database or a jupyter environment making it very convenient to share and analyze offline.

qprof.to_html("my-report.html") # Where "my-report.html" is the path of the file to save.