Loading...

Joins

When working with datasets, we often need to merge data from different sources. To do this, we need keys on which to join our data.

Let’s use the US Flights 2015 datasets. We have three datasets.

First, we have information on each flight.

import verticapy as vp

flights  = vp.read_csv("flights.csv")
flights.head(100)
123
YEAR
Int
100%
...
123
LATE_AIRCRAFT_DELAY
Int
32%
123
WEATHER_DELAY
Int
32%
12015...920
22015...00
32015...[null][null]
42015...[null][null]
52015...[null][null]
62015...00
72015...00
82015...[null][null]
92015...[null][null]
102015...[null][null]
112015...00
122015...[null][null]
132015...[null][null]
142015...[null][null]
152015...[null][null]
162015...[null][null]
172015...230
182015...[null][null]
192015...00
202015...[null][null]
212015...00
222015...[null][null]
232015...[null][null]
242015...[null][null]
252015...[null][null]
262015...470
272015...00
282015...500
292015...[null][null]
302015...[null][null]
312015...[null][null]
322015...[null][null]
332015...[null][null]
342015...[null][null]
352015...[null][null]
362015...[null][null]
372015...[null][null]
382015...00
392015...1840
402015...[null][null]
412015...[null][null]
422015...460
432015...[null][null]
442015...[null][null]
452015...[null][null]
462015...[null][null]
472015...350
482015...[null][null]
492015...[null][null]
502015...00
512015...740
522015...[null][null]
532015...[null][null]
542015...[null][null]
552015...[null][null]
562015...00
572015...00
582015...[null][null]
592015...[null][null]
602015...[null][null]
612015...[null][null]
622015...[null][null]
632015...[null][null]
642015...[null][null]
652015...[null][null]
662015...[null][null]
672015...[null][null]
682015...[null][null]
692015...[null][null]
702015...[null][null]
712015...960
722015...[null][null]
732015...[null][null]
742015...[null][null]
752015...[null][null]
762015...[null][null]
772015...[null][null]
782015...[null][null]
792015...[null][null]
802015...[null][null]
812015...[null][null]
822015...[null][null]
832015...[null][null]
842015...[null][null]
852015...[null][null]
862015...[null][null]
872015...[null][null]
882015...00
892015...[null][null]
902015...[null][null]
912015...00
922015...[null][null]
932015...700
942015...[null][null]
952015...[null][null]
962015...[null][null]
972015...[null][null]
982015...[null][null]
992015...1140
1002015...1100

Second, we have information on each airport.

airports = vp.read_csv("airports.csv")
airports.head(100)
Abc
IATA_CODE
Varchar(20)
100%
...
🌎
LATITUDE
Float
100%
🌎
LONGITUDE
Float
100%
1ABI...32.41132-99.6819
2ABR...45.44906-98.42183
3ACY...39.45758-74.57717
4AEX...31.32737-92.54856
5ALB...42.74812-73.80298
6ALO...42.55708-92.40034
7ATL...33.64044-84.42694
8AVL...35.43619-82.54181
9BGM...42.20848-75.97961
10BHM...33.56294-86.75355
11BMI...40.47799-88.91595
12BOI...43.56444-116.22278
13BOS...42.36435-71.00518
14BQN...18.49486-67.12944
15BTM...45.9548-112.49746
16BTR...30.53316-91.14963
17BUF...42.94052-78.73217
18CAK...40.91631-81.44247
19CEC...41.78016-124.23653
20CHA...35.03527-85.20379
21CHS...32.89865-80.04051
22CLD...33.12723-117.27873
23CRW...38.37315-81.59319
24DAB...29.17992-81.05806
25DAY...39.90238-84.21938
26DCA...38.85208-77.03772
27DHN...31.32134-85.44963
28DIK...46.79739-102.80195
29DLH...46.84209-92.19365
30DSM...41.53493-93.66068
31EVV...38.03799-87.53063
32GNV...29.69006-82.27178
33GRB...44.48507-88.12959
34GRK...31.0649-97.8278
35GRR...42.88082-85.52277
36GSO...36.09775-79.9373
37GUC...38.53396-106.93318
38HLN...46.60682-111.98275
39HOB...32.68753-103.21703
40HPN...41.06696-73.70757
41IAH...29.98047-95.33972
42ILG...39.67872-75.60653
43ISP...40.79524-73.10021
44JAX...30.49406-81.68786
45JFK...40.63975-73.77893
46LAR...41.31205-105.67499
47LAW...34.56771-98.41664
48LAX...33.94254-118.40807
49MCO...28.42889-81.31603
50MFR...42.37423-122.8735
51MKG...43.16949-86.23822
52MLI...41.44853-90.50754
53MLU...32.51087-92.03769
54MOB...30.69142-88.24283
55MQT...46.35364-87.39536
56MRY...36.58698-121.84295
57MSN...43.13986-89.33751
58OMA...41.30252-95.89417
59OME...64.5122-165.44525
60ORD...41.9796-87.90446
61ORH...42.26734-71.87571
62OTH...43.41714-124.24603
63PBI...26.68316-80.09559
64PHL...39.87195-75.24114
65PIB...31.46715-89.33706
66PSE...18.0083-66.56301
67PSG...56.80165-132.94528
68RDM...44.25407-121.14996
69RDU...35.87764-78.78747
70ROA...37.32547-79.97543
71RSW...26.53617-81.75517
72SBA...34.42621-119.84037
73SBN...41.70895-86.31847
74SCE...40.85121-77.8463
75SEA...47.44898-122.30931
76SMF...38.69542-121.59077
77SMX...34.89925-120.45758
78SNA...33.67566-117.86822
79SPI...39.84395-89.67762
80STL...38.74769-90.35999
81STX...17.70189-64.79856
82SWF...41.50409-74.10484
83TPA...27.97547-82.53325
84TRI...36.47521-82.40742
85TTN...40.27669-74.81347
86TUL...36.19837-95.88824
87YAK...59.50336-139.66023
88ABY...31.53552-84.19447
89ACK...41.25305-70.06018
90ACT...31.61129-97.23052
91ADQ...57.74997-152.49386
92AUS...30.19453-97.66987
93BDL...41.93887-72.68323
94BET...60.77978-161.838
95BFL...35.4336-119.05677
96BLI...48.79275-122.53753
97BRD...46.39786-94.13723
98BTV...44.473-73.15031
99CAE...33.93884-81.11954
100CID...41.88459-91.71087

Third, we have the names of each airline.

airlines = vp.read_csv("airlines.csv")
airlines.head(100)
Abc
IATA_CODE
Varchar(20)
100%
Abc
AIRLINE
Varchar(56)
100%
1AAAmerican Airlines Inc.
2VXVirgin America
3ASAlaska Airlines Inc.
4B6JetBlue Airways
5DLDelta Air Lines Inc.
6HAHawaiian Airlines Inc.
7MQAmerican Eagle Airlines Inc.
8NKSpirit Air Lines
9EVAtlantic Southeast Airlines
10F9Frontier Airlines Inc.
11WNSouthwest Airlines Co.
12OOSkywest Airlines Inc.
13UAUnited Air Lines Inc.
14USUS Airways Inc.

Notice that each dataset has a primary or secondary key on which to join the data. For example, we can join the flights dataset to the airlines and airport datasets using the corresponding IATA code.

To join datasets in VerticaPy, use the join() method.

help(vp.vDataFrame.join)
Help on function join in module verticapy.core.vdataframe._join_union_sort:

join(self, input_relation: Annotated[Union[str, ForwardRef('vDataFrame')], ''], on: Union[NoneType, tuple, dict, list] = None, on_interpolate: Optional[dict] = None, how: Literal['left', 'right', 'cross', 'full', 'natural', 'self', 'inner', None] = 'natural', expr1: Optional[Annotated[Union[str, list[str], ForwardRef('StringSQL'), list['StringSQL']], '']] = None, expr2: Optional[Annotated[Union[str, list[str], ForwardRef('StringSQL'), list['StringSQL']], '']] = None) -> 'vDataFrame'

Joins the :py:class:`~vDataFrame` with another
one or an ``input_relation``.

.. warning::

    Joins  can  make  the  vDataFrame  structure
    heavier.  It is recommended that you check
    the    current     structure    using    the
    ``current_relation``  method  and  save  it
    with the ``to_db`` method, using the parameters
    ``inplace = True`` and ``relation_type = table``.

Parameters
----------
input_relation: SQLRelation
    Relation to join with.
on: tuple | dict | list, optional
    If using a list:
    List of 3-tuples. Each tuple must include
    (key1, key2, operator)  where ``key1`` is
    the key of the :py:class:`~vDataFrame`,
    ``key2`` is the key of the ``input_relation``,
    and ``operator`` is one of the following:

    - '=':
        exact match
    - '<':
        key1  < key2
    - '>':
        key1  > key2
    - '<=':
        key1 <= key2
    - '>=':
        key1 >= key2
    - 'llike':
        key1 LIKE '%' || key2 || '%'
    - 'rlike':
        key2 LIKE '%' || key1 || '%'
    - 'linterpolate':
        key1 INTERPOLATE key2
    - 'rinterpolate':
        key2 INTERPOLATE key1

    Some operators need 5-tuples:
    ``(key1, key2, operator, operator2, x)``
    where  ``operator2`` is  a simple operator
    ``(=, >, <, <=, >=)``, x is a ``float`` or
    an ``integer``, and ``operator`` is one of the
    following:

    - 'jaro':
        ``JARO(key1, key2) operator2 x``
    - 'jarow':
        ``JARO_WINCKLER(key1, key2) operator2 x``
    - 'lev':
        ``LEVENSHTEIN(key1, key2) operator2 x``

    If using a dictionary:
    This parameter must include all the different
    keys. It must be similar to the following:
    ``{"relationA_key1": "relationB_key1" ...,"relationA_keyk": "relationB_keyk"}``
    where ``relationA`` is the current :py:class:`~vDataFrame`
    and ``relationB`` is the ``input_relation`` or
    the input :py:class:`~vDataFrame`.

on_interpolate: dict, optional
    Dictionary of all unique keys. This is used
    to join two event series together using some
    ordered attribute. Event series joins let you
    compare values from two series directly, rather
    than having to normalize the series to the same
    measurement interval. The dict must be similar
    to the following:
    ``{"relationA_key1": "relationB_key1" ...,"relationA_keyk": "relationB_keyk"}``
    where ``relationA`` is the current :py:class:`~vDataFrame`
    and ``relationB`` is the ``input_relation`` or the
    input :py:class:`~vDataFrame`.

how: str, optional
    Join Type.

    - left:
        Left Join.
    - right:
        Right Join.
    - cross:
        Cross Join.
    - full:
        Full Outer Join.
    - natural:
        Natural Join.
    - inner:
        Inner Join.

expr1: SQLExpression, optional
    List of the different columns in pure SQL
    to select from the current :py:class:`~vDataFrame`,
    optionally as aliases. Aliases are recommended
    to avoid ambiguous names. For example: ``column``
    or ``column AS my_new_alias``.
expr2: SQLExpression, optional
    List of the different columns in pure SQL
    to select from the current :py:class:`~vDataFrame`,
    optionally as aliases. Aliases are recommended
    to avoid ambiguous names. For example: ``column``
    or ``column AS my_new_alias``.

Returns
-------
vDataFrame
    object result of the join.

Let’s use a left join to merge the airlines dataset and the flights dataset.

flights = flights.join(
    airlines,
    how = "left",
    on = {"airline": "IATA_CODE"},
    expr2 = ["AIRLINE AS airline_long"],
)
flights.head(100)
123
YEAR
Integer
100%
...
123
WEATHER_DELAY
Integer
32%
Abc
airline_long
Varchar(56)
100%
12015...0American Airlines Inc.
22015...0American Airlines Inc.
32015...[null]American Airlines Inc.
42015...[null]American Airlines Inc.
52015...[null]American Airlines Inc.
62015...0American Airlines Inc.
72015...0American Airlines Inc.
82015...[null]American Airlines Inc.
92015...[null]American Airlines Inc.
102015...[null]American Airlines Inc.
112015...0American Airlines Inc.
122015...[null]American Airlines Inc.
132015...[null]American Airlines Inc.
142015...[null]American Airlines Inc.
152015...[null]American Airlines Inc.
162015...[null]American Airlines Inc.
172015...0American Airlines Inc.
182015...[null]American Airlines Inc.
192015...0American Airlines Inc.
202015...[null]American Airlines Inc.
212015...0American Airlines Inc.
222015...[null]American Airlines Inc.
232015...[null]American Airlines Inc.
242015...[null]American Airlines Inc.
252015...[null]American Airlines Inc.
262015...0American Airlines Inc.
272015...0American Airlines Inc.
282015...0American Airlines Inc.
292015...[null]American Airlines Inc.
302015...[null]American Airlines Inc.
312015...[null]American Airlines Inc.
322015...[null]American Airlines Inc.
332015...[null]American Airlines Inc.
342015...[null]American Airlines Inc.
352015...[null]American Airlines Inc.
362015...[null]American Airlines Inc.
372015...[null]American Airlines Inc.
382015...0American Airlines Inc.
392015...0American Airlines Inc.
402015...[null]American Airlines Inc.
412015...[null]American Airlines Inc.
422015...0American Airlines Inc.
432015...[null]American Airlines Inc.
442015...[null]American Airlines Inc.
452015...[null]American Airlines Inc.
462015...[null]American Airlines Inc.
472015...0American Airlines Inc.
482015...[null]American Airlines Inc.
492015...[null]American Airlines Inc.
502015...0American Airlines Inc.
512015...0American Airlines Inc.
522015...[null]American Airlines Inc.
532015...[null]American Airlines Inc.
542015...[null]American Airlines Inc.
552015...[null]American Airlines Inc.
562015...0American Airlines Inc.
572015...0American Airlines Inc.
582015...[null]American Airlines Inc.
592015...[null]American Airlines Inc.
602015...[null]American Airlines Inc.
612015...[null]American Airlines Inc.
622015...[null]American Airlines Inc.
632015...[null]American Airlines Inc.
642015...[null]American Airlines Inc.
652015...[null]American Airlines Inc.
662015...[null]American Airlines Inc.
672015...[null]American Airlines Inc.
682015...[null]American Airlines Inc.
692015...[null]American Airlines Inc.
702015...[null]American Airlines Inc.
712015...0American Airlines Inc.
722015...[null]American Airlines Inc.
732015...[null]American Airlines Inc.
742015...[null]American Airlines Inc.
752015...[null]American Airlines Inc.
762015...[null]American Airlines Inc.
772015...[null]American Airlines Inc.
782015...[null]American Airlines Inc.
792015...[null]American Airlines Inc.
802015...[null]American Airlines Inc.
812015...[null]American Airlines Inc.
822015...[null]American Airlines Inc.
832015...[null]American Airlines Inc.
842015...[null]American Airlines Inc.
852015...[null]American Airlines Inc.
862015...[null]American Airlines Inc.
872015...[null]American Airlines Inc.
882015...0American Airlines Inc.
892015...[null]American Airlines Inc.
902015...[null]American Airlines Inc.
912015...0American Airlines Inc.
922015...[null]American Airlines Inc.
932015...0American Airlines Inc.
942015...[null]American Airlines Inc.
952015...[null]American Airlines Inc.
962015...[null]American Airlines Inc.
972015...[null]American Airlines Inc.
982015...[null]American Airlines Inc.
992015...0American Airlines Inc.
1002015...0American Airlines Inc.

Let’s use two left joins to get the information on the origin and destination airports.

flights = flights.join(
    airports,
    how = "left",
    on = {"origin_airport": "IATA_CODE"},
    expr2 = [
        "LATITUDE AS origin_lat",
        "LONGITUDE AS origin_lon",
    ],
)
flights = flights.join(
    airports,
    how = "left",
    on = {"destination_airport": "IATA_CODE"},
    expr2 = [
        "LATITUDE AS destination_lat",
        "LONGITUDE AS destination_lon",
    ],
)
flights.head(100)
123
YEAR
Integer
100%
...
123
destination_lat
Float(22)
99%
123
destination_lon
Float(22)
99%
12015...28.42889-81.31603
22015...26.07258-80.15275
32015...26.07258-80.15275
42015...40.63975-73.77893
52015...30.19453-97.66987
62015...40.63975-73.77893
72015...35.21401-80.94313
82015...33.94254-118.40807
92015...40.63975-73.77893
102015...42.36435-71.00518
112015...38.69542-121.59077
122015...26.07258-80.15275
132015...18.43942-66.00183
142015...36.08036-115.15233
152015...43.11887-77.67238
162015...40.77724-73.87261
172015...40.63975-73.77893
182015...42.36435-71.00518
192015...27.97547-82.53325
202015...18.43942-66.00183
212015...33.81772-118.15161
222015...40.63975-73.77893
232015...40.63975-73.77893
242015...40.77724-73.87261
252015...28.42889-81.31603
262015...38.85208-77.03772
272015...41.06696-73.70757
282015...42.36435-71.00518
292015...40.63975-73.77893
302015...18.43942-66.00183
312015...42.36435-71.00518
322015...40.6925-74.16866
332015...42.36435-71.00518
342015...42.36435-71.00518
352015...40.63975-73.77893
362015...26.53617-81.75517
372015...26.07258-80.15275
382015...42.36435-71.00518
392015...40.63975-73.77893
402015...26.53617-81.75517
412015...40.63975-73.77893
422015...43.11887-77.67238
432015...33.94254-118.40807
442015...42.94052-78.73217
452015...27.97547-82.53325
462015...40.78839-111.97777
472015...40.6925-74.16866
482015...27.97547-82.53325
492015...42.36435-71.00518
502015...40.63975-73.77893
512015...37.619-122.37484
522015...40.63975-73.77893
532015...28.42889-81.31603
542015...38.85208-77.03772
552015...42.36435-71.00518
562015...33.43417-112.00806
572015...26.07258-80.15275
582015...36.08036-115.15233
592015...33.94254-118.40807
602015...26.07258-80.15275
612015...18.0083-66.56301
622015...37.36186-121.92901
632015...40.63975-73.77893
642015...29.99339-90.25803
652015...30.49406-81.68786
662015...26.53617-81.75517
672015...28.42889-81.31603
682015...40.63975-73.77893
692015...33.94254-118.40807
702015...38.85208-77.03772
712015...33.94254-118.40807
722015...40.6925-74.16866
732015...42.36435-71.00518
742015...18.43942-66.00183
752015...28.42889-81.31603
762015...27.39533-82.55411
772015...40.63975-73.77893
782015...40.63975-73.77893
792015...26.68316-80.09559
802015...42.36435-71.00518
812015...40.63975-73.77893
822015...30.19453-97.66987
832015...40.78839-111.97777
842015...28.42889-81.31603
852015...42.36435-71.00518
862015...41.06696-73.70757
872015...42.36435-71.00518
882015...38.85208-77.03772
892015...41.50409-74.10484
902015...26.07258-80.15275
912015...18.43942-66.00183
922015...36.08036-115.15233
932015...35.87764-78.78747
942015...40.63975-73.77893
952015...41.93887-72.68323
962015...42.36435-71.00518
972015...40.63975-73.77893
982015...18.49486-67.12944
992015...42.36435-71.00518
1002015...42.36435-71.00518

To avoid duplicate information, splitting the data into different tables is very important. Just imagine: what if we wrote the longitude and the latitude of the destination and origin airports for each flight? It would add way too many duplicates and drastically impact the volume of the data.

Cross joins are special: they don’t need a key. Cross joins are used to perform mathematical operations.

Let’s use a cross join of the airports dataset on itself to compute the distance between every airport.

distances = airports.join(
    airports,
    how = "cross",
    expr1 = [
        "IATA_CODE AS airport1",
        "LATITUDE AS airport1_latitude",
        "LONGITUDE AS airport1_longitude"
    ],
    expr2 = [
        "IATA_CODE AS airport2",
        "LATITUDE AS airport2_latitude",
        "LONGITUDE AS airport2_longitude",
    ],
)
distances.filter("airport1 != airport2")

import verticapy.sql.functions as fun

distances["distance"] = fun.distance(
    distances["airport1_latitude"],
    distances["airport1_longitude"],
    distances["airport2_latitude"],
    distances["airport2_longitude"],
)
123
distance
Float(22)
17450.70997424659
26421.84779178945
35391.61496246993
46523.04980811255
54844.61059668544
65276.14927676841
74972.32495308659
87287.89561709663
92126.1820031486
107269.62662402063
116281.2924573778
125231.27187027905
135165.70779431547
147334.43604776347
156825.29092241442
165492.87923122046
175480.20089285439
186655.0263408654
195358.90426145315
206849.70391597978

VerticaPy offers many powerful options for joining datasets.

In the next lesson, we’ll learn how to deal with duplicates.