Loading...

verticapy.read_json

verticapy.read_json(path: str, schema: str | None = None, table_name: str | None = None, usecols: list | None = None, new_name: dict | None = None, insert: bool = False, start_point: str = None, record_terminator: str = None, suppress_nonalphanumeric_key_chars: bool = False, reject_on_materialized_type_error: bool = False, reject_on_duplicate: bool = False, reject_on_empty_key: bool = False, flatten_maps: bool = True, flatten_arrays: bool = False, temporary_table: bool = False, temporary_local_table: bool = True, gen_tmp_table_name: bool = True, ingest_local: bool = True, genSQL: bool = False, materialize: bool = True, use_complex_dt: bool = False, is_avro: bool = False) → vDataFrame

Ingests a JSON file using flex tables.

Parameters

path: str

Absolute path where the JSON file is located.

schema: str, optional

Schema where the JSON file will be ingested.

table_name: str, optional

Final relation name.

usecols: list, optional

list of the JSON parameters to ingest. The other parameters will be ignored. If empty, all the JSON parameters will be ingested.

new_name: dict, optional

Dictionary of the new column names. If the JSON file is nested, it is recommended to change the final names because special characters will be included in the new column names. For example, {"param": {"age": 3, "name": Badr}, "date": 1993-03-11} will create 3 columns: “param.age”, “param.name” and “date”. You can rename these columns using the new_name parameter with the following dictionary: {"param.age": "age", "param.name": "name"}

insert: bool, optional

If set to True, the data is ingested into the input relation. The JSON parameters must be the same as the input relation otherwise they will not be ingested. If set to True, table_name cannot be empty.

start_point: str, optional

str, name of a key in the JSON load data at which to begin parsing. The parser ignores all data before the start_point value. The value is loaded for each object in the file. The parser processes data after the first instance, and up to the second, ignoring any remaining data.

record_terminator: str, optional

When set, any invalid JSON records are skipped and parsing continues with the next record. Records must be terminated uniformly. For example, if your input file has JSON records terminated by newline characters, set this parameter to \n. If any invalid JSON records exist, parsing continues after the next record_terminator. Even if the data does not contain invalid records, specifying an explicit record terminator can improve load performance by allowing cooperative parse and apportioned load to operate more efficiently. When you omit this parameter, parsing ends at the first invalid JSON record.

suppress_nonalphanumeric_key_chars: bool, optional

boolean, whether to suppress non-alphanumeric characters in JSON key values. The parser replaces these characters with an underscore (_) when this parameter is True.

reject_on_materialized_type_error: bool, optional

boolean, whether to reject a data row that contains a materialized column value that cannot be coerced into a compatible data type. If the value is False and the type cannot be coerced, the parser sets the value in that column to None. If the column is a strongly-typed complex type, as opposed to a flexible complex type, then a type mismatch anywhere in the complex type causes the entire column to be treated as a mismatch. The parser does not partially load complex types.

reject_on_duplicate: bool, optional

boolean, whether to ignore duplicate records (False), or to reject duplicates (True). In either case, the load continues.

reject_on_empty_key: bool, optional

boolean, whether to reject any row containing a field key without a value.

flatten_maps: bool, optional

boolean, whether to flatten sub-maps within the JSON data, separating map levels with a period (.). This value affects all data in the load, including nested maps.

flatten_arrays: bool, optional

boolean, whether to convert lists to sub-maps with integer keys. When lists are flattened, key names are concatenated in the same way as maps. lists are not flattened by default. This value affects all data in the load, including nested lists.

temporary_table: bool, optional

If set to True, a temporary table will be created.

temporary_local_table: bool, optional

If set to True, a temporary local table will be created. The parameter schema must be empty, otherwise this parameter is ignored.

gen_tmp_table_name: bool, optional

Sets the name of the temporary table. This parameter is only used when the parameter temporary_local_table is set to True and if the parameters table_name and schema are unspecified.

ingest_local: bool, optional

If set to True, the file will be ingested from the local machine.

genSQL: bool, optional

If set to True, the SQL code for creating the final table is generated but not executed. This is a good way to change the final relation types or to customize the data ingestion.

materialize: bool, optional

If set to True, the flex table is materialized into a table. Otherwise, it will remain a flex table. Flex tables simplify the data ingestion but have worse performace compared to regular tables.

use_complex_dt: bool, optional

boolean, whether the input data file has complex structure. If set to True, most of the other parameters are ignored.

Returns

vDataFrame

The vDataFrame of the relation.

Examples

In this example, we will first create a JSON file using vDataFrame.to_json() and ingest it into Vertica database.

We import verticapy:

import verticapy as vp

Hint

By assigning an alias to verticapy, we mitigate the risk of code collisions with other libraries. This precaution is necessary because verticapy uses commonly known function names like “average” and “median”, which can potentially lead to naming conflicts. The use of an alias ensures that the functions from verticapy are used as intended without interfering with functions from other libraries.

We will use the Titanic dataset.

import verticapy.datasets as vpd

data = vpd.load_titanic()
123
pclass
Integer
123
survived
Integer
Abc
Varchar(164)
Abc
sex
Varchar(20)
123
age
Numeric(8)
123
sibsp
Integer
123
parch
Integer
Abc
ticket
Varchar(36)
123
fare
Numeric(12)
Abc
cabin
Varchar(30)
Abc
embarked
Varchar(20)
Abc
boat
Varchar(100)
123
body
Integer
Abc
Varchar(100)
110male71.000PC 1760949.5042[null]C[null]22
210male45.00011378435.5TS[null][null]
310male[null]0011379831.0[null]S[null][null]
410male17.00011305947.1[null]S[null][null]
510male27.01013508136.7792C89C[null][null]
610male37.011PC 1775683.1583E52C[null][null]
710male31.010F.C. 1275052.0B71S[null][null]
810male50.010PC 17761106.425C86C[null]62
910female36.000PC 1753131.6792A29C[null][null]
1010male37.01011380353.1C123S[null][null]
1110male24.000PC 1759379.2B86C[null][null]
1210male45.0103697383.475C83S[null][null]
1310male40.0001120590.0B94S[null]110
1410male42.00011303842.5B11S[null][null]
1510male[null]001746351.8625E46S[null][null]
1610male42.01011378952.0[null]S[null]38
1710male[null]00PC 1760030.6958[null]C14[null]
1810male29.00011350130.0D6S[null]126
1910male46.0001305075.2417C6C[null]292
2010male54.0001746351.8625E46S[null]175
2110male47.00011379642.4[null]S[null][null]
2210male58.00235273113.275D48C[null]122
2310male45.50011304328.5C124S[null]166
2410male29.01011377666.6C2S[null][null]
2510male47.00011046552.0C110S[null]207
2610male38.000199720.0[null]S[null][null]
2710male22.000PC 17760135.6333[null]C[null]232
2810male31.000PC 1759050.4958A24S[null][null]
2910male50.0101350755.9E44S[null][null]
3010male56.0001776430.6958A7C[null][null]
3110male57.010PC 17569146.5208B78C[null][null]
3210female63.010PC 17483221.7792C55 C57S[null][null]
3310male61.0003696332.3208D50S[null]46
3410male21.0013528177.2875D26S[null]169
3510male51.001PC 1759761.3792[null]C[null][null]
3611female63.0101350277.9583D7S10[null]
3711female32.0001181376.2917D15C8[null]
3811female58.00011378326.55C103S8[null]
3911female44.000PC 1761027.7208B4C6[null]
4011female41.00016966134.5E40C3[null]
4111female53.000PC 1760627.4458[null]C6[null]
4211male36.001PC 17755512.3292B51 B53 B55C3[null]
4311female58.001PC 17755512.3292B51 B53 B55C3[null]
4411male11.012113760120.0B96 B98S4[null]
4511female76.0101987778.85C46S6[null]
4611female[null]0111350555.0E33S6[null]
4711female39.011PC 1775683.1583E49C14[null]
4811female27.012F.C. 1275052.0B71S3[null]
4911female[null]0017421110.8833[null]C4[null]
5011female35.000113503211.5C130C4[null]
5111female22.00111237859.4[null]C7[null]
5211female25.0101176555.4417E50C5[null]
5311male48.010PC 1757276.7292D33C3[null]
5411female35.0103697383.475C83SD[null]
5511male27.000PC 1757276.7292D49C3[null]
5611female24.0001176783.1583C54C7[null]
5711female52.0111274993.5B69S3[null]
5811female44.00111136157.9792B18C4[null]
5911female15.00124160211.3375B5S2[null]
6011male30.0101323657.75C78C11[null]
6111female31.01035273113.275D36C6[null]
6211female39.000PC 17758108.9C105C8[null]
6311female22.00111350961.9792B36C5[null]
6411male52.00011378630.5C104S6[null]
6511female43.00124160211.3375B3S2[null]
6611female33.00011015286.5B77S8[null]
6711male45.01116966134.5E34C3[null]
6811female40.01116966134.5E34C3[null]
6911male48.0101999652.0C126S5 7[null]
7011female[null]00PC 1758579.2[null]CD[null]
7111female35.000PC 17755512.3292[null]C3[null]
7211female60.01011081375.25D37C5[null]
7311male21.001PC 1759761.3792[null]CA[null]
7420male23.000C.A. 3103010.5[null]S[null][null]
7520male28.00024435826.0[null]S[null][null]
7620male60.0112975039.0[null]S[null][null]
7720female44.01024425226.0[null]S[null][null]
7820male29.010200326.0[null]S[null][null]
7920male18.000S.O.C. 1487973.5[null]S[null][null]
8020male18.000S.O.C. 1487973.5[null]S[null][null]
8120male54.0002840326.0[null]S[null][null]
8220male18.00023617113.0[null]S[null][null]
8320male36.00022923613.0[null]S[null]236
8420male34.0102866421.0[null]S[null][null]
8520male21.0102813311.5[null]S[null][null]
8620male21.0102813411.5[null]S[null][null]
8720male24.00023386613.0[null]S[null]155
8820male34.0001223313.0[null]S[null][null]
8920male30.00025065313.0[null]S[null]75
9020male44.00024874613.0[null]S[null]35
9120male49.01222084565.0[null]S[null][null]
9220male21.020S.O.C. 1487973.5[null]S[null][null]
9320male21.000S.O.C. 1487973.5[null]S[null][null]
9420female60.0102406526.0[null]S[null][null]
9520male24.020C.A. 3102931.5[null]S[null][null]
9620male22.020C.A. 3102931.5[null]S[null][null]
9720male35.00023373412.35[null]Q[null][null]
9820male31.000C.A. 1872310.5[null]S[null]165
9920male36.000SC/Paris 216312.875DC[null][null]
10020male[null]00SC/A.3 286115.5792[null]C[null][null]
Rows: 1-100 | Columns: 14

Note

VerticaPy offers a wide range of sample datasets that are ideal for training and testing purposes. You can explore the full list of available datasets in the Datasets, which provides detailed information on each dataset and how to use them effectively. These datasets are invaluable resources for honing your data analysis and machine learning skills within the VerticaPy environment.

Let’s convert the vDataFrame to a JSON file.

data[0:20].to_json(
    path = "titanic_subset.json",
)

Let’s ingest the json file into the Vertica database.

from verticapy.core.parsers.json import read_json

read_json(
    path = "titanic_subset.json",
    table_name = "titanic_subset",
    schema = "public",
)
123
boat
Integer
123
body
Integer
Abc
cabin
Varchar(20)
123
age
Numeric(10)
Abc
home.dest
Varchar(64)
Abc
embarked
Varchar(20)
Abc
sex
Varchar(20)
123
parch
Integer
123
pclass
Integer
123
fare
Numeric(13)
Abc
name
Varchar(64)
123
sibsp
Integer
123
survived
Integer
Abc
ticket
Varchar(20)
1[null][null]E46[null]Brighton, MASmale0151.8625Hilliard, Mr. Herbert Henry0017463
2[null][null][null]17.0Montevideo, UruguaySmale0147.1Carrau, Mr. Jose Pedro00113059
3[null]62C8650.0Deephaven, MN / Cedar Rapids, IACmale01106.425Douglas, Mr. Walter Donald10PC 17761
4[null][null]B1142.0London / MiddlesexSmale0142.5Head, Mr. Christopher00113038
5[null][null][null][null][null]Smale0131.0Cairns, Mr. Alexander00113798
6[null]22[null]71.0Montevideo, UruguayCmale0149.5042Artagaveytia, Mr. Ramon00PC 17609
7[null]292C646.0Vancouver, BCCmale0175.2417McCaffry, Mr. Thomas Francis0013050
8[null][null]A2936.0New York, NYCfemale0131.6792Evans, Miss. Edith Corse00PC 17531
9[null][null]C8345.0New York, NYSmale0183.475Harris, Mr. Henry Birkhardt1036973
10[null][null]E5237.0Lakewood, NJCmale1183.1583Compton, Mr. Alexander Taylor Jr10PC 17756
11[null][null]T45.0Trenton, NJSmale0135.5Blackwell, Mr. Stephen Weart00113784
12[null]110B9440.0[null]Smale010.0Harrison, Mr. William00112059
13[null][null]B7131.0Montreal, PQSmale0152.0Davidson, Mr. Thornton10F.C. 12750
14[null][null]B8624.0[null]Cmale0179.2Giglio, Mr. Victor00PC 17593
15[null][null]C12337.0Scituate, MASmale0153.1Futrelle, Mr. Jacques Heath10113803
16[null][null]C8927.0Los Angeles, CACmale01136.7792Clark, Mr. Walter Miller1013508
17[null]38[null]42.0New York, NYSmale0152.0Holverson, Mr. Alexander Oskar10113789
18[null]126D629.0Springfield, MASmale0130.0Long, Mr. Milton Clyde00113501
19[null]175E4654.0Dorchester, MASmale0151.8625McCarthy, Mr. Timothy J0017463
2014[null][null][null]New York, NYCmale0130.6958Hoyt, Mr. William Fisher00PC 17600
Rows: 1-20 | Columns: 14

Let’s ingest the json and rename some columns.

read_json(
    path = "titanic_subset.json",
    table_name = "titanic_sub_newnames",
    schema = "public",
    new_name = {
        "fields.fare": "fare",
        "fields.sex": "sex",
    },
)
123
boat
Integer
123
body
Integer
Abc
cabin
Varchar(20)
123
age
Numeric(10)
Abc
home.dest
Varchar(64)
Abc
embarked
Varchar(20)
Abc
sex
Varchar(20)
123
parch
Integer
123
pclass
Integer
123
fare
Numeric(13)
Abc
name
Varchar(64)
123
sibsp
Integer
123
survived
Integer
Abc
ticket
Varchar(20)
1[null][null]E46[null]Brighton, MASmale0151.8625Hilliard, Mr. Herbert Henry0017463
2[null][null][null]17.0Montevideo, UruguaySmale0147.1Carrau, Mr. Jose Pedro00113059
3[null]62C8650.0Deephaven, MN / Cedar Rapids, IACmale01106.425Douglas, Mr. Walter Donald10PC 17761
4[null][null]B1142.0London / MiddlesexSmale0142.5Head, Mr. Christopher00113038
5[null][null][null][null][null]Smale0131.0Cairns, Mr. Alexander00113798
6[null]22[null]71.0Montevideo, UruguayCmale0149.5042Artagaveytia, Mr. Ramon00PC 17609
7[null]292C646.0Vancouver, BCCmale0175.2417McCaffry, Mr. Thomas Francis0013050
8[null][null]A2936.0New York, NYCfemale0131.6792Evans, Miss. Edith Corse00PC 17531
9[null][null]C8345.0New York, NYSmale0183.475Harris, Mr. Henry Birkhardt1036973
10[null][null]E5237.0Lakewood, NJCmale1183.1583Compton, Mr. Alexander Taylor Jr10PC 17756
11[null][null]T45.0Trenton, NJSmale0135.5Blackwell, Mr. Stephen Weart00113784
12[null]110B9440.0[null]Smale010.0Harrison, Mr. William00112059
13[null][null]B7131.0Montreal, PQSmale0152.0Davidson, Mr. Thornton10F.C. 12750
14[null][null]B8624.0[null]Cmale0179.2Giglio, Mr. Victor00PC 17593
15[null][null]C12337.0Scituate, MASmale0153.1Futrelle, Mr. Jacques Heath10113803
16[null][null]C8927.0Los Angeles, CACmale01136.7792Clark, Mr. Walter Miller1013508
17[null]38[null]42.0New York, NYSmale0152.0Holverson, Mr. Alexander Oskar10113789
18[null]126D629.0Springfield, MASmale0130.0Long, Mr. Milton Clyde00113501
19[null]175E4654.0Dorchester, MASmale0151.8625McCarthy, Mr. Timothy J0017463
2014[null][null][null]New York, NYCmale0130.6958Hoyt, Mr. William Fisher00PC 17600
Rows: 1-20 | Columns: 14

Let’s ingest only two columns from the json.

read_json(
    path = "titanic_subset.json",
    table_name = "titanic_sub_usecols",
    schema = "public",
    usecols  = [
        "fields.fare",
        "fields.sex",
    ],
)
123
boat
Integer
123
body
Integer
Abc
cabin
Varchar(20)
123
age
Numeric(10)
Abc
home.dest
Varchar(64)
Abc
embarked
Varchar(20)
Abc
sex
Varchar(20)
123
parch
Integer
123
pclass
Integer
123
fare
Numeric(13)
Abc
name
Varchar(64)
123
sibsp
Integer
123
survived
Integer
Abc
ticket
Varchar(20)
1[null][null]E46[null]Brighton, MASmale0151.8625Hilliard, Mr. Herbert Henry0017463
2[null][null][null]17.0Montevideo, UruguaySmale0147.1Carrau, Mr. Jose Pedro00113059
3[null]62C8650.0Deephaven, MN / Cedar Rapids, IACmale01106.425Douglas, Mr. Walter Donald10PC 17761
4[null][null]B1142.0London / MiddlesexSmale0142.5Head, Mr. Christopher00113038
5[null][null][null][null][null]Smale0131.0Cairns, Mr. Alexander00113798
6[null]22[null]71.0Montevideo, UruguayCmale0149.5042Artagaveytia, Mr. Ramon00PC 17609
7[null]292C646.0Vancouver, BCCmale0175.2417McCaffry, Mr. Thomas Francis0013050
8[null][null]A2936.0New York, NYCfemale0131.6792Evans, Miss. Edith Corse00PC 17531
9[null][null]C8345.0New York, NYSmale0183.475Harris, Mr. Henry Birkhardt1036973
10[null][null]E5237.0Lakewood, NJCmale1183.1583Compton, Mr. Alexander Taylor Jr10PC 17756
11[null][null]T45.0Trenton, NJSmale0135.5Blackwell, Mr. Stephen Weart00113784
12[null]110B9440.0[null]Smale010.0Harrison, Mr. William00112059
13[null][null]B7131.0Montreal, PQSmale0152.0Davidson, Mr. Thornton10F.C. 12750
14[null][null]B8624.0[null]Cmale0179.2Giglio, Mr. Victor00PC 17593
15[null][null]C12337.0Scituate, MASmale0153.1Futrelle, Mr. Jacques Heath10113803
16[null][null]C8927.0Los Angeles, CACmale01136.7792Clark, Mr. Walter Miller1013508
17[null]38[null]42.0New York, NYSmale0152.0Holverson, Mr. Alexander Oskar10113789
18[null]126D629.0Springfield, MASmale0130.0Long, Mr. Milton Clyde00113501
19[null]175E4654.0Dorchester, MASmale0151.8625McCarthy, Mr. Timothy J0017463
2014[null][null][null]New York, NYCmale0130.6958Hoyt, Mr. William Fisher00PC 17600
Rows: 1-20 | Columns: 14

Note

You can ingest multiple JSON files into the Vertica database by using the following syntax.

read_json(
    path = "*.json",
    table_name = "titanic_multi_files",
    schema = "public",
)

See also

read_file() : Ingests an input file into the Vertica DB.
read_avro() : Ingests a AVRO file into the Vertica DB.
read_csv() : Ingests a CSV file into the Vertica DB.
read_pandas() : Ingests the pandas.DataFrame into the Vertica DB.