Load Spec Tool Options

The vkconfig script's load spec tool lets you provide parameters for the COPY statement that loads streaming data.

Syntax

$ vkconfig load-spec {--create | --read | --update | --delete} [--load-spec spec_name] [other_options...]
Option Description
--create

Creates a new loadspec. Cannot be used with --delete, --read, or --update.
--read

Outputs the current settings of the load specs defined in the scheduler. This output is in JSON format. Cannot be used with --create, --delete, or --update.

By default, this option outputs all of the load specs defined in the scheduler. You can limit the output by supplying a single value or a comma-separated list of values to the --load-spec, --filters, --uds-kv-parameters, --parser, --load-method, --message-max-bytes, and --parser-parameters options. The vkconfig script only outputs the configuration of load specs that match the values you supply.

You can use LIKE wildcards in these options. See LIKE-predicate for more information about using wildcards.
--update

Updates an existing load-spec. Cannot be used with --create, --delete, or --read.
--delete

Deletes a load-spec. Cannot be used with --create, --read, or --update.
--load-specspec_name A unique name for copy load spec to operate on. This option is required for --create, --update, and --delete.
--dump

When you use this option along with the --read option, vkconfig outputs the Vertica query it would use to retrieve the data, rather than outputting the data itself. This option is useful if you want to access the data from within Vertica without having to go through vkconfig. This option has no effect if not used with --read.

--filters "filter_name" A Vertica FILTER chain containing all of the UDFilters to use in the COPY statement. For more information on filters, refer to Parsing Custom Formats.
--load-method AUTO|TRICKLE |DIRECT

The COPY load method to use for all loads with this scheduler. See the COPY statement for more information.

Default Value:

TRICKLE
--message-max-bytes max_message_size

Specifies the maximum size, in bytes, of a Kafka protocol batch message.

Default Value:

25165824

You may need to manually update this value if you created a scheduler using Vertica 9.1.0 or earlier. The meaning of Kafka's max.message.bytes setting changed between version 0.10 and 0.11. See Changes to the message.max.bytes Setting in Kafka Version 0.11 and Later for more information.
--new-load-spec new_name A new, unique name for an existing load spec. Requires the --update parameter.
--parser-parameters " key=value, key=value" A list of parameters to provide to the parser specified in the --parser parameter. When you use a Vertica native parser, the scheduler passes these parameters to the COPY statement where they are in turn passed to the parser.
--parser parser name

Identifies a Vertica UDParser to use with a specified target. This parser is used within the COPY statement that the scheduler runs to load data. If you are using a Vertica native parser, the values supplied to the --parser-parameters option are passed through to the COPY statement.

Default Value:

KafkaParser

--uds-kv-parameters key=value, key=value A comma separated list of key value pairs for the user-defined source.
--validation-type{ERROR|WARN|SKIP}

Controls the validation performed on a created or updated load spec:

  • ERROR - Cancel configuration or creation if vkconfig cannot validate the load spec. This is the default setting.
  • WARN - Proceed with task if validation fails, but display a warning.
  • SKIP - Perform no validation.

Renamed from --skip-validation.

See Common vkconfig Script Options for options that are available in all of the vkconfig tools.

Examples

These examples show how you can use the Load Spec utility options.

Create the load spec, Streamspec1:

$ /opt/vertica/packages/kafka/bin/vkconfig load-spec --create --load-spec Streamspec1 --conf myscheduler.conf

Update the load spec, Streamspec1, to the name Streamspec2, and also update the load method to Direct:

$ /opt/vertica/packages/kafka/bin/vkconfig load-spec --update --load-spec Streamspec1 \
                                                     --new-load-spec Streamspec2  --load-method direct \
                                                     --conf myscheduler.conf

Update the load spec, Filterspec, to use the KafkaInsertLengths filter and a custom decryption filter:

$ /opt/vertica/packages/kafka/bin/vkconfig load-spec --update --load-spec Filterspec \
                                                     --filters "KafkaInsertLengths() DecryptFilter(parameter=Key)" \
                                                     --conf myscheduler.conf

Read the current settings for the load spec named streamspec1:

$ vkconfig load-spec --read --load-spec streamspec1 --conf weblog.conf
{"load_spec":"streamspec1", "filters":null, "parser":"KafkaParser", 
"parser_parameters":null, "load_method":"TRICKLE", "message_max_bytes":null, 
"uds_kv_parameters":null}