Kafka and Vertica Configuration Settings
The following sections lists settings for Vertica and Kafka that you can set to optimize performance.
Vertica Producer Settings
These settings change how Vertica acts as a Kafka producer when using either the KafkaExport function or through notifiers. How you change these settings depends on the method you are using to export data (KafkaExport or Notifiers).
| Setting | Notes |
|---|---|
| queue.buffering.max.messages | Specifies the size of the Vertica producer queue. If Vertica generates too many messages too quickly, the queue can fill, resulting in dropped messages. Increasing this value consumes more memory, but reduces the chance of lost messages. |
| queue.buffering.max.ms | Specifies the frequency with which Vertica flushes the producer message queue. Lower values decrease latency at the cost of throughput. Higher values increase throughput, but can cause the producer queue (set by queue.buffering.max.messages) to fill more frequently, resulting in dropped messages. |
| message.max.bytes | Specifies the maximum size of a Kafka protocol request message batch. This values should be the same on your sources, brokers, and producers. |
| message.send.max.retries | Specifies the number of attempts the producer makes to deliver the message to a broker. Higher values increase the chance of success. |
| retry.backoff.ms | Specifies the interval Vertica waits before resending a failed message. |
| request.required.acks | Specifies how many broker replica acknowledgments Kafka requires before it considers message delivery successful. Requiring acknowledgments increases latency. Removing acknowledgments increases the risk of message loss. |
| request.timeout.ms | Specifies the interval that the producer waits for a response from the broker. Broker response time is affected by server load and the number of message acknowledgments you require. Higher values increase latency. |
| compression.type |
Specifies the compression algorithm used to encode data before sending it to a broker. Compression helps to reduce the network footprint of your Vertica producers and increase disk utilization. Vertica supports gzip and snappy. |
Kafka Broker Settings
Kafka brokers receive messages from producers and distribute them among Kafka consumers. Configure these settings on the brokers themselves. These settings function independently of your producer and consumer settings. For detailed information on Apache Kafka broker settings, refer to the Apache Kafka documentation.
| Setting | Notes |
|---|---|
| message.max.bytes | Specifies the maximum size of a Kafka protocol request message batch. This values should be the same on your sources, brokers, and producers. |
| num.io.threads | Specifies the number of network threads the broker uses to receive and process requests. More threads can increase your concurrency. |
|
num.network.threads |
Specifies the number of network threads the broker uses to accept network requests. More threads can increase your concurrency. |
Vertica Consumer Settings
The following settings changes how Vertica acts when it consumes data from Kafka. You can set this value using the kafka_conf parameter on the KafkaSource UDL when directly executing a COPY statement. For schedulers, use the --message_max_bytes settings in the scheduler tool.
| Setting | Notes |
|---|---|
| message.max.bytes |
Specifies the maximum size of a Kafka protocol request message batch. Set this value to a high enough value to prevent the overhead of fetching batches of messages interfering with loading data. Defaults to 24MB for newly-created load specs. The meaning of this setting changed between Kafka version 0.10 and 0.11. If you have a scheduler you created using Vertica version 9.1.0 or earlier, see Changes to the message.max.bytes Setting in Kafka Version 0.11 and Later for settings you may need to change. |