Apache Kafka Integration
Vertica version 8.1.1 includes the following changes to its integration with Apache Kafka.
Setting a Starting Offset for a Microbatch
Previously, a microbatch always started loading data from the start of a stream. You can now use the --offset
parameter to set an offset in the stream where the microbatch will start loading data . This new parameter lets you have the microbatch skip older data in a stream or to force it reload old data. See Microbatch Utility Options for details.
Setting an Ending Offset for KafkaSource
Previously, KafkaSource could end streaming either after a timeout or on reaching the end of a stream. You can now have it end streaming when it reaches a specific offset in the stream. See Using COPY with Data Streaming for more information.