Why is Vertica not Ingesting Data From Kafka?
Prerequisite: Verify that Vertica is up and running.
If you want to troubleshoot why Vertica is not ingesting data from Kafka, follow this checklist.
|1||Check whether Kafka is up and running.
On a broker node, use Kafka CLI producer and consumer to demonstrate that Kafka is working as expected.
|If Kafka is up and running, go to Step 2.
If Kafka is not up and running, start Kafka using zookeeper_start or kafka_server_start.
|2||Check whether Kafka produce and consume works from the Vertica nodes.
To confirm the Kafka connection settings are correct, run kafkacat produce and consume from each Vertica node, using exactly the same broker list that you used to configure the scheduler. The command checks whether the network and firewall work without causing issues.
|If Kafka produce and consume works with Vertica nodes, go to Step 3.
If Kafka produce and consume does not work with Vertica nodes, check your network configuration and consult the Kafka documentation.
|3||Perform an ad-hoc copy into a dummy table using the kafkaparser.
||If Vertica can load raw data from Kafka, o to Step 4.
If KafkaSource does not use the same configuration as kafkacat, the KafkaSource is the issue instead of the parser. Please contact Vertica Technical Support.
|4|| Check whether the parser is configured correctly.
||If the copy acts as expected, there are no data formatting problems, go to Step 5.
If the query errored out or rejected data, there is a data formatting problem. Please contact Vertica Technical Support.
|5||Check whether the scheduler is configured and running properly.
The scheduler requires a few inputs that are common to each subcommand. The scheduler stores the configuration information inside a Vertica database.
The scheduler uses Vertica schema to store configuration and runtime state.You can check the stream_lock configuration table and the stream_scheduler_history table to see if the scheduler is running. The scheduler should know the following:
If the scheduler is running properly, go to Step 6.
If the scheduler is not running, launch with vkconfig.
|6||Check whether the scheduler is making progress.
If you do not see any progress, there could be an issue with the COPY statement.
If no messages are loaded, the scheduler is not working as expected.
|If the scheduler is making progress, the checklist is complete.
If the scheduler is not making progress, please contact Vertica Technical Support.