Prerequisite: Verify that Vertica is up and running.
If you want to troubleshoot why Vertica is not ingesting data from Kafka, follow this checklist.
| Step | Task | Results |
|---|---|---|
| 1 | Check whether Kafka is up and running.
a.
$ BROKER_ROOT/logs/server.log
$ ZOOKEEPER_ROOT/logs/zookeeper.log
$ KAFKA_INSTALL/Kafka_server_stop.sh
$ KAFKA_INSTALL/Kafka_server_start.sh
$ KAFKA_INSTALL/zookeeper_server_stop.sh
$ KAFKA_INSTALL/zookeeper_server_start.shd.
|
If Kafka is up and running, go to Step 2.
If Kafka is not up and running, start Kafka using zookeeper_start or kafka_server_start.
|
| 2 | Check whether Kafka produce and consume works from the Vertica nodes. To confirm the Kafka connection settings are correct, run kafkacat produce and consume from each Vertica node, using exactly the same broker list that you used to configure the scheduler. The command checks whether the network and firewall work without causing issues. | If Kafka produce and consume works with Vertica nodes, go to Step 3. If Kafka produce and consume does not work with Vertica nodes, check your network configuration and consult the Kafka documentation. |
| 3 | Perform an ad-hoc copy into a dummy table using the kafkaparser.
=> CREATE TABLE test (x varchar(65000));
=> COPY test source KafkaSource(brokers='broker:9092,broker2:9092'duration=interval'10second') Parser KafkaParser() rejected data as table test_rejects;=> SELECT* FROM test;
The command shows raw message data.
=> SELECT* FROM test_rejects;
The command returns no value or character.
=> SELECT KafkaOffsets() over partition (best) FROM dual;
The command returns load statistics for each partition that you attempted to load from. |
If Vertica can load raw data from Kafka, o to Step 4. If KafkaSource does not use the same configuration as kafkacat, the KafkaSource is the issue instead of the parser. Please contact Vertica Technical Support. |
| 4 | Check whether the parser is configured correctly.
a.
|
If the copy acts as expected, there are no data formatting problems, go to Step 5. If the query errored out or rejected data, there is a data formatting problem. Please contact Vertica Technical Support. |
| 5 | Check whether the scheduler is configured and running properly.
The scheduler requires a few inputs that are common to each subcommand. The scheduler stores the configuration information inside a Vertica database.
The scheduler uses Vertica schema to store configuration and runtime state.You can check the stream_lock configuration table and the stream_scheduler_history table to see if the scheduler is running. The scheduler should know the following:
|
If the scheduler is running properly, go to Step 6.
If the scheduler is not running, launch with vkconfig.
|
| 6 | Check whether the scheduler is making progress.
a.
|
If the scheduler is making progress, the checklist is complete. If the scheduler is not making progress, please contact Vertica Technical Support. |
Learn More
Learn more about Apache Kafka in the Vertica Documentation. Also, see Apache Kafka documentation.About the Author
Soniya Shah
Information Developer
Currently, a first year law student with a background in science and technology. Experienced technical writer, with specializations in software documentation, big data, blog development, and website development. I build user-centered content to communicate complex and technical information more easily.
I used to work for Vertica full time for about 3 years. I still work at Vertica part time while going to law school.
Update: Soniya is now doing her law internship, and no longer working at Vertica. Good luck, Soniya!
