Kafka

Bryan Herger, Vertica Big Data Solution Architect at Micro Focus
July 31, 2019

Remember that game show, "Name That Tune", where contestants were challenged to name a tune in as few notes as possible? Today's tip converts that for Vertica Big Data, showing how we can ingest streaming data in just one line! A simple way to stream data is to write CSV rows to a network socket....

See You at IoT World Conference Next Month

Ben Smith, Senior Product Marketing Manager
April 25, 2019

Are you attending the IoT World Conference in Santa Clara from May 13-16th? Come by booth #1254 to hear why more and more companies, including medical device manufacturers, wearable product developers, network service providers, and manufacturing intelligence software companies, are turning to Vertica to manage their growing volumes of IoT data. Vertica is purpose-built for...

Commercial passenger plane with Vertica painted on the tail

Blog Post Series: Using Vertica to Track Commercial Aircraft in near Real-Time — Part 7

Mark Whalley, Corporate Systems Engineer
April 4, 2019

Preparing Vertica to consume messages from Kafka How time has flown (no pun intended!). At long last getting back to continuing this project on tracking commercial aircraft… Having covered ADS-B, the Raspberry Pi, DUMP1090, Zookeeper and Kafka and then in the how to develop an Extract, Transform and Load (ETL) application to present a continuous...

Why is Vertica not Ingesting Data From Kafka?

Soniya Shah, Information Developer
June 14, 2018

Prerequisite: Verify that Vertica is up and running. If you want to troubleshoot why Vertica is not ingesting data from Kafka, follow this checklist. Step Task Results 1 Check whether Kafka is up and running. a. Examine the server log files for broker errors: If there are errors, consult the Kafka documentation. b. Examine the...

Handling Duplicate Records in Input Data Streams

Soniya Shah, Information Developer
April 17, 2018

This blog post was authored by Ravi Gupta. We have often found that sources or operational systems that provide data for further analysis have duplicate records and these are sent to a downstream application or EDW for processing. This post shows a few scenarios of how to handle these duplicate records using various SQL options,...

Vertica Wins First Place for Edge-Based Analytics at University of Central Arkansas Raspberry Pi Bakeoff

Tim Donar
April 2, 2018

We had the opportunity to enter our Vertica flight tracking demo at the University of Central Arkansas’s university raspberry pi bakeoff. We won first place, competing against a number of other raspberry pi-based projects! For more information on how to use Vertica to track commercial aircraft in near real time, read Mark Whalley’s blog series...

Blog Post Series: Using Vertica to Track Commercial Aircraft in near Real-Time – Part 6

Mark Whalley, Corporate Systems Engineer
March 27, 2018

Part 6: Extract, Transform and Load ADS-B messages into Kafka I have discussed in previous blog posts the continuous stream of messages from aircraft transponders, captured and decoded using the DUMP1090 application, which we are planning on feeding into a series of Kafka topics, prior to loading into their corresponding tables in a Vertica database....

Introducing the Parallel Streaming Transformation Loader (PSTL) Solution

Soniya Shah, Information Developer
March 19, 2018

This blog post was authored by Soniya Shah. At Vertica, we understand how important it is that our customers can make decisions in near real time. Being able to do this not only requires the massive parallel processing that Vertica offers, but the ability to transform and ingest your data into Vertica as quickly as...

Blog Post Series: Using Vertica to Track Commercial Aircraft in near Real-Time — Part 5

Mark Whalley, Corporate Systems Engineer
March 8, 2018

Part Five: Apache Kafka As discussed in an , we have a continuous stream of messages from aircraft transponders captured and decoded using the DUMP1090 application. We now want to load this data into a series of Vertica tables. As with the streaming nature of the incoming data, with many 10s, possibly 100s of messages...

The Kafka Streaming Load Scheduler

Soniya Shah, Information Developer
March 1, 2018

This blog post was authored by Tom Wall. Vertica’s streaming load scheduler provides high-performance streaming data load from Kafka into your Vertica database. Whether you already use Kafka or not, it is worth considering it as a solution to your data loading challenges. Kafka complements Vertica very nicely, and the scheduler removes many complexities of...

Blog Post Series: Using Vertica to Track Commercial Aircraft in near Real-Time — Part 4

Mark Whalley, Corporate Systems Engineer
February 13, 2018

Part Four: Installing DUMP1090 to decode ADS-B messages The detailed the hardware requirements used in this project for tracking commercial aircraft in near real-time. In this blog post I will touch on installing the operating system on the Raspberry Pi (RPI) and the DUMP1090 software used for decoding the ADS-B signals being received by the...

Blog Post Series: Using Vertica to Track Commercial Aircraft in near Real-Time — Part 3

Mark Whalley, Corporate Systems Engineer
December 5, 2017

Building a Raspberry Pi to Capture ADS-B signals Picking apples, pears, blackberries or raspberries In a , I provided a very high-level overview of ADS-B, and that with the appropriate pieces of hardware and some open-source software, it was possible to capture and decode the radio signals being broadcast from commercial aircraft, with the view...

Vertica Blog