Vertica Blog

Vertica Blog

Kafka

Streaming Data in One Line!

Remember that game show, "Name That Tune", where contestants were challenged to name a tune in as few notes as possible? Today's tip converts that for Vertica Big Data, showing how we can ingest streaming data in just one line! A simple way to stream data is to write CSV rows to a network socket....

See You at IoT World Conference Next Month

Are you attending the IoT World Conference in Santa Clara from May 13-16th? Come by booth #1254 to hear why more and more companies, including medical device manufacturers, wearable product developers, network service providers, and manufacturing intelligence software companies, are turning to Vertica to manage their growing volumes of IoT data. Vertica is purpose-built for...
Commercial passenger plane with Vertica painted on the tail

Blog Post Series: Using Vertica to Track Commercial Aircraft in near Real-Time — Part 7

Preparing Vertica to consume messages from Kafka How time has flown (no pun intended!).  At long last getting back to continuing this project on tracking commercial aircraft… Having covered ADS-B, the Raspberry Pi, DUMP1090, Zookeeper and Kafka and then in the how to develop an Extract, Transform and Load (ETL) application to present a continuous...

Why is Vertica not Ingesting Data From Kafka?

Prerequisite: Verify that Vertica is up and running. If you want to troubleshoot why Vertica is not ingesting data from Kafka, follow this checklist. Step Task Results 1 Check whether Kafka is up and running. a. Examine the server log files for broker errors: If there are errors, consult the Kafka documentation. b. Examine the...

Handling Duplicate Records in Input Data Streams

This blog post was authored by Ravi Gupta. We have often found that sources or operational systems that provide data for further analysis have duplicate records and these are sent to a downstream application or EDW for processing. This post shows a few scenarios of how to handle these duplicate records using various SQL options,...
Commercial passenger plane with Vertica painted on the tail

Vertica Wins First Place for Edge-Based Analytics at University of Central Arkansas Raspberry Pi Bakeoff

We had the opportunity to enter our Vertica flight tracking demo at the University of Central Arkansas’s university raspberry pi bakeoff. We won first place, competing against a number of other raspberry pi-based projects! For more information on how to use Vertica to track commercial aircraft in near real time, read Mark Whalley’s blog series...
Commercial passenger plane with Vertica painted on the tail

Blog Post Series: Using Vertica to Track Commercial Aircraft in near Real-Time – Part 6

Part 6: Extract, Transform and Load ADS-B messages into Kafka I have discussed in previous blog posts the continuous stream of messages from aircraft transponders, captured and decoded using the DUMP1090 application, which we are planning on feeding into a series of Kafka topics, prior to loading into their corresponding tables in a Vertica database....

Introducing the Parallel Streaming Transformation Loader (PSTL) Solution

This blog post was authored by Soniya Shah. At Vertica, we understand how important it is that our customers can make decisions in near real time. Being able to do this not only requires the massive parallel processing that Vertica offers, but the ability to transform and ingest your data into Vertica as quickly as...
Commercial passenger plane with Vertica painted on the tail

Blog Post Series: Using Vertica to Track Commercial Aircraft in near Real-Time — Part 5

Part Five: Apache Kafka As discussed in an , we have a continuous stream of messages from aircraft transponders captured and decoded using the DUMP1090 application. We now want to load this data into a series of Vertica tables. As with the streaming nature of the incoming data, with many 10s, possibly 100s of messages...

The Kafka Streaming Load Scheduler

This blog post was authored by Tom Wall. Vertica’s streaming load scheduler provides high-performance streaming data load from Kafka into your Vertica database. Whether you already use Kafka or not, it is worth considering it as a solution to your data loading challenges. Kafka complements Vertica very nicely, and the scheduler removes many complexities of...
Commercial passenger plane with Vertica painted on the tail

Blog Post Series: Using Vertica to Track Commercial Aircraft in near Real-Time — Part 4

Part Four: Installing DUMP1090 to decode ADS-B messages The detailed the hardware requirements used in this project for tracking commercial aircraft in near real-time. In this blog post I will touch on installing the operating system on the Raspberry Pi (RPI) and the DUMP1090 software used for decoding the ADS-B signals being received by the...
Commercial passenger plane with Vertica painted on the tail

Blog Post Series: Using Vertica to Track Commercial Aircraft in near Real-Time — Part 3

Building a Raspberry Pi to Capture ADS-B signals Picking apples, pears, blackberries or raspberries In a , I provided a very high-level overview of ADS-B, and that with the appropriate pieces of hardware and some open-source software, it was possible to capture and decode the radio signals being broadcast from commercial aircraft, with the view...