Apache Kafka seems to be everywhere these days. I’m feeling a bit surrounded by Kafka right now, in fact, like I’m at the center of a convergence of the planets, and data flowing through Kafka is the gravitational pull. Let me give you an idea of what I mean.
So, Vertica just announced the release of the new version 9.3.1, and it has a nice list of new features. One of the features is the addition of the ability to parse incoming Avro messages in a Kafka topic that were compressed with the deflate codec. (Read about all the new features here.)
Every year, I make a point to go to one of my favorite conferences, Data Day Texas in Austin. Coming up on Jan 25, I’m teaching a short presentation on Architecting Production IoT Analytics at Data Day Texas. Jesse Anderson will be there. He’s been a past presenter who I enjoyed hanging out with, and he edited the upcoming from O’Reilly “97 Things Every Data Engineer Should Know,” which I contributed to. He is the Managing Director of the Big Data Institute, a cool place to learn many things related to data pipelines, architecture, and data engineering. Guess what he’ll be doing at Data Day? Teaching an entire, full day course on Jan 24 on … three guesses … yeah, it’s Kafka.
One of the big points in my Architecting Production IoT Analytics talk is that I looked at several of the architectural choices made by Vertica customers who had to analyze a lot of data very quickly. And I found patterns. The patterns didn’t depend on use case or industry. They were general patterns in dealing with mountains of data quickly and efficiently to get real production value from it. Reducing the gap in time between receiving a message and being able to query it is always a goal.
Want to guess what one of the technologies consistently chosen in virtually every IoT architecture is?
Yeah, that one was kind of obvious. It’s Kafka.
Everyone needs fast pipelines that scale if they’re dealing with a lot of data. It’s the most common technology choice made in a remarkably common architectural structure pattern. Of course, being Vertica customers, there is one other choice they all made, but that goes without saying.
Come check out my talk at Data Day Texas if you get the chance,. and if you do, see if you can come a day early. Jesse’s all-day practical guide to using Kafka is bound to be worth the extra day. I’ll be there, that’s for sure.
And if you’re just not a Texas kind of person, I’ll be doing the same Architecting Production IoT Analytics talk at Strata San Jose in March, and at Women in Analytics in Columbus in June. I’d love to see you there!
Something tells me there will also be a few talks on Kafka. The planets are aligned.
Related Posts:
Datadog Announces Integration with Vertica to Monitor Performance and Availability
The Kafka Streaming Load Scheduler
The Vertica Academy Is Open for Learning
Blog Post Series: Using Vertica to Track Commercial Aircraft in near Real Time — Part 1
Data Day Texas: Keep Your Architecture Open and Avoid Mindset Lock-in
Related Pages:
Internet of Things Analytics
Open Source Innovation
About the Author
Paige Roberts
Vertica Open Source Relations Manager
In over two decades in the data management industry, I have worked as an engineer, a trainer, a marketer, a product manager, and a consultant. Contributor to O'Reilly's 97 Things Every Data Engineer Should Know. Now, I promote understanding of Vertica, MPP data processing, open source, high scale data engineering, and how the analytics revolution is changing the world.
