Vertica

Archive for the ‘HP’ Category

Building Bridges with Gumdrops and Toothpicks

soniyaBlogPic2

In June 2014, the HP Vertica summer interns headed to the East End House in East Cambridge, MA to work with students through a community service project. Sarah Perkins, a business planner on the Project Management team, organized the project. Since 1875, the East End House has offered innovative programs to the community and continues to strive for excellence. Their programs help support families and individuals through curricula that enhance education standards. Programming supports the whole family with comprehensive services like a Food Pantry, Community Workshops, Parent Education and Senior Programming.

The interns, accompanied by mentors Sarah Lemaire and Jaimin Dave, helped students build bridges with very limited materials: fifty gumdrops and one hundred toothpicks! The goal was to build a bridge that spanned a six-inch gap and would hold at least 300 grams, or 120 pennies.

Teams of interns paired up to work with four students, ranging from third to eighth grade. They watched and assisted as the students discussed strategies, drew prototypes and started to build and re-build their structures. As the students worked on their bridges, they got to know more about the Vertica interns, including their majors, hometowns, and the projects they are working on for the summer. Throughout the course of the day, if students could correctly answer questions about their interns, they would win HP Vertica swag, including toy bulldozers, stress balls, flashlights, and more.

Once the bridges were built, the interns and students tested them across a six-inch gap. Students placed pennies on a paper plate on top of the bridge, one-by-one, until the bridge collapsed under the weight. The winning team’s bridge, led by interns Swikriti Jain and Jun Yin, held 255 pennies, which weigh more than 1 1/3 pounds! The top two teams won a bundle of HP Vertica swag, including t-shirts, water bottles, and baseballs.

The HP Vertica interns had a great time learning about students at the East End House, and helping them build successful bridges. It was a unique opportunity to interact with students of many ages, while also encouraging them to remain active in school and participate in extracurricular activities.

The Single-Solution Elf

The “De-mythification” Series

Part 3: The Single-Solution Elf

In this part of the de-mythification series, I’ll address another common misconception in the big data marketplace: that there exists a single piece of technology that will solve all big data problems. Whereas the first two entries in this series focused on market needs, this will focus more on the vendor side of things in terms of how big data has driven technology development, and give some practical guidance on how an organization can better align their needs with their technology purchases.

Big Data is the Tail Wagging the Vendor

Big data is in the process of flipping certain technology markets upside-down. Ten or so years ago, vendors of databases, ETL, data analysis, etc. all could focus on building tools and technologies for discrete needs, with an evolutionary eye – focused on incremental advance and improvement. That’s all changed very quickly as the world has become much more instrumented. Smartphones are a great example. Pre-smartphone, the data stream from an individual throughout the day might consist of a handful of call-detail records and a few phone status records. Maybe a few kilobytes of data at most. The smartphone changed that. Today a smartphone user may generate megabytes, or even gigabytes of data in a single day from the phone, the broadband, the OS, email, applications, etc. Multiply that across a variety of devices, instruments, applications and systems, and the result is a slice of what we commonly refer to as “Big Data”.

Most of the commentary on big data has focused on the impact to organizations. But vendors have been, in many cases, blindsided. With technology designed for orders of magnitude less data, sales teams accustomed to competing against a short list of well-established competitors, marketing messages focused on clearly identified use cases, and product pricing and packaging oriented towards a mature, slow-growth market, many have struggled to adapt and keep up.

Vendors have responded with updated product taglines (and product packaging) which often read like this:

“End-to-end package for big data storage, acquisition and analysis”

“A single platform for all your big data needs”

“Store and analyze everything”

Don’t these sound great?

But simple messages like these mask the reality that there are distinct activities that which comprise big data analytics, and that these activities come with different technology requirements, and much of today’s technology was born in a very different time – so the likelihood of there being a single tool that does everything well is quite low. Let’s start with the analytic lifecycle, depicted in the figure below, and discuss the ways this has driven the state of the technology.

analytic_lifecycle

This depicts the various phases of an analytic lifecycle from the creation and acquisition of data through the exploration and structuring to analysis and modeling, to putting the information to work. These phases often require very different things from technology. Let’s take the example of acquiring and storing of large volumes of data with varying structure. Batch performance is often important here, as is cost to scale. Somewhat less important is ease of use – load jobs tend to change at a lower rate than user queries, especially when the data in a document-like format (e.g. JSON). By contrast, the development of a predictive model requires a highly interactive technology which combines high performance with a rich analytic toolkit. So batch use will be minimal, while ease of use is key.

Historically, many of the technologies required for big data analytics were built as stand-alone technologies: a database, a data mining tool, an ETL tool, etc. Because of this lineage, the time and effort required to re-engineer these tools to work effectively together as a single technology, with orders of magnitude more data, can be significant.

Despite how a vendor packages technology, organizations must ask themselves this question: what do you really need to solve the business problems? When it comes time to start identifying a technology portfolio to address big data challenges, I always recommend that customers start by putting things in terms of what they really need. This is surprisingly uncommon, because many organizations have grown accustomed to vendor messaging which is focused on what the vendor wants to sell as opposed as to what the customer needs to buy. It may seem like a subtle distinction, but it can make all the difference between a successful project and a very expensive set of technology sitting on the shelf unused.

I recommend engaging in a thoughtful dialog with vendors to assess not only what you need today, but to explore things you might find helpful which you haven’t thought of yet. A good vendor will help you in this process. As part of this exercise, it’s important to avoid getting hung up on the notion that there’s one single piece of technology that will solve all your problems: the single solution elf.

Once my colleagues and I dispel the single solution myth, we can then have a meaningful dialog with an organization and focus on the real goal: finding the best way to solve their problems with a technology portfolio which is sustainable and agile.

I’ve been asked, more than once “Why can’t there be a single solution? Things would be so much easier that way.” That’s a great question, which I’ll address in my next blog post as I discuss some common sense perspectives on what technology should – and shouldn’t – do for you.

Next up: The Automagic Pixie

 

Work hard, have fun and make a difference!

JaminBlogPic

My name is Jaimin and I work as a Software Engineer in the Distributed Query Optimizer Team at HP Vertica. I wanted to share with you what I think makes Vertica the best place to work! I will explain the kind of impact you can make as an employee/intern at HP Vertica, while sharing my personal experiences.

As a student, I researched many companies I might want to work for to get a better understanding of the everyday life of software engineers. However, what I was most interested in learning about was the kinds of things engineers might do that went above and beyond the normal day-to-day stuff.

Is writing code something unique to the job?

No! Right?

As Software Engineers, we write code, develop algorithms, and implement them. But here at HP Vertica, we do lots of other things besides simply writing code.

Go above and beyond!

Vertica is different from other companies as far as normal day-to-day stuff goes.

Let me ask you this question: How many new graduates would you guess could get a chance to file a patent within their first 6 months of joining a company? How many would get chance to write a paper within first six months? Not a lot, right?

In my experience at HP Vertica, I’ve seen that just about all new graduate engineers file at least one patent in their first year at work. This speaks to the fact that the work we do here at Vertica is completely innovative. Our projects have a huge business impact.

Be the captain of your ship!

Vertica offers engineers incredible opportunities! All you have to do is be willing to accept them. One of the best things about HP Vertica is that you work in an environment where other engineers are smarter than you! You’ll find yourself constantly challenged to learn new, interesting, and exciting things. You’ll get better exposure and, more importantly, you have a massive role to play in the company’s growth and development.

Something else that’s unique about HP Vertica—the projects you work on as an intern become part of the shipping product! As a result, you’ll get the chance to see your code in action and sometimes you can learn what customers have to say about your feature in particular. You won’t be allowed to sit idle for a minute because we have a very short release cycle. This will keep you on your toes and encourage you to think something new day in and day out.

Here, engineers are not forced to work on this and that—they have a great deal of autonomy and frequently get to choose the things they work on. If you have an idea you think can help improve the product, you are encouraged to see it through. And, you’ll also get a chance to participate in various technical events that take place within HP and submit your ideas.

Taking initiative is always encouraged and you’ll be expected to make, discuss, and defend your design decisions with your mentors instead of just following directions. You’ll also be able to learn about the complexities of building a database and how we achieve the performance advantages in HP Vertica.

It is also easy to move between the teams. It is entirely up to you and the only question is what you want to do?

Share and gain knowledge!

Knowledge Sharing is another important thing at Vertica. We do a lunch talk where we discuss any new paper related to database systems. Every now and then people from various teams give tech talks so that each team is aware of what people in other groups are doing.

As a fresh graduate before joining Vertica, I did not have any experience working on a database optimizer product, though I had worked a bit on optimizations when I took a compiler class. Because of the great culture and environment at Vertica, I didn’t find the transition difficult at all. Sometimes it was challenging, but it allowed me to learn a lot by working with incredibly smart people at the company while working on challenging projects (I wonder how many people have the opportunity to work on the design and implementation of queries involving Set Operators during their first year of work).

Have fun!

We frequently unwind doing fun things at work, including watching the Olympics games or other sporting events during lunch, or playing table-tennis and board games when we can. Vertica provides a lot of flexibility and it comes with huge responsibility. You’re expected to get your work done on time—if you do that, no one will have any problem with having a little fun. Interns also go on outdoor field trips, including horseback riding, hiking to Blue Hills, going for a movie, participating in a bocce tournament, and water activities such as motor boat racing. Once, we went to the Boston Harbor and tried to learn how to sail a boat from one of our in-house experts in Vertica.

We are looking for people to join Vertica! Do you have any interest in being challenged in an innovative design environment? Then apply today!

Enter the Flex Zone – Modernizing the Enterprise Data Warehouse

I’ve had the privilege of attending the Data Warehouse Institute’s (TDWI) conference this week. The Las Vegas show is usually one of their biggest gatherings. This year, there were about 600 of us gathered together to talk about the latest and greatest in the data warehouse and business intelligence world. HP Vertica was a sponsor.
The latest buzz was around many of the new data discovery tools that were announced by some vendors. Vendors recognize that there is a significant amount of undiscovered data in most businesses. As data warehouse teams go merrily along delivering daily analytics, piles and piles of dark data builds within that might have value. To innovate, users are recognizing that some of this unexplored data could be quite valuable, and it’s spurring on the development of a new breed of data discovery tools that enable users to develop new views of structured, semi-structured, and unstructured data.

Of course, this is the very reason that we have developed HP Vertica Flex Zone. The ability to ingest semi-structured data and use current visualization tools are one of the key tenets of HP Vertica Flex Zone. With HP Vertica Flex Zone, you can leverage your existing business intelligence (BI) and visualization tools to visually explore and draw conclusions from data patterns across a full spectrum of structured and semi-structured data. Analysts, data scientists, and business users can now explore and visualize information without burdening or waiting for your IT organizations to use lengthy and costly ETL tools and processes typical with legacy databases and data warehouses.
Most agreed that special data discovery tools should converge with standard analytical platforms in the coming months. Discovery should be as much a part of your business as daily analytics.

There were some first-rate executive sessions led by Fern Halper and Philip Russom, who talked about the transformation of analytics over the years. Analytics has become more mainstream, more understood by the masses of business users. Therefore innovation comes when we can deliver business intelligence for this new generation of information consumers.

The panel discussions and sessions focused very much on business value and put forth a call-to-action for some. Innovate. Feed the business users needs for information that will help drive revenue, improve efficiency, and achieve compliance with regulations. It was clear that data warehouse must be modernized of data warehouse (and that is happening today). Data warehouse pros aren’t satisfied with daily static analytics that they delivered in the past. They are looking for new data sources, including big data, and new-age data analytic platforms to help achieve their business goals.

Get started modernizing your enterprise data warehouse – evaluate HP Vertica 7 today.

Social Media Insights with HP Vertica Pulse

Get insight into what your customers are saying about your products and services with HP Vertica Pulse, one of the new innovations available on the recently announced HP Vertica Marketplace.

HP Vertica Pulse allows you to analyze Tweets automatically, scalably, and in-database so that you can:

  • understand the top-of-mind topics for your community and the sentiment associated with those topics.
  • combine results with other business data already stored in Vertica
  • identify advocates and detractors
  • analyze how sentiment changes over time
  • view high-level aggregate results as well as low-level comments
  • customize results for your domain.

 

Below are some visual examples of how HP Vertica Pulse can help you to make meaning of your data.  In this case, we used our free Social Media Connector on Github to collect and automatically load Tweets about some NFL teams over one weekend in November.  After running Pulse on the data, we used Tableau to visualize.  The visual on the left gives an aggregate view of the most popular topics.  Although we collected Tweets on football teams, a number of other topics appear, like “Tom Brady”, “Cam Newton”, and “call”.  The size of the circle reflects the volume of the tweets, while the color indicates sentiment (green for positive and red for negative).  As it turns out, the Patriots, 49ers, Jets, and Chiefs all lost that weekend.

Bubbles

The next visual shows an excerpt of a drill-down into one of the compelling results, namely the negative sentiment on ‘call’. This extract shows that there was a controversial call in the New England-Carolina game.

Bubble list

By using Pulse, and without actually watching the games, we were able to get an understanding of the most talked-about topics.  We could drill-down into the most compelling results to get an understanding of why the sentiment of the topic is negative, and take action if needed.  In the case of products and services, you could similarly see the aspects that mattered most to your community.  Negative sentiment could highlight a problem that needs attention.

You can try HP Vertica Pulse on tweets that impact your business.  To download, visit the HP Vertica Community Marketplace, and look for HP Vertica Pulse on the Innovations tab.

The HP Vertica Pulse download contains the following resources:

  • Installation files and documentation
  • three instructional videos
  • A quick start guide,
  • A Tableau template

These resources will help you to evaluate HP Vertica Pulse and make meaning of your Tweets as quickly and easily as possible.  When you create your Marketplace account, please review the Terms and Conditions, as they contain important information about the Innovations efforts.

We want to hear from you!  Post your questions, comments, and feedback to the HP Vertica Community.  Would you use HP Vertica Pulse to make an impact on your business?  What changes would you like to see?  We look forward to your feedback!

Boom Times for Boston’s Biggest Data

It’s boom times at HP Vertica – as the Boston area’s first and biggest Big Data technology provider, we continue to grow and expand our employee base and ecosystem.

We had a full house last week for our spring Open House, which gave us a chance to have friends of the company including current (of course) and future employees, strategic partners and even members of our alumni network.

As with any growth business in a highly dynamic market, occasionally people decide to move on but it was gratifying to have a number of former colleagues join us – some of whom have become customers, and a few of which have even re-joined the company recently!

Over the next few weeks, HP Vertica will be taking things on the road to HP Discover Las Vegas and will be involved in some major strategic announcements regarding HP’s Big Data strategy, so watch this space for more.

We’re also going to be on the road for our Discover Performance series and a number of industry events – check our website for details.

And last but not at all least, we’ve got our HP Vertica Big Data Conference in early August – click here or on the image below to learn more and to register. We’re expecting another full house including our base of current and future customers, strategic partners and of course many of our colleagues from worldwide HP – sign up now, since space is limited and Early Bird Pricing expires June 28!

Register here

Taking a Moonshot at Big Data Analytics for Everyone

HP Vertica is very excited about Monday’s announcement of the HP Moonshot system.

Why? Because we believe that the combination of the HP Vertica Analytics Platform running on the HP Moonshot Servers offers a truly game-changing value proposition for a variety of customers, and new segments of the market.

Moonshot is, simply put, a groundbreaking system which offers customers the ability to rapidly deploy, scale and manage with dramatically lower space and energy constraints. While traditional IT services that support business functions will continue to be served by general purpose server infrastructure, a new computing platform is required for specialized workloads that can deliver innovative solutions to market at unprecedented speed and scale.

 

We’ve already successfully tested the HP Vertica Analytics Platform on HP Moonshot Servers, and achieved very comparable performance to traditional Big Data Analytics hardware across certain performance ranges, which for a large segment of the market is more than sufficient to handle their Big Data Analytics loads – while offering very significant potential cost, space and energy savings.

Running Vertica on Moonshot offers yet another proof point of the unmatched value provided by HP’s combination of Information Optimization solutions, and a great example of the opportunity created by innovation that makes us so excited to be a part of the greater OneHP.

To learn more about HP Project Moonshot, visit http://www.hp.com/go/moonshot

Get Started With Vertica Today

Subscribe to Vertica