Archive for the ‘big data’ Category

HP Discover: Introducing HP Vertica OnDemand


Yesterday the sun rose over the beautiful city of Barcelona, and with it, HP Discover. Back in Cambridge MA the excitement is just as high as we finally get to reveal to the world what we’ve all been tirelessly working on. We have some great announcements for you from the HP Vertica platform, one of which is the exciting launch of HP Vertica OnDemand.

So let’s say you’ve got a startup with an awesome product , but you still have a pretty tight budget. We understand, we’ve been there. Small organizations like yours face challenges in Big Data and analytics every day. But now there’s good news—HP Vertica OnDemand!

HP Vertica OnDemand, provides a solution that allows for flexible scaling and pricing, and also provides the same speed and performance our customers have come to expect from our Enterprise Edition of HP Vertica Analytics Platform.

From the start, we designed HP Vertica OnDemand to be easy to set up, hassle-free, and fast. You can get set up in under 30 minutes and be running complex SQL queries in no time, performing geopatial and predictive analytics, advanced timeseries, and much, much more.

Finally there’s the price. Budgets can be tight, and we wanted to provide a service that could fit an organization’s needs without costing an arm and a leg. Starting at $999 a month, HP Vertica OnDemand is the perfect solution to start small, and scale easily to whatever your needs might be in the future. Interested? You can learn more here

You can also follow along with all of the action in Barcelona with the HP DISCOVER Insider

HP Women at Grace Hopper

A couple weeks ago, the Anita Borg: Grace Hopper Conference for Women in Technology took place in Phoenix, Arizona, from October 8 -11, 2014. A Platinum Sponsor of the event, HP sent many women from across the organization to attend the conference.

HP Vertica had a significant presence at the event. Bo Hyun Kim, an HP Vertica developer, was chosen to present a technical paper she authored on Sentiment Analysis and Big Data. Shilpa Lawande, now General Manager of HP Software’s Big Data Platform Business unit, spoke at a breakfast given for technical women at HP. Seven of our own HP Vertica women attended the conference with the purpose of recruiting and screening talented candidates for the growing HP Vertica team.

Bo Hyun’s Presentation on Sentiment Analysis

In March 2014, Shilpa Lawande reached out to the technical women of HP Vertica, encouraging volunteers to submit session proposals for the conference.

Bo Hyun Kim, of the HP Vertica Management Console team, accepted the challenge. She collaborated with Lina Chen in authoring a paper called “Lexicon-Based Sentiment Analysis Using the Most-Mentioned Word Tree.” After several months of waiting anxiously, Bo Hyun learned that her paper was accepted!

Bo Hyun

Bo Hyun’s presentation became part of a larger Data Science in Social Media Analysis presentation. On Friday, October 10, she presented to a room packed with technical women of all ages and backgrounds.

Bo Hyun rocked—she handled the presentation with good humor, poise, and confidence. The presentation was held in one of the large ball rooms, and Bo Hyun prepared herself well before she stood in front of hundreds of bright, technical women who came from all over the world. She engaged with the audience right away by stepping down from the podium and asking them questions related to her research work, which was enhancing the sentiment analysis analytic package by HP Vertica. She had proposed to improve the performance of HP Vertica Pulse earlier this year, and the proposal was accepted. With many nights of hard work, Bo Hyun created a presentation that was both informative and educational. Bo Hyun made sure the audience members followed and understood each slide.

After the presentation ended, audience members asked her various questions about her work on sentiment analysis. Most were students, seeking advice and guidance in knowing more about sentiment analysis for their senior thesis research projects. Students also asked about the skill set required to work at a software engineering company. Bo Hyun did not have enough time to answer all their questions, so she handed out her business card so that the students could contact her later and ask as many questions as they wanted. On her way out, Bo Hyun was stopped by the Panel, who congratulated her on the talk. After the conference, Bo Hyun received many emails from the eager, enthusiastic students, who completed the presentation questionnaire, and she was happy to be able to help them further.

Bo Hyun Presentation


HP VertiGals at Bo Hyun’s talk.

HP Technical Women’s Breakfast

On Thursday, 10/9, conference attendees from HP were invited to a technical women’s breakfast. Shilpa Lawande, General Manager of HP Software’s Big Data Platform business unit, was the key speaker at the breakfast. Shilpa shared stories of her own journey as a woman in the male-dominated field of computer science. She spoke of the experience of being “the only woman in the room,” spanning from her undergraduate education in India, as an individual contributor at Oracle and Vertica, to her leadership roles as VP of engineering and, most recently General Manager of HP Software’s Big Data Platform business unit. Among the questions she answered for the attendees, Shilpa shared that her own personal role model is her mother, who raised her family while pursuing a career, and always met every challenge with a smile.


Shilpa Lawande speaking at the HP Technical Women’s breakfast


Lina Chen, Beth Favini, Dan Huang, Bo Hyun Kim, Shilpa Lawande, Pratibha Rana, and Amy Miller were among the women from HP Vertica who staffed the HP recruiting booth located in the career fair area of the conference. The staff met countless talented young candidates eager to learn more about HP and about Vertica. The recruiting team collected literally hundreds of resumes and even held several “on-the-spot” interviews.

recruiting booth

The HP Recruiting Booth

One of the big attractions of the HP booth was the daily raffle. At the end of each of the three days of the college fair, an HP Slate was raffled off. The raffle was immensely popular, drawing visits (and repeat visits) from women across the conference.

booth 2

Crowds gathering at our daily HP Slate raffle
The HP Vertica women also scoped out the surrounding scene. Companies like Google, LinkedIn, Facebook, and Pinterest brought incredible creativity and enthusiasm to their booths. The team gathered ideas and hope work with HP recruiting to make HP recruiting booth even more eye-catching and memorable than ever.


The conference was nothing short of a success for HP Vertica attendees, who gathered a large pool of candidates, in addition to cheering on both Bo Hyun and Shilpa at their presentations. And, last but not least, everyone got to know each other a bit better, laughed a lot, and enjoyed a sense of camaraderie, teamwork, and enthusiasm at being part of HP Vertica.

What’s New in Dragline (7.1.0): Installing HP Vertica Pulse

Installing HP Vertica Pulse Video

Installing Pulse from Vertica Systems on Vimeo.

HP Vertica 7.1.0 introduces the general availability of HP Vertica Pulse, our add-on sentiment analysis package for HP Vertica. Pulse provides a suite of functions that allow you to analyze and extract the sentiment from text, directly from your HP Vertica database. For example, you can use HP Vertica Pulse to analyze sentiment from Tweets or online product reviews to get a feel for how satisfied your customers are about your products or services.

HP Vertica Pulse automatically discovers attributes included in text and scores them using a built-in system dictionary. You can tune user-dictionaries to detect certain words or phrases, to determine how words are scored, and to filter out attributes that are of no interest to you. Because of this flexibility, you can tune HP Vertica Pulse to work for your specific business needs.

Currently, HP Vertica Pulse allows you to analyze English language text only. You can download HP Vertica Pulse as an add-on package for your Enterprise Edition or as a trial for your Community Edition, from Additionally, the Innovations section of the HP Vertica Marketplace offers a beta version of Pulse for Spanish only. Take a look at this video to learn how to install Pulse and stay tuned for our next video, ‘Using Pulse’.

HP Vertica Pulse documentation.

Is Big Data Giving You Grief? Part 5: Acceptance

“We can do this”

Over the last month or so, this series has discussed how organizations often deal with a missed big data opportunity in ways that closely resemble the grieving process, and how that process maps to the commonly understood five stages of grief: denial, anger, bargaining, depression, and acceptance. This is the last entry in the series; it focuses on how an organization can move forward effectively with a big data project.

While big data is big, complicated, fast, and so forth, it is also very vague to most businesses. I was at an event recently where a poll question was asked of a room full of technology professionals – “How important is big data to your business?” A surprisingly high number of respondents felt that big data wasn’t relevant to them. Afterwards, I spoke with one of the attendees over lunch. I asked him what the primary challenges were to his business. It turns out that their business costs rely primarily on commodity costs – if the price of an input such as oil goes up or the supply is disrupted, the entire business is affected. I asked him whether he thought social media was relevant to his business, and he didn’t believe so. I then talked about how hedge funds have found that Tweets can be a very effective way of predicting commodity prices and availability disruptions. Until that moment, he was unaware that this was possible. This was what I call a “light bulb” moment. Suddenly, the appeal of big data became clear.

This experience highlighted for me a fundamental issue I see daily in the big data space – that it’s just too big (and vague) for many organizations to grasp its tangible value – an important pre-requisite to moving forward. So even while they go through all the stages of grief and struggle with the fact that their competitors may be outperforming them due to big data, companies also struggle with how to turn that into a plan of action.

Once they’ve worked their way through the realization that something’s wrong, organizations are often ready to take action. Here are some of the most helpful techniques I’ve seen businesses take over the years to begin an effective big data program – to accept the reality of the situation, and move forward.

Execute tactically, think strategically
For the organization first tackling big data, this is probably the most important thing to keep in mind. Big data projects rarely start with a crystal clear vision of what the strategic outcome should be. Uncertainty and hype around the opportunity, unfamiliarity with how to handle big data, lack of a data science competence, and so forth all create challenges that make it tough to articulate an up-front strategic vision.

But don’t interpret that as a pass to ignore the potential impact of a big data project. Thus the advice. Execute the project tactically – be prepared to move fast with the aim to demonstrate value quickly. And when the project is complete, a debrief with the business leadership is essential. In this debrief, answer two questions: How did applying big data matter to the business? And given what we’ve learned, how can our next project impact the business in a bigger way?

The answers are inputs to the next project, and over time can serve as a powerful guide to articulating a big data strategy for the business.

Don’t boil the ocean
Very often, when a group of people from an organization attend a big data event, they all come back very enthused about big data projects. Vendors love to talk about big-picture, blue sky notions of transforming businesses or industries with big data. It’s exciting stuff, but doesn’t lend itself to immediate action – especially for a business new to big data.

So don’t start there.

A much better approach is to identify measurable goals that can be tied to actions that can be completed in the right timeframe. What’s “the right timeframe”? Good question! In part, it depends on how open the business is to a big data initiative – if the leadership team is bearish on the idea and needs powerful convincing, it’ll be important to demonstrate value quickly. Also, immediacy is a powerful guide to enthusiasm – so don’t tell the IT team to disappear for a year and come back with a big data architecture. There’s no immediacy, and as a result there likely won’t be much focus. So don’t boil the ocean and try to do everything at once, in a big hurry. Start with focus, and retain it as you progress.

One foot in front of the other (and sometimes…baby steps!)
When an organization wakes up and realizes that it’s at risk of being left behind or otherwise outperformed by others due to big data, the first response can be panic. The CEO or CMO may set a goal for the team – catch up. This can kick everyone into overdrive quickly, which is great. But it can also set everyone running in different directions with a vague charter to do something to change the business…now!

The tendency is to start chasing the Big Goal – maybe something dramatic like “reinvent the business”. For the organization new to big data, this is a recipe for trouble. Developing any new core competence takes time, and nobody starts as an expert. Learning to incorporate big data into your business is the same thing. It’s probably not realistic to expect a team accustomed to managing enterprise applications (which might all be running on a twenty-year-old technology stack) to learn massively parallel technologies, large scale data management and data science in a week. Or a month. Or a year.

So put one foot in front of the other. Don’t expect to master big data overnight, and instead take measured steps. Pick a project with a strong return on investment to get stakeholders on board and get the technology team’s feet wet in new technology. Then make the next project somewhat more ambitious. As the team learns more about delivering these projects, it’ll be much more natural to assess larger questions such as revising technology architecture.

It’s not too late
Marketing is marketing and reality is reality. Just because one of your competitors released a success story about their big data program last week doesn’t mean that there’s no benefit for your company. And when an article shows up online or in the printed media that declares that the big data war is over, and you lost if you’re not one of a handful of companies – take it with a huge grain of salt. There’s nothing wrong with a big data project that makes your business more profitable, or drives more top line revenue. And while it’s fun to contemplate reinventing your company, there are plenty of practical (and do-able) opportunities for improving revenue, customer experience, efficiency, etc. So don’t think for a moment that it’s too late.

Furthermore, by waiting a bit, organizations can take advantage of the learnings of others – things to do, things to avoid, and so forth. And the tools will usually improve. And successful use cases will become easier to spot. All these factors will reduce the risk to your big data project, and increase the likelihood of success. So it’s not too late.

To Accept or Not
Sadly, not all organizations make it to this stage. I’ve seen companies get stuck in finger pointing exercises, or trapped in endless cycles of ill-defined big data “science projects” that never seem to produce anything tangible and never end, or even put on blinders and avoid big data completely. But for companies who get to a place where they’re ready to accept the challenge, there are opportunities to meaningfully impact the business. And there are frequently increasing returns on well-crafted big data projects – which is to say that for every additional dollar spent over time, the value to the business actually increases. I’ve seen this cycle unfold time and time again, and in every single case of which I’m aware, the organization has reached the stage I’m referring to as “acceptance”, and is moving forward in a well-planned fashion with an effective big data program.

In fact, as I write this I’m listening to the HP Vertica Customer Advisory Board talk about their experiences to date with Vertica. And every one of them has approached their big data program in the ways described above. And every one of them has discovered increasing returns to their big data investment over time.

So put big data grief aside, accept that big data can help your business, and get started!

Is Big Data Giving You Grief? Part 4: Depression

“The problem is too big.  How can we possibly address it?”

Continuing the five part series which explores how organizations coping with big data often go through a process that closely resembles grief, this segment addresses the point at which the organization finally grasps the reality of big data and realizes the magnitude of the opportunity and challenge…and gets depressed about the reality of it.

Having seen this more than once, I’ve observed a few ways this shows in an organization.  Here are the most common reactions.

It’s too big

This reaction makes sense.  After all, as much as we in the industry say that “big data” is more than big and describe it with a laundry list of varying attributes, we all agree that it’s big.  It represents addressing data at a scale never before attempted by most organizations.  It represents analytic abilities perhaps never done before – and a capability pivot towards being an analytics-driven company.  And it may represent opportunities that are so big they appear to be nebulous: “If I capture ten thousand times as much data about my product, how does that translate into value?  Does that mean I’ll sell ten thousand times as many widgets?  How do I quantify the payoff?”

It may be challenging just to get a handle on the costs of a big data program for reasons mentioned in earlier parts of this series, much less the potential payoff.  This can make for a very challenging return-on-investment calculation.

We’re not ready

I believe I may have heard this particular form of worry more than anything else.  The infrastructure isn’t ready, the people aren’t ready to build big data applications, the business isn’t ready to consume the new data, and so on.  And, in fact, the company may not be prepared to size the big data effort because the team may not have the know-how for the ROI calculation (see above).  Also, the executive leadership may be unprepared to make a strategic wager on the program because of the uncertainty around the risks and benefits.

This can seem like a true show-stopper.  It’s not easy to change an organization.  Skills and technologies may not appear to be aligned with big data needs.  The various lines of business may not realize the ways they can improve or revolutionize their business.  The leadership team may be unaccustomed to making big bets on unproven technologies, or may believe that big data is a fad and will pass.

We’re too late

I hear this a lot too.  Everywhere a business turns today there’s a story about how someone has transformed their business, created new markets, broken old barriers, etc.  It’s easy to believe that all the opportunity is gone – that there’s no more benefit to tackling big data because it’s already been done.  It’s also easy to believe that it would be impossible to “catch up” with others because of all the time and effort required.

While this can be an intimidating belief, it can also be hard to characterize accurately.  After all, do you think your competitors will announce that the big data project they recently publicized in the media is a year late and $10M USD over budget?  Instead, they’ll play it up as if it’s a runaway success.  Vendors help this along too – who wouldn’t want to tout that their product helped a company?

So the saying goes – “The darkest hour is just before the dawn.”  Sage words written long before computers that apply to this situation.   But this is actually a positive place to be, because once a team has moved through anger, denial, bargaining , and into depression, it’s ready to come to terms with the situation and make an action plan to move forward.  I’ll discuss that next week in the final part of this series: acceptance.

Next week the series concludes with…acceptance.  “We can do this.”

Is Big Data Giving You Grief? Part 3: Bargaining

Is Big Data Giving You Grief?  Part 3: Bargaining

“Can’t we work with our current technologies (and vendors)? But they cost too much!”
Continuing the five part series about the stages of big-data grief that organizations experience, this segment focuses on the first time organizations explore the reality of the challenges and opportunities presented by big data and start to work their way forward…with bargaining.
Coping with a missed opportunity often brings some introspection. And with that comes the need to explore what-ifs that may provide a way forward. Here are some of the more common what-ifs that organizations explore during this phase.

What if I go to my current vendor? They’re sure to have some great technology. That’ll fix the problem.
This is a perfectly fine path of inquiry to explore. The only issue with this, as mentioned in my previous post (Part 2: Anger) is that vendors have a tendency to re-label their technology to suit a desirable market. So their technology offerings may not actually be suited to big data needs. And spending time and effort exploring these technologies to verify this can distract and prevent you from moving forward.

Also, vendors may have a business that relies on high-margin technology or services that were priced for a time before the big data explosion. So, the economics of their technology may suit them, but not the organization in need – your company. For example, if I need to store a petabyte of data in a data warehouse, I might require a several hundred node data warehouse cluster. If my current vendor charges a price of a hundred thousand US dollars per node, this isn’t economically feasible since I can now find alternatives that are purpose-built for large scale database processing and are priced at 1/5th or 1/10th that (or less!).

What if I hire some smart people? They’ll bring skills and insight. They’ll fix the problem.
Like the question above, this is a perfectly reasonable question to ask. But hiring bright people with the perfect skills can be very difficult today – the talent pool for big data is slim, and the hiring for these folks is highly competitive. Furthermore, hiring from outside doesn’t bring in the context of the business. In almost every business, there are nuances to the products, culture, market, and so forth that have a meaningful impact on the business. Hired guns, no matter how skilled, often lack this context.

Also, just bringing in new people doesn’t necessarily mean that your organization’s technology will suit them. Most analytic professionals develop their way of operating—their “game plan”—early in their career, and often prefer a particular set of technologies. It’s likely your new hires will want to introduce technologies they’re familiar with to your organization. And that can introduce additional complexity. A classic example of this is hiring a data science team who have spent the last decade analyzing data with the SAS system. If the organization doesn’t use SAS to begin with, the new team will likely press to introduce it.. And that may conflict with how the how the organization approaches analytics.

What if I download this cool open source software? I hear that stuff is magic, so that’ll fix the problem.
Unlike the first two what-ifs, this one should be approached with great caution! As mentioned in my previous post, open source software has something of a unique tendency to be associated with vague, broad, exaggerated, and often contradictory claims of functionality. This brings to mind a classic bit of satire by the Saturday Night Live crew, first aired in 1976: “New Shimmer is both a floor wax and a dessert topping!” The easy mistake to make here is for the technology team to rush forward, install the new stuff and start to experiment with it to the exclusion of all else. Six months (and several million dollars of staff time) later, the sunk cost in the open source option is so huge that it becomes a fait accompli. Careers would be damaged if the team admitted that it just wasted six months proving that the technology does not do what it claims, so it becomes the default choice.

What if I do what everybody else is doing? Crowds have wisdom, so that’ll fix the problem.
The risk with this thinking is similar to that posed by open source. This often goes hand-in-hand with hiring big data smarts – companies often bring in people from the outside and pay them to do what they’ve done elsewhere. It can definitely accelerate a big data program. But it can also guarantee that the efforts are more of a me-too duplication of something the rest of the industry has already done rather than true innovation. And while this may be suited for some businesses, the big money in big data is in being the first to derive new insights.

These are all perfectly acceptable questions that come up as organizations begin to acknowledge, for the first time, the reality of big data. But this isn’t the end of the discussion by any means. It’s important to avoid getting so enamored with exploring one or two of the above options that you don’t follow through on the “grief” process. But the natural next step is to be intimidated by the challenge, which will serve as an important reality check. I’ll cover this in the next segment: depression. So stay tuned!

Next up: Depression “The problem is too big. How can we possibly tackle it?”

Get Started With Vertica Today

Subscribe to Vertica