Q&A: Analyst Sees Bright Future in Big Data

Huge data sets can reveal hidden patterns, analyst explains.

• By Linda L. Briggs
• 03/27/2012

One of the most interesting aspects of big data, says author and industry analyst Judith Hurwitz, is the ability to examine huge sets of data for trends that might be overlooked otherwise. Big data is "not about what you already know," she says. "You want to ... find out what you don't know."

Hurwitz is president and CEO of Hurwitz & Associates, Inc., a strategy consulting and research firm focused on the business value of emerging computing technologies, and a well-known analyst, author, and commentator with decades of experience observing the technology arena. In the first segment of this two-part interview, she discussed her new book, Smart or Lucky: How Technology Leaders Turn Change Into Success. In this portion, she focuses on trends and innovations she sees taking place in business intelligence.

BI This Week: As a long-time observer of the technology space, how important is the "big data" explosion?

Judith Hurwitz: Well, everybody is suddenly talking about "big data," certainly. Companies are coming out of the woodwork -- companies that did something else for years but are suddenly a "big data" company. The trend is real, there are real products, and it is huge.

One of the most important advantages with big data isn't about asking the questions you already know you want to ask about your data. Where big data starts is when you don't necessarily know what questions to ask. Instead, you have lots of information and you're trying to figure out whether there's something there you should know about.

Big data allows you to take huge volumes of data and say, "OK, if I look at this data, what is it telling me?" Maybe it looks at geography. "Where were my customers located over the last two years?" You start to look at patterns in the largest context. Once you do that, you get insights you never imagined.

You can then say, "OK, knowing that, I no longer need three trillion terabytes or whatever of that data anymore. I want to hone in on this subsection of data." Now, you can apply more traditional analytics tools because now you've figured out what you need to focus on.

That's the difference; that's where big data really begins to have a role. It's not what you already know. You want to look at things from a much more predictive standpoint, to find out what you don't know. When you look back, you might well say, "Oh, that's so obvious. It was staring me right in the face. ... Had I realized that, I would have changed my whole marketing strategy."

So I might be able to tease patterns out of big data that I couldn't see in smaller subsets?

That's exactly right. Typically, when you're doing the smaller subsets, you're asking questions such as, "Tell me how much my sales are growing. Tell me what's happening in this territory." That's fine. You need that, but where big data comes in is that it starts to look for patterns, and it starts to look at issues that don't just jump out at you. Maybe it's because there's too much information there and maybe your search is too broad. Maybe three-quarters of it is white noise, but you're not going to know until you look further.

After doing that, you need to build an integration bridge between show-me-everything and OK-now-let-me-look-at-that.

Regarding predictive analytics, which you mentioned in context of big data, where are we headed? Half of TDWI members say they want to be using predictive analytics productively in the next three years, but few are using it today. Are we at a tipping point there?

Predictive analytics does have a relationship to big data. What we're looking at is the technology that helps us not just to assess where we are, but to start to make connections between pieces of information. [By doing that], I'm going to get more value from not just what I can understand but what this means for the future. So, being able to predict the future, being able to see if certain factors are true and if I can understand the connective tissue between elements, I can start to say, "Oh, there's something going on here."

Maybe that means that I need to sell differently. Maybe that means I need to develop new go-to-market strategies.

For example, there is technology out there that lets you look across a broad swath of the market at what people are saying about certain products or certain television programs or whatever. I can look across television, Twitter feeds, Facebook, information from call centers, information from what critics say, and what is written in local newspapers. If you can start taking information like that -- that you could not have put together in the past – and you can start to understand what it's telling you, then you're in a much better position to act. ...

It's not a new idea. It's really what we have, for a long time, hoped that business intelligence would prove to be. We just didn't have the maturity of technology, and at a reasonable cost...

When we talk about analytics, is it becoming something that everyday users can take advantage of, or are BI analytics tools still for experts?

I think it depends on what you're trying to do. Something as simple as a Google search is a type of analytics, after all. Thirty years ago, the idea of being able to search a vast warehouse of data and say, "Where is the closest gas station to the corner of 5th and Main?" would have been unthinkable.

With more sophisticated analytics, though, you still need a skilled, trained person.

We're at a fascinating time in information technology, and you've been watching this space for a while. What do you see as one of the most important trends heading into the next 18 months or so relating to business intelligence in particular?

One thing is the ability to aggregate more and more data, [and have] systems that do a much better job of understanding context. It's very complicated, but we're getting into an era where more can be put into algorithms [and] can be used by people to start learning and understanding the context of how things are related. Our ability to turn that data into usable solutions is improving because we are getting much better at understanding what the context is.

Really, that's the whole background of [IBM's] Watson project. It isn't just feeding in information, then [letting the computer] figure out the answers. Watson actually starts to understand context, and over time, it learns more and more as it finds the right answers to things, and as the people who are tuning the systems feed it more information. Systems like that will become useful tools for figuring out solutions to problems we've never been able to tackle.

That takes us back to our original discussion about artificial intelligence and how far we've come.

That's right. There really is nothing new under the sun. People such as Marvin Minsky, the father of artificial intelligence, were having these discussions 30 years ago. All of the seeds of this, all of the potential to transform industries and businesses, has been around for a long time, but making it practical, making it usable, and making it so that people can take those concepts and make it practical and affordable -- that's the difference between success and failure in technology.