In-Depth

The Confusing Future of BI and Data Warehousing

Thanks to a combination of social, mobility, cloud, Big Data, and other forces, business intelligence and data warehousing are on the cusp of profound transformations. The question is: which way are these technologies headed?

• By Stephen Swoyer
• 03/27/2012

At last month's TDWI World Conference in Las Vegas, folks were buzzing about change. Transformation. Disruption. Uncertainty.

Things are certainly changing fast; the status quo is quickly being upended and the familiar fixtures of business-as-usual no longer seem quite so fixed. Recently, for example, Apple Inc. disclosed that sales of its iOS-based products -- the iPod Touch, iPhone, and iPad -- had surpassed those of its Mac desktops. Five years of iOS sales as against 28 years of Mac sales.

The upshot was a conference experience in which newly coined terms such as "gamification" were combined with familiar acronyms such as ACID to describe BI-in-the-not-too-distant-future.

Thanks to some combination of mobility, cloud, Big Data, CEP, NoSQL, and a matrix of other technical and social forces, business intelligence and data warehousing are in the midst of profound transformations. The caveat, of course, is that even though things are changing and will never again be the same, no one can quite agree about what's going to change and how. This is particularly true when it came to DW futures.

Take John Cruppi, CTO of JackBe Inc., an upstart BI and analytics vendor that specializes in connectivity to non-traditional information sources. Cruppi says most enterprise data warehouse (EDW) practices are grounded in a misguided premise.

"[Consider all of] the data warehousing guys, all of the data architects and [data] modelers, the MDM guys, [and] the data quality guys. They've worked for years -- decades -- getting the data [into the warehouse] and making sure it's right," he argues. "But say you're actually able to achieve that. What happens then? Isn't it the case that [the EDW] becomes somewhat of a silo again? If the only thing you're really going to be able to get from your data warehouse is ad hoc [reporting] and dashboards, and if the self-service side is very limited, in a sense you've gone ahead and you've created another silo."

Cruppi's point concerns the inertia of traditional data warehouse environment. To the extent that traditional DW architectures -- or, more precisely, the data management processes and disciplines that evolved to support the data warehouse -- impose limitations on the freshness of data or the practical ability of DW architects and BI pros to respond to the needs of business users, Cruppi claims the traditional data warehouse has itself become a barrier.

Most people accept this argument to some degree. It explains not only the persistence of the operational data store (ODS) -- which has been given a new lease on life thanks to real-time/right-time BI -- but the emergence of so many new (or non-traditional) data management tools, from MPP appliances to technologies such as MapReduce (implemented natively or via Hadoop) to NoSQL repositories.

Cruppi takes it one step further, arguing that the traditional EDW shouldn't so much be abolished as set aside. That's what's happened over the last decade, he claims, as frustrated business users went out-of-band around IT, approaching vendors directly, to redress their BI or analytic needs.

It's the Warehouse, Stupid

Few others put the matter quite so starkly, but many vendor attendees at the TDWI World Conference were happy to talk to BI This Week about the shortcomings or failures of the (once sacrosanct) EDW, or, for that matter, of ETL-centric approaches to data integration -- to say nothing of the decline in the importance or prestige of relational data.

The TDWI exhibit hall, for example, played host to several consulting and services vendors that specialize in tackling emerging data integration and data warehousing problems. Companies such as Impetus Technologies Inc., Intellicus Technologies, and, Lunexa LLC joined Hexaware Technologies in promoting their specialty services to show attendees.

All of these vendors had some kind of Big Data pitch, too. This was fitting, as the show floor positively bristled with Big Data-oriented vendors. In addition to first-wave players such as Teradata Corp., seminal competitors such as IBM Corp. (now the steward of DW appliance pioneer Netezza Inc.) and appliance mainstays such as Kognitio, ParAccel, and Vertica (a Hewlett-Packard Co. company), a bevy of NoSQL players was also present.

In this respect, vendors such as MarkLogic Corp., RainStor, Inc., and SAND Technology spoke their version of truth (NoSQL) to DW professionals, championing new -- typically Hadoop-oriented -- ways to solve old and new problems, all while maintaining the atomicity, consistency, isolation, and durability (ACID) that are sacred to so many in the data management Old Guard.

Plenty of established vendors got into the act. Take Kalido, which -- in the spirit of NoSQL -- recently took up what might be called the "NoETL" banner. (Its booth featured a sign with the unicode "No" symbol superimposed over the letters "ETL.")

"In our latest release, we built in a lot of functionality to eliminate the need for a lot of ETL," says Kalido CEO Bill Hewitt. "We aren't eliminating the need to extract and transform data, we're eliminating the need for a separate tool to do that," adding that "we can take data directly from the source, determine what kind of data it is, put it in the right staging tables, and load it directly."

Kalido has long championed an iterative design philosophy that incorporates what are now recognized as agile concepts and methods. Its Kalido Information Engine likewise uses a "generic" data model that the company says helps to insulate it from business change.

Hewitt, too, offered a vision of where things are heading, invoking the Wild West-like atmosphere of the networking segment prior to Bob Metcalfe's invention of Ethernet to illustrate the problem of data governance.

"Network management today is virtual. If something breaks, it repairs itself or it finds an alternate route. It hasn't always been that way. In my early days at IBM, we had different data centers with multiple networks. When something went wrong, it was difficult to determine who was at fault: was it the carrier or the hardware vendor?" he explains, noting that both parties tended to blame one another.

"With governance, the same thing needs to happen. At a minimum, I need to be able to go to those underlying systems and figure out which systems are more problematic [i.e., from a governance perspective] than others. I can't do that now, because the tools [or] the standards [for connectivity, for interoperability] don't exist."

Goodbye Excel, Hello Avatar

A distinct Futurist bent was on display, at least among a few of the vendor representatives.

Composite VP of marketing Bob Eve, for example, wasn't alone in describing a future in which BI and analytics are consumed primarily through an AppStore-like delivery channel. What's more, Eve predicted, the BI user experience will itself be less explicitly application-like and altogether more real: immersive, interactive -- the stuff of virtual reality.

Eve wasn't the only one talking immersion, either. QlikTech Inc.'s Donald Farmer -- the kind of Big Thinker who can turn a discussion involving the anesthetic properties of peat moss (Farmer is a Scot) into a compelling object-lesson in BI -- also championed immersion, describing it as characteristic of the user experience of the not-too-distant future. (Farmer likewise had a lot to say about the concept of "gamification." We'll hear more from him in a future BI This Week article.)

If Composite Software's Eve sounds optimistic about the future of BI and DW, his assessment of the BI and DW status quo -- or its IT aspect, at least -- sounds considerably more pessimistic, if not downright acerbic. Business dissatisfaction with IT is nothing new, Eve points out, citing a 2011 TDWI survey in which more than half (57 percent) of respondents expressed dissatisfaction with IT's inability to respond to requests in a timely manner.

After a decade of disruption -- from software-as-a-service (SaaS) to data warehouse appliances, from agile practices to "self-service" BI -- things aren't improving fast enough.

"IT's full-steam-ahead mentality ... is not a course for long-term survival," he observes, comparing the imminence of significant disruption in the near-term to the "proverbial iceberg" that sunk the RMS Titanic. IT, too, will founder on the iceberg of change, warns Eve, who -- in addition to the rise of an AppStore-like delivery channel -- predicts a massive expansion in cloud computing capacity.

Eve likewise foresees the development of a culture in which "mavens" or "super users" have more freedom to lead by influence or example, without being hobbled by IT constraints or "standards." "What will become of IT as we know it?" he wonders, rhetorically. "At their retirement clubs, someone needs to rearrange the deck chairs."