In-Depth

BI and DI Challenges Loom, Worrying Analysts

Data integration needs are shifting, but vendors, many content with their market share and revenues, simply aren't keeping up with the needs imposed by new data types.

• By Stephen Swoyer
• 06/08/2011

Surveys, Magic Quadrants, Waves, and other popular metrics paint a picture of a neatly divided data integration (DI) market. Think of it as a big pie chart, with giants Microsoft Corp. and Oracle Corp. enjoying large slices thanks to the ubiquity of their flagship DBMSes, which bundle built-in DI tools.

Add in the sizeable piece claimed by at least one DI pure-play (Informatica Corp.), the large share allotted to middleware powerhouse IBM Corp., and a slice for SAP AG (which inherited its most powerful DI technologies from Business Objects SA), and the thin wedges (or niches) staked out by a host of specialty players, and you have neatly divided DI market.

Unfortunately for BI professionals, many DI vendors seem content with the market, which yields reliable revenues, so there's little motivation to shake up the status quo.

Several industry veterans point out that the status quo could use a good bit of shaking up. Integration has become an inescapably integrated practice, such that the priorities or considerations which have traditionally divided integration products (chiefly by mandating the use of distinct tools for application-oriented and data-oriented integration) haven't so much disappeared as been obviated, perhaps even obliterated.

"[Look at] all the files from server logs, and many SaaS or hosted app companies, which have APIs and no SQL access and [which] deliver flat records, objects, or files," observes Mark Madsen, a veteran data warehouse architect and a principal with business intelligence (BI) consultancy Third Nature Inc.

One issue, he says, is that enterprises are struggling to integrate new kinds of "stuff" -- e.g., traditional structured or semi-structured data, unstructured data, traditional application messages, and newfangled event-oriented data.

A second issue concerns timeliness: as demands more real-time data increase, IT departments don't have the luxury of packaging, formatting, or transforming it to suit the preferences or requirements of business users.

A third issue concerns performance. In addition to the push for real-time, enterprises are collecting more historical data, typically structured data. If they plan to crunch hundreds of terabytes of (mostly structured) historical data or consume gigabytes of (increasingly heterogeneous) operational data at close to real-time speeds, they're going to need fast and flexible connectivity that can handle transformations and transactions.

The final issue is that the roles of both IT and the line-of-business users are moving to a process-centric model, where analytics are used not just to predict but to influence outcomes. It's the world of complex event processing (CEP) in which, for example, the seemingly benign act of querying the balance of one's mortgage can kick off events across a number of related business processes.

"Who looks up their loan amount?" asked data warehousing expert Claudia Imhoff, founder and president of Intelligent Solutions Inc., during a seminar on operational business intelligence (OBI) at TDWI's World Conference in Washington, D.C.

"Nobody, unless -- what am I getting ready to do? [I'm either going to] pay it off or [I'm going to] look for a lower rate. Now what should that activity, that event, trigger at the bank?" At the very least, Imhoff told attendees, it should have precipitated a contact overture of some kind. "It should have triggered somebody calling that person and saying, 'Hey, what you doing? Let me talk about what we can offer you!' Anything: some contact from the bank should have occurred."

DI Vendors Behind the Times

DI vendors, says Madsen, have mostly ignored the changing requirements of their customers.

"I see files rising faster than databases, and the DI vendors [are] still living in a world of SQL access that's going [in] the other direction," he argues. Although Madsen believes change is imminent, he concedes it isn't "as visible yet inside IT, which appears to be where [the DI players] pay attention."

We're almost half a decade into Web 2.0, he grouses, and "the big guys still can't do Web APIs ... properly." As Madsen sees it, a lot of DI players are "married to [the idea of] following the market rather than leading it."

He's hardly alone. At April's TDWI World Conference in Washington, D.C., Imhoff served up a similar critique. In her OBI session, Imhoff discussed the challenges posed by CEP as well as by other kinds of non-traditional data. She described CEP, in particular, as "an entire world of analytics that we [BI professionals] don't know anything about."

For the most part, data warehousing is stuck in a pre-CEP mindset.

OBI, as envisioned by Imhoff and co-presenter Colin White (president of Database Associates Inc.), proposes to change this. It replaces data centricity with process centricity. It's as concerned with the routine (e.g., optimizing and automating the workflows that drive a particular process) as with the outstanding or anomalous (e.g., successfully matching a sequence of events with the fraudulent use of a credit card). It's less about past performance -- the bread-and-butter of traditional BI -- and more about the Now.

"By and large, strategic BI and tactical BI are very data centric. [They're] comparing this data to that data ... [or] looking at long patterns [inside the] data," Imhoff explained. "When I [get] to operational BI, ... our whole paradigm kind of shifts from a data centricity to a process centricity."

Right now, traditional (data-centric) BI architectures can barely keep up with the flow and variety of stuff-types. Some day soon, she predicts, conventional DW-powered BI environments will run out of gas. This isn't in any sense a bad thing, Imhoff stressed; it simply means that BI professionals -- and the vendors which service them -- are going to have to start thinking outside the COTS box.

"[A]t some point, traditional BI architectures [will] stick their tongues out and go, 'I can't go any faster. I'm done,'" Imhoff told attendees. "At some point, we all have to recognize that traditional BI has gone as fast as it can, and if I want to go faster, I'm going to have to [change my architecture]."

Madsen, for his part, doesn't think enterprise application integration (EAI) vendors are any more enlightened than their DI counterparts. If the latter come at integration from a data-centric perspective, the former focus on application-level phenomena such as messages and transactions. Neither camp seems overly interested in building a bridge toward the other. As Madsen put it in a February presentation at TDWI's World Conference in Las Vegas, "traditional EAI vendors tried to provide some level of operational DI, but the[se] tools [use] heavyweight SOA tooling with poor features for the actual tasks of DI."

If the OBI vision championed by Imhoff and White is ever to come to fruition, Madsen says, EAI and DI vendors are going to have to do some bridge building.

In both cases, that means disrupting a neatly divided market share pie.