In-Depth
Enterprise Information Integration: Lead-Pipe Cinch or Pipe Dream?
Is EII a lead-pipe cinch or a lead-pipe dream?
For a long time now, enterprise information integration (EII) has been a technology category waiting to take off.
There’s no shortage of EII vendors: Avaki Corp. (now part of Sybase Inc.), Composite Software Inc., IBM Corp., MetaMatrix Inc., and a host of other players have shopped EII solutions for several years now (apparently with success), but as far as many business intelligence (BI) and data warehousing professionals are concerned, EII is still an unknown category.
One apparently informed TDWI member spoke on background, noting that while EII looks good on paper, the reality on the ground is that adoption is limited, perhaps due to alternate solutions for filling data-timeliness gaps, such as replication, or by embedding BI tools within/against source systems. This anonymous source stated that a certain ETL vendor says that demand for the technology is weak.
There are a few explanations for this. One, of course, is that EII remains a nascent technology category, at least from the point of view of many potential adopters (despite the fact that it’s been around for several years). The data warehouse didn’t become a fait accompli overnight (and arguably still isn’t a fait accompli), and it’s possible that many companies are still feeling their way as far as EII is concerned. Given the din (or apoclasm, more accurately) of competing application and process visions, it’s not surprising some organizations are taking their time. However, forward-thinking companies such as financial services giant ING Group are already neck-deep into EII: at TDWI’s February conference, for example, keynote speaker (and ING CTO) Raymond Karrenbauer told attendees his company uses EII as the backbone of its global BI architecture.
There’s another possible explanation, says Cindi Howson, a member of TDWI’s extended research collaborative and a principal with BIScorecard.com. It could be that many BI pros (and TDWI members, especially) approach EII from a data warehousing-centric perspective. This fact negatively colors the potential value and applicability of EII technology in practice, she suggests; as the real magic (in her opinion) is a direct connection of EII with BI, not necessarily DW. “I do think EII with BI tools can be powerful,” Howson concludes, citing Cognos Inc.’s partnership with Composite as one example.
Another example of EII and BI synergy, suggests Philip Russom, senior manager of research and services with TDWI, could be a killer app: frequently refreshed reports. Typically exposed in the form of dashboard or operational reporting solutions, frequently refreshed reporting powered by EII can help bolster time-sensitive applications, such as just-in-time inventory management systems, or yield-intense manufacturing efforts. “I'd guess that a single-digit percentage of reports would benefit from frequent refresh—i.e., more than once every 24 hours—so the real-world demand for EII is inherently limited, though increasing slowly,” says Russom.
Russom says that EII's real killer app “is to bolster the data access and integration capability of BI platforms that refresh reports directlyfrom app databases.” In this model, Russom suggests, “EII is not married to ETL, although it's an alternative to ETL in a very short list of situations.”
Wayne Eckerson, TDWI’s research and services director (and resident dashboard guru) expands on this theme, linking the availability of frequently refreshed data with the emergence of a new class of highly interactive dashboards. “Many new dashboard products and ones that are built in-house pull data from multiple systems on demand to display on one screen in a dashboard,” Eckerson comments. “Except in a few cases, these products don't use EII products per se… but provide direct links via adapter APIs to source systems—some use Web services APIs, others event-processing models, others intelligent agents. But dashboards are a prime place for EII if it is to take off.”
Are BI Powers-that-Be Against EII?
EII is a touchy subject for some BI vendors. To the extent that it promotes a platform (data federation) and a vision (frequent refresh) that in turn beg a few potentially uncomfortable questions—if the major BI or data integration suites provide pervasive access to data sources (or canned integration a-go-go), then why have vendors such as Cognos Inc. or Informatica Corp. notched licensing agreements with EII pure-plays like Composite?
Cognos officials, for one thing, say they don’t see a problem here. Cognos has its own ETL tool (which it acquired from the former DecisionStream)—a product that has arguably languished over under its stewardship—but Paul Hulford, senior product marketing manager with Cognos, says he doesn’t see EII as the bête noire of ETL (or of the all-in-one BI suite, for that matter). Instead, Hulford takes the idea of frequently refreshed reporting and runs with it, positioning EII as an important complement to ReportNet and other flagship technologies—including the company’s new Go! Enterprise search offering. “With Composite, we can actually search transaction systems now [in real-time],” he maintains. “Search plus Composite really opens up a whole new angle for exploring enterprise data that wasn’t opened up before.”
Representatives from Informatica, on the other hand, have been careful to position EII as an important, albeit niche, technology. (http://tdwi.net/Publications/display.aspx?id=7471&t=y) Informatica last year notched an agreement with Composite to help bolster its own data integration stack. And last October, Composite’s EII technology was also given prominence of place in Informatica’s PowerCenter 8 announcement. (http://www.esj.com/business_intelligence/article.aspx?t=y&EditorialsID=7754)
“We’re hearing from our customers that [federation] is a stop-gap measure,” said Ivan Chong, vice-president of business development with Informatica, at the time. “It’s a way they can temporarily expose data that’s SILOed in resources; or if there’s an acquisition, it’s a way they can quickly expose data in [the acquired company’s] new systems. It’s for situations like that.” In the long run, Chong maintained, organizations will embrace full-blown data integration—with ETL as its centerpiece—to more robustly connect their enterprises.
Business Objects, for its part, has bet big on EII. Earlier this month, the Paris-based BI giant outlined an enterprise information management (EIM) strategy that incorporates ETL technology it acquired last year from the former Medience. (http://www.tdwi.org/News/display.aspx?ID=7943) Business Objects, too, is an ETL vendor (its Data Integrator tool experienced 120 percent revenue growth last year, officials claim), but representatives reject the idea that data integration—or EIM, as they call it—is an either (ETL) or (EII) scenario. “We’re seeing that there are two drivers for data federation or EII. Rather than taxing a BI product to do an ETL kind of work, you have the option of using data federation to accelerate connectivity [to data sources], especially for reporting,” says Philip On, group manager for product marketing with Business Objects. “The second use case is extending the data warehouse. A lot of organizations who are trying to bring information into a data warehouse very rapidly—say in a merger and acquisition-type scenario—these new data sources have to be brought in ASAP, and the data warehouse cycle may take several months, so data federation offers a very rapid way of bringing data sources in.”
The future of EII
Rare is the BI professional who takes seriously the idea—propounded by some EII triumphalists—that this technology will replace the ETL-powered data warehouse. Nor do most knowledgeable industry watchers endorse the idea of EII as the backbone of a “virtual” data warehouse. Russom, for his part, scoffs at this notion.
“The way I see it, some requirements demand a warehouse and others don't. Some reports are best when populated with fresh data straight from app databases—which EII can do fast, with little hit of databases—others demand cleansed, aggregated and time series data, which a BI Framework (complete with a warehouse) does and EII alone can't do,” he indicates.
Mike Schiff, a principal with data warehousing consultancy MAS Strategies Inc. and a member of TDWI’s extended research collaborative, agrees. “You can access five different databases without moving the data around, but how do you freeze the data? How do you provide a snapshot view in time of the data [in those data sources]?” he asks. “Years ago, people used to talk about the virtual data warehouse. You don’t have to build it, they argued; you can go out to all of these systems directly and look at those systems in real time. It was a nice idea, but it doesn’t give you a time-stamped image with everything frozen beneath you. So how are you supposed to make sense of it?”
That’s why the data warehouse won’t ever go out of style, Schiff argues.EII, for its part, will survive, too—albeit (like ETL) in a somewhat different form. “I don't think EII will survive over the long haul as a self-standing tool-type,” concludes Russom. “It will eventually be a collection of useful capabilities in larger platforms, like report servers, RDBMSes, ETL servers, and application integration platforms. We already see this precedence in products from Actuate, IBM, Informatica, and BEA, respectively.”
Adds Russom: “After all, EII is an interesting combination of two ancient capabilities: database views and distributed queries, upgraded to include more modern modeling tools and query optimizations that these two require.”