In-Depth

BI Tech Talk: Behind the Wild Profusion of ESBs

Why are so many data integration players selling enterprise service buses of their own? Is it just a matter of vendor opportunism?

• By Stephen Swoyer
• 04/17/2012

IBM Corp. has one. So do Oracle Corp., Microsoft Corp., SAP AG, Software AG, and TIBCO Software Inc. Even Informatica Corp., Information Builders Inc. (IBI), and Talend have one, too. So does Pervasive Software Inc., which OEMs one from Progress Software.

What's the common denominator? It isn't ETL, data quality, data profiling, or master data management.

In this list of giant enterprise applications vendors, middleware specialists, and data integration (DI) specialists, the common denominator is .. the enterprise service bus (ESB).

What's an ESB? In the broadest sense, it's a channel for message traffic. Messages, in this context, are communications between applications. Usually, they take the form of event notifications. Until recently, a distinction obtained between events, which are associated with applications, and queries, which are associated with databases. The former was the province of enterprise application integration (EAI); the latter the stuff of DI.

Given what ESBs are, why are so many nominal DI specialists now marketing them?

It's a sign of the times. Like so many other practices, the worlds of DI and application integration used to be effectively siloed: DI had its own standards (chiefly SQL, which drives much of ETL) and application integration had a different set. That's changed over the last half-decade, partly as a consequence of the un-siloing of the enterprise, partly as a result of an insatiable appetite for analytics.

Today's enterprise very much wants to examine itself. That's set the stage for the emergence of complex event processing (CEP), which applies analytics to IT "events" (as distinct from application events). The events of interest to CEP involve applications, databases, and system hardware -- literally any resource that can generate information in any form.

Events Everywhere

This has encouraged at least one DI industry veteran to coin a new term. "The big question for any customer is how do they eventify their current infrastructure without having to change everything?" asks Scott Fingerhut, senior director of product marketing with Informatica.

Fingerhut says "eventification" is a logical consequence of the availability -- of the increasing commodification -- of the ESB.

"If you're doing change data capture out of a database and pushing it out over a [service] bus, you're essentially eventifying that transaction, so the conversation that we'd have with the customer is instead of being intrusive to the application or the database, we can watch for changes and push the change out. We can eventify it."

If there's a place for the ESB in the DI toolkit, why are there so many ESBs? After all, when the concept itself first came to the fore about a decade ago, it prescribed a single messaging backbone through which an enterprise could funnel all of its application event traffic. Fast forward to 2012 and it's a very different story -- we have a wide choice of service buses.

At a very basic level, the message bus gives each vendor a means to shunt its own event traffic into an internetwork of enterprise event traffic. An Informatica ESB can use well-defined protocols (such as the Java Message Service) to communicate events to a TIBCO ESB.

The ESB, in this context, becomes an integration interface. To use the language of traditional data integration, it becomes a "connector" or "adapter" to another application or platform.

That's a far cry from the original vision of a single enterprise messaging backbone. According to Fingerhut and other industry veterans, however, the ESB vision might have been just a tad over-optimistic because a single ESB can't possibly scale to support all message traffic in large enterprise environments, especially one where non-traditional traffic -- information from databases and BI applications; servers, routers, switches, and network devices; control systems, ATMs, and other resources -- is likewise "eventified."

The case of the ESB is similar to that of the microkernel; 25 years ago it was predicted to replace monolithic kernel designs. Unfortunately, microkernels were much slower than their monolithic counterparts and couldn't keep up with a barrage of remote procedure call (RPC) traffic.

That's one reason why Linus Torvalds developed Linux, and it's why so many vendors offer their own ESBs, industry watchers say.

"The ESB was this idealistic single bus that was service-oriented that people could build. It was very simple to say, 'I'm going to move things on it: I'm going to move everything on it.' Unfortunately, the ESB has yet to scale because of all of the service calls that it has to make," says Fingerhut, who now heads up Informatica's CEP practice.

He sees the proliferation of ESBs as a pragmatic response to the failure of what might be called ESB 1.0. "In the beginning, people said, 'This is great. It's like a universal outlet [for electricity]. In practice, it really didn't scale for operations to be the single bus for all [message] traffic. It ran out of steam. It didn't get as far," Fingerhut suggests.

Open source software (OSS) DI specialist Talend introduced its ESB just over a year ago. It suggests another reason why ESBs have started to proliferate: the availability and robustness of OSS ESBs, such as Apache ServiceMix, FUSE ESB (an enterprise-grade version of ServiceMix), and, of course, Mule.

Talend's ESB implementation "uses a lot of Apache projects," says Yves de Montcheuil, vice president of marketing. It isn't simply a relabeling of OSS offerings, however. Far from it, de Montcheuil maintains, pointing to Talend's acquisition 18 months ago of the former Sopera, which specialized in service and support for OSS ESB projects. Then there's the DI-specific value-add that Talend brings to the table.

"Applications and data have to come together," de Montcheuil explains, noting that application integration architectures typically center around the needs of software developers. "Any time you need to transform data between the connector and your ESB, there's this very strong mentality among ESB developers to just code [their] way through it," he continues.

"[Talend's approach] is enabling [customers] to implement data quality routines or deploy data quality as a service and make it [usable] by the ESB. It's bringing data quality and data integration services to the ESB, [and] bring[ing] application integration to data integration."

BI and DI stalwart IBI introduced its first CEP offering, iWay Event Manager, two years ago, says Kevin Quinn, vice president of marketing with Information Builders. That product sits on top of IBI's ESB, called iWay Service Manager, that in turn can interoperate with more than two dozen messaging products, Quinn continues. Although most IBI customers opt to deploy Service Manager, some are using it in tandem with other service bus offerings.

"Because of our uniqueness, there have been cases where we've coexisted, so if somebody's got an existing ESB, they don't even have to rip it out. Service Manager can couple with that existing ESB and extend it and give it access to things that it normally wouldn't be able to touch," he points out. "We like to think of this as [being consistent with] our being the integration vendor for integration vendors."

A Question of Necessity

IBI isn't the sole claimant to this title, objects Informatica's Scott Fingerhut.

To the extent that a service bus provides a well-defined interface -- nominally hewing to JMS specifications -- for exchanging message traffic between systems, Fingerhut argues, it isn't so much a question of proliferating ESBs as it is one of intractable heterogeneity.

He sees the emergence of ESBs in the DI toolkit as a response to this problem: in spite of the best efforts of The Big Four (IBM, Oracle, Microsoft, and SAP), shops use a mix of different applications. In many cases, they'll have all four of The Big Four. For this reason, he argues, the service bus has become a pragmatic necessity.

"There's no vendor that's a center of a customer's world. No matter which vendor draws a diagram and shows themselves at the center of it -- no one's like that. [Nowadays ] a vendor shows you an architectural diagram, and it's like 20 technologies. What customer is going to buy [all of] this?" Fingerhut deadpans.

Outmoded Thinking?

Industry veteran Mark Madsen, a principal with BI and DW consultancy Third Nature, is a little uncomfortable with this perspective. After all, to the extent that one or more DI vendors introduce an ESB, it then becomes a checklist feature -- something that all DI vendors have to offer. That's not what makes Madsen most uncomfortable, however.

The ESB may even comprise a kind of remnant -- a leftover from CORBA and the outmoded thinking of client-server. "I suspect that part of the ESB challenge is that it's still sort of server-centric without meaning to be. It's like the old models. Today, SOA is in and ESB isn't quite the same approach and doesn't encapsulate the same way," he argues, noting that services are scaled independently in the SOA model.

In other words, he explains, "there is no big message passing -- an application calls services, maintains its state, and the remote invocations do the work." Besides, he concludes, a lot of real-time, Big Scale applications don't even use an ESB. "Look at how to do real-time architecture at large data scale. No more ESBs. Instead they talk about real-time flows and use custom-built systems. LinkedIn, Amazon, Netflix are a few who [are doing that]."