In-Depth

And the Biggest BI Development of 2005 Is...

Some say dashboards; others master data management; still others tout the maturation of integrated business intelligence suites. But beneath all that is a powerful undercurrent.

• By Eric Kavanagh
• 12/21/2005

In his seminal poem, The Waste Land, Thomas Stearns Eliot captured the imagination of generations by integrating a world of history, language, hope and despair into a precious paradox of symbol and sign. Oddly enough, the most poignant part of this poetic masterpiece is rarely referenced, even by college professors:

PHLEBAS the Phoenician, a fortnight dead,
Forgot the cry of gulls, and the deep sea swell
And the profit and loss.

A current under sea
Picked his bones in whispers. As he rose and fell
He passed the stages of his age and youth
Entering the whirlpool.

Gentile or Jew
O you who turn the wheel and look to windward,
Consider Phlebas, who was once handsome and tall as you.

So, in the world of enterprise technology, who—or what—is Phlebas?

Consider this commentary from John Senor, president of iWay, the integration arm of Information Builders: “A year ago, people were asking about it. Today it’s in full bloom. And in two years it will be pervasive. You just won’t do things any other way.”

Compare that prediction to the foresight of David Clarke, senior vice president of products for Cape Clear: “This is a business bigger than the application server segment. Could be as influential as the database over time. If there’s a new Oracle, this is the space.”

Compound those statements with a widely publicized memo, reportedly sent by the Big Kahuna himself, Bill Gates, on the eve of Halloween: “The next sea change is upon us. We must recognize this change as an opportunity to take our offerings to the next level, compete in a manner commensurate with our industry responsibilities, and utilize our assets and our broad reach to reshape our business for the benefit of the users of our products, our customers, our partners and ourselves.”

Phlebas, then, is the enterprise software industry as a whole; business intelligence and data warehousing included. The development, so to speak, is the triggering of a whirlpool called the service-oriented architecture (SOA).

Anatomy of an SOA

What exactly comprises an SOA? MicroStrategy chief operating officer Sanju Bansal says that’s a very good question. “First, I’d say there’s a lot of confusion. No one has defined it in a clean, clear way; so all the vendors are picking it up as a buzz word. Whether or not they have it…” Bansal also questioned the tangible value of an SOA: “It’s unclear what benefit an end-user customer has, if an SOA is in place. There’s lots of theoretical benefit.”

One commonly accepted definition (as evidenced by its replication around the Internet) appears in an article entitled, Delving into Service-Oriented Architecture. Authors Bernhard Borges, Kerrie Holley and Ali Arsanjani offer the following definition (SP stands for service provider, SC for service consumer):

“SOA is the architectural style that supports loosely coupled services to enable business flexibility in an interoperable, technology-agnostic manner. SOA consists of a composite set of business-aligned services that support a flexible and dynamically re-configurable end-to-end business processes realization using interface-based service descriptions. The inherent, salient advantage of today's service descriptions is the decoupling of the SP and SC through open standards and protocols, and the deferment of implementation decisions from the SC to the SP. Moreover, individual or collections of services that enjoy various levels of granularity can be combined and choreographed to produce ‘new’ composite services that not only introduce new levels of reuse but also allow the dynamic reconfiguration of business systems.”

Senor of iWay offers a 30,000-foot view in slightly simpler terms: “SOA is obviously a big topic. The whole idea behind SOA is that we recognize—or have come to recognize—that as we begin to scale out any enterprise architecture, the large-scale monolithic model does not work well. That’s fundamental to understanding SOA. It’s more about a distributed type of architecture. Really, it mimics the way human beings and business processes work: distributed response; levels of aggregation; all roll up into the business processes. [SOA] is a step toward the realization that the way software should be architected, should mimic the way people work.”

Deep Impact?

Senor’s words are mimicked by Alex Rosen, who manages the SOA practice for consultancy MomentumSI, which employs 50 full-time consultants, and an equal number of 1099s. When asked how the enterprise software industry will change, he didn’t mince words: “It’s going to change dramatically. What we’re seeing with SAP and NetWeaver, is that essentially they’re taking their monolithic product and re-architecting it into a set of services and processes, and delivering them to the enterprises in a way that will allow customers to much more easily customize such a major industry application.”

Hanns-Christian Hanebeck, director of enterprise applications at GlobeRanger Corp., an RFID concern, offers a similar assessment of SAP’s activities: “They’re fully embracing SOA. They have over 200 business processes available for subscription to NetWeaver. By 2007, it’ll be 700… SAP is forcing the industry to move along.”

Rosen went on to address the bottom-line impact of SOA on software vendors: “When you think about that, breaking monolithic applications into smaller services, that could potentially have a major impact on how you buy your enterprise software, because you may be able to buy it in a smaller granularity to just perform the pieces that you need. And there’s also the whole concept of software as a service, which we believe is a wave that’s picking up steam. You see that with Salesforce.com, which has focused on small to mid-sized business. But now some larger enterprises are starting to make use of that. What’s the impact of that type of solution? It’s huge.”

Hyperion chief technical officer John Kopcke says the pay-as-you-go model remains too problematic to have any impact on Hyperion’s licensing model, however: “…software in this case is a service on your computer delivered via the Internet; but there’s still a complex set of software to manage all that, and it becomes more difficult because you have lots of detail you must track from a usage perspective. Organizations don’t like you reaching in and pulling out that kind of data. It gets CIOs kind of twitchy as you go in to read the meters, so to speak.”

Plenty of other analysts are quick to offer caveats. Philip Russom, senior manager of research and services for TDWI, says that while SOA does represent a sea change in the broader application market, its footprint is less pronounced in the data-oriented world of BI and DW. "Data warehousing professionals today use Web services opportunistically here and there, treating each service like just another interface. It's nice to have services as a new option that complements older interfaces -- and this is a great place to start using services -- but this kind of isolated service is not an architecture by any stretch of the imagination."

On the other hand, Russom notes there are distinct advantages to SOA. "If you think about how the layers of a BI technology stack interact, it resembles a composite application. And many companies need these pieces to interoperate more directly, as well as integrate with operational applications, to support right-time and on-demand business practices. To plug BI into more composite apps and to enable a greater diversity of information delivery speeds, many companies will eventually turn to SOA."

SOA, DW and BI

iWay’s Senor is very animated about the impact of SOA on the realm of reporting. “Fundamentally in SOA, there’s probably a set of core reports, and more importantly, methods for creating those reports, that you’d like as much as possible for everyone to use. It’s the single version of the truth at the base level of your reporting architecture.”

How will SOA affect data warehousing, specifically? Based on his comments, it would appear that Senor sees a future of enhanced enterprise information integration (EII)—possibly to the detriment of the traditional data warehousing world:

“Customer information does not exist in a single place. In previous architectures, someone derived a view of a customer, but the time value of data warehouse data is sometimes old; so it costs a lot of money to set up and maintain a data warehouse. Now, we want to do the same thing, but rather than creating a data warehouse, loading the database, let’s recreate a process: I get customer record from DB2, combine that with SAP, combine that with call center info from Siebel; combine that with the problem ticket reports in my own home-grown case reporting system—all of this info from my sources, I aggregate and format it in a pretty way, and I expose that with a method called GetCustomerInformation.”

He continues: “By doing that, and doing it that way, what I’ve done is I’ve first of all created a single, common, governance-able, auditable means for producing customer information in the context of service. That means anybody in the enterprise, any application in the enterprise, whose need for customer info has a service-oriented orientation, providing some kind of service to a customer—I can reuse that real-time process simply by calling the service.”

TDWI's director of research and services, Wayne Eckerson, comments: “On the data side, I think SOA can only make a big impact in BI/DW if companies get their operational data squared away first. If they do that, then distributed queries can really start to hum. However, there will always be a need for a historical archive to analyze historical data and a real-time cache to avoid bogging down operational systems.”

And Senor makes no mistake about tying reporting accuracy to the increasingly influential world of compliance: “The purpose of reporting as a service is to decide what are the core pieces of information or core derivations of data that we want [in order] to be consistent. So we need a set of reusable report services that can be incorporated into any application, so that when and if we produce reports, we all do it the same way, documented and auditable for compliance.”

SOA Software?

While SOA itself isn’t an application, but more of a methodology or architecture, there is one key product space taking shape: the enterprise service bus (ESB). Cape Clear’s Clarke explains the role of an ESB, noting that its “function is to provide the backbone on which you can build a SOA. It handles all of the service definitions, service creation, integration, and deployment and management. It deals with the whole lifecycle of building, deploying, and managing services; your ESB handles all that stuff. It acts like an application server, but for services, not just for Java programs. It’s like a descendant of the application server, lots of the same types of capabilities.”

Clarke says most of their customers these days are big enterprises. “We had thought early on it would be small companies. Really, we have very few companies less than 500 people. In market segments, we see different profiles between North America and Europe. In America, we do lots of work with states and local governments; Wisconsin is our most prominent client: they’ve used us as their backbone for their architecture, saved lots of money. And we’re engaged with lots of other state, county and city governments.”

Depending where you look, ESB isn’t referred to as a product. The IBM Web site defines it accordingly: “An enterprise service bus (ESB) is a pattern of middleware that unifies and connects services, applications and resources within a business. Put another way, it is the framework within which the capabilities of a business' applications are made available for reuse by other applications throughout the organization and beyond.”

MomentumSI’s Rosen notes that IBM has recently changed its position, however. “In that arena, we had until recently, IBM saying ESB is a pattern not a product; but then within the last few months, they changed their tune and now provide a product that’s called their ESB.”

Indeed, elsewhere on the IBM site, you’ll find the following product descriptions and commentary (note use of the word “true”):

Market Penetration

Gauging overall market penetration is difficult, especially in light of how fluid the ESB evolution remains. However, significant signs show a gathering storm. The Yankee Group has issued survey results based upon interviews with 300 IT personnel from a range of industries. The findings show some remarkably high numbers for SOA adoption in the coming year. According to the report, “2006 will be the year of initial SOA project completion on a broad basis -- not on a hit or miss trend, but through a rising tide of broad and deep adoption of SOA across the market.”

The survey reveals some startlingly high numbers for SOA adoption in 2006: wireless (93%), retail (92%), financial (89%), manufacturing (76%), and government (75%). TDWI also conducted an informal, Web-based poll asking companies about their SOA strategy: a full two-thirds of 68 respondents said they’re either fully embracing SOA for their enterprise architecture, or are at least seriously investigating SOA.

Of course, one man’s adoption is another man’s dabbling. MomentumSI’s Rosen: “Of those who say they’ve fully embraced SOA, I do not believe that it’s that large. There are many companies who say they’re doing it, and have begun an initiative, but it runs into some challenges. Six months or a year into it: have we fully achieved what we believed? The problem that is most often the case is a failure to look at what you’re doing from a strategic point of view, and looking at all the various things that this new approach can affect. When you think back to object-oriented approaches, in that transition, you’d get a survey result of lot of people saying yes, but it took training, mentoring, a lot of changes in the software development methodology, before you had mass adoption.”

Still, real-world examples are cropping up. Take Cullen/Frost Bankers Inc., a financial holding company based in the Southwest. At TDWI’s World Conference in San Diego this year, Louis Barton, their director of data warehousing, said his enterprise is ready to use Web services to handle one-fifth of what they’ve traditionally purchased via licenses from BI vendors in 2006. He said that will begin to impact BI license expenditures. Twenty percent at a mid-sized bank is no small potatoes; rather, it sounds more like a clarion call.

So far, major BI vendors aren’t expressing any concern. When asked if Hyperion was seeing any erosion of license revenue in its BI stack as a result of SOA adoption, CTO Kopcke stated flatly: “Absolutely not.” However, he did extol the value of SOA, and when asked whether his development team had embraced a service-oriented approach, he explained: “In earnest across the enterprise, we started that four years ago. I’d describe us as being in mid-stride with that. It’s not something we’re starting now; we’re ahead of game.” He went on to note that Hyperion’s architecture group makes “sure all functional areas in System 9 conform to the architecture; and the architecture is all SOA in nature.”

1. Recognize that SOA is a philosophy, a best-practices methodology; it’s how you go about doing something, not just what you do;
2. Secure a very high-level sponsor; it’s about collaboration, not monolithic implementation, so having organizational buy-in and top-down support is extremely important, or it’s not going to succeed;
3. SOA is an evolutionary, not revolutionary implementation; if you do it right, you won’t have to rip and replace anything.

Ah, but there’s the rub... or will be. With dashboards firmly cemented as the point of interface between humans and the enterprise architecture, and with a services-oriented, loosely coupled engine under the hood, ripping and replacing specific applications (read: services) will no longer be a painful process. In well architected infrastructures, it will become all but seamless. Once this happens—whether it’s one year hence or five—look for price points to drop precipitously on any number of applications, across all industries. By then, Phlebas had better learn to swim.

Developing...