In-Depth

Is Your Data Warehouse Dead? (Pshaw!)

As customer data integration rises, will data warehousing fall? Not so fast!

• By Eric Kavanagh
• 11/08/2006

‘This is a test of the emergency data warehousing system. This is only a test. Were this a real emergency, your data warehouse would be dead.’

Sparking concern among the rank-and-file attendees at TDWI’s quarterly conference in Orlando this week, Monday keynote speaker Jill Dyche bravely took the podium as her cover slide silently, yet audaciously proclaimed the unthinkable: ‘Your Data Warehouse Is Dead! The Rise of CDI’

(Gasp!)

Fear not, gentle practitioners: the message Dyche evangelized was one of hope (and a bigger slice of the pie, specifically) springing eternal: “The stuff that we’re doing in our data warehousing and business intelligence teams, should actually transcend the data warehousing platform.” In other words, she argued, the pastures of customer data integration (CDI) are a plenteous green, and who better to graze them than the practiced, polished crews of decision-support trenches everywhere?

So, your data warehouse is not really dead. “It could have the sniffles,” joked Dyche, a founding partner of Baseline Consulting.

Framing the argument for the rise of CDI, Dyche first outlined common challenges with customer data. Unlike the historical data that comprises the nuts and bolts of a traditional data warehouse, customer data is much more volatile, with frequently changing values. Complicating matters, customers can have more than one relationship with an enterprise: as a customer, vendor, partner, even board member, for example.

There are other challenges as well. Various operational systems can have widely ranging views on what, exactly, constitutes a customer. Such systems can include customer relationship management applications, supply-chain programs, order-taking and fulfillment apps—really, any transactional system that an enterprise might have. Furthermore, the real world presents an array of potential hierarchies for enterprise systems, whether financial reporting hierarchies, organizational or product hierarchies. Said Dyche: “Hierarchical relationships are not conducive to SQL.”

Then there’s the timeliness issue. In essence, CDI seeks to fulfill the long-promoted promise of CRM: to maximize every customer touch-point by feeding systems and thus workers with the cleanest, most up-to-date information available about a particular customer. Such right-time data can hardly be expected from the historical-minded warehouse. Rather, a more expedient method must be devised to deliver such data to front-line systems while a customer is on the phone, logged into a Web site, or possibly even standing at a checkout counter in an old-fashioned, brick-and-mortar store.

Baked In, Not Bolted On

On the issue of data quality, Dyche makes no bones: quality is absolutely key. The garbage-in, garbage-out paradigm obviously applies to customer interactions, arguably even more so than with inward-looking processes such as reporting. The reason is because customer interactions are where rubber meets road in any organization, and if customers see an organization’s dirty laundry—whether by receiving a mailer that misspells their name; or by being pitched on a product upgrade for something they don’t own—the net effect will likely be a missed opportunity, if not a lost customer.

Consequently, Dyche stressed the importance of tightly coupling data quality with any CDI initiative. The way to accomplish that, she said, is to implement a CDI hub, which is ideally an actual server loaded with sophisticated data quality software, such as complex matching algorithms, as well as data retrieval and delivery functionality. With such a solution, the data quality component is “baked in, not bolted on.”

The definition for CDI provided by Dyche states that it is “the automation of the integration, reconciliation, management, and certification of customer master data from enterprise systems to enterprise systems.” In other words, a CDI hub is a form of middleware that’s designed for the express purpose of determining and delivering the “golden customer record” to enterprise systems. And CDI, per se, is a subset of the increasingly popular discipline of master data management (MDM).

Dyche explained that an effective CDI hub should incorporate eight core functions, outlined in these layers:

• client interface
• business/data rules
• matching engine
• merge processor
• validation and transformation
• source system interface
• data administration
• system configuration

Because a CDI hub is designed to feed operational systems in near-real-time, that means speed is obviously critical. In fact, Dyche noted that such a hub really should be treated as an OLTP system, as opposed to an analytical system such as an OLAP cube. Therefore, reliability and scalability are extremely important, especially since the hub must have constant connectivity to front-line, customer-focused systems.

Dyche also offered some key recommendations for building a CDI solution. First and foremost, a service-oriented architecture can make implementation and maintenance easier, since CDI is essentially a service provided to enterprise systems. With an SOA, an organization can loosely couple a customer’s golden record to all connected operational systems. On the other hand, the data quality component must be tightly coupled (e.g. “baked in”) to the hub itself.

The good news, said Dyche: “It’s actually pretty easy to configure a CDI hub. You don’t have to really change the source systems, which is a boon to those operational guys who don’t want to be bothered anyway.” In terms of where and how to begin, she recommends that companies “think big but start small. Let’s start with a small project around MDM, and use it as the jumping-off point.”

On the issue of whether or not to jump, Dyche was clear: “MDM is transcendent,” she concluded. “It’s the future.”

'This has been a test of the emergency data warehousing system. We will now return to our regularly scheduled ETL processes.'