In-Depth

Personal Data Warehouses: Challenging a Single Version of the Truth

The data warehouse might not survive a radical re-definition of "single version of the truth."

The data warehouse might be dead. If not now, possibly soon.

That fleeting thought crossed the minds of two data warehousing heavyweights at TDWI's recent World Conference in San Diego as they listened to a Bank of America executive argue for a "new class of service.".

Larissa T. Moss, co-author of BI Roadmap and a regular TDWI instructor, organized "Peer Networking: EDW vs. PDW"—enterprise data warehouse vs. personal data warehouse—after she heard rumblings of several tools she fears could weaken the enterprise data warehouse's dominance.

One such tool is Alan Cherdak's Personal Data Warehouse. It operates on a PC and essentially hot-wires operational data sources while semi-automated functions clean and align data. It repulses Moss the way Alice Waters of Chez Panisse fame might react to a greasy cheeseburger. "I totally reject his use of the term 'personal data warehouse,'" she said.

Just five people showed up. It was the show's final night, long after most attendees had gone home. Almost by chance, one articulate business intelligence practitioner took part whose resonating voice spoke for more than just new tools. He advocated a radical redefinition of the single-version-of-the-truth concept.

"To most folks," explained Bank of America senior vice president of finance operations Norbert Sonner, "a single version of the truth means that you get all this data together and create one semantic layer." He stood between a newsprint easel and his four seated listeners and drew a chart: complexity on one axis and needs on the other. "What has happened is this gap has formed." Far off on the side, at the extremes of complexity and range of needs satisfied, he put the enterprise data warehouse. The "gap" covered most of the rest.

"Every asset has a cost," he said, and the data warehouse's cost may be too high for some needs.

He sees another possibility: one in which some data comes from the data warehouse --while other data comes straight from a system of record through a semantic layer. "We might go to that paradigm that says, 'Don't even bring it into the data warehouse—just go and get it wherever it lies.'"

What's important, Sonner said, is that the people who need to communicate with each other can talk about the same data. "That's why we seek one version of the truth." For example, party A talks to party B about the same data, but that doesn't mean that party C cares. In fact, C talking about same concept may talk to party D, and they may use a different source of data.

"To my mind, where we are today, that's OK," he said. "What's important is that A and B talk together and that C and D talk, but if A never talks to C, that's not so important."

"One version of the truth now means that everybody speaks with the same data," he said. "Maybe too much. Maybe that's not the right business decision."

Sonner claims huge financial savings. He calculated, for example, that one 20-day project eliminated an opportunity cost of roughly $21 million. Other projects around the bank have received tremendous notice, he said, for saving one-tenth as much.

He also builds solutions for his internal clients quickly. He described one that he helped develop, and Moss remarked, "You probably did it in days or weeks, not months or years."

Short-Term Thinking

Sid Adelman, also a regular TDWI instructor and a co-author with Moss of Data Warehouse Project Management, listened but was skeptical.

"Norbert seems to be successful," he told me by phone a week later, "but is it good for the whole organization? Maybe not. There's lots of sub-optimization going on … lots of redundancy and inconsistency."

"It's short term mentality," he said, "People get used to getting their stuff this way." He compares them to World War II temporary housing that in many areas is still in use. "Some things just don't go away."

These quicker and cheaper arrangements also undermine support for the more efficient and valuable enterprise data warehouse.

"It takes strength and power and commitment" to build a data warehouse, he said. Without the long-term view, it's not going to happen.

Moss wrote in an e-mail, "I think Norbert definitely has a point with the 'new class of services,'" but she questioned how IT departments could provide the service, already stretched as most resources are.

As it happens, Sonner is looking outside of IT.

"Find me a tinkerer," he said, recalling a requisition to his human resources department. "Give me someone who likes to look under the hood—a soup-to-nuts person." Such a person makes prototypes, works with clients, hears requirements, tests, and writes documentation. Such tinkerers have held a variety of positions and "carry with them a wealth of knowledge on how the business really works."

He avoids IT when possible. A "plug-and-play" mentality, fed by globalization and off-the-shelf software, has displaced creativity, he said. In IT he finds "arterial sclerosis"—a term Adelman and Moss picked up and used.

Moss agreed that IT's role could be partly absorbed into the business units eventually—just not yet. They're not ready.

Ready or not, she said, business will continue to "play in the sandbox" while IT looks on, knowing the business groups' projects won't produce enterprise-wide assets.

"What can possibly be done about that? [The business units] have the tools, they buy their own servers, the CEO and COO support them," she said. "Not a thing IT can do."

To avoid having data chaos rain down on them in a post-EDW world, she said, business and IT have to do several things: get master data on the operational side under control, ensure transaction records are unique and "certified," and set up a BI center of excellence to ensure that data and processes are reused efficiently.

Ultimately, she said, business will have to adopt many of the same controls IT uses now.

"It's just like when PCs came out: totally uncontrolled, no security, coffee spilling on the computers and ruining the processor." Once they "hurt themselves badly enough," they'll introduce controls. "My guess is they'll go and reinvent the wheel."

The conversation wound up after an hour and 15 minutes with Sonner, Moss, and Adelman arriving at a new page on the newsprint easel—a hypothetical model of where things could be headed: On one side is "IT" with two circles: transactions and master data management. On the other side of a red line is "business" running business intelligence. Data flows up from the system of record, perhaps then through an enterprise information integration layer, and into BI. There's no data warehouse in sight.

"I'm not sure the Norberts of the world can be stopped," wrote Moss in e-mail a week later, "and I'm not sure they should be."

Must Read Articles