Business Intelligence: Data Warehousing Raison D’Etre
After six years as an industry analyst covering the cutting edge of information technology (IT) trends, I joined TDWI last year as director of education and research. Many colleagues and industry acquaintances were perplexed by my move. Their question to me (whether spoken or unspoken) was, "Data warehousing is not the hot trend it was five years ago, so why are you putting your career in reverse?"
I would politely explain that although data warehousing does not get the press coverage it once did, it has become a mainstream IT discipline. Thousands of IT professionals are making careers in data warehousing. As evidence, I mention that TDWI’s in-depth educational conferences are as packed as ever and its membership rolls growing.
More importantly, I state that data warehousing is the foundation for all the hot new trends currently splashed across the pages of business and trade magazines, including customer relationship management (CRM), one-to-one marketing, personalization, e-business intelligence, supply chain intelligence, "intelligent" B2B portals, and so on.
Rather than chasing the next new IT fad, I add, I am now at the heart of the information revolution and an eyewitness to the birth and development of all new important technology-driven business strategies. Data warehousing, I insist, is just now heating up.
By this time, I have either gained a convert or lost an admirer. Part of the problem is that most of my fellow analysts and vendors tend to be two to four years ahead of the mainstream market. What’s "hot" now will start to become common practice in three to four years. So I figure that I am right on time.
Data Warehousing Creates Intelligent Businesses
I also perhaps have a broader definition of data warehousing than most people. I believe that data warehousing is more than just the process of creating a database geared to support end-user query and reporting activity. To me, data warehousing is all about using information for competitive gain. It involves empowering workers and organizations with information to discover new operating efficiencies, new ways of interacting with customers and supply chain partners, and new ways of doing business.
In essence, data warehousing is all about creating intelligent businesses. Today companies compete based on their ability to gather, interpret, and act on information rather than the inherent attributes of their products or services. In a classroom, we call this ability "intelligence" or "knowledge." The same holds true for business. That’s why many vendors now use the term "business intelligence" or "knowledge management" to describe what they do. To me, these terms are synonymous with data warehousing.
I am not saying there is a data warehouse in every information pot, to paraphrase the late Huey Long. However, data warehousing concepts, tools, platforms, or practices are inevitably used to create all serious information repositories that provide companies a competitive advantage.
Data Warehousing 2001
For example, the article The New Old Economy: Oil, Computers, and the Reinvention of the Earth in the January, 2001 edition of The Atlantic Monthly paints a wonderful portrait of how oil exploration companies are analyzing terabytes of seismic data to pinpoint potential oil deposits within range of previously drilled wells. The article does not mention data warehousing – in fact, it is unlikely that the companies are using "classical" data warehouses to analyze the data, although much data warehousing componentry is probably there, including databases, storage devices, access tools, data mining tools, and data integration and cleansing software.
Among other things, the article describes how modern day oil explorers use advanced 3-D imaging tools running on Silicon Graphics supercomputers to give geologists the ability to "zoom" miles under the earth and look at geologic features from every potential angle. More importantly, the article explains how other firms use simpler data visualization tools running on Pentium class Windows NT servers to deliver the same results albeit with much less pizzazz.
The ability to cost-effectively visualize complex seismic data has helped accelerate the deployment of two other technologies: computer-aided drilling and directional drilling, both of which rely heavily on information to deliver value. The convergence of these three technologies has dropped the average cost of finding new oil has dropped from $12 to $16 a barrel in the 1970s and 1980s to $4 to $8 today. Since oil is much harder to find today than 20 or 30 years ago, these results are staggering.
In short, the oil industry has undergone a sizable transformation. In the face of dwindling supplies of a finite resource, the oil industry continues to prosper and deliver the fuel that drives our modern economy. The article’s author, Jonathon Rauch, writes,
Knowledge, not petroleum, is becoming the critical resource in the oil business; and though the supply of oil is fixed, the supply of knowledge is boundless… Every day the planet becomes less an object and more an idea.
We used to say that "knowledge is power." Today, we also must say that knowledge is the basis of a profitable business.
"Getting Data Out"
When I tell my family and friends who are not in the IT field what I do, I explain that for the past 30 years, we have used computers to capture information or "get data in" for the purpose of recording various types of transactions.
The next 30 years, I explain, is all about using computers "to get data out."— to tap reservoirs of information and knowledge that lay buried deep within corporate transaction systems. Like oil exploration firms, companies that extract, learn from, and act on information will be tomorrow’s winners.
I conclude by saying, "I focus on getting data out, and that is where all the action is."
Wayne Eckerson is Director of Research and Education at TDWI.