In-Depth
New Imperative for Business Intelligence
Major data warehouse providers are seeing an expansion of demand for their services, as well as demand for more value-added capabilities, including the capability to communicate across disparate data marts within an organization. Though some companies started building data warehouses as early as the mid 1980s, the data mining capability is moving away from use by a select few to wider usage across ever smaller companies.
A study by META Group forecasts growth in the "business intelligence population" – suppliers, business partners and consumers – from 400,000 in 1987, to more than 10 million by 2010.
According to AMR Research, a Boston industry analysis firm, total aggregate revenue of the customer relationship market – of which data warehousing is a part – will grow from $1.2 billion in 1997, the last full year for which figures are available, to $11.5 billion in 2002. Software revenue alone will top $7.5 billion by the year 2002, compared to $762 million in 1997.
"Companies have always had all kinds of data on their customers; now they’re beginning to use it," said Peggy Menconi, Research Director for AMR. To get the best and most comprehensive customer data, companies are trying to "use information from all customer touch points, including call centers, the Web and the mail so they can better tailor their products and services to meet customer demands."
According to a Palo Alto Consulting Group study of 400 users and information technology professionals, $113.5 million will be invested in data warehouse and data mining hardware, software and outside services by the year 2002, up sharply from an estimated $6.6 million in 1996. The figure for 1999 was expected to be $36.6 million.
Some of this growth in the use of data is a byproduct of the millennium bug, according to John Riehm, Senior Management Consultant with Grant Thornton.
Though larger companies began installing enterprise resource planning systems several years ago, these systems were initially seen as too expensive for small- and mid-size companies. According to Pine Cone Systems, a data warehouse management company in Denver, the average data mart costs about $1 million to build, while the average data warehouse costs $2.5 million. With Y2K forcing numerous companies to replace their computer systems – including database applications – many have decided now is the time to add enterprise relationship capabilities, and are choosing the less expensive data management solution, at least initially.
Additionally, vendors have been able to show prospective customers proof of the value of these systems. According to an IBM study of 65 data warehouse projects, the mean return on investment, including the cost of building, operating and maintaining the systems, was 401 percent, with the payback period covering about two years.
According to Ben Barnes, General Manager of IBM’s Business Intelligence Solutions Unit, several factors contribute to this outstanding ROI. There are cost savings due to:
• Reduced costs to develop and maintain reports for end users.
• Lower inventory levels from use of just-in-time delivery.
• Reduced transportation and mailing costs.
• More cost-effective target marketing campaigns.
Additionally, there is revenue and profit potential from:
• Increased cross-selling and upselling of products and services.
• Better inventory control, meaning fewer lost sales due to scarcity.
• The ability to charge fees or discontinue customers that are unprofitable. An example is credit card holders who incur large balances but avoid interest charges – income to the card issuer – by paying the balance before the due date.
Another reason for the growth in data marts and data warehouses among smaller companies is that many of the technology vendors are offering more scaleable solutions to fit the different needs of different companies, as well as targeting companies in different industries.
The enterprise resource planning systems give companies much of the information necessary to build data marts – effectively scaled down versions of data warehouses often used in single departments, Riehm said. "This gives them the ability to try a data mart and see how it performs and how they can use it before going with a full-blown data warehouse. Companies have had this data, but haven’t had any way to get at it."
Even though data marts are more focused and less costly than data warehouses, many companies – large and small alike – have installed data marts within different departments without the ability to communicate with each other, according to Bob Butchko, Vice President of Marketing for Pine Cone Systems.
To properly obtain and use the information from an enterprise resource planning system, data mart or data warehouse, the information needs to come from across all sections of the business enterprise. The information must also be usable in different areas of the business (management, sales, human resources, etc). This expansion in usage is leading to combinations of companies to address this need.
"In many ways, we’re going back to where we started from with the data warehouses," Butchko said.
Electronic Data Systems, A.T. Kearney and NCR Corporation plan to team up to offer customers management, consulting and technical capabilities in data warehousing. Data warehousing has matured, according to EDS officials, and most of the future benefits will be in the synergies of the data shared across a company.
"It’s commonplace now for information technology departments to build data warehouses. But very few of these data warehouses allow organizations to share information across the enterprise, let alone across the extended enterprise," said Aaron Zornes, Executive Vice President and Director, Application Delivery Strategies, META Group, said of the partnership, adding that the greatest difficulty is integrating marketing systems with the rest of a company.
Companies are also including the Internet as at least part of their data warehousing solution. The Internet enables a company to push data information and analysis to appropriate parties via personal computers, e-mail and even cellular phones.
IBM’s SurfAid product enables companies to use IBM-developed algorithms to analyze click screens of a company’s Web site.
Oracle announced in January that it would implement the Java programming interface, SQLJ, in its Oracle version 8i database and JDeveloper 2.0 Java programming tool when they are released this quarter. Oracle also released a Web-based version of Oracle Discoverer Viewer, its ad hoc query and analysis tool to access databases, data warehouses and data marts and to create reports.
SQLJ, adopted as a standard by the American National Standards Institute last December, embeds SQL statements directly in Java. This enables Java developers to access a database by writing one line of code, rather than 10 as before, according to Oracle officials.
Some Oracle competitors, including Cognos and BusinessObjects, already had Web versions. In December, Toronto-based Cognos strengthened its offering with the acquisition of United Kingdom-based Relational Matters. Relational Matters’ DataStream software is designed to integrate large volumes of transaction data with different data structures. Like the NCR-EDS-A.T. Kearney partnership, the goal is to enhance reporting and analysis across a business enterprise.
Despite these partnerships and outright consolidations among other companies, these business combinations are more about fixing the problem of non-communicating data silos rather than addressing the problem at the core, according to Butchko. Pine Cone founder and chairman Bill Inmon’s philosophy is to use the data warehouse to populate different data marts. This provides the ability to communicate across an enterprise, with the capability to focus on a specific set (data mart) of information when necessary.
Companies can do this through the use of metadata – data about data – much like a card catalogue in a library. Along with using one of the newest media, the Internet, some companies are also using one of the oldest, text, to provide companies with data warehousing advantages. IBM’s Intelligent Miner for Text can search, analyze, organize and index various kinds of free-form documents, including e-mail messages, memos, reports, notes, outlines, business plans, technical papers, documentation and Web pages. The software automatically generates metadata – structured information about the documents – to turn the mass of text files into information.
While they are now promoting data warehousing and analysis across an entire business, many data warehousing vendors are also becoming more narrowly focused in some areas, as they develop products and services aimed at vertical markets, such as telecommunications, banking and retail.
IBM uses different data warehouse packages aimed at banking/finance, telecommunications, retail, insurance, utilities/energy, healthcare, government, travel/transportation and manufactured goods (see "Data Mining Growth Reflects Strategic Mission" on page 28). IBM in mid-January announced a partnership with Oracle to target the manufacturing industry.
Despite the above developments, Informix Corp. (Menlo Park, Calif.), sees the offerings of IBM, Oracle and the like to be little more than variations built on top of old data information structures. So in January Informix announced a two-year plan to launch new versions of its database servers aimed at customers needing high-end data warehousing, high-performance OLTP and high-volume e-commerce.
Informix officials say the new products will bridge the gap between database servers and application servers. The idea is to help the flow between OLTP and data warehousing applications by Java, ActiveX and OLE DB standards. By 2000, these products will also support an environment in which data can be accessed, analyzed and manipulated enterprisewide regardless of location, data engine, OS or hardware.
In a related announcement, Informix introduced Informix Visionary, a business "visualizer" that provides a portal into all corporate data, a lower-cost way of accessing information. This type of access is needed, Informix officials contend, due to the sheer volume of data – reaching multiple terabytes in some instances – as well as the explosion in the use of the Internet as a means of financial transactions, e-commerce and communications.
In 1998, Informix had acquired Red Brick Systems, with the scaleable Red Brick Warehouse data mart engine, which was incorporated into the Informix Decision Frontier software. By the middle of the year, Informix expects to launch a new Internet extensible data engine that will include application server functionality. This data engine, code-named "Centaur," is based on Informix’s object relational technology. Centaur will integrate server-managed, data with both Java and COM+ server programmability. In the second half of 1999, Informix plans to launch a data engine, code-named "Yellowstone," that will be based on the company’s extended parallel servers and linear scalability.
Informix also plans to offer prefabricated solutions in e-commerce and other areas that will rely on Centaur as the underlying platform.
Riehm also sees the Internet playing an increasingly important role in data warehousing applications in the future. The ability to share information via the Internet greatly reduces the cost of the data warehousing infrastructure. To access data via the Internet, a user needs only a browser and a computer, rather than a server, software and other technology.
About the Author:
Phillip Britt is a freelance writer, specializing in technical analysis and reporting.