Smart Ways of Managing Data: Data Warehousing and Business Intelligence

Data. It is accumulating so rapidly, and in so many different sources, that organizations are struggling to consolidate the information and provide access to it across the enterprise and within the supply chain. UNIX remains the number one platform for data warehousing, and HP-UX servers provide the scalability needed to run BI/DW solutions.

• 05/01/2000

No one’s complaining about a shortage of data. In fact, data is accumulating so rapidly, and in so many different sources, that organizations are struggling to consolidate the information and provide access to it across the enterprise and within the supply chain. One way of dealing with data consolidation and access is, of course, by creating a data warehouse (DW). DWs offer a means of collecting, cleansing, transforming and storing transactional and external information in a structured format for query analysis and reporting. But, they are just one tool for managing data.

Thanks largely to the Internet, the nature of data has changed. Today, about 90 percent of data exists in unstructured formats – in word processing programs, e-mail and so on. That’s one reason why organizations are looking beyond data warehouses to business intelligence (BI) software and solutions. BI allows organizations to address specific business questions by accessing and analyzing information from a DW, or from other structured or unstructured data sources.

In any case, BI/DW presents huge opportunities – Survey.com reported that BI/DW became a $25-$30 billion market worldwide last year. While IBM remains the dominant player here, offering a complete mix of hardware, software and services, HP seems poised to capture substantial market share.

One of HP’s big advantages is hardware. UNIX is still the number one platform for data warehousing, according to IDC, and HP-UX servers provide the scalability needed to run BI/DW solutions.

HP is also taking advantage of its partnering expertise to forge new alliances in the growing enterprise information portal (EIP) space. Portals are essential to BI because they bring intelligence to "the so-called masses in an organization," says Thomas Wuerz, HP’s Business Intelligence Program Manager. Because of partnerships it has forged with vendors, like Cognos, Brio and Viador, HP can offer EIP solutions that allow information previously available only to a few specialized employees to be distributed and accessed throughout an organization.

In perhaps its most interesting – and certainly its most innovative – initiatives, HP is developing BI solutions predicated on e-services. Portals also play a central role here, as does e-speak, HP’s brokering technology for dynamically creating e-services. HP thinks it’s on to a good thing with e-intelligence. "Our e-intelligence initiative isn’t just business intelligence with a Web interface slapped on it," says Jennie Grimes, HP’s Director of e-Intelligence and Portal Solutions.

As HP moves forward on different fronts in the BI/DW space, partnering remains a core strategy across initiatives and programs. Take, for example, the HP Open Warehouse Program, which was launched in 1993, and by 1998, had evolved into the HP Business Intelligence Program. This partnership program includes major initiatives in the areas of ERP/CRM, high-end data warehouses and EIPs.

"When HP talks about ERP, there’s a strong focus on SAP and the SAP Business Information Warehouse (SAP BW)," Wuerz says. SAP BW is the ERP vendor’s initiative for BI solutions in an R/3 environment, as well as the underlying piece for new SAP initiatives, like mySAP.com (its portal solution), CRM applications, and procurement solutions.

BI solutions for a SAP R/3 environment are a natural for HP, which boasts more than 1,000 installed SAP BW solutions and more than 6,000 installations running R/3. "HP’s value proposition in this market," Wuerz explains, "is a combination of hardware, storage, tools and services to help businesses obtain vital reporting and analysis out of their SAP data."

HP delivered on that business proposition by unveiling the Impact Analysis for SAP BW. The analysis shows businesses how SAP BW can complement a current or planned data warehouse architecture, and identifies and costs all the resources needed to implement and operate SAP BW successfully. HP also developed a configuring and sizing tool, called CAST, that, Wuerz says, is used to "effectively reduce the risk involved in sizing a SAP BW solution, while making it easy to estimate the hardware needed."

HP also offers the Certified Data Warehouse program for SAP. Forged out of an alliance between HP, Informix and Cognos, the program is designed to reduce the time it takes to build an SAP BW solution. Program partners test and certify SAP BW solutions for high-performance warehouses in three configurations – small, medium and large.

HP knows all about the need for large data warehouses. In one huge deal, HP agreed to build a 40-TB data warehouse for Cap Gemini over two years. The telecommunications company will use the warehouse mainly to analyze call detail records (CDRs) for the purpose of determining call behavior trends.

The Cap Gemini deal isn’t an anomaly in the market. More and more businesses consider data warehouses in the multi-terabyte range a mission-critical necessity. In fact, IDC predicts that by 2001, 5 percent of the data warehouses installed will hold more than 16 TB of data, and 20 percent will hold more than 2 TB.

Again, HP is relying on partnerships to serve this market. An essential partner is Oracle. Toward the end of 1998, HP and Oracle launched the Terabyte Plus initiative to build warehouses that can be accessed by hundreds to thousands of enterprise and Internet users. The two companies’ combined offerings are designed to "look as though they come from only one company," Wuerz says.

Under terms of the partnership, HP and Oracle integrated HP 9000 data warehouse servers with the Oracle8i database and established an Enterprise Technology Center in Atlanta to help customers design and deploy multi-terabyte data warehouses. The partnership was also marked by an industry "first" when the companies aligned their separate support organizations in the Service Integration Program.

HP’s interest in the EIP market is fueled by the two big advantages of these portals. First, EIPs allow for wider distribution of information within an organization. And second, their use of search engines allows them to offer a means of analyzing unstructured data – the data residing outside databases, data warehouses, ERP systems and data marts.

"All of the BI vendors are moving into the EIP world to add collaboration or the capability of analyzing unstructured data to their offerings," says Wuerz, who points to Cognos, Information Advantage (the OLAP software vendor acquired by Sterling Software last year), and Viador as examples.

Viador, in fact, is a major HP partner in the BI space. "About 80 percent of Viador’s business today is aimed at employees in companies who are analyzing data sources that exist within the company," says Wuerz. "That’s very different from other portal trends, but something all other EIP vendors are morphing into."

Viador’s E-Portal Suite, which has embedded within it Infoseek’s Ultraseek Server enterprise search application, allows an organization’s users to access and analyze information across databases and documents. Last October, HP and Viador forged a strategic alliance to deliver an online implementation service that enables the rapid deployment of an EIP.

HP took great strides to move out from its more traditional offerings in the BI/DW space last December when it launched its e-intelligence initiative. The company began talking about e-intelligence, says Jennie Grimes, HP’s Director of e-Intelligence and Portal Solutions, "When we tried to see what the Internet was going to do to the traditional business intelligence world. And we figured out that there wouldn’t be just an evolution. A revolutionary set of activities would emerge." The Web interface stage is just the beginning of entry into the e-intelligence world, Grimes says, although she expects brick-and-mortar, or chapter one Internet customers to ease themselves into the e-services world by taking what is "almost a first-generation step."

"When we look at the end state of e-intelligence, what we see are a couple of key things that tell me it’s a functional shift than rather a little step-wise movement," Grimes explains. "What we realized first was that in this e-services world, as our customers try to figure out what to sell, how to sell it, to whom to sell it, for how long to sell it, and at what price to sell it – all of that is fueled by intelligence. All of it is fueled by my supply chain information, and that’s fueled by my customer relationship information, and that’s fueled by my current business performance information. And, we said, ‘Look, that’s intelligence seeding e-services.’"

HP realizes that’s not how BI is viewed now. To most businesses, BI is seen primarily as a downstream reporting activity. In an e-services world, though, everything is dynamic. And BI is an inherent part of every transaction.

"Look at it this way," Grimes says. "Every transaction has intelligence embedded in it, and every transaction had intelligence that drove it." Companies already sell intelligence – psychographic and demographic profiling, for example. "But," Grimes says, "what if I’m selling the clickstream analysis or what if I’m selling the customer profiling in the pharmaceutical segment? What if I’m selling the data extract capability? What if I’m selling the data mining capability or the cleansing or the scrubbing – all the parts of a traditional data warehouse?" At this point, intelligence becomes an e-service.

It’s one thing, of course, to talk about e-intelligence, and quite another to deliver on it. But, HP is pulling together a lot of the pieces that make the initiative seem real and compelling.

One of these pieces involves ISVs, who play a major role in HP’s e-intelligence efforts. Last December, HP launched the E-Intelligence Partner Program, which initially embraced more than three dozen ISV partners, including Acta, Hyperion, Infoseek, Informatica, e.phiphany, PeopleSoft and SAS. The new partner program welcomes traditional, or brick-and-mortar, ISVs, like SAS and Oracle, which have business models built on software license transfer.

For example, HP announced that it had collaborated with SAS on an eCRM knowledge portal, leveraging SAS’ decision-support technologies. The knowledge portal combines SAS data mining solutions and services with HP hardware, support and consulting to educate customers on conducting customer evaluation, campaign analysis, and segmentation profiling.

Dotcom ISVs, companies just getting started in the BI space, are also embraced by the E-Intelligence Partner Program. HP has, for example, developed an e-service with ShortCycles. The application is designed to replace intranet and extranet sites with a single solution for exchanging realtime information between sales and marketing departments, business partners, and key customers.

The program is also platform-independent insofar as HP platforms go. In other words, ISVs receive assistance in porting applications not only to HP-UX, but also to MPE/iX, Windows NT and Linux.

HP’s ISV partners are expected to embrace e-speak as part of their strategic direction. Brick-and-mortar ISVs must work within product upgrades and release cycles to find the appropriate release in which to embed e-speak. Dotcom ISVs, because they’re just getting started, can snap e-speak into their architecture rather than retrofitting it.

HP e-intelligence partners that are piloting e-speak projects include Ericsson, VHx, Hitel and Jobpartners.com. Ericsson is using e-speak in the area of service repair dispatch on a solution whereby a request for services will be routed to the appropriate technician via cell phone. Healthcare company VHx, just acquired by Claimsnet.com, is using e-speak to enable its healthcare exchange. Korea-based Hitel is using the technology in its portal solutions for online gaming. Finally, Jobpartners.com is using it to develop a recruiting portal.

HP also is enticing ISVs by providing them with a portal through which they can tap into HP’s resources. The portal, which provides information on customer successes, additional solution sets and additional feature functionality, is an avenue open to HP customers, system integrators, sales reps and "really anyone," Grimes says.

E-intelligence reaches far beyond ISV partnerships to embrace new service delivery models (like Viador), financing options, support, storage, ASPs and new intellectual property yet to be developed.

"When looking at financing options with ISVs and, in some cases, with customers, we ask ourselves, for example, if the relationship is with an ASP? Is this a lease space? A click space? How do we fund it? And we get HP Technology Finance involved," says Grimes.

As far as support models are concerned, HP is finding that e-intelligence creates its own support needs. Grimes and her group, for example, are working with HP’s support organization to create a program similar to HP’s recently announced critical systems support program for e-commerce. The new program will map specifically to e-intelligence.

E-intelligence also affects storage and storage support. A business running an ERP e-commerce application may be able to cache off last week’s transactions every day, says Grimes. In that case, 500 GB is a lot of storage. In the e-intelligence world, though, businesses engage in activities like clickstream analysis, causing storage requirements to soar. "Each week Amazon.com generates 500 GB of clickstream data alone, and that’s not counting all the repositories they’re caching off to do customer profiling and trend analysis," Grimes says.

"An HP support contract in a traditional area would concentrate very heavily on the high availability of the server, for example, and MCService/Guard failover strategies," Grimes points out. "We’d offer a support contract that guarantees six-hour repair on servers." An e-intelligence application is an entirely different matter. "It’s actually the storage you’re worried about," Grimes explains. "You can failover your servers, but if you don’t have a good robust disk array or three-way mirroring on the storage side, and you take an Oracle database corruption hit, it can take you 14 to 18 hours to cycle through the corrupted database." In other words, support contracts must be adjusted to focus not only on the servers, but on the kind of storage area network (SAN) strategy used to help keep the storage devices in synch and available.

As HP moves into uncharted territory, driving intelligence as an e-service, it is focusing on a couple of specific areas. One is the notion of a single customer view, or the integration of data from all customer touchpoints. "HP has flirted with this notion for years through key programs like our CRM program and our CBA, or Call Behavior Analysis, program," Grimes says. "One [program] stemmed largely from the financial community, and the other from the telecommunications world, but they were both after that single customer view. Now take something we’re working on now – the virtual single customer view." With the virtual single customer view, businesses may use business process technology and create a ripple effect in which data is distributed not only from a CRM application, but also from a supply chain application, or perhaps some of the operational systems.

HP is also expanding its portal activity for e-intelligence. Along with the EIP arena, the company is pursuing partnerships and solutions for e-commerce and trading community portals.

The company sees yet another opportunity in the BI space. It hopes not only to leverage existing intellectual property, like e-speak, but to foster development of intellectual property that will enable e-intelligence applications. "This is a real opportunity for HP," Grimes says. "We think we’re out ahead of the industry here and able to develop product – actual software product – that we think can uniquely drive some e-intelligence solutions."

About the Author: Jean Nattkemper is the Editor at Large for HP Professional. She can be reached at jnattkemper@hppro.com.