OLAP Council Releases Revised Benchmark
The OLAP Council has a revised benchmark that the council’s member vendors and industry observers say should provide better information to buyers of OLAP tools.
Analytic Processing Benchmark (APB-1) Release 2 came out Nov. 16 in Boston. An upgrade of the group’s two-year-old original APB-1, the goal of the benchmark is to provide an audited test any vendor can use to compare the speed and performance of its OLAP tool with any other.
"A customer needs to go buy an OLAP technology. They start to build a short list," says Dan Druker, vice president of product marketing at Hyperion Solutions Inc. (Sunnyvale, Calif., www.hyperionsolutions.com). After determining what sort of OLAP tool is required and contacting analysts, the buyer’s next question, Druker says, should be, "How did the vendor do on the benchmark? It’s just about the only metric."
Major changes include a new data generation program that runs on Unix platforms as well as Windows NT, opening the door to larger scale tests. The previous version ran only on NT, and the program limited the randomized data set for the test to about 1 GB. The randomized query generator has also been modified to create queries that run against cells that contain data. Guidelines and disclosure requirements have also been clarified to minimize infighting among vendors.
Robert Craig, an analyst with the Hurwitz Group (Framingham, Mass.), says scalability improvements to the benchmark were key. "One knock on multidimensional databases is a lack of scalability, and one of the goals is to make this benchmark address that."
Increased scalability could also help with another problem, getting vendors to publish their results. Under the first benchmark, results have been published only five times: twice by Arbor Software, which is now Hyperion, twice by Oracle Corp. and once by Applix Inc. (Westboro, Mass., www.applix.com). Nigel Pendse, editor of the OLAP Report (www.olapreport.com), says more vendors may publish because they can compete directly with vendors similar to themselves in the diverse OLAP market.
"There's at least scope for some stratification, with ‘big slow’ [ROLAP] vendors able to compete among themselves without risking being humiliated by nimbler [multidimensional-database OLAP] competitors, who can have a different sprint race," Pendse says.
However results are published under the new benchmark, Dave Menninger, vice president of OLAP product marketing at Oracle, believes the test will continue to spur improvements in the speed of OLAP products. He credits the direct comparison allowed by the original benchmark for helping to make OLAP tools more than 20 times faster.
Arbor was first to publish in June 1997 with its Essbase tool taking nearly 11 hours to perform the incremental load and process 250,000 queries. Oracle and Applix responded with better times. Arbor came back this March with a time of 49 minutes, 6 seconds. "Then the worm [turned], and things come back over to Oracle, and we go, ‘Wow, they did a good job,’" Menninger says. In May of this year, Oracle published a benchmark just above 30 minutes.
The revised benchmark’s release coincided with Microsoft Corp.’s Comdex launch of SQL Server 7.0, which bundles OLAP Services at no extra cost. Steve Murchie, Microsoft data warehouse product manager, confirmed that Microsoft plans to run SQL Server OLAP Services against the council’s new benchmark. He cautioned, however, that Microsoft had not yet seen the benchmark, and that company officials hoped it hadn’t been skewed in such a way as to prohibit Microsoft’s participation.
"If Microsoft decides to publish results -- which is far from certain -- then that will probably force many of the more expensive server vendors to do the same," the OLAP Report’s Pendse says. "It would be even more fun if Microsoft publishes decent results with a reasonably large [i.e. gigabytes] input data set. That would [put] pressure on the others to demonstrate why their products cost 10 or 20 times as much, despite being possibly less scalable."
Members of the OLAP Council include Hyperion, Oracle, Applix, Business Objects Inc., Cognos, NCR Corp., IBM Corp., Pilot Software, Platinum Technology Inc., Gentia/Planning Sciences International Ltd., Management Science Associates Inc. and Kenan Technologies. More information about the benchmark is available at www.olapcouncil.org.