In-Depth

Oracle’s OLAP Triumph Questioned

Using competitor's four-year-old version makes for "meaningless comparison", analyst says

• By Stephen Swoyer
• 04/30/2003

Oracle achieved the results using a clustered version of its Oracle 9i database Release 2, which features an integrated OLAP engine. Since the introduction of Oracle 9i Release 1, the database giant—which traditionally marketed a separate OLAP tool, called Oracle Express Server—has integrated OLAP functionality into the database itself.

In a statement that attended the announcement, Oracle VP Ray Roccaforte suggested that the new benchmark result amounts to nothing short of a vindication of Oracle’s 9i OLAP integration strategy. “This new world record demonstrates the scalability and performance benefits of having a truly embedded OLAP engine in the database.”

There are a few caveats, of course. First, the benchmark result that Oracle shattered—the OLAP Council’s APB-1 Release II test—hasn’t been used in several years. Moreover, points out Mike Schiff, principal of data warehousing consultancy MAS Strategies, the OLAP Council itself apparently isn’t a going concern anymore. Finally, the APB-1 benchmark that Oracle trounced was indeed achieved by Essbase 5.0.2—in April of 1999.

Needless to say, several versions of Essbase have come and gone since then. Essbase XTD shipped in March 2002, for example, and features parallel processing load improvements.

There’s also the issue of the applicability of the APB-1 Release II benchmark, which was finalized by the OLAP Council in November of 1998.

According to Schiff, Release II of the APB-1 benchmark was designed to help quantify some key performance metrics for OLAP applications. There’s a particular emphasis on analytical processing, for example, which—for the purposes of the benchmark—is understood as the amount of time required to complete an incremental update to warehouse data and complete a given set of queries, divided by the number of queries in the data set. The result, explains Schiff, is the APB-1 metric: analytical queries per minute.

The APB-1’s relevance may be an issue because the OLAP Council is no longer in existence—it was succeeded by the Analytical Solutions Forum, which also appears to be a nonviable concern. In fact, Schiff says, the general phone number that was associated at different times with both the OLAP Council and the Analytical Solutions Forum is still in service—but now rings at a retirement community in Washington state.

Then there’s the fact that Oracle fudged on a similar benchmark in the past—the APB-1 Release I, to be precise—when it conducted a test of its Oracle Express OLAP Server running on SPARC/Solaris. At the time, Oracle sought to demonstrate that Express was a better performer than Essbase, and to that end, trumped the fact that its OLAP server ran the same benchmark 34 percent faster than did Essbase. But while Oracle Express ran on top of version 2.6 of the mature Solaris operating environment, Essbase leveraged the scalability-plagued Windows NT 4.0 operating system. At the time, this platform mismatch prompted Schiff, among other analysts, to speculate that perhaps the real performance crown should go to Sun Microsystems Inc. and not to Oracle.

The upshot, Schiff says, is that the APB-1 Release II benchmark is still a good indicator of OLAP performance. Oracle’s comparison with a four year-old version of Essbase, on the other hand, is far from meaningful, Schiff notes. “It is unfortunate that rather than simply focusing on showing the power of the Oracle9i Database OLAP option in combination with its Real Application Clusters, and thus demonstrating its continuing commitment to data warehousing, Oracle chose to dilute this with a meaningless comparison to a four-year-old prior version of Essbase.”

Instead, suggests Schiff, Oracle would be better served by comparing the OLAP performance of the 9i database to its Express OLAP Server. “Oracle could have done much better by comparing the performance of the Oracle9i Database OLAP option to Oracle Express and highlighting [that] much more ‘Express-like’ functionality is incorporated in Oracle9i Database Release 2, than existed in the initial release 1.”