In-Depth

Hyperion Claims OLAP Performance Crown

Company taps little-used APB-1 benchmark to best its own OLAP best result

• By Stephen Swoyer
• 12/01/2004

For the second year running, the Essbase OLAP server from Hyperion Solutions Corp. notched a performance record in standard OLAP benchmark.

The rub, of course, is that—at this point, anyway—Hyperion seems mostly to be competing against itself.

Last December, Hyperion bested Oracle Corp.’s then-record in the OLAP Council’s standard APB-1 benchmark, posting results that were almost 40 percent better than Oracle’s 9i Release 2 database.

The new benchmark tapped Essbase Analytic Services 6.6 to more than double the performance of its predecessor, handling a maximum of 247,524 Average Queries per Minute (AQM)—at an average of 2.4 seconds per query—while supporting 10,000 concurrent users. Last year, Essbase Analytic Services 6.6 recorded 119,085 AQM, with an average response time of 5.04 seconds. Both tests were powered by 64-bit Itanium 2 servers.

For the record, the Essbase 6.6 mark Hyperion announced last week is almost three times faster than Oracle’s best-posted APB-1 result.

A couple of important caveats obtain with respect to Hyperion’s benchmark triumph, however: The OLAP Council, for starters, ceased to exist several years ago, and the APB-1 benchmark has itself fallen into disuse.

In fact, before Oracle dusted off APB-1 in April 2003 to lay claim to the OLAP performance crown, the once-popular benchmark had been all but left for dead. Just how dead? When Oracle resurrected APB-1, the previous record—achieved by a Hyperion Essbase server running on top of a four-way RS/6000 system from IBM Corp.—had stood for five years.

For a brief time, some industry watchers even speculated that Hyperion, Oracle, and other vendors could revive the once-contentious OLAP wars of the mid- and late-1990’s. Alas, it was not to be: Hyperion’s recent APB-1 test is the first of its kind since last year.

Of course, there’s no end of debate about how reliable benchmarks are as indicators of a system’s performance in real-world situations. In the past, some database and application vendors have specifically tweaked their products to exploit the requirements of certain benchmarks.

So given that the OLAP Council is no more and that APB-1 is so rarely used, just what are prospective customers to make of the Essbase results?

According to Mike Schiff, a senior analyst with consultancy Current Analysis, APB-1 is still a reliable indicator of an OLAP server’s performance. Part of the reason is that the benchmark itself is designed to test analytical processing: The AQM metric is based on the amount of time it takes the OLAP server to complete an incremental update to warehouse data and complete a given set of queries, divided by the number of queries in the data set.

Schiff, for his part, believes the latest Essbase result is an impressive achievement, even by the often-questionable standards of the industry benchmark game. “While business performance management and other analytic applications may be getting more attention than OLAP engines, they are the enabling technology for building analytic applications. The APB-1 benchmark results certainly reinforce Hyperion’s claim that Essbase is able to support enterprise-class analytic applications,” he says.

Why Not Essbase 7X?

He also expresses surprise that Hyperion ran the benchmark on Essbase 6.6, instead of the new Essbase 7X. “Since Essbase is now in version 7.1, this raises questions as to why a more current version was not used,” he concludes.

Indeed, Hyperion claims that the new Essbase 7X (which began shipping to select customers in June of this year, and which became generally available in September) delivers a radical performance improvement over its predecessor.

The new version of Essbase ships with Hyperion’s much-promoted Aggregate Storage Option, a technology that addresses an issue (namely, the storage of sparse dimensions in an OLAP data cube) that can hamper OLAP performance in many situations. “The example I usually show is an application where the cube creation time went from around three hours down to just under two minutes,” said Hyperion product manager Srikant Gokulnatha in a September interview. “This gives [customers] a dramatic speed up in performance that is just more significant than anything we’ve ever had before.”

Hyperion did not respond at press time to BI This Week's request for comment.

During the heyday of the OLAP wars, most high-performance benchmarking was done on RISC-Unix platforms, such as Sun Microsystems Inc.’s Solaris and IBM’s AIX. But Hyperion’s most recent OLAP benchmarks have been powered by volume—or near-volume—Intel-based servers from Hewlett-Packard Co. In its most recent test, the OLAP specialist tapped a quartet of HP Integrity servers, each populated with four 1.5 GHz Itanium 2 chips.