Microsoft Meets Oracle Challenge (Sort of)
Larry Ellison, Oracle Corp.’s CEO, has a reputation for talking big. But during his keynote speech at Fall Comdex ‘98 in Las Vegas, he put his money where his mouth was. Ellison challenged the IT community at large -- Microsoft Corp. included -- to run a standard business query using Microsoft SQL Server 7.0 and a 1 TB TPC-D database at a rate better than 1 percent of Oracle's best published performance.
"If you can make SQL 7 run no less than 100 times slower than Oracle8i, we'll give you a million dollars," Ellison challenged. Four months later, and after Oracle officially ended the challenge, Microsoft has issued a response by publicizing results on SQL Server.
In mid-March, the software giant posted a benchmark result -- although not based on the standard TPC-D query 5 test -- of 1.075 seconds in executing what the company characterized as an OLAP-based solution that met the original intention of TPC-D.
As it turns out, Microsoft didn’t quite meet the letter of the Oracle CEO’s challenge -- but may have claimed a moral victory in the process. Although Ellison threw down his gauntlet in November 1998, Oracle recently publicized a new benchmark result -- submitted to the Transaction Processing Performance Council (TPC, www.tpc.org) Feb. 12, 1999, and posted on March 23 -- of 0.7 seconds on the standard TPC-D Query 5 test.
To meet the requirements of Ellison’s challenge, then, Microsoft would have had to demonstrate TPC-D query 5 performance of less than 70 seconds using its SQL Server 7.0 database.
According to Microsoft, it chose to eschew the TPC-D query 5 benchmark because it says the record-holding Oracle8i database is effectively playing with a loaded deck in TPC-D query 5 testing by virtue of its use of a new database technology called a "materialized view." A materialized view is essentially a software mechanism implemented at the database level that allows it to pre-aggregate data for a specific query or queries. In many environments, materialized views serve a vital purpose. The technique allows database administrators to predefine an aggregation of commonly accessed data to make it available for end user queries, thereby significantly reducing response times.
SQL Server 7.0 does not support the use of materialized views; consequently, Microsoft could not have duplicated Oracle's TPC-D query 5 benchmark result. Barry Goffe, Microsoft’s SQL Server product manager, says materialized views will be supported in the next version of SQL Server, code-named "Shiloh."
But the very notion of materialized views runs counter to the ad hoc nature of the TPC-D query 5 benchmark, which measures a database’s performance in dynamically aggregating data from across its disparate pages, rows and tables.
"The goal of that benchmark was to be able to make apples to apples comparisons in terms of ad hoc query results, and materialized views are simply not appropriate to an ad hoc environment," Goffe maintains. "[Because of this,] the TPC understands that the TPC-D benchmark is in dispute, and they’ve actually proposed benchmarks to replace it."
The performance benefits that accrue as a result of the use of materialized views in TPC-D testing is undeniable. In April 1998, for example, Oracle’s high-water mark on the TPC-D query 5 benchmark was 1,315.5 seconds, achieved on a Sun Ultra Enterprise 10000 with 64 336-MHz UltraSparc processors and 64 GB of memory from Sun Microsystems Inc. By February 1999, the same TPC-D query 5 test -- executed this time on an Ultra Enterprise 10000 system that now featured 64 400-MHz UltraSparc processors and 60 GB of memory -- clocked in at .75 seconds.
For its part, the TPC essentially agrees that materialized views degrade the benchmark. In the March 1999 edition of its TPC Newsletter, the organization notes, "The problem [with the use of materialized views] is that TPC-D was intended to represent an ad hoc environment in which queries are submitted on a random basis and are not known in advance." As a result, the TPC proposed two alternate benchmark tests, TPC-R and TPC-H. TPC-H, specifically, would restore the "ad hoc-ness" of the TPC-D 2.1 benchmark.
While Microsoft may not have met the letter of Ellison’s challenge, says Mike Schiff, a principal analyst at Current Analysis (www.currentanalysis.com), it appears to have staked out a firm position with regard to the "ad hoc-ness" of the TPC-D benchmark.
"Oracle will claim that there’s a business reason for [the use of these materialized views], but it looks like what they’re doing is putting a lot into the preparation and loading of the database," Schiff comments. "Microsoft has apparently taken the position that these are supposed to be ad hoc queries and [it] has tried to demonstrate the ability of SQL Server to solve these real types of business problems."
With regard to the validity of his company’s undocumented and nonstandardized ad hoc query testing solution, Microsoft’s Goffe is unequivocal. "We had a third-party analyst audit the benchmark for us, so he did standard things like confirm the size of the database and confirm that the query we were using was similar in nature to the query that Oracle was using," Goffe explains. "He also confirmed that we weren’t using memory cache but were in fact reading the data off of the disk." The third-party analyst was Richard Winter, president of Winter Corp. (www.wintercorp.com), known for his VLDB (Very Large Database) Survey Program, which identifies the world’s largest databases.