Microsoft Resubmits Withdrawn Benchmarks

• 08/16/2000

Benchmark results that Redmond unveiled at the February launch of Windows 2000 had shattered all previous performance records on the Transaction Processing Performance Council's (TPC) OLTP performance benchmark, the TPC-C. But the results were disputed.

In late June, the TPC disqualified the Microsoft numbers. Saying the ruling was over a technicality, Microsoft (www.microsoft.com) vowed to resubmit audited results shortly.

In late July, Microsoft announced that it reran the tests, and once again, the audited benchmarks are posted on TPC's site (www.tpc.org). The announcement came at a meeting of financial analysts where Microsoft was discussing SQL Server.

The company achieved the record benchmark in February with Windows 2000 Advanced Server and prerelease code of SQL Server 2000 on a cluster of Compaq Computer Corp. (www.compaq.com) eight-processor ProLiant servers. Since then, IBM Corp. (www.ibm.com) reclaimed the top spot in raw performance with Microsoft's help using a cluster of 32 Netfinity servers running Windows 2000 Advanced Server on IBM's DB2 database software.

The resubmitted numbers from Microsoft and Compaq are an improvement over the February results, but remain far behind IBM's achievement.

The SQL Server 2000 on Windows 2000 Advanced Server results are important to Microsoft. For the first time, a Microsoft database on a Microsoft operating system was at the top of the raw performance chart: A rank Microsoft products never came close to competing for before. The placement represented a huge jump in status for Microsoft, which had barely broken into the top 10 on raw performance with a Compaq result shortly before February's vault to the top spot.

To get there, Microsoft and Compaq built a complex cluster using a new feature of SQL Server 2000 called Distributed Partition Views. In that architecture, each server in the cluster controls its own piece of the larger database. Microsoft posted two record results for first and second place on TPC-C performance. One configuration used 12 server nodes, the other used eight nodes. Some observers question whether the architecture is feasible for most IT departments to set up and support.

Microsoft got tripped up when a challenge was mounted within the TPC over a rule that covers the primary key of a database.

The primary key is the field that uniquely identifies a record and often links records in different tables of a relational databases. In a personnel database, for example, the primary key might be an employee's social security number. While some database experts contend that a primary key should be something that never changes, the challenge came over the inability of SQL Server to update primary keys in the distributed configuration.

There has been some fallout since the TPC decision. IBM found the primary key interpretation surprising, and it needed to add the primary key update capability to DB2 to get its 32-node configuration submitted to the TPC. In the month since the Microsoft results came down, another 10 TPC-C benchmark results dating back to 1996 have been withdrawn. Those tests had been performed using Oracle and Sybase databases.

Some trade press reports say Oracle was behind the TPC challenge. Microsoft officials declined to comment on the assertion.

In a news release on the resubmission, Microsoft pointed out that no TPC-C result using an Oracle database ranks higher than seventh place on the TPC performance list, and that performance on Oracle systems has improved only 2 percent while the SQL Server system performance jumped 15 percent.

For the resubmission round of tests, Microsoft and Compaq chose to rerun the original systems with their 550-MHz Pentium III Xeon processors for a slight gain in performance over the original results. The companies then ran the benchmark on the systems using the 700-MHz Xeons Intel Corp. (www.intel.com) released recently. The four rerun benchmarks now hold positions two through five on the TPC raw performance list.

[InfoBox]

TPC-C Benchmark Performance
Results in transactions per minute (tpmC)

*Top non-Windows 2000 result on TPC-C raw performance.