Throwing Cold Water on TPC-C Numbers

• 11/08/2000

Enoughalready.

Don’t get mewrong. I’m impressed as all get out with the numbers that Microsoft, withCompaq and IBM, have put up on the Transaction Processing Performance Council’sOLTP benchmark, the TPC-C.

These arethe numbers Microsoft executives talk about every time they open their mouths,usually in a context like, “Not only are we on top when it comes toprice/performance, but we’ve got the best raw performance numbers, too.”

What I saidwhen Microsoft and Compaq first achieved the best TPC-C raw performance resultback in February was that Microsoft had arrived. Competitors could no longercriticize whether Microsoft software scales, only how Microsoft softwarescales.

I stand bythat opinion. TPC-C benchmarks are important scalability indicators, but theyrepresent a utopian dream of unlimited IT budgets and the benefit of being setup and tuned by the people who wrote the software. I’m guessing this is not thesituation in your enterprise.

InFebruary, Microsoft and Compaq topped the list with more than 250,000transactions per minute on the TPC-C test (tpmC). Later IBM set a record atalmost 450,000 tpmC with DB2 and Advanced Server. The Microsoft-Compaq teamresponded recently with a rate of more than half a million tpmC.

Rather thanrunning up the lead any further, Microsoft and its partners need to move toStep 2: Show us customers running scale-out databases on Windows servers.

In caseyou’ve been busy over the last year, let me explain scale-out, which Microsoftalso sometimes refers to as scaling with software. The traditional path todatabase scalability is by installing a bigger server with more processors andmore memory -- a solution that usually involved some flavor of Unix and anOracle or Sybase database. Microsoft plans to pursue that strategy with Windows2000 Datacenter Server and Unisys’ 32-way CMP servers.

Scale-outrefers to ganging up many smaller four- or eight-processor servers and stripinga database across all the nodes so each server handles part of the workload.

ENT hasinvited IBM, Microsoft, and Compaq to point us to customers who are usingdatabases this way on Windows. We haven’t heard back yet. No 24-node customerconfigurations, no 12-node customer configurations, no eight-node customerconfigurations, not even a two-node customer configuration.

Granted,SQL Server 2000 -- which has the Distributed Partition View (DPV) technologythat makes scaling-out possible for Microsoft -- was just released, butMicrosoft first posted a benchmark using DPV back in February. And the companyseems to have no trouble getting partners to make major commitments to theirbeta code. Witness all the major corporations that rolled out both Windows 2000and SQL Server 2000 well before the products shipped.

Microsoftis putting a lot of emphasis on scaling out. For now, I’ll take the company atits word that it is committed to making scale-out databases work.

But beforeMicrosoft and Compaq or IBM go spending $10 million to $15 million on yetanother laboratory system to scale to 1 million transactions or so on the TPC-C,let’s see a couple of real customer implementations of this scale-out stuff.