In-Depth

HP Leapfrogs IBM in Industry Benchmark

Analyst says real performance champ is still the IBM Power4 processor

• By Stephen Swoyer
• 08/05/2003

Although HP’s benchmark provided a rare opportunity to compare the performance of Microsoft Corp.’s new Windows Server 2003 operating system—the reigning TPC-C test champ—with that of a mature Unix operating environment on similar hardware, analysts say that the real TPC-C performance champ remains IBM Corp.’s Power4 processor.

For the record, the HP/Oracle combination notched a score of 824,164 transactions per minute on the TPC-C (tpmC) at a cost of $8.28/tpmC. Powered by the same 64-processor Itanium 2-based Superdome server, HP-UX slew Windows Server 2003’s April result of 707,102 tpmC by almost 15 percent. However, the Microsoft-based result still has the edge on price/performance, with a cost of $7.16 per tpmC.

HP's test favors its Unix variant. The company used PA-RISC servers, not conventional workstations, as clients, and also deeply discounted the Oracle/HP-UX system—to the tune of 49 percent. In the previous benchmark, HP discounted its Microsoft-based test system by 45 percent.

Benchmark Leapfrog

Over the last four months, HP and IBM have gone back and forth for control of the top spot in the TPC-C benchmark.

When HP and Microsoft notched record TPC-C results in late April, for example, IBM struck back in early May, riding one of its 32-way pSeries p690 “Regatta” system to the top of the TPC-C charts. Weeks later (in late May), HP and Microsoft again displaced IBM from its perch, this time on the strength of another 64-way Itanium 2-powered SuperDome system running Windows Server 2003 Datacenter Edition. IBM has proven to be irrepressible, however: In early July, Big Blue again exploited a proven combination—a 32-way p690, AIX, and DB2—to recover first place.

If recent history is any indication, IBM will probably vault back to the top of the TPC-C charts. Between them, Big Blue and HP have moved the TPC-C performance bar significantly—by almost 39 percent—since Japanese computer giant NEC posted a record score of 514,035 tpmC on April 23rd. Since then, IBM has improved its own performance in the TPC-C test by 11 percent, moving from 680,613 to 763,898 in just two tests.

The upshot is that if Big Blue wants to spend the millions of dollars involved in setting up and auditing a TPC-C test, it can probably pass HP once again. “As you can see, benchmarks change on a pretty regular basis as vendors leapfrog each other,” comments Gordon Haff, a senior analyst with consultancy Illuminata. “There’s setting up these different configurations and tuning operating systems to squeeze out more and more performance.”

At the same time, Haff concedes, it’s pretty clear that IBM’s Power4 architecture is the real performance champ in the TPC-C. After all, he points out, IBM’s Power4-powered p690 systems have twice outperformed Itanium 2-powered HP systems with twice as many processors. “A Regatta system using Power4 processors does seem to deliver more performance on a per-processor basis than SuperDome running Itanium,” he acknowledges.

Oracle on Top

The new benchmark also marks the return of Oracle to the top tier of TPC-C results, and the resumption of the three-way battle for scalability bragging rights among Oracle, IBM, and Microsoft.

Interestingly, the published results make Oracle look like the cheaper database when compared with SQL Server. The documentation for HP's new result puts the price for Oracle Database 10G Enterprise Edition at $20,000 per processor for a total of $1,280,000 on 64 processors, plus $6,000 for a three-year support package. But the final price tag takes off an "Oracle E-Business Discount" of $321,500, leaving the total Oracle price at $964,500.

The May result with Microsoft puts the per-processor price of SQL Server at $16,541 for a 64-processor total of $1,058,624 plus $5,850 for three years of maintenance. No specific discount was broken out for SQL Server in that run. Presumably, any standard discount for SQL is lumped into the total discount for that result. A call to Microsoft for an estimate of the standard discount for SQL Server in the scenario was not immediately returned.

Scott Bekker of ENTmag.com contributed to this report