Benchmark Variety Reveals True Power of 800 Series
When IBM pushed its new AS/400e servers into general availability on July 28, the line had set world records for three very different benchmarks. In June, a 24-way model 840 captured the VolanoMark 2.1.2 for server-side Java Virtual Machine performance. Then, just days before its official release, IBM announced the same model had raised the benchmarks for Lotus Domino scalability (NotesBench R5), as well as single-server transaction processing (TPC-C).
Running VolanoMark, the 840 handled 108,513 messages per second with 200 concurrent user sessions. For NotesBench R5, it managed 75,000 concurrent mail users with an average response time of 276 milliseconds. And, operating under the Transaction Processing Performance Council's TPC-C stress test, IBM's highest-end AS/400e processed 152,346.25 transactions per minute (tpmC) at $59.35 per transaction ($/tpmC).
| Despite the inadequacies of the benchmarking process, the AS/400e's top-rated standing for Java, Lotus Domino, and transaction processing establishes the new line as a best-in-class series. |
"We felt we needed to make a bold statement to the marketplace, and we've done that," says IBM spokesperson Tim Dallman. Supporting Dallman's sentiment, the 840's VolanoMark performance more than quadruples Sun Microsystems' on a comparable server. Its NotesBench mark sustains 10 times as many concurrent users as Sun's largest audited enterprise server, and five times as many as a similar offering from Compaq. And, based on TPC-C, the 840 processes transactions at a rate 25-percent faster than Sun's 64-way E10000 at just over half the price per tpmC.
Considered in the light of such statistics, the 800 server series is positioned as a best-in-class midrange platform. However, Brad Day, VP, Giga Information Group, says benchmarks should never be taken too seriously. "Any single benchmark can be flawed," says Day, who cites rather broad measurements as reason to maintain perspective when considering the 840's performance under VolanoMark, NotesBench, and TPC-C.
TPC-C, for example, figures throughput (tpmC) by examining how many new orders per minute a system is able to process while, at the same time, executing payment, order-status, delivery, and stock-level transactions. In addition, it calculates price/performance ratios, which attribute a dollar amount to each tpmC, based on a system-wide price.
Due to the speculative parameters under which both tpmC and cost per tpmC are figured, their representative numbers may vary somewhat from those generated within the unique specifications of a particular midrange shop.
An FAQ on the TPC Web site, pertaining to how the council computes price/performance, reads, "TPC benchmarks are system-wide benchmarks, encompassing almost all cost dimensions of an entire system environment the user might purchase, including terminals, backup storage, and three years maintenance cost." This indicates the TPC-C price/performance ratio is computed based not only on the cost of the server, but also on highly-variable ancillary costs. As a result, the computation doesn't necessarily translate into an accurate measuring stick for every transaction-intensive midrange environment.
Michael Famolare, global alliance manager for the Industry Standard Business Server Unit at Compaq says of high-end benchmarking, "we're all just trying to prove we've got bigger, better, faster iron." But, he explains, while benchmarks for high-end servers may paint a nice picture of raw capability, they don't translate to a real-life environment.
Speaking of the 840's NotesBench R5 performance, Famolare believes the ability to support 75,000 concurrent mail and calendar users is irrelevant because a typical enterprise doesn't have a need for anywhere near that capacity. Benchmarks offer buyers a much more accurate depiction of real-life value when run on lower-end systems, says Famolare.
A low-end AS/400 820-2396 running at 70-percent utilization can support 2,570 concurrent Notes mail and calendar users and is priced from $67,000. Meanwhile, a high-end 840, similar to the one on which the benchmarks were run, supports 55,610 concurrent Notes mail and calendar users at 70-percent utilization and is priced in the millions of dollars.
While Giga's Day is hesitant to point to the 840's benchmarking performance as a symbol of Java, Lotus Domino, or transaction-processing superiority, he is not entirely dismissive of the records. "Benchmarks are best used as a process for shortlisting a number of vendors," says Day. He explains that once a buyer has narrowed a search to two or three vendors, then they should request the running of a series of custom benchmarks tailored more specifically to their needs.
In terms of the 800 server series, Day says the benchmarks indicate a propensity for scalable mixed-transaction workloads, mid-tier Web serving or application serving, and compatibility with special purpose application segments like Domino. Day also says, "IBM's selection of benchmarks shows that AS/400 can compete with the high-end Unix systems."
Despite the inadequacies of the benchmarking process, the AS/400e's top-rated standing for Java, Lotus Domino, and transaction processing, IBM believes establish the new line as a best-in-class series. The 800 series combines the traditional reliability of AS/400 with dramatically increased scalability and response time. "The fact that they did a swath of benchmarks and garnered first place in each one…" says Day. "That's the issue here."
Related Editorial:
Benchmark Testing Shows Power of 8xx Series IBM Delivers New AS/400 Models, OS/400 V4R5 IBM Passes Goal of Backing UP 1TB in an HourRelated Information:
IBM AS/400 Division (new window)Volano Java Benchmarks (new window)NotesBench Consortium (new window)Transaction Processing Performance Council (new window)