In-Depth
Calculating Storage Value
Companies need some sort of yardstick for comparing the product offerings of different vendors.
I attended a presentation by Rob Peglar, VP of technology for Xiotech, at the University of Minnesota about a month ago in which he explained his company's calculus for measuring value in storage purchases. I liked it a lot because it relied more on open statistical and cost data published by the Storage Performance Council than on any mysterious or proprietary methodology.
The Storage Performance Council (SPC) has been around for about eight years. Their membership roster reads like a Who's Who of the storage industry -- with the notable exception of EMC, which doesn't like its storage performance or value to be measured against a common standard.
That doesn't mean that EMC wares haven't been subjected to testing. A year or so ago, NetApp bought an EMC rig and conducted the SPC-1 benchmark test on it. The results were submitted to SPC for certification, which was granted. This action probably invalidated NetApp's warranty on its EMC platform, since Hopkinton imposes a gag order via its warranty and maintenance contract on the unsanctioned publication of test data about their equipment (as does NetApp, by the way), but it got EMC numbers included in a spreadsheet that enables vendor equipment to be compared on an apples-to-apples basis.
I was almost as tickled by Peglar's jovial recounting of the story as he obviously was in the telling. More to the point, he contrasted the results of the EMC SPC-1 test with several other products, including Xiotech's Emprise products. The EMC Clarion CX3 Model 40 with Snapshot turned in the worst cost of any array tested using SPC metrics: $59.49 per IOP. Without Snapshot, the box delivered a cost per SPC-1 IOP of $20.72. By contrast, the cost per IOP on the Emprise 5000 with 146GB drives was $3.53; it was $4.19 per IOP for the Emprise with 72GB drives. Peglar delighted in stating the obvious, "They are the same IOPS, so why are you paying so much more for EMC's [IOPS]."
Of course, cost per IOPS is not the only measure of value in storage, conceded the speaker. That would be as ridiculous as buying storage the way you buy meat -- by cost per pound. Xiotech has borrowed another metric from SPC to build its business value case, represented by the equation: Capacity x Performance x Resiliency/Cost = Value (that is, CxPxR/$ = V). This is a calculus I would like readers to think about.
Capacity should be presented as tested capacity, not raw capacity, in this equation -- as it is in SPC-1 benchmarks. That means we are examining the capacity over which I/O was actually performed, and not inclusive of spare drives held in reserve.
Performance should be measured using a standardized workload (like SPC-1) and measured in IOPS. Resiliency should be, according to Peglar, the warranty period measured in years.
I'm not completely comfortable with resiliency being mainly a function of warranty period, though that works to Xiotech's advantage given its unique five-year warranty. Many technical aspects also factor into resiliency, including things such as RAID (or RAGS in Xiotech's case) and a variety of hardware redundancies. That said, I get the two points that make resiliency difficult to compare on an apples-to-apples basis. For one, there is no practical way to test for every failure condition and no actuarial table for storage disaster; for another, resiliency may have a lot to do with things done outside of the array, such as array-to-array mirroring.
Lacking standard measures for resiliency, I guess hardware warranty periods will have to do the job. This definitely stacks the deck in favor of Xiotech, both because of its warranty period and because most of its competitors' products have a life of 18 months from release; competitors outsource the support contract to a third party before the warranty expires.
The ratio created by the CxPxR/$ equation should be greater than 20 to demonstrate value, Peglar noted, but this line of reasoning had its caveats. Xiotech Emprise 5000 with 146GB drives turned in a whopping value ratio of 75.97. The unit with the smaller drives showed a "V" of 30.79. However, a couple of JBODs from Sun also produced high value numbers of 52.89 and 38.68, both outdoing the much faster Xiotech array. Peglar was quick to point out that these products did not protect data and came with only a one-year warranty. He also noted that the Clarion with snapshot delivered a paltry value ratio of 2.31 -- among the lowest ratios in the spreadsheet.
To be honest, until I saw Peglar present this data, I wasn't really sure what to make of SPC benchmarks or whether they had any real value in the storage decision-making process. Not only am I generally suspicious of performance benchmarks, understanding that products can be "tweaked" for testing using configurations you would probably never use in real life, but I also see benchmarking as the rough equivalent of NASCAR racing, which tells us only how fast a car will go if it is always turning left.
However, these days I am softening my cynical view. Companies need some sort of yardstick for comparing the product offerings of different vendors. At least the SPC benchmarks don't allow storage vendors to hide behind things such as thin provisioning, which FalconStor CEO ReiJane Huai recently referred to as a "scam that allows vendors to sell 16 cents-per-GB hard disks for $60 per GB."
Now that I know how to interpret the spreadsheets from SPC-1, I encourage everyone to get and read them. They are available from the Storage Performance Council Web site at no charge. Visit http://www.storageperformance.org/results to get your copy.
Your comments are welcome: jtoigo@toigopartners.com