Cloud Storage Benchmark Report: Amazon S3 is Standout
Not all cloud storage providers (CSPs) are created equal. That’s the unmistakable conclusion of a 26-month stress test of 16 major providers in which only six could meet the test’s minimum performance, stability, availability, and scalability requirements. The six included (alphabetically) Amazon S3, AT&T Synatpic Storage as a Service (powered by EMC ATMOS), Microsoft Azure, Nirvanix, Peer1 Hosting (also powered by EMC ATMOS), and Rackspace Cloud.
The intensive tests were conducted by Nasuni, an enterprise storage company that combines on-premise security and storage resources from leading CSPs. The tests looked at how the services perform for mid-sized enterprises in three areas: performance (including response time), stability/availability (how often is the service down, and for how long), and scalability (does the service truly offer unlimited capacity).
Nasuni wrote a custom connector using Python using the CSP’s API documentation, broke down the service into components for unit testing (to test basic functions such as reading and writing files of different sizes, using a different connection to re-read previously written data, and simultaneous access). Six providers failed in these tests; services designed for archiving files were severely stressed by the system, “sometimes to the breaking point” according to the report.
The company tested performance for concurrency, object size, and type of workload (reads only, writes only, and a combination of the two tasks). The benchmarks adjusted for location bias by running the benchmarks from a trio of geographic areas and hosts. Of the 10 providers that based the basic functions tests, two were eliminated because their performance was “too low to be acceptable to the vast majority of end-user organizations.”
Performance speeds showed significant differences in only some tests. For example, the average write speed for 1 MB files varied among providers between 2.0 and 2.38 MB/sec (except for Peer1, in last place with a speed of just 1.49 MBs/sec); read speed variability for 1MB files was more distinctive, with Nirvanix and Azure leading the pack (at 13.3 and 13.2 MB/sec respectively) and Amazon coming in third (at 11.28 MB/sec); Peer1 again was in last place at just 3.1MB/sec). Peer1’s performance was also 70 percent slower than first-place Azure when writing 128KB files and less than 25 percent as good as first-place Amazon in reading those files.
The report warns that organizations may be satisfied with slower speeds depending on how the enterprise actually uses the service; sometimes slower performance is just fine (especially if premium performance comes at a premium price).
Stability tests checked re-reading files written to ensure no data loss as well as how frequently CSPs experienced unplanned outages. Scalability checked how many objects can be placed in the cloud and whether performance is maintained as more objects are added; it wrote small (1KB) files using concurrent threads until 100,000,000 objects were written.
I asked Nasuni’s CEO, Andres Rodriguez, what the low "pass" rate says about these services. Did he, for example, expect the 10 service providers that failed to eventually shut down? Will there be more competition in the industry?
“I think the biggest lesson is that, while it’s easy to make something that looks like cloud storage, it’s very difficult to create a service with the performance, scalability, and stability that enterprises require for primary storage. We do expect that the market will continue to consolidate, but as quality improves, competition will intensify and, like any commodity market, prices will come down.”
The survey targeted mid-size enterprises. Would the results have differed for larger enterprises? Rodriguez said that Nasuni focuses on “the mid-size enterprise because they are the ones facing the most challenges -- they are most resource-constrained in terms of investment ability and size of their IT staff, and thus their pain around adopting the cloud is most acute.
“The cloud, in many ways, is not usable by itself and thus why we package it as part of an overall solution. Our evaluation wouldn't vary based on whether the organization is small, medium, or large. For organizations that need access to their data from anywhere in the world, at any time, and want to leverage the cloud for redundancy, scalability, and its on-demand benefits, the tests performed reflect that and are appropriate regardless of an organization's size.”
-- James E. Powell
Editorial Director, ESJ
Posted on 12/15/2011