Nasuni 2015 State of Cloud Storage Report

The State of Cloud Storage

2015 Industry ReportA Benchmark Comparison of Speed, Availability and Scalability

White Paper: The State of Cloud Storage

[email protected]

White Paper:The State of Cloud Storage

United States: 1.800.208.3418International: 1.508.433.62002

Executive SummaryBoth 2013 and 2014 were record-setting years for adoption of cloud services in the enterprise. More than 50 percent of large enterprises reported using cloud services as an integral part of their infrastructure, and it is safe to say that cloud services are no longer just for early adopters. Cloud service provider (CSP) offerings have matured and are quickly becoming an essential component of any leading enterprises critical infrastructure.

One of the most widely adopted and fastest growing cloud services is public cloud storage. Some analysts project a 33 percent compound annual growth rate for public and private cloud storage over the next five years, as enterprises find new ways to leverage the unlimited capacity and enterprise-grade reliability offered by most leading CSPs. The challenge for enterprises, however, has been how to take full advantage of what these providers have to offer without increasing internal costs for support, maintenance and custom development.

At Nasuni, we have created a unique service that enables enterprises to scale their storage capacity without scaling their internal operations, to provide unlimited back-ups of their data, and to provide global access from any device. To do this, we leverage integrations with the major CSPs for unlimited capacity and global footprint. Our goal is to always provide the best infrastructure for our customers, and to that end, we regularly test these CSPs services.

This year, we have restricted our tests to the three leading CSPs: Microsoft Azures Blob Storage, Amazons Simple Storage Service (S3), and Googles Cloud Storage. We found that the results are similar to last year: Microsoft has come out as the highest performer, particularly in the benchmark tests. Amazon, as expected, is a close second, and edged out Microsoft in some aspects of scalability and availability, but ultimately fell short in some of the critical benchmark evaluations. Google continues to show promise albeit without a significant difference in its status from 2013.

1 Cohen, Reuven, The Cloud Hits the Mainstream: More than Half of U.S. Businesses Now Use Cloud Computing, Forbes, April 16, 2013.

2 Rebello, Jagdish, Enterprise cloud computing: future market size, growth and competitive landscape, IHS Quarterly.

[email protected]



Why Does Nasuni Publish This Report?We regularly benchmark the major cloud service providers to ensure that we offer our customers the most reliable and best performing cloud storage available. A few years ago, we decided to publish these results to 1) demonstrate the reasoning behind our choice to work with specific technology partners, and 2) provide some useful benchmarking and analysis in a market that is becoming increasingly commoditized.

How We Evaluate CSPsOne of the things that we take pride in doing well at Nasuni is making the most out of what all the cloud service providers have to offer. The Nasuni Service employs many different cloud services including NoSQL databases, elastic computing, and messaging queues from a variety of providers. But storage is central to what we offer, so, accordingly, we are invested in evaluating the public cloud storage capabilities of the major providers.

Our thinking around how to test public cloud storage comes from our own experience providing mission-critical storage services to hundreds of enterprises. Our benchmark tests reflect the patterns of use that we see from our customers. The scalability tests are designed to stress the critical aspects of the storage environment, such as the durability of containers and the behavior of the service under stress, as well as to simulate some of our toughest use cases, such as disaster recovery and initial migrations. Where we set up our tests and how frequently we test are also considerations that are designed to replicate the global footprint of our customers and help us to identify weak points not just in the storage itself, but also in the network infrastructure that is delivering it.

Sharing Critical Performance InformationPublic cloud storage is often referred to as a commodity that a CSP offers: a basic service that differs little from its competitive alternatives. Since most of the major cloud storage services are at feature parity with each other, choosing the right

provider is more a decision about the price and performance trade-off than about features. We hope that, by sharing our findings on the performance of these providers, we are informing the market with critical information to enable readers to distinguish between these so-called commodity services.

Although this report represents a detailed review of Nasunis latest findings, it is important to note that we conduct these tests primarily to identify CSPs that meet the needs of Nasuni and its customers. They do not necessarily represent the optimal metrics for any provider tested. However, these independent test results should be of interest to any organization that is considering the use of cloud storage as a component within a larger IT infrastructure.

Comparison MetricsAs with most technology evaluations, three main criteria govern the decision to purchase:

Functionality: what a service offers Price: the cost of the service Performance: how well that service is operates

For each of the cloud service providers, the functionality is essentially the same: we are able to create containers for objects, write objects into that container, read objects from that container, and then delete objects.

For most of the major CSPs, price is based on some combination of data stored per month, and the bandwidth consumed with both writes and reads. Price competition has been happening for years in this industry, and the real distinction in pricing is around the capacity tiers of service. But, for the most part, the price differences are minimal and only matter on the margin. This leaves us with performance as the main set of comparison metrics.

[email protected]



Performance Performance is the primary yardstick by which Nasuni measures any publicly available CSP, testing the operation and stability of CSPs over long periods of time. In fact, Nasuni has been testing and comparing CSPs since 2009. Before considering any CSP for use in a production environment, it must meet minimum performance benchmarks across three areas:

Speed Availability Scalability

Speed This simple test measures the raw ability of each CSP to handle large numbers of writes, reads, and deletes (W/R/D). We test each CSP with files of varying sizes:

1 KB 10 MB

10 KB 100 MB

100 KB 1 GB

1 MB

Using different levels of concurrency:

1 Thread 25 Threads

10 Threads 50 Threads

This test runs for 12 hours, using multiple testing machine instances and several non-serial test runs to reduce the likelihood that external network issues could bias the results.

Availability: This test takes place over a 30-day period and measures each CSPs response time to a single W/R/D process at 60-second intervals:

Write a randomly-generated 1 KB file. Read a randomly-selected previously-written file. Delete a selected file.

Reading and deleting a random file forces each CSP to prove their ability to be responsive to all of the data, all of the time, and not merely to the last piece of cached data. This test calculates the entire time required to complete the three requests, including

any required retries. This ensures examination of not only responsiveness, but also of CSP reliability and latency.

Scalability: Similar to the availability test, this is also an extended test that measures each CSPs ability to perform consistently as the number of objects under management increases. Performance under increasing object counts is often the Achilles heel of a cloud storage system, and this test measures each CSPs ability to maintain performance levels as the total number of objects stored in a single container increases to hundreds of millions. This is particularly important to Nasuni, because unlimited scalability is a key feature of our service. We have customers with large data sets that have been keeping version snapshots for years.

MethodologyDue to dynamics in the marketplace, the list of platforms evaluated continues to change from year to year. The CSPs tested this year are Amazon S3, Microsoft Azure, and Google Cloud Storage. HPs Cloud Object Storage and IBMs SoftLayer were considered, and even tested in a limited capacity; however, we ultimately decided not to include them. In HPs case, it was partly because there was a change in their strategic direction and we were unsure what their service would look like over the coming year. With IBM, our experience with their service was checkered with scheduled outages on their part. Working around those proved to be difficult and a concern for us. We hope to be able to re-visit both services in the future.

We also looked to outside analysts to help validate our decisions to focus our efforts this year. Gartner has listed our three target CSPs in their most recent Magic Quadrant report as the only Leaders (Amazon and Microsoft) and Challengers (Google) in the public cloud storage space. That additional perspective helped us justify our decision to restrict the scope of this years report.

Unlike last year, we did not invite the CSPs to participate in the evaluation process.

[email protected]



Test SetupNasuni engineers conducted all tests between October 2014 and February 2015, using virtual machines across most of the major cloud-compute platforms. The virtual machines had the following specifications:

RAM: 3.5-4 GB vCPUs: 1 Operating system: Ubuntu 14.04,

64 bit - Ubuntu 14.04 LTS

Each CSPs benchmark tests were run using four outside machines. For example, Amazon EC2 was not used to test Amazon S3. Instead, S3 was tested using Microsoft, HP, Google, and IBM.

The test centers were also spread geographically throughout the eastern and central regions of the United States. We ran all tests using a variety of times, locations, virtual machines, and dates to minimize the risk of external network bias.

Benchmark Tests: The benchmark tests are designed to evaluate the performance of CSPs under file-server data load, and the file data set represents the same distribution of file sizes used by actual Nasuni enterprise customers across thousands of installations over several years. Here is our typical breakdown of files by size:

1KB 10KB 100KB 1MB 10MB 100MB 1GB

16.8% 24.6% 26.2% 9.7% 22.2% 0.4% 0.1%

Table 1: File size distribution

In addition, this distribution of file sizes closely matches a well-documented breakdown that Microsoft did over a number of years.

For each speed test, the test evaluated 23 combinations of file sizes and thread counts as shown in Table 2.

File Size & Threads Tested

1 KB

10 KB

100 KB

1 MB

10 MB

100 MB

1 GB

Num

ber

of

C

onc

urre

nt T

hrea

ds 1

10

25

50

Table 2: File size and thread count combinations tested

The results are averaged based on the weighting of customer file-server data (Table 1) and are then indexed to the performance of the top performer. The results compare all the CSPs to the performance of the top performer across all file sizes and thread counts. This allows hundreds of individual tests to be evaluated using a single benchmark metric. Detailed raw results by CSP are included in the appendix.

As the results show, the raw speed performance varies significantly as object sizes and thread counts vary. Specifically, small object sizes and smaller thread counts highlight the transactional overhead of any platform. The effect of transactional overhead becomes most noticeable during writes, which contains three steps:

Preparation Transmission Acknowledgment

For small files, Transmission is only a small portion of the total transaction, so any inefficiency in the performance of Preparation and Acknowledgment has greater impact. Those object stores that are built with efficient Preparation and Acknowledgment steps perform best when handling small files.

However, as file sizes or thread counts increase, the time associated with Transmission increasingly dominates the overall time associated with the transaction. Inefficiencies in Preparation or Acknowledgement become less and less critical. Many CSPs overly focus their efforts to improve the efficiency of the Transmission stage of the

[email protected]



ResultsWrite Benchmark Similar to last year, Microsoft was the top write performer, excelling in 13 of the 23 individual combinations tested, thereby making it the optimal write target for file-based data. Amazon is a strong second and Google performed at less than half of the average response time of Microsoft.

However, for files larger than 1MB, Amazon had the overall best write performance. In Figure 2, Google and Microsoft are on par with one another, essentially tying for second place to Amazon.

Figure 1: Indexed write speed with all file sizes

Figure 2: Indexed write speed with file sizes >1MB

transaction, and thus perform better under the load of larger object sizes or thread counts. This may be fine for use-cases such as media archives, but for file-server data, which is often dominated by small files, performance on small files is critical.

Availability Test: Availability tests were run from a single VM running in Rackspace. The metric used is response time, which measures each CSPs response time to a single W/R/D process at 60-second intervals. Because response time also includes any time associated with retries or delays, it is a more effective metric for availability than a simple ping test.

Scalability Test: Scalability tests were conducted using internal machines to reach the highest scalability numbers in the shortest amount of time, for example, Amazon EC2 writing to Amazon S3. As object counts increase, the performance of some CSPs degrades or becomes variable. Depending on CSP architecture, some systems are designed to scale across containers, not within them. This type of architectural limitation can become a significant bottleneck after months or years of usage.

An ideal scenario for anyone seeking to leverage cloud storage is to partner with a CSP whose performance and responsiveness are unchanging, regardless of the number of objects under management. Just as with traditional in-house storage, customers expect a consistent level of performance.

Under this test, all of the CSPs were loaded with new objects as quickly as possible: up to 100 million objects or 30 days, whichever came first. The variance represents how much the speed of loading objects changed over time, causing inconsistency and variability as objects were loaded. This year we added a new dimension write speed vs. write speed variance so others can evaluate the tradeoff of time-to-complete their large tasks vs. the variability in the performance.

[email protected]



Read Benchmark In terms of read performance, Microsoft still consistently outperforms the other CSPs. However, Amazon trails Microsoft by less than it did in the write performance tests. Google, however, consistently performs at about half of the level of Microsoft in both read and write.

Delete Benchmark Microsofts real performance superiority is seen in the delete benchmark: it is more than twice as fast at deleting files as Amazon and nearly 5x as fast as Google. Different CSPs implement delete in different ways. Some CSPs acknowledge the delete and then do the work in the background. Others do the actual delete operation before responding to the request. This could explain some of the results in this benchmark.

Similar to the write benchmarks, Amazon significantly outperforms both Google and Microsoft on the large file sizes.

Figure 3: Indexed read speed with all file sizes

Figure 4: Indexed read speed with file sizes >1MB

Figure 5: Indexed DELETE speed with all file sizes

[email protected]



Examining the results over the month of testing also gives some insight into the variability of the numbers. Amazon shows the most consistency and the smallest daily uptime values. Microsoft is a close second. Google, however, is both more variable and slower.

ScalabilityIn previous reports, we measured scalability resilience using three metrics: 1) the variance of write speed in writing 100M objects, 2) the number of write misses and 3) the number of read misses. This year we decided to add a fourth dimension to this analysis, the tradeoff between variance (measured as (obj/s)2) and write speed.

Google had by far the lowest variance of the three, an order of magnitude smaller than both Microsoft and Amazon. Microsoft had the worst of the three with exceptionally large variance.

The read/write error analysis showed less of a difference between CSPs. All CSPs had zero read errors for 100M objects. Google and Microsoft showed zero write errors, while Amazon had five write errors for an error rate of .000005 percent.

In addition to system and data availability, the test also measures overall uptime or percent of the time that the CSP is reachable. Unlike in previous reports, each CSP had perfect scores of 100 percent uptime.

AvailabilityAmazon and Microsoft nearly tied on our availability metrics, averaging a response time of .1 and .14 seconds, respectively, over a 30-day period. Google, again, trailed the top two with an average response time of .5 seconds, nearly 5x slower.

Figure 6: Average response time (shorter bars are better)

Figure 7: Average daily response time

Figure 8: Variance in write speed (shorter bars are better)

[email protected]



The average write speed/variance trade-off tells a different story. Microsoft performs worst with the lowest average write speed and the highest variance. Amazon appears to be the top performer with the highest write speed; however, that comes at a price of a substantially higher variance over Google, which has the lowest variance but also a substantially lower average write speed. For most applications, however, the ability to write quickly is more important than the ability to write consistently, which makes Amazon the preferred choice for simply scaling quickly.

ConclusionSimilar to our 2013 report, we find that there are only two significant competitors in the public cloud storage market: Microsoft and Amazon. And, for the second year in a row, Microsoft is the top CSP for public cloud storage.

In our most important suite of tests the benchmark tests Microsoft consistently performed better than Amazon. It delivered the best speeds across small and medium-sized files and, in some cases, beat Amazon by nearly 2x. For large files, Amazon showed better write and read performance, but Microsofts delete speeds topped Amazons.

Amazon performed slightly better than Microsoft in our availability test, both in overall average response time and in daily average response time. However, Google was a distant third, with almost five times the response time.

Scalability results were mixed. Amazon had a relatively low variance and the highest average write speed, but also was the only CSP to show any write errors. Although Microsoft had the highest variance, it also posted the second highest speed and a perfect record of zero read and zero write errors.

Overall, Microsoft and Amazon are the two clear leaders in public cloud storage. This year, Microsoft out-performed Amazon, specifically, in our critical benchmark tests. However, Amazon continues to demonstrate the robustness of its platform, particularly in its ability to scale quickly and reliably.

Disclaimer

The tests reported upon in this document are conducted by us using our own test tools under test conditions chosen by us. The test conditions were chosen by us to reasonably represent what our customers would experience using our Service with their representative environments and workloads. The tests have been designed by us to only look at the performance aspects of the CSPs that we believe are relevant to our customers it is intentionally narrow in scope. Nasuni is not in the business of benchmarking CSPs, certifying test results or selling performance metrics. We have attempted to make sure the tests are fair and consistent within our selected parameters and have worked with several of the vendors to confirm our results. Our tests are not meant to indicate performance from each CSP under ideal conditions to the CSP, and, in any event, performance should only be one factor of many in a CSP selection process.

Figure 9: Variance (1/V) vs. average write speed

[email protected]



Appendix Raw Results

Microsoft Azure

Write Benchmark Results (KB/s)

Threads File Sizes

1 KB 10 KB 100 KB 1 MB 10 MB 100 MB 1 GB

1 62 531 3,485 9,654 18,075

10 35 3,242 17,502 28,947 27,182

25 472 4,316 19,745 29,838 28,896

50 519 4,606 19,973 30,014 30,411

Read Benchmark Results (KB/s)

Threads File Sizes

1 KB 10 KB 100 KB 1 MB 10 MB 100 MB 1 GB

1 113 863 4,617 10,271 22,886

10 631 6,014 22,748 33,854 27,462

25 1,229 9,600 27,061 72,474 25,773

50 1,285 9,778 26,128 79,116 25,493

Delete Benchmark Results (Obj/s)

Threads File Sizes

1 KB 10 KB 100 KB 1 MB 10 MB 100 MB 1 GB

1 111 107 105 105 129

10 755 799 806 795 695

25 1,267 1,250 1,294 1,032 659

50 1,272 1,294 1,276 1,244 435

Note: Microsoft Azure Blog Storage does not support objects larger than 64MB in a single upload.

[email protected]



Amazon S3


Threads File Sizes

1 KB 10 KB 100 KB 1 MB 10 MB 100 MB 1 GB

1 9 91 560 3,003 9,241 11,737 11,851

10 95 916 5,551 27,123 48,188 43,771

25 231 2,241 14,141 65,409 102,036

50 481 4,707 27,849 112,971 119,267


Threads File Sizes

1 KB 10 KB 100 KB 1 MB 10 MB 100 MB 1 GB

1 19 175 1,480 8,729 20,067 32,651 36,042

10 190 1,881 13,783 62,196 89,801 86,477

25 474 4,560 31,833 83,684 82,460

50 996 9,354 56,899 86,346 75,570


Threads File Sizes

1 KB 10 KB 100 KB 1 MB 10 MB 100 MB 1 GB

1 17 18 18 17 18 16 13

10 174 166 182 175 180 157

25 470 448 473 461 441

50 957 955 968 945 850

[email protected]



Google Cloud Storage


Threads File Sizes

1 KB 10 KB 100 KB 1 MB 10 MB 100 MB 1 GB

1 5 53 343 2,496 13,189 23,172 28,214

10 54 525 3,533 23,094 32,050 32,040

25 125 1,267 8,769 30,958 32,327

50 261 2,533 15,736 30,891 32,086


Threads File Sizes

1 KB 10 KB 100 KB 1 MB 10 MB 100 MB 1 GB

1 23 188 1,436 5,095 19,658 28,966 31,598

10 210 1,901 12,271 27,971 32,930 33,994

25 518 4,740 19,056 18,520 29,843

50 975 7,488 22,440 14,731 24,288


Threads File Sizes

1 KB 10 KB 100 KB 1 MB 10 MB 100 MB 1 GB

1 9 8 6 9 10 8 8

10 87 88 61 86 77 60

25 212 209 149 218 192

50 441 419 296 310 264

About Nasuni

Files are everywhere and they are a pain. Nasuni eliminates this pain forever by delivering file storage for distributed enterprises using a combination of cloud capacity, Nasuni software, and NAS appliances. Nasuni gives customers unlimited storage with built-in data protection and DR, secure global file sharing and mobile access, all managed from a single web console. Nasuni is cloud-based NAS for the distributed enterprise.

Our team is made up of enterprise storage, security and networking industry veterans with a shared vision of transforming the way enterprise organizations view data storage. We believe that storage should be as easy to purchase, consume and manage as the electricity that keeps the lights on.

Nasuni Corporation 2015, All Rights Reserved

313 Speen Street, Natick, MA 01760-1538

WP_3.15 1.508.433.6200 www.nasuni.com

Nasuni 2015 State of Cloud Storage Report

Documents

Transcript of Nasuni 2015 State of Cloud Storage Report