Commercialization of OpenStack Object Storage

April 26, 2010

Joe Arnold, Cloudscaling

Dr. Jinkyung Hwang, KT

Dr. Jaesuk Ahn, KT

Commercialization of OpenStack: Object Storage

Wednesday, April 27, 2011

Building cloud infrastructure for telcos and service providers


- Thanks to the core Swift team. They've been invaluable in sharing their knowledge about the system. - We've brought to market several OpenStack Object Storage systems for our customers. We’re leading the charge on large-scale deployments of OpenStack Object Storage.- Our focus is on building infrastructure cloud services for telcos and service providers. To do this we've focused on integrating the hardware, software and operational components so that our customers can go to market with a fully-integrated stack.

•Cloud Visionaries•Infrastructure Cloud Services•End-user Cloud Products•Very involved in Korean OpenStack Community


- KT has been visionaries in the cloud computing space. - Cloudscaling has been working with KT for about a year. In that time, Cloudscaling has helped KT launch infrastructure compute clouds including an object storage system based on Swift. - Released end-user cloud products- Kicked-off Korean OpenStack Community

0

75

150

225

300

Q4 2006Q4 2007

Q4 2008Q4 2009

Q4 2010

Billions of Objects in S3


- Storage is growing. - Applications are sprouting up for Tablets/Games/mobile devices. That application data is living in the cloud- Media consumption over the internet is increasing. Volume of that data is increasing. - Need for asset storage is large.- Users are participating and consuming more than they ever have. Social media, online video, user-generated content are all contributing to the vast need for easily-consumable storage systems.

Today’s storage systems need to supply endless storage.

Rackspace runs billions of objects and petabytes of files.

Clearly there is demand for these types of services.

Brief Refresher on OpenStack Object

Storage


Data Storage

API

Object Storage


- objects via HTTP- Not traditional filesystem- not blocks- GET/PUT/Delete over REST API- Object storage is not a traditional filesystem, or a raw block device. - It’s just containers (folders) and objects (files) that’s available via an HTTP API. - It can’t be mounted like a folder in your OS directly.- There isn’t random-access to files and there can be multiple concurrent writers, so it’s unsuitable for transactional applications like traditional relational databases. Also, it doesn’t provide raw data blocks that an operating system can form into a filesystem, so it’s unsuitable for booting an OS.

- Applications need to be designed with object storage in mind. As object storage is partition tolerant, it’s not possible to create file-system locks. The newest file wins. Applications need to be designed designed with this in mind.

Data Storage

PUT

Upload


- A simplified view of of upload.- A client makes a REST API request to PUT an object into an existing Container. The request is received by the cluster. - The data then is sent to three locations in the cluster. At least two of the three writes must be successful before the client is notified that the upload was successful.

Data Storage

GET

Download


- A request comes in for an Account/Container/Object. One of the location is determined. A lookup in the Ring reveals which storage nodes contain that partition. A request is made to one of the storage nodes to fetch the object and if that fails, requests are made to the other nodes.

Horizontal Growth & Concurrency


- OpenStack Object Storage is designed to have linear growth characteristics. As the system gets larger and requests increase, the performance doesn’t degrade. To scale up, the system is designed to grow where needed — adding storage nodes to increase storage capacity, adding compute capacity as requests increase and growing network capacity where there are choke points.

- Space available isn’t a useful statistic alone. A key benchmark is the storage system’s concurrency. Swift it able to be configured to handle a great number of simultaneous connections.

- It’s great to have the ability to scale the storage system as your customers’ applications grow.

Fantastic Durability/Availability properties

Auditors

Replicators

Independent Zones

Durability - Data PersistsShared-nothing access tier

Data served by any Zone

Availability - Access to the data


- Durability:- As we all know, the 2nd worst thing you can do in this business is loose someone’s data. The first, or course, being to corrupt customer’s data. Durability refers to the systems ability to not lose or corrupt data. - These systems are extremely durable. To achieve extreme durability numbers, -- objects are distributed in triplicate across the cluster. -- Auditors run to ensure the integrity of data to check for bitrot. -- Replicators run to ensure that enough copies are in the cluster. In the event that a device fails, data is replicated throughout the cluster to ensure there remains three copies.

- Availability: Ability to for the data to be accessed. - The servers that handle incoming api requests scale up just like any “front-end” tier for a web application. The system is architected to use a shared-nothing approach and can use the same proven techniques that have been used to provide high-availability by many web applications.- Early in a client deployment we went into pre-production (closed BETA) without monitoring and a server had failed without noticing it. There was no service interruption and Swift dutifully replicated data across to other nodes to keep 3 copies of data in place. We finally noticed when peak throughput numbers weren’t quite as high as they were previously. This really points out the robustness of the Swift architecture.

Zones: Failure Boundaries

1 2 3


- Another feature is the ability to define failure zones. Failure zones allow a cluster to be deployed across physical boundaries which could individually fail. For example, a cluster could be deployed across several, nearby data centers and be able to survive multiple datacenter failures.- 3 copies of each bit of data is distributed across zones- We go for rack-per-zone. That means we plan for rack outages of storage servers. - At Swift’s smallest, a zone could be a single drive or a grouping of a few drives. This scale of deployment is quite useful for creating development / staging environments.

Five Zones

1 2 3 4 5


How this translates into a deployment-- Everything in Swift is stored, by default, three times. There are three copies of just about everything the system needs to store data.- In order for three copies of the data be stored, at first blush, it seems like it would make sense for there to be three zones. However, Swift is designed to be a durable, highly-available system. It needs its three copies of everything – at all times.- If a Zone goes down from a three-Zone system, there will only be two zones left!- Five Zones is recommended as a starting point because if a Zone goes down, there will be other zones for data to be replicated to. Having at least five zones leaves enough wiggle room to accommodate the occasional Zone failure and enough capacity to replicate data across the system.

Object Storage for Service Providers



- I’d like to recommend OpenStack Object Storage (Swift), what else?- The software that has been battle tested by the huge deployment at Rackspace. Billions of objects & Petabytes of storage.- Something is never ‘proven’ until it’s running at scale. So, by that measure, Rackspace Cloud Files (and Swift), which is known to be proven. No other object storage type system available is proven deployed at this scale.

- We at Cloudscaling have been working with Swift from it’s initial launch in July of last year. - Now, with KT and other commercial installations, momentum is building behind this project.

- who should? what does it look like? What should you know going in?

Storage is Not an Island


- Must have a reason to offer storage- Storage is an anchor service- Grounded with other compelling services where storage is a component.

- Data is sticky. Application migration is easy. Data migration is tricky. Moving data around is difficult, often requires downtime, or is tricker to orchestrate.-- Bring customer data into your ecosystem/platform.- AWS S3 offered free TX-in for a very long time. Offers low cost physical media moving, so that it can get as much of customers data in their ecosystem as possible. -- S3 grew like crazy with EC2 right next door with 150% y/o/y growth. This is staggering.

- When building a storage product, there must be /compelling/ reasons for customers to put data into it.-- That can be:--- convenience--- access to compute resources--- features associated with the uploaded data (transcoding, data processing)--- even legal or compliance reasons.

Have an Advantage

leading South Korean landline, mobile, internet, IPTV.


What's your unfair advantage?- KT is the leading provider in Korea for internet, mobile, and IPTV. - They have a huge network advantage for providing services to end-users. - Not only that, South Korea intends to connect every home in the country with gigabit speeds. http://www.nytimes.com/2011/02/22/technology/22iht-broadband22.html- KT is in a unique position from a network prospective to offer the platform of services to serve this market. The media assets, consumer media assets, need a place to reside that is well connected to the Korean consumer of these services. There is a distinct edge that regional service providers have an edge in providing services to their local market. - Other, out of country providers won’t have the same cost-advantages or quality of services for that market.

- Other unique assets from some of our other customers include-- Colocation facilities and an existing customer base of managed hosting customers-- Extensive CDN services. Object Storage serves as a jumping-off point for CDN services.

Be Compatible


- The contrarian point here is that for all the advantages you are going to present to your users, the service needs to remain compatible with the tooling ecosystem.

- At one client meeting we were going down the path of 'differentiation' -- What makes this product unique? The answer of course, was -- nothing! That's the point. In fact, we've been working hard to make sure that you are compatible with the ecosystem of tools that are available for end-users of the service. -- We've worked with and contributed back to the open-source libraries-- We've worked with OpenStack vendors like Nasuni and Gladinet to make our 'outside of Rackspace' implementations work.- What is distinct is the bundle services that you provide your customers, the customer base that you already have, the network access that you enjoy.

- one of the huge assets that OpenStack brings is the ecosystem of tools that come to the party-- Commercial vendors such as Nasuni and Gladinet-- OpenSource tools such as Cyberduck, fog, and Rackspace's own Cloudfiles language bindings/libraries for C#, java, ruby, php, python.- You don't need to build these per-se... but you do need to ensure compatibility with your service. -- Lots of little issues that needed to be addressed (adding alternate Cloudfiles urls, fixing port issues with cyberduck, ssl cert issues with Gladinet, different format of keys, user names, passwords) So you will need to make sure that these tools are compatible with your deployment.

- The differentiation is still important! Differentiation should be in providing services on top of the infrastructure and building platform services or other infrastructure services based on storage.

Online Service Providers / Private

Huge Flat Namespace

'Repatriation' from public clouds


I know that this is the service providers track. But it's worthwhile to address folks who are building online services or who have a need to provide private solutions.

Huge Flat Namespace- Accounts -> Containers -> Objects- Proliferation of storage systems requires knowledge of what data is located where. The extreme scaling options of Swift can solve some of these issues.- Each storage cluster can grow to be several petabytes, and for regional or additional scaling the authentication service can route users to different clusters if need be.

'Repatriation' from public clouds- For those who are thinking about bringing their data back in house, using an architecturally-compatible system to the popular cloud storage products out there like S3 and CloudFiles can make a lot of sense.- For the major reason that an application doesn't need to be re architected.- Using something that still delivers the durability and reliability and not just API compatible.

Building


Building the System

Authentication

Billing Portal

Installer Front-End

Hardware

Data Center

Ecosystem

Network Ops


You must build it- Development effort. So you must consider the R&D expense.- Ramp-up a development team to understand the core of swift- Development of integration components

OpenStack Object Storage provides a core of services and functionality.- You can't just sudo apt-get install openstack- OpenStack Object Storage is a solid foundation. But must be supported by a host of services.

Let’s go into a few.

Billing


Billing- There is utilization tracking as part of Swift in the Cactus release. It's much better, but it's still 'tricky'. - Many steps involved here. I'll address two things that I think are unique to the object storage system. - Charge per GB Stored- Charge for TX ingress/egress- Charge for # of API requests

Pricing

vs

Consumption Pricing Capacity Pricing


Further, there is a decision to be made on consumption-pricing vs. capacity pricing.

When you typically go to buy bandwidth, you are charged 95-percentile. You pay for bandwidth that goes unused because you’re paying for the capacity to be available with some wiggle-room for extra-ordinary bursts.

So service providers are having to figure out how to deal with this. It’s a bigger deal at a smaller scale. A single customer could but-in and consume a large amount of a cluster on a percentage basis.

Authentication & User Management


Authentication & User Management- Two real options. - 1) Use the existing authentication service that is built into Swift. Swift comes with an authentication service that stores account information within the cluster itself. -- Benefit is that it the cluster is more self-contained and not dependent on any external services that could result in availability issues for your customers.-- However, that means integration. If you're supporting a large customer base that has access to other services and you want a way to centralize that so that the customers accounts/credentials/authentication credentials are manageable. More integration effort is required.- 2) Build your own authentication service. There is an API defined. Build to that spec, make sure it's scale-out / HA properties are something that you're comfortable with.-- Benefit is that an authentication system remains centralized and can service a range of services for the customer. If this is part of a larger IT initiative or part of a broader cloud computing offering, it’s desirable to provide end-users with a consistent way to manage and use credentials. -- Downside is that it's another component to build and maintain.

Load Balancing


Load Balancing- One of the great properties about the architecture of Swfit is it's ability to horizontally scale-out to handle increasing API access (GET/PUT/DELETE)- An incoming request does not need to be processed by a centralized storage controller.- Load balancing can handled by many mechanisms that have been refined over the past 15 years- The complexity of this setup will vary with the needs of the deployment. It can be as simple as using round-robin DNS or using Pound to using commercial load balancing solutions like a Netscaler. For whatever load balancer is used, a health check needs to written for the load balancer to monitor.

Storage Nodes

24-48 GB RAM36-48, 2TB Drives

SATANo RAID

Newish XeonWednesday, April 27, 2011

The Hardware

Storage Nodes- 36-48 disk JBODs- 24-48 GB RAM- Go for good price/performance CPUs - Xeon E5620s / E5640s. -- Not just data, also replicators, auditors- While commodity, these are not JBODs (Just a Bunch of Disks). There is a reasonable amount of memory and CPU. Metadata needs to be readily available to quickly return objects. The object stores each run services not only to field incoming requests from the Access Tier, but to also run replicators, auditors, reapers, etc. - Our configurations currently run 2TB disks, SATA disks without RAID. We use desktop-grade drives where we have more-responsive remote hands in the datacenter and enterprise-grade drives elsewhere.- SATA desktop drives (not green drives). -- We placed an order with another drive vendor (who will go nameless). Based on the order size of ~$300k worth of drives, one hard drive vendor refused to fill the order because to them it was obvious that we were not using them for desktop application.

Proxy Nodes

24 GB RAM10 GbE

Newish Xeon

Proxy ServersAuthentication Servers


Proxy Nodes- Go for “sweet spot” in price/performance (Xeon E5620s / E5640s). As it's better to have many of them and scale out, than have fewer monster machines. - Dual 10GbE- 12-44GB RAM- Cloudscaling’s deployments segment off an “Access Tier”. This tier is the “Grand Central” of the Object Storage system. It fields incoming API requests from clients and moves data in and out of the system. This tier is composed of front-end load balancers, ssl-terminators, authentication services, and it runs the Proxy server processes.- These access servers are in their own tier. This enables read/write access to be scaled-out independently of storage capacity. For example, if the cluster is on the public internet with demanding needs on ssl-termination and data access, many access servers can be provisioned. However, if the cluster is on a private network and it is being used primarily for archival purposes, fewer access servers are needed.- We deploy a collection of 1U servers to service this tier. These systems are use a moderate amount of RAM and are CPU intensive. As these systems field each incoming API request, we recommend two high-throughput (10GbE) interfaces. One interface for “front-end” incoming requests, the other for “back end” access to the object stores to put and fetch data. Factors to consider:- For most publicly-facing deployments, or private deployments available across a wide-reaching corporate network, SSL will be used to encrypting traffic to the client. SSL adds significant processing load to establish sessions between clients and more capacity in the access layer will need to be provisioned. SSL may not be required, for private deployments on a trusted networks.- Application intensive vs archive oriented. Simply put, the volume of requests will have an impact on the provisioning of the access tier.

Aggregation Aggregation

Proxy

Proxy

Switch Switch

Proxy

Proxy

Networking

Object

Object

Object

ObjectWednesday, April 27, 2011

Networking- An pair of aggregation switches with two links back to the access network / border network. The aggregation switches connect to two pools of the Access Tier and to each of the five Zone switches that connect the Object Stores. All connections to the Access Tier and the Zones are 10GbE.- Zone Network-- Each Zone has a switch to connect itself to the aggregation network. We run a single, non-redundant switch as the system is designed to sustain a Zone failure. Depending on overall concurrency desired, Cloudscaling will deploy either an 1GbE or a 10GbE network to the object stores.- Remember that when you have a write coming into the proxy server, you have 3x going to the object stores to write the three replicas. Be sure to account for that when figure out the theoretical limits for read/write traffic. Typically, the expected bandwidth coming in is the celling.

Raw System Costs


Raw System Costs:- TCO caveat: There are many components that are part of the TCO of the entire cluster.-- Facilities, power, cooling, network, NOC staff-- Many of those factors are site-specific

1 Petabyte

2 Agg Switches6 Proxy/Auth Servers5 ToR Switches50 Object Stores...and cables, racks, etc

~$750,000$0.75/GB

120 Terabyte

2 ToR Switch2 Proxy/Auth Servers5 Object Stores...and cables, rack, etc

~$95,000$0.79/GB

Raw System Costs


- Illustrate hardware pricing as a baseline - All-in hardware costs (switching, load balancing, storage nodes, optics, cabling, forged metal for the racks, PDUs)-- (To note: Amazon's retail pricing for S3 is $0.140 - $0.055)-- That price is going to go down as hardware prices go down.

Understanding TCOWednesday, April 27, 2011

- Total-cost of ownership for the cluster should include development costs, hardware and ongoing costs. These include:-- Design/Development-- Hardware-- Hardware Standup-- Datacenter Space-- Power/Cooling-- Networking-- Ongoing Software Maintenance and Upgrades-- Operational Support-- Customer Support

Understanding TCO

•Design/Development/Integration•Hardware•Hardware Standup•Datacenter Space•Power/Cooling•Network Access•Ongoing Software Maintenance•Operational Support•Customer Support


- Total-cost of ownership for the cluster should include development costs, hardware and ongoing costs. These include:-- Design/Development-- Hardware-- Hardware Standup-- Datacenter Space-- Power/Cooling-- Networking-- Ongoing Software Maintenance and Upgrades-- Operational Support-- Customer Support

Planning Checklist•Product Service Requirements•Hardware Selection•Network Design•Facilities Planning•Hardware Standup•Software Provisioning•System Configuration•Load Balancing•Authentication Integration•Utilization & Billing Integration•Additional Platform Services•Monitoring Integration•Operational Tooling•Operator Training and Documentation•Customer Training and Documentation


- There are many pieces that need to come together for a successful project. Many groups that must come together to design, build, deploy, integrate, operate and onboard customers. Consider the these of activities during your planning phase:

Assemble a cross-functional team as there are many hats that are needed for a successful standup. Data center technicians to help plan the power/cooling needed at the DC, networking experts to help design and plan out the network, a great software development team to write the integrations needed and fix issues related to the software systems of the cluster, Swift is built around common unix tools and folks who are good systems administrator skills can really help tune a running system.Product/Sales team who can communicate the value to customers. Who can bring the product to market.

-- Customer Discovery / Determining Service Requirements-- Hardware Selection-- Network Design-- Facilities Planning-- Hardware Standup-- Software Provisioning-- System Configuration-- Load Balancing-- Authentication Integration-- Utilization & Billing Integration-- Additional “Value Add” Services-- Monitoring Development and Integration-- Operational Tooling-- Operator Training and Documentation-- Customer Training and Documentation

Storage as a Service


Yes, you can offer storage as a service. - don't be 'just storage' offer as a suite of services- use OpenStack Object Storage with a commodity hardware stack to develop a cost-competitive product offering- Put together a cross-functional team. Many roles are needed.- Get help. Feel free to reach out to us, we've deployed over a 6 of petabytes in several environments and can help design a solution for your needs.

KT ucloud storage service with openstack object storage

Jinkyung HwangKT Cloud Business Unit/PEG

[email protected]

ACTION의 실천이 기업문화 혁신을 완성합니다.

OpenStack conference


What we did

□ Swift start up at Sept. 2010 and initial build-up with

Chef deploy at Dec. 2010 SAIO --> Swift on multi-servers --> Swift on VM --> Swift with Chef

□ Deployment on KT data center1 peta bytes

□ Customer service & Interworking portal, cdn interworking, api server, and other cloud services in KT

□ Beta test service from March 2011~ hundred of customers

with performance testing and system tunings

Bexar 1.2

with SWauth

Austin 1.1

Middleware

addition for CDN

& OpenAPI


What we did – automatic deployment

□ Swift deployment with Chef

Automatic deployment

cleanhardware

Swift ReadyHardware

DHCPserver

TFTPserver

mirrorrepository

CHEFserver

IP alloc for MAC

OS url, Kickstart url per MAC

Roles per IP

booting

MACIP

MAC

OS Image

IProle

OS url,kickstart url

Kickstart install(OS)

role install

OS url

Autodeployserver

kickstarturl

kickstartfile

success

IPMI


What we did - services□ user portal : cs.ucloud.com/ss

* cs: compute service* ss: storage service

< products >


What we did – Cyberduck, Gladinet, Cloudfuse IW


What we did - architecture□ KT Swift is based on , designed with

□ Currently, interworking with Cloud services of KT and 3rd party

services with API are underway

Swift Cluster

proxies Storage servers

repository

Portal

CDN Swift API

3rd party tools

Computecloud

Monitoring systems

Management ConsoleRDB Backend auth &

billing systems

DR,AS


What we did – performance test□ Internal performance tests are underway with massive loads

□ ‘Advanced’ Swift bench code is used & submitted to launchpadhttp://bazaar.launchpad.net/~jkyoung0/+junk/bench_server/files

auth create, delete, authenticate (get url & token), container create, delete, file upload, download

and delete

□Still Tuning Cluster before Launch


Issues to solve□ Tunings for best/optimal performance

seems like disk IO bottlenecks rather than network bandwidth

tunings with system parameters as well as Swift config values are necessary

□ Lookup ID middleware for CDN, API server interworking

kt add a ‘cdn-uri lookup’ and ‘portal-id lookup’ middleware to retrieve Swift URI

with CDN URI, user ID

general lookup middleware is necessary for service interworking

□ Statistics (1.2.0)

seems incorrect values and bugs existed

□ Management & operations tools are necessary

system monitoring and Swift mgmt such as ring re-balancer etc

□ Revision control visibility for commercial services

As a service provider, update is almost-no-down-time is important.

Need more visibility on the upgrade path.

e.g. ubuntu latest v 10.10 vs. ubuntu LTS (long term support) v10.4Wednesday, April 27, 2011

THANK YOU!감사합니다

April 26, 2010

Joe Arnold, Cloudscaling

Dr. Jinkyung Hwang, KT

Dr. Jaesuk Ahn, KT


Commercialization of OpenStack Object Storage

Technology

Transcript of Commercialization of OpenStack Object Storage