CASE STUDY:CLOUDSTACK WITH OPENSTACK SWIFT

Authentication, usage and more…

WHO AM I?

Will Stevens – Lead Developer at CloudOps

Work with Python, Ruby, PHP, Java, jQuery, JavaScript, JSON, HTML, CSS and MySQL

Recent project: Custom Python development

building reports for a customer using data from the XenServer API.

WHO IS CLOUDOPS?

Founded in 2005.

We build and manage private and hybrid clouds.

Focus on managed cloud operations.

Cloud infrastructure built on Citrix/CloudStack solutions, including CloudStack/CloudPlatform and XenServer.

WHAT IS THE PROJECT?

Customer background Launching a Canadian-owned public

cloud in Canada.

Planning to launch an object storage service first, followed by a compute service.

Customer Requirements Looking for best open-source solutions

for compute and object storage.

Decided on CloudStack for compute and OpenStack Swift for object store.

Required custom integration between the platforms.

REQUIREMENT FOR CUSTOM INTEGRATION

Swift is an open source, Apache-licensed massively scalable redundant storage system.

CloudStack already allows for VM snapshots and template images to be stored on Swift via Secondary Storage.

CloudStack does not enable Swift to be used as an object store by its users.

WHAT IS NEEDED TO INTEGRATE?

Swift needs to authenticate CloudStack users to enable them to use the service. Implemented via ‘cs_auth’ or ‘mauth’.

Swift usage data needs to be made available to the billing implementation. Implemented via ‘swift_usage’.

Swift needs to be integrated into the CloudStack UI to enable the functionality for the CS users. Requires custom development.

WHAT IS ‘CS_AUTH’?

Swift authentication middleware which enables the authentication of CloudStack users via the CS API.

Uses a caching mechanism to store a CloudStack user identity in Swift, so no database syncing is required.

Leverage CloudStack’s user management instead of introducing an addition auth system.

WHAT DOES ‘CS_AUTH’ ENABLE?

Implements role based ACL, including public access.

Handles S3 requests via the ‘swift3’ middleware.

Improved performance by using identity caching, not requiring the auth middleware to hit a 3rd party auth system via the network on every request.

Works out of the box with common tools: Cyberduck, Swift Bench, Swift Dispersion, Swift Recon

‘CS_AUTH’ REQUEST FLOW

ANATOMY OF ‘CS_AUTH’

token = hashlib.sha224('%s%s' % (cs['secretkey'], cs['apikey'])).hexdigest()

if self.reseller_prefix != '': account_url = '%s/v1/%s_%s' % \ (self.storage_url, self.reseller_prefix, quote(cs['account']))else: account_url = '%s/v1/%s' % (self.storage_url, quote(cs['account']))

identity = dict({ 'username':cs['username'], 'account':cs['account'], 'token':token, 'account_url':account_url, 'domain':dict({ 'id':cs['domainid'], 'name':cs['domain'] }), 'roles':[self.cs_roles[cs['accounttype']], cs['account']], 'expires':expires})

WHAT IS ‘MAUTH’?

‘mauth’ is a project we are working on with SwiftStack.

‘mauth’ reuses the 3rd party identity caching used in ‘cs_auth’, but implements it in an extensible way.

Extensions could be written to enabling Swift auth support for ActiveDirectory, SAML, LDAP, etc…

By default, ‘mauth’ includes an extension for

authenticating against the CloudStack API.

WHAT ABOUT THE SWIFT USAGE?

Slogging is a production ready, open source usage logging middleware for Swift.

Unfortunately the slogging logs are not usable by most billing systems.

I developed the ‘swift_usage’ middleware for Swift to simplify the process of getting the usage data into a billing system.

HOW DOES ‘SWIFT_USAGE’ WORK?

‘swift_usage’ is made up of two parts: 1. slogging log parser and processor. 2. a REST service to expose the usage data.

The processor continually parses the logs and stores the usage data in mongodb objects.

The REST service exposes the mongodb objects securely using an api_key and signature mechanism which is validated with a secret_key.

WHAT ABOUT A UI?

Custom development is required to integrate Swift into the CloudStack UI.

CHALLENGES DEVELOPING A UI

Extending the CloudStack UI presents some challenges due to its single page implementation.

Representing an object store as a file system has its own challenges. (managing pseudo folders, etc…)

Uploading large files (> 50Mb) through the browser can cause unwanted behavior.

Most browsers only support POST (not PUT), so uploading to a strict REST API requires preprocessing.

WHAT IS NEEDED TO INTEGRATE?

Swift needs to authenticate CloudStack users to enable them to use the service. Implemented via ‘cs_auth’ or ‘mauth’.

Swift usage data needs to be made available to the billing implementation. Implemented via ‘swift_usage’.

Swift needs to be integrated into the CloudStack UI to enable the functionality for the CS users. Requires custom development.

IS ANY OF THIS OPEN SOURCE?

‘cs_auth’, ‘mauth’ and ‘swift_usage’ are all open source and available for you to use.

Get the source at: github.com/cloudops

Learn more about CloudOps at: www.cloudops.com

Follow us at: twitter.com/CloudOps_

+