Image Magick in the Cloud

Scalable Image Processing Service

Sanketh BeerabbiUniversity of Central Florida

Image processing is a very compute intensive task which requires multiple compute units to provide near instantaneous results.

It’s more challenging in online applications involving many simultaneous, “impatient” users creating varying workloads.

Cloud computing provides the scalability needed for running such applications online.

Introduction

Allows users to upload photos and store them in the cloud.

Allows photos to be manipulated by applying Instagram style image filters.

Allows users to create profile pics by detecting faces and cropping them out of the original photo.

Service

Amazon 64 bit Linux AMI AWS PHP SDK with cURL and PEAR packages GD and Image Magick for image manipulation

AWS services:• EC2 & S3• SimpleDB (SDB)• Simple Queue Service (SQS)• Elastic Load Balancer (ELB)• Cloudfront CDN

Implementation

Plan and Design (~2 weeks)

LAMP, AWS PHP SDK, GD and ImageMagick (~2 weeks)

Browser based uploading to S3 and server side operations using S3 API (~2 weeks)

Database operations using Simple DB(1 week)

Message passing using SQS API (~2 weeks)

What took time

Architecture

AWS CLOUD

Simple DB

Load Balancing & Redundancy

Upload – • Stores file names and task status in SDB • Creates bucket and uploads image to S3• Sends message to SQS

Process – • Reads details from SDB and updates task status. • Receives and deletes message from SQS• Downloads from S3 and processes image . Uploads processed image to S3.

Modules

A single Amazon SQS queue can be shared by multiple instances simultaneously.

An instance that is processing a message can prevent other instances from processing the same message at the same time by temporarily “locking” a message. When the instance is done processing the message, it deletes the message.

If the instance fails while processing the message, another instance can get the message after the lockout period.

The number of processing servers can be changed

dynamically to handle varying load.

Message Queing

PHP, hundreds of new *NIX commands, tools and Linux from the “terminal” point of view.

Virtual networking, load balancing and DNS.

Batch processing on the cloud using multiple instances

Inter-instance communication using message queuing.

What I Learnt

Main Website:• http://imagecloud.vf1.us/

AWS Public address:• http://ec2-54-204-45-246.compute-1.amazo

naws.com/

Cloudfront CDN:• http://d3emazpmic2hkb.cloudfront.net/

DEMO

Questions?