AHM 9-11 Nov 2009 Bologna Engineering Ingegneria Informatica S.p.A. INFSO-RI Andrea Turli e Daniele Rufini 9-11 nov 2009 AWS Submitter Analysis and Design 2 Bologna, 9-1 November 2009 ETICS SubmissionManager and AWS Power of Cloud Architectures From GrepTheWeb-AWS white paper Processing Pipelines Batch processing systems o Nightly builds, automated unit/deployment testing Websites Amazon AWS Amazon EC2 for running a large distributed processing cluster on-demand Amazon S3 for retrieving/storing persistent datasets Amazon SQS for buffering requests - loosely coupled Amazon SimpleDB for storing data - status, log 3 Bologna, 9-1 November 2009 Architectural Abstraction : Level 1 AWS as a Grid Input System sendJob : send ETICS jobs to Amazon Cloud getStatus : receive information about the status of ETICS job. Output System GetOutput : result of processing ETICS jobs. 4 Bologna, 9-1 November 2009 Architectural Abstraction : Level 2 sendJob getSatus getOutput 5 Bologna, 9-1 November 2009 Architectural Abstraction : Level 2 The ETICS Submitter send a message in to the SQS queue for each request The controller (EC2 instance): takes a message from the SQS queue use SimpleDB to store Job information launches EC2 instances (worker nodes) to run the job monitor the job status Store the output on S3 6 Bologna, 9-1 November 2009 Architectural Abstraction : Level 3 7 Bologna, 9-1 November 2009 Architectural Abstraction : Level 3 On application start SQS queue creation and start all the controller Launch Controller pick up message from launch queue, populated by ETICS send the job request to EC2 cluster update status in SimpleDB (ready) delete the message in the launch queue insert a message in monitor queue Monitor Controller pick up message from monitor queue ping EC2 cluster to check the job status take/update the status in SimpleDB (running) insert a message in delete queue Shutdown Controller shutdown EC2 instances pick up message from shutdown queue update the status in SimpleDB delete a message after processing