Combing MATLAB with Amazon Web Services to produce highly ... · Combing MATLAB with Amazon Web...
31
Combing MATLAB with Amazon Web Services to produce highly scale cloud based applications James P. Doherty AOG 2016
Transcript of Combing MATLAB with Amazon Web Services to produce highly ... · Combing MATLAB with Amazon Web...
Combing MATLAB with Amazon Web Services to produce highly scale cloud based applications
James P. Doherty
AOG 2016
j
• MATLAB is the world’s leading scientific computing software. • Amazon Web Services is the world’s leading cloud computing platform. • Combining the two enables applications written in MATLAB to access
almost unlimited computational power and durable data storage. This enables us to:
– solve extremely large computational problems rapidly, without the cost of setting up and maintaining a super computer or a cluster.
– deploy highly available MATLAB based web applications that can be accessed by one or one-million users.
– centralise data storage and streamline workflows.• By using the MATLAB Complier and MATLAB SDK toolbox, this can be
done without re-writing any MATLAB code.
Overview
j
• What is the cloud?• AWS• Why cloud is useful to engineering design?• Example application• Architecture behind the application• Conclusions
Outline
j
Somewhere at the other end of your internet connection – a place where you can access apps and services, and where your data can be stored securely. The cloud is a big deal for three reasons:
– No effort on your part to maintain or manage it.– You can access cloud-based applications and services from anywhere – all you
need is a device with an Internet connection.– It's effectively infinite in size, so you don't need to worry about it running out of
capacity and scales on demand so you only pay for what you use.
Some of the major cloud computing companies
What is the Cloud?
Traf
fic/c
ompu
te
pow
er
Provision for peak
Wasted $$
Scale on demand
Victorian Racing Club
Jan Dec
Melbourne Cup
j
Every day, AWS adds enough new server capacity to support all of Amazon’s global infrastructure when it was a $7B retailer
AWS
j
• Access virtually unlimited computing power– Specific machines for RAM, storage, or GPU– For highly parallel tasks, like Monte Carlo or
optimisation problems you can spread work across 1000s of machines
– Highly durable data storage
• Enhanced QA/QC– Desktop applications are hard to control and maintain.
With cloud you have one fully QA ed copy of an application.
– Access from anywhere.– Improved project management and collaboration.
• Secure intellectual property
What role can cloud computing play in engineering design?
j
The application: design foundations for offshore oil and gas infrastructure
Source: Norske Hydro
Pazfloor, Angola. Image courtesy of Subsea 7
Particular challenges• Weak soils• Complex load history• Capacity depends on
load history• Must simulate many
cases
j
Acknowledgements
Ali AmodioCarl ErbrichDave WilliamsFraser BransbyHenry KrisdaniMarcelo SilvaMike O’NeillPhil WatsonSusan GourvenecDave WhiteMark Randolph
j
Process• Input soil properties• Consolidation analysis to get change in effective stress under dead loads• Change in effective stress leads to change in soil strength• Using modified strength, conducted a series of H-V capacity calculations
for different effective widths and CLR values• Compare with 1000’s of load cases• Document
Offshore foundation design
H
Vseabed
Image courtesy of Susan Gourvenec
j
System architecture
All AWS regions have at least two physically separate “availability zones”, with a guaranteed uptime of 99.95%.
Design your system so that if one zone goes down, the system still works as normal
j
System architecture
All AWS regions have at least two physically separate “availability zones”, with a guaranteed uptime of 99.95%
Some services reside in specific availability zones
Computer servers
j
System architecture
All AWS regions have at least two physically separate “availability zones”, with a guaranteed uptime of 99.95%
Others automatically work across zones
j
System architecture
User fills in a web form and hits “run” and send a package of data to the web server
j
System architecture
The web server sends the package to a data base for storage
Web server is in constant communication with data base to keep user informed of status of the job
Job queueing
j
System architecture
The web server sends meta data to a queue
Status=queueing
Job queueing
j
System architecture
An application server reads the meta data from a queue
Job queueing
Status=queueing
j
System architecture
Using this meta data, and the MATLAB Database Toolbox the application server can get the full input data object from the database and can then run the job
Job Running
j
System architecture; more than serving up a web page!
After the job finishes and app server generates output data and writes this to the database using the MATLAB Database Toolbox. Images are stored in S3
Status=Complete
Job Complete
j
System architecture
EC2 instances with the following• MATLAB Runtime Complier
Library (MCR)• Your complied MATLAB code as
a dll• A wrapper function that can read
from SQS and call the dll
Then from this, create an Amazon Machine Image (AMI).
Can then “spin up” as many EC2 instances based on this AMI
The computational engine(s)
j
Cloud computing offers opportunities to access massive compute power and gives flexibility and freedom to a workforce. It also assists with QA and QC and helps secure valuable IP.• This has been demonstrated by developing the Fugro AG shallow
foundation design package, BearCon.• The web application combines several design processes into a single
workflow, resulting in a large reduction in manual data processing, making design activities more efficient.
• The scaling capability of cloud based computing will allow large volumes of foundation design calculations to be conducted. This will allow a range of design options to be explored, resulting in optimised foundation solutions. i.e. better engineering design.
Reliability based design can be developed on the back of this. – 2000 Monte Carlo simulations of a calculations that takes 5 minutes, would
take 1 week on a single machine.– Or, using 200 machines (say AWS M3 instances at 15 cents per hr), would get
the job done in around 48 minutes for $30.
Conclusions
j
• By compiling MATLAB into dll using the Complier and SDK toolbox, MATLAB can be seamlessly combined with AWS.
• The MATLAB Database Toolbox simplifies data storage and integration of the data into MATLAB algorithms. Found to be fast and reliable.
• Even though MATLAB was one of about 8-10 different technologies used it was probably the most important in enabling an application such as BearCon to be written.
Conclusions
