[email protected] Advanced Computing Interfaces ......ADAMA - data services Event driven...
Transcript of [email protected] Advanced Computing Interfaces ......ADAMA - data services Event driven...
![Page 1: jstubbs@tacc.utexas.edu Advanced Computing Interfaces ......ADAMA - data services Event driven compute containers… coming soon Elastic Storage and Compute Create Docker hosts in](https://reader033.fdocuments.us/reader033/viewer/2022050503/5f95b8188ec7956a9d507280/html5/thumbnails/1.jpg)
Docker For Scientific Applications
Joe StubbsAdvanced Computing Interfaces - TACC
![Page 2: jstubbs@tacc.utexas.edu Advanced Computing Interfaces ......ADAMA - data services Event driven compute containers… coming soon Elastic Storage and Compute Create Docker hosts in](https://reader033.fdocuments.us/reader033/viewer/2022050503/5f95b8188ec7956a9d507280/html5/thumbnails/2.jpg)
Why Docker?Building scientific libraries is painful...
>>> import numpy Traceback (most recent call last):Import multiarray ImportError: /usr/lib/python2.7/dist-packages/numpy/core/multiarray.so: undefined
Dependency management is complicated and error prone...
$virtualenv foo$ pip install -r requirements.txt
Distributing the app to a new host means repeating the effort all over
![Page 3: jstubbs@tacc.utexas.edu Advanced Computing Interfaces ......ADAMA - data services Event driven compute containers… coming soon Elastic Storage and Compute Create Docker hosts in](https://reader033.fdocuments.us/reader033/viewer/2022050503/5f95b8188ec7956a9d507280/html5/thumbnails/3.jpg)
Why Docker?
Reproducible Scientific Computations
Easy InstallationSimplified Distribution Isolated Environment
![Page 4: jstubbs@tacc.utexas.edu Advanced Computing Interfaces ......ADAMA - data services Event driven compute containers… coming soon Elastic Storage and Compute Create Docker hosts in](https://reader033.fdocuments.us/reader033/viewer/2022050503/5f95b8188ec7956a9d507280/html5/thumbnails/4.jpg)
How Easy Is It?
# Install Docker$ wget -qO- https://get.docker.com/ | sh# Install and Run an application$ docker run taccsciapps/bwa --helpPulling repository taccsciapps/bwa...Usage: intro [options] ...
![Page 5: jstubbs@tacc.utexas.edu Advanced Computing Interfaces ......ADAMA - data services Event driven compute containers… coming soon Elastic Storage and Compute Create Docker hosts in](https://reader033.fdocuments.us/reader033/viewer/2022050503/5f95b8188ec7956a9d507280/html5/thumbnails/5.jpg)
Docker Still Young
$ docker run -v /tmp:/data tooError: start: Cannot start container 56b: Error getting container 56b from driver devicemapper: Error mounting '/dev/mapper/
$ uname -astampede.tacc.utexas.edu 2.6.32-431.17.1.el6.x86_64
Multi-host deployments challenging No support for older kernels
bugs...
![Page 6: jstubbs@tacc.utexas.edu Advanced Computing Interfaces ......ADAMA - data services Event driven compute containers… coming soon Elastic Storage and Compute Create Docker hosts in](https://reader033.fdocuments.us/reader033/viewer/2022050503/5f95b8188ec7956a9d507280/html5/thumbnails/6.jpg)
Docker Rapidly Evolving
Docker Machine Docker Compose Docker Swarm
SocketPlane
Flocker/Cluster HQ
Powerstrip
Mesos
Kubernetes
CoreOSBrooklyn
Shipyard
Registrator
Synapse
Consul
![Page 7: jstubbs@tacc.utexas.edu Advanced Computing Interfaces ......ADAMA - data services Event driven compute containers… coming soon Elastic Storage and Compute Create Docker hosts in](https://reader033.fdocuments.us/reader033/viewer/2022050503/5f95b8188ec7956a9d507280/html5/thumbnails/7.jpg)
What is Agave?Agave is a Science-as-a-Service web API platform● Run scientific codes
your own or community provided codes● ...on HPC, HTC, or cloud resources
your own, shared, or commercial systems● ...and manage your data
reliable, multi-protocol, async data movement● ...from the web
webhooks, rest, json, cors, oauth2● ...and remember how you did it
deep provenance, history, and reproducibility built in
![Page 8: jstubbs@tacc.utexas.edu Advanced Computing Interfaces ......ADAMA - data services Event driven compute containers… coming soon Elastic Storage and Compute Create Docker hosts in](https://reader033.fdocuments.us/reader033/viewer/2022050503/5f95b8188ec7956a9d507280/html5/thumbnails/8.jpg)
Docker @TACC● Elastic Provisioning - Compute and Storage● Cloud Runner for Execution● endofday - Workflow engine● ADAMA - data services● Event driven compute containers… coming soon
![Page 9: jstubbs@tacc.utexas.edu Advanced Computing Interfaces ......ADAMA - data services Event driven compute containers… coming soon Elastic Storage and Compute Create Docker hosts in](https://reader033.fdocuments.us/reader033/viewer/2022050503/5f95b8188ec7956a9d507280/html5/thumbnails/9.jpg)
Elastic Storage and Compute
Create Docker hosts in public clouds and register them in Agave
![Page 10: jstubbs@tacc.utexas.edu Advanced Computing Interfaces ......ADAMA - data services Event driven compute containers… coming soon Elastic Storage and Compute Create Docker hosts in](https://reader033.fdocuments.us/reader033/viewer/2022050503/5f95b8188ec7956a9d507280/html5/thumbnails/10.jpg)
Cloud RunnerLaunch Jobs In The Cloud With A Single Command
1. Specify a work directory with executable and data
2. (Optional) Include a Dockerfile to customize your image
![Page 11: jstubbs@tacc.utexas.edu Advanced Computing Interfaces ......ADAMA - data services Event driven compute containers… coming soon Elastic Storage and Compute Create Docker hosts in](https://reader033.fdocuments.us/reader033/viewer/2022050503/5f95b8188ec7956a9d507280/html5/thumbnails/11.jpg)
Workflow Engineinputs: - input 1<- /home/jstubbs/workflows/examples/input.txt
outputs: - S.output
processes: P: image: user/image_P inputs: - inputs.input1 -> /data/input.txt outputs: - /data/output.txt -> output_P command: python p.py /data/input.txt N1: image: user/image_Q inputs: P.output_P -> /tmp/input.txt . . . .
Represent Complex Workflows using YAML
Execute on localhost or using Cloud Runner (experimental)
![Page 12: jstubbs@tacc.utexas.edu Advanced Computing Interfaces ......ADAMA - data services Event driven compute containers… coming soon Elastic Storage and Compute Create Docker hosts in](https://reader033.fdocuments.us/reader033/viewer/2022050503/5f95b8188ec7956a9d507280/html5/thumbnails/12.jpg)
ADAMA
JSON request
Legacy Datasource
MySQL
FTP Server
cgi script
FTP Server
A
C
C
A
A
JSON response
Create Dataservices that scale by writing single Python or JavaScript functions
def search(args): print json.dumps(...)
![Page 13: jstubbs@tacc.utexas.edu Advanced Computing Interfaces ......ADAMA - data services Event driven compute containers… coming soon Elastic Storage and Compute Create Docker hosts in](https://reader033.fdocuments.us/reader033/viewer/2022050503/5f95b8188ec7956a9d507280/html5/thumbnails/13.jpg)
Compute Containers(In early development…)
Run arbitrary containers in response to Agave events:- file uploads- job completions- metadata updates
Trigger execution with messages (think actor model)Event context/message injected into the container environment
![Page 14: jstubbs@tacc.utexas.edu Advanced Computing Interfaces ......ADAMA - data services Event driven compute containers… coming soon Elastic Storage and Compute Create Docker hosts in](https://reader033.fdocuments.us/reader033/viewer/2022050503/5f95b8188ec7956a9d507280/html5/thumbnails/14.jpg)
Thanks!
Questions?