Shane Canon!Lisa Gerhardt!NERSC Data and Analytics Services!

Shifter at NERSC

-1-

December19,2016

Shifter: Bringing Containers to HPC

•  Docker:opensource,automatedcontainerdeploymentservice–  Dockercontainerswrapupapieceofso3wareinacompletefilesystemthatcontainseverythingitneedstorun(code,run>me,systemtoolsandlibraries)

–  Guaranteedtooperatethesame,regardlessoftheenvironmentinwhichitisrunning

•  NERSChaspartneredwithCraytodeliverDocker-likecontainertechnologythroughanewsoEwarepackageknownasShiEer

-2-

Shifter at NERSC

•  SecureandscalablewaytodelivercontainerstoHPC

•  ImplementedonCoriandEdison•  SupportsDockerimagesandotherimages(vmware,ext4,squashfs,etc.)

•  BasicIdea–  UserscreatecustomimagesindesiredOS–  UploadimagetodockerhubandpulldownonHPCsystem

–  Hookedintothebatchsystem

-3-

NSHIFTERhJp://www.nersc.gov/users/so3ware/using-shi3er-and-docker/

Why Use Shifter?

•  ShiEerallowsyoutofullycustomizeyouroperaRngenvironment– WantSL6.Xwith32bitlibraries?UseShi3er–  Haveaverycomplicatedso3warestackwithlotsofdependencies?UseShi3er

•  Portability–  Canvolumemountdirectoriesintoshi3erimages

•  Havean/inputand/outputthatarelinkedtodirectoriesinyourscratchdirectory

–  ImagesareNERSC-independent,canberunanywhere

-4-

Shifter is Fast

-5-

Anotherbenefit:Improvedsharedlibraryloading>mescomparedtootherfilesystem

File System flow – Traditional vs Shifter

-6-

/udi/image1/udi/image1/usr/udi/image1/etc…

LustreClient

LustreOST

/udi/image1.ext4

LustreClient

LustreOST

Process Process

ext4orsquashfs

LustreMDS

Remote

Local

Even Big Images Load Quickly •  Asproofofconceptcreated“Mega”CVMFSshiEerimage

–  FullCVMFSstackpulleddownanddedupedwithuncvmfsso3warestack.1–3TBext4fileuncompressed,300GBcompressedw/squashfs

•  UseShiEertoloadjob–  Addasingleflagtobatchscript“--image=<imagename>”–  ATLAScvmfsrepositoryisfoundat/cvmfs/atlas.cern.chlikenormal

-7-

BurstBufferLustreShi3er

Loop Mounted FS for Super Fast I/O

-8-

57GiB/s!

•  ShiEercanmountanxfsfilesystemoneachnode–  Createdwhenjobstartsanddestroyedwhenjobends–  Computenode“localdisk”–  ExcellentI/Orates:

•  Smalldatabases•  Alsogoodfor“badIO”

Shifter and MPI

•  MPIcommunicaRonoverAriesnetworkisavailablebydefaultforShiEeronCori

•  Inimage,buildandlinkagainststandardMPICHlibraries

•  CraylibrariesswappedinatrunRmebyfrontloadingLD_LIBRARY_PATH

-9-

Shifter: How to Use It •  CreateaDockerimageanduploadittoDockerhub

-10-

FROMubuntu:15.10RUNapt-getupdate&&apt-getinstall-yautoconfautomakegccg++makegfortran&&apt-getcleanallRUNmkdir/build/COPY./input_files/Python-2.7.11.tgz/buildCOPY./input_files/mpich-3.2.tar.gz/buildCOPY./input_files/mpi4py-1.3.1.tar.gz/buildRUNcd/build&&tarxvzfPython-2.7.11.tgzRUNcd/build/Python-2.7.11&&./configure&&make-j4&&makeinstallRUNcd/build&&tarxvzfmpich-3.2.tar.gzRUNcd/build/mpich-3.2&&./configure;make-j4&&makeinstallRUNcd/build&&tarxvzfmpi4py-1.3.1.tar.gzRUNcd/build/mpi4py-1.3.1&&pythonsetup.pybuild&&pythonsetup.pyinstallRUNcd/build&&rmPython-2.7.11.tgz&&rmmpich-3.2.tar.gz&&rmmpi4py-1.3.1.tar.gz

Shifter at NERSC: Demo

-11-

Outlook

•  ShiEerisbeingsuccessfullyusedbymanyusersincludingusersfromHEP,NP,andBES

•  FutureShiEerplans–  Abilitytooverlaymul>pleshi3erimages–  Privateshi3erimagesforgroupswithaccesslimita>ons

•  ShiEerisaneasywaytoimproveperformanceandgetportabilityforyourscienceenvironment

-12-

National Energy Research Scientific Computing Center

-13-

Pulling Down an Image from Dockerhub

-14-

shi3erimgpulldocker:lgerhardt/mpi-test:v5

Formatissource:image_name:tag

OnCoriorEdison

Existing Shifter Images

-15-

shi3erimgimages

cori11>shi3erimgimagescoridockerREADYce20c473cd2015-12-11T09:53:47centos:7coridockerREADY17583c7dd02015-12-04T08:32:58busybox:latestcoridockerREADY92b9f2dbf72015-12-04T10:48:12scanon/shanetest:latestcoridockerREADYd120eb68e82015-12-04T15:36:13jcorrea/spot2:v1coridockerREADY621a496b0d2015-12-08T06:26:58registry.services.nersc.gov/lcls_xfel_edison:201509181024coridockerREADYd55e68e6cc2015-12-22T09:16:46ubuntu:14.04coridockerREADY86314f27ce2015-12-09T00:33:46registry.services.nersc.gov/psana:latestcoridockerREADYd6bf89bd0b2016-02-08T09:32:43registry.services.nersc.gov:nersc-py/2.7_edisoncoridockerREADYb345ea6cf62015-12-09T07:57:41registry.services.nersc.gov/nersc-py:latestcoridockerREADY33200b4db52015-12-12T00:00:06kbase/kbase_base:developcoridockerREADYe5ff6dcec02015-12-14T11:03:53registry.services.nersc.gov/psana:ana-0.17.4coridockerREADY109b72e23c2015-12-15T05:50:38fedora:23coridockerREADY2bc6cdd62f2015-12-15T06:06:25miguelgila/wlcg_wn:20151214coricustomREADY6378f30b512015-12-16T17:13:54atlas_cvmfs:latestcoridockerREADY9ff944a24c2015-12-18T03:08:09marius311/run_sim:latestcoridockerREADY56e5a8a6e02016-01-07T15:24:40marius311/mpitest:latestcoridockerREADYdf9675993b2015-12-21T14:47:19paterno/centos67-art_v1_18_02-e9-prof:latestcoridockerREADYc0f009e6672015-12-22T09:37:10fenicsproject:dev:latestcoridockerREADY0f73fcv8a2015-12-22T13:08:00fenicsproject/stable:latestcoridockerREADYf1c24227b72015-12-22T10:54:20jbkowalkowski/art_test:latestcoricustomREADY3f2d105bb22015-12-23T17:06:47cms_cvmfs:latestcoridockerREADYe51ffe812d2016-01-07T11:46:19paterno/centos-uboone_v04_31_00-e9-prof:latestcoricustomREADY9e6c507dae2016-01-11T22:40:36cvmfs_test:latestcoricustomREADY8e244988952016-01-12T16:45:12cvmfs_mpitest:latestcoricustomREADYe5da39eb022016-01-13T17:37:11alice_cvmfs:latest

Running A Job in A Shifter Image

-16-

#!/bin/bash#SBATCH--image=docker:image_name:latest#SBATCH--volume="/global/cscratch1/sd/lgerhard:/output”#SBATCH--volume=“/global/cscratch1/sd/lgerhard/shi3er_tmp:/tmp:perNodeCache=size=200G”#SBATCH--nodes=1#SBATCH--par>>on=regular#SBATCH-Chaswellsrun-n32shiEerpythonmyPythonScript.pyargs

ManymorecommandsathJp://www.nersc.gov/users/so3ware/using-shi3er-and-docker/using-shi3er-at-nersc/