Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x....
Transcript of Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x....
![Page 1: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/1.jpg)
LLNL-PRES-806064 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC
github.com/spack/spack
Spack:APackageManagerforHPCToddGamblin
AdvancedTechnologyOffice,LivermoreComputingLawrenceLivermoreNationalLaboratory
NCSA Blue Waters Webinar October 30, 2019
![Page 2: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/2.jpg)
LLNL-PRES-806064github.com/spack/spack 2
SoftwarecomplexityinHPCisgrowing
Ascent: Lightweight, in-situ, many-core visualization and analysis
![Page 3: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/3.jpg)
LLNL-PRES-806064github.com/spack/spack 3
MFEM: Arbitrary high-order finite elements
Ascent: Lightweight, in-situ, many-core visualization and analysis
Software complexity in HPC is growing
![Page 4: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/4.jpg)
LLNL-PRES-806064github.com/spack/spack 4
LBANN: Artificial Neural Nets for HPCMFEM: Arbitrary high-order finite elements
Software complexity in HPC is growing
Ascent: Lightweight, in-situ, many-core visualization and analysis
![Page 5: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/5.jpg)
LLNL-PRES-806064github.com/spack/spack 5
Thecomplexityoftheexascaleecosystemthreatensproductivity.
▪ Everyapplicationhasitsownstackofdependencies.▪ Developers,users,andfacilitiesdedicate(many)FTEstobuilding&porting.▪ Oftentradereuseandusabilityforperformance.
80+ software packagesx 5+ target architectures/platforms Xeon Power KNL
NVIDIA ARM Laptops?x
Up to 7 compilers Intel GCC Clang XL
PGI Cray NAGx
= up to 1,260,000 combinations!
15+ applications
x10+ Programming Models
OpenMPI MPICH MVAPICH OpenMP CUDA OpenACC Dharma Legion RAJA Kokkos
2-3 versions of each package + external dependencies
x
We must make it easier to rely on others’ software!
![Page 6: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/6.jpg)
LLNL-PRES-806064github.com/spack/spack 6
HowtoinstallsoftwareonaMaclaptop,circa2013
![Page 7: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/7.jpg)
LLNL-PRES-806064github.com/spack/spack 7
Howtoinstallsoftwareonasupercomputer
1. Download all 16 tarballs you need
2. Start building!
![Page 8: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/8.jpg)
LLNL-PRES-806064github.com/spack/spack 7
Howtoinstallsoftwareonasupercomputer
configure
make
Fight with compiler...
make
Tweak configure args...
make install
make
configure
configure
make
make install
cmake
make
make install
1. Download all 16 tarballs you need
2. Start building!
![Page 9: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/9.jpg)
LLNL-PRES-806064github.com/spack/spack 7
Howtoinstallsoftwareonasupercomputer
configure
make
Fight with compiler...
make
Tweak configure args...
make install
make
configure
configure
make
make install
cmake
make
make install
1. Download all 16 tarballs you need
2. Start building!
3. Run code4. Segfault!?5. Start over…
![Page 10: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/10.jpg)
LLNL-PRES-806064github.com/spack/spack 8
▪ Mostsupercomputersdeploysomeformofenvironmentmodules— TCLmodules(datesbackto1995)andLmod(fromTACC)arethemostpopular
▪ Modulesdon’thandleinstallation!— Theyonlymodifyyourenvironment(thingslikePATH,LD_LIBRARY_PATH,etc.)
▪ Someone(likelyateamofpeople)hasalreadyinstalledgccforyou!— Also,youcanonly`moduleload`thethingsthey’veinstalled
Whataboutmodules?
$ gcc-bash: gcc: command not found
$ module load gcc/7.0.1$ gcc –dumpversion7.0.1
![Page 11: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/11.jpg)
LLNL-PRES-806064github.com/spack/spack 9
▪ Containersprovideagreatwaytoreproduceanddistributeanalready-builtsoftwarestack
▪ Someoneneedstobuildthecontainer!— Thisisn’ttrivial— Containerizedapplicationsstillhavehundredsofdependencies
▪ UsingtheOSpackagemanagerinsideacontainerisinsufficient— Mostbinariesarebuiltunoptimized— Genericbinaries,notoptimizedforspecificarchitectures
▪ HPCcontainersmayneedtoberebuilttosupportmanydifferenthosts,anyway.— Notclearthatwecaneverbuildonecontainerforallfacilities— Containerslikelywon’tsolvetheN-platformsprobleminHPC
Whataboutcontainers?
We need something more flexible to build the containers
![Page 12: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/12.jpg)
LLNL-PRES-806064github.com/spack/spack 10
• Spackautomatesthebuildandinstallationofscientificsoftware• Packagesaretemplated,sothatuserscaneasilytuneforthehostenvironment
• Easeofuseofmainstreamtools,withflexibilityneededforHPCtuning
• Majorvictories:• ARESportingtimeonanewplatformwasreducedfrom2weeksto3hours• Deploymenttimefor1,300-packagestackonSummitsupercomputerreducedfrom2weekstoa12-hourovernightbuild
• UsedbyteamsacrossECPtoacceleratedevelopment
$ spack install [email protected] $ spack install [email protected] %[email protected] $ spack install [email protected] +threadssafe
$ spack install [email protected] cppflags="-O3 –g3" $ spack install [email protected] target=haswell $ spack install [email protected] +mpi ^[email protected]
$gitclonehttps://github.com/spack/spack$spackinstallhdf5
No installation required: clone and go
Simple syntax enables complex installs
github.com/spack/spack
SpackisaflexiblepackagemanagerforHPC
![Page 13: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/13.jpg)
LLNL-PRES-806064github.com/spack/spack 11
PeoplewhowanttouseordistributesoftwareforHPC!
1. EndUsersofHPCSoftware— InstallandrunHPCapplicationsandtools
2. HPCApplicationTeams— Managethird-partydependencylibraries
3. PackageDevelopers— Peoplewhowanttopackagetheirownsoftwarefordistribution
4. UsersupportteamsatHPCCenters— PeoplewhodeploysoftwareforusersatlargeHPCsites
WhocanuseSpack?
![Page 14: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/14.jpg)
LLNL-PRES-806064github.com/spack/spack 12
Spackisusedworldwide!
Over 450 contributors from labs, academia, industry
Over 3,500 software packages Over 2,000 monthly active users (on docs site)
Plot shows users on spack.readthedocs.io for one month
![Page 15: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/15.jpg)
LLNL-PRES-806064github.com/spack/spack 13
ActiveUsersonthespack.readthedocs.io
![Page 16: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/16.jpg)
LLNL-PRES-806064github.com/spack/spack 14
SpackisbeingusedonmanyofthetopHPCsystems
▪ OfficialdeploymenttoolfortheU.S.ExascaleComputingProject
▪ 7ofthetop10supercomputers▪ HighEnergyPhysicscommunity
— Fermilab,CERN,collaborators▪ Astra(Sandia)▪ Fugaku(JapaneseNationalSupercomputerProject)
Summit (ORNL), Sierra (LLNL) Edison, Cori, Perlmutter (NERSC)SuperMUC-NG (LRZ, Germany)
Fugaku coming to RIKEN in 2021 DOE/MEXT collaboration
![Page 17: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/17.jpg)
LLNL-PRES-806064github.com/spack/spack 15
▪ InNovember2015,LLNLprovidedmostofthecontributionstoSpack
▪ Sincethen,we’vegonefrom300toover3,500packages
▪ Mostpackagesarefromexternalcontributors!
▪ Manycontributionsincore,aswell.
▪ WearecommittedtosustainingSpack’sopensourceecosystem!
ContributionstoSpackcontinuetogrow!
![Page 18: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/18.jpg)
LLNL-PRES-806064github.com/spack/spack 16
Spackisnotthefirsttooltoautomatebuilds— Inspiredbycopiouspriorwork
1. “Functional”PackageManagers— Nix https://nixos.org/— GNUGuix https://www.gnu.org/s/guix/
2. Build-from-sourcePackageManagers— Homebrew http://brew.sh— MacPorts https://www.macports.org
OthertoolsintheHPCSpace:
▪ Easybuild http://hpcugent.github.io/easybuild/— AninstallationtoolforHPC— FocusedonHPCsystemadministrators–differentpackagemodelfromSpack— Reliesonafixedsoftwarestack–hardertotweakrecipesforexperimentation
▪ Conda https://conda.io— Verypopularbinarypackagemanagerfordatascience— NottargetedatHPC;generallyunoptimizedbinaries
RelatedWork
![Page 19: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/19.jpg)
LLNL-PRES-806064github.com/spack/spack 17
SpackBasics
![Page 20: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/20.jpg)
LLNL-PRES-806064github.com/spack/spack 18
▪ Eachexpressionisaspecforaparticularconfiguration— Eachclauseaddsaconstrainttothespec— Constraintsareoptional–specifyonlywhatyouneed.— Customizeinstallonthecommandline!
▪ Specsyntaxisrecursive— Fullcontroloverthecombinatorialbuildspace
SpackprovidesaspecsyntaxtodescribecustomizedDAGconfigurations
$ spack install mpileaks unconstrained$ spack install [email protected] @ custom version$ spack install [email protected] %[email protected] % custom compiler$ spack install [email protected] %[email protected] +threads +/- build option$ spack install [email protected] cppflags="-O3 –g3" set compiler flags$ spack install [email protected] target=skylake set target microarchitecture$ spack install [email protected] ^[email protected] %[email protected] ^ dependency information
![Page 21: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/21.jpg)
LLNL-PRES-806064github.com/spack/spack 19
`spacklist`showswhatpackagesareavailable
$ spack list==> 303 packages.activeharmony cgal fish gtkplus libgd mesa openmpi py-coverage py-pycparser qt tcladept-utils cgm flex harfbuzz libgpg-error metis openspeedshop py-cython py-pyelftools qthreads texinfoapex cityhash fltk hdf libjpeg-turbo Mitos openssl py-dateutil py-pygments R the_silver_searcherarpack cleverleaf flux hdf5 libjson-c mpc otf py-epydoc py-pylint ravel thriftasciidoc cloog fontconfig hwloc libmng mpe2 otf2 py-funcsigs py-pypar readline tkatk cmake freetype hypre libmonitor mpfr pango py-genders py-pyparsing rose tmuxatlas cmocka gasnet icu libNBC mpibash papi py-gnuplot py-pyqt rsync tmuxinatoratop coreutils gcc icu4c libpciaccess mpich paraver py-h5py py-pyside ruby trilinosautoconf cppcheck gdb ImageMagick libpng mpileaks paraview py-ipython py-pytables SAMRAI uncrustifyautomaded cram gdk-pixbuf isl libsodium mrnet parmetis py-libxml2 py-python-daemon samtools util-linuxautomake cscope geos jdk libtiff mumps parpack py-lockfile py-pytz scalasca valgrindbear cube gflags jemalloc libtool munge patchelf py-mako py-rpy2 scorep vimbib2xhtml curl ghostscript jpeg libunwind muster pcre py-matplotlib py-scientificpython scotch vtkbinutils czmq git judy libuuid mvapich2 pcre2 py-mock py-scikit-learn scr wgetbison damselfly glib julia libxcb nasm pdt py-mpi4py py-scipy silo wxboost dbus glm launchmon libxml2 ncdu petsc py-mx py-setuptools snappy wxpropgridbowtie2 docbook-xml global lcms libxshmfence ncurses pidx py-mysqldb1 py-shiboken sparsehash xcb-protoboxlib doxygen glog leveldb libxslt netcdf pixman py-nose py-sip spindle xerces-cbzip2 dri2proto glpk libarchive llvm netgauge pkg-config py-numexpr py-six spot xzcairo dtcmp gmp libcerf llvm-lld netlib-blas pmgr_collective py-numpy py-sphinx sqlite yasmcallpath dyninst gmsh libcircle lmdb netlib-lapack postgresql py-pandas py-sympy stat zeromqcblas eigen gnuplot libdrm lmod netlib-scalapack ppl py-pbr py-tappy sundials zlibcbtf elfutils gnutls libdwarf lua nettle protobuf py-periodictable py-twisted swig zshcbtf-argonavis
elpa
gperf
libedit
lwgrp
ninja py-astropy py-pexpect py-urwid szipcbtf-krell
expat
gperftools
libelf
lwm2
ompss py-basemap py-pil py-virtualenv tarcbtf-lanl
extrae
graphlib
libevent
matio
ompt-openmp py-biopython py-pillow py-yapf taskcereal exuberant-ctags graphviz libffi mbedtls opari2 py-blessings py-pmw python taskd cfitsio fftw gsl libgcrypt memaxes openblas py-cffi py-pychecker qhull tau
▪ Spackhasover3,500packagesnow.
![Page 22: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/22.jpg)
LLNL-PRES-806064github.com/spack/spack 20
▪ Alltheversionscoexist!— Multipleversionsof
samepackageareok.
▪ Packagesareinstalledtoautomaticallyfindcorrectdependencies.
▪ Binariesworkregardlessofuser’senvironment.
▪ Spackalsogeneratesmodulefiles.— Don’thavetousethem.
`spackfind`showswhatisinstalled
$ spack find==> 103 installed packages.-- linux-rhel6-x86_64 / [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]@3.9.1 [email protected] [email protected] [email protected] [email protected]@1.0 [email protected] [email protected] [email protected] [email protected]@2.14.0 [email protected] [email protected] [email protected] [email protected]@1.55.0 [email protected] [email protected] [email protected] [email protected]@1.14.0 [email protected] [email protected] [email protected] [email protected]@1.0.2 jpeg@9a [email protected] [email protected] [email protected]@8.1.2 libdwarf@20130729 [email protected] [email protected] [email protected]@8.1.2 [email protected] ocr@2015-02-16 [email protected] [email protected]@2.11.1 [email protected] [email protected] [email protected]@2.5.3 [email protected] [email protected] [email protected]@2.31.2 [email protected] [email protected] [email protected]
-- linux-rhel6-x86_64 / [email protected] [email protected] [email protected] [email protected] libdwarf@20130729 [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]
-- linux-rhel6-x86_64 / [email protected] [email protected] [email protected] [email protected]
-- linux-rhel6-x86_64 / [email protected] [email protected] [email protected] libdwarf@20130729 [email protected] [email protected]
-- linux-rhel6-x86_64 / [email protected] [email protected] [email protected] libdwarf@20130729 [email protected] [email protected] [email protected] [email protected] [email protected]
![Page 23: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/23.jpg)
LLNL-PRES-806064github.com/spack/spack 21
Userscanquerythefulldependencyconfigurationofinstalledpackages.
$ spack find callpath==> 2 installed packages.-- linux-rhel6-x86_64 / [email protected] —— -- linux-rhel6-x86_64 / [email protected] [email protected] [email protected]
Expand dependencies with spack find -d
$ spack find -dl callpath==> 2 installed packages.-- linux-x86_64 / [email protected] ----------- -- linux-x86_64 / [email protected] -----------xv2clz2 [email protected] udltshs [email protected] ^[email protected] rfsu7fb ^[email protected] ^[email protected] ybet64y ^[email protected] ^[email protected] aa4ar6i ^[email protected] ^[email protected] tmnnge5 ^[email protected] ^[email protected] ybet64y ^[email protected] ^libdwarf@20130729 g2mxrl2 ^libdwarf@20130729cj5p5fk ^[email protected] ynpai3j ^[email protected] ^[email protected] ynpai3j ^[email protected] ^libdwarf@20130729 g2mxrl2 ^libdwarf@20130729cj5p5fk ^[email protected] ynpai3j ^[email protected] ^[email protected] ynpai3j ^[email protected] ^[email protected] aa4ar6i ^[email protected]
$ spack find -dl callpath==> 2 installed packages.-- linux-rhel6-x86_64 / [email protected] ----- -- linux-rhel6-x86_64 / [email protected] -----xv2clz2 [email protected] udltshs [email protected] ^[email protected] rfsu7fb ^[email protected] ^[email protected] ybet64y ^[email protected] ^[email protected] aa4ar6i ^[email protected] ^[email protected] tmnnge5 ^[email protected] ^[email protected] ybet64y ^[email protected] ^libdwarf@20130729 g2mxrl2 ^libdwarf@20130729cj5p5fk ^[email protected] ynpai3j ^[email protected] ^[email protected] ynpai3j ^[email protected] ^libdwarf@20130729 g2mxrl2 ^libdwarf@20130729cj5p5fk ^[email protected] ynpai3j ^[email protected] ^[email protected] ynpai3j ^[email protected] ^[email protected] aa4ar6i ^[email protected]
▪ Architecture, compiler, versions and variants may differ between the builds.
![Page 24: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/24.jpg)
LLNL-PRES-806064github.com/spack/spack 22
Spackmanagesinstalledcompilers
▪ Compilersareautomaticallydetected— AutomaticdetectiondeterminedbyOS— Linux:PATH— Cray:`moduleavail`
▪ Compilerscanbemanuallyadded— IncludingSpack-builtcompilers
compilers:- compiler: modules: [] operating_system: ubuntu14 paths: cc: /usr/bin/gcc/4.9.3/gcc cxx: /usr/bin/gcc/4.9.3/g++ f77: /usr/bin/gcc/4.9.3/gfortran fc: /usr/bin/gcc/4.9.3/gfortran spec: [email protected] compiler: modules: [] operating_system: ubuntu14 paths: cc: /usr/bin/clang/6.0/clang cxx: /usr/bin/clang/6.0/clang++ f77: null fc: null spec: [email protected] - compiler: ...
$ spack compilers==> Available compilers-- gcc [email protected] [email protected]
-- clang [email protected]
compilers.yaml
![Page 25: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/25.jpg)
LLNL-PRES-806064github.com/spack/spack 23
CoreSpackConcepts
![Page 26: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/26.jpg)
LLNL-PRES-806064github.com/spack/spack 24
▪ Traditionalbinarypackagemanagers— RPM,yum,APT,yast,etc.— Designedtomanageasinglestack.— Installoneversionofeachpackageinasingleprefix(/usr).— Seamlessupgradestoastable,welltestedstack
▪ Portsystems— BSDPorts,portage,Macports,Homebrew,Gentoo,etc.— Minimalsupportforbuildsparameterizedbycompilers,dependencyversions.
▪ VirtualMachinesandLinuxContainers(Docker)— Containersallowuserstobuildenvironmentsfordifferentapplications.— Doesnotsolvethebuildproblem(someonehastobuildtheimage)— Performance,security,andupgradeissuespreventwidespreadHPCdeployment.
Mostexistingtoolsdonotsupportcombinatorialversioning
![Page 27: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/27.jpg)
LLNL-PRES-806064github.com/spack/spack 25
▪ Eachuniquedependencygraphisauniqueconfiguration.
▪ Eachconfigurationinstalledinauniquedirectory.— Configurationsofthesamepackagecancoexist.
▪ Hashofentiredirectedacyclicgraph(DAG)isappendedtoeachprefix.
▪ Installedpackagesautomaticallyfinddependencies— SpackembedsRPATHsinbinaries.— NoneedtousemodulesorsetLD_LIBRARY_PATH— Thingsworkthewayyoubuiltthem
Spackhandlescombinatorialsoftwarecomplexity.
spack/opt/ linux-x86_64/ gcc-4.7.2/ mpileaks-1.1-0f54bf34cadk/ intel-14.1/ hdf5-1.8.15-lkf14aq3nqiz/ bgq/ xl-12.1/ hdf5-1-8.16-fqb3a15abrwx/ ...
InstallationLayout
DependencyDAG
Hash
![Page 28: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/28.jpg)
LLNL-PRES-806064github.com/spack/spack 26
▪ SpackensuresoneconfigurationofeachlibraryperDAG— EnsuresABIconsistency.— UserdoesnotneedtoknowDAGstructure;onlythedependencynames.
▪ Spackcanensurethatbuildsusethesamecompiler,oryoucanmix— WorkingonensuringABIcompatibilitywhencompilersaremixed.
SpackSpecscanconstrainversionsofdependencies
$ spack install mpileaks %[email protected] ^[email protected]
![Page 29: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/29.jpg)
LLNL-PRES-806064github.com/spack/spack 27
SpackhandlesABI-incompatible,versionedinterfaceslikeMPI
$ spack install mpileaks ^[email protected] $ spack install mpileaks ^[email protected]:
$ spack install mpileaks ^mpi@2
▪ mpiisavirtualdependency
▪ InstallthesamepackagebuiltwithtwodifferentMPIimplementations:
▪ LetSpackchooseMPIimplementation,aslongasitprovidesMPI2interface:
![Page 30: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/30.jpg)
LLNL-PRES-806064github.com/spack/spack 28
Concretizationfillsinmissingconfigurationdetailswhentheuserisnotexplicit.
mpileaks ^[email protected]+debug ^[email protected] User input: abstract spec with some constraints
Concrete spec is fully constrained and can be passed to install.
Abstract, normalized spec with some dependencies.
Norm
alize
Concretize Store
spec:- mpileaks: arch: linux-x86_64 compiler: name: gcc version: 4.9.2 dependencies: adept-utils: kszrtkpbzac3ss2ixcjkcorlaybnptp4 callpath: bah5f4h4d2n47mgycej2mtrnrivvxy77 mpich: aa4ar6ifj23yijqmdabeakpejcli72t3 hash: 33hjjhxi7p6gyzn5ptgyes7sghyprujh variants: {} version: '1.0'- adept-utils: arch: linux-x86_64 compiler: name: gcc version: 4.9.2 dependencies: boost: teesjv7ehpe5ksspjim5dk43a7qnowlq mpich: aa4ar6ifj23yijqmdabeakpejcli72t3 hash: kszrtkpbzac3ss2ixcjkcorlaybnptp4 variants: {} version: 1.0.1- boost: arch: linux-x86_64 compiler: name: gcc version: 4.9.2 dependencies: {} hash: teesjv7ehpe5ksspjim5dk43a7qnowlq variants: {} version: 1.59.0...
spec.yaml
Detailed provenance is stored with the installed package
![Page 31: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/31.jpg)
LLNL-PRES-806064github.com/spack/spack 29
Use`spackspec`toseetheresultsofconcretization
$ spack spec mpileaksInput spec------------------------------ mpileaks
Concretized------------------------------ [email protected]%[email protected] arch=darwin-elcapitan-x86_64 ^[email protected]%[email protected] arch=darwin-elcapitan-x86_64 ^[email protected]%[email protected]+atomic+chrono+date_time~debug+filesystem~graph ~icu_support+iostreams+locale+log+math~mpi+multithreaded+program_options ~python+random +regex+serialization+shared+signals+singlethreaded+system +test+thread+timer+wave arch=darwin-elcapitan-x86_64 ^[email protected]%[email protected] arch=darwin-elcapitan-x86_64 ^[email protected]%[email protected] arch=darwin-elcapitan-x86_64 ^[email protected]%[email protected]~mxm~pmi~psm~psm2~slurm~sqlite3~thread_multiple~tm~verbs+vt arch=darwin-elcapitan-x86_64 ^[email protected]%[email protected] arch=darwin-elcapitan-x86_64 ^[email protected]%[email protected] arch=darwin-elcapitan-x86_64 ^[email protected]%[email protected] arch=darwin-elcapitan-x86_64 ^[email protected]%[email protected]+sigsegv arch=darwin-elcapitan-x86_64 ^[email protected]%[email protected] arch=darwin-elcapitan-x86_64 ^[email protected]%[email protected] arch=darwin-elcapitan-x86_64 ^[email protected]%[email protected]~stat_dysect arch=darwin-elcapitan-x86_64 ^libdwarf@20160507%[email protected] arch=darwin-elcapitan-x86_64 ^[email protected]%[email protected] arch=darwin-elcapitan-x86_64
![Page 32: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/32.jpg)
LLNL-PRES-806064github.com/spack/spack 30
SpackpackagesaretemplatesTheyuseasimplePythonDSLtodefinehowtobuild
Metadata at the class level
Versions
Install logic in instance methods
Dependencies (note: same spec syntax)
Not shown: patches, resources, conflicts, other directives.
from spack import *
class Kripke(CMakePackage): """Kripke is a simple, scalable, 3D Sn deterministic particle transport proxy/mini app. """
homepage = "https://computation.llnl.gov/projects/co-design/kripke" url = "https://computation.llnl.gov/projects/co-design/download/kripke-openmp-1.1.tar.gz"
version(‘1.2.3’, sha256='3f7f2eef0d1ba5825780d626741eb0b3f026a096048d7ec4794d2a7dfbe2b8a6’) version(‘1.2.2’, sha256='eaf9ddf562416974157b34d00c3a1c880fc5296fce2aa2efa039a86e0976f3a3’) version('1.1’, sha256='232d74072fc7b848fa2adc8a1bc839ae8fb5f96d50224186601f55554a25f64a’)
variant('mpi', default=True, description='Build with MPI.’) variant('openmp', default=True, description='Build with OpenMP enabled.’)
depends_on('mpi', when='+mpi’) depends_on('[email protected]:', type='build’)
def cmake_args(self): return [ '-DENABLE_OPENMP=%s’ % ('+openmp’ in self.spec), '-DENABLE_MPI=%s' % ('+mpi’ in self.spec), ]
def install(self, spec, prefix): # Kripke does not provide install target, so we have to copy # things into place. mkdirp(prefix.bin) install('../spack-build/kripke', prefix.bin)
Base package (CMake support)
Variants (build options)
Don’t typically need install() for CMakePackage, but we can work around codes that don’t have it.
![Page 33: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/33.jpg)
LLNL-PRES-806064github.com/spack/spack 31
Spackbuildseachpackageinitsowncompilationenvironment
SpackProcess
Setupenvironment
CC = spack/env/spack-cc SPACK_CC = /opt/ic-15.1/bin/iccCXX = spack/env/spack-c++ SPACK_CXX = /opt/ic-15.1/bin/icpcF77 = spack/env/spack-f77 SPACK_F77 = /opt/ic-15.1/bin/ifortFC = spack/env/spack-f90 SPACK_FC = /opt/ic-15.1/bin/ifort
PKG_CONFIG_PATH = ... PATH = spack/env:$PATHCMAKE_PREFIX_PATH = ...LIBRARY_PATH = ...
do_install()
Install dep1 Install dep2 Install package…
BuildProcess
Fork
install() configure make makeinstall
-I /dep1-prefix/include-L /dep1-prefix/lib-Wl,-rpath=/dep1-prefix/lib
Compilerwrappers(spack-cc, spack-c++, spack-f77, spack-f90)
icc icpc ifort
▪ Forkedbuildprocessisolatesenvironmentforeachbuild.Usescompilerwrappersto:— Addinclude,lib,andRPATHflags— Ensurethatdependenciesarefoundautomatically— LoadCraymodules(userightcompiler/systemdeps)
![Page 34: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/34.jpg)
LLNL-PRES-806064github.com/spack/spack 32
Someadvancedfeatures
![Page 35: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/35.jpg)
LLNL-PRES-806064github.com/spack/spack 33
AdvancedTopicsinPackaging
▪ Spacktriestoautomaticallyconfigurepackageswithinformationfromdependencies— Buttherearemanyspecialcases.Oftenyouneedtoretrievedetails
aboutdependenciestoconfigureproperly
▪ Thegoalistoanswerthefollowingquestionsthatcomeupwhenwritingpackagefiles:— HowdoIretrievedependencylibraries/headerswhenconfiguringmy
package?— Howdoesspackhelpmeconfiguremybuild-timeenvironment?
▪ We’llstartwithaclientviewandthenlookathowweaddfunctionalitytopackagestomakeiteasierfordependents
Dependent (client)
dependency
![Page 36: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/36.jpg)
LLNL-PRES-806064github.com/spack/spack 34
AccessingDependencyLibraries
▪ AlthoughSpackperformssomeworktohelpabuildfindlibraries,youmayneedtoexplicitlyspecifydependencylibrariesduringconfiguration
▪ Specsprovidea.libspropertywhichretrievestheindividuallibraryfilesprovidedbythepackage
▪ Accessing.libsforavirtualpackagewillretrievethelibrariesprovidedbythechosenimplementation
.libs.joined() expresses the list of libraries as a single string like: "/…/lib1.so;/…/lib2.so" (e.g. for cmake)
.libs.search_flags expresses the libraries as linker arguments like: "-L/…/libdir1/ -L/…/libdir2/" (e.g. as an argument to the compiler)
![Page 37: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/37.jpg)
LLNL-PRES-806064github.com/spack/spack 35
AccessingDependencyLibraries:Virtuals
arpack-ng
netlib-lapack +external-blas essl
spec[‘blas’].libsspec[‘lapack’].libs
arpack-ng
netlib-lapack ~external-blas
spec[‘lapack’].libs spec[‘blas’].libs
▪ Theclientsidecodeforaccessing“.libs”isthesameregardlessofwhichimplementationofblasisused
▪ Asaclient,youdon’thavetocarewhether‘blas’and‘lapack’areprovidedbythesameimplementation
![Page 38: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/38.jpg)
LLNL-PRES-806064github.com/spack/spack 36
What’sNew?What’sontheRoadMap?
![Page 39: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/39.jpg)
LLNL-PRES-806064github.com/spack/spack 37
WeareworkingtoenableoptimizedsoftwaredistributionforHPC
• Distribution effort required is similar to efforts like Red Hat, Debian, Ubuntu, etc. – Curation and vetting of software – Packaging, building – Wide distribution
• HPC community is not as mainstream, not as widespread as these distributions
• HPC platform complexity poses challenges – Many (often unique) platforms – Many software ecosystems – From-source distribution – Must support Optimization, GPUs, fast networks
• Much more automation is required to practically support our ecosystem!
![Page 40: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/40.jpg)
LLNL-PRES-806064github.com/spack/spack 38
Ourstrategyistoenableexascalesoftwaredistributiononbothbaremetalandcontainers
– New capabilities to make HPC packaging easy and automated • Optimized builds and package binaries that exploit the hardware • Workflow automation for facilities, developers, and users • Strong integration with containers as well as bare metal deployments
– Work with ECP and other partners to harden packages • Build pipelines at facilities • Coordination on multi-site testing • Security integration
– Outreach to users • Tutorials, workshops, BOFs
Container Runtimes
Spack Packaging
Bare Metal
Build / Deployment Automation
![Page 41: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/41.jpg)
LLNL-PRES-806064github.com/spack/spack 39
▪ Wehavedevelopedacross-platformlibrarytodetectandcomparemicroarchitecturemetadata— Detectsbasedon/proc/cpuinfo(Linux),sysctl(Mac)— Allowscomparisonsforcompatibility,e.g.:
▪ Keyfeatures:— Knowwhichcompilerssupportwhichchips/whichflags— Determinecompatibility— Enablecreationandreuseofoptimizedbinarypackages— Easilyqueryavailablearchitecturefeaturesforportable
buildrecipes
▪ Wewillbeextractingthisasastandalonelibraryforothertools&languages— Hopetomakethisstandard!
Spacknowunderstandsspecifictargetmicroarchitectures
$spackinstalllbanntarget=cascadelake
$spackinstallpetsctarget=zen2
Specialized installationsSimple feature query
Extensive microarchitecture knowledge
skylake > broadwell zen2 > x86_64
![Page 42: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/42.jpg)
LLNL-PRES-806064github.com/spack/spack 40
▪ AllowsdeveloperstobundleSpackconfigurationwiththeirrepository
▪ CanalsobeusedtomaintainconfigurationtogetherwithSpackpackages.— E.g.,versioningyourownlocalsoftwarestackwithconsistentcompilers/
MPIimplementations
▪ Manifest/LockfilemodelpioneeredbyBundlerisbecomingstandard— spack.yamldescribesprojectrequirements— spack.lockdescribesexactlywhatversions/configurationswereinstalled,
allowsthemtobereproduced.
Spackenvironmentsenableuserstobuildcustomizedstacksfromanabstractdescription
Simple spack.yaml file
installbuild
projectspack.yaml file with names of required
dependencies
Lockfile describes exact versions installed
Dependency packages
Concrete spack.lock file (generated)
![Page 43: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/43.jpg)
LLNL-PRES-806064github.com/spack/spack 41
▪ WerecentlystartedprovidingbaseimagesonDockerHubwithSpackpreinstalled.
▪ VeryeasytobuildacontainerwithsomeSpackpackagesinit:
WehavesimplifiedcontainerdeploymentsusingSpackEnvironments
spack-docker-demo/Dockerfilespack.yaml
Base image with Spack in PATH
Copy in spack.yaml Then run spack install
List of packages to install, with constraints
Build with docker build .
RunwithSingularity(orsomeothertool)
![Page 44: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/44.jpg)
LLNL-PRES-806064github.com/spack/spack 42
▪ Allowuserstoeasilyexpressahugecross-productofspecs— Allthepackagesneededforafacility— Generatemodulestailoredtothesite— Generateadirectorylayouttobrowsethepackages
▪ Buildontheenvironmentsworkflow— Manifest+lockfile— Lockfileenablesreproducibility
▪ Relocatablebinariesallowthesamebinarytobeusedinastack,regularinstall,orcontainerbuild.— Differenceishowtheuserinteractswiththestack— Single-PATHstackvs.modules.
WehavedevelopedSpackstacks:combinatorialenvironmentsforentirefacilitydeployments
![Page 45: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/45.jpg)
LLNL-PRES-806064github.com/spack/spack 43
▪ WehaveaddedsecurityfeaturestotheopensourceGitLabproduct.— Integrationwithcenteridentitymanagement— IntegrationwithschedulerslikeSLURM,LSF
▪ WearedemocratizingtestingatLivermoreComputing— Userscanruntestsacross30+machinesbyeditingafile— Previously,eachteamhadtoadministerownservers
▪ ECPsitesaredeployingGitLabCIforusers— AllHPCcenterscanleveragetheseimprovements— NNSAlabsplantodeploycommonhigh-sideCIinfrastructure— Wearedevelopingnewsecuritypoliciestoallowexternal
opensourcecodetobetestedsafelyonkeymachines
WehavebeenheavilyinvolvedintheECPCIproject.
. . .
User commits to GitLab
GitLab test runners are now integrated with HPC machines
![Page 46: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/46.jpg)
LLNL-PRES-806064github.com/spack/spack 44
▪ BuildsonSpackenvironments— Supportauto-generatingGitLabCIjobs— CanruninaKubeclusteroronbaremetalrunnersatan
HPCsite— SendsprogresstoCDash
SpackhasaddedGitLabCIintegrationtoautomatepackagebuildpipelines
![Page 47: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/47.jpg)
LLNL-PRES-806064github.com/spack/spack 45
ECPisworkingtowardsaperiodic,hierarchicalreleaseprocess
• ECP teams work to ensure that libraries and components work together – Historically, HPC codes used very few dependencies
• Now, groups of teams work together on small releases of “Software Development Kits”
• SDKs are rolled into a larger, periodic release.
Develop
Package
Build
Test
Deploy
Math Libraries
Develop
Package
Build
Test
Deploy
Visualization
Develop
Package
Build
Test
Deploy
Programming Models
…
Build
TestDeploy
Integrate
E4S
ECP-wide software release
https://e4s.io
![Page 48: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/48.jpg)
LLNL-PRES-806064github.com/spack/spack 46
LLNLLANLSNLANL
ORNL
NERSC
AutomatedbuildsusingECPCIwillenablearobust,widelyavailableHPCsoftwareecosystem.
Spack users
Automated package builds
WithpipelineeffortsatE6labs,userswillnolongerneedtobuildtheirownsoftwareforhighperformance.
Per-laboratory pipelinesPublic and private
package repositories
![Page 49: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/49.jpg)
LLNL-PRES-806064github.com/spack/spack 47
SpackfocusareasinFY20
• Multi-stage container generation with Spack– Add support to Spack to generate multi-stage container builds that
exclude build dependencies from artifacts automatically
• Build Hardening with Spack Pipelines– Continue working with E4S team to harden container builds
• Parallel builds– “srun spack install” will use the entire allocation to build
• New concretizer based on fast ASP/SAT solvers
• Improved dependency models for compilers– icpc depends on g++ for its libstdc++, and other ABI nightmares
Build-time artifacts
Run-time artifacts
1
2 5
3 4
B
B
76
L
L
8
R
BL
Multi-stage build analysis
spack container build
![Page 50: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/50.jpg)
LLNL-PRES-806064github.com/spack/spack 48
▪ Therearelotsofwaystogetinvolved!— Contributepackages,documentation,orfeaturesatgithub.com/spack/
spack— Contributeyourconfigurationstogithub.com/spack/spack-configs
▪ Talktous!— JoinourSlackchannel(seeGitHubrepoforinfo)— JoinourGoogleGroup(seeGitHubrepoforinfo)— SubmitGitHubissuesandpullrequests!
▪ Docsandafulldaytutorialareavailableat:
spack.readthedocs.io
JointheSpackcommunity!
Tweet at us!@spackpm
spack.io
![Page 51: Spack: A Package Manager for HPC - Blue Waters...Often trade reuse and usability for performance. x. 80+ software packages. 5+ target architectures/platforms . Xeon Power KNL ... —](https://reader035.fdocuments.us/reader035/viewer/2022080719/5f7945370cc3e17fa5173c44/html5/thumbnails/51.jpg)