Draft version February 8, 2018Typeset using LATEX twocolumn style in AASTeX61

CYBERHUBS: VIRTUAL RESEARCH ENVIRONMENTS FOR ASTRONOMY

Falk Herwig,1, 2, 3 Robert Andrassy,1, 2 Nic Annau,1 Ondrea Clarkson,1, 2, 3 Benoit Cote,4, 1, 2, 3 Aaron D’Sa,5, 2 Sam Jones,6, 2, 3 BelaidMoa,7

Jericho O’Connell,1 David Porter,8 Christian Ritter,1, 2, 3 and PaulWoodward5, 2

1Department of Physics & Astronomy, University of Victoria, Victoria, Canada2Joint Institute for Nuclear Astrophysics, Center for the Evolution of the Elements, Michigan State University, USA3NuGrid Collaboration4Konkoly Observatory, Research Centre for Astronomy and Earth Sciences, Hungarian Academy of Sciences, Budapest, Hungary5LCSE, University of Minnesota, Minneapolis, USA6Los Alamos National Laboratory, Los Alamos, New Mexico, USA7Research Computing Services, University of Victoria, Victoria, Canada8MSI, University of Minnesota, Minneapolis, USA

(Received December 8, 2017; Revised Feb 1, 2018; Accepted acceptance date; Published published date)

Submitted to ApJS

ABSTRACT

Collaborations in astronomy and astrophysics are faced with numerous cyber infrastructure challenges, such as large data sets,the need to combine heterogeneous data sets, and the challenge to effectively collaborate on those large, heterogeneous datasets with significant processing requirements and complex science software tools. The cyberhubs system is an easy-to-deploypackage for small to medium-sized collaborations based on the Jupyter and Docker technology, that allows web-browser enabled,remote, interactive analytic access to shared data. It offers an initial step to address these challenges. The features and deploymentsteps of the system are described, as well as the requirements collection through an account of the different approaches todata structuring, handling and available analytic tools for the NuGrid and PPMstar collaborations. NuGrid is an internationalcollaboration that creates stellar evolution and explosion physics and nucleosynthesis simulation data. The PPMstar collaborationperforms large-scale 3D stellar hydrodynamics simulation of interior convection in the late phases of stellar evolution. Examplesof science that is presently performed on cyberhubs, in the areas 3D stellar hydrodynamic simulations, stellar evolution andnucleosynthesis and Galactic chemical evolution, are presented.

Corresponding author: Falk Herwigfherwig@uvic.ca

arX

iv:1

802.

0223

3v1

[as

tro-

ph.I

M]

6 F

eb 2

018

2

1. INTRODUCTION

New astronomical observatories, large surveys, and the lat-est generation of astrophysics simulation data sets, providethe opportunity to advance our understanding of the universeprofoundly. However, the sheer size and complexity of thenew data sets dictate rethinking of the current data analyticpractice which can often be a barrier to fully exploiting thescientific potential of these large data sets. The followingchallenges may be identified.

The typical size of data sets in astronomy & astrophysicscontinues to grow substantially. To name a few examples,optical waveband projects such as the Large Synoptic SurveyTelescope (LSST), radio facilities such as the Square Kilo-meter Array (SKA) pathfinders, as well as large data setsproduced by cosmological or stellar hydrodynamics simula-tions which will, in combination, produce 10s of petabytes ofscientific data that would not typically be held in one place.Such data sets can not be downloaded for processing andanalysis. Instead combining remote computer processing ca-pacity where the data is stored with the appropriate analyticand data processing software stacks is required. On top ofthat efficient remote access with visualization and interac-tion capabilities are needed to enable a distributed commu-nity to collectively explore these data sets. Instead of movingthe data to the processing machines, the processing pipelinesneed to be moved to where the data resides.

One of the great promises of the age of data science forastronomy is that the many multi-physics, multi-messenger,multi-wavelength and multi-epoch constraints that bear onmost problems in astronomy, can in fact be combined suc-cessfully when complex data interactions and collaborativecyber-research environments are constructed. This notion ofdata fusion between different types of observational data andsimulation data can reach its full potential when differentcommunities can come together in combining and sharingdata sets, analytical tools and pipelines, and derived re-sults.

A significant barrier in accomplishing this goal in practiceare authentication and access models. Resources are oftenprovided by national bodies or consortia with often burden-some access requirements and limitations. An effective re-search platform system would accept easily-available socialor otherwise broadly used third-party identities to allow flex-ible, international, collaborative access. Although singlesign-on technologies are emerging, the scientific communityis still far from adopting them.

Decades worth of investments in legacy software, tools,and workflows by many research groups may be lost andnever shared with a broader community or applied to newdata sets. The reproducibility of science is suffering whendata processing and analytic workflows can not be shared. Amodern research platform would provide a uniform execu-

tion environment that will liberate legacy software, unlock-ing its analytic value and that of associated data sets, makingthem available to others, and allowing them to reproduciblyinteract with analysis procedures and data sets of other re-searchers.

In this paper we describe cyberhubs, an easily-deployedservice that combines Juypter notebook or Jupyterlabdata and processing in a containerized environment with aprescribed software application toolbox and data collection.The system described here is the result of multi-year develop-ments and evolution, with the goal to address the limitationsand challenges, initially of the international NuGrid collabo-ration, and then in addition, those of the PPMstar collabora-tion. Both of these brought somewhat different demands andrequirements, that in combination are likely typical for a verywide range of use cases in astronomy and astrophysics.

The international Nuclesoynthesis Grid (NuGrid1) collab-oration has members from ≈ 20 institutions in many coun-tries (Pignatari & Herwig 2012). Since 2007, NuGrid hasbeen combining the required expertise from many differentscientists to generate the most comprehensive data sets forthe production of the elements in massive and low-mass starsto date (Pignatari et al. 2016; Ritter et al. 2017b). Around2009, when the first data set was created it had a size of≈ 5 TB, which is relatively small by today’s standards, how-ever still large enough that it is not easily transferred aroundthe globe on demand. The collaboration was faced with prob-lems that are common to any distributed, data-oriented col-laboration. Problem-specific processing and analytic toolsare developed, but deployment in different places is alwayscomplicated by a diverse set of computing environments.In the NuGrid collaboration, which brings together differ-ent communities, all of the three major operating systemsare used. Initially, three copies of the data sets were alwaysmaintained at three institutions in the UK, Switzerland andCanada through a tedious, time-consuming and error-pronesyncing system. Still, analytic and interactive access for re-searchers who were not at one of those three institutions wasvery limited through VNC sessions or X11 forwarding, andrequired account-level access sharing that would be probablyimpossible at most institutions today.

As a next step, the collaboration adopted CANFAR’sVOspace as a shared and mountable storage system. CAN-FAR is the Canadian Advanced Network for AstronomyResearch2, a consortium between NRC (National ResearchCouncil) Herzberg’s Canadian Astronomy Data Center(CADC) and Canadian university groups which aims tojointly address astronomy cyber-infrastructure challenges.

1 http://www.nugridstars.og2 http://www.canfar.net

3

VOspace provides shareable user storage with a web inter-face, and identity and group management system, similar tocommercial cloud storage systems such as Google Drive andDropbox. It also has a Python API vos that allows commandline access to VOspace and POSIX mounting to a local lap-top or workstation. The mounting option, in particular, madeVOspace very promising for the collaboration, as it allowsfor the replacement of three distributed storage copies withjust one master copy, which can be mounted from anywhere.In addition, it includes a smart indexing algorithm whichensures that only the data needed for an analysis or plot istransferred. Although this system was a significant improve-ment, it did not work for remotely executed analysis projectsrequiring high data throughput of most of the available dataset, and it did not solve the problem that many in the collab-oration felt restricted by complications in establishing andmaintaining the NuGrid software stack. This software stackis not particularly complex, from the viewpoint of a computa-tionally experienced user, but in order to address the diverseset of science challenges, the collaboration includes mem-bers that deploy a diverse set of science methodologies, andespecially entry-level researchers and researchers-in-traininghave often found establishing and maintaining the NuGridsoftware stack to be a substantial barrier.

We tried to overcome this challenge by using virtual ma-chines based on Oracle’s VirtualBox technology. We have,for example, developed VMs for the Nova project3, whichis admittedly now mostly defunct. While in principle VMsallow one to load a pre-defined software stack into a VM aswell as the VOspace access tools, in practice, this technol-ogy has not been adopted broadly. The reasons for that werea combination of rather time-consuming and ridged mainte-nance requirements and reports of usability issues. In addi-tion to their heavy weight nature, the VMs also did not prop-erly address the need for distributed teams to collaborate onthe same project space, because they limit any VM instanceto only one researcher.

From the beginning in 2007, the NuGrid collaboration hadadopted Python as the common analytic language. Around2012/2013 the ipython notebook technology became increas-ingly popular in the collaboration. In collaboration withCANFAR, we developed as one of two applications of theproject Software-as-a-Service for Big Data Analytics fundedby Canarie4 the Web-Exploration for Nugrid Data Interac-tive (WENDI Jones et al. 2014) service, which had someof the functionality that is now offered by JupyterHub.This project was very successful in establishing a proto-type for web-enabled analytic remote data access with a pre-

3 http://www.nugridstars.org/data-and-software/

virtual-box-releases/copy2_of_readme4 http://www.canarie.ca

defined, stable analytic software stack and network proxim-ity to the data for a fast and interactive, remote data explo-ration experience. For some years NuGrid served graphi-cal user-interfaces built on ipywidgets of the SYGMA andOMEGA tools of the NuGrid Python Chemical Evolution En-vironment (NuPyCEE, Ritter & Cote 2016) as well as theNuGridSetExplorer which allows GUI access to the Nu-Grid stellar evolution and yield data sets (Pignatari et al.2016; Ritter et al. 2017b). The project was to a large extentfocused on improvements to the storage backend, enablingfor example VOspace to work well with indexed hdf5 files,and thus did not add authentication and access managementto the service. The usage was therefore limited to anonymousand time-limited access. The service was deployed on virtualmachines of the Compute Canada5 cloud service6.

Another set of requirements for the cyberhubs facilitypresented here originates from a stream of efforts in pro-viding enhanced data access within a collaboration, and ul-timately to external users, undertaken by the PPMstar col-laboration (Woodward et al. 2015; Herwig et al. 2014). Thetypical size of the aggregate data volume for a single projectinvolving three-dimensional stellar hydrodynamics simula-tions is ≈ 200 TB, and the collaboration would at any giventime work simultaneously on two to three projects. The sim-ulations are performed at super-computing centers, such asthe NSF’s Blue Waters computer at the NCSA in Illinois,or high-performance computing facilities in Canada, such asWestGrid’s orcinus cluster at UBC or the cedar cluster atSFU. Blue Waters has relatively restrictive access require-ments which make it somewhat burdensome to add interna-tional collaborators to access the data, especially temporaryaccess for students. For interactive access or processing ac-cess by collaboration members who cannot login to Blue Wa-ters, data has to be moved off the machine, which is not prac-tical since such external collaboration members do not havethe required storage facilities, and the network bandwidth isinsufficient. In addition, over decades the LCSE has devel-oped custom and highly optimized software tools to visual-ize and analyse the algorithmically compressed data outputsof the PPMstar codes. More recently, new tools have beendeveloped using Python. The data exploration and analysisecosystem of the collaboration is heterogeneous and difficultto maintain even for core members in view of the constantlychanging computing environments on the big clusters and thehome institutions. The challenge for this collaboration was tostabilize and ease the use of legacy software, the very largedata volumes and the access and authentication when trying

5 https://www.computecanada.ca6 https://www.computecanada.ca/research-portal/

national-services/compute-canada-cloud/

4

to broaden the group of users that can have analytic and ex-ploratory access to these large and valuable data sets.

Based on these requirements, and through experience, wehave combined the latest technologies, including Docker andJupyter and designed cyberhubs, a system that allowseasy deployment of a customized virtual research environ-ment (VRE). It offers flexible user access management, andprovides mechanisms to combine the research area specificsoftware applications and analytic tools with data and pro-cessing to serve the needs of a medium-sized collaborationor user group. At a larger scale, an architecture similar tocyberhubs has been deployed, for example, by the NOAOin their NOAO data lab, and has been selected for the LSSTScience Platform7. JupyterHub-based systems are also usedin teaching large data science classes, such as the UC Berke-ley Foundations of Data Science course8. At the Universityof Victoria a precursor of the architecture described here hasbeen used in both graduate and undergraduate classes for thepast three years. Another broad installation of this type isthe Syzygy project9 that allows institutional single-sign-onto Jupyter Notebook servers across many Canadian universi-ties to access Compute Canada resources.cyberhubs, although scaleable in the future, is at this

point addressing the needs of medium-sized collaborationswhich require an easy to setup and maintain shared researchenvironment. The cyberhubs software stack is availableon GitHub10 and the docker images are available on DockerHub11. In §2 we describe the system architecture and imple-mentation, in §3 we briefly sketch the typical steps involvedin deploying cyberhubs, and in §4 we present the two maindeployed applications and how to add new applications. Weclose the paper with some discussion of limitations and fu-ture developments in §5.

2. SYSTEM ARCHITECTURE ANDIMPLEMENTATION

In this section we describe the architecture and design fea-tures as well as the implementation of cyberhubs. Here, acyberhub administrator configures and deploys the service,while a cyberhub user is simply someone who connects tothe deployed service, and is not burdened with the details de-scribed here.

7 https://docushare.lsst.org/docushare/dsweb/Get/LSE-3198 https://data.berkeley.edu/education/foundations, http:

//data8.org9 https://syzygy.ca10 The multiuser and corehub single user is in https://github.

com/cyberlaboratories/cyberhubs, the application hubs that arebuilt on top of corehub is in the repository https://github.com/cyberlaboratories/astrohubs

11 https://hub.docker.com/u/cyberhubs

2.1. General design features

To satisfy the requirements of the cyberhubs system, wecombine the following components:

1. A thin user interface component allows the users toeasily interact with their virtual research environment(VRE).

2. The authentication component ensures the right re-searchers are accessing the proper data, processing andsoftware analytic tools in their VRE.

3. The docker spawner component that allows the usersto spawn their selected choice of application containersand allows selection of Jupyter environment (notebookor lab).

4. The image repository offers prebuilt container imagesfor all components and can be expanded to host newVREs for other applications.

5. The notebook templates and interactive environmentshelp the users to get started in their VREs and prepar-ing their own analytic workflow.

6. A deployment component enables administrators, suchas the data infrastructure experts in a collaboration, todeploy all of the above components with minimal ef-fort and customization.

The first three components are offered by JupyterHub,with some modifications we made to the second one to al-low dynamic white and black listing for user management.The fourth is partially enabled by the Cyberhubs Docker Hubrepository12. The fifth and sixth components, and the cus-tomization images are offered by cyberhubs.cyberhubs provides a complete, packaged system that can

easily be installed and customized and allows the cyberhubsadministrator to quickly deploy VREs. There is no reasonwhy a particular cyberhubs instance could not be near-permanent, with little in maintenance needed because thesystem is based on docker images. However, the cyberhubssystem is designed for easy deployment with all essentialconfiguration options, like attaching local or remote storagevolumes, provided through external configuration files thatthen will be absorbed by the pre-built docker images.

As opposed to other packaging solutions that rely on toolssuch as ansible, puppet and others, cyberhubs requiresonly setting a minimal number of environment variables todeploy any of the pre-built application hubs which could pos-sibly enable many astronomy use cases. Extending an appli-cation hub, or building a new one on top of corehub (part

12 https://hub.docker.com/u/cyberhubs

5

White and black list

Hub

SQlite Database

Image

Registry

Image Selector

Thin client

Data

Repository

Astrohub Dashboard

Configurable HTTP Proxy

Authenticator

Spawner

NoteBook

Repository

NoteBook

Repository

NoteBook

RepositorySingle Server

Authenticator

Selector Github

Container Monitor

Data

SelectorAdministrators

Notebook

Type selector

Figure 1. cyberhubs general system architecture.

of cyberhubs repository) is straight forward and well doc-umented. An example if the application SuperAstroHub(available in the Cyberhubs Docker repository) available onthe WENDI server (see §4.1.1) that combines all of ourpresently available cyberhub applications.

The general system architecture for the cyberhubs isshown in Fig. 1. JupyterHub is a multi-user environmentthat allows the users to authenticate and launch their ownnotebook and terminal server, while sharing access to cer-tain storage areas. By doing so, the same collaboration in-frastructure can be used by multiple users and therefore pro-vide a platform for sharing resources, analytic tools, as wellas research content and outcomes. The main components ofJupyterHub are:

• A configurable HTTP proxy that allows the users tointeract with the system and directs their requests tothe appropriate service.

• A hub that handles users and their notebooks. In moredetail, the hub offers the following services:

– An authentication service that supports many au-thentication backends (including PAM, LDAP,OAuth, etc.). Currently, cyberhubs uses an

extended Github authenticator that allows userswith GitHub account to log into cyberhubs.

– A spawner service that allows the user to selectthe singleuser application hub and the interfacetype.

– An SQLite database which keeps track of theusers and the state of the hub.

2.2. Spawner and authentication extensions

Our extended spawner is integrated in the JupyterHubconfiguration file. It provides at this point two selection op-tions. First the user selects between the default Jupyter Note-book option and the experimental option JupyterLab. BothJupyterLab and Notebooks offer Python, bash and other lan-guage notebook options as well as terminal access to the sin-gleuser hub container. JupyterLab is a significantly enhancedJupyter interface that overcomes the restrictions imposed bysingle linear notebooks or terminals, and allows one to com-bine multiple sessions in parallel in one web browser win-dow. For the second option, the user can select the applica-tion hub image if multiple options are configured to be of-fered by the cyberhubs administrator. In the future a simpleextension could allow one to choose between a variety of dataaccess options in this spawner menu.

A major challenge in a shared environment is access andauthentication administration. For most collaborations, na-tional or institutional authentication models are not practi-cal. We have adopted the third-party OAuth authenticationmethod available for JupyterHub and allow authenticationof users with their GitHub accounts. Other third-party OAuthapplications, such as Google, could be used as well.

To dynamically control the GitHub users who can accessthe system, our authentication extension provides a simplewhitelist and blacklist mechanism that can be updated in therunning, fully deployed cyberhubswithout service interrup-tion. The authenticator relies on a whitelist and/or black-list file to dynamically grant or deny access to GitHub users.When a whitelist is supplied, only the users in that list are al-lowed to login. When the blacklist is present, the users in theblacklist are blocked from accessing the system and will geta 403: forbidden page even if they are in the whitelist.

This is a rather simple yet powerful access model that al-lows, in combination with the easy configuration of the stor-age additions to the cyberhubs, a flexible access control todata and processing that can serve in a flexible way the accessand sharing requirements of medium-sized collaborations. Itcan be easily combined with temporary unrestricted accessto any user.

Unlike some JupyterHub installations that propagate thehost system identity to the hub user, or systems that create in-side the singleuser application container an identity accord-ing to the login identity, we are adopting a simpler approach

6

with the goal of enabling the most transparent and seamlesssharing and collaboration. In cyberhubs all users have intheir own application hub container the identity user. Typ-ically, a read-write data volume of considerable size is at-tached, and all users appear as users on that read-write vol-ume with the same identity.

Another major component in a shared environment is re-source allocation. By resources we are referring to cpu,memory, swap space and disk storage. Currently, we arenot enforcing any resource limits and we are not offering anyscheduling capabilities. We are considering to add the abil-ities to specify resource limits for every container launchedand to alert users if no more resources are available. Sincethese abilities do not ensure a scalable system, our roadmapincludes a plan to use Docker Swarm and/or Kubernetes forscaling the resources and scheduling the containers.

2.3. Features and capabilties

Both Jupyter Notebook and Jupyterlab offer web-basednotebook user interfaces for more than 50 programming lan-guages, and we included by default python 2 and 3 and bash.But other popular languages, such as R or Fortran are eas-ily added. Both bash notebooks as well as terminals pro-vide shell access to the singleuser application docker con-tainer (that we call here hub). Any simulation or processingsoftware that can be executed on the Linux command line,such as the MESA stellar evolution code (Paxton et al. 2010,2013, 2015) or the NuGrid simulations codes (Pignatari et al.2016), can be run by each user in their identical instance onthe full hardware available on the host. Other examples in-clude legacy analysis and processing tools that require a spe-cial software stack. If such software is once expressed inthe singleuser application hub docker image it can be easilyshared with anyone accessing the cyberhubs.cyberhubs are typically configured with an openly

shared, trusted user space that is equally available to allusers that have access to a particular cyberhub. This userspace is mounted on a separate, persistent volume. A trustedcollaboration, would establish some common sense rules onhow to access this shared space, in which all participantshave seamless access to any project or individual directories.We have found that this arrangement allows for very effectivecooperation between team members with very different skillsets, including students.

A typically small amount of private non-persistent storageis available inside the user’s singleuser hub instance, whichwill disappear when the user container is restarted. In orderto create some level of persistence beyond the shared userspace, users of cyberhubs rely heavily on external, remoterepositories, such as git repositories, for storing and sharingnon-data resources, such as software, tools, workflows, doc-umentation, and paper writing manuscripts.

In addition to analytic access to data the cyberhubs pro-vide documentation and report writing through inline Mark-down cells, a latex typesetting environment where pdf filesare seamlessly viewed in the browser for paper manuscriptwriting and editing, as well as slide presentation exten-sions to Jupyter which allow one to create presentationswith live plots and animations. In addition, graphical user-interface applications can be built, and examples are providedin wendihub (§4.1.1).

In terms of the maintenance of the multiuser environment,where many containers are running, the cyberhubs admin-istrator should monitor the state of the cyberhub, includingthe available resources. Astray and blacklisted user contain-ers should be removed.

A typical cyberhub includes a repository of examples andtemplate notebooks to help users getting started in exploringthe data resource. These notebooks can be copied into theimage or provided via mounted volumes. Each cyberhubin our cyberhubs family has strict version-specific require-ments files for python and Linux packages, that ensure thatthe same versions of each component of the entire softwarestack are always used, until an update is made. In that case,past docker image versions will be still available as taggedimages on the docker hub repository. Each user has there-fore a completely controlled and reproducible environment.It is then straight-forward to create a stable, shareable and re-producible science workflow. Simulation software and anal-ysis packages are shared via repository platforms such asGitHub, GitLab or BitBucket, and include information onexactly which cyberhub including the version, it is to bedeployed.

At the core of the cyberhubs13 design is the multiuserimage and a basic corehub singleuser application. The lat-ter is a skeleton and has no application software installed.The main elements of these core elements of cyberhubs areshown in Fig. 2. They are:

1. multiuser image: It is composed of our customizedJupyterHub image, available via dockerhub repos-itory, or via build package that includes Dockerfileand all other necessary scripts and docker-related filesto customize, compose and then launch the multiuserJupyterHub service.

2. singleuser corehub image: The most basic, bare sin-gleuser docker image, also available via dockerhubrepository or build package.

corehub is the starting point on top of which all of the otherapplication hubs are built (§3), as shown in Fig. 3. Obvi-

13 These are available in the GitHub repository https://github.com/cyberlaboratories/cyberhubs.

7

Jupyter

Python

Ubuntu

Jupyterhub

Python

Ubuntu

Cyberhubsingleuser application hub multiuser

SSL Keys

Authentication Environment Variables

Data Storage Environment Variables

Required Inputs

MultiuserDocker

SingleuserDocker

SkeletonDockerfile

Figure 2. Main elements of the core cyberhubs system. The multiuser takes care of receiving the initial service request from a user,handles user authentication and and data storage attachment. It launches a singleuser application hub from the appropriate image or reconnectsa returning user to that user’s existing application container. The singleuser component contains the application-specific analytic software andis the processing home for the user. Each user has a separate container instance of the application image.

ously, not only astronomy cyberhubs can be built on topof corehub, but applications from other disciplines or usecase are possible. The astronomy application hubs are theastrohubs14.

2.4. Storage staging

Docker containers typically do not have much storage, anda few options of storage staging are typically deployed:

• Read-only data volume: Most cyberhubs are aboutproviding access to a particular data universe. This isimmutable data that is staged on read-only data vol-umes. It is mounted on the singleuser container to al-low the users to read the data in their notebooks andprocessing as needed.

• Persistent data volume: This volume is also mountedand all users have the ability to write to and read fromit. The volume lives on the host or externally attachedstorage, and is protected against singleuser containershutdowns.

• Local ephemeral storage: This is the local storage allo-cated to each container when created. It is available tothe users only when the container is running and getspurged once the container is removed. This holds acopy of example notebooks added to the image to beavailable to all users. This it the home directory of

14 They are available from the GitHub repository https://github.com/cyberlaboratories/astrohubs. As explained in the documen-tation provided with these repositories the docker images are staged athttps://hub.docker.com/u/cyberhubs.

each user. Since this area is inaccessible to other usersof the cyberhub it is the right place to store, for ex-ample, a .gitconfig file or other configuration filesas well as a .ssh directory.

• Individual remote data storage: Users can use sshfs,mountvos, google-drive-ocamlfuse, and other fusetools to mount their remote data storage. This, how-ever, requires elevated privileges for the containers andis not yet supported.

Currently, read-only and persistent data volumes are spec-ified via environment variables, and we do not support in-dividual remote data storage yet. However, the cyberhubsadministrator can now configure both read-only and persis-tent data volumes as remote data volumes, an option that al-ready would serve the needs of many collaborations. We areconsidering to add a data selector that allows users to selectvolumes to mount that they have privilege for.

3. SYSTEM DEPLOYMENTS

The goal of cyberhubs is to make deployment aseasy as possible, and there are two deployment options.cyberhubs is based on dockers whose images are cre-ated and stored in a repository either locally, or on theDocker Hub repository. A Docker container is an in-stance of a Docker image. A cyberhub deployment alwaysconsists of two Docker containers (Fig. 2). One is an in-stance of the cyberhubs/multiuser image, the other isone container per user of one application hub image, suchas cyberhubs/corehub. A particular configuration maychoose to offer users more than one application image in the

8

spawner menu dialogue after login, such as those offered inthe cyberhubs family.

For ease of use, the first deployment option is recom-mended. It involves launching containers as instances fromthe images available in the cyberhubs organization athttps://hub.docker.com/u/cyberhubs which allowsdeployment of a cyberhub without building any docker im-ages. The only requirement is one must prepare the hostmachine and the cyberhub configuration file and launch.

The second option is for the administrator to build one orseveral of the required Docker images using the Dockerfileand configuration files provided. This involves building thesingleuser application hub on top of one of the providedapplication hub images. This option allows the administra-tor to add features and software otherwise not available bydefault, and to provide specialized configurations.

For specializations that require fundamentally differentdata access or different user authentication models, it may benecessary to rebuild the multiuser image. In any case the

Corehub

Machine Learning Hub

Wendihub

PPMstarhub

Customhub1Customhub2

Figure 3. cyberhubs administrators can build application im-ages by chaining existing application images and adding customiza-tion through dockerfile-based builds. By starting from the pre-builtPPMstarhub that is itself based on Codehub one can build in se-quential steps Wendihub and Machine Learning Hub each on topof the previous. At this point the capabilities and analytic tools ofthree application hubs are combined, and can be the bases for an-other addition of tools and data stores to create Customhub2. Sim-ilarly, for example, Customhub1 can be built as a combination ofCorehub, Wendihub and PPMstarhub.

administrator follows the step-by-step documentation in thecyberhubs repository15 on GitHub. A deployment wouldinvolve the following steps:

• Preparing the host machine:

15 https://github.com/cyberlaboratories/cyberhubs

– Select a host machine, such as a Linux worksta-tion. In our case cyberhubs are deployed in acloud environment, such as the Compute CanadaCloud, and require launching a suitable virtualmachine, attaching external storage volumes, andassigning IPs. We use a CentOS7 image, butother Linux variants should work as well.

– The host machine only needs a few addi-tional packages, most important of which isthe docker-ce package. A small amount ofdocker configuration is followed by launchingthe docker service on the host machine.

– Any external data volumes to be made availableneed to be mounted. sshfs mounted volumeswork well.

• Configuring the cyberhubs. This invariably startswith pulling the cyberhubs Github repo16. The mainsteps that always must be done are:

– Register an OAuth authentication applicationwith a GitHub account and enter the callback ad-dress, authentication ID and secret into the singleconfiguration file jupyter-config-script.sh.

– Specify the admin user IDs as well as white-listedusers. If white-listed users are specified, either inthe configuration file (static white listing) or inthe access/wlist file (dynamic white listing),then only white listed users are allowed. Other-wise everybody with a github account can haveaccess. Dynamic black listing is also possibleonce the service runs.

– Specify or modify data storage mapping from thehost to the cyberhub as the user sees it, for bothread-write and read-only storage.

– Specify the application hub name. Either a singleapplication hub is offered or the spawner menucan offer a number of different application hubs.

– Finally, create SSL key/certificates as describedin multiuser/SSL/README. While a commer-cial certificate can certainly be used, our defaultinstallation includes letsencrypt which pro-vides free three-month certificates. Such cer-tificates can be created and updated using thedocker blacklabelops/letsencrypt. How-ever, the present instructions recommend the useof certbot-auto17 which works well for ourreference host system CentOS.

16 https://github.com/cyberlaboratories/cyberhubs17 https://dl.eff.org/certbot-auto

9

– Source the config file and launch the cyberhubaccording to the instructions in multiuser/README.

• The above assumes a deployment from the pre-builtdocker images. This is the recommended mode. Bydefault, the multiuser image will be automaticallypulled from the repository during the execution of thedocker-compose up command. However, the sin-gleuser application hub will have to be pulled manuallyusing the docker pull command. In addition to thebasic corehub application there are several pre-builtapplications available in the docker hub repository18,such as WENDI for NuGrid data analysis, mesahubfor MESA stellar evolution, for machine learning hubmlhub to be used, for example, by StarNet (Fabbroet al. 2017), PPMstarhub for PPMstar stellar hydrodata analysis.

• To add more application functionality, packages, tools,etc. it is necessary to rebuild the singleuser applicationhub. The Docker and configuration files of all of theexisting application hubs available on Docker Hub arein the astrohubs GitHub repository19. All of theseapplication hubs start with the cyberhubs/corehubimage, as indicated in the first line of the Dockerfile:FROM cyberhubs/corehub. Building or extendingan application hub could start from one of the exist-ing hubs, or from corehub. Within the cyberhubsfamily, all application hubs can be combined with allothers. This is shown schematically in Fig. 3. In prin-ciple a super-application hub can be built by daisy-chaining all other application hubs together, collect-ing and adding in the process the capabilities fromeach participating application hub. If the commu-nity builds application hubs that are consistent withthe cyberhubs model, they could be added to thecyberhubs docker hub repository.

• The multiuser image and all application hub imagescan also be built from scratch using the dockerfilesand configuration files provided in the GitHub repos-itories, starting with Ubuntu images. This allows fullcustomization of all components, or improvements ofthe cyberhubs facility. The complete rebuild of thedocker images would also be required when an updateof the software stack is needed.

4. APPLICATIONS

In this section we describe the application hubs we havebuilt and deployed. Due to our research areas, these en-

18 https://hub.docker.com/u/cyberhubs19 https://github.com/cyberlaboratories/astrohubs

able simulation-based data exploration. Applications forobservationally-oriented data exploration and processingtasks are equally possible and supported.

4.1. NuGrid

As described in §1, the challenges of the NuGrid collabora-tion have contributed a significant portion to the requirementsof cyberhubs. NuGrid develops and maintains a numberof simulation codes (NuPPN) and utilities that allow users toperform nucleosynthesis production simulations and nuclearphysics sensitivity studies. These codes, the MESA code (Pax-ton et al. 2010, 2013, 2015) and the GENEC stellar evolu-tion code (Eggenberger et al. 2008) have been used to cre-ate the NuGrid data sets (Pignatari et al. 2016; Ritter et al.2017b). The NuGrid data consists of stellar evolution tracksfor twelve initial masses and five metallicities. Each trackis made up of between 30,000 and 100,000 time steps. Foreach time step profile information including at least density,temperature, radius, mass coordinate, mixing coefficient, anda small number of isotope abundances is saved. Each profilehas between 1,000 and 5,000 radial zones, and for all pro-files all zones are written out. The data set also contains post-processed data which reports profiles for about 1000 isotopesevery 20 time steps.

The time-dependent nature of stellar evolution simulationoutput suggests for this data a particular structure. For eachsaved time step, or cycle, a number of scalar quantities haveto be saved, as well as a number of profile vectors. A num-ber of such cycles are combined into one data file, or packet.The scalar quantities are the cycle attributes, the profile vec-tors are the data columns, and each packet has a number ofheader attributes which are repeated in each file and pro-vide global information for the run, such as initial conditions,code version used, units of quantities in the data columns orcycle attributes, etc. This is the SE (Stellar evolution) dataformat, shown schematically in Fig. 4 and, within NuGrid, iscurrently implemented using the hdf5 data format.SE output from MESA is written with NuGrid’s mesa h5

and then used for the post-processing simulations using Nu-Grid’s NuPPN codes, which in turn write output again in theSE format. Libraries and modules to write SE from Fortran,C or Python are available20. SE data output can be exploredvia data access, standard plots, visualisations, and standardanalysis procedures, using the NuGridPy21 Python package.

We want to accommodate three types of user:

• Internal users, including for example students, who areless experienced in the analysis of the NuGrid simula-tion data.

20 https://github.com/NuGrid/NuSE21 https://nugrid.github.io/NuGridPy

10

HEADER ATTRIBUTES(attributes of the file; these are global properties of the model

and are fixed for all files in that model, e.g. the model name)

cycle0000020(group)

CYCLE ATTRIBUTES(group attributes; properties of the model at that cycle, e.g. age)

SE_DATASET(dataset; typically between 5 and 10 columnsand the number of rows is a group attribute)

Lagrangian coordinate of

the zone

Eulerian coordinate

of zonezone density zone

temperature

zone diffusion

coefficient

array containing about 1000

isotopic abundances

isomeric stateA Z(datasets; one-column tables containing information about the contents of the

iso_massf columns in the SE_DATASET contained in each cycle group)

M25.0Z2.0e-02.0000001.out.h5(HDF5 file; initial mass 25 M_sun, initial metallicity Z = 0.02)

cycle0000040(group)

⋮

dataset

file

group

attributes

⋮cycle0001000

(group)

Wednesday, 25 June 14

Figure 4. Schematic of SE data format for one-dimensional time-dependent stellar evolution and explosion data.

• Users external to the collaboration to whom we wouldlike to provide the option to explore the NuGrid dataset to find answers to their specific research questions.

• Expert users who want to carry out NuGrid and/orMESA simulations and analyse the simulation outputconveniently in the same location, and possibly sharerun directories and workflows. These users can share acommon development platform that is exactly identicalto each participant who remotely accesses the platformand all of its content.

The first two types of users are served by WENDI, while thethird requires additional compilers and libraries which aredelivered in mesahub.

4.1.1. WENDI

Web-Exploration for NuGrid Data Interactive WENDI pro-vides

• python and bash notebooks, terminal and text editor;

• example notebooks; and

• self-guided graphical user interface (GUI) notebooks(widgetized notebooks).

The WENDI widget notebook NuGridStarExplorer pro-vides GUI access to plotting and exploring the NuGrid datasets, specifically the library of stellar evolution and detailednucleosynthesis simulations. As an example, the evolutionof a low-metallicity, intermediate mass star is shown dur-ing the Asymptotic Giant Branch evolution. The Kippen-hahn diagram shows the Lagrangian coordinates of recur-ring He-shell flashes, each of which drives a pulse-drivenconvective zone. In the pulse-driven convective zone the22Ne(α, n)16O reaction creates neutrons with neutron den-sities reaching Nn ≈ 1012cm−3. At these neutron densitiess-process branchings, such as at 95Zr, are activated and theneutron-heavy 96Zr is produced. Fig. 6 shows the isotopicabundance distribution of s-process elements. Note, how themass fraction of 96Zr exceeds that of 94Zr in the shown modelat the end of the pulse-driven convective zone. In the 13Cpocket where the bulk of the s-process exposure takes place94Zr/96Zr ≈ 600 because the neutron denisty is low and thebranching is closed. In the solar system 94Zr/96Zr = 6.2.This demonstrates the different neutron density regimes ofthe s process. Although this example is for low metal con-tent, conditions at solar-like Z are similar. The solar systemabundance distribution originates from a mix of low- and in-termediate mass stars, involving both the 13C and the 22Neneutron source, which each produce isotopic ratios in differ-ent proportions (Gallino et al. 1998; Herwig 2005, 2013).

While the widget notebooks provide easy and powerful ac-cess to the data, customized analysis may require the addedcontrol of programming access to the platform. In order tomake it easy for users to get started analyzing the NuGriddata we keep adding to a collection22 of short example anal-ysis tasks, such as abundance profiles at collapse, or C13-pocket analysis. All data and software dependencies of theseexamples are satisfied on wendihub. Although users caneasily clone their own copy of any external repository, thewendi-examples are preloaded in WENDI, and are a con-venient starting point for further analysis. Users who createan interesting new example are encouraged to fork the exam-ple repository on the terminal command line, add their newexample and make a pull request to the original repository.Although executing notebooks is just a matter of clicking theplay button, any further interaction for this type of notebook-based analysis requires basic knowledge in Python.

Two additional widget notebooks23 are presently availablein WENDI. The OMEGA self-guided interface provides exam-ple applications of the NuGrid Python Chemical EvolutionEnvironment (NuPyCEE, Ritter & Cote 2016) code One-zone Model for the Evolution of Galaxies (OMEGA), such

22 https://github.com/NuGrid/wendi-examples23 https://github.com/NuGrid/WENDI

11

Figure 5. Self-guided exploration of the NuGrid stellar evolutionand yield data base via the graphical user interface NuGridStarEx-plorer in WENDI. Kippenhahn diagram of a MZAMS = 3M� stellarmodel with Z = 0.001, zoomed in around the core-envelope inter-face where the He- (dashed, orange line) and H-burning (solid, blueline) shells are located. Grey and blue areas mark convectively un-stable regions and regions of energy generation. The isotopic abun-dance distribution in the thermal pulse ending around model 16880is shown in Fig. 6.

Figure 6. Mass-averaged isotopic abundance distribution of thepulse-driven convective zone for model 16880 of the MZAMS = 3M�

stellar model with Z = 0.001 shown in Fig. 5. The abundance dis-tribution shows the stable isotopes for the first- and second-peaks-process elements (see text for more details).

as models for dwarf galaxies Fornax, Carina and Sculptor.For OMEGA WENDI allows arbitrary, complex program-ming of analysis through ipython notebooks as well. For ex-ample, the top panel of Fig. 7 shows that NuGrid massive staryields overproduce some iron-peak elements like Cr and Ni,

12

but produce a consistent amount of other elements like Ti, V,Cu, and Ga. Further analysis of the yield source as shownin the IMF-weighted yields (bottom panel) reveals that theoverestimation of Cr in the galaxy evolution models origi-nates from the 20M� model at Z = 0.01. This analysis toolallows to identify sources of discrepancies between numeri-cal predictions and observations. In this case, further anal-ysis of the underlying stellar evolution model using similarapproaches as those demonstrated below for intermediate-mass stars, that will be presented elsewhere in more detail,connects this overproduction of some iron-group elements tothe convective merger of an O-Si shell in the stellar evolu-tion model that may not be realistic. This analysis can beperformed by anyone on WENDI, and is available there in anotebook as part of the pre-loaded WENDI examples24.

Another widgetized notebook provides an interface forStellar Yields for Galactic Modelling Applications (SYGMA,Ritter et al. 2017a), which allows to generate simple stel-lar population models and retrieve chemical yields amongother properties in table formats that can be used as build-ing blocks for GCE models or hydrodynamic simulations ofgalaxy evolution. The SYGMA section of WENDI also con-tains a folder with notebooks25 that run the SYGMA codeand generate all plots shown in the SYGMA code paper Rit-ter et al. (2017a). This serves as an example how cyberhubscan be a tool in support of the goal of reproducible science,by providing not only access to data and code, but also to thecapability to execute the analysis on the data in a controlledand specified environment.

A technical detail concerns how the widget notebooks arelaunched. The cyberhubs configuration allows automaticself-start up that hides Python code cells, creating a rela-tively polished final experience. For this to work, the wid-get notebooks have to be designated as trusted. Jupyter ref-erences a database in a notebook signatures database filewhich contains instances of the notebooks that are config-ured to be trusted. In order for a notebook to be trusted, theraw .ipynb notebook file must exactly match the file used tosign said notebook in the database. This system implementedby JupyterHub is rather sensitive to minute changes in thenotebook, and does not easily incorporate notebook updatesand preserve the trusted state of the previously signed note-book. In order to ensure flexibility of the application hubswith trusted notebooks in their respective singleuser images,even after a notebook has been updated, a bash script is usedto trust the notebooks in the state that they exist when staged.

24 https://github.com/NuGrid/wendi-examples/blob/master/

Stellar\evolution\and\nucleosynthesis\data/Examples/

Solar_abundance_distribution.ipynb25 https://github.com/NuGrid/NuPyCEE/tree/master/DOC/

Papers/SYGMA_paper

20 22 24 26 28 30 32 34Z (charge number)

10 10

10 8

10 6

10 4

10 2

X (m

ass f

ract

ion)

Ti

V

Cr

Mn

Fe

Co

Ni

Cu

Zn

Ga

SolarAll sourcesMassiveSNe IaAGB

5 10 15 20 25Stellar initial mass [M ]

10 3

10 2

10 1

Cr IM

F-we

ight

ed y

ield

s [M

] Z=0.02Z=0.01Z=0.06

Figure 7. Top: Elemental abundance distribution of the Galacticgas, when the Sun formed, predicted by the chemical evolution codeOMEGA using NuGrid yields for low-mass and massive stars andThielemann et al. (1986) yields for Type-Ia supernovae (SNe Ia).The solar distribution is taken from Lodders et al. (2009). The blue(dashed with triangles), red (dashed with squares), and green (solidwith crosses) lines represent the individual contribution of massivestars (winds and core-collapse supernovae), SNe Ia, and low-massstars, respectively. The orange solid line shows the combined con-tribution of all sources. Bottom: Cr yields weighted by the initialmass function as a function of stellar mass for different metallici-ties (different colors) in a 10000M� simple stellar population, usingNuGrid yields and the SYGMA code. The contribution of SNe Ia isnot shown in this panel.

This ensures that even if a notebook has changed remotelyprior to building a singleuser application hub with a refer-ence to an outdated signatures database, the database will beupdated automatically. This script contains a series of pathsto the notebooks in the singleuser environment that the userrequires to be trusted. When its run, it signs the notebooksignatures database file in the singleuser notebook directory.

13

WENDI is provided by the wendihub docker image thatcan be found on docker hub (cyberhubs/wendihub) as wellas in the astrohubs GitHub repository26.

4.1.2. NuGrid / MESA experts

The third of the above cases adds the requirement to in-stall and run multi-core, parallel simulations. The NuGridcollaboration uses the MESA code that uses OpenMP to pro-vide shared memory parallelism scaling to up to ≈ 10 cores.Like many sophisticated simulation codes MESA relies on asignificant number of dependencies. The installation pro-cesses has been significantly eased with the MESA-SDK. Still,installation can be a challenge, especially for inexperiencedusers. The NuGrid code NuPPN for single-, multi-zone andtracer particle processing adopts MPI parallelism and exhibitsgood strong scaling to ≈ 50 cores for typical 1D multi-zone problems. Both applications are compiled with thegfortran compiler, and require hdf and se libraries, as wellas numerical libraries, such as SuperLU and OpenBLAS. Thesingleuser application mesahub combines the compilers, li-braries and environment variable settings needed to installand run MESA and NuGrid simulation codes, and proba-bly several other codes with the same requirements. Thepresently latest mesahub docker image (version 0.9.5) runsthe NuGrid codes NuPPN/mppnp for parallel multi-zone sim-ulations, and NuPPN/ppn for single-zone simulations, andhas been tested for MESA versions 8118, 8845, and 9331.It should be straight-forward to update this application to ac-commodate newer as well as older MESA versions. The re-sources of the entire host machine can be accessed by eachuser through their application docker container. We are cur-rently running cyberhubs on several servers, including oneinstance on a virtual workstation with 16 cores and 120GBmemory which allows several concurrent MESA runs as wellas using all 16 cores for NuPPN multi-zone simulations.

In addition, this application includes a wide range ofPython packages for data analysis, including NuGrid’sNuGridPy tool box. The application also includes com-mon command-line editors and a complete LATEX installationthat allows manuscript generation, complemented with thebrowser’s pdf viewer. Jupyter extensions that allow easygeneration of interactive slide shows from notebooks for pre-sentations are included. It is therefore possible to perform allsteps needed for a research project just inside the mesahubapplication.

4.2. PPMstar

Another application that we want to highlight is thePPMstarhub which provides analytic access to stellar hy-drodynamics simulations (Herwig et al. 2014; Woodward

26 https://github.com/cyberlaboratories/astrohubs

et al. 2015; Jones et al. 2017). The challenges in this case area combination of very large data sizes and the benefits fromusing legacy software in a shared environment. This sectionstarts with a historical perspective that provides context forthe current development described in this paper.

4.2.1. Data representation strategies for 3D hydrodynamicssimulations with PPMstar

Over many years, the team at the University of Min-nesota’s Laboratory for Computational Science & Engineer-ing (LCSE) has developed a series of tools to deal with thevoluminous data that is generated by collections of large 3-Dfluid dynamics simulations. The LCSE was formed in 1995,but the activity began in 1985 as a result of the University ofMinnesota’s purchase that year of the largest and most pow-erful supercomputer then available, the Cray-2. Only 3 ofthese machines existed in the world at that time. This pur-chase, coupled with the rarity at the time of academic re-searchers with simulation codes capable of exploiting thismachine, produced an unprecedented opportunity. The uni-versity had purchased the machine, but not a data storage sys-tem. An early way around that problem for our research teamwas to take advantage of a holiday sale of disk drives by Con-trol Data, and later the purchase of a tape drive and a smallcomputer to drive it. With this experience, a long tradition ofever greater data compression and development of data anal-ysis and visualization tools began. At the time, there were nodata file format standards, nor were there tools, aside fromprograms one could write oneself, to read such files. Theresult of this combination of circumstances was that, in theLCSE, we developed our own very powerful tools and tech-niques to analyze and visualize 3-D simulation data. As thefield has grown, other groups have taken it upon themselvesto produce, enhance, and maintain such tools for communityuse, which is a full-time activity that we chose not to engagein. The infrastructure described in this article has provided aframework in which we can embed our LCSE tools, givingthem an interface that can be quickly understood and utilizedby others through a Web browser. Python is the glue that con-nects our utilities to the framework and to external users. Allthis can now make our simulation data available to a commu-nity of interested parties around the globe.

Simulations in three dimensions pose special challenges tothe understanding of the computational results. LCSE did notembark on 3-D simulation until we heard from a Pixar rep-resentative in the mid 1980’s about the invention of volumerendering. Once we saw on a workstation screen the rotat-ing image of the volume rendered water rat that constitutedthe first demo of the technique, it was obvious to us that thiswas the solution to the data exploration problem for our fluiddynamics domain. Soon thereafter, we had our first volumerendering program, written by David Porter, running on theCray-2. This new visualization technique (Porter & Wood-

14

ward 1989; Ofelt et al. 1989) prompted the conversion of our2-D hydrodynamics code to do 3-D simulations. Initially,we tried to save as much of our simulation data as we pos-sibly could, because 3-D fluid dynamics simulation was sonew that we thought that we could not possibly predict whatrepresentations of the data we might later wish to make. Wequickly discovered that we could compress saved data downfrom 64 to 16 bits per number, saving a factor of 4 in datavolume. Even so, the data volume was enormous. Decadesof experience with visualizing and analyzing 3-D simulationdata (Woodward 1992a,b, 1993; Tucker & Woodward 1993)have led us to the approach described below that is connectedto the Python-based cyberhubs framework described in thisarticle.

Our simulations fall into fairly simple categories, such ashomogeneous turbulence, stellar convection in slab or spheri-cal geometry, or detailed studies of multifluid interface insta-bility growth in slab or spherical geometry. In each category,we do many simulations that all have common features. Thishas meant that after the first few simulations are completed,we have a very clear idea of what data we want to preserveand what visualizations we want to make of the flows in anynew category. We have also developed highly robust nonlin-ear maps from the real line, or the positive real line, to theinterval from 0 to 255. Each such map is determined by aninitial functional transformation, such as a logarithm, for ex-ample, followed by the standard nonlinear map given by thevalues of just 2 constant parameters. Such mappings of sim-ulation data to the 256 color levels used in volume renderingcan work over all the runs in a single category of simulationswithout any modification. This is not only tremendously con-venient, but it also allows direct and meaningful visual com-parisons of data from different runs. It turns out that certaincolor and opacity maps can work well for a particular sim-ulation variable, such as vorticity magnitude, over an entirecategory of runs without any modification. These to somedegree unexpected findings have profound consequences fordata compression.

The robustness of nonlinear mappings from the real lineto color levels allows us to have our codes dump out only asingle byte per grid cell per variable field. This is an enor-mous savings in data volume. It can only be helpful if onecan know before doing the simulation which variables areuseful for visual exploration of the simulation results. Onecan save even more data volume if one knows which viewsof such variables one wants to preserve. Such images canthen be compressed further by standard image file formats.Making this data compression also requires that one knowthe color and opacity mapping one wishes to use. After a runis completed, these images can now be animated using stan-dard movie making software, such as mencoder, that havereplaced our own movie animation software. However, to

save only images and not the much more voluminous rawvoxel data, one must build the volume renderer into the sim-ulation code. We have done this using the srend softwarepackage (Wetherbee et al. 2015). In this way rendering withsrend happens right in the code rather than as a separateactivity as part of a complex workflow, which has many ben-efits. The full-resolution voxel data need never be writtento disk at all, although, for now, we still write this out justin case. This new capability replaces the previous workflowinvolving the LCSE HVR volume renderer that required slowand difficult data format conversion, GPUs, and special soft-ware libraries to run. srend is written in Fortran, is compiledalong with the simulation code, and it has no dependenciesupon software libraries. Using either the new srend capa-bility or the previous HVR volume renderer, our strategy is tocreate default image views of a pre-defined set of variablesfor all dumps. These image libraries are available throughthe cyberhub.

Even if we were to continue to save full-resolution voxeldata, these data sets are very clumsy to work with due totheir sheer size of 45GB per dump depending on how manyvariables are saved. At the same time, a flexible capabil-ity to make any visualization of any variable after the sim-ulation is highly desirable. We accomplish this by perform-ing an additional data compression. This final data compres-sion has evolved from our use of simulation data to validateand develop statistical models of turbulence (Woodward et al.2006).

Turbulence closure models deal in averages. They tendto be based on comparisons of averages of products and theproducts of the corresponding averages. To work with suchmodels in the early 2000s, we used averages of our simula-tion data taken over cubes 32 cells on a side. Our filter rep-resented the behavior of a quantity inside the filter cube by aquadratic form determined by the 10 lowest order momentsof the quantity. This filtering technique was derived fromour work with the PPB advection scheme (Woodward 1986;Woodward et al. 2015), which also works with the 10 lowestmoments. To be able to construct such filtered representa-tions using a moving filter volume, from our simulations wesaved averages of many different variables taken over cubes4 grid cells on a side. We saved these with 16-bit precision,after first passing them through our robust nonlinear maps.

Our present simulation codes all now work with fundamen-tal data structures consisting of briquettes of 4 grid cells on aside. The problem domain is subdivided into regions, whichare rectangular solids, and the regions are subdivided intogrid bricks, which are smaller rectangular solids. Each gridbrick is a brick of briquettes, augmented all around by a sin-gle layer of ghost briquettes from the 26 neighbor bricks.

For the storage required to save just one byte of data fromeach grid cell in our simulation at any dump time level, we

15

8 7 6 5 4 3 2 1 0

log10 FV

0.00 0.01 0.02 0.03 0.04 0.05

| × u | / s 1

50

00

0 k

m

Figure 8. Volume renderings of the volume mixing fraction (left) of hydrogen-rich material pulled into the pulse-driven convective zone in a2M�, low metallicity star and of the magnitude of vorticity (right) in this star. In the left column the back hemisphere of the star’s central regionis shown, while the vorticity images on the right render a thin slice through the 3D 4π simulation domain. The volume renderings at the top aremade using the full-resolution simulation data, while those at the bottom use the briquette-averaged data (see text for details).

can instead save, with 16-bit precision, 32 variable aver-ages for each grid briquette of 64 cells. 32 variables is somany that we can save several other quantities which requiredifferentiating the simulation data and are useful for modelbuilding, in addition to storing variables that we nearly al-ways look at. We include, for example, the magnitude of thevorticity, the divergence of the velocity, and both volume-weighted and mass-weighted averages of the velocity compo-

nents. The idea is that from the 32 quantities one could derivealmost anything one may need when analyzing the data.

Data cubes with the 32 variables representing each of the216 or 512 regions of a large simulation are saved directly bythe code as it runs in separate disk files. In this way the ana-lytic tools have immediate and targeted access to any desiredregion. In practice, it has been somewhat a surprise how use-ful the briquette-averaged data actually is. While a fine grid

16

is needed in order to advance the solution in time with highaccuracy, this same fine grid is not needed to represent the so-lution. Volume-rendering from full-resolution images of themixing fraction and the vorticity are compared with render-ings based on the briquette-averaged lower-resolution datasets in Fig. 8. The full-resolution data consists of a single-byte voxel value at each cell of the uniform Cartesian grid torepresent each variable we wish to volume render. For themixing fraction full-resolution means double resolution, i.e.in this case 30723 voxels, due to the sub-grid resolving powerof the higher-order PPB advection scheme (see appendix ofWoodward et al. 2015). The high-resolution rendering of thevorticity and other quantities is based on the grid-resolutiondata. The lower-resolution renderings of both mixing frac-tion and vorticity are based on data cubes with four timesfewer grid points along each axis. The 3843 data cubes con-tain averages of 4×4×4-cell briquettes, which for the mixingfraction represent 8 × 8 × 8 significant data values. The im-ages shown on the top row of Fig. 8 use the full-resolutiondata, and of course give the best representation. The mostimportant role of renderings and 3D visualisations is to al-low a qualitative assessment of the flow, that will ultimatelyguide quantitative model building. Even renderings based onthe 64-times less voluminous briquette-averaged data shownon the bottom row still expose most of the key features of theflow.

Fig. 8 shows the far hemisphere of the He-shell flash con-vection or pulse-driven convective zone (similar to thoseshown in Fig. 5 and Fig. 6, see §4.1) in a low-metallicity AGBstar. The inner core, including the He shell and a stable layerabove that contains H-rich, unprocessed envelope material,is included in the simulation volume. The gas is confined bygravity and also by a reflecting boundary sphere at a radiusof 33.5 Mm. The star is shown in the midst of a global os-cillation of shell hydrogen ingestion (GOSH, Herwig et al.2014). H-rich gas is entrained into the pulse-driven convec-tive zone from just above the top of the convection zone, ata radius of about 28 Mm. Waves of combustion involvingthis mixed-in H and the 12C are propagating around the outerportion of the pulse-driven convective zone. The convectionzone has been formed by the helium shell flash, with heliumburning located at the bottom of the convection zone, arounda radius of 13 Mm. The combustion of entrained gas at radiiaround 17 Mm drives strong local updrafts, which greatly en-hance convective boundary mixing as the combustion wavespropagate. This is of course best seen in a movie animation.

In Fig. 8, two counter-propagating wave fronts have re-cently collided in the region of the lower-left, and a clearlyvisible puff of entrained gas has been forced downward there,helping to form what will soon become a shell of H-enrichedgas floating near the top of the convection zone. In theimages at the left the gas of the star’s carbon-oxygen core

as well as the helium-and-carbon mixture of the convec-tion zone are rendered as transparent. The color map rep-resents only the H-rich gas component, that initially is con-fined to the stable layers above the convective boundary. Atthe point of the simulation shown, some amount of the H-rich fluid has been entrained into the pulse-driven convectivezone. From the assumed viewing perspective one sees thelowest H-concentrations first. Where these, rendered darkblue, are sufficiently low, one sees through them into themore highly enriched gas. At this stage, the lower surfaceof the newly formed shell of H-enriched gas is fairly easilydeformed as rising plumes of hotter, more buoyant gas tendto force it aside as they decelerate and ultimately reverse theirupward flow. Ridges of dark blue in these images show thelanes that separate neighboring upwellings. These ridges ofH-enriched gas are descending, pulling the gas from abovethe convection zone deeper into the hot region below, wherethe hydrogen will burn and drive new waves of combustion.All of these key features are clearly discernible in the low-resolution renderings shown in the bottom row of Fig. 8.

Volume renderings of the vorticity magnitude (right col-umn, Fig. 8) reveal a thin, highly turbulent region at the frontof the descending puff of entrained gas at the lower left. Atypical, unstable shear layer induced by boundary-layer sep-aration (Woodward et al. 2015) can be seen in the upper rightquadrant. These features have been identified as an importantcomponent of the 3D entrainment mechanism at boundariesthat are Kelvin-Helmholtz stable according to the 1D radialstratification. The images based on the briquette-averageddata gives a fuzzy, slightly out-of-focus impression. But theystill clearly reveal these key features of the flow, and, whensuch images are animated, their dynamics.

Another more quantitative example of how the briquette-averaged data can be used is shown in Fig. 9. Even at amoderately-sized grid of 7683 the down-sampled data pro-vides a good initial impression of the overall structure ofthe flow at a computational- and data-related cost that canbe easily accommodated in the analysis scenario of thePPMstarhub. For the overall speed profile the low- andhigh-resolution data representations can hardly be discerned.Even when zooming in to the upper convective boundary thelow resolution data provides meaningful exploratory infor-mation.

4.2.2. The PPMstar application hub

We have used the cyberhubs technology to build thePPMstarhub application which is addressing several issues.The PPMstar simulations of stellar convection are performedon the Blue Waters computing system at the NCSA center onas many as 400,000 cores. The resulting data sets are in theirown way unique, similar to astronomical surveys. Althoughour team is exploiting them for their primary scientific pur-

17

1020304050607080

|u| /

km s

1

4 5 6 7 8 9r / Mm

0

10

20

30

40

50

|u| /

km s

1

full-resolutionbriquette-averaged

7.6 7.7 7.8 7.9 8.0 8.1 8.2 8.3 8.4r / Mm

0

10

20

30

40

50

|u| /

km s

1

full-resolutionbriquette-averaged

Figure 9. A center-plane slice showing the convective speed fromthe 7683-grid O-shell convection simulation D1 presented by Joneset al. (2017), based on briquette-averaged 3D data (top panel), aswell as spherically averaged profiles based on full-resolution dataand on briquette-averaged data.

pose, there are potentially a number of additional questionsthat these data sets could answer. Sharing the raw data byjust making it available for download is impractical due tothe size of the data, as well as the specialized analytic toolsthat are needed to access and explore the data. cyberhubsallows us to expose to interested users these simulation datasets together with our specialized software stack for analysis.

Over the years, a full range of tools have been developedat the LCSE that exploit the briquette data sets, both for vi-sualization as well as for further analysis that would be in-volved, for example, in model building. As mentioned above,such models could be turbulence models, or mixing mod-els, to ultimately be deployed in 1D stellar evolution models.These tools are now available along with access to severalof our published data sets to interested users through PPM-starHub27.

In addition, we have developed additional new Python-based analytic tools that work with the briquette data as wellas with single and multiple radial profile data. These, alongwith collections of example notebooks are available on thePPMstar GitHub repository28. Specifically, the examples in-clude notebooks that contain the analysis of all plots shownin our recent study on stellar hydrodynamics of O-shell con-vection in massive stars (Jones et al. 2017), as well as thenotebooks of our project of simulations of low-Z AGB H-ingestion into a thermal-pulse He-shell flash (Woodward etal., in prep). All of these example notebooks are staged onour PPMstarHub server where the necessary data sets arestaged as well, so that interested users can follow our dataanalysis. This is an example of how cyberhubs can play anessential role in making scientific analysis of raw simulationor observational data more transparent and accessible, andthe process of data analysis reproducible.

4.3. How to add new applications

The previous sections have described two use cases thathave guided the requirements for cyberhubs. Adoptingcyberhubs for a different application would start either withthe corehub application, or with one of the already existingapplications. One would modify the requirements files thatspecify the Python and Linux software stack, and, followingthe examples provided, add any custom software and toolsrequired. For example, we have built a basic cyberhubs ap-plication image for machine learning (mlhub) and intend toevolve this into a StarNet application for users and develop-ers. StarNet is an application of deep neural networks forthe analysis of stellar spectra and especially abundance de-termination (Fabbro et al. 2017). If one builds a StarNet hubon top of WENDI hub users could perform combined anal-

27 https://hickory.lcse.umn.edu28 https://github.com/PPMstar

18

ysis of stellar abundance determination and interpretation ofthese abundances using, for example, the NuPyCEE tools.

We have also created targeted application for teaching spe-cific courses, such as the second-year computational physicsand math course29 at the University of Victoria, that we arecurrently teaching with 90 students on the mp248 applicationthat can be optionally launched on the server that also offersthe WENDI application.

5. CONCLUSIONS

Leveraging dockers, jupyterhub, jupyterlab and jupyternotebook, we designed, implemented and deployed thecyberhubs system. It provides collaborations and researchgroups with a common collaboration platform in which data,analytic tools, processing capacity as well as different levelsof user interactions (Python or bash notebooks, terminals,GUI/widget notebooks). cyberhubs adopts a simple, flexi-ble and effective access and authorization model.

The system is easy to deploy, to customize, and is alreadyin production. By pulling a docker image, cloning a GitHubrepository and specifying a few environment variables, ad-ministrators can launch VREs for their users. Existing coreor more advanced application hub images can be customizedto suit specialized needs. In addition to the basic corehub,our specialized hubs are in production and used for collabo-rations such as NuGrid and PPMstar, as well as in the classroom teaching classes with dozens of students.

5.1. Limitations and future development

As with any multiuser platform, the cyberhubs requiredesignated personnel to administrate and maintain, thoughthe deployment of our hubs is straight-forward. The fact thatwe are dealing with leading edge technologies makes thesystem susceptible to major changes at any time. We haveprotected cyberhubs to some degree by enforcing version-locking of each included Python and Linux software pack-age. Although we have frozen the pip and apt requirementsby specifying for each component the version to be used atbuild time to avoid package incompatibility issues, securityupdates of any package may require the users to update thesystem. Another limitation of the system is that it does notscale with the number of users, and does not offer resourceallocation or scheduling capabilities. As the number of usersincreases, a cyberhub may run out of resources and largerservers may become necessary. Therefore, we are explor-ing Docker Swarm and Kubernetes to scale the resources andschedule containers into distributed resources. Volume selec-tion is also an important feature that we plan to add. Depend-ing on the credentials of a user, we are interested in ensuring

29 https://github.com/cyberlaboratories/teachinghubs,https://hub.docker.com/r/cyberhubs/mp248

that the user has access to special/private volumes. We alsoplan to add letsencrypt capability to our hubs so that ad-ministrators are freed from dealing with SSLs directly.

We invite those who are creating new application hub im-ages to share these through adding the build files to theastrohhubs GitHub repository and to submit such imagesto be pushed to the cyberhubs Docker Hub organisationcyberhubs.

The cyberhubs project is building on a previous Canariefunded CANFAR project Software-as-a-service for Big DataAnalytics in which the first version of WENDI was builtwith pre-JupyterHub tools. Further funding was providedby NSERC USRA, NSERC Discovery, EcoCanada and theNational Science Foundation (NSF) under Grant No. PHY-1430152 (JINA Center for the Evolution of the Elements).Previous undergraduate students in the Coop program ofthe Department of Physics and Astronmy at the Universityof Victoria who have directly or indirectly contributed areWilliam Hillary and Daniel Monti, who developed the initialversions of the NuGridPy software. Luke Siemens has madesignificant contributions to an initial version of the new, andmore general version of WENDI based on JupyterHub. Thedata sets and software tools in NuGrid’s WENDI cyberhubwere developed by members of the NuGrid collaboration(http://www.nugridstars.org). The motivation for this pa-per described in the introduction was previously expressed,in part, in CANFAR’s CFI proposal ”Astronomy Cyber-laboratories Platform”, PI Falk Herwig, submitted in Octo-ber 2017. We also acknowledge support for our large simu-lations on the Blue Waters machine at NCSA with PPMstarfrom NSF PRAC awards 1515792 and 1713200, and supportfor work at Minnesota on these simulations and constructionof means to serve and share the data from NSF CDS&E grant1413548.

Software: Juypter notebook http://jupyter.

org, Jupyterlab https://github.com/jupyterlab,VOspace http://www.canfar.net/en/docs/storage,vos https://pypi.python.org/pypi/vos, VirtualBoxhttps://www.virtualbox.org, JupyterHub https://jupyterhub.readthedocs.io/en/latest/, ipywidgetshttps://ipywidgets.readthedocs.io, NuPyCEE http://nugrid.github.io/NuPyCEE, NuGridSetExplorer

https://github.com/NuGrid/WENDI, hdf5 https:

//www.hdfgroup.org, Cyberlaboratories cyberhubshttps://github.com/cyberlaboratories/cyberhubs,Cyberlaboratories astrohubs https://github.com/cyberlaboratories/astrohubs, Cyberhubs Docker

repository https://hub.docker.com/u/cyberhubs,Docker https://www.docker.com, NOAO data lab

http://datalab.noao.edu, ansible https://www.ansible.com, puppet https://puppet.com, mesa h5

19

https://github.com/NuGrid/mesa_h5, Python https://www.python.org, MESA http://mesa.sourceforge.net, WENDI http://wendi.nugridstars.org, OpenMPhttp://www.openmp.org, MESA-SDK http://www.astro.

wisc.edu/˜townsend/static.php?ref=mesasdk, MPIhttps://www.open-mpi.org, gfortran https://gcc.gnu.org/fortran, SuperLU http://crd-legacy.lbl.gov/˜xiaoye/SuperLU, OpenBLAS http://www.openblas.net, mencoder http://www.mplayerhq.hu

REFERENCES

Eggenberger, P., Meynet, G., Maeder, A., et al. 2008, Ap&SS, 316,43

Fabbro, S., Venn, K., O’Briain, T., et al. 2017, eprintarXiv:1709.09182, 1709.09182

Gallino, R., Arlandini, C., Busso, M., et al. 1998, ApJ, 497, 388Herwig, F. 2005, ARAA, 43, 435—. 2013, in link.springer.com (Dordrecht: Springer Netherlands),

397–445Herwig, F., Woodward, P. R., Lin, P.-H., Knox, M., & Fryer, C.

2014, ApJL, 792, L3Jones, S., Andrassy, R., Sandalski, S., et al. 2017, MNRAS, 465,

2991Jones, S., Herwig, F., Siemens, L., et al. 2014, poster contribution

at Nuclei in the Cosmos NIC XIII conference, available athttp://www.nugridstars.org/publications/

conference-contributions/

nuclei-in-the-cosmos-xiii/NIC_CADC_poster_best.

pdf

Lodders, K., Palme, H., & Gail, H. P. 2009, in Solar System(Berlin, Heidelberg: Springer Berlin Heidelberg), 712–770

Ofelt, D., Porter, D., Varghese, T., et al. 1989, PPM GraphicsTools, Tech. rep., Minnesota Supercomputer Institute

Paxton, B., Bildsten, L., Dotter, A., et al. 2010, ApJS, 192, 3Paxton, B., Cantiello, M., Arras, P., et al. 2013, ASTROPHYS J

SUPPL S, 208, 4Paxton, B., Marchant, P., Schwab, J., et al. 2015, ASTROPHYS J

SUPPL S, 220, 15Pignatari, M., & Herwig, F. 2012, Nuclear Physics News, 22, 18Pignatari, M., Herwig, F., Hirschi, R., et al. 2016, ASTROPHYS J

SUPPL S, 225, 24Porter, D. H., & Woodward, P. R. 1989, in ACM SIGGRAPH

Video Review, Vol. 44, Volume Visualization State of the Art,ed. L. Herr, 10-minute video segment: Simulations ofCompressible Convection with PPM

Ritter, C., & Cote, B. 2016, NuPyCEE: NuGrid Python Chemical

Evolution Environment, Astrophysics Source Code Library,

ascl:1610.015

Ritter, C., Cote, B., Herwig, F., Navarro, J. F., & Fryer, C. 2017a,

ApJS submitted, 1711.09172v1

Ritter, C., Herwig, F., Jones, S., et al. 2017b, MNRAS, submitted,

arxiv:1709.08677

Thielemann, F.-K., Nomoto, K., & Yokoi, K. 1986, A&A, 158, 17

Tucker, L., & Woodward, P. R. 1993, System Software and Tools

for High Performance Computing Environments (SIAM),

109–113, Paul Messina and Thomas Sterling, eds.

Wetherbee, T., Jones, E., Knox, M., Sandalski, S., & Woodward, P.

2015, in Proceedings of the 2015 XSEDE Conference: Scientific

Advancements Enabled by Enhanced Cyberinfrastructure

(ACM), Article No. 35. https:

//dl.acm.org/citation.cfm?doid=2792745.2792780

Woodward, P. R. 1986, in Astrophysical Radiation

Hydrodynamics, ed. K.-H. Winkler & M. L. Norman, Reidel,

245–326, online at http://www.lcse.umn.edu/PPMlogo

Woodward, P. R. 1992a, Segment of Art of Science program

broadcast nationally, PBS, Scientific American Frontiers,

produced at Showcase 92 exhibit at SIGGRAPH

Woodward, P. R. 1992b, in Proc. Supercomputing Japan, scientific

Visualization of Complex Fluid Flow, also available as

Minnesota Supercomputer Institute Research Report UMSI

92/233

Woodward, P. R. 1993, in IEEE Computer, Vol. 26

Woodward, P. R., Herwig, F., & Lin, P.-H. 2015, ApJ, 798, 49

Woodward, P. R., Porter, D. H., Anderson, S., Fuchs, T., & Herwig,

F. 2006, Journal of Physics Conference Series, 46, 370