Leveraging Distributed Research Cloud Infrastructures for Domain Science Research and Experimentation
Anirban Mandal, Cong Wang, Paul Ruth, Komal Thareja (RENCI, UNC – Chapel Hill)
Ewa Deelman, George Papadimitriou (USC/ISI)
Michael Zink, Eric Lyons (UMass Amherst)
Ivan Rodero, J. J. Villalobos (Rutgers University)
Funded by the National Science Foundation
Grant #1826997
Drew Angerer, Getty ImagesOpen Cloud Workshop, Boston University, March 2020
Distributed Research Clouds for Domain Sciences
• Develop novel algorithms and mechanisms to offer optimized data-flows across different kinds of national CI.
• Dynamic multi-cloud resource provisioning - ExoGENI, Chameleon, XSEDE JetStream, MOC.
• Network-centric platform to bridge the abstraction gap.
• Data-aware scheduling in Workflow Management Systems (Pegasus).
• Deploy solutions for use in observational science communities - CASA and OOI.
Major challenge is integration of data CI to science workflows: data movement across diverse infrastructure, complex workflows, distributed repositories, and elastic application compute/storage/network requirements.
Multi-cloud Provisioning and Network Orchestration
• Resource requirements from applications are generated using a Gantt chart abstraction
• Mobius network-centric platform• Multi-cloud provisioning of compute and
storage resources• Layer 2 network provisioning• Resource monitoring and control• https://github.com/RENCI-NRIG/Mobius• Exposes a REST API for Applications and
Workflow Management Systems• Management, Optimization & Prioritization
of Data Flows with virtualized SDX
CASA OOI
CASA: Collaborative Adaptive Sensing of the Atmosphere
• Traditional Next Generation Weather Radars (NEXRAD)
• High power, long range• Limited ability to observe the lower
part of the atmosphere because of the earth's curvature
• CASA• Network of short range Doppler
radars deployed in DFW area• Adjustable sensing modes in response
to quick weather changes• Suitable for near-ground weather
events: tornado, hail, high winds
CASA Workflows
> 7M people, >100K businesses, >1500 Corporate HQs
SC’19 SCinet Tech Challenge
Distributed Research Cloud Deployment with Mobiusfor CASA Applications
CASA Operational Workflows on Chameleon and ExoGENIwith Layer 2 Provisioned Data Flows
• Distributed testbeds• Heterogeneous: ExoGENI and
Chameleon testbeds
• ExoGENI• 11 workers (VMs)• 4 cores and 12 GB RAM• NFS storage
• Chameleon• 4 workers (bare metal)• 24 cores and 192 GB RAM
• Radar data repository• UNT via L2 stitching port
• Connected via 10 Gbpsnetwork.
CASA Hail Workflow
Multi-workflow display of hail (orange) and wind (red) contours, with GIS boundaries and infrastructure overlaid during a severe weather event.
Pegasus workflow for Hail
CASA radar data
Current Research Thrusts
• Optimization & prioritization of science data flows with virtualized SDX.
• Active monitoring and control for maintaining QoS.
• Supporting wider federation of clouds, including EC2, CloudLab, and OCT.
• Data-aware workflow scheduling. • Deploying novel CASA workflows (e.g.
drone path planning).• Support for streaming data and on-
demand workflows from Ocean Observatory Initiative (OOI) NSF Large Facility.
Thank you !
Funded by the National Science Foundation
Grant #1826997
Funded by the National Science FoundationGrant #1826997
Top Related