Overview of Science Data Processor

Paul AlexanderRosie BoltonIan CooperBojan NikolicSimon RatcliffeHarry Smith

Consortium Management and System Engineering

Aims of the SDP Work

• Detailed design of complete SDP system to CDR• Hardware platform• Complete software stack including system and

management layers

• Delivery is the design documentation supported by• Prototyping required for verification of the design

• Requirements driven approach to SE

Management Structure

Management Team

• Lead: Paul Alexander• PM: Harry Smith (acting)• Deputy PM: Ian Cooper• PE: Bojan Nikolic• SE: Simon Ratcliffe• PS: Rosie Bolton

Consortium Members

Management Groupings Partner Status

Workshare (%)

University of Cambridge (Astrophysics & High Performance Computing Groups) (UK)

Full 8.3

Netherlands Institute for Radio Astronomy Full 8.6International Centre for Radio Astronomy Research (AUS) Full 7.5SKA South Africa Full 7.2STFC Laboratories (UK) Full 3.1NIP Team Full 6.3

University of Manchester Max-Planck-Institut für RadioastronomieUniversity of Oxford (Physics)

University of Oxford (OeRC) (UK) Full 3.9Chinese Universities Collaboration Full 22.0New Zealand Universities Collaboration Full 2.7Canadian Collaboration Full 12.2

Canadian Universities CollaborationCADCCANARIE

Forschungszentrum Jülich (Germany) FULL/HPCF 2.1Centre for High Performance Computing (SA) HPCF/FULL 3.6iVEC (AUS) HPCF 1.0Centro Nacional de Supercomputación (ESP) HPCF 1.4Fundación Centro de Supercomputación de Castilla y León (ESP) HPCF 1.0

Instituto de Telecomunicações (Portugal) Associate 3.1University of Southampton (UK) * Associate 1.5University College London (UK) Associate 1.5University of Melbourne (AUS) Associate 1.0French Universities Collaboration Associate 1.0Universidad de Chile Associate 1.0

Consortium Industry Partnerss

Organization Area of ExpertiseAmazon SKA related verification projectsAMD E E accelerator options (GGPU), E E Accelerator software stacksAVANTEK E E ARM based architecturesARM E E ARM based architectures, software stacks for E E processing coresBull File & object store architecturesCISCO SKA related verification projectsDDN Storage system Architecture & streaming data handling, File & object store

architectures, SKA related verification projectsDell Storage system Architecture & streaming data handling, File & object store

architecturesGeomerics E E Accelerator software stacksGNODAL Interconnects and NICSHP Architecture, EE computing, BigData tools

Intel E E accelerator options (MIC), SKA related verification projectsMellanox Interconnects and NICSNAG Software models, software stacks for EE processing coresNVIDIA E E ARM based architectures, EE accelerator options (GGPU), E E

Accelerator software stacksOracle Storage system Architecture & streaming data handling, File & object store

architecturesParallel Scientific Software models, File & object store architectures SGI Storage system Architecture & streaming data handling,

SKA related verification projectsThoughtworks SKA related verification projectsTilera E E accelerator options (GGPU), E E Accelerator software stacksXyratex Storage system Architecture & streaming data handling, File & object store

architectures

Original Proposed Schedule

• Motivation was the expectation of major system changes following re-baselining

• Follow standard SE approach of element PDRs following system PDR

• Baseline element architecture at PDR

Agreed Delivery Milestones Stage 1

CODE Description Date Value (M €)

M1 Kick-off of the project Nov 2013 1

M2 Confirmation of Requirements Feb 2014 1

M3 Reconciliation of requirements with initial Functional Analysis

March 2014

0.5

M4 Analysis of the impact of SDP requirements on performance critical algorithms (including baseline-dependent averaging)

May 2014 0.5

M5 Analysis of scaling of algorithms and components to computational scale of SKA

June 2014 1

M6 Preliminary element architecture July 2014 2

M7 Key parameters for inputs into the System PDR (to be defined)

Sep 2014 3

M8 Participation in system-level review Dec 2014 1

M9 Closure of stage 1 Dec 2014 0.8

Agreed Delivery Milestones Stage 2

CODE Description Date Value (M €)

M10 Kick-off of Stage 2 Jan 2015 1

M11 Element PDR Feb 2015 5

M12 Test Report: Component Interface Completeness Jun 2015 1

M13 Test Report: Key Performance Metrics of Hardware in the Open Architecture Lab (LAB)

Sep 2015 1

M14 Test Report: Functional Completeness Dec 2015 1

M15 Test Report: Scaling of System Software Prototype on ISP HPC facilities

March 2016

1

M16 Test Report: Performance vertical prototypes Jun 2016 1

M17 Test Report: Test of System Software Prototype with revised components from vertical prototyping

Aug 2016 1

M18 Element CDR submission Sep 2016 5

M19 CDR close-out and wind-up of work Nov 2016 3.9

Work Breakdown and Scope

Scope

• Added WBS element to consider technical aspects of data delivery to users– Designed to inform final decision on edge of

observatory, does not imply adding additional responsibilities to the observatory

– Outside of costing envelope• Assume ingest visibilities, pulsar

candidates after search, other time-series data from CSP/SADT (TBC)

Design Approach

• Adopt Incremental and Iterative Design approach to the system engineering

• Fully exploited in the prototyping work.

• Distinguish between horizontal and vertical prototyping.

• Horizontal prototyping aims to provide a system-wide prototype

• Vertical prototyping provides detailed prototyping for performance and functionality of individual components

Design Approach

• Horizontal prototype of whole software system • Be used to test and verify the system decomposition and specification

of internal interfaces which emerge from the architectural analysis• Test models for scalability to SKA1 and give consideration to scaling

of the software system to SKA2. • Prototype the system architecture to test for the required flexibility in

architectural design• Prototype and test the design models for loose versus tight coupling

between software and hardware.

• Vertical prototyping of key components• Close interaction with industry on emerging technologies• Aim to roadmap into technology available for SKA1

Design Approach

• Open Architecture Lab • Approach based on Laurence Livermore Hyperion initiative:

• Integrated work with industry partners• Emphasis on a determining an appropriate scalable element• Emphasis on system-level components of the software stack

• Create an evaluation and prototyping testbed for new hardware and software technologies to address:

• Petascale I/O technology scaling for SKA1 and future capacity to SKA2

• Processor, memory, networking, storage, visualization, etc.• Designed for future technology refresh, expansion, and upgrades• Open source software stacks

Element Concept

Principles Behind Technical Solution

• Data parallelism provides a scaleable model through SKA1 to SKA2

• Emphasis is on the framework to manage the throughput

• Hardware platform will be replaced on a short duty cycle c.f. any HPC facility

• The approach to data analysis for the SKA will evolve during operations as more is learnt about the system

• Require a framework in which observatory staff can develop efficient radio-astronomy specific code

• Integration with TM distinguishes the system from a normal HPC environment

Example Data Rates and Data Products

• Baseline Design

• Aperture Array Line experiment (e.g. EoR)• 5 sq degrees; 170000 channels over 250 MHz bandwidth

~ 30 GB/s reducing quickly to ~ 1GB/s Up to 500 TB UV (Fourier) data; Images (3D) ~ 1.5 TB

• Imaging experiment with long baselines• 50 km baseline with the low-frequency AA or SKA1_Survey

1.5 TB/s reducing to ~ 50 GB/s Up to 1000 TB/day to archive if we archive raw UV data Images (3D) ~ 27 TB

Element Previous Data Rate (GB/s)

BL Design(Ingest)(GB/s)

Use Case(maximal)(GB/s)

LFAA 420 842 245Survey 42 4670 995Mid 8.5 1800 255

Overall Architecture

• Heterogeneous hardware architecture• Homogeneous software stack

…

IncomingData from

Correlator,beamformer

Switch

Buffer store

Switch

Buffer store

HPC

BulkStore

Ingest Processor

UV

Processor

Imaging:

Non-Imaging:

CornerTurning

CourseDelays

Ingest, flagging VisibilitySteering

ObservationBuffer

GriddingVisibilities Imaging

ImageStorage

CornerTurning

CourseDelays

BeamSteering

ObservationBuffer

Time-seriesSearching

Searchanalysis

Object/timingStorage

Regional Data

Centres

Ingest, flagging

Imaging Processing Model

Correlator

RFI excision and phase

rotation

Subtract current sky model from visibilities using current calibration model

Grid UV data to form e.g. W-projection

Major cycle

Image gridded data

Deconvolve imaged data (minor cycle)

Solve for telescope and image-plane calibration model

Update current sky model

Update calibration model

Astronomicalquality data

UV data store

UV processors

Imaging processors

Use Case requirements and estimates (preliminary)

SKA1 LOW CH2 EOR HI emission

CH2EOR source subtraction (continuum

survey)

CH3HI absorption

CH4High redshift HI

absorption

Bmax (km) 5 50 10 10G(out) GB/s 3.7 13 245 233Nchan 2500 Varies with bl 80,000 280,000GSamples/s 0.94 3.2 61 214Gridding ops / sample 6.3e3 63e3 13e3 3.2e4

PFlops/s (Gridding) 0.05 1.6 6.2 15

UV Buffer, TBytes 130 560 1100 10000Observation length, hours 5 5 6 6

Archive, 1000hrs of experiment (PBytes)

13 1.4 0.45 3100

System Sizing

SKA1 LOW / SURVEY (36 beams):• Data rate out of correlator:

– 4670 GBytes/s (SURVEY), – 842 GBytes/s (LOW)

• Max data rate into SDP: 995 GBytes/s – (SURVEY: DRM Ch 3 H1 absorption, proportional to Nbeams, assuming 36)

• Max computing load (flops/s): 32 Pflops– (SURVEY: DRM Ch 3 H1 absorption, proportional to Nbeams, assuming 36)

• Max UV buffer: 14 PBytes– (SURVEY: DRM CH3 H1 absorption)

SKA1 Mid:• Data rate out of correlator:

– 1800 GBytes/s (BL design page 49)• Max data rate into SDP: 255 GBytes/s

– (DRM CH3: H1 absorption, band 1)• Max computing load: 10.0 Pflops/s

– (DRM CH3: H1 absorption, band 1)• Max UV buffer: 11.0 PBytes

– (DRM CH3: H1 absorption, band 1)

Data Flow

Science Data Processor Local M&C

Science Data Processor

Telescope Manager

Cor

rela

tor /

B

eam

form

er

Data Routing Ingest

Visibility processing

Multiple Reads

Time Series Search

Multiple Reads

Data BufferData Routing

Time Series Processing

Image Plane Processing

Data Prodcuts

Sky Models, Calibration

Parameters ...

Meta Data

Master ControllerMaster Controller Local M&C Database

Tiered Data Delivery

Data Flow

Tiered Data Delivery

Astronomer

Regional Centre

Cloud

Sub-set of Archive

Data routing

Regional Centre

Sub-set of Archive

Regional Centre

Sub-set of Archive

Cloud access

SDP Core Facility South Africa

SDP Core Facility Australia

Example Implementation (current technology)

Host processorMulti-core X86

M-Core

->10TFLOP/s

M-Core

->10TFLOP/s

To rackswitches

Disk 1≥1TB

56Gb/s

PCI Bus

Disk 2≥1TB

Disk 3≥1TB

Disk 4≥1TB

Processing blade 1Processing blade 2Processing blade 3Processing blade 4Processing blade 5Processing blade 6Processing blade 7Processing blade 8Processing blade 9

Processing blade 10

Processing blade 11Processing blade 12Processing blade 13Processing blade 14Processing blade 15Processing blade 16Processing blade 17Processing blade 18Processing blade 19Processing blade 20

Leaf Switch-1 56Gb/sLeaf Switch-2 56Gb/s

42U Rack

Processing Blade:

GGPU, MIC,…?

• 20 TFlop• 2x56 Gb/s comms• 4 TB storage• <1kW power

• Capable host (dual Xeon)

• Programmable• Significant RAM

Blade Specification

Functional Analysis

Software Stack

SKA subsystems and service components

SKA Common Software Application FrameworkUIF Toolkit

Access Control Monitoring Archiver

Live Data Access

Logging System Alarm Service Configuration

ManagementScheduling

Block Service

Communication Middleware Database Support Third-party tools and

libraries Development tools

Operating System

High-level APIs and Tools

Core Services

Base Tools