SKA Science Data Processing · SKA imaging Drivers for data processing Computing Requirements...

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging

Drivers for dataprocessing

ComputingRequirements

SKA Science Data Processing

B. Nikolic(SDP Project Engineer)

&SDP Consortium Team

Astrophysics Group, Cavendish Laboratory, University of Cambridgehttp://www.mrao.cam.ac.uk/˜bn204/

1 April 2014AASTCS 2: Exascale Radio Astronomy

Monterey, CA, USA

http://www.mrao.cam.ac.uk/~bn204/

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Outline

Introduction

Big picture

SKA imaging

Drivers for data processing

Computing Requirements

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Introduction

SKA1-Mid SKA1-Low

SKA1-Survey

Computers

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Introduction

SKA1-Mid SKA1-Low

SKA1-Survey Computer

s

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Introduction

SKA1-Mid SKA1-Low

SKA1-Survey Computers

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



SKA Progress

Dec 2011 Incorporation of the SKA Organisation as anot-for-profit company registered in England

May 2012 Decision on site and scope of SKA-1

March 2013 Issue of the first Baseline Design, call forconsortia proposals

May 2013 Opening of SKA International HQ at JodrellBank

Sept 2013 Acceptance of consortia, signatures of MoUsand CAs, fully funded design phase

Nov 2013 Issue of L1 requirements, commencement ofthe work of consortia

March 2014 UK Government announces GBP 100 millionfor its share of SKA1 construction costs

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



SKA Project Structure

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Square Kilometre Array Simplified ContextW

est

ern

Au

stra

lia

So

uth

Afr

ica

Central Signal ProcessingScience Data Processor

Dish

LFAA

SADTManager

SKA1 Low:Low Frequency Aperture

Array

SKA1 Survey:

Dish Antennas with PAFs

SKA1 Mid:Dish

Antennas with Single-Pixel feeds

LFAA Correlator/Beam Former

Science Data Processor

Implementation(Australia)

Science Data Processor

Implementation(South Africa)

Survey Correlator

SKA1 Mid Correlator/

Beam Former

Pulsar Search

Processor

Monitor and Control

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Context of the SKA Science Data Processor

Central Signal Processor

Science DataProcessor

Telescope Manager Astronomers

Visibilities

Pulsar Candidates

Beam Formed Data

Telescope State

Control

Control

Monitoring

Image Cubes

Catalogues

· · ·

∼ 1000 km

∼ 1–10000 km

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Outline

Introduction

Big picture

SKA imaging



SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



We can see the route to SKA1-SDP

I FLOPS: ∼300 PetaFLOPS⇒ Expect won’t be at top of Top500

I Scratch storage capacity: ∼ 100 PetaBytes⇒ Both disks and solid state possibility

I Scratch storage throughput: ∼ 20 TByte/s⇒ ∼ 10000 nodes, lots of inherent parallelism

I Ingest rate ∼ 5 TByte/s⇒ FDR Infiniband switches almost there now

I Capital & Operational costs⇒ Commoditisation and general drive to low power

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Risks

I Convergence with rapidly evolving technology?

I Unknown aspects of data processing?

I Implementing the Software?

I Actual costs of scaling up?

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Outline

Introduction

Big picture

SKA imaging



SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Data levels

Raw Visibilities

CalibratedVisibilities

Image Cubes

CataloguesStackedImages

PowerSpectra

ConfirmedPulsars

PulsarTimings

PulsarCandidates

Time series

Archive

Raw data

‘Science Ready’

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Measurement Process

Pointsource

TurbulentIonosphere

Astronomicalwavefront

Corruptedwavefront

Collectors

Correlator

∑

q

∫ ∫

Ai

dA · Eq(x, y, t) ·W qpi (x, y, t)

∑

q

∫ ∫

Aj

dA · Eq(x, y, t) ·W qpj (x, y, t)

V pij

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Major cycle imaging iteration

Sky Model

FilterImage PlaneCorrections

Combine

FFT Grid Convolve

Visibilities

Subtract SolveCalibration

PhaseRotate

Predict

visibilities

Esti-

matesky

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Outline

Introduction

Big picture

SKA imaging



SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Data Rate – time smearing

Vi j = (a+ ib)

∼ D/λ

u

v

θθ̇ ≤ 7.3×10−5 rads−1

Bλ

⇒ tint ∼ D8Bθ̇

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Data Rate – bandwidth smearing

∼ D/λ2

u

v

Bλ2

Bλ1 B

λ3

⇒ ∆ ff ∼ D

8B

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Wide-fields & three dimensional geometry

A1 A2

Incomingwave-front

Off-axiswave-front

Referenceplane

A2’

cosλw

λw

θ

λw − λw cos θ = λw(1− cos θ) =

= λw(1−√

1− sin2 θ) = λw(1−√

1− l2)

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Outline

Introduction

Big picture

SKA imaging



SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



SKA-1 Parameters

SKA1-Low SKA1-Mid SKA1-SurveyNa 1024 254 96

tdump 0.6 s 0.08 s 0.3 sNf 256k 256k 256k

Nbeam 1 1 36DS 35 m 15 m 15 m

Bmax 100 km 200 km 50 km

NGW Size of the w-projection kernel

NAA Size of the anti-aliasing kernel

Nfacet Number of facets (on the side)

Tion Ionospheric timescale

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Requirements – ingest

Input Visibility Rate2 (Na−1)Na Nbeam Nf

tdump

SKA1-Low SKA1-Mid SKA1-Survey9× 1011/s 4× 1011/s 6× 1011/s

Input Data Rate16 (Na−1)Na Nbeam Nf

tdumpbyte

SKA1-Low SKA1-Mid SKA1-Survey7.2 Tbyte

s 3.3 Tbytes 4.5 Tbyte

s

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Requirements – w-snapshots

Snapshot Duration (in terms of w-kernel size)27510.Ds Nfacet NGW

Bmaxs

SKA1-Low SKA1-Mid SKA1-Survey9.6 Nfacet NGW s 2.1 Nfacet NGW s 8.3 Nfacet NGW s

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Requirements – pixel based

FFT nominal FLOP rate.012 B3

max Nbeam Nf log(

16 B2max

D2s N2

facet

)log 2 D3

s Nfacet NGW

Opss

SKA1-Low SKA1-Mid SKA1-Survey(assume Nfacet = 1 )

1.3×10+15

log 2 NGW

Opss

1.5×10+17

log 2 NGW

Opss

7.5×10+16

log 2 NGW

Opss

Re-projection Rate FLOPS.12 B3

max Nbeam NfD3

s Nfacet NGW

Opss

SKA1-Low SKA1-Mid SKA1-Survey6.9×10+14

Nfacet NGW

Opss

7.1×10+16

Nfacet NGW

Opss

4.0×10+16

Nfacet NGW

Opss

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Requirements – visibility based

Gridding FLOPS rate16 (Na−1)Na Nbeam Nf N2

facet (N2GW+N2

AA)tdump

Ops

SKA1-Low SKA1-Mid SKA1-Survey(assume Nfacet = 1, NAA = 9 )

7.2 · 1012(

N2GW + 81

)Ops

s 3.3 · 1012(

N2GW + 81

)Ops

s 4.5 · 1012(

N2GW + 81

)Ops

s

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Requirements – combined

Combined gridding+fft+reprojection16 (Na−1)Na Nbeam Nf N2

facet (N2GW+N2

AA)tdump

Ops +

.012 B3max Nbeam Nf log

(16 B2

maxD2

s N2facet

)log 2 D3

s Nfacet NGW+ .12 B3

max Nbeam NfD3

s Nfacet NGW

Ops

s

Optimised w-kernel sizes

SKA1-Low SKA1-Mid SKA1-Survey6 35 25

Combined gridding+fft+reprojection

SKA1-Low SKA1-Mid SKA1-Survey1.3× 1015 Ops

s 1.3× 1016 Opss 9.7× 1015 Ops

s

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Requirements – convolution kernels

Computing the convolution kernels128 (Na−1)Na Nbeam Nf N2

facet (N2GW+N2

AA) log(

8√

N2GW+N2

AA

)log 2 Tion Ops

SKA1-Low SKA1-Mid SKA1-Survey

(assume 60 second ionospheric timescale)

4.3× 10+14 Opss 3.7× 10+14 Ops

s 9.8× 10+14 Opss

Computing convolution functions over gridding rate8 tdump log

(8√

N2GW+81

)log 2 Tion

SKA1-Low SKA1-Mid SKA1-Survey31.Tion s 5.2

Tion s 19.Tion s

SKA SDP

B. Nikolic / SDP

Introduction

Big picture

SKA imaging



Scratch storage

Capacity27 (Na−1)Na Nbeam Nf

78125000 tdumpTbyte s

SKA1-Low SKA1-Mid SKA1-Survey154. Pbyte 71. Pbyte 97. Pbyte

Chunk Size (10 channels, each snapshot)(Na−1)Na s62500 tdump

Gbyte

SKA1-Low SKA1-Mid SKA1-Survey28. Gbyte 13. Gbyte .49 Gbyte

SKA Science Data Processing · SKA imaging Drivers for data processing Computing Requirements...

Documents

Transcript of SKA Science Data Processing · SKA imaging Drivers for data processing Computing Requirements...