University College Dublin

MSc Space Science & Technology

Detecting Transient Events InLOFAR Data

Author:

Dualta O Fionnagain

Supervisors:

Peter T. Gallagher (TCD)

Brian A. Coghlan (TCD)

CONTENTS CONTENTS

Contents

1 Introduction 1

2 LOw Frequency ARray (LOFAR) 3

2.1 Interferometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

3 LOFAR Stations 6

3.1 Station Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3.2 Low-band Antenna (LBA) . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.3 High-band Antenna (HBA) . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.4 Digital Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.5 Central Processing (CEP) . . . . . . . . . . . . . . . . . . . . . . . . . 13

4 Key Science Projects (KSPs) 15

5 Irish LOFAR 16

6 Transient Buffer Boards (TBBs) 18

6.1 TBB pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

6.2 ARTEMIS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

7 TBB Data & Software 24

7.1 TBB Data Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

7.2 HDF5 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

7.3 PyCRTools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

8 TBB Transient Cluster Testing 28

8.1 TCD Computational Resources . . . . . . . . . . . . . . . . . . . . . . 28

8.2 Data Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

i

CONTENTS CONTENTS

8.3 Birr-Groningen Connection . . . . . . . . . . . . . . . . . . . . . . . . . 31

9 Results 32

9.1 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

9.2 TBB data processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

9.3 Processing times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

9.4 Antenna Positions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

9.5 Network Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

9.6 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

10 Conclusion 41

Appendix a

Glossary g

ii

LIST OF FIGURES LIST OF FIGURES

List of Figures

1 Basics of interferometry . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Superterp Station, Exloo, The Netherlands . . . . . . . . . . . . . . . . 5

3 LOFAR station layout types . . . . . . . . . . . . . . . . . . . . . . . . 7

4 LBA Antenna . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

5 HBA Tile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

6 RCU Filter/Amplifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

7 Dat Path Visualisation . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

8 Beamformer Mechnanism . . . . . . . . . . . . . . . . . . . . . . . . . . 13

9 Updated ILT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

10 Transient Buffer Board . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

11 Parallel Data Stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

12 Station Processing Pipeline . . . . . . . . . . . . . . . . . . . . . . . . 21

13 ARTEMIS Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

14 I-LOFAR Cluster Architecture . . . . . . . . . . . . . . . . . . . . . . . 29

15 Average Spectrum & Timeseries Data . . . . . . . . . . . . . . . . . . . 34

16 Timeseries Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

17 Histogram Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

18 Averaged Spectra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

19 HBA Tile Positions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

20 Network Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

21 All-Sky Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

22 Dynamic Spectrum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

23 VHECR Spectra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . a

24 tbbctl help file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . b

iii

LIST OF TABLES LIST OF TABLES

List of Tables

1 LOFAR station specifications . . . . . . . . . . . . . . . . . . . . . . . 8

iv

LIST OF TABLES LIST OF TABLES

Acknowledgements

I would like to gratefully acknowledge various people who helped over the course of this

project. In particular my supervisors Prof. Peter T. Gallagher & Dr. Brian Coghlan for

guiding each of the steps in this project and invaluable information relating to LOFAR,

radio astronomy, computation and networking.

I would like to thank John Walsh of the School of Computer Science for guiding

the computational and networking aspect of this project, configuring the hardware and

running network tests.

My thanks also goes to Diana Morosan for imparting her knowledge of LOFAR and

LOFAR data.

I would also like to thank the members of LOFAR and ASTRON for helping retrieve

TBB data to work with during this project.

v

1 INTRODUCTION

1 Introduction

Our understanding of the universe and cosmic objects was dramatically expanded with

the birth of radio astronomy. It allows a whole new multitude of observations, with

atmospheric windows allowing operations to be carried out from the surface. With

successful projects such as the Karl G. Jansky Very Large Array (VLA) proving that

radio interferometry is a powerful method to increase the spatial resolution available

at these long wavelengths, radio astronomy has gained a surge in support and interest.

The LOw Frequency ARray 1 (LOFAR) (van Haarlem, 2013) is at the forefront of this

new wave of radio astronomy. It was built in Europe (with its core in The Netherlands)

with the purpose or creating a powerful radio telescope that pushes the boundaries of

what not long ago was impossible, and to do so without any mechanically moving parts

(unlike other large radio dishes). This proved to be a significant undertaking in terms

of international cooperation, infrastructure, data rates and computational power. The

success of this radio array has led to the planning of larger and exceptional undertakings

such as the Square Kilometre Array (SKA) across southern Africa (the technology for

which LOFAR is pioneering).

This project studies how LOFAR operates, and in particular the possibility of using

Transient Buffer Boards (TBB), that are part of each LOFAR station, as a source

for raw and relatively unprocessed data. This is a precursor to the construction of

Irish LOFAR2 (I-LOFAR), the newest addition to the international LOFAR consortium

which will reside on the grounds of Birr Castle in Birr, Co. Offaly. This will be Ireland’s

first arrayed radio telescope and one which will establish Ireland at the forefront of this

new wave of radio astronomy.

If this TBB data can be accessed and processed, it provides a unique opportunity for

1www.lofar.org2www.lofar.ie

1

1 INTRODUCTION

Irish and international radio astronomy to gain additional data and science that would

otherwise be lost. The uniqueness of this system lies in the parallelisability of the TBB

data stream. Allowing for data streaming from these channels that would otherwise

not be used. This new source of data from the phased array that is the Low Band

Antenna (LBA) and High Band Antenna (HBA) could be synthesised on an I-LOFAR

local scale and maximise the Irish scientific benefit from the station.

As the I-LOFAR station is not constructed during the time of this project, TBB

data was acquired from ASTRON, which was then analysed on the high performance

computing cluster dedicated to I-LOFAR at TCD. This part of the project included

delays in schedule due to software issues.

This project is conceptually similar to a project named ARTEMIS (Serylak et al.,

2013) (Advanced Radio Transient Event Monitor and Identification Systems). This is

a software/hardware backend project carried out in England on the Rawlings Array,

Chilbolton telescope. It attaches to the existing station without disrupting the current

set-up. This is a similar concept as to what is being attempted with the I-LOFAR

station, with the difference that this project aims to use the TBB output not the final

station output.

2

2 LOW FREQUENCY ARRAY (LOFAR)

2 LOw Frequency ARray (LOFAR)

LOFAR is a radio astronomy project led by ASTRON. The main application of LO-

FAR was to study astronomical radio phenomena at the lower end of the frequency

spectrum, operating from 10-240MHz. LOFAR consists of 50 stations, this includes 24

core stations, 14 remote stations and 12 international stations. These stations have

different number of antennae and different layouts. The antennae act as a large phased

array or interferometer, a method that has been used in radio astronomy for a long

time and was pioneered by UK and Australian radio astronomers (One-mile Telescope

(MPIFR, 2007)) from the 1950s/1960s. Each LOFAR station is a fully functional radio

telescope on its own, with the larger international array giving increased resolution.

2.1 Interferometry

Radio interferometry began by creating basic radio dishes that would have one moveable

component, to alter baselines lengths and positions. The basic principle behind inter-

ferometry is that temporal delays between antenna receiving signals causes the signals

to interfere and cause constructive (in phase) or destructive (out of phase) interference.

LOFAR (and I-LOFAR) is a slightly different system to previous radio telescopes, such

as the VLBA, as it does not includes these large dishes that are already beamformed.

3

2 LOW FREQUENCY ARRAY (LOFAR) 2.1 Interferometry

Figure 1: Basics of interferometry. The geometrical time delay (τg) between antenna 1 and 2 canbe used to correlate each antenna voltage and average it to create a summed signal. This correlationallows for a much larger aperture than each individual telescope3

.These large dishes are directional antenna that favour signals originating from a

certain direction. These dishes can be thought of as beamformed and the data collected

is bias to this directional gain. With antenna like the LBA in LOFAR, there is only a

directional gain in the upward direction, with a very wide beam, so are omnidirectional.

The advantage being that all of this data can then be beamformed in any direction

after the observation has been made. This is a major advantage for looking at multiple

sources with the same observation. Simply by beamforming the data (which involves

adding time delays to different antenna) in a certain direction, different sources can

be focused upon, and directional gain can be modified. This provides a completely

unique observing capability (the Square Kilometre Array (SKA) will incorporate a

similar concept). This electronic beamforming technique makes LOFAR a very agile

instrument with capabilities for rapid re-pointing and simultaneous observations of

3http://www.cv.nrao.edu/course/astr534/Interferometers1.html

4

2 LOW FREQUENCY ARRAY (LOFAR) 2.1 Interferometry

different objects in the sky.

Figure 2: The Superterp station near Exloo, The Netherlands. This image shows 6 stations on theSuperterp and also some near surrounding stations.

With the dense core and very long baselines across Europe, LOFAR enables un-

matched sensitivity and spatial resolution at these extremely low frequencies. An im-

portant part of radio interferometry is good coverage in the uv plane. This uv plane is

called the complex visibility and is a 2D Fourier transform of the sky brightness. Insuf-

ficient coverage in the uv plane, will lead to undesirable effects in images. UV coverage

depends on the number of differing baselines. LOFAR complies to this requirement by

providing many different baselines between antenna.

5

3 LOFAR STATIONS

3 LOFAR Stations

The LOFAR stations of spread across most of Europe, with the central core located

near Dwingeloo in The Netherlands. The antenna in each of these stations act as

traditional radio dishes, providing collecting area, sensitivity with capabilities to be

pointed and track objects. However, past radio telescopes would move these large radio

dishes to point and track their targets. LOFAR has no moving parts so employs a

separate technique. This involves combining signals from different antenna elements

hence creating a phased array of elements. This is an advantage in many ways, but

requires a lot more computing resources. Therefore each station has its own LCU (Local

Control Unit) with significant computing resources, and each antenna includes its own

digital electronics components.

3.1 Station Configurations

LOFAR stations are categorised into three different types of stations, dependant on their

distance from the centre of the array. These are Core, Remote and International type

stations. The core and remote stations consist of 96 signal paths, which allow simul-

taneous operation of either 48 dual-polarised antenna, or 96 single-polarised antenna.

International stations are equipped with 196 signal paths which allow 96 dual-polarised

antenna, so as not to limit sensitivity whilst observing with the entire international

array.

The LOFAR core stations consist of 48 High Band Antenna (HBA) and 48 Low

Band Antenna (LBA). These are laid out in the formation shown in Figure 3. There

is a central hub (The Superterp) that is located near Exloo, The Netherlands. This

houses 6 of these core stations, shown in Figure 2 and is the centre of the LOFAR array.

Note the two differing fields in the Core station layout. These separate fields can be

6

3 LOFAR STATIONS 3.1 Station Configurations

Figure 3: Each of the station layouts; Core, Remote and International. Square gridded antenna arethe HBA antenna, while large circles indicate LBA fields. These stations are not to scale, Internationalstations are considerably larger than core stations (van Haarlem, 2013).

used together or as their own separate LOFAR stations, which allows shorter baselines

and greater uv coverage. Remote stations are categorised as stations located across

the Netherlands, that are not part of the core of LOFAR. These stations include 96

LBA in a single field and 48 HBA tiles in a single field. An International Station layout

is what will be employed in Birr, Co. Offaly. These are the largest of all the LOFAR

stations. They include 96 LBA and 96 HBA, with a LCU to control observations and

data transport.

As there is a finite amount of antenna, sidelobes are an issues when designing each

station. To reduce the overall contribution of each sidelobe, each separate station is

rotated relative to the others. This projects each sidelobe on a different part of the sky

so they do not constructively interfere and increase sensitivity off-axis.

Each station includes RCUs (digital receiver units) which are the first step in the

digital processing pipeline. These are each connected to an antenna, with 3 inputs on

each RCU board. For core and remote stations, 2 inputs are used for 96 LBA with the

3rd used for 48 HBA. As only one of these inputs can be used at one time, not all LBAs

7

3 LOFAR STATIONS 3.1 Station Configurations

can be read at once, restricting usage to LBA Inner mode (inner circle of 48 antenna)

or LBA Outer mode (outer annulus of 48 antenna). International stations have 192

signal paths so this problem can be avoided, with an additional empty input on each

RCU for future expansion (van Haarlem, 2013).

Station Config. # Stations LBA HBA Signal Paths Min. baseline(m) Max. baseline(km)

Superterp 6 2x48 2x24 96 68 0.24Core 24 2x48 2x24 96 68 3.5

Remote 14 2x48 48 96 68 121International 12 96 96 192 68 1158

Table 1: Summary of all the LOFAR stations specifications. The 6 Superterp stations are a subset ofthe Core 24 stations. Since the publication of (van Haarlem, 2013), three new international stationshave been added (all in Poland). When I-LOFAR is constructed, this will bring the total internationalstations to 13 and total staions to 51.

8

3 LOFAR STATIONS 3.2 Low-band Antenna (LBA)

3.2 Low-band Antenna (LBA)

Figure 4: The LBA antenna final design. Inset:The LNA board inside the head cap.

The low-band antenna at each LOFAR

station cover the frequency range of 30-

80MHz. They are designed so that they

work from the ionospheric cutoff fre-

quency (10MHz) to the commercial FM

radio band range (varies, but usually

around 90-100MHz). Due to RFI inter-

ference around these two frequency limits,

the LBAs became operationally limited to

their current range. The LBA antenna de-

sign for LOFAR was one that needed to be able to endure weathering for long periods

of time to ensure constant maintenance was not an issue. A design was required that

would also allow mass production, with 50 stations and 96 antenna each, amounts to

nearly 5000 antenna.

The LBA element can detect two perpendicular linear polarisations. Each of these

signals are detected by two copper wires, connected at the head of the antenna which

houses the Low Noise Amplifier (LNA) board. They are then anchored to the ground

via a synthetic spring or rope. Each polarisation has its own output, so there are two

separate coaxial cables running to each LNA, through the PVC piping. These coaxial

cables also supply power to the antenna. These antenna are omnidirectional, meaning

they have a view of the entire sky, and can be digitally beamformed to focus on different

astrophysical phenomena.

9

3 LOFAR STATIONS 3.3 High-band Antenna (HBA)

3.3 High-band Antenna (HBA)

Figure 5: Inside each HBA tile, 16 of these“crossed bow-tie” antenna sit encased in protectivepolystyrene (Falcke and for the LOPES collabora-tion, 2008).

The HBA for the LOFAR array incorpo-

rates a different design to monitor the

higher frequency range of 110-240MHz.

Again, the lower band (100-110MHZ) is

dominated by commercial FM stations so

is full of noise, as is the 240-250MHz

range, which limits the observing frequen-

cies. The design of the HBA is different,

incorporating 16 antenna elements into

one HBA tile. These tiles are then laid

out in a regular array. These 16 antenna

are beamformed on a tile-by-tile level by the LOFAR station. Each HBA tile consists

of square 4x4 element phased array, which is dual polarised and includes amplifiers and

analog beamformers. Each tile is 5mx5m in size, made of a polystyrene support with

aluminium antenna and a spacing between tiles of 15cm. The signals from these HBA

are transported using two coaxial cables to the RCU, as with the LBA.

3.4 Digital Processing

The Reciever Units (RCUs) are all stored in the cabinet on site at each station. This

cabinet is heavily shielded and contains all on-site computational power for the first

digital processing steps. At these RCUs, the signals are filtered, amplified and digitised.

The system is designed for sky-noise limited operations, so a 12-bit A/D converter is

used. This number of bits is enough for detection of signals ranging up to 40dB above

sky noise with a 48MHz bandwidth. The LBA signals can pass through two different

10

3 LOFAR STATIONS 3.4 Digital Processing

high pass filters (10 or 30 MHz), which is amplified and run through a 10-90 MHz

bandpass, amplified and passed through a low pass 90MHz filter. The HBA signal is

filtered through a 110-290 MHz bandpass, which can then be chosen to pass through

other bandpass filters to clean the signal further. It is then amplified and passed through

a low pass 270 MHz filter to attentuate signal above this cutoff. Both signals then are

amplified and pass through the ADC, sampled at 200MHz at 12 bits. This system is

shown in Figure 6.

Figure 6: Schematic showing each step inside the RCUs. For the Core and Remote stations all threeswitches are full, two LBA and one HBA. Filters and amplifiers are discussed in text. (Schoonderbeek,2007).

The sampling rate of 200MHz is chosen as this causes the Nyquist edge to occur at

100MHz, which is where FM radio stations operate. This band would be unobservable

anyway due to FM radio, so placing the Nyquist edge there means the overlap reduces

frequency bands that are inaccessible to an observer. Since the HBA frequency range

covers 200MHz, a 160MHz sampling option is available if observations are required

at this point, so that signals do not suffer from aliasing during the analog-to-digital

conversion. Note the previously mentioned three input design on these RCUs, with an

option only for inner or outer LBA antenna. In the international design, there are more

RCUs and one of the inputs is left empty, while the full LBA or HBA field can be used

11

3 LOFAR STATIONS 3.4 Digital Processing

at one time.

After the RCUs digitise each of the signals, they enter the digital processing stage.

It is at this stage the signal is either beamformed as required, but the raw digital data

can also be stored in a circular buffer. This circular buffer is the Transient Buffer Board

that allows for parallel storage of data from the antennas. The remote station processing

(RSP) boards process the data further. These boards are low-cost Field Programmable

Gated Arrays (FPGA). This means they can be programmed and patched remotely, for

a specific update. These RSPs separate the signal into 512 sub-bands. These sub-bands

allow for faster processing, and are either 156kHz or 195kHz in bandwidth depending

if the 160MHz or 200MHz clock is selected in the A/D step. By default 244 of these

sub-bands can be selected, leading to a total bandwidth of 48MHz per polarisation.

Figure 7: The digital processing steps directly succeeding the RCUs A/D step. Note the sub bandsplitting, which can be seen along the top section of the block. Control and Sync procedures operatethroughout each step (Schoonderbeek, 2007).

After the separation into sub-bands the main processing step is the phase rota-

tional beamformer. This sums the signals from each of the RCUs, multiplying them

12

3 LOFAR STATIONS 3.5 Central Processing (CEP)

by complex weights which represent the geometrical phase rotation. These weights

are calculated by the Local Control Unit (LCU) and sent to the RSPs. The actual

beamforming is completed separately for each sub-band, resulting in what are called

‘beamlets’, a smaller sub-banded beam. Many of these beamlets can be combined to

produce larger width beams. After the data is beamformed, it is sent over the fibre

optic network to the computational center in Groningen, described in Section 8.3.

Figure 8: This schematic depicts the beamformer architecture, including the buffer (the TBB processthat is essential to this project). The first stage of the beam forming is done on the same digital boardas the filtering process. In the second stage the result is added by the result of the previous board.The result of this transported to the next board in the ring. (Gunst A. W., 2005)

3.5 Central Processing (CEP)

Data from each of the LOFAR stations is sent over a high speed fibre network to the

central processing unit in Groningen. Initially the BlueGene/P supercomputer was

used to correlate and process LOFAR data but was replaced in 2014 by the COBALT

supercomputer, which is now the main hub of the CEP). The COBALT cluster was

a custom made supercomputer by the ASTRON team (Broekema, 2014). By utilising

both CPUs and GPUs COBALT reaches a processing power of 1.2PFLOPS which is

more powerful than the previous BG/P. This new COBALT cluster is composed of

16 GPUs (nVidia K10s) connected to Dual Xeon E’s, which are incorporated together

13

3 LOFAR STATIONS 3.5 Central Processing (CEP)

using Infiniband switches. Each K10 includes 1536 cores, with a max clock speed of

745 MHz (nVidia, 2012).

14

4 KEY SCIENCE PROJECTS (KSPS)

4 Key Science Projects (KSPs)

The LOFAR project includes Key Science Projects (KSPs). These are designed to study

key areas of radio astronomy and help keep an organised structure within the LOFAR

consortium. These key science projects are:

• Epoch of Reionisation

• Deep extragalactic surveys

• Transient sources

• Ultra high energy cosmic rays

• Solar science and space weather

• Cosmic magnetism

The utilisation of the TBBs will be most applicable to the transient sources and

solar science and space weather KSPs. The transient sources include many astronomical

objects, but the high temporal resolution nature of the TBBs will allow for very precise

temporal measurements. This is useful for pulsars, x-ray binaries and numerous other

transient sources in the sky (FRBs, flaring stars, lightning on Saturn, Jovian radio

emission, exoplanets). The solar science and space weather will benefit from the use of

TBB boards in particular due to the capability to record the Sun in very fine temporal

detail. Tiny variations on short timescales will be visible in any number of solar radio

bursts, flares and CMEs. The UHECR/VHECR research would also be applicable, as

these cosmic rays occur over very short time-scales.

Any Irish research group that are active in any of the above mentioned fields will

find a working TBB data acquisition system on I-LOFAR very useful for recording

scientific data.

15

5 IRISH LOFAR

5 Irish LOFAR

Figure 9: What the new International LOFAR Telescope (ILT) will look like when the Birr stationis incorporated with the rest of the international array.

I-LOFAR (www.lofar.ie) is a current project funded led by ASTRON and Science

Foundation Ireland to construct a new radio telescope as part of the international

LOFAR consortium in Birr, Co Offaly (shown in Figure 9 is what the International

LOFAR Telescope (ILT) will look like once I-LOFAR is incorporated). This project

was undertaken with the objective in mind to further the advancement of Ireland in

the field of astronomy, in this case the radio regime. It also will increase the size of the

international LOFAR array by a considerable amount, soon being the most westerly

station in the LOFAR contingent (which at the moment is Chilbolton station in the

UK).

The addition of the I-LOFAR site will increase the East-West baseline to nearly

1900km. This will push the angular resolution of the LOFAR International Telescope

to 0.1”, which is an amazing achievemnt at these low frequencies.

16

5 IRISH LOFAR

Birr, Co. Offaly provides a unique site for the position of the new ILT member. It

is a large flat site, with large views of the sky and very little RFI. This makes it an

ideal location for the new I-LOFAR telescope. The site will be leased to the LOFAR

consortium for a 35 year period.

Construction on the station is due to begin in October, with the breaking of the

ground and levelling for the forthcoming antenna, which will be laid next spring. The

ground works will include raising an area of land for the High Band Antennas (HBAs),

while a ditch and dyke will be installed for the Low Band Antennas (LBAs). The aim

for first light is summer 2017. This project is a major milestone in Irish STEM research,

making large strides like this in astronomy and astrophysics will increase prosperity in

these fields, producing much more research into this field. It also marks a significant

achievement in scientific outreach in Ireland. With a visitor centre to be constructed

near the station, it will encourage people to learn and inspire students to be more

involved in STEM based subjects.

17

6 TRANSIENT BUFFER BOARDS (TBBS)

6 Transient Buffer Boards (TBBs)

These boards are the central component to this project. Essentially a circular ring

buffer that is stored on the RAM for each of the LOFAR stations, it can be thought

of as memory for each individual LOFAR dipole or tile. They can be frozen and

read out (or ‘dumped’) at any time. This data consists of the raw voltages for each

of the antenna/tile (both x and y polarisations), before being processed (The HBA

tiles will have already gone through a tile-level pre-processing stage to form beamlets,

Section 3.3). They come in the form of Field Programmable Gated Arrays (FPGAs),

and originally included 1GB of RAM, which can now be upgraded from 5GB all the

way up to 32GB of RAM, depending on the station. This option to increase the RAM

allows for larger buffers, and therefore slower reaction times to fast transients, or more

data for longer observations ( 30 seconds max). Note that TBBs may operate in either

raw timeseries mode, or sub-band mode, but not both.

Figure 10: The entire TBB board, with RAM visible, and the three coaxial inputs. The on-sitecabinet will contain multiple copies of this board to buffer each antenna’s data streams (James, 2010).

18

6 TRANSIENT BUFFER BOARDS (TBBS) 6.1 TBB pipeline

Figure 11: Data stream flow within each station,indication the parallel system of the TBB. (vanHaarlem, 2013).

The TBB boards are made of four

FPGA boards that allow for low-cost,

easy maintenance and patching. This is

just like the RSP boards mentioned in

Section 3.4. Each FPGA uses 4 channels

(so 2 antenna with 2 polarisations each),

hence each TBB holds 16 channels. These

data streams through each TBB run in

parallel to the main pipeline. When the

TBB receives a dump command, the data is frozen and read out over the LOFAR net-

work to the central hub in Groningen, where it is kept in storage in the post-processing

section of the CEP. These ‘triggers’ that cause the read-out of data can originate from

a station-level command, ILT-level, or even from external sources, such as detectors

(Cosmic Rays) or other telescopes/satellites. For each station, the TBBs will be moni-

toring the data stream by running a detection algorithm. This algorithm is continuously

generating data that is processed by the LCU for events. If these events match with

some pre-defined criteria, a trigger occurs and the TBB data is read out.

Triggers for the TBBs have been around for quite some time (Horneffer, 2009; ter

Veen S., 2016), they are not a new concept for the LOFAR array. They will be trig-

gered using the pre-set monitoring algorithm. However, for this project, triggering

won’t necessarily be an issue. These TBBs will trigger automatically with the built in

algorithm.

6.1 TBB pipeline

The TBB pipeline on-site at the international LOFAR stations is key to the goal of this

project. The design of the station pipeline allows for the TBB pipeline to a be an entirely

19

6 TRANSIENT BUFFER BOARDS (TBBS) 6.1 TBB pipeline

parallel system to the rest of the on-going observations. This means that the TBB is

constantly buffering data streams from all antenna (Enriquez, 2012). In Figure 12 we

can see the different processing pipelines for each LOFAR station. Clearly the TBB

trigger at the bottom (in orange) can be triggered without halting other operations.

This feature has already been exploited by the CR research group by using external

high energy detectors to trigger a TBB data dump when they detect incoming CRs,

and also by the ARTEMIS team in Chilbolton, who don’t use the TBB data directly

but the output from the station itself (the data from the RSP that goes to Groningen).

Another team attempting something similar is the AARTFAAC team in Amsterdam

(Amsterdam-ASTRON Radio Transients Facility and Analysis Centre4 (Prasad and

Wijnholds, 2012)). AARTFAAC aims to create an all-sky monitor with the LOFAR

telescope. Their aim is to enable real-time, 24x7 monitoring for low frequency radio

transients over most of the sky locally visible to the LOFAR at timescales ranging from

milliseconds to several days. Using rapid triggering, it would then carry out follow-up

observations on transient candidates using LOFAR. For I-LOFAR, ideally another pipe

could be placed directing data to TCD from Birr to provide raw radio data of the sky.

4www.aartfaac.org

20

6 TRANSIENT BUFFER BOARDS (TBBS) 6.1 TBB pipeline

Figure

12:

Th

est

atio

np

roce

ssin

gp

ipel

ine,

show

ing

the

para

llel

isab

ilit

yof

the

TB

Bsy

stem

,all

owin

gfo

rp

ara

llel

ob

serv

ati

on

san

dop

erati

on

sof

the

mai

np

ipel

ine

and

the

TB

Bp

ipel

ine.

21

6 TRANSIENT BUFFER BOARDS (TBBS) 6.2 ARTEMIS

6.2 ARTEMIS

Figure 13: The ARTEMIS pipeline, showing the additional hardware used and provided by theARTEMIS project in orange. Using GPUs to dramatically speed up processing performance. Existinghardware to the LOFAR station is shown in blue (UK, 2010).

ARTEMIS is a combined hardware/software backend, which connects non-disruptively

to the LOFAR station hardware (Serylak et al., 2013). The pipeline for the ARTEMIS

system is shown in Figure 13. The Artemis backend hardware includes:

• Dell Poweredge C6100 - 8 Xeon 5650 CPUs

• Dell Poweredge C410x expansion - 4 Tesla M2050 GPUs

22

6 TRANSIENT BUFFER BOARDS (TBBS) 6.2 ARTEMIS

It should be noted that this ARTEMIS system is located on site at the Chilbolton

telescope. It has also been fully implemented at the French site at Nancay with test

installations existing at both the Swedish (Onsala) and German (Julich) LOFAR sites.

These two test installations can process 12MHz of sky bandwidth, with the Chilbolton

and Nancay telescopes being able to process the full available 48MHz sky beamformed

bandwidth available from a LOFAR station (Serylak et al., 2013).

ARTEMIS can perform real-time operations necessary to discover short duration

radio pulses from FRBs, thanks to a modular software structure operating in a C++

scalable framework developed at the University of Oxford. ARTEMIS includes pro-

cessing modules for receiving the data, poly-phase filtering for finer channels, Stokes

parameters generation, removal of terrestrial RFI, real-time brute force de-dispersion

and detection of interesting signals, temporal integration, in high-throughput CPU and

GPU code (Serylak et al., 2013).

The difference between this framework and the concept of portrayed in this report, is

the ARTEMIS system does not seem to gather their data from TBB boards, but rather

from the station output. This allows them to have un-processed data as it has not

passed through the CEP in Groningen. However, it is not as flexible as the TBB data

as it is partially beamformed, and TBBs have access to the lowest level of unprocessed

data from each antenna. However, if the TBB project reaches certain impasses that are

unforeseen, the ARTEMIS system could be considered as an alternative.

23

7 TBB DATA & SOFTWARE

7 TBB Data & Software

There is a massive demand for processing power when it comes to TBB data. The

sheer volume of data produced by each station is staggering. The sample data set for

TBBs on one of the core stations indicated a 80MB file for 2ms of data. This amounts

to 40GBps of data. With I-LOFAR having twice as many antenna as this, roughly

80GBps of data can be expected from the TBBs.

To circumvent this, smaller bandwidths can be selected, less antenna can be used,

or shorter data sizes can be acquired. This limits the observations as smaller bandwidth

decreased the size of the averaged spectrum, and less antenna will reduce sensitivity

and angular resolution on the sky, it will also decrease the coverage in the uv plane.

Attempts have been made to accelerate processing times for beamforming and data

reduction, using many core systems to achieve this (Sclocco A., 2012). This is where

LOFAR becomes a very cross-disciplinary project with knowledge in HPC and computer

science being of great value to GPU processing of LOFAR data.

To process the data, a software package known as PyCRTools was used, this is

discussed in Section 7.3.

7.1 TBB Data Acquisition

At station level there is a library of commands to control the TBBs. These commands

are run by two programs at station level. These are TBBcontrol and TBBdriver. The

TBBdriver is the low level controller, that performs basic operations such as clearing

the memory, assigning memory, starting and stopping recording into the ring buffer

and data read out from the ring buffer. TBBcontrol performs higher level functions,

such as correlating triggers that run on different TBB channels or datadumps from

external triggers. However, TBBcontrol is not fully operational during the writing of

24

7 TBB DATA & SOFTWARE 7.1 TBB Data Acquisition

this thesis, so TBBdriver is used to operate the TBBs. tbbctl is used to test TBBdriver

functionality. It can be used to start the TBB boards, to stop, read out data, and restart

the recording of data. For future reference (when operation of the I-LOFAR station

begins) the help file for the tbbctl command is included in Appendix A, Figure 24.

When reading out data from the TBB boards, they must be stopped with the --stop

command. TBB data can then be read out, until the buffers are restarted or recording

begins again. There are 3 commands to read data from TBB boards, these are:

• tbbctl --readpage

• tbbctl --readall

• tbbctl --read

--readpage is designed as an inspection tool, it read a certain number of pages from

a selected rcu to the hard disk at the station. Data is stopped at an arbitrary moment

relative to the start of the buffer. This command can also be used while still recording

data. This feature might be an invaluable feature while trying to send data to TCD

without stopping the buffer recording.

--readall command reads the last number of pages stored. It stops all RCUs by

default but can also select certain ones. As all RCUs are stopped at different times, the

data will have relatively different start times. The total buffer length must be longer

than these times to have data that can be correlated. After this command, the TBBs

will start to record data into the buffers again automatically. This commands stops the

buffers if they are not stopped already.

--read is used to read for a given amount of time before and after a central defined

time. This is useful for obtaining data around an event that has a precisely known

time. Disadvantages of this command is that it would include a writing to the local

25

7 TBB DATA & SOFTWARE 7.2 HDF5 Data

disk and then a reading from that disk to TCD, which would take up time. A way to

move the data directly over the 10G to TCD would be ideal.

7.2 HDF5 Data

This is the form of the TBB data. This dataset type was chosen as it allows for

the quickest access and clean segmentation of the LOFAR data. Allowing subsets

of ‘groups’ for each station in the dataset, and other subgroups inside each station

for each antenna dataset. It was a decision made by the LOFAR group as it was

the most flexible and suitable for high volumes of data with efficient input/output.

https://www.hdfgroup.org/HDF5/.

7.3 PyCRTools

TBB data is unique in that it is saved in the form of raw voltages. This makes the TBB

data very flexible, which is desirable for the end user of the data. It has the disadvantage

that all the processing must also be carried out by this end user. Converting raw

voltages to discernable data is a complex task. Hency why ASTRON have created

multiple software packages for this purpose. The most applicable for the purposes of

this project is called PyCRTools. This is an open source library written in Python.

Documentation on this software package can be found at https://www.astro.ru.

nl/software/pycrtools/. Installation instructions can be found at https://www.

astro.ru.nl/software/pycrtools/installation.html for MAC and Ubuntu. How-

ever, the builds in this documentation are for quite an old version of Ubuntu (in this

case 12.04 LTS). The first hurdle with this software was introducing it to a Ubuntu 16.04

LTS environment. This raised multiple issues and slowed progress. Eventually by using

an older version of casacore ( a suite of C++ libraries for radio astronomy data pro-

cessing, https://github.com/casacore/casacore) and tweaking some of the other

26

7 TBB DATA & SOFTWARE 7.3 PyCRTools

dependencies (such as Boost, another suite of C++ libraries), this could be achieved

(courtesy of John Walsh, CAG).

The end product was essentially a Python coding environment for the user, which

acted as a wrapper for lower level C++ routines. The API library for PyCRTools can be

found at https://www.astro.ru.nl/software/pycrtools/api.html. Tutorials can

also be found at the linked website. This software is still not perfect and is constantly

being updated and improved by the team at ASTRON. The results of using this software

are presented below.

Ideally, this software could be used to make dynamic spectra, which could be used to

interpret short timescale variabilities in many radio phenomena. Another end product

could be all-sky maps created from TBB data (Figure 21) showing what the radio sky

looks like at any point during the day (allowing for delays in data transfer), with 96

radio antenna.

27

8 TBB TRANSIENT CLUSTER TESTING

8 TBB Transient Cluster Testing

8.1 TCD Computational Resources

TCD has computational resources that will be essential to the implementation of this

project. The Computer Architecture and Grid (CAG) research group in TCD have

recently changed their infrastructure, leaving some resources available to support the

I-LOFAR project. The computational resources consist of a Blade Realtime Cluster

and a Dell2950 Realtime Cluster.

The Blade Realtime Cluster consists of 16 blades (Dell M600), each with 8 cores,

16GB DRAM and 240GB RAID1. The Dell2950 Realtime Cluster consists of 8 Dell

2950 blades, each with 8 cores, 16GB DRAM and 6TB RAID6. Each of these clusters

will be reconfigured to run Ubuntu Server 16.04 LTS, with Python and MPI.

It is proposed that the 16 blades in the Blade Realtime Cluster be replaced with new

Intel Haswell/Broadwell or nVidia Pascal or Intel Phi blades with extra 10Ge switches.

There is long term storage included of 130TB which will be utilised for the storage

of any raw TBB data deemed valuable. This large data storage is necessary for the

data volumes that I-LOFAR will produce. Shown in Figure 14 is the current design

and infrastructure of the CAG clusters for the I-LOFAR project.

28

8 TBB TRANSIENT CLUSTER TESTING 8.1 TCD Computational Resources

Figure

14:

Cu

rren

tin

frast

ruct

ure

of

the

CA

GI-

LO

FA

Rcl

ust

ers

as

of

30th

Au

gu

st2016.

29

8 TBB TRANSIENT CLUSTER TESTING 8.2 Data Rates

8.2 Data Rates

The data rate that will be produced by I-LOFAR is very large. The LOFAR interna-

tional array produces 13Tbit/s of raw uncorrelated data (2700 LBA antenna (or HBA

tiles) × 2 polarisations × 12 bit ADC × 200MHz sampling rate) and this is only in-

creasing with the addition of 3 international stations in Poland and 1 in Ireland. This

is much too large to be transported to the central processing unit in Groningen. This

data is reduced significantly, through beam-forming at a station level, yet still produces

a massive aggregate data rate of approximately 150Gbps for the current set of 50

LOFAR stations. This requires fibre networks that can handle such data volumes.

These large data rates produce challenges in data storage. The data rates men-

tioned above can relate to storage (post-processing) on the order of 68 TB/hr of

raw,uncorrelated visibilities. This data is sent over high speed, partially dedicated

fibre optic network infrastructure to the central facility in Groningen. This CEP is de-

scribed in Section 2. Originally the CEP used a BlueGene/P supercomputer to process

the data from all stations (this includes aligning in time, combining and processing all

correlated signals for interferometric imaging, tied-array beamforming for high time-

resolution observations and real-time triggering on incoming data streams). This Blue-

Gene/P (BG/P) offered about 28TFLOPS of processing power. However this BG/P

was replaced with a COBALT supercomputer which uses GPUs as well as CPUs. This

data rate of 150Gbps is for the entire international array, and if just the Irish station

is considered, approximately 3.2Gbps can be expected as the data rate. This equates

to 0.375GBps.

30

8 TBB TRANSIENT CLUSTER TESTING 8.3 Birr-Groningen Connection

8.3 Birr-Groningen Connection

There is a 10G fibre link from Birr to Groningen, where the CEP is housed. The data

stream from the I-LOFAR station is expected to take 3.2 Gbps. This will pass on a

(10Gbps Eir) link from Birr to Athlone, and on the 10Gbps HEAnet link to Dublin, over

the Dublin ROADM ring (HEAnet 2x40Gbps) to the Netherlands (GEANNT 10Gbps)

(Coghlan, 2016b). This last connection is for international purposes, but as this link

passes through Dublin, it provides an opportunity to send more data over the network

to the cluster in TCD, namely the TBB data. With only approximately 3.2 Gbps traffic

this leaves excess bandwidth for the purposes of this project. (Note anything over 7

Gbps would be considered excess traffic on a 10Ge link) This allows for more data to

be sent, hopefully in the form of TBB data from the buffers in I-LOFAR. Tests to see

if a similar connection to this one could exceed its purpose and transfer large amounts

of raw TBB data from I-LOFAR were carried out with results given in Section 9.

31

9 RESULTS

9 Results

The goal of this project was to investigate the feasibility of using the upcoming Irish

LOFAR station to detect astrophysical transient sources. This would be carried out

in parallel to the main observations made by LOFAR. Initially, real-time detection

monitoring was the ultimate goal of this project, and still is a possibility depending

on a number of temporal factors (read-out times, processing times, algorithm detection

times).

Using the transient buffer boards in a LOFAR station in this fashion raises numerous

technical challenges: operating the TBBs as required, transporting large volumes of

data, low latency processing, autonomous operation (triggers etc) and RFI mitigation.

Presented below are results from testing that was carried out on a number of different

aspects of the project. Testing on real TBB data from a core station taken in March

of last year (received courtesy of Dr. Sander ter Veen) was shown to produce averaged

spectra, with future work leading towards dynamic spectra and all sky maps. If these

spectra/maps could be used to create real-time analysis of the radio sky that would

be a significant achievement. Network tests were carried out by Dr. Brian Coghlan

on the 10Ge Dublin-Amsterdam link, to simulate LOFAR traffic that will exist on the

Dublin-Birr 10Ge link. The transient TBB high performance cluster in TCD was tested

to see how well it would process TBB data using the PyCRTools software.

9.1 Applications

If this comes to fruition, the benefits will traverse many fields of radio astronomy which

require high temporal resolution to study transient phenomena. The ability to observe

a maximum of 30s of TBB data would allow for the study of many astrophysical events.

For solar astronomy this will enable the study of solar flares and type I and III

32

9 RESULTS 9.2 TBB data processing

radio bursts (which occur on timescales of 1s or greater). It will also aid in the study

and prediction of space weather.

It will allow detailed study of Fast Radio Bursts (FRBs). A FRB is a high-

energy astrophysical event that manifests as a transient radio pulse lasting only a few

milliseconds. The short lifetime of these events makes them ideal candidates, with the

main issues being coincidentally catching them in data (unlikely) or triggering them

accurately to cause a TBB dump to occur (difficult). The high temporal resolution also

makes this perfect for observing FRBs. This would also be useful for studying pulsars.

Planetary radio emission could also be studied using these TBBs. As the data is

unprocessed, it can be beamformed after the observation. This could be used to follow

planetary movement across the sky, and detect lightning on Saturn or radio emission

from Jupiter.

9.2 TBB data processing

Currently data read out times from the TBB boards are a decisive factor in this project.

Data read-out times of 30s per antenna are implied (ter Veen, 2015) (although the

possibility of very short acquisition periods is currently being explored). This would

cease any real-time data from being used to process current radio events. 30s per

antenna accumulates to 48 minutes for the entire array. Of course, if required, data

from only a handful of antennas could be used; creating real-time maps from 12 antenna

with a 6 minute delay for example. If real-time data is not required it would still provide

unique data from I-LOFAR that is not available anywhere else, and in this sense could

be a great resource to have while studying transient radio events.

After much troubleshooting of the PyCRTools software, averaged spectra were ex-

tracted from the raw data, using code that was written using Python (shown in List-

ing 1). These average spectra were averaged over each tile in the dataset and plotted

33

9 RESULTS 9.2 TBB data processing

Figure 15: Shown in the top window is the averaged spectrum for the first 20 antenna. Clearly visibleseparately from the rest of the data are the noisy antenna/datasets (coloured in red and blue). Thesecould be distorted through RFI or the antenna themselves could be malfunctioning. Other antennaagree considerably well. This is a major advantage to accessing TBB data directly; being able toanalyse individual antenna separately. In the bottom window is an example of one of the antenna’sraw voltages (in this case antenna 10).

together (Figure 15).This data shows a HBA spectrum, which agrees well with standard

HBA spectrum provided in the LOFAR document, (van Haarlem, 2013).

Before the operation of the PyCRTools software, and an averaged spectrum could

be extracted, the timeseries data was examined. This was extracted from the LOFAR

TBB file with code written in Python. This code essentially looped over each of the

antenna files and extracted the data to be plotted. It resulted in the timeseries data

shown in Figure 15 and Figure 16.

From this timeseries data (Figure 16) it is impossible to distinguish anything by

eye. Histograms of this timeseries data were produced to see if any of the timeseries

contained any peculiar signals, or other important information, shown in Figure 17.

Panels on the left are x-polarisations and on the right are y-polarisations. From

these histograms it was concluded that it was mostly noise that was prevalent in the

34

9 RESULTS 9.2 TBB data processing

Figure 16: Top: Raw voltages retrieved from Tile 5 of the HBA array, x-polarisation. This is a 50µssample. Bottom: Raw voltages extracted from tile 8, y-polarisation, sample length of 12.5µs. Notethe x-axes are in units of 5ns.

signal as they produced Gaussians. This analysis showed that most of the voltages

represent themselves as white noise in the raw data files. It also helped as an exercise

in dealing with HDF5 data, which is not trivial.

The averaged spectrum was also analysed and different plots were created, removing

noisy antenna (also one bad antenna). The cleaned spectrum can be seen in Figure 18.

The peaks near the lower end of the spectrum corresponding to the same RFI peaks in

other HBA sample spectra (van Haarlem, 2013). These RFI peaks have been identified

in Figure 18.

The RFI sources in Figure 15 were identified from (Offringa and et al, 2013). We

35

9 RESULTS 9.2 TBB data processing

Figure 17: Panel A: Raw voltage histograms for tile 4. Panel B: Histograms for tile 11. Panel C:Histograms for tile 45. Note the left panels represent x-polarisations, while right hand side representy-polarisations.

can see that there are many sources in the LOFAR HBA regime (this is also the case

in the LBA regime). RFI detection is a difficult but essential part of recording reliable

and accurate observations. RFI down at the lower end of the spectrum is due to

FM radio stations. Towards the middle of the spectrum (125MHz-135MHz) there is

interference due to air traffic, with navigation systems interfering at 150MHz. The two

large broad band features near 178MHZ/183MHz are from digital audio broadcasting

(DAB) or digital radio. This is a relatively new source of RFI will become increasingly

36

9 RESULTS 9.2 TBB data processing

Figure 18: An averaged spectra for each antenna 10-20 in the HBA set. Bad antenna have beenremoved from this graph. Note the units on the y-axis are A/D units. Identified RFI signals (discussedbelow) include air traffic and digital radio.

problematic in the future. Currently in Ireland there are two multiplexes used for

broadcasting; MUX1 and MUX2. These operate in block 12C (227.360MHz) and block

12A (223.936MHz) respectively (Media, 2013). The list of radio stations carried by these

frequencies can be found at https://en.wikipedia.org/wiki/DAB_in_Ireland. Note

there has been testing for DAB multiplexes at lower frequencies in the past, such as

block 9B (204.640MHz). It is important to know where these DAB stations operate to

rapidly and accurately label them as RFI. Note that at these frequencies they will only

effect the higher end of the HBA spectra.

37

9 RESULTS 9.3 Processing times

9.3 Processing times

These averaged spectra took 51.471329 seconds to run on one of the blades in the

LOFAR cluster in TCD. This was nearly identical each time the process was run. This

figure could most definitely be decreased, as more cores are available for processing

(only one blade was utilised on this occasion) and there is a framework in place for

the use of GPUs to accelerate this even further. As mentioned in Section 3.5 the main

CEP in Groningen uses GPUs to accelerate processing. The PyCRTools package also

provides an interface for use with CUDA, although there was no time to incorporate

this into the scope of this project, it would be interesting to attempt to parallelise the

PyCRTools to process TBB data at an optimal level. Using MPI would also optimise

the processing times for this data, making use of all the high performance cluster nodes.

9.4 Antenna Positions

Figure 19: It is very clear that there are two antenna fields for the core HBA stations. This agreeswith the layouts presented in Figure 3. These positions are important to know precisely if one wantsto beamform data, as temporal delays must be incorporated into signals depending on the geometricalposition of their respective antenna.

38

9 RESULTS 9.5 Network Testing

The position of each of the antenna was precisely known. As the HDF5 data format

includes keys. These keys can be used to store additional data about each dataset, in

this case their precise position was known relative to the station. This is an essential

parameter required for beamforming. The positions of each of the antenna are shown in

Figure 19. This figure was created by looping over each HBA tile in the dataset, extract-

ing positional information from the HDF5 data format, and plotting these coordinates.

A z-component was also plotted but was left out here for clarity.

9.5 Network Testing

Figure 20: 10Ge networking tests on the Amsterdam-Dublin connection (mimicking the connectionfrom Birr-Dublin) during August 2016. These mimic 1ms (80MB) and 1s (80GB) LOFAR data trans-fers. Week 30 shows a data transfer rate of ¿3.6Gbps which is more than required. Overall the datarates reach 50% the required speeds without network fine tuning. Tested by Dr. Brian Coghlan /John Walsh, CAG.

Tests were carried out in August 2016 on the Amsterdam to Dublin connection (de-

scribed in Section 8.3. The results of these tests are shown in Figure 20. We can see

39

9 RESULTS 9.6 Future work

that for one week of the tests, download speeds reached 3.6Gbps, which is more than

necessary for our purposes. Of course, this is intended to represent how the Birr to

Dublin connection would act under the same circumstances. This is reassuring that the

fibre link can handle what is required for this TBB project to be successful. Proposed

is that the Birr-Dublin link is used to send short snapshots of data (1ms samples) from

the I-LOFAR station to TCD to run triggering algorithms on it. When an event is de-

tected, this triggers a data dump from the TBB. This is discussed in detail in (Coghlan,

2016a).

9.6 Future work

Ideally PyCRTools could be used to create dynamic spectra and sky-maps, both of which

have been accomplished by one of the software’s authors. These results are shown in

Figure 21 and Figure 22 (ter Veen S., 2010). However this was not achieved due to

time constraints on this project and to lengthy troubleshooting while the installation

of PyCRTools.

Note that this sky map in Figure 21 is using 43 LBA, so not the full capacity of

an international station. Even so, there are multiple objects visible; the Sun, M87,

Cassiopeia A and the Crab Nebula. Dynamic spectra like Figure 22 could be very

useful for monitoring solar activity using the I-LOFAR TBBs. The temporal resolution

in Figure 22 is enough to detect the frequency modulation in the dutch FM channels.

This dynamic spectrum was also created using LBA.

It is important that the TBBdriver and TBBcontrol are studied in depth (Sec-

tion 7.1) so that the most use can be extracted from these commands, in a way that

will maximise the ability and scientific reward of the TBB project.

40

10 CONCLUSION

Figure 21: This is an all-sky map made from one station’s TBB data, using 43 LBA from LOFARstation CS302. This ranges from 35-80MHz, with an integration time of 5ms. This is an example ofwhat could be accomplished if this set up is implemented at I-LOFAR (with an increased resolutionas there will be more than double the antennas) (ter Veen S., 2010)

Figure 22: Dynamic spectrum using a core station. The local dutch FM radio stations are clearlyvisible in this plot. Note the high temporal resolution of this plot. The frequency modulation of theFM band that is used to encode audio is visible in each FM channel. (ter Veen S., 2010)

10 Conclusion

Throughout this project the advantages of using TBB data has become clear. It is a

highly flexible data source with unmatched temporal resolution at these low frequencies.

41

10 CONCLUSION

If this TBB is not retrieved from the I-LOFAR station it will be overwritten by the

TBB itself after max 30 seconds. This seems a waste and this source of data should be

utilised to maximise scientific benefit for Ireland and the international community.

The hardware mechanics of how a LOFAR station has been investigated, revealing

the interesting operation of a radio telescope station. It is possible to retrieve the data

from I-LOFAR and send it to TCD, where the data will be processed and analysed. It

is a matter of choice as to what format this is carried out in, to send small frequent

packets of data, or a wait longer to send larger portions of data. This is provided TBB

board read out times do not become an issue. Local station controls have been explored

to the best they have been documented, and it seems that TBB commands on a station

level should allow acceptable read out times.

Software has been installed on the LOFAR cluster in TCD, which will be used to

process incoming TBB data and output it to be analysed further. This software is

running on one blade at the moment, with plans to expand to the entire cluster once

sufficient testing has been carried out. It has been shown that this software (lofarsoft-

PyCRTools) can be used to form averaged spectra per antenna in a station dataset.

Future work would allow this to be expanded to produce dynamic spectra and all-sky

maps.

Once I-LOFAR construction is nearly completed the possibility of employing this

TBB infrastructure will become a reality and more detailed testing could be performed

to increase productivity and performance. There will most likely be unforeseen obsta-

cles (for example data read out times), but none seem implausible to overcome given

sufficient expertise on the hardware and software of a LOFAR station.

More testing is required on the networks while the LOFAR station is being set up

in Birr. A more practical understanding of the inner workings and operation of the

LOFAR station would benefit the project. This is something that will become easier

42

10 CONCLUSION

when the station is built. A post-grad or post-doc fully dedicated to the progression of

this project is most likely necessary.

43

10 CONCLUSION

Appendix

1. VHECR test figure

Figure 23: Left: Averaged spectrum of high energy cosmic ray incident on two LBA antenna. Right:Corresponding TBB voltages from one of the antenna. This image was created using PyCRTools withgiven example data with the software package.

a

10 CONCLUSION

Figure 24: Help file for the tbbctl command.

b

10 CONCLUSION

1 from p y c r t o o l s import ∗2 import matp lo t l i b . pyplot as p l t34 p l t . f i g u r e ( )5 f i l e n a m e l o f a r = ”L401459 D20151020T144127 .608 Z CS001 R000 tbb . h5”6 d a t a f i l e = open ( f i l e n a m e l o f a r )7 f f t d a t a = d a t a f i l e . empty ( ”FFT DATA” )89 nblocks = d a t a f i l e [ ”MAXIMUM READ LENGTH” ] / d a t a f i l e [ ”BLOCKSIZE” ]

10 avspectrum = hArray ( f l o a t , d imensions=f f t d a t a , name=”Average spectrum” )1112 # Calcu la te s p e c t r a l power13 f o r b lock in range ( nblocks ) :14 d a t a f i l e [ ”BLOCK” ] = block15 f f t d a t a . read ( d a t a f i l e , ”FFT DATA” )16 hSpectralPower ( avspectrum [ . . . ] , f f t d a t a [ . . . ] )1718 # Create 1 s t p l o t19 f r e q u e n c i e s = d a t a f i l e [ ”FREQUENCY DATA” ] . s e tUni t ( ”M” , ”” )20 p l t . subp lot (1 , 2 , 1)21 p l t . t i t l e ( ”Average spectrum f o r two antennas ” )22 p l t . x l a b e l ( f r e q u e n c i e s . getKey ( ”name” ) + ” [ ” + f r e q u e n c i e s . getUnit ( ) + ” ]

” )23 p l t . y l a b e l ( avspectrum . getKey ( ”name” ) + ” [ ” + avspectrum . getUnit ( ) + ” ] ” )24 f o r i in range (2 ) :25 p l t . semi logy ( f r e q u e n c i e s . vec ( ) , avspectrum [ i ] . vec ( ) )2627 # Prepare block28 d a t a f i l e [ ”BLOCK” ] = 229 d a t a f i l e [ ”BLOCKSIZE” ] = 2 ∗∗ 1630 t i m e a l l = d a t a f i l e [ ”TIME DATA” ]31 t i m e a l l . s e tUni t ( ”m” , ” s ” )32 f x a l l = d a t a f i l e [ ”TIMESERIES DATA” ]3334 # Create 2nd p lo t35 p l t . subp lot (1 , 2 , 2)36 p l t . t i t l e ( ”Time s e r i e s o f antenna 10” )37 p l t . p l o t ( t i m e a l l . vec ( ) , f x a l l [ 0 ] . vec ( ) )38 p l t . x l a b e l ( t i m e a l l . getKey ( ”name” ) + ” [ ” + t i m e a l l . getUnit ( ) + ” ] ” )39 p l t . y l a b e l ( ” E l e c t r i c F i e ld [ADC counts ] ” )

Listing 1: Python code using PyCRTools to display averaged spectrum and timeseries spectrum forTBB data.

1 #! / usr / bin /env python2 # −∗− coding : utf−8 −∗−34 import h5py as h55 import matp lo t l i b as mpl6 from m p l t o o l k i t s . mplot3d import Axes3D7 import matp lo t l i b . pyplot as p l t8 import numpy as np

c

10 CONCLUSION

910 f = h5 . F i l e ( ’ . . / L401459 D20151020T144127 .608 Z CS001 R000 tbb . h5 ’ )1112 s = f [ ’ / Stat ion001 ’ ]13 da ta s e t s = s . i tems ( )1415 names = [ x [ 0 ] f o r x in da ta s e t s ]1617 x = [ ]18 y = [ ]19 z = [ ]20 names = names [ : : 2 ]21 l ength = len ( names )2223 #READ POSITION DATA24 f o r i in xrange ( l ength ) :25 pos = s [ names [ i ] ]26 POSITION = pos . a t t r s [ ’ANTENNA POSITION VALUE ’ ]27 x . append (POSITION [ 0 ] )28 y . append (POSITION [ 1 ] )29 z . append (POSITION [ 2 ] )3031 x [ : ] = [ a − 3826880.7895200001 f o r a in x ]32 y [ : ] = [ a − 460876.04822 f o r a in y ]33 z [ : ] = [ a −a f o r a in z ]3435 l a b e l = [ ]36 f o r i in xrange ( l ength ) :37 l a b e l . append ( i +1)3839 l ength = len ( names )40 coords = z ip (x , y , z )4142 #3D PLOTTING TOOL43 mpl . rcParams [ ’ l egend . f o n t s i z e ’ ] = 1044 f i g = p l t . f i g u r e (1 )45 ax = f i g . gca ( p r o j e c t i o n=’ 3d ’ )46 ax . p l o t (x , y , z , ’ o ’ , l a b e l=’ Sta t i on Antenna P o s i t i o n s ’ )47 ax . l egend ( )4849 p l t . f i g u r e (3 )50 p l t . p l o t (x , y , ’ o ’ )51 p l t . x l a b e l ( ’X Coordinate (m) ’ )52 p l t . y l a b e l ( ’Y Coordinate (m) ’ )53 p l t . t i t l e ( ’ Core Stat i on ’ )54 p l t . g r i d ( )55 p l t . show ( )

Listing 2: Python code used to determine tile positions of the HBA array.

d

REFERENCES REFERENCES

References

Broekema, C. (2014). COBALT: A new correlation for LOFAR. http:

//on-demand.gputechconf.com/supercomputing/2013/presentation/

SC3132-COBALT-Correlator-LOFAR.pdf. [Online; accessed 22-August-2016].

Coghlan, B. A. (2016a). CAG I-LOFAR Realtime Event Trig-gering Proposal. https://www.cs.tcd.ie/coghlan/I-LOFAR/

CAG-ILOFAR-RealtimeEventTriggering-proposal-July2016-DRAFT-20160725-1254-v1.

pdf. [Online; accessed 27-August-2016].

Coghlan, B. A. (2016b, May). CAG I-LOFAR Sci-ence Proposal. https://www.cs.tcd.ie/coghlan/I-LOFAR/

CAG-ILOFAR-science-proposal-May2016-DRAFT-20160104-1041-v7.pdf. [On-line; accessed 26-August-2016].

Enriquez, J. E. (2012). Identifying Real Fast Radio Transients usingTBBs. http://www.lofar.org/wiki/lib/exe/fetch.php?media=public:lsm_

new:2014_03_19_enriquez.pdf. [Online; accessed 14-August-2016].

Falcke, H. and for the LOPES collaboration (2008, April). Radio Detection of Ultra-High Energy Cosmic Rays. ArXiv e-prints .

Gunst A. W., K. G. W. (2005). Signal transport and processing at the LOFAR remotestations. http://ursi.org/Proceedings/ProcGA05/pdf/J06.4%280464%29.pdf.[Online; accessed 11-August-2016].

Horneffer, A. (2009). Status of the VHECR pipeline. http://www.lofar.org/wiki/

lib/exe/fetch.php?media=public:meetings:2009-05_technical_status:

2009-05-18_horneffer_-_vhecr_status.pdf. [Online; accessed 10-August-2016].

James, C. W. (2010). The LOFAR Cosmic Ray KSP and associated pipelines. https://www.astron.nl/sites/astron.nl/files/cms/PDF/LDS2010_james.pdf. [Online;accessed 11-August-2016].

Media, W. (2013). DAB frequency tables. http://www.wohnort.org/DAB/freqs.html.[Online; accessed 21-August-2016].

MPIFR (2007). Introduction to Radio Interferometers. https://www.mpifr-bonn.

mpg.de/1040947/Garrett_RadioAstronomy_red.pdf. [Online; accessed 27-July-2016].

nVidia (2012). Tesla K10 GPU Accelerator. http://www.nvidia.com/content/

PDF/kepler/Tesla-K10-Board-Spec-BD-06280-001-v07.pdf. [Online; accessed17-July-2016].

e

REFERENCES REFERENCES

Offringa, A. R. and et al (2013, January). The LOFAR radio environment. A&Ap 549,A11.

Prasad, P. and S. J. Wijnholds (2012, May). AARTFAAC: Towards a 24x7, All-skyMonitor for LOFAR. ArXiv e-prints .

Schoonderbeek, A. W. G. . G. (2007). LOFAR: Architectural DesignDocument. http://www.lofar.org/wiki/lib/exe/fetch.php?media=public:

documents:14_station-add-013.pdf. [Online; accessed 8-August-2016].

Sclocco A., e. (2012, May). Radio Astronomy Beam Forming on Many-Core Architec-tures. IEEE .

Serylak, M., A. Karastergiou, C. Williams, W. Armour, M. Giles, and LOFAR PulsarWorking Group (2013, March). Observations of transients and pulsars with LOFARinternational stations and the ARTEMIS backend. In J. van Leeuwen (Ed.), NeutronStars and Pulsars: Challenges and Opportunities after 80 years, Volume 291 of IAUSymposium, pp. 492–494.

ter Veen, S. (2015). Searching for Fast Radio Transients with LOFAR. http://hdl.

handle.net/2066/147186. [Online; accessed 14-August-2016].

ter Veen S. (2016). Testing Extenal TBB triggers from Effelsberg tele-scope. http://www.lofar.org/wiki/lib/exe/fetch.php?media=public:lsm_

new:2016_02_17_terveen.pdf. [Online; accessed 10-August-2016].

ter Veen S., Falcke H., e. (2010, June). FRATs: a real-time search for Fast RadioTransients with LOFAR . ISKAF2010 science meeting .

UK, L. (2010). ARTEMIS. http://www.lofar-uk.org/wiki/uploads/Main/

ARTEMIS.pdf. [Online; accessed 14-August-2016].

van Haarlem, M. e. a. (2013, August). LOFAR: The LOw-Frequency ARray. Astronomy& Astrophysics 556, A2.

f

Acronyms Acronyms

Acronyms

CAG Computer Architecture Grid. 28

CEP CEntral Processing. 13

CR Cosmic Ray. 19

FPGA Field Programmable Gated Array. 12

HBA High Band Antenna. 1, 6

ILT Internation LOFAR Telescope. 16

LBA Low Band Antenna. 1

LCU Local Control Unit. 6

LNA Low Noise Amplifier. 9

LOFAR LOw Frequency ARray. 1

RCU Receiver Unit. 7

TBB Transient Buffer Board. 1

g