Docker and Radio Astronomy - Containing Fragile Scientific Software by Gijs Molenaar (University of...
31
Docker & radio astronomy 1
-
Upload
docker-inc -
Category
Technology
-
view
6.939 -
download
0
description
As a scientific software engineer in the field of radio astronomy, I'm involved in creating, improving and maintaining a broad range of tools used by scientists. These tools are used to process and analyse data coming from various radio telescopes, particularly LOFAR (LOw Frequency ARray) and SKA (Square Kilometre Array). Radio astronomy has a long and rich computing related history, and with that comes a big pile of poorly written, fragile, and badly maintained legacy code. This problem is not unique to radio astronomy but is a common in various scientific fields. Although no quick solution exists when it comes to re-writing lots of legacy code, Docker helps to contain this fragile software, simplifies the installation and ensures that the software works. In this talk I will discuss how Papino, our umbrella project for containing various radio astronomy libraries and RODRIGUES, an online radio telescope simulator leverage Docker.
Transcript of Docker and Radio Astronomy - Containing Fragile Scientific Software by Gijs Molenaar (University of...
Docker & radio astronomy
1
Who Am I
Gijs Molenaar
Scientific Software Engineer in NL
PhD student in SA
MSc Artificial Intelligence
2
@gijzelaerr http://pythonic.nl
3
4
5
6
7
8
The problem
9
Scientist Q wants to crunchData X with
software Y onPlatform Z
10
Scientist Q
Smart person
Self educated on the field of software engineer
Gets the job done
Good at duct tape programming
11
Data X
Really big data
Datasets of peta, exa bytes
LOFAR - 50 GB/s
SKA - 10 times global Internet traffic
12
Software Y
13
Complex math heavy software
Written during PhD of scientist Q
Often bad coding practices
(Unit) tests?
Version control?
‘Runs on my computer’
Platform Z
14
Solution until now
15
Solution until now
16
Compile everything by hand
Still common practice (!)
17
18
Canonical launchpad
19
$ docker run skasa/papino ipython notebook
21
Rodrigues - telescope simulator
22
23
24
25
26
27
science vs scalable reliable web apps
Much less users
Much more data
High availability less important
Security less important
Performance very important
Durability very important
28
Lessons learned
Don’t build virtual machine images
use small minimal containers
use exec to examine problems inside container
29
Problems to solve / future work
GPU acceleration
Adaptation
‘Mixing’ containers (nixos?)
Container size (South Africa)
Container aware programming language / library?
30
