LOFAR - finding transients in the radio spectrum
-
Upload
gijs-molenaar -
Category
Technology
-
view
343 -
download
2
Transcript of LOFAR - finding transients in the radio spectrum
![Page 2: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/2.jpg)
Agenda
● Transient detection
● Pipeline layout
● Software
![Page 3: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/3.jpg)
Transients Detection
● Static sources are boring
● Transient – something that changes
● Huge amount of data – automation
![Page 4: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/4.jpg)
Image data
![Page 5: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/5.jpg)
The data
● 1 datacube per second● 10 frequency bands
● In the future 10 images per second● In the future 4 different polarization
● Non stop
![Page 6: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/6.jpg)
Database
Transient detection Pipeline
Source extraction
Association
Source extractionQuality checkExtract metadata
Detection Classification
![Page 7: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/7.jpg)
● The glue
● Distributing computation● Image processing● Statistics● Source extraction● Database interactions
![Page 8: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/8.jpg)
Distributed Computation
● Home made libary● SSH based
● Difficult to debug● Difficult to profile● Doing research on Celery and Hadoop
![Page 9: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/9.jpg)
Database
● Move calculation to the data● Highly structured● Independent data● Naturally separable by sky coordinates
● ~100 TB/year● 10.000 insert/second
![Page 10: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/10.jpg)
MonetDB
● Relational● Column store DB● Fast● Auto tuning!● Developers next door (CWI)
![Page 11: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/11.jpg)
Challenges
● Debugging queries
● MonetDB still in active development
![Page 12: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/12.jpg)
INSERT INTO tempbasesources(xtrsrc_id,datapoints,I_peak_sum,I_peak_sq_sum,weight_peak_sum,weight_I_peak_sum,weight_I_peak_sq_sum)SELECT b0.xtrsrc_id,b0.datapoints+ 1 AS datapoints,b0.I_peak_sum+ x0.I_peak AS i_peak_sum,b0.I_peak_sq_sum+ x0.I_peak * x0.I_peak AS i_peak_sq_sum,b0.weight_peak_sum+ 1 / (x0.I_peak_err * x0.I_peak_err) AS weight_peak_sum,b0.weight_I_peak_sum+ x0.I_peak / (x0.I_peak_err * x0.I_peak_err)AS weight_i_peak_sum,b0.weight_I_peak_sq_sum+ x0.I_peak * x0.I_peak / (x0.I_peak_err * x0.I_peak_err)AS weight_i_peak_sq_sumFROM basesources b0,extractedsources x0WHERE x0.image_id = @imageidAND b0.zone BETWEEN CAST(FLOOR((x0.decl - @theta) / x0.zoneheight) AS INTEGER)AND CAST(FLOOR((x0.decl + @theta) / x0.zoneheight) AS INTEGER)AND ASIN(SQRT((x0.x - b0.x)*(x0.x - b0.x)+(x0.y - b0.y)*(x0.y - b0.y)+(x0.z - b0.z)*(x0.z - b0.z)) / 2)/SQRT(x0.ra_err * x0.ra_err + b0.ra_err * b0.ra_err+x0.decl_err * x0.decl_err + b0.decl_err * b0.decl_err)< @assoc_r;
![Page 13: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/13.jpg)
MonetDB and Python
● We maintain the MonetDB Python API
● http://pypi.python.org/pypi/python-monetdb/
● Problems? Ask me :)
![Page 14: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/14.jpg)
Djonet
● MonetDB backend for Django● https://github.com/gijzelaerr/djonet
● brew install monetdb● pip install python-monetdb djonet
● Contributions are welcome!
![Page 15: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/15.jpg)
VO events
● Standardized language● Report observations of
astronomical events
● Hey world, check this supernova out over there
● http://comet.transientskp.org
![Page 16: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/16.jpg)
Visualisation
● Web interface● Django!● Not public (yet)
![Page 17: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/17.jpg)
More
● http://www.transientskp.org/● http://www.lofar.org/● http://www.aartfaac.org/
![Page 18: LOFAR - finding transients in the radio spectrum](https://reader034.fdocuments.us/reader034/viewer/2022042816/5595acb61a28ab7b678b487e/html5/thumbnails/18.jpg)
Questions?