eResearch Australasia Online Conference - Scalable platform for...
Transcript of eResearch Australasia Online Conference - Scalable platform for...
![Page 1: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/1.jpg)
Australia’s National Science Agency
Scalable platform for geophysical data analysis and visualisation
Pavel Golodoniuc | 24 October 2019
![Page 2: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/2.jpg)
Why?
Leveraging data interoperability to enable online interactive processing platforms
Data availability
Data initiatives
Interoperable services
Natural progression
Applications and toolkits
Algorithms availability and integration
![Page 3: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/3.jpg)
• Hand-crafting control files and inputs• Setting the environment• Compiling codes• And then…
$WIGLAF_MPIEXEC –np $WIGLAF_NP $WIGLAF_BIN/lsqmod –f0 ${wdir}/priormod.bin –f1 ${wdir}/observed.bin –f2 ${wdir}/priormod.bin –f3 ${wdir}/Lpi.bin –f4 ${wdir}/Cdi.bin –f5 ${wdir}/weights.bin –f6 ${wdir}/lsqmod.bin –tao_type ntr –tao_gatol 25.0 –tao_grtol 0.1 –tao_steptol 0.01 –tao_max_it 10 –tao_view -malloc_dump
Feature richness vs Usability
Don’t we all hate doing this?
![Page 4: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/4.jpg)
GeophysicalProcessingToolkit
Our candidate
Data-intensive processing
Compute-intensive processing
Multi-dimensional data sets that benefit from interactive visualisations
Routinely performed survey processing
Costly commercial options
![Page 5: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/5.jpg)
Geophysical Processing ToolkitRemote detection and measurement of the electrical conductivity structure of the subsurface:
– Green- or brown-fields exploration for minerals
– Groundwater detection
– Environmental
– Geo-technical applications
through geophysical inversion for conductivity-depth information
Spatially and temporally dense AEM data sets:
– Ranging from 1000 km to tens of thousands of km
– Sounding stations are taken every 0.2 seconds
– About 30-50 measurements taken at each site
Which translates to:
– Millions of individual stations of data, with tens of millions of measurements
![Page 6: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/6.jpg)
Capabilities
Browser-based
Desktop- or Cloud-hosted
Interactive visual analytics tools
Signal processing
Integrated inversion codes
See our demo
![Page 7: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/7.jpg)
• Multi-view interactive dashboards
• Interactive data visualisations
• Integration with signal processing codes
• Integration with forward modelling and inversion codes
• Complete scientific workflows:• Data processing / filtering / optimisation
• Data QA/QC
• Modelling, geophysical inversions
Researcher-focused interactivity
![Page 8: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/8.jpg)
Toolkit architecture
Cloud-hosted user application
Cloud data storage
Multiple processing workers deployed in a variety of environments
![Page 9: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/9.jpg)
![Page 10: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/10.jpg)
Another Case Study:Data Mosaic
![Page 11: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/11.jpg)
Same framework, different skin
![Page 12: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/12.jpg)
• One month sprint
• Python code, taken as is, packaged in a container:• Easily updatable / re-deployable
• Simpler server-side rendered visualisations
• Angular-based client application
• Cloud-hosted (storage/compute)
Data Mosaic
![Page 13: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/13.jpg)
Where to next Data integration
Missing data
Converters and optimisers
Development of a new NetCDF metadata standard for geophysical applications
Scientific application store
Publications and versioning of algorithms/modules
Workflows integration
Python notebooks, R, etc.
Payment system / billing
![Page 14: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/14.jpg)
Where to next Platform-level interoperability
Deployment options — from desktop to the Cloud
Scalable on-demand deployments
Software Development Kits (SDKs) and Application Programming Interfaces (APIs) for future extensions
![Page 15: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/15.jpg)
Online processing platform concept
![Page 16: eResearch Australasia Online Conference - Scalable platform for …conference.eresearch.edu.au/wp-content/uploads/2019/11/3... · 2019. 11. 25. · Australia’s National Science](https://reader034.fdocuments.us/reader034/viewer/2022051607/60381724888fcb56967cdd6c/html5/thumbnails/16.jpg)
Australia’s National Science Agency
Thank youMineral ResourcesPavel GolodoniucResearch Team Leader
+61 8 6436 [email protected]/en/Research/MRF