Virtual Geophysics Laboratory (VGL) VGL v1.2 NeCTAR Project Close R.Fraser, T.Rankine, J.Vote,...
-
Upload
adrian-roberts -
Category
Documents
-
view
219 -
download
0
Transcript of Virtual Geophysics Laboratory (VGL) VGL v1.2 NeCTAR Project Close R.Fraser, T.Rankine, J.Vote,...
Virtual Geophysics Laboratory (VGL)
VGL v1.2 NeCTAR Project CloseR.Fraser, T.Rankine, J.Vote, L.Wyborn, B.Evans, R.Woodcock, C.Kemp
July 2013
CSIRO | MINERALS DOWN UNDER FLAGSHIP
Hardcopy of data
SSH Client
Maths PackageProprietary Package
Let’s map it out for Geophysics …
Transform to a regular grid
Crop data to area of interest
Reformat data for processing
Upload data to HPC
resource
Configure job and start
processing
Download results
Get handed field data Visualise data
Virtual Geophysics Laboratory v1.2 | Ryan Fraser3 |
The Past• Compile raw data using proprietary FORTRAN
• Also use other proprietary software • Transform to a regular grid using more software
• MATLAB, Intrepid, ER Mapper, ESRI ArcGIS, QGIS, etc• Crop data spatially
• eg: everything in Victoria• Transform data into a file format that can be read by proprietary
scientific code• This is usually done with some handwritten python or c• There is no version control, code is often rewritten / redone
• Upload data to HPC• Manually enter input parameters/start job• Where did my results go, how do I visualise?
Virtual Geophysics Laboratory v1.2 | Ryan Fraser4 |
There seems to be a problem…
• Reproducibility – there is none• What was the input of your model?• What transformations occurred?
• It’s a manual process• Time consuming• Error prone
• Expensive • Licensing costs• Effort/time
Virtual Geophysics Laboratory v1.2 | Ryan Fraser5 |
Scientific workflow – Virtual Geophysics Laboratory (VGL)• Automated - Simplify access and integration• Reproducible process• Open access (less expense)• Allow scientists to focus on the science and not the tech• Exploiting the Cloud and HPC• Operational or education tool
• Automates and massively expands geophysics computational capacity via the Cloud
Virtual Geophysics Laboratory v1.2 | Ryan Fraser6 |
VGL
• VGL - Virtual Geophysics Laboratory• NeCTAR funded activity• Collaboration with multiple partners (CSIRO, NCI, GA, UQ, Monash, ANU)• Supporting multiple workflows• New data types and collections• New Use – Not just exploration.
Done
Virtual Geophysics Laboratory v1.2 | Ryan Fraser7 |
Hardcopy of data
SSH Client
Maths PackageProprietary Package
Let’s map it out…
Transform to a regular grid
Crop data to area of interest
Reformat data for processing
Upload data to HPC resource
Configure job and start
processing
Download results
Get handed field data Visualise data
Virtual Geophysics Laboratory v1.2 | Ryan Fraser8 |
…to thisVirtual Geophysics Laboratory
Build “science” from existing
libraries
Run jobCollect and
publish results
Discover raw data
Select spatial bounds
Virtual Geophysics Laboratory v1.2 | Ryan Fraser9 |
Data discovery
11 |
Layers discovered via remote registries
Layers consist of numerous remote data services
Virtual Geophysics Laboratory v1.2 | Ryan Fraser
Data discovery
12 |
Some data services support subsetting
Some data services support reformattinge.g. CSV, NetCDF, GeoTIFF
Virtual Geophysics Laboratory v1.2 | Ryan Fraser
Data processing
14 |
A variety of different scientific codes are already available in the form of “Toolboxes”
Flexibility in what computing resources to utilise
Virtual Geophysics Laboratory v1.2 | Ryan Fraser
Data processing
15 |
The steps so far have been building an environment to run a processing script
...or build from existing templates
Either write your own...
Virtual Geophysics Laboratory v1.2 | Ryan Fraser
Managing results - provenance
Presentation title | Presenter name16 |
All of a job’s outputs are also accessible
Each job has a lifecycle that can be managed
16 |
A job’s console log can be inspected
Virtual Geophysics Laboratory v1.2 | Ryan Fraser
Managing results - provenance
17 |
Successful jobs can have their entire process captured in a ISO 19115 ‘provenance record’
Each provenance record tracks all inputs, outputs, processing scripts and other metadata.... Spatial bounds...
Input/output data...
Virtual Geophysics Laboratory v1.2 | Ryan Fraser
Powered by the Spatial Information Services StackCommon Platform
MarineEnvironment, Water
Groundwater GeologyGeophysics
Virtual Geophysics Laboratory v1.2 | Ryan Fraser18 |
VGL: Opportunities
• Free for research use• Supporting multiple workflows• Model Registry (3D) – researcher contributed models• New Scientific Codes – Underworld, eScript, UBC, Airborne EM inversion
codes + others (relatively “easy” to add)• Data sets from GA: National Airborne Geophysical DB including
– Gravity, Radiometric, AEM, Magnetics– Data sets from State Surveys
• Broad application and generic implementation• Scalable Infrastructure (Cloud)
Virtual Geophysics Laboratory v1.2 | Ryan Fraser19 |
What’s next?
• VGL sustainability strategy kick-ins • More Data and More Codes
• Extensions for new domains beyond the geosciences• Reuse of infrastructure by additional clients• Commercial connections• Roadshow for state government geologic surveys (funded by
AuScope)• Provenance Service – take learnings of VGL and research to create
a generic prov service
20 | Virtual Geophysics Laboratory v1.2 | Ryan Fraser
Thank you
Ryan FraserResearch Manager
t +61 8 6436 8760e [email protected] siss.auscope.org
VGL is Power by:
VGL acknowledges funding from the NeCTAR project http://www.nectar.org.au NeCTAR is an Australian Government project conducted as part of the Super Science initiative and financed by the Education Investment Fund.
VGL is a collaboration between:
VGL is supported by: University of Queensland, Monash University and the Australian National University