Daily Oral Language XVIII Set 086 Set 087 Set 088 Set 089 Set 090.
Model Database Interface - GFDL's Data Portal · The template script contains variables to be...
Transcript of Model Database Interface - GFDL's Data Portal · The template script contains variables to be...
![Page 1: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/1.jpg)
Model Database Interface (MDBI)
Aparna Radhakrishnan, Kyle Olivo, Serguei Nikonov, V Balaji
&Applying User-developed Climate
Analytics as a web-service
GFDL Summer School 2012, Princeton NJ
GFDL SUMMER SCHOOL 2012GFDL Summer School 2012, MDBI and
Curator1
![Page 2: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/2.jpg)
1. Model Database Interface (MDBI) –a) what is it? Where is it?b) Learning to navigate to different experiments and lookup for information on GFDL experiments from the MDBI.(Some features will be demonstrated)
1.1. Class work/Exercise -1a) Try a couple of features of MDBI in class
2. Applying user-developed climate analytics on CMIP5 datasets a) What is it?
2.1. Class work /Exercise-2a) Apply time-averaged spectrum climate analysis on a pool of
CMIP5 datasets
•
GOALS FOR THE SESSION
GFDL Summer School 2012, MDBI and Curator
2
![Page 3: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/3.jpg)
3May 2011 GO-ESSP Workshop - Asheville, NC
Facebook DB schema ?
GFDL Summer School 2012, MDBI and Curator
![Page 4: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/4.jpg)
CURATOR MODEL DATABASE
GFDL experiments
metadata
GFDL climateField metadata
CMIP5/ACCMIPVariable
mappings
Etc etc..GFDL Summer School 2012, MDBI and
Curator4
![Page 5: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/5.jpg)
MDBI LAYOUT
Left panel – Model/Experiment Tree
Right panel – Content Display
Menu bar
GFDL Summer School 2012, MDBI and Curator
5
![Page 6: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/6.jpg)
DEMO WITH USE-CASES
Where is this MDBI??!
http://cobweb.gfdl.noaa.gov:8080/extmdbCW(Open this url in your Firefox web browser)
GFDL Summer School 2012, MDBI and Curator
6
![Page 7: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/7.jpg)
DEMO WITH USE-CASES CONTD..
Can I get a brief description of any experiment
On MDBI?
GFDL Summer School 2012, MDBI and Curator
7
Yes, please select an experiment and Click on the Description tab under the
Experiment Info tab
![Page 8: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/8.jpg)
DEMO WITH USE-CASES. CONTD..
•.
When and by whom was this Experiment checked into the
Database?
Please select the experiment you’re Interested in and go to
Experiment Info tab at the top-level, andAdministration tab at the bottom-level.
GFDL Summer School 2012, MDBI and Curator
8
![Page 9: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/9.jpg)
DEMO WITH USE-CASES. CONTD..
What about the Runtime Settings For an experiment?
Can I find them on MDBI?
GFDL Summer School 2012, MDBI and Curator
9
Sure, please select an experiment and click on the tab
“Run Descriptions” -to get the runtime settings defined for a
particular experiment.
![Page 10: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/10.jpg)
DEMO WITH USE-CASES. CONTD..
How can I find the Coupler Diag andNML-coupler info?
06/27/12 GFDL Summer School 2012, MDBI and Curator
107/19/2012
Please select an experiment and Click on the Component Configuration
tab underExperiment Info
![Page 11: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/11.jpg)
DEMO WITH USE-CASES. CONTD..
How can I find the INPUT FILES-Diag tables/
Datafile/namelistsdefined for this experiment?
06/27/12 GFDL Summer School 2012, MDBI and Curator
117/19/2012
Please select an experiment and Click on the INPUT FILES tab under
Experiment Info
![Page 12: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/12.jpg)
DEMO WITH USE-CASES. CONTD..
•.
What variables were defined in thePostprocess section of the XML?
Please select an experiment and Click on the “Post Processing” tab under
Experiment Info tab.
GFDL Summer School 2012, MDBI and Curator
12
![Page 13: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/13.jpg)
DEMO WITH USE-CASES. CONTD..
I think I need to look at the XML for this Experiment. Where is it?
GFDL Summer School 2012, MDBI and Curator
13
Please click on “Generate XML” tab on MDBI and then
choose the experiment that you'reInterested in.
Clicking on “View Original XML”Will open up the XML in a new window.
![Page 14: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/14.jpg)
DEMO WITH USE-CASES. CONTD..
Where can I find the procedures toCheckout an experiment's code?
06/27/12 GFDL Summer School 2012, MDBI and Curator
147/19/2012
Please select an experiment and click on the tab
“Checkout procedures” -Under experiment info tab
![Page 15: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/15.jpg)
DEMO WITH USE-CASES
I ran frepp yesterday on experiment A.Is there a way to tell if all the postprocess
Output files are available already?
Yes. Please click on the top-level
Tab named “Monitoring” , and then
“PP monitoring” in the lower level tab.
Now, select the experiment you'd like to monitor.
GFDL Summer School 2012, MDBI and Curator
15
![Page 16: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/16.jpg)
DEMO WITH USE-CASES (last use-case!)
Ok.. where can I find the files/figuresthat
I post-processed?
Please select an experiment and
Click on the “PP Directory” tab under
Experiment Info tab.
06/27/12 GFDL Summer School 2012, MDBI and Curator
167/19/2012
![Page 17: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/17.jpg)
End of Part-1
GFDL Summer School 2012, Princeton NJ
GFDL SUMMER SCHOOL 2012GFDL Summer School 2012, MDBI and
Curator17
![Page 18: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/18.jpg)
Applying User-developed ClimateAnalytics as a web-service
Aparna Radhakrishnan1, Balaji Venkataramani2
GFDL Summer School 2012, Princeton NJ
GFDL SUMMER SCHOOL 2012GFDL Summer School 2012, MDBI and
Curator18
![Page 19: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/19.jpg)
7/19/2012
OPeNDAPaggregation
UI
AnalysisOutput:Postscript,netCDF,etc
LAS(Live Access Server)
Ferret,Python ..
ESGF(Earth System Grid Federaion)
CMIP5 Datasets
Request
Response
Browser
Back-end- User Analysis Plugin
LAS-User-analysis Webservice Architecture
Input Templater
![Page 20: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/20.jpg)
Templater Script – (from yesterday’s frepp talk)
The template script contains variables to be filled by frepp, ie:set in_data_dir #pp directory containing files to be analyzedset in_data_file #list of all filenames to be analyzedset descriptor #experiment nameset out_dir #directory to write output filesset yr1 #first year to analyzeset yr2 #last year to analyze
GFDL Summer School 2012, MDBI and Curator
20
![Page 21: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/21.jpg)
7/19/2012
OPeNDAPaggregation
UI
AnalysisOutput:Postscript,netCDF,etc
LAS(Live Access Server)
Ferret,Python ..
ESGF(Earth System Grid Federaion)
CMIP5 Datasets
Request
Response
Browser
Back-end- User Analysis Plugin
LAS-User-analysis Webservice Architecture
Input Templater
![Page 22: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/22.jpg)
OPeNDAP aggregations
• OPeNDAP provides software which makes local data accessible to remote locations regardless of local storage format.
• The data provider wants to group together a collection of DAP URLs so that they can also function as a single URL. This will enable data sets made up of many files to appear as a single entity.
Ref: www.opendap.org
GFDL Summer School 2012, MDBI and Curator
22
![Page 23: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/23.jpg)
7/19/2012
yes? sh dacurrently SET data sets:1>
http://esgdata.gfdl.noaa.gov/thredds/dodsC/cmip5.output1.NOAA-GFDL.GFDL-ESM2M.esmrcp85.mon.atmos.Amon.r1i1p1.ts.20111228.aggregation.1 name title I J K LTS Surface Temperature 1:144 1:90 ... 1:1140
OPeNDAP aggregations (contd..)
![Page 24: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/24.jpg)
7/19/2012
![Page 25: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/25.jpg)
7/19/2012
OPeNDAPaggregation
UI
AnalysisOutput:Postscript,netCDF,etc
LAS(Live Access Server)
Ferret,Python ..
ESGF
CMIP5 Datasets
Request
Response
Browser
Back-end- User Analysis Plugin
LAS-User-analysis Webservice Architecture
Input Templater
![Page 26: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/26.jpg)
7/19/2012
![Page 27: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/27.jpg)
GFDL Summer School 2012, Princeton NJGFDL Summer School 2012, MDBI and
Curator27
What did we do just now? –“BRING ANALYSIS TO DATA”
A TEMPLATE-DRIVEN APPROACH WHERE USER-DEVELOPED ANALYSES IS APPLIED TO PUBLICLY AVAILABLE CMIP5 DATASETS USING LIVE ACCESS SERVERS AND OPeNDAP AGGREGATION URLS, RIGHT FROM YOUR WEB BROWSER.
Acknowledgement: This work was partly funded by the international ExArchproject under the G8 initiative by National Science Foundation Award 1119308. Many thanks to: V Balaji (Princeton University/GFDL), ExArch, Andrew Wittenberg (GFDL), Roland Schweitzer (Weather Top Consulting/PMEL), Hans Vahlenkamp (UCAR/GFDL).
![Page 28: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/28.jpg)
Input: Get dataset Identifiers (D1,D2) for the experimentsE1,E2 in comparison .(This includes model-name, experiment, ensemble_member,frequency, realm, CMIP table)Eg: NOAA-GFDL.GFDL-ESM2G.historical.mon.atmos.Amon.r1i1p1
Input: Get start_time (t0E1,t0E2) and end_time (t1E1,t1E2) for experiments E1, E2 in comparison – as input
Input: Get CMIP5 variable name (V) to be analyzed
Input: Get climate analytics plot type to be applied to datasets (D1,D2)
More background info:
1GFDL Summer School 2012, MDBI and
Curator28
![Page 29: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/29.jpg)
THREDDSCATALOG
FEEDER (python-based)
1. Crawls through ESGF Root THREDDS catalogs and locates datasets D1,D2. 2. Fetches the OPeNDAP aggregation URL for variable “V “ in datasets D1,D2.3. Prepares arguments to be passed to the analysis script templates along with the start_time(s) (t0E1,t0E2) and end_time(s) (t1E1,t1E2).4. Runs the analysis scripts (any language. Currently, tested with Ferret) server-side.5. Sends analysis products back to Thredds Catalog feeder.6. Throws exceptions if the timer ranges are not available for specified experiments or if specified variables are not part of a given experiment.
1
2
GFDL Summer School 2012, MDBI and Curator
29
![Page 30: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/30.jpg)
NetCDF
files
Analysis Products2
GFDL Summer School 2012, MDBI and Curator
30
![Page 31: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/31.jpg)
DEMO AND CLASSWORK Exercise -2
Please open this page for –http://cobweb.gfdl.noaa.gov/~a1r
GFDL Summer School 2012, MDBI and Curator
31
![Page 32: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/32.jpg)
7/19/2012
![Page 33: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/33.jpg)
Useful References
GFDL Summer School 2012, MDBI and Curator
33
External links-
1. CMIP5 Controlled Vocabulary
http://cmip-pcmdi.llnl.gov/cmip5/output_req.html
2. IPCC AR5 - List of available variables
http://data1.gfdl.noaa.gov/nomads/forms/ipcc_var/
![Page 34: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/34.jpg)
BACKUP SLIDESLAS-CLIMATE ANALYSIS
GFDL Summer School 2012, MDBI and Curator
34
![Page 35: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/35.jpg)
GFDL Summer School 2012, MDBI and Curator
35
![Page 36: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/36.jpg)
GFDL Summer School 2012, MDBI and Curator
36
![Page 37: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/37.jpg)
GFDL Summer School 2012, MDBI and Curator
37
![Page 38: Model Database Interface - GFDL's Data Portal · The template script contains variables to be filled by frepp, ie: set in_data_dir #pp directory containing files to be analyzed set](https://reader036.fdocuments.us/reader036/viewer/2022090605/6059bea203a23021b93a2b2f/html5/thumbnails/38.jpg)
06/27/12 GFDL Summer School 2012, MDBI and Curator
38