Better Data Analytics, Better Processes for Consumer Products
The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for...
Transcript of The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for...
![Page 1: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/1.jpg)
IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th
The new IAGOS data portal
Damien Boulanger
CNRS – Observatoire Midi-Pyrénées
IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th
![Page 2: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/2.jpg)
IAGOS data workflow
IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th
![Page 3: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/3.jpg)
New database
IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th
• New software (MongoDB)• Better performances
• Better for heterogeneous data management
• Simplified database management (backup, etc.)
• Data processing• A lot of processes : validation, formatting, merging, added-value products
• Use of the tool Apache Camel (rule-based routing and mediation engine)
• Better management of the data flow
• Content• IAGOS-MOZAIC and IAGOS-CORE data
• IAGOS-CARIBIC (Aerosols and cloud data on going)
• BCP data on going
• Flags and uncertainties (on going for CARIBIC)
• Added-value products: PV, tropopause height, CO representation error, etc.
![Page 4: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/4.jpg)
Live demo
IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th
http://iagos.sedoo.fr
![Page 5: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/5.jpg)
Future improvements
IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th
• Data download• Save user requests
• Generation of a single file containing only the observations matching the request
• Download any levels of data for the IAGOS team
• Web service for scripts
• Data plotting• Plot two parameters at a time for comparison
• Tools for the PIs• Manage instruments
• Manage comparisons
• Data• New parameters : Boundary layer height, potential temperature, etc.
• Metadata• Available through AERIS portal, GAW and ENVRI+ metadata portals
• Implementation of DOI (common work with AERIS)
![Page 6: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/6.jpg)
IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th
Released next week.
http://www.iagos-data.fr
Any comments are welcome!
![Page 7: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/7.jpg)
IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th
Screenshots
![Page 8: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/8.jpg)
IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th
Home
![Page 9: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/9.jpg)
![Page 10: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/10.jpg)
![Page 11: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/11.jpg)
IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th
Registration
![Page 12: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/12.jpg)
![Page 13: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/13.jpg)
![Page 14: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/14.jpg)
![Page 15: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/15.jpg)
IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th
Data status
![Page 16: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/16.jpg)
![Page 17: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/17.jpg)
![Page 18: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/18.jpg)
![Page 19: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/19.jpg)
![Page 20: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/20.jpg)
![Page 21: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/21.jpg)
![Page 22: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/22.jpg)
![Page 23: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/23.jpg)
IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th
Metadata
![Page 24: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/24.jpg)
![Page 25: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/25.jpg)
![Page 26: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/26.jpg)
![Page 27: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/27.jpg)
![Page 28: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/28.jpg)
![Page 29: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/29.jpg)
![Page 30: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/30.jpg)
![Page 31: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/31.jpg)
![Page 32: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/32.jpg)
![Page 33: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/33.jpg)
![Page 34: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/34.jpg)
![Page 35: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/35.jpg)
![Page 36: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/36.jpg)
![Page 37: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/37.jpg)
IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th
Search and download
![Page 38: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/38.jpg)
![Page 39: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/39.jpg)
![Page 40: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/40.jpg)
![Page 41: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/41.jpg)
![Page 42: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/42.jpg)
![Page 43: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/43.jpg)
![Page 44: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/44.jpg)
![Page 45: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/45.jpg)
![Page 46: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/46.jpg)
IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th
Administration
![Page 47: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/47.jpg)
![Page 48: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/48.jpg)
![Page 49: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/49.jpg)
![Page 50: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/50.jpg)
![Page 51: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/51.jpg)
![Page 52: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/52.jpg)
![Page 53: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/53.jpg)
![Page 54: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f874193c9157c1342498ecc/html5/thumbnails/54.jpg)