Stetl: Preparing Rich GML Data for deegree - The ETL Challenge
-
Upload
just-van-den-broecke -
Category
Technology
-
view
441 -
download
3
description
Transcript of Stetl: Preparing Rich GML Data for deegree - The ETL Challenge
![Page 1: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/1.jpg)
Preparing Your Rich GML Data for deegree
-
the ETL Challenge
Just van den Broeckedeegree Community Space 2012, Bonn
November 13, 2012www.justobjects.nl
![Page 2: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/2.jpg)
About MeIndependent Open Source Geospatial Professional
Trailblazer OSGeo Dutch Local Chapter
Just van den [email protected] www.justobjects.nl
![Page 3: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/3.jpg)
![Page 4: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/4.jpg)
![Page 5: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/5.jpg)
THE DUTCH KADASTER
GETS INSPIRED WITH
deegree day - nov 16, 2010
![Page 6: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/6.jpg)
THE DUTCH KADASTER
GETS INSPIRED WITH
deegree Community Space - nov 13, 2012
![Page 7: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/7.jpg)
![Page 8: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/8.jpg)
50+ DATASETS
WMS/WFS/WCS/ATOM
![Page 9: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/9.jpg)
METADATA
+
NL INSPIRE ACCESS POINT
![Page 10: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/10.jpg)
PDOK – Open Source & Open Standards
ApplicationsOGC/ISO/INSPIRE Web Services
Storage
Conversion
Services
sETLFME?
![Page 11: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/11.jpg)
OSGeo - Bolsena - 2010
![Page 12: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/12.jpg)
BOLSENA2012
![Page 13: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/13.jpg)
ALLES VORBEI ?
BOLSENA2012
![Page 14: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/14.jpg)
BOLSENA2012
![Page 15: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/15.jpg)
Preparing Your Rich GML Data for deegree
-
the ETL Challenge
Just van den Broeckedeegree Community Space 2012, Bonn
November 13, 2012www.justobjects.nl
![Page 16: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/16.jpg)
We have a Problem
![Page 17: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/17.jpg)
The Rich GML Problem
![Page 18: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/18.jpg)
Rich GML = Complex Mess
![Page 19: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/19.jpg)
INSPIREDutch National DSsAFIS-ALKIS-ATKIS
.
.
![Page 20: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/20.jpg)
![Page 21: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/21.jpg)
The Streetname!
![Page 22: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/22.jpg)
Complex Model
Transformations
![Page 23: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/23.jpg)
Millionsof
Objects
![Page 24: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/24.jpg)
10s of Millionsof
<Elements>
![Page 25: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/25.jpg)
MultipleTransformation
Steps
![Page 26: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/26.jpg)
Solution is Spatial ETL
![Page 27: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/27.jpg)
A.K.A.
![Page 28: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/28.jpg)
Thank You for your
Attention!
![Page 29: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/29.jpg)
But what about.......FOSS ? ... Stetl?
![Page 30: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/30.jpg)
FOSS ETL - High Level
![Page 31: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/31.jpg)
FOSS ETL - Lower Level
But Each Powerful by Itself
ogr2ogr
![Page 32: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/32.jpg)
FOSS ETL - DIY ? (No!)
![Page 33: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/33.jpg)
FOSS ETL - How to Combine?
=+ + ?ogr2ogr
![Page 34: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/34.jpg)
FOSS ETL - Add Python to Equation
=+ + ?( )ogr2ogr
![Page 35: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/35.jpg)
=+ +
Stetl
( )ogr2ogr
![Page 36: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/36.jpg)
Stetl=
SimpleStreaming
SpatialSpeedy
ETL
![Page 37: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/37.jpg)
Process Chain
Input Filter Outputgml
Filter
Stetl concepts
![Page 38: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/38.jpg)
Speed: Streaming
Input Filter Outputgml
Stetl concepts
![Page 39: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/39.jpg)
Speed: Going Native
Input Filter Outputgml
ogr2ogr sETLsETL
Native C Libs/Progs
Calls
Stetl concepts
![Page 40: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/40.jpg)
Example: GML to PostGIS
Reader XMLSplitter ogr2ogr
gml
Stetl concepts
![Page 41: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/41.jpg)
Example: INSPIRE Model Transform
ogr2ogr XSLT Writergml
Stetl concepts
![Page 42: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/42.jpg)
Example: deegree Store
ogr2ogr XSLT deegreeWriter
Stetl concepts
![Page 43: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/43.jpg)
Process Chain - How?
Input Filters Output
Stetl concepts
![Page 44: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/44.jpg)
Example: XML to Shape
The Source
![Page 45: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/45.jpg)
Example: XML to Shape
First: XSLT Transform to GML
![Page 46: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/46.jpg)
Example: XML to Shape
XMLInput XSLT ogr2ogr
![Page 47: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/47.jpg)
Example: XML to Shape
The SETL Chain Config File
ProcessChain
Reader
XSLT
ogr2ogr
![Page 48: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/48.jpg)
Example Components
Input Filters Output
Stetl concepts
XMLFile XSLT GMLFile
ogr2gml GMLSplitter gml2ogr
LineStream XMLValidator WFS-T
deegree* FeatureExtractor deegree*
YourInput YourFilter YourOutput
![Page 49: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/49.jpg)
Data Structures
Stetl concepts
✴Components exchange Packet✴Packet contains data✴Data format:
xml_line_stream etree_docetree_feature_arrayxml_doc_as_stringany
![Page 50: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/50.jpg)
deegree Integration
Stetl concepts
✴Input DeegreeBlobstoreInput✴Output DeegreeBlobstoreInput DeegreeFSLoaderOutput WFSTOutput
![Page 51: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/51.jpg)
Cases✴INSPIRE Download Services publish to deegree store (WFS) GML files (for Atom Feed)
✴National GML Datasets GML to PostGIS
![Page 52: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/52.jpg)
Case: Dutch Addresses
Source<GML>
sETL sETLdeegree
WFS
sETL INSPIRE<GML>
AtomFeed
Other
INSPIREAddresses
DutchAddresses+
Buildings
DutchGeocoder
deegreeblobstore
![Page 53: Stetl: Preparing Rich GML Data for deegree - The ETL Challenge](https://reader033.fdocuments.us/reader033/viewer/2022042814/555077d2b4c905cc0f8b51c2/html5/thumbnails/53.jpg)
Thank You !
stetl.orgwww.inspire-foss.org