Discussion: Coverage data and WCS...
Transcript of Discussion: Coverage data and WCS...
www.jrc.ec.europa.eu
Serving societyStimulating innovationSupporting legislation
Discussion:Coverage data and WCS implementation
INSPIRE Thematic Cluster onElevation, Orthoimagery, Reference systems and Geographical grids
Peter Baumann, Dominique Laurent, Jordi Escriu
Topics / Issues selected for discussion• Issue 1
How to handle the huge volume of coverage data
• Issue 2How to implement the concept of coverage aggregation
• Issue 3How to implement the concept of mosaicking
• Issue 4INSPIRE Coverage extensions vs. OGC standard coverages
• Issue 5Purpose of the metadata hook
Issue 1 - How to handle the huge volume of coverage data
Description
• It is generally difficult to transfer huge volume of coverage data through network services and data producers must analyse how to handle it.
Issue 1 - How to handle the huge volume of coverage data
Server side:big data
Client side: user requests
(various sizes)
(small) Internet pipe
Issue 1 - How to handle the huge volume of coverage data
Server side:big data
Client side: user requests
(various sizes)
(small) Internet pipe
???
Issue 1 - How to handle the huge volume of coverage data
Choice of download services
predefined data sets WCS
Make small coverages (tile,
small AU)
Other solution at encoding
level?
Server may handle big data and « cut »it
Client side?
Issue 1 - How to deal with huge volume of coverage data
Open questions
Case 1 - Delivery through Predefined data sets• Experiences in making “small” coverages (at conceptual
level)? Issues?• Alternative approaches at encoding level?
Case 2 - Delivery through WCS• Feed-back from experience?• Are there any issues on client side?
Issue 1 - How to deal with huge volume of coverage data
Case 1 - Delivery through Predefined data sets• Most data providers consider each tile / map sheet as
a coverage - general trend among NMCAs.• The coverage over a certain territory is split in several
pieces, for both organizational and efficiency purposes. • To be implemented using the concept of tiling. • Need for guidelines to select: The most appropriate tiling approach.
Maximum size for each tile.
• Tiling approach may have a direct impact on efficiency of delivery.
Issue 1 - How to deal with huge volume of coverage data
Case 1 - Delivery through Predefined data sets
• When each tile is considered as a coverage The coverage identifier depends on the tiling system and
so on the chosen CRS. Is it an issue?
What should be a data set? A unique coverage/tile or a set of coverages/tiles?
If a data set is composed of one coverage/tile, it implies a lot of data sets. Might be difficult to be handled both by producer (many metadata to be produced) and by user (how to find relevant tiles? => is there need for user friendly-tools?
Issue 1 - How to deal with huge volume of coverage data
Case 2 - Delivery through WCS• A coverage is the natural response of a WCS to a data
request (GetCoverage operation).• However, some Efficiency issues has been identified
when using such kind of services: WCS implementations build the final output of the
GetCoverage request first totally in the RAM of the server before sending it out to the client, e.g. MapServer / GeoServer [Jukka Rahkonen].
• How to solve this efficiency issues?
Issue 1 - How to deal with huge volume of coverage data• Big Data = “ship code to data”, extract what you want Who wants images > RAM?
• Coverage image For user convenience, pack many images in one coverage Tentatively no tiling prescribed, WCS is interface spec Many tilings possible for different use cases
– cf PhD thesis Paula Furtado WCS implementers can exercise ingenuity in That said, CIS 1.1 supports flexible tiling
– Space, time, range, …– Can exploit data format tiling (TIFF, NetCDF, …)
• Predefined data sets: no such restriction in WCS Any request, anytime – with tools existing today!
• Proven implementation evidence of WCS scalability WCS/WCPS : 145 TB space/time datacubes @ ESA (rasdaman)
WCS Core GetCoverage• Download a coverage (or a subset thereof), values guaranteed unaltered• Ex: „download coverage c001“
http://www.acme.com/wcs ? SERVICE=WCS & VERSION=2.0& REQUEST=GetCoverage & COVERAGEID=c001
http://standards.rasdaman.com
http://www.acme.com/wcs ? SERVICE=WCS & VERSION=2.0& REQUEST=GetCoverage & COVERAGEID=c001& SUBSET=Long(100,120) & SUBSET=Lat(50,60)& SUBSET=time("2009-11-06T23:20:52")
Ex: „coverage c001, lat/long cutout, time slice t=2009-11-06T23:20:52”
http://www.acme.com/wcs ? SERVICE=WCS & VERSION=2.0& REQUEST=GetCoverage & COVERAGEID=c001 & FORMAT=“image/tiff“
Ex: “coverage c001, in GeoTIFF”
Issue 2 - How to implement the concept of coverage aggregationDescription• The concept of tiling is introduced in the INSPIRE
TG on Elevation and Orthoimagery. • Tiles describe logical structures (e.g. map sheets,
administrative units like regions or districts, etc.).• Each tile is implemented as an individual Coverage.• This splitting approach is helpful to handle raster
data in manageable units.• Coverages can be aggregated (associated) to
indicate all of them cover certain areas of territory.
Issue 2 - How to implement the concept of coverage aggregation
Issue 2 - How to implement the concept of coverage aggregation
A number of Tiles
Their aggregation and corresponding BBOX
Issue 2 - How to implement the concept of coverage aggregation
Implementation of each Tile
Aggregation of a number of Tiles
covering a certain area
Issue 2 - How to implement the concept of coverage aggregation
Contributing footprints corresponding to each Orthoimage coverage / Tile which participates in the aggregation
The aggregated orthoimage coverage (blue footprint)
Issue 2 - How to implement the concept of coverage aggregationOpen questions• Is it clear how to implement coverage / tile
aggregations in GMLCOV?
• Might be probably different ways in practice!
• TC #3 discussion thread:https://themes.jrc.ec.europa.eu/discussion/view/50412/how-to-implement-tiling-model-mosaic-elements-coverages-and-coverage-aggregations-in-gmlcov-files
• Are coverage aggregations really needed?
Issue 2 - How to implement the concept of coverage aggregation
• Why are coverage aggregations really needed? Requested only by ESA early in TG OI [Markus Seifert] Doubt usefulness for standard agency cases
• Adds complexity [Dominique Laurent, Peter Baumann] For users: what you get what is advertised For service ops: tedious to keep consistent
• Recommendation optional advanced feature Simple core coverage model
– Coverage = seamless mosaic (many files!)– Coverage = image timeseries– Coverage = weather datacube– Etc.
Inset: WaterML 2.0 Time Handling• WaterML 2.0: timeseries = time slices layout hardwired
• OGC Coverages: time just another axis Implementation can choose efficient layout
Tiling• Goal: faster tile loading by adapting storage units to access patterns• Approach: partition n-D array into n-D partitions („tiles“)• Tiling classification based on degree of alignment [ICDE 1999]
chunking [Sarawagi, Stonebraker, DeWitt, ... ]
Tiling in rasdaman• Any partitioning („tiling“)
145+ TB datacubes[IJDE 2015]
• Why irregular tiling?
[OpenStreetMap]
Tiling in rasdaman
• tiling strategies as service tuning [Furtado]:regular directional ... area of interest
• rasdaman storage layout languageinsert into MyCollection values ...tiling area of interest [0:20,0:40], [45:80,80:85]tile size 1000000index d_index storage array compression zlib
Issue 3 - How to implement the concept of mosaickingDescription
• The concept of mosaicking is introduced in the INSPIRE data specification on Orthoimagery.
• Mosaic elements are intended to identify the contributing area and the acquisition time of one or several input images used to generate an orthoimage coverage, i.e. precisely describe the temporal characteristics of a mosaicked product.
• Their modelling corresponds to a thematic need. • Not clear how to implement mosaicking in
GMLCOV.• Might be probably different ways in practice.
Issue 3 - How to implement the concept of mosaicking
Issue 3 - How to implement the concept of mosaicking
Issue 3 - How to implement the concept of mosaicking
Use of‘SingleMosaicElement’
objects
Use of ‘AggregatedMosaicElement’
objects
Issue 3 - How to implement the concept of mosaicking
Open questions• Again! - Is it clear how to implement mosaic
elements in GMLCOV?
• Might be probably different ways in practice!
• Are there any other ways to add mosaic information to each individual coverage, without providing ‘MosaicElement’ objects?
Issue 3 - How to implement the concept of mosaicking
• No mosaicking in Google Maps! ;-)• Tiling vs mosaicking? As before: complex concept!
• Internal implementation detail,should not be visible to users Similar to tiling WaterML vs Coverage
• Good tools will do it automatically Should not be constrained
Issue 4 – INSPIRE Coverage extensions vs. OGC standard coverages
• INSPIRE models based on coverages partially extends the standard way of modelling and implementing OGC coverages.
• Such INSPIRE extensions corresponds to thematic needs identified by TWGs.
• Danger: They are ignored by WCS interface.• Agreement: Minimize INSPIRE extensions as
possible - Need to examine them case-by-case. • Worth to align this task with CIS v1.1, if this new
standard is adopted by INSPIRE in the future.
Issue 4 – INSPIRE Coverage extensions vs. OGC standard coverages
INSPIRE Coverage extensions (Elevation)
Issue 4 – INSPIRE Coverage extensions vs. OGC standard coverages
INSPIRE Coverage extensions (Orthoimagery)
Issue 4 – INSPIRE Coverage extensions vs. OGC standard coverages• EXAMPLE - Identification of coverage extent –
There are 2 attributes to deal with it: boundedBy (GML): Optional / Approximate / Does not
allow identification of discontinuous spatial extents.
domainExtent (INSPIRE extension): It supports the identification of discontinuous spatial extents (needed e.g. when coverage aggregations are identified) / Not recognized by WCS.
• Largely discussed in TC #3: https://themes.jrc.ec.europa.eu/discussion/view/12901/domainextent-vs-gmlboundedby-el-oi-coverages-encoding
Issue 4 – INSPIRE Coverage extensions vs. OGC standard coveragesOpen questions• Are INSPIRE extensions really needed?
• For each of them, is there any equivalent element foreseen in the implementations standards? e.g. in GMLCOV, CIS v1.1?
• Is there specific GMLCOV component to implement additional items not covered by OGC standards?
• If an INSPIRE extension which is not considered / foreseen in the implementation standards proves to be useful, could it be considered in a future update of such standards?
Issue 4 – INSPIRE Coverage extensions vs. OGC standard coverages• Problem 1: INSPIRE coverages have strange
definitions domainSet: <any>; rangeType: <any>
• Problem 2: INSPIRE coverages add in wrong place <metadata> slot foreseen for this purpose
• Suggestions: Reinspect each item Use metadata for extensions Make voidable (allow vanilla WCS to serve INSPIRE) INSPIRE to contribute to further OGC CIS/WCS work
Issue 5 - Purpose of the metadata hook
• Coverages may have metadata at different levels: Data set level (xml metadata file of the data set) –
For discovery purposes (CSW).
Image file (header) – Required by specific format.
Coverage level (optional metadata hook, as proposed in GMLCOV).
• Guidelines about the purpose and content of the metadata hook are needed.
Issue 5 - Purpose of the metadata hook
Metadata hook
Issue 5 - Purpose of the metadata hook
Open questions
• Which metadata elements are expected in the metadata hook?
• Why? This content is expected to cover any WCS functionality?
• Is there any need to include in the metadata hook specific metadata elements?
Issue 5 - Purpose of the metadata hook
• …to transport any additional “meta”data Coverage metadata = CoverageDescription
Discovery (catalog): <metadata>
• Image “header” ~ covg descr + <metadata>• Example: CIS 1.1 00_metadata.gml
• Recommendation: own schema per unit ISO 19115, EO Metadata Profile, … Can coexist