State ofGeoServer 2.12

Andrea AimeTechnical Leadandrea.aime@geo-solutions.it@geowolf

GeoSolutionsInnovative, robust and cost-effective solutions leveraging best-of-breed Open Source products.

WelcomeOpen Source ProjectsGeoToolsGeoServerImageIO-ExtJAI-ExtJAI Tools

OSGeo FoundationCharter MemberGeoServer PSCGeoTools PSC

Jody GarnettCommunity Leadjgarnett@boundlessgeo.com@jodygarnett

BoundlessProvides geospatial tools and services for managing data and building applications.

WelcomeOpen Source ProjectsGeoToolsGeoServeruDig

OSGeo FoundationBoard MemberOSGeo Incubation ChairGeoTools Project Officer

Eclipse FoundationLocationTech Steering CommitteeLocationTech Technology Project

Open Source ProjectsGeoToolsGeoServerGeoWebCache

OSGeo FoundationGeoServer PSCGeoWebCache Lead

Kevin SmithSoftware Engineerksmith@boundlessgeo.com smithkm

BoundlessProvides geospatial tools and services for managing data and building applications.

Welcome

GeoServer 2017

GeoServer at a GlanceJava Web Application to share and edit geospatial data.

Publish data from any major spatial data source using open standards.

Core Protocols

WMS – mapsWFS – vector WFS-T – editing WCS – coverageWPS* – processCSW* – search

Tile Protocols

WMTS – tilesTMS – tilesWMS-C – tiles

28 → 37 members of team geoserver (to be updated624 → 619 No clue this year (check with Kevin)500+ → 525 pull requestsNew contributors being added(others stepping back)Smaller base of active contributors this year.Pull requests still going strong, very active codebase.Simone replaces Andrea as OSGeo Officer.13 releases this year

GeoServer Project StatusHow about team user-list?

2005 → 1400 (SF now checking on people not active on list)

Also geoserver-italia, espanol and opengeo.cn mailing lists

Release Schedule

stable maintenance

July 2.12.5

June

May 2.12.4

April

March 2.12.3

February 2.11.5

January 2.12.2

December 2.11.4

November 2.12.1

October 2.11.3

September 2.12.0

August 2.10.5 2.11.2

User List Participation● Answering users questions relies on a low number of people

Action:

● We are sending reminder on effective use of the user list on subscription and at the bottom of each mail

● Community building activity - remind developers about the definition of “supported”(not only reviewing pull requests, also answering user questions, used to be obvious, but it’s not obvious to devs hired into open source)

● Bug stomp introduced in 2016○ Relying on volunteer time not working out○ allowed developers to reserve time

● Moved to fixed schedule in 2017○ Arranging day each month too random

● Last Friday of each month○ Check blog post for tips on participation

Standing “last friday of the month” bug stomp

Held “sprint” to update from Restlet to Spring MVC for our REST configuration API.

Thanks to GeoSolutions for hosting, to Boundless and Astun for sending developers, and to all the sponsors who helped fund this important and badly needed, but boring work.

REST API Migration

2.12Viareggio Code SprintCommunity

REST API Migration

● Migration complete● REST API documented

○ Available online now!

2.12Viareggio Code SprintCommunity

Vector data sources

GeoPackage:

● Provides an sqlite database containing vector or raster data○ Initial idea from spatialite project○ Now an OGC Standard to replace shapefile and personal file geodatabase

● GeoServer module○ GeoPackage is a supported plugin in GeoTools○ We can now promote it to a GeoPackage extension ○ Needs some work, splitting into 4 GeoServer packages, store, wfs output format, wms output

format, wps output format○ Bug fix - raster mosaic geopackage had y-axis inverted.

GeoPackage (community)

2.12GeoSolutionsBoundless

CommunityBoundless Suite

● GeoServer importer supports GeoJSON files○ Allows GeoJSON to be imported into PostGIS database and published in one step○ Based on unsupported module in GeoTools

● Wish to package as a community module for GeoServer○ Allow GeoServer to directly publish GeoJSON files○ Text based, non indexed format, only suitable for small files

GeoJSON store

2.12Ian TurtonCommunity

● GeoServer importer supports CSV files○ Allows CSV to be imported into PostGIS database and published in one step○ Based on unsupported module in GeoTools (used as a tutorial for programmers)

● Package as a community module for GeoServer○ Allow GeoServer to directly publish CSV files○ Text based, non indexed format, only suitable for small files

CSV store

2.12BoundlessCommunity

CSV input/output for WPS● Allows the input and output of CSV for use in WPS

processes● This will lead to GeoTools CSV Datastore being

upgraded to community (and hopefully to supported)● Come see more at “Building a Table Joining Like

Service with Web Processing Services” Friday Afternoon

2.12Ian TurtonAstunRnD

Raster data andprocessing

● Initial version in 2.11● Several fixes in 2.12, e.g.

accuracy, performance, dateline crossing handling

● Being tested against global mosaics of Sentinel2 (each image in a different UTM zone), more improvements/fixes incoming

Heterogeneous CRS image mosaics

2.11/2.12Boundless/GeoSolutionsRnD/DLR

UTM60 image footprints UTM1 image footprints

Dateline wrapped image mosaic

● Reprojected images could have shown border effects

Better handling of reprojection boundaries

2.10/2.11GeoSolutionsCommunity

● Higher order interpolation did not play well with nodata, causing border effects:

○ Bilinear○ Bicubic

● Requires jai-ext enabled

Improved no-data handling in interpolation

2.10/2.11GeoSolutionsCommunity

Before

After

● Respect scale/offset attributes● Improved support for related dimensions (e.g. time/runtime in forecast),

automatically finding the best related value when just one dimension is provided

NetCDF reader improvements

2.11Niels/Devon/Ben/DanieleScitus/Boundless/Transient/GeoSolutionsCustomer

Runtime

S3 GeoTIFF Community ModuleSupport for GeoTiffs hosted on Amazon S3

● Reader based off of the GeoTiff reader.● Very basic caching of images from S3 based off of EhCache● InputStream implementation from JAI for reading imagery from S3

2.12DevonBoundlessBoundless

s3://landsat-pds/L8/001/002/LC80010022016230LGN00/LC80010022016230LGN00_B1.TIF?useAnon=true&awsRegion=US_WEST_2

Styling

Style Page

2.10BoundlessCommunity

Refactored CSS Style Page into the main application.

Works withSLD, CSS, YSLD, Mapbox

Improved experience for everybody!

New features for CSS (2.10)● Rendering transformations supported (feature parity with SLD!)● Rule nesting, helps with selector duplication and makes styles more readable

/* @title Levels */* { transform: ras:Contour(levels: 1100 1200 1300 1400 1500 1600 1700); stroke: black; z-index: 0;}

/* @title Values */[@scale < 3000] { transform: ras:RasterAsPointCollection(); mark: symbol('square'); :mark { size: 2; fill: black; } label: [GRAY_INDEX]; label-anchor: 0 0.5; label-offset: 5 0; font-family: Arial; font-fill: black; z-index: 1;}

Extract contours at the desired levels

When zoomed-in enough, extract point values, display center with a mark, and value on top of it

2.10Andrea AimeCommunity

New features for CSS (2.12)● Code completion in the editor● LessCSS compatible color functions

○ Darken, lighten, saturate, ….

● More readable scale denominator rules○ E.g. @sd < 100k

● Impressive CSS->SLD translation speedup○ 50 times faster on large CSS sheets

● Easier env variable integration (in progress, not yet landed)

○ @varname or @varname(defaultValue)

2.10Andrea AimeCommunity

YSLD extension module● Style Layer Descriptor represented in

YAML○ Uses indentation rather than XML○ Easier to read○ More compact○ More flexible syntax○ Contains variables for reusable code○ Compatible with SLD

● REST API can convert formats○ Preview your existing styles as YSLD

using REST API○ View the SLD representation of YSLD

2.11Boundless

symbolizers:- polygon: stroke-color: 'blue' stroke-width: 1 fill-color: '#7EB5D3'- text: label: ${name} fill-color: 'black' anchor: [0.5, 0.5] x-maxDisplacement: 40 x-autoWrap: 70

Community

● JSON representation of styling○ More limited than SLD/YSLD/CSS

● Works well with vector tiles○ Cross platform styling (OpenLayers, MapBoxGL)

● Work in progress:○ Currently a GeoTools community module ○ Should package and go to extension if we get time!

MapBox Style Community Module

2.12David VickBoundless Suite

MapBox Style in GeoServer andOpenLayers

Labelling improvements

● Kerning on by default(adjusts spacing between chars, e.g.Check space between T and r in Trimble)

● Underline and Strikethrough● Char and word spacing control

● Work in progress:○ Labelling border of polygon

(instead of the centroid)○ Filtering labels based on

polygon screen size in pixels

2.112.12

Andrea AimeNuno Oliveria

Community/Geoinfo/OpenGeoGroep

Improved QGIS Compatibility ● Additional custom “Well Known” marks● Support for SLD 1.1 External Marks (e.g. TTF font based)● SVG Parameters 1.0 spec

<PointSymbolizer> <Graphic> <ExternalGraphic> <OnlineResource xlink:type="simple" xlink:href="firestation.svg?fill=#FF0000" /> <Format>image/svg</Format> </ExternalGraphic> <size>128</size> </Graphic> </PointSymbolizer>

2.11Andrea AimeCommunity

● Label export, most shared configurations exported● Some in the works (word/char spacing, strikethrough, polygon border labelling)● Some changes needed in GeoServer SE 1.1 parser too

QGIS 3.0 style export improvements

2.12Andrea AimeGeoSolutionsOpenGeoGroep

WMS

Vector Tiles ExtensionExtension for publishing vector tiles

- Great solution for modern High Definition Screens

- Use for GeoWebCache for “MapBox without MapBox”

- Couple with MapBox style for consistent styling

2.11Gabriel RoldanBoundless Suite

Better centroid / label placement in “vector” outputsKML

● generate placemarks that are always inside polygon(for concave polygons)

Vector Tiles

● Pregenerate label points in tiles

○ Reduce effort on client

2.12Kevin SmithJust DeOliveraRnD

Label Labelvs

WMS sortBy vendor param● Sort by recency, cloud cover, and so on● Supported by vector data and image mosaic

2.12Andrea AimeGeoSolutionsCommunity

Sort by recencySort by cloud cover

ncWMS like extensions to GetMap (community)● A new styling language designed for dynamic colormaps● Applies to all rasters based on their statistics● Use can control min/max, log scale, animation● Part of OGC TestBed 12 activities

2.10GeoSolutionsOGC

WFS

“null” prefix bug finally solved

● Occasionally GeoServer could return WFS GML with “null” as the feature prefix

● Solved for simple features● Solved for complex features

on global services

2.10Nuno OliveiraGeoSolutionsIGEA/NLS

App-schema improvements

● Improvements to connection pool usage (only one connection used per request now)

● Some improvements helping while using MongoDB as a source for app-schema records

● Hale mapping support being extended to support MongoDB

2.12Nuno OliveiraGeoSolutionsNPRA

GeoJSON output for complex features● Complex features not limited to GML anymore

2.10Nuno OliveiraGeoSolutionsNPRA

{ "type":"FeatureCollection", "totalFeatures":"unknown", "features":[ { "type":"Feature", "id":"gu.93", "geometry":{ "type":"Polygon", "coordinates":[ ... ] }, "properties":{ "description":"Olivine basalt, tuff, microgabbro, minor sedimentary rocks", "occurrence":[ { "shape":{ "type":"Polygon", "coordinates":[ ... ] }, "name":"urn:x-test:GeologicUnit:16777549126931093", "observationMethod":{ "CGI_TermValue":{ "value":"urn:ogc:def:nil:OGC:missing" } }, "positionalAccuracy":{ "CGI_TermValue":{ "value":"urn:ogc:def:nil:OGC:missing" } },

{ "shape":{ "type":"Polygon", "coordinates":[ ... ] }, "name":"urn:x-test:GeologicUnit:16777549126931093", "observationMethod":{ "CGI_TermValue":{ "value":"urn:ogc:def:nil:OGC:missing" } }, "positionalAccuracy":{ "CGI_TermValue":{ "value":"urn:ogc:def:nil:OGC:missing" } }, "specification":{

} } ], "name":"urn:x-test:GeologicUnit:16777549126931093" } // properties ends here}

WCS

Vendor param for mosaics: cqlFilter and sortBy ● Sort by recency, cloud cover,

and so on● Filter on the same● Maps to image mosaic read

parameters● Uses attributes in the mosaic

index● Consistent with WMS now,

first see and then download

2.12Andrea AimeGeoSolutionsCommunity

NetCDF input/output improvements

2.12NielsScitusCommunity

● Can configure NCML files as sources● Support variable's attributes definition in

NetCDF output● Preserve scalars from input

(NetCDF in/NetCDF out)

Tiling

● GeoServer supports “cascading” remote WMTS services

○ GeoServer acts as a client, rather than just rerouting each request

○ Same as WMS cascading

● WMTS cascade is a work in progress

○ Should land in 2.12

WMTS cascading

2.12Ian Turton/Emanuele TajariolGeoSolutionsCustomer

Parameter filter cache drop improvements● GeoWebCache can cache based on parameters

like style, time, or elevation.○ Could not truncate (remove cached tiles) across all

parameter values, each must be known and truncated separately.

○ Tiles could be orphaned if the allowed values changed.○ GeoServer can truncate tiles when the data within them

changes, but only caught the default values for parameters other than style

● Added operations for○ Cross parameter truncate○ Orphan purge

● Enabled cross parameter truncate on data updateGWC 1.12

GS2.12

Kevin SmithBoundlessMapStory

WMS/WMTS multidim domain discovery module (community)● Data with N dimensions (time, elevation, custom ones)● Dimensions can be related (forecasts, time and run time), the data can be

scattered and found in pockets, hard to locate data in the ND cube● Add extra calls to query the domains● Developed as part of OGC Testbed 12● See the engineering report and give us feedback:

2.10Andrea/NunoGeoSolutionsOGC

http://docs.opengeospatial.org/per/16-042r1.html#WMTS_Multidimensional_GeoSolutions

Configuration and management

Security Consideration - Disabled writing by default● WFS-T allows editing using

Transaction requests○ May not be what you expect?

● Default configuration includes security restrictions on WFS-T functionality

● Restricting editing of data to the administrator account.

● Makes service read-only(while still advertising wfs-t)

2.11Nuno OliveiraCommunity

● Great performance improvement!● Test with 10000 layers

Faster startup times for large data dirs

2.11AndreaCommunity

Version COLD(Startup time seconds)

HOT(Startup time seconds)

2.10.1 428 39

2.10.1 + JDCBConfig 62 49

GSIP 155 68 29

● Great performance improvement!● Tested on topp:states to make catalog lookup times more evident (assume diff is a constant)

Faster runtime times for large data dirs

2.11AndreaCommunity

Version Throughput req/sec Avg resp. time ms

2.10.1 169 47

2.10.1 + JDCBConfig 68 117

GSIP 155 233 34

● When loading a shapefile the PRJ file only matched on AUTHORITY code

● Faster lookup implementation allows GeoServer to search the EPSG database and match far more often

Better/Faster EPSG code lookups

2.11AndreaCommunity

This reduces a common frustration when configuring new layers in GeoServer

Module “status” REST API (and GUI)● Verifying what modules are installed was a

challenge● REST API access to the “status page”● Easy to add additional entries● Used to check environment for difficult to

configure modules that use native code

2.11Morgan ThompsonBoundlessBoundless Suite

geoserver/rest/about/status

Parametric configuration● Stick parameters in store configurations and

some security bits ● Helps migrating a data dir between

environments (e.g., integration/test/beta/production)

● Set the -DALLOW_ENV_PARAMETRIZATION=true system var

● Create a property file with variable values

2.10Alessio FabianiGeoSolutions

EumetsatWorld Bank

store_url = file:///var/geoserver/store/teststore

Backup/restore (community)● Save the configuration to a zip file (not the

data!)● Restore later, in the same or different

environment● UI to run backup and restore● Asynch operations

○ During backup the configuration is forced read only○ During restore it’s fully locked

2.10GeoSolutionsEumetsatWorld Bank

Web interface in Tools / Resource Browser

● Manage and edit resources

● Data Directory or JDBCStore

● Community module

● Packaged plugin coming soon

● REST API sporting equivalent functionality

found in core

Resource Browser (community)

2.10Scitus DevelopmentBoundless/NGA

Security

OAuth (community)● Allows GeoServer authentication to be

provided by OAuth● Allows users to login with:

○ GeoNode○ Google○ Github○ … add yours here, it’s pluggable

● Developed for integrationwith GeoNode

2.10Alessio FabianiGeoSolutionsBoundless

SecurityConsiderations

Take care to report the issue in a responsible fashion:

● Keep exploit details out of issue report● Mark the issue as a vulnerability.● Be prepared to work with Project Steering Committee (PSC) on a solution● Keep in mind PSC members are volunteers

and an extensive fix may require fundraising / resources

If you are not in position to communicate in public please consider commercial support, contacting a PSC member, or reaching us via the Open Source Geospatial Foundation at info@osgeo.org.

Security Vulnerabilities - Responsible Disclosure

Responsible disclosure policy is working:

● Most users contacted us in a responsible fashion● One user shared confidential customer report on the user list

Where do reports come from?

● Majority reported from security audits○ Usually performed under contract○ Automated tools tend to focus on web administration ui, rather than web services

● GeoServer is subject to several security audits a year

Security Vulnerabilities

Upgrade to get security fixes● Security fixes being added to supported versions only● Please upgrade your GeoServer installations!

Looking Ahead

Java 9 CompatibilityJava 9 has been delayed until 2017, but we would like to ensure GeoServer will function. (Oracle stops free security updates to Java 8)

We have a couple incompatibilities with the GeoTools “service provider interface” plugin system that we will look at resolving during the code sprint!

The Java Advanced Imaging library enables GeoServer to do all kinds of great image processing! The JAI-EXT project extends this library with replacement no-data / footprint aware operations.

Sadly Oracle has not maintained the project, and has not released it as open source.

(see Replace-JAI on GeoTools wiki)

Raster Processing EngineJoint OSGeo / LocationTech game plan:

● Raster Processing Engine APIdefinition of raster-processing-api

● Raster Processing Enginereference pure-java implementation

● Raster Processing Wrapperbenchmark jai-wrapper implementation

● GeoTools Replace JAImigrate to raster processing engine api and coordinate transition affected projects (GeoServer, GeoNode, uDig, GeoScript, etc...)

Thanks!

Earth observation for EO

OpenSearch for Earth Observations● Recent addition to OGC protocol for search of earth imagery● Simpler alternative to CSW with ebRIM● Two steps search

○ First find collection of similar products (by satellite/sensor)○ Then search products in it (by other attributes, e.g., cloud cover, off nadir)

● Keeps database of searchable metadata● Links to OGC services providing said data● Allows downloading ISO/O&M metadata

2.12GeoSolutionsDLR/EVO-ODAS

OpenSearch for Earth Observations● Result is RSS with human readable descriptions and

○ Links to OGC services providing said data○ Links to download ISO/O&M metadata○ Links to download the raw data

● Exposes one feature type per collection, allowing easy mosaic setup with same filtering and ordering

2.12GeoSolutionsDLR/EVO-ODAS

OpenSearch for Earth Observations

http://cloudsdi.geo-solutions.it/geoserver/oseo/search?parentId=SENTINEL2&cloudCover=30]

● E.g., search all SENTINEL2- images with cloud cover less than 30%

2.12GeoSolutionsDLR/EVO-ODAS

OpenSearch for EO: admin REST API● Create/Update/Delete the collections and its information● Create/Update/Delete the product and its information● Either one shot (throw a zip file with all the info) or granular (edit

searchables first, then thumbnail, then metadata, links, …)

Swagger documentation

2.12GeoSolutionsDLR/EVO-ODAS