Status of GeoServer WPS

Status of GeoServer WPS

Ing. Andrea Aime, GeoSolutions

FOSS4G-NA 2015, San Francisco March 9th-12th 2015


Introductions

Your host:

Andrea Aime (geowolf on Twitter)

Long time developer and project steering committee member of GeoTools and GeoServer

Map rendering, spatial analysis, data access, reprojection, security, …

His company

GeoSolutions

Specialized in open source solutions based on GeoTols/GeoServer/GeoNetwork/MapStore, strong focus on image processing, spatial analysis, catalogs, web and mobile front ends for mapping and data analysis

WPS: five minute intro


Web Processing Service

Wikipedia introduces OGC WPS as:

[A service] designed to standardize the way that GIS calculations are made available to the Internet.

WPS can describe any calculation including all of its inputs and outputs, and trigger its execution

The specific processes served up by a WPS implementation are defined by the owner of that implementation.

Although WPS was designed to work with spatially referenced data, it can be used with any kind of data.


WPS 1.0 operations

Standard operations:

GetCapabilites

DescribeProcess

Execute

GeoServer extended operations:

GetExecutionStatus

GetExecutionResult

Dismiss (new, borrowed from WPS 2.0)


GetCapabilities

Some metadata about the server and its owner

The list of available processes, with a description

Let’s say we’re interested in the JTS:buffer process

FOSS4G 2011, Denver 12th-16th September 2011


DescribeProcess

List of inputs and outputs

Descriptions

List of accepted formats (not shown in the xml)



Execute

Buffer a L shaped geometry with distance “2”

Get the result back as GML


Synchronous vs asynchronous

WPS client

WPS

Launch process

Send back results

Simple Suitable for fast executions

Synchronous

WPS client

WPS

Launch process

Status URL

Check progress GetExecutionStatus

50%

Check progress GetExecutionStatus

100% Full results

More complex Suitable for longer computations

Asynchronous


Inline vs reference outputs

WPS client

WPS

Launch process

a) Get back XML with results embedded

Simple Suitable for smallish outputs

Inline XML/raw

WPS client

WPS

Launch process

Get back XML with links to results

Download each result

GetExecutionResult

More complex Suitable for large downloads

(allows for retry/resume)

Reference outputs

b) Get back result directly

GeoServer: Integrated WPS


Common WPS setup

WPS

Remote WCS

Remote WFS

HTTP server

WPS client

Request +

data or links to data

Result data

Fetch data

Local filesystem


GeoServer WPS integration

WPS

Remote WCS

Remote WFS

HTTP server

WPS client

GeoServer Layers

WMS client WMS

GeoServer UI

Local filesystem


Demo request builder

List processes

Describe

Set parameters and execute

All in one form


Local layers and chaining


WFS

WMS


Rendering transformations

On-the-fly data transformations inside rendering

chain

Calling WPS processes from SLD docs

Optimized for performance


Rendering transformations


Writing processes


First, do you need to?

• 80+ built-in processes

• JTS geom manipulation

• Compound WFS query

• Raster to vector

• Vector to raster

• Stats analysis processes

• Designed for chaining


But if you really need to …

Python Groovy

JavaScript Ruby

JSR-223

http://geoscript.org


• ProcessFactory

• Enumerates processes it can generate

• Describes them

• Creates instances of them

• Suitable for wrapping another set of existing processes (e.g., a GRASS, a Sextante, a GDAL command line utils wrapper)

Low level Java API


• You write the javabean, annotate it to advertise and describe inputs and outputs, finally declare the bean in the Spring application context:

Java annotations


• Community modules to write processes in scripting languages for the JVM: Jython, Javascript, Groovy, …

• Just drop a script file in the right directory and the process is ready to be used

GeoScript scripting languages

Python Groovy JavaScript Ruby

http://geoscript.org


Sample scripting application

http://localhost:8080/geoserver/www/wps.html


Land property distrib. in the polygon


A look at the data dir

1. Data

2. WPS process scripted in Jython

3. The GUI


The Jython script

http://geoscript.org/py/index.html

What’s new in GeoServer 2.7.x


• So far, only the node that was running the async process knew about it

Asynchronous process clustering

Client Get Process

Status

Load balancer

WPS1

WPS2

WPS3

??? Get Process Status


• ProcessStatusStore, shared state of the currently running processes

• ProcessArtifactsStore, shared outputs store

Asynchronous process clustering

Client Get Process

Status

Load balancer

WPS1

WPS2

WPS3

Get Process Status

Hazelc

ast

sh

are

d s

tatu

s

!!

Sh

are

d f

ilesyste

m

for

larg

e o

utp

uts


• GUI to ProcessStatusStore

• Also allows admin to dismiss process executions

Status page available to admins


• Allows the client to dismiss the execution of an asynchronous process

• Borrowed from WPS 2.0 (as a new vendor op in WPS 1.0)

Dismiss

http://host:port/geoserver/ows?service=WPS&version=1.0.0&

request=GetExecutionStatus

&executionId=397e8cbd-7d51-48c5-ad72-b0fcbe7cfbdb

http://host:port/geoserver/ows?service=WPS&version=1.0.0&

request=Dismiss

&executionId=397e8cbd-7d51-48c5-ad72-b0fcbe7cfbdb


• All processes get immediately dismissed if they are in the following states

• Queued

• Reading inputs (poison pill on inputs)

• Writing outputs (poison pill on outputs)

• A process that’s running needs to give up willingly (it gets a dismiss notification). We cannot just kill a a thread in Java ;-)

Dismiss


Process security

• GeoServer comes with 80+ built-in processes

• It was already possible to selectively disable them

• Now, each can be associated to one or more user roles


Global process limits

• Make sure each Execute request does not chew too much resources

• Limit how many processes run in parallel

• Limit how much time a process can run

• Automatic dismiss past the limit


Local process limits

• Add limits on input sizes, range of acceptable values, max number of values for repeated inputs

• All advertised in DescribeProcess

• Pluggable framework to add more input validators


New output formats: GPX and KML

• Java processes have java objects as input and outputs

• ProcessParameterIO instances translate common java Object into external formats

• E.g., FeatureCollection GML, GeoJson, CSV

• In GeoServer 2.7.x there are two new community modules to transform FeatureCollection also into GPX and KML


New «Download» process

• Single process for «advanced clip and ship»

• Filter, reproject, cut on mask both raster and vector data

• WPS async support key for large data extractions (WFS and WCS not really suitable for this case)

• Evaluate in advance download size to enforce max extraction size limits


New «Download» process

• A GUI to the process, as a MapStore customization

http://mapstore.geo-solutions.it/mapstore/

What’s cooking


ogr2ogr WPS output formats

• Extension to the ogr2ogr WFS output format

• Can turn any FeatureCollection output type into any of the ogr2ogr supported output formats

• Can be already downloaded in 2.8.x nightly builds

WPS GML ogr2ogr Desired format


WPS-remote

• You have a bunch of data processing servers, with native/command line «processes»

• Want to discover them, expose as WPS, invoke, track progress, and publish results

WPSAgent.py

Ge

oS

erv

er

WPS remote module

XMPP WPSAgent.py

WPSAgent.py

Flooding model

Fire propagation

model

Snow levels forecast


WPS-remote

• Asset Allocator example: on demand counter-piracy ship positioning wrt wheather & other conditions


WPS-remote

• Automatic publishing of result as new layers via the importer module

Ge

oS

erv

er

WPS remote module

XMPP WPSAgent.py

Snow levels forecast

Importer

Shared filesystem

1) Execute 2) Get results location

3) Call importer

4) Side effect: publish result as new layer

5) Return OWS context


More WPS integration

• Mind, still on the wish list, but with good chance of getting funding

• «Automate» module, would allow periodic operations of new data in GeoServer. Focus on data ingestion, temporary layer management, it would integrate internal WPS process calls

• Allow calling WPS processes from Importer module, for more complex input data transformations before publication


WPS 2.0

• Still looking for sponsors!

• Could it be you?


The End

Questions? [email protected]

[email protected]

Status of GeoServer WPS

Documents

Transcript of Status of GeoServer WPS