U.S. Department of the InteriorU.S. Geological Survey

Web Presence, Data Sharing, Real-time Analysis and Crowdsourcing

GFSAD30 January 2015 Workshop Justin Poehnelt, Student Developer

jpoehnelt@usgs.gov

2014 Achievements

Data products are being distributed from multiple sources.

An interactive map with multiple layers and a legend is available for public viewing.

A beginning framework for crowdsourcing training and validation data has been established.

2014 Challenges

Requirements are beginning to stabilize for croplands web application.

Custom web application requires custom architecture, design, codebase and testing.

Development has been a very iterative process.

USGS Web Portal

While project updates have been posted, more significant changes will be delayed until USGS migration to new content management system. A storage solution for publicly available large files has been implemented on Flagstaff servers.

Central location to point visitors.

Google Earth Engine

Layers can be used in derivative products through the distributed power of Google Earth Engine.

https://earthengine.google.org/#detail/USGS%2FGFSAD1000_V1 https://earthengine.google.org/#detail/USGS%2FGFSAD1000_V0

Google Earth Engine brings together the world's satellite imagery — trillions of scientific measurements dating back over 40 years — and makes it available online with tools for scientists, independent researchers, and nations to mine this massive warehouse of data to detect changes, map trends and quantify differences on the Earth's surface. Applications include: detecting deforestation, classifying land cover, estimating forest biomass and carbon, and mapping the world’s roadless areas.

Croplands Website

Purpose: The web application provides a data repository for the team and enables interaction not possible through the USGS website system.

Status: Currently in internal testing and use by the team.

Croplands Website: Services, Platforms and Libraries Used

Croplands Website: Objectives

Crowdsourcing Training Data: Allow experts and non-experts to contribute local knowledge to training and validation data through a simple interface

Interactive Maps: Interactively display multiple data products with easy to understand metadata and legend

Data Storage: Provide a standardized data storage solution to serve numerous.

Real-Time Distributed Spatial Computing: Enable real-time distributed computing operations through Google Earth Engine

Croplands Website: Crowdsourcing

Reference Data: Creating a mechanism for others to contribute their own knowledge on a location to the project.

Team Use: Allows our team to review and modify ground data using high resolution satellite imagery, NDVI time series and other data sources as available.

Croplands Website: Interactive Map

Layer Legend: Easy to understand layer classes

Opacity Controls: Controls allow users to view data products over high resolution imagery.

Future: Easily expandable for new data products or additional background layers.

Croplands Website: Data Storage

Standardized Data Format: Uses REST web architectural style

Restricted Access: Multiple Layers of Access according to Role

Backup: Database is backed up on a daily basis and API level data is cached every ten minutes.

Capacity: Tested with over 500,000 data entries

Public API Server Database

Croplands Website: Computing

Task Queue: Distributed tasks to send digest emails of data changes, process reference photos, find data anomalies or complete other parallel tasks.

Real-Time Spatial Calculations: Ability to do computations on spatial data using server-side libraries and interface with Google Earth Engine.

NDVI Time Series Point in Polygon Generate Product on Periodic Basis

Data Models

Questions still remain but a more flexible structure is being developed.

Is a secondary crop from a mixed pixel or a later season?

Locations

Parent object for records, photos, etc. Point based, but may also be represented by a

polygon. NDVI time series pulled as requested from

Google Earth Engine.

Records

Each location may have many records. Currently a unique constraint of a single record per month in a year.

Does not contain spatial data. Includes land cover type, primary and secondary

crop type, water source, and intensity. May be expanded to include crop health. Each record has a flag for training or validation.

Common Interface

An Application Programming Interface(API) can allow many individuals to manipulate data in a standardized form.

FAO employees and volunteers may want to access the data from a custom application.

A mobile can consume and manipulate the data in the same way as a web application using the same endpoints.

API Security

JSON Web Tokens: A digitally signed token is generated for a user with a specific expiration time in seconds.

The token is attached to all data requests and validated by the server.

Currently limited to authorizing specific users.

API Permissions

  Task Public User Partner Team Admin

Records

Create x x x x x

Review   x x x x

Update       x x

Delete         x

DownloadsPublic x x x x x

Private     x x x

# Todo

Get SSL certificates for server endpoints that handle user data.

Rebuild some Angular Directives and Controllers. Add additional unit tests for frontend application. Implement timed release for team collected data. Move source code to public repository for

anyone to contribute to. Move all static content to Amazon Cloudfront and

point www domain to Amazon.

# Todo (Continued)

Separation of client application and server application.

Finalize API and data models for rating mechanism.

Input existing global data into system. Build CSV file import into web application. Allow filtering of locations by country and

continent. Get appropriate libraries running on server.

# Todo (Continued)

NDVI time series aggregation statistics for records generated in browser as requested by user.

Allow selection of records by specific NDVI time series parameters. Find records similar to an existing record.

Simple mobile application for field data collection. Automatic creation of CSV files saved to GFSAD

drive.

#Todo (Continued)

Allow user to overlay kml, geojson or other wms layers on map.

Find alternative to Google Map Engine which is being discontinued 1/16.

Other Ideas

Very simple application allowing user to sketch polygon(field) for a location.

Even simpler application asking ‘Is this cropland?’ with two buttons to help refine the data.

Querying Google Maps for street view data. Using Google Earth Engine to determine pixel

validity.

Mobile Application

What is best approach for converting point data with photo and bearing outside of location to the correct pixel? Manipulation within mobile application by field

personnel. Manipulation within web application by team members Auto suggest best location with spatial segmentation

and analysis.

Mobile application data should be placed in staging area until approved by team member.