GBIF API Hackaton, March 2015, Leiden, Sp2000/GBIF

GBIF & Species2000 Hackathon March 2015 Naturalis, Leiden

GBIF portal API

Dag Endresen GBIF Norway UiO Natural History Museum in Oslo University of Oslo Tuesday, March 3rd, 2015 Slides: CC-BY-4.0

Credits, some slides from:

Daniel Amariles, Colombia (2013) Nodes training at GBIF GB20 in Berlin [link]

Gallien Labeyrie, France (2014) Mentoring project France-Spain-Portugal [link]

GBIF DATA PORTAL

SPECIES SEARCH

GBIF DATA PORTAL API

An interface to access data published through the GBIF network using web services.

DATA PORTAL API

GBIF Data Portal API: http://api.gbif.org/v1/ (+parameters)

Summary and information:

http://www.gbif.org/developer/summary The RESTful API take search parameters as key=value pairs and respond with json content type.

RESTful query formatJSON response type

GBIF API sections

•  Registry informa)on about the datasets, organiza)ons (e.g. data publishers), networks and the means to access them (technical endpoints)

•  Species informa)on about species and higher taxa, and u)lity services for interpre)ng names and looking up the iden)fiers (access to all published checklists in the GBIF checklist bank)

•  Occurrence occurrence informa)on crawled and indexed by GBIF and search services to do real )me paged search and asynchronous download services to do large batch downloads

•  Maps simple services to show the maps of GBIF mobilized content

•  News services to stream useful informa)on (RSS)

API example : dataset Search for datasets by publishing country: http://api.gbif.org/v1/dataset/search?publishingCountry=NO

Dataset information (UiO NHM Lichens):http://api.gbif.org/v1/dataset/7948250c-6958-4a29-a670-ed1015b26252

Contact persons for a dataset:http://api.gbif.org/v1/dataset/7948250c-6958-4a29-a670-ed1015b26252/contact

Dataset endpoint (get download URL): http://api.gbif.org/v1/dataset/7948250c-6958-4a29-a670-ed1015b26252/endpoint

http://www.gbif.org/developer/registry

API example : dataset Download activity metrics for dataset (UiO NHM Lichens): http://api.gbif.org/v1/occurrence/download/dataset/7948250c-6958-4a29-a670-ed1015b26252

=> records from this dataset, included in 2650 download sets

Records lastInterpreted since November 2014: http://api.gbif.org/v1/occurrence/search?datasetKey=7948250c-6958-4a29-a670-ed1015b26252&lastInterpreted=2014-11-01,* (=> 168 316 occ.)

Metrics for dataset data contents: http://api.gbif.org/v1/dataset/66dd0960-2d7d-46ee-a491-87b9adcfe7b1/metrics

=> count records by dimensions such as Kingdom, Rank, Vernacular name langue, Extensions provided, … NB! only implemented for species checklists, not (yet?) for occurrences!

http://www.gbif.org/developer/registry

API example : species List all name usages (across all checklists): http://api.gbif.org/v1/species?name=Beta%20vulgaris

Name usage across checklists (Beta vulgaris, 5383920): http://api.gbif.org/v1/species/5383920/related

Name parsed into epithets and author etc.: http://api.gbif.org/v1/parser/name?name=Abies%20alba%20Mill.%20sec.%20Markus%20D.

{"scientificName": "Abies alba Mill. sec. Markus D.", "type": "SCINAME", "genusOrAbove": "Abies", "specificEpithet": "alba", "authorsParsed": true, "authorship": "Mill.", "sensu": "sec. Markus D.", "canonicalName": "Abies alba", "canonicalNameWithMarker": "Abies alba", "canonicalNameComplete": "Abies alba Mill." }

http://www.gbif.org/developer/species

API example : occurrence List occurrences of Beta vulgaris: http://api.gbif.org/v1/species/match?name=Beta+vulgaris => taxonKey http://api.gbif.org/v1/occurrence/search?taxonKey=5383920

List occurrences from Norway (of Beta vulgaris): http://api.gbif.org/v1/occurrence/search?publishingCountry=NO http://api.gbif.org/v1/occurrence/search?publishingCountry=NO&taxonKey=5383920

Information about a single occurrence record: http://api.gbif.org/v1/occurrence/1040970640 http://api.gbif.org/v1/occurrence/1040970640/fragment http://api.gbif.org/v1/occurrence/1040970640/verbatim

List occurrence counts for datasets of country (or taxon):

http://api.gbif.org/v1/occurrence/counts/datasets?country=NO

http://www.gbif.org/developer/occurrence

API example : identifiers (not implemented -- yet)

Searching by occurrenceID is unfortunately not supported yet … http://dev.gbif.org/issues/browse/POR-2451 http://dev.gbif.org/issues/browse/POR-2337 So, we cannot yet list or count occurrences with pattern “urn*” in occurrenceID

http://api.gbif.org/v1/occurrence/search?occurrenceID=urn* http://api.gbif.org/v1/occurrence/search?occurrenceID=urn%3Acatalog http://api.gbif.org/v1/occurrence/search?occurrenceID=urn%3Alsid http://api.gbif.org/v1/occurrence/search?occurrenceID=http* …

API example : download data Lookup speciesKey (1) and download occurrences (2): http://api.gbif.org/v1/species/match?verbose=false&kingdom=Plantae&name=Beta+vulgaris

=> usageKey/speciesKey = 5383920

http://api.gbif.org/v1/occurrence/search?taxonKey=5383920 [&limit=1000&offset=0]

=> notice: count = 25 513 => then: page through results… (using offset & limit)

http://api.gbif.org/v1/occurrence/download/request [POST] => downloadKey (see next slide)

API example : asynchronous (1) Request asynchronous download:

$ curl -i --user yourGbifUserName:yourGbifPassord -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d @filter.json http://api.gbif.org/v1//occurrence/download/request >> log.txt

Search parameters in a json text file: filter.json (in current directory or located in a “PATH-directory”):

{ "creator":”yourGbifUserName", "notification_address": [“[email protected]"], "predicate": { "type":"and", "predicates": [{"type":"equals","key":"HAS_COORDINATE","value":"false"}, {"type":"equals","key":"TAXON_KEY","value":"5383920"}] } }

Downloads are available in the portal (from your user profile)

API example : asynchronous (2a) Request asynchronous download:

function gbifapi { curl -i –user yourGbifUserName:yourGbifPassword -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d "{\"creator\":\”yourGbifUserName\", \"notification_address\": [\”[email protected]\"], \"predicate\": {\"type\":\"and\", \"predicates\": [{\"type\":\"equals\",\"key\":\"HAS_COORDINATE\",\"value\":\"true\"}, {\"type\":\"equals\", \"key\":\"TAXON_KEY\", \"value\":\"$1\"}] }}" http://api.gbif.org/v1/occurrence/download/request >> log.txt echo -e "\r\n$1 $2\r\n\r\n----------------\r\n\r\n" >> log.txt

} $ gbifapi 4140730 "Aciachne acicularis" $ gbifapi 4140704 "Aciachne flagellifera" $ gbifapi 5289784 "Aegilops comosa” … (work in progress…)

API example : asynchronous (2b) (…clean log.txt with the downloadKeys using regular expressions…)

function gbifwget { echo -e "\n\n----------------\n$1 $2 $3\n" >> log_wget.txt wget http://api.gbif.org/v1/occurrence/download/request/$1.zip 2>&1 | tee /dev/tty >> log_wget.txt mv $1.zip ./dwca/$2.zip 2>&1 | tee /dev/tty >> log_wget.txt } $ gbifwget 0006050-141024112412452 4140730 "Aciachne acicularis" $ gbifwget 0006053-141024112412452 4140704 "Aciachne flagellifera" $ gbifwget 0006056-141024112412452 5289784 "Aegilops comosa" … (work in progress…)

MAPPING API v1.0 You can easily overlay GBIF content on your own maps. http://www.gbif.org/developer/maps

Slide by Daniel Amariles, 2013

This service is intended for use with commonly used map clients such as the Google Maps API, Leaflet JS library or Modest maps JS library. These libraries allow the GBIF layers to be visualized with other content, such as those coming from Web Map Service (WMS) providers. It should be noted that the mapping API is not a WMS service, nor does it support WFS capabili)es.

hNp://leafletjs.com/

MAPPING API v1.0

hNp://modestmaps.com/


CUSTOMIZING LAYER CONTENT

The format of the URL is as follows: With the following required parameters:

type : TAXON, DATASET, COUNTRY or PUBLISHER

key : The appropriate key for the chosen type (a taxon key, dataset/publisher UUID or 2 leNer ISO country code)

Other supported parameters: resolu)on, layer, paleNe, colors, satura)on, hue

hNp://www.gbif.org/developer/maps

hNp://api.gbif.org/v1/map/density/)le?x={x}&y={y}&z={z}


Useful Tools (JSON & REST)

•  REST client … •  JSON client/parser …

•  JSONView (Firefox, Chrome, …)

•  http://jsonview.com/ •  Display formatted JSON in browser

•  R CRAN : jsonlite •  http://cran.r-project.org/web/packages/jsonlite/

•  E.g. read json into a dataframe [link]

•  OpenRefine •  http://openrefine.org/

R CRAN

rOpenSci provides programmatic access to scientific data with R (rgbif, taxize, EML, geonames, …). https://github.com/ropensci http://ropensci.org/packages/ http://ropensci.org/tutorials/rgbif_tutorial.html http://ropensci.org/tutorials/taxize_tutorial.html

rOpenSci : rgbif library(rgbif) key <- name_backbone(name='Beta vulgaris', kingdom=‘Plantae')$speciesKey bv <- occ_search(taxonKey=key, return='data', hasCoordinate=TRUE, limit=1000) gbifmap(bv)

raster : WorldClim, BioClim layers # using GBIF data (bv) from the previous slide… library(raster) xy <- cbind('lon'=bv$decimalLongitude, 'lat'=bv$decimalLatitude); env <- getData('worldclim', var='bio', res=10) # bioclim (pkg raster) plot(env, 1) # plot the first bioclim layer points(xy[,'lon'], xy[,'lat'], col='red') # plot points bio <- extract(env, xy); # extract environment to points (pkg raster) bv_bio <- cbind(bv, bio); # column-bind GBIF-data and bioclim

rOpenSci : rWBclimate library(rWBclimate, ggplot2) country_dat <- get_historical_temp(c("NOR", "SWE", "DNK", "FIN"), "year") ggplot(country_dat, aes(x = year, y = data, group = locator)) + theme_bw(base_size=18) + geom_point() + geom_path() + labs(y="Average annual temperature of Nordic countries", x="Year") + stat_smooth(se = F, colour = "black") + facet_wrap(~locator, scale = "free")

Resolve taxonomic names library(taxize) # rOpenSci Taxize gnr <- gnr_resolve(names = "Beta vuulgariss") # Misspelled name gnr$results # display suggested names submitted_name matched_name data_source_title score 1 Beta vuulgariss Beta vulgaris L. Catalogue of Life 0.75 2 Beta vuulgariss Beta vulgaris L. ITIS 0.75 3 Beta vuulgariss Beta vulgaris NCBI 0.75 4 Beta vuulgariss Beta vulgaris var.-gr. crassa Alef. GRIN Taxonomy for Plants 0.75

specieslist <- c("Beta vulgaris", "Phleum pratensis", "Nicotiana glauca") classification(specieslist, db = 'itis') # lookup higher taxonomy

Global Names Resolver: http://resolver.globalnames.org/ rOpenSci Taxize: http://ropensci.org/tutorials/taxize_tutorial.html

db = ’col' db = ’itis'

rOpenSci : EML

library(EML, rfigshare) description <- "My dataset published in GBIF"

eml_write(dat = dat, meta, title = "My Dataset", description = description, creator = "Your Name <[email protected]>", file = "dataset.xml") eml_publish("dataset.xml", description = description, categories = "Ecology", tags = "biodiversity", destination = "figshare", visibility = "public") meta <- eml_read("eml_example.xml")

GBIF API support

Subscribe to the mailing-list for help and information messages:

[email protected]

GBIF Hackathon March 2015 Naturalis, Leiden

GBIF portal API

Dag Endresen GBIF Norway UiO Natural History Museum in Oslo University of Oslo Tuesday, March 3rd, 2015 Slides: CC-BY-4.0

GBIF API Hackaton, March 2015, Leiden, Sp2000/GBIF

Technology

Transcript of GBIF API Hackaton, March 2015, Leiden, Sp2000/GBIF