Post on 31-Mar-2015
The Process of Data Ingestion in ÆKOS
Andrew Graham and Matt SchneiderTERN Ecoinformatics Data Analysts
Logos used with consent. Content of
this presentation except logos is
released under TERN Attribution Licence Data Licence v1.0
Introduction
The Data Analyst Role with TERN Ecoinformatics• Analysis of source data and methods• ÆKOS system development and domain modelling• Contextual description of the data• Publication of data into ÆKOS
The AEKOS Framework
1. Upper Context: Party, Project, Scope etc
2. Domain Model (Ontology): Observed entities, their features and relationships
3. Description Model: Methods and definitions
4. Indexing Model: Search and federation
Upper Context
Provides context for Datasets:• Contact details• High level objectives of program• Licensing details and conditions of use• Statement of scope• Alignment with national metadata standards
(ANDS)• Statement of curation processes applied to data
Understanding Field SamplingSchematic view of sampling configuration
Methodological work-flowStudy Location
Selection
Study Location Visit
Study Location Establishment
Sampling Unit Selection Vegetation
Assessment
Physical Assessment
Landscape AssessmentSoil Assessment
Fire EvidenceSurface Cover
Disturbance EvidenceVertebrate Evidence
Climate Evidence
Species AssessmentSpecies Life Stage
Vegetation Assemblage
Voucher Collection
Canopy Age-classCanopy AssessmentStructural Formation
Overstorey Measurement
Authored Method Descriptions• Start with published
method manuals
• Enrich existing method descriptions (protocols) with external web links and other resources
• Clarify questions about methods
• Divide the protocol into smaller method descriptions
Authored Method Descriptions
• Use a consistent format across datasets to allow comparison
• Direct linkage between the data value and the specific method of measurement
• Allows rapid assessment of suitability of data for re-use
• Eventually a method catalogue for researchers
Definition of source datasets
Analysis and definition of source data types:
• Observation data• Taxonomic concepts (a
specific type of ref. data)• Reference data (i.e. Lookup
tables)• Images and other artefacts.
Mapping to the ÆKOS Domain ModelStudy Location
Sampling Unit
Study Location Visit Spatial Point
mudmapcomment
visit dateobserversdisturbance
datumx coordy coord
identifiermarkertype
Species Organism Group
Voucher Specimendetermined identityaccession No.determiner
field identitylife formcover/abundancelife stagephenologydominance
Landscapeslopeaspectlandform pattern
selectsrepresents
contains
contains
represented by
IndexingEnrichment of data with common indexes:• Project level traits• Data management traits• Ecological process traits (disturbance and land-use)• Measurement details• Species taxonomy• Vegetation Assemblage (e.g. NVIS Major Veg. Groups)• Jurisdictional and Bio-geographic boundaries• Spatially derived features (e.g. distance from road,
slope, aspect, etc.)
Federated Taxonomy
The AEKOS Ingestion “DSL”
Screen cap of Eclipse...
• Source data query• Vocabulary management• Method description• Mapping to the common model• Populate indexes• Upper context authoring• Sandbox testing
Data Work-flow• Point of truth is always the source database• Data values are not changed• Data issues fed back to Data Providers• Automatic data refresh mechanism developed• Corrections made in source database and fed back
to AEKOS on next “push”• Just new records and edits after the first load• Update frequency defined for each dataset
Quality Assurance
ÆKOS QA and review:• Team review domain modelling of every dataset
ingested• “Sandbox” test ingestion before publishing to
ÆKOS• Review of method description by other team
members• Internal code validation and error checking
Quality AssuranceData Providers QA:• Review method descriptions• Review upper context
Portal feedback:• Review data content in the portal• Use the portal and suggest enhancements and changes• Look and feel• Index traits• Data accuracy and representation
• Feedback survey and email facility on portal
Thank you
Contact Details
Data Analyst – Matt Schneider matt.schneider@adelaide.edu.au
Data Analyst – Andrew Graham andrew.graham@adelaide.edu.au
Website www.aekos.org.au