talk · Thursday 24 September · 12:03–⁠12:15 · SAL B

Robot-ready by construction: an ontology-driven, SHACL-validated submission pipeline for national ecological monitoring data

Andrew Tokmakoff · AI-Readiness Metrics and Metadata for Biodiversity Data

Recording time 3:51:42–4:04:11Open on Vimeo ↗

Summaries and key moments are machine-produced. Check the recording for the speaker’s exact words.

The short versionDriving both field data capture and RDF export from one semantically annotated specification makes monitoring data standards-compliant and machine-ready from the start.

Overview

What this was about

Andrew Tokmakoff described how TERN Ecosystem Surveillance moved from a decade-old Android app and Postgres database to EMSA (Ecological Monitoring System Australia), built for the federal government, where a single data model specification drives both the offline-capable field app (a PWA) and the exporter that pushes RDF to Australia's national Biodiversity Data Repository. Protocol steps and attributes are mapped to IRIs, an OpenAPI 3.0 specification generates UI elements (e.g. lookup-table references) and validates JSON on the way in, and the exporter maps flat JSON into the graph-based ABIS and TERN ontologies, resolving vocabularies live via SPARQL and validating output with SHACL shapes. Data therefore arrive 'robot-ready by construction' rather than by later remediation.

Why it matters. It demonstrates a working alternative to retrofitting standards onto legacy data: specification-driven collection with validation at both boundaries, feeding a national biodiversity knowledge graph.

Key ideas

In the room

  • TERN is Australia's national terrestrial ecosystem research infrastructure (funded through NCRIS); Ecosystem Surveillance, based in Adelaide since 2011, runs a long-term standardised plot network, mainly in the rangelands.
  • Legacy system: AusScribe Android app, Postgres and CouchDB; new system: EMSA protocols and software, with a PWA field app and a Strapi headless CMS back end ('Core') on Postgres.
  • Two routes to standards-compliant data: collect first and map later, or, as with EMSA, start from protocol definitions that drive both collection and RDF creation.
  • Data are structured surveys, not just occurrences; protocol steps and procedures are mapped to IRIs, making data self-describing.
  • An OpenAPI 3.0 specification (close to JSON Schema) is a machine-readable contract: the app builds its UI from it (e.g. an x-lutRef veg growth form lookup) and the exporter reads the same key to map codes into the graph.
  • The exporter maps flat JSON to graph models (ABIS national profile, TERN ontologies, EMSA protocol-specific SHACL shapes), resolving EMSA vocabularies via SPARQL against GraphDB with caching, so it adapts as protocols evolve.
  • Quality gating at both boundaries: JSON Schema validation in the app and on submission; SHACL validation of RDF on export.
Jump in

Notable moments

In their words

Transcript

Automatically generated captions can contain mistakes, especially in names and technical terms. Times are relative to the room recording.

Read the transcript ↓
Loading transcript…
Programme abstract

Robot-ready by construction: an ontology-driven, SHACL-validated submission pipeline for national ecological monitoring data

Andrew Tokmakoff

Some approaches to robot-ready biodiversity data work backwards: collect first, then retrofit standards-compliant metadata before submission. The Ecological Monitoring System Australia (EMSA) takes the opposite approach. Its Monitor platform captures field data across 38 ecological survey protocols, and the export pipeline emits standards-aligned RDF at source - so submissions arrive at the national Biodiversity Data Repository (BDR) robot-ready by construction, not by remediation.

The pipeline resolves every attribute, property, and observation-procedure URI live against EMSA vocabularies in a GraphDB SPARQL endpoint, with caching and vocabulary harmonisation mapping field terms to controlled vocabularies. Data quality is enforced at both boundaries: JSON Schema validation governs what enters the system from the field, and per-protocol SHACL shapes gate what leaves - submissions that fail validation are not exported. Provenance is tracked end-to-end from field collection through to BDR ingest, and a companion XLSX gives curators a human-readable view of what the machine submitted. The full pipeline is config-driven, containerised, and reproducible.

We discuss what works well - SPARQL resolution keeps the pipeline aligned as vocabularies evolve, and SHACL gating catches structural errors before they reach BDR curators - and the practical lessons of maintaining a config-driven exporter across 38 protocols as both the monitoring program and its target vocabularies continue to grow.

Submitted to the conference programme before the talk, so the talk itself may differ.