talk · Monday 21 September · 14:45–⁠15:00 · Aula

DwC-DP Integration at GBIF

Federico Mendez, Mikhail Podolskiy · Monday plenary programme

Recording time 52:57–1:08:49Open on Vimeo ↗

Summaries and key moments are machine-produced. Check the recording for the speaker’s exact words.

The short versionGBIF is close to supporting Darwin Core Data Packages end to end, with typed and stricter validation, while keeping Darwin Core Archives fully supported.

Go back to 2025Data Package pilot · 3 connections

Revisit Living Data 2025 alongside this talk. These connections were checked against captions; related examples may come from different teams rather than a direct follow-up.

Continuing work

Public review → reported ratification

2025 Living Data

Steve Baskauf describes the proposal as under an extended 90-day public review, open until at least 16 December.

Watch 2025 at 1:06:56 ↗
Read the source captions

Darwin Core Data Package public review process

moving the proposal to public comment in September. The public review is ongoing until at least December 16th. This period of review is 90 days rather than the normal 30 days due to the complexity of the proposal. If you want to participate in the review of the two documents, you can follow the links in the participating in the review section of the landing page. This will lead you to the appropriate place in the GitHub issue tracker for the Darwin Core
2026 This talk

Federico Méndez reports that the conceptual model was ratified in May. He also says a later no-dissent period ended on 15 September and package-standard release was still expected.

Open on Vimeo ↗
Read the source captions

DwC-DP Integration at GBIF

sense. There are two important milestones that I want to highlight for this year that in May 26, the Darwin Core Data Package conceptual model was ratified as part of the Darwin Core standard. And then the 15th of September, that's less than a week ago, that the non-descent period after the final review just finished. Okay? And it's expected by the cohort that the executive of that week ratifies the standard and releases the standard. So we are super close now to get the standard out. If you look into a Darwin Core data package, it resembles a Darwin Core archive. It has data file similar to a Darwin Core archive. It has an EML, the same EML that you have been using for Darwin Core archives, but

The recordings document a reported change from proposal review to ratification of the conceptual model.

About this connection

These are historical statements in conference recordings. They do not independently verify the complete ratification history or imply that every implementation was in production.

Open question

Versioning remains an engineering question

2025 Living Data

GBIF says it needs a versioning strategy to accommodate changes after public review.

Watch 2025 at 48:08 ↗
Read the source captions

Implementation Status of the Darwin Core Data Package (DwC DP) at GBIF

versioning strategy for progressive improvements. We we are totally aware that this is just after the public review is the first and draft implementation of the standard. So we have to anticipate how are we going to accommodate the changes in the future. Uh point four uh we would like to implement the services and tools for comparation between both DA core archives into Darwin core data packages and in in in in cases when it's also possible how to convert from from data packages into DR archive and that will
2026 This talk

The GBIF presentation discusses how future releases of the schemas will require support for multiple versions.

Open on Vimeo ↗
Read the source captions

DwC-DP Integration at GBIF

raw data that has been published. Yep. Now, what's next for us? What are the plans now? So first, when I talk with the technical executive, it's going to release the standard, so we have to align the software components to the official release of that. We're going to enable the ingestion pipeline and also in the IPP to publish the Darwin Core data packages into the test environment. And we're going to invite users to use the test platform and to give us feedback about that. Then we're going to think of when it's a feasible time to move this to production. There's one outstanding issue. The maintenance group for Darwin Core data package is going to periodically release new versions of that, and then we need to find an engineering solution on how to keep up with different multiple versions of that. It's not something new, it's something that we have done, for example, for ABCDE standards. And then later, that's for next year. We know that we're trying to squeeze in data packages into a more strict format. So we need to start developing the addition for the new data services around richer data, and that's the work for next year. And then this is one of the dreams that we have. This is the... Don't take it as a compromise that we're going to build that, but it will be something that we would like to have. There is some kind of integrated view where you can go and see all the different

The same infrastructure is still working through the operational consequences of an evolving standard.

About this connection

A recurring concern does not establish that no versioning work happened in between; the 2026 endpoint is a test-environment presentation.

Continuing work

Validation: planned services → concrete checks

2025 Living Data

GBIF proposes validation in publishing tools and a portal service for packages assembled elsewhere.

Watch 2025 at 47:11 ↗
Read the source captions

Implementation Status of the Darwin Core Data Package (DwC DP) at GBIF

slides about that and then uh we will also we like to implement data validation tools in in mapping tools not only in the context of those tools but also as a data services that a user can access in the GB portal to validate the D core data packages that have been assembled out of the context of those tools. Erh milestone two er we we would like to mature the technical and and user documentation and that's that's the work that that has been also presented that's
2026 This talk

The later presentation explains uniqueness, primary-key and referential-integrity checks, including failure when a referenced resource is missing.

Open on Vimeo ↗
Read the source captions

DwC-DP Integration at GBIF

Once we are past the schema, we are going in to check the actual data. So in actual data checks, we are now doing unique constraint checks. So there can only be a single this value in an entire resource. We're checking primary key constraints. We are also checking this new thing with referential integrity, so that whenever you have something that points to another resource, you are actually guaranteed that it's there, else our validation will simply not pass. In addition to this, we're also supplying cell statistics, such as unique column counts. We're also having information about how many values are actually present for a given column, so that you have a bit more insight into your data about, "Was my values actually filled out?" Or what happened with them. Once we are past our validation, we then allow our metadata synchronization to

The 2026 talk supplies concrete validation behaviour for part of the earlier plan.

About this connection

This does not establish delivery of every planned service or a production rollout. The second GBIF presenter’s identity is uncertain in the recording.

This pilot covers one 2025 symposium and selected 2026 moments. Automatic captions can contain mistakes; audio has not been checked.

Overview

What this was about

Federico Mendez and a GBIF colleague (introduced as Nikolaj) described how GBIF is integrating the Darwin Core Data Package (DwC-DP) into its publishing and ingestion infrastructure. Mendez stressed that Darwin Core Archives are not going away. He explained that DwC-DP adds a Frictionless-style datapackage.json with typed fields, constraints and referential integrity, and that the IPT will support mapping and publishing DwC-DP. Nikolaj covered the stricter validation, metadata synchronisation to the registry, and a new ingestion that maps DwC-DP into GBIF's internal model, with examples from the Nice Botanic Garden herbarium. Everything is currently only in GBIF's test environment.

Why it matters. DwC-DP is the biggest change to the Darwin Core publishing model in years. How GBIF validates, ingests and displays it will decide how data publishers can express richer, relational biodiversity data.

Key ideas

In the room

  • Darwin Core Archives (core plus extensions, meta.xml and EML) will stay supported. Publishers who are happy with them can keep using them.
  • GBIF's 20-year review found that publishing around central entities (occurrences, events, taxa) was not enough for existing data or for engaging new communities.
  • The DwC-DP conceptual model was ratified in May 2026. The final review's no-dissent period ended on 15 September, and ratification and release by the TDWG Executive is expected soon.
  • A DwC-DP has data files, the same EML, and a new datapackage.json (from the Frictionless Data specification) that describes tables, fields, constraints, data types and relationships between tables.
  • The IPT will support mapping, metadata editing, validation and publishing of DwC-DP, but only in the test environment for now.
  • Validation checks the datapackage.json against its schema and against the standardised TDWG definitions, then checks unique and primary-key constraints and referential integrity, and reports statistics such as unique and populated values per column.
  • Metadata synchronisation mainly uses EML, falling back to datapackage.json where needed.
Jump in

Notable moments

In their words

Transcript

Automatically generated captions can contain mistakes, especially in names and technical terms. Times are relative to the room recording.

Read the transcript ↓
Loading transcript…
Programme abstract

DwC-DP Integration at GBIF

Federico Mendez, Mikhail Podolskiy

Following the public review of the Darwin Core Data Package (DwC-DP) standard, the GBIF Secretariat has committed to integrating DwC-DP into its existing data publishing infrastructure and ingestion pipeline. This talk presents recent engineering progress across four key areas:

IPT Extensions: The Integrated Publishing Toolkit (IPT) is being extended to handle data packages, enabling publishers to map and publish DwC-DP datasets through a familiar interface.

Data Validation: GBIF's validation services are being expanded to support DwC-DP, offering publishers meaningful quality metrics, including column population statistics, primary key uniqueness checks, referential integrity validation across schemas, and data type conformance reporting.

Registry and Metadata Synchronization: The GBIF Registry will automatically synchronize dataset metadata from Data Package descriptors, ensuring records remain accurate and up to date without manual intervention.

Ingestion Pipeline: The core ingestion pipelines are being updated to index occurrence and sampling-event records, together with their related extensions, published via DwC-DP. This work lays the foundation for broader community adoption of DwC-DP.

Together, these developments mark a significant step toward full DwC-DP support across the GBIF data publishing lifecycle, from authoring and validation to registration and indexing.

Submitted to the conference programme before the talk, so the talk itself may differ.