Artificial Intelligence Amplifies Paleontology's Reproducibility Problem
Brooke Long-Fox · AI-Readiness Metrics and Metadata for Biodiversity Data
Summaries and key moments are machine-produced. Check the recording for the speaker’s exact words.
The short versionAI can make uncertain paleontological data look clean and authoritative, so AI-ready data must also be verifiable back to evidence and interpretation.
What this was about
Brooke Long-Fox, a data curation scientist at MorphoBank and data editor for Historical Biology, argued that AI is entering a field that already has reproducibility problems of availability (data not deposited or on personal websites), documentation (no README, undefined characters) and traceability (unclear links from evidence to result). AI can turn ambiguous or incomplete sources into perfectly formatted, schema-valid tables that get DOIs and feed further AI, improving structure without improving scientific validity. Using morphological character matrices, she showed that a binary score can encode expert judgements about homology, preservation and ontogeny, and proposed adding verifiability to AI readiness: tracing assertions to evidence, recording whether they were transcribed, transformed or inferred, and who validated them.
Why it matters. It distinguishes technical validity from scientific validity, a warning relevant to any biodiversity data being mass-extracted or inferred by AI.
In the room
- This is not an argument against AI; she uses AI-assisted curation, but AI can amplify existing reproducibility problems.
- Paleontology is data-intensive; the digital extended specimen links a specimen to images, occurrences, stratigraphy, measurements, matrices and outputs, so errors propagate further.
- Availability problems: data never deposited, manuscripts claiming no data exist, files on personal websites or cloud links, private or incomplete repository records.
- Documentation problems: no README or data dictionary, unexplained abbreviations, 3D models without links to specimens, incomplete character definitions; a PID says where a file is, not what it means.
- Traceability problems: data statements not matching deposits, scores not linked to sources, matrices modified across studies without documentation, unclear versions.
- AI can produce schema-valid, DOI-bearing structured outputs from ambiguous sources (e.g. OCR column shifts, inconsistent translation), which then feed other AI systems.
- Technical validity is not scientific validity: a matrix cell such as character 37 = 0/1 may encode reasoning about homology, absence vs non-preservation, ontogeny, polymorphism or inapplicability.
Notable moments
Transcript
Automatically generated captions can contain mistakes, especially in names and technical terms. Times are relative to the room recording.
Read the transcript ↓
Related talks.
From Images to Structured Data: A Scalable AI Workflow for Natural History Museums
A simple 'just do it' AI transcription pipeline with schema prompts, an intent step and full provenance can make label digitisation scale.
Rediscovering Archives - Using LLM's for the Transcription and Extraction of Ecological Data from Historical Archives
Off-the-shelf multimodal LLMs can transcribe varied handwritten archives well, with closed models more robust on structured tables.
Biological occurrence data from historic scientific correspondence: enabling automated approaches for structured data extraction from body text
A Darwin Core-aligned annotated corpus from MfN journals provides a reference for evaluating automated occurrence extraction from historical texts.