TDWG2026
Home / Talks / SAL A
talk · Tuesday 22 September · 16:25–⁠16:35 · SAL A

Multimodality for Knowledge Extraction from Historical Entomology Literature

Jana Hoffmann, Sefika Efeoglu · BHL at 20: A New Chapter for Biodiversity Literature and Data - Transforming BHL: Data Extraction, Semantic Navigation and Future Interfaces

Recording time 7:58:30–8:07:25Open on Vimeo ↗

Summaries and key moments are machine-produced. Check the recording for the speaker’s exact words.

The short versionAligning text and illustrations in a multimodal knowledge graph can generate evidence-grounded captions and recover knowledge lost by text-only extraction.

Overview

What this was about

Jana Hoffmann and Sefika Efeoglu of Senckenberg's new Semantic Data Systems and Knowledge Engineering group presented a prototype linking text and illustrations in historical entomological literature, using Maria Sibylla Merian's 1679 'Der Raupen wunderbare Verwandlung' (50 plates, digitised and published on Zenodo, soon in BHL). The workflow uses olmOCR for text, an ontology of concepts, detection of visual entities, relations and attributes in illustrations, annotation of text entities using those visual entities, merging into a multimodal sub-knowledge graph, and verbalisation into evidence-grounded captions, since historical plates often lack captions. Next steps are scaling to all 50 articles, provenance, and expert human-in-the-loop validation.

Why it matters. Illustrations carry complementary morphological, life-stage and interaction evidence that text-only pipelines ignore.

Key ideas

In the room

  • Text describes organisms, host plants and feeding interactions; illustrations show morphology, developmental stages and spatial relations.
  • Merian volume from the Senckenberg German Entomological Institute library in Müncheberg.
  • Generated captions are restricted to what is supported by the graph; freer generation adds hallucinated content.
  • Aim to provide a reusable workflow for other historical sources and downstream BHL uses.
Jump in

Notable moments

In their words

Transcript

Automatically generated captions can contain mistakes, especially in names and technical terms. Times are relative to the room recording.

Read the transcript ↓
Loading transcript…
Keep exploring

Related talks.

See the map ↗
talk · 22 Sept · 16:05–⁠16:15 · SAL A

Biological occurrence data from historic scientific correspondence: enabling automated approaches for structured data extraction from body text

A Darwin Core-aligned annotated corpus from MfN journals provides a reference for evaluating automated occurrence extraction from historical texts.

Christian Bölling
talk · 22 Sept · 14:40–⁠14:55 · SAL A

BHL and the Planetary Knowledge Base: Structuring Biodiversity Literature for the Next Generation

NHM is re-OCRing all of BHL with modern models and extracting entities into a knowledge graph, with confidence gating to protect BHL data quality.

Qianqian Hiris Gu
talk · 22 Sept · 16:15–⁠16:25 · SAL A

A closer look at steps needed to extract research-ready biodiversity data from legacy literature

Legacy literature data can only be unlocked by combining better OCR/NLP with taxonomic rigour that accounts for synonymy, shifting concepts, coreference and mixed languages.

Gerwin Kasperek
Programme abstract

Multimodality for Knowledge Extraction from Historical Entomology Literature

Jana Hoffmann, Sefika Efeoglu

Historical entomological publications combine textual descriptions and scientific illustrations that contain complementary biodiversity knowledge. While texts describe insect taxa, host plants, and ecological interactions, illustrations capture life cycles, morphological traits, and species interactions. Existing biodiversity knowledge extraction approaches mainly focus on text and rarely integrate visual information from historical illustrations.

We propose a multimodal knowledge graph for representing entities, traits, and ecological interactions extracted from historical entomological literature. As a case study, we analyze 50 articles and illustrations from Maria Sibylla Merian’s Der Raupen wunderbare Verwandelung, und sonderbare Blumennahrung (1679). Our workflow combines OCR, semantic annotation, image-region linking, and knowledge graph construction to align textual and visual information. The workflow includes: (i) extracting text with OCR, (ii) defining an annotation ontology for concepts and events in illustrations, (iii) annotating articles with these concepts, and (iv) linking illustration regions to corresponding text passages.

The workflow generates semantic triples from text and illustrations, e.g., {(tulip, instance_of, petal), (petal, has_color, purple), (caterpillar, has_shape, elongated)}. Beyond visual traits, extracted relations represent biologically relevant information such as insect-host plant associations and developmental stages. The resulting knowledge graph supports semantic enrichment and automated caption generation for historical biodiversity collections.

Submitted to the conference programme before the talk, so the talk itself may differ.