What they did
The authors created a dual-layer knowledge graph for pharmaceutical process development. Its base layer preserves documents in a Document–Section–Chunk hierarchy, including digital, scanned, handwritten and multilingual material; its intelligence layer extracts ontology-aligned entities and connects concepts across documents and projects while retaining links to source passages.
LLM agents route questions to the appropriate retrieval path, from lexical lookup to domain-graph reasoning. The authors evaluated the lexical layer using a three-tier protocol and 505 questions curated from 38 proprietary development reports covering a Sanofi small-molecule program.
Key findings
- Tier-1 multiple-choice accuracy reached 95%, indicating reliable retrieval and answer selection on the curated test set.
- Under the stricter Tier-2 evaluation by an LLM judge, the pass rate fell to 85%.
- Comparative and corpus-wide questions produced more failures than simpler lookups, exposing weaknesses that multiple-choice accuracy did not capture.
- A router agent selected between the lexical and ontology-based graph layers according to question type.
Why it matters
CMC knowledge is commonly dispersed across incompatible documents, making technology transfer, investigation and regulatory preparation labor-intensive. The architecture offers a way to preserve source-level traceability while also supporting structured queries across documents, and the evaluation protocol provides a practical template for testing retrieval systems when the underlying industrial corpus cannot be released publicly.
Caveats
The reported benchmark covers 38 reports from a single proprietary small-molecule program, so performance may not generalize to other projects, organizations or document distributions. The quantitative results primarily evaluate the lexical layer rather than the full domain graph and routing system, and the Tier-2 score depends on an LLM judge whose calibration and reproducibility are not described in the supplied material.