LLM Evaluation · Regulatory AI
CRL-Bench
A benchmark pipeline that extracts deficiencies from 448 real FDA Complete Response Letters and checks whether model-cited regulations and guidance actually exist and apply.
- Context
- LLM benchmark over FDA Complete Response Letters · two-person project, I built the pipeline
- Stack
- Python, LLM evaluation, Hybrid retrieval
How it works
Recorded terminal session, replayedRecorded terminal session, replayed (the test run is sped up 2.5x). The typed commands and prompt are decoration; the output is the logged output of the real run. No benchmark result is shown: the reference counts are an inventory of citations in the letters, not a model score.
Two-person project with Anish Indurthy.
List the commands
uv run crl-bench --help prints the benchmark's commands.
Run the checks
make verify runs the linter, formatter, type checker, schema check and test suite: 1,493 passed and 12 skipped, 92% coverage.
Resolve the cited references
crl-bench references letters resolves the citations in 448 letters: 2,895 references, of which 2,587 resolved, 27 ambiguous and 281 unresolved.
Problem
When the FDA declines to approve a drug, it sends a Complete Response Letter listing deficiencies. A model that drafts a response has to find each deficiency, cite the right regulation or guidance, and not invent one. I wanted a benchmark that checks whether a model's citations exist, apply on the letter's date and support its plan.
My contribution
This is a two-person project. I wrote the pipeline: ingestion, extraction, the citation resolver, retrieval, the experiment runner and the scoring tooling. My teammate Anish Indurthy verifies each change and is the second annotator.
Approach
448 letters come from openFDA with https-only downloads, SHA-256 hashes and atomic writes. Text comes from pypdfium2 with a Tesseract OCR fallback. A point-in-time resolver maps every cited reference to a catalog of 2,792 FDA guidance records, the 21 CFR parts in force on the letter's date and 90 ICH guidelines. Retrieval combines BM25, BGE-M3 dense embeddings and reciprocal rank fusion over 1,713 documents and 63,760 chunks. The runner caches each trial by content hash, keeps a cost ledger, refuses paid models without an explicit budget and keeps the test split locked until the gold labels are frozen.
How it was built
I wrote the pipeline in Python: openFDA ingestion with hashed downloads, pypdfium2 extraction with an OCR fallback, a point-in-time resolver that checks every CFR, ICH and FDA guidance citation, and BM25 plus BGE-M3 hybrid retrieval. A resumable runner caches trials by hash and locks the test split until gold labels are frozen. My teammate verifies each change and is the second annotator. Limits: there are no gold labels yet, so no benchmark result exists.
Evidence and limitations
Extraction covered 2,361 pages, 364 of them OCR'd, and recovered 2,504 FOIA redaction markers against 630 in openFDA's own text. Of 2,895 cited references, 2,587 resolved, 27 were ambiguous, 281 unresolved and none was verified invalid.
A pilot with a local Qwen2.5-14B on 10 development letters produced schema-valid output on the first attempt every time (T1 10 of 10, T2 and T3 49 of 49), with 81.6% of deficiency quotes verbatim (40 of 49). The resolver caught one CFR citation that does not exist.
Limits: there are no benchmark results yet. Gold labels have not been annotated, so no model comparison, retrieval score or verifier threshold is reported, and the pilot is 10 letters of development data.