Historical-text OCR · IMPACT-BHL ground truth

BHL OCR Leaderboard

How well OCR and vision-language models transcribe 18th–19th-century book pages — a broken-out scorecard, not a single grade. Ranked on micro-averaged character error rate, read as confidence intervals, not ranks.

The test material is a ground truth made by hand long before today’s models: 2,165 pages from six natural-history volumes, expert-transcribed for the EU’s IMPACT digitisation programme (with BHL-Europe) in English, German, French and Latin. This board reuses that older ground truth to ask a current question — which of the new models actually reads old books best — and breaks the answer out, because models fail differently: some misread letters, some invent text on near-empty pages, some cannot stop repeating themselves.

clara intra Philoſophiæ Thalamos admiſsâ luce → clara intra philosophiæ thalamos admissâ luce one line of the ground truth as each lane scores it — diplomatic keeps the long ſ, reading folds it · Piscium querelae, 1708

Type

Size

Reads

click a row for per-language CER

Model ▾	Type i	Params i ▾	CER · reading i ▾	CER · dip i ▾	Content CER i ▾	Sparse CER i ▾	Recall i ▾	Over-x i ▾	Loop % i ▾	Reads i

Sparse and blank pages are where models diverge most: strong content OCR can coexist with runaway hallucination on near-empty pages, and the aggregate CER shows it.

Size vs. accuracy

params (log) vs CER · lower-left is better

Reading the board. Where 95% bootstrap CIs overlap, differences are within noise — that overlap is the finding, not a defect. CIs resample volumes (six books), so they are wide by design.

A page can end three ways. Scored; looped (hit its token cap mid repetition loop — excluded from the row's aggregates and counted in Loop %); or errored (the producer failed — any error makes the row ineligible for ranking under fail-closed scoring).
CER · reading folds long-s/case/ligatures (reading ability); CER · dip keeps them (diplomatic fidelity). Recall is format-immune coverage, read co-equally with CER.
Reads is a policy, not a quality score: whether a model emits page furniture (running heads, page numbers). The body-only headline neither rewards nor penalizes it.
Params are measured, not quoted. Every weight in the model repo's safetensors index, summed across dtypes, so vision towers count. Names mislead: GLM-OCR is 1.33B, not the 0.9B it is usually listed as, and olmOCR-2-7B is 8.29B. Classical engines (Tesseract, PP-OCRv6) have no comparable count and are omitted from the scatter.
We ran the inference ourselves, for every model on the board — a pinned container image, a pinned model revision, a pinned script commit, and the id of the job that produced each score. A hosted inference router does not record which provider served a request, at what quantization, or under what serving configuration — and CER is sensitive to all three — so routed scores are not admissible here.
Held for v1.1: surya-ocr-2 and PP-OCRv6 join the board once their runs meet the same bar.
Docs & data: bhl-ocr-eval — methodology in DESIGN.md, plain-language guide in GLOSSARY.md, findings and caveats in RESULTS.md. Ground truth: finebooks/bhl-impact-gt (IMPACT Centre of Competence / BHL-Europe, CC-BY).