# PDF → Markdown corpus results, document by document URL: https://mindmax.me/open-source/pdf-markdown/benchmark/ Last verified: 2026-11-09 > Every document the open-source PDF engine is tested against, what came out of it, the assertions a person verified against the PDF, and the defects still open. ## The corpus, document by document This is the whole regression corpus, not a selection of the good cases: every document the [engine](https://mindmax.me/open-source/pdf-markdown/) is tested against, what came out of it, and what is still wrong with the output. It runs on every pull request, so a table like this is the difference between "we care about correctness" and a sentence claiming it. | | | | --- | --- | | Documents | 22 real PDFs, 612 pages | | Assertions | 120 of 120 passing — facts about the output that a person checked against the PDF | | Known defects | 17, in 10 documents, listed below | | Languages | 20 English, 1 bilingual, 1 Chinese | | Licences | 16 CC BY 4.0 · 5 US federal public domain · 1 PRC official document | | Engine | @mindmax/pdf-markdown v0.1.0, canonical text v1 | | Document | Lang | Pages | Headings | Tables | Links | Checks | Open | | --- | --- | --- | --- | --- | --- | --- | --- | | Chain-of-Thought Prompting Elicits Reasoning in LLMsarxiv-2201.11903-cot | en | 24 | 32 | 12 | 77 | 8/8 | — | | Mamba: Linear-Time Sequence Modelingarxiv-2312.00752-mamba | en | 37 | 65 | 21 | 3 | 5/5 | 1 | | Mistral 7Barxiv-2310.06825-mistral | en | 9 | 13 | 4 | 2 | 5/5 | 1 | | LLaMA: Open and Efficient Foundation Language Modelsarxiv-2302.13971-llama | en | 27 | 44 | 18 | 30 | 9/9 | 1 | | Constitutional AI: Harmlessness from AI Feedbackarxiv-2212.08073-constitutional-ai | en | 34 | 47 | 5 | 0 | 4/4 | — | | Phi-3 Technical Reportarxiv-2404.14219-phi3 | en | 12 | 10 | 2 | 0 | 5/5 | — | | 3D modelling of orbits in wide binary gravity testsarxiv-2610.40388-wide-binaries | en | 12 | 6 | 2 | 17 | 8/8 | 2 | | GæniceN: relaxing magnetization states self-consistentlyarxiv-2610.37889-magnetization | en | 7 | 11 | 5 | 0 | 5/5 | — | | Causal framework for autonomous mobile robotsarxiv-2611.04027-ieee-ral | en | 12 | 26 | 2 | 0 | 4/4 | — | | Annotator-free object detectionarxiv-2611.06195-detection | en | 12 | 27 | 5 | 0 | 5/5 | — | | Cutoff for simple random walk on preferential attachment graphsarxiv-2611.00717-random-walk | en | 49 | 13 | 22 | 0 | 4/4 | 1 | | Mapping AI economic complexityarxiv-2609.17200-ai-complexity | en | 8 | 17 | 2 | 4 | 5/5 | — | | Mismatch-tolerant high-gain OTAarxiv-2610.39699-ota | en | 6 | 8 | 1 | 0 | 8/8 | — | | Speech brain-computer interfaces (review)arxiv-2609.36736-speech-bci | en | 36 | 68 | 1 | 1 | 5/5 | — | | Luo's spectral large sieve inequalityarxiv-2608.29558-math-fonts | en | 17 | 81 | 42 | 0 | 4/4 | 3 | | Bilingual manuscript: English then Chinesearxiv-2609.27618-bilingual | zh+en | 76 | 103 | 9 | 2 | 6/6 | 2 | | NIST SP 800-63B, Digital Identity Guidelinesnist-sp800-63b | en | 80 | 44 | 19 | 69 | 4/4 | 2 | | The Constitution of the United Statesus-constitution | en | 85 | 45 | 72 | 0 | 6/6 | — | | Federal Register: Airworthiness Directives (Airbus)federal-register-2023-28802 | en | 4 | 14 | 0 | 21 | 6/6 | — | | USGS Fact Sheet 2018-3025usgs-fs-2018-3025 | en | 4 | 11 | 7 | 0 | 4/4 | 1 | | NACA Report 614 (scanned, OCR text layer)naca-19930091690-ocr-scan | en | 20 | 10 | 32 | 0 | 3/3 | — | | State Council Gazette of China, 2025 No. 35gov-cn-gazette-2025-35 | zh | 41 | 60 | 1 | 0 | 7/7 | 3 | | All 22 documents | | 612 | 755 | 284 | 226 | 120/120 | 17 | ## How to read a row[​](https://mindmax.me/#how-to-read-a-row) **Pages, headings, tables, links** are what came out: the page count of the PDF, and how many of each structure the Markdown contains. They say nothing about whether those are the _right_ ones — a table detected where the PDF has none is counted here like any other. **Checks** is the number of assertions that document carries, and how many pass. An assertion is a fact a person verified by opening the PDF: this line is a heading, this running header must not appear, there are at least five tables, the equations on these pages must be reported as flattened. They are written by hand, one document at a time, which is why there are 120 of them and not 10,000. **Open** is the count of known defects, and it is the column that makes the rest worth reading. Each one is a `todo` in the manifest: an assertion describing the correct output, expected to fail today. The document's name links to the PDF, so any row here can be checked without trusting us. ## What this table does not measure[​](https://mindmax.me/#what-this-table-does-not-measure) - **Structural accuracy.** There is no ground-truth document tree behind these numbers, so there is no F1 score to quote. Counting headings and tables is not the same as knowing that each one is correct; the assertions are the only correctness signal here, and they cover what a person looked at, not every line. - **Your PDFs.** The corpus is chosen by us, from failure modes we hit — two-column papers, CJK manuscripts, tagged government documents, a scan with an OCR text layer. It is not a random sample of the PDFs in the world, so it cannot tell you that your file will convert. It tells you exactly what happens on these 22. - **Snapshot quality.** `golden/.md` is what the engine produces today, not what it should produce. It catches change; the assertions catch wrongness. - **OCR.** One document (NACA Report 614) is a scan whose text layer came from someone else's OCR. It is here to check that the engine reports what it finds instead of inventing structure, not to measure OCR quality. - **Other converters.** Nothing here runs a competing tool. Comparing would be a different table, with the other tool's version pinned and its settings stated. ## The defects we know about[​](https://mindmax.me/#the-defects-we-know-about) 17 failing checks across 10 documents. They are listed in public because a converter with a defect list is easier to trust than one that claims to have none: these are the cases where someone looked at the output, saw it was wrong, and wrote down what right would look like. Each one is expected to fail, so CI stays green while they are open. When a fix makes one pass, the runner says so by name and the assertion moves from `todo` to `expect` — the defect cannot be fixed quietly, and it cannot be forgotten either. - **Mamba: Linear-Time Sequence Modeling** (1) - `should not appear: "### 𝐴"` - **Mistral 7B** (1) - `missing whole line: "| Parameter | Value |"` - **LLaMA: Open and Efficient Foundation Language Models** (1) - `missing whole line: "| Dataset | Sampling prop. | Epochs | Disk size |"` - **3D modelling of orbits in wide binary gravity tests** (2) - `not a heading: "1. INTRODUCTION"` - `should not appear: "[1. INTRODUCTION]("` - **Cutoff for simple random walk on preferential attachment graphs** (1) - `tables 22 > 0` - **Luo's spectral large sieve inequality** (3) - `should not appear: "#### ÿ"` - `should not appear: "#### ˇ"` - `should report garbled text, did not` - **Bilingual manuscript: English then Chinese** (2) - `should not appear: "尺寸 持续缩小"` - `should not appear: "\n\n。瞬态平面热源法"` - **NIST SP 800-63B, Digital Identity Guidelines** (2) - `not a heading: "5.1.1 Memorized Secrets"` - `should not appear: "###### 5 Authenticator and Verifier Requirements"` - **USGS Fact Sheet 2018-3025** (1) - `should not appear: "now more the Lower Elwha Klallam Tribe"` - **State Council Gazette of China, 2025 No. 35** (3) - `should not appear: "�"` - `should not appear: "!!第一条"` - `should report garbled text, did not` Read them literally and they are legible: `should not appear: "### 𝐴"` means a stray mathematical glyph was promoted to a heading, `tables 22 > 0` means a page of theorems was read as 22 tables, and `should report garbled text, did not` means the health check missed fonts it should have flagged. ## Reproduce it[​](https://mindmax.me/#reproduce-it) The PDFs are not in the repository — most licences cover redistributing the converted text, not the file. The runner downloads them by URL and refuses a file whose sha256 does not match the manifest. ` git clone https://github.com/woodpeng/mindmax-pdf-markdown cd mindmax-pdf-markdown && pnpm install pnpm eval --fetch # first run only: fetch the 22 PDFs (~82 MB), sha256 checked pnpm eval # convert and compare; exit code 1 on a regression ` It ends with a line you can compare against this page, and it needs Node 22 (the runner drives pdf.js 6, which uses `Promise.withResolvers`; the library itself runs on Node 18+): ` ran 22 · 0 assertion failures · 0 differ from snapshot · 10 open todos ` Adding `--report eval/results.json` writes the table on this page as JSON, and this page renders that file — nothing in the table above is typed by hand. The JSON is published so you do not have to take our word for any number here: [eval/results.json](https://raw.githubusercontent.com/woodpeng/mindmax-pdf-markdown/main/eval/results.json). If your own PDF converts badly, that is how the corpus grows: the [corpus case template](https://github.com/woodpeng/mindmax-pdf-markdown/issues/new?template=corpus_case.yml) asks for the document, or a page you have the right to share, and the failing assertion gets written first. - [How to read a row](https://mindmax.me/#how-to-read-a-row) - [What this table does not measure](https://mindmax.me/#what-this-table-does-not-measure) - [The defects we know about](https://mindmax.me/#the-defects-we-know-about) - [Reproduce it](https://mindmax.me/#reproduce-it)