The corpus, document by document

This is the whole regression corpus, not a selection of the good cases: every document the engine is tested against, what came out of it, and what is still wrong with the output. It runs on every pull request, so a table like this is the difference between "we care about correctness" and a sentence claiming it.

How to read a row​

Pages, headings, tables, links are what came out: the page count of the PDF, and how many of each structure the Markdown contains. They say nothing about whether those are the right ones — a table detected where the PDF has none is counted here like any other.

Checks is the number of assertions that document carries, and how many pass. An assertion is a fact a person verified by opening the PDF: this line is a heading, this running header must not appear, there are at least five tables, the equations on these pages must be reported as flattened. They are written by hand, one document at a time, which is why there are 120 of them and not 10,000.

Open is the count of known defects, and it is the column that makes the rest worth reading. Each one is a todo in the manifest: an assertion describing the correct output, expected to fail today. The document's name links to the PDF, so any row here can be checked without trusting us.

What this table does not measure​

• Structural accuracy. There is no ground-truth document tree behind these numbers, so there is no F1 score to quote. Counting headings and tables is not the same as knowing that each one is correct; the assertions are the only correctness signal here, and they cover what a person looked at, not every line.
• Your PDFs. The corpus is chosen by us, from failure modes we hit — two-column papers, CJK manuscripts, tagged government documents, a scan with an OCR text layer. It is not a random sample of the PDFs in the world, so it cannot tell you that your file will convert. It tells you exactly what happens on these 22.
• Snapshot quality. golden/<id>.md is what the engine produces today, not what it should produce. It catches change; the assertions catch wrongness.
• OCR. One document (NACA Report 614) is a scan whose text layer came from someone else's OCR. It is here to check that the engine reports what it finds instead of inventing structure, not to measure OCR quality.
• Other converters. Nothing here runs a competing tool. Comparing would be a different table, with the other tool's version pinned and its settings stated.

The defects we know about​

17 failing checks across 10 documents. They are listed in public because a converter with a defect list is easier to trust than one that claims to have none: these are the cases where someone looked at the output, saw it was wrong, and wrote down what right would look like.

Each one is expected to fail, so CI stays green while they are open. When a fix makes one pass, the runner says so by name and the assertion moves from todo to expect — the defect cannot be fixed quietly, and it cannot be forgotten either.

• Mamba: Linear-Time Sequence Modeling (1)
  • should not appear: "### 𝐴"
• Mistral 7B (1)
  • missing whole line: "| Parameter | Value |"
• LLaMA: Open and Efficient Foundation Language Models (1)
  • missing whole line: "| Dataset | Sampling prop. | Epochs | Disk size |"
• 3D modelling of orbits in wide binary gravity tests (2)
  • not a heading: "1. INTRODUCTION"
  • should not appear: "[1. INTRODUCTION]("
• Cutoff for simple random walk on preferential attachment graphs (1)
  • tables 22 > 0
• Luo's spectral large sieve inequality (3)
  • should not appear: "#### ÿ"
  • should not appear: "#### ˇ"
  • should report garbled text, did not
• Bilingual manuscript: English then Chinese (2)
  • should not appear: "尺寸 持续缩小"
  • should not appear: "\n\n。瞬态平面热源法"
• NIST SP 800-63B, Digital Identity Guidelines (2)
  • not a heading: "5.1.1 Memorized Secrets"
  • should not appear: "###### 5 Authenticator and Verifier Requirements"
• USGS Fact Sheet 2018-3025 (1)
  • should not appear: "now more the Lower Elwha Klallam Tribe"
• State Council Gazette of China, 2025 No. 35 (3)
  • should not appear: "�"
  • should not appear: "!!第一条"
  • should report garbled text, did not

Read them literally and they are legible: should not appear: "### 𝐴" means a stray mathematical glyph was promoted to a heading, tables 22 > 0 means a page of theorems was read as 22 tables, and should report garbled text, did not means the health check missed fonts it should have flagged.

Reproduce it​

The PDFs are not in the repository — most licences cover redistributing the converted text, not the file. The runner downloads them by URL and refuses a file whose sha256 does not match the manifest.

git clone https://github.com/woodpeng/mindmax-pdf-markdown
cd mindmax-pdf-markdown && pnpm install
pnpm eval --fetch      # first run only: fetch the 22 PDFs (~82 MB), sha256 checked
pnpm eval              # convert and compare; exit code 1 on a regression

It ends with a line you can compare against this page, and it needs Node 22 (the runner drives pdf.js 6, which uses Promise.withResolvers; the library itself runs on Node 18+):

ran 22 · 0 assertion failures · 0 differ from snapshot · 10 open todos

Adding --report eval/results.json writes the table on this page as JSON, and this page renders that file — nothing in the table above is typed by hand. The JSON is published so you do not have to take our word for any number here: eval/results.json.

If your own PDF converts badly, that is how the corpus grows: the corpus case template asks for the document, or a page you have the right to share, and the failing assertion gets written first.