5 posts tagged with "Accuracy"

Checking that an AI's quote really appears in a PDF sounds like pressing Ctrl+F. In practice the text inside a PDF is not the text you see: words are hyphenated across lines, paragraphs are split by page breaks and running headers, and letter pairs like "fi" are stored as single symbols. A quote check that ignores this either rejects real quotes or accepts paraphrases. Getting it right is most of the work.

We said every quote in MindMax is checked against the source. This post is about why that sentence hides more than it shows — and what you should expect from any tool that claims the same.

Asking AI about a screenshot of a chart works, but it throws away the text under the picture, the page it came from and the document around it. Boxing a region inside the document keeps all three, so numbers are read from the text rather than guessed from pixels, and the answer links back to the page.

Everybody has the same workaround. You hit a figure you cannot parse, you screenshot it, you paste it into a chat window, you type "what is this saying". It works often enough to become a habit.

It also fails in a specific, repeatable way, and the failure is not the model being bad at pictures. It is that by the time the picture arrives, three things that were sitting right next to it have been thrown away.

Every few days someone who has built their own retrieval setup asks the same question about MindMax, and it is a good question: if there is no embedding pass over my folder, how does it find anything?

The short answer is that it finds material the way you would — by structure, by the words on the page, and by trying other wordings — and that documents small enough to read are read rather than retrieved from. The longer answer is that this is a trade with a real cost, and pretending otherwise would be the kind of claim that falls apart the first time someone tests it.

To chat with multiple PDFs reliably, ask questions that name what you want compared, require every claim to say which document it came from and quote it, and look at which documents the answer actually read. The common failure is not a wrong sentence — it is a source that was never looked at, and an answer that sounds complete anyway.

One PDF is the easy case. The questions worth asking usually span ten: which of these papers use the method, where do these reports disagree, what does the contract say that the summary left out.

AI invents citations because, to a language model, a reference is just more text that should look plausible — and a plausible title, author and page number are easy to produce without any document behind them. The check that catches most of it is simple: require the exact quoted words, and match them against the source text before trusting the claim.

That is a mechanical check, so software can do it. Most tools do not, and the gap is where the embarrassing stories come from.