Extracting data from PDFs with AI: how to verify what it reads

A system that reads an invoice and returns ten filled-in fields makes a good impression. The problem comes later: across two thousand documents, how do you know which ten fields are wrong? Automatic extraction isn't judged by a successful demo, but by what happens when it fails silently. This guide helps you design the checks before you let that data get written anywhere.

Fields first, then the tool

The first decision isn't technological: it's the list of fields you need, with the format and the rule for each one, written down before trying any tool.

• Which fields. Not "everything in the document": only the fields someone will actually use. Every extracted field is a field to check.
• What format. A date is a specific day, not a line of text; an amount has a currency and decimals; a code has a length. Without a format, there's no automatic check.
• Which documents are allowed. A file generated by another system is read one way, a crooked scan of a stamped sheet is a different problem. Saying "we accept anything" means having no quality threshold.
• What to do when a field isn't there. This is the most frequent case and the least designed for: the document is valid but that piece of data doesn't appear. The answer can't be a value guessed by intuition.

If you can't write this list, the problem isn't artificial intelligence: it's that the downstream process isn't defined and no one knows what will be done with the data.

Three different operations called "reading"

The same sentence often lumps together three things that fail differently and need to be checked differently.

• Text recognition: turning an image into characters. Here the errors are physical — an eight read as a three, a line cut off by the fold of the page, a part outside the frame. A document born digital skips this step and is far more reliable.
• Extraction: deciding which, among the numbers present, is the total, and which of the three dates is the issue date. Here the errors are interpretive, and they're the most dangerous because the result is always plausible.
• Checking: establishing whether the extracted data is consistent with the rest. It's the only one of the three steps that can notice something doesn't add up, and it's the one almost no one designs for.

A system that skips the third step isn't automated: it has simply shifted the checking onto whoever receives the data, without telling them.

The field table

The most useful project document is a table with one row per field. An example, based on an invoice received from a vendor.

The last two columns are the real guide. A field with no check and no destination in case of error is a field no one will ever verify.

The checks you write by hand

The useful checks aren't statistical: they're arithmetic and reference-based, and that's exactly why they're reliable.

• Internal consistency: the totals add up, the line items sum correctly, the percentages recalculate.
• Consistency with what you already know: the vendor exists, the order number exists, that customer really has that address.
• Plausibility: a date from last century, an amount off by three orders of magnitude, a negative quantity where it can't be one.
• Duplicates: the same document uploaded twice is a more common error than a badly read field, and a more costly one.

When a check fails, it doesn't get corrected automatically. It's sent for review: correcting on its own a piece of data the system has already shown itself uncertain about is the most direct way to create invisible errors.

Where human review fits in

The realistic goal isn't to remove the person: it's to have them look at a few documents instead of all of them, and know which ones.

1. Documents that pass every check move forward, along with their trail.
2. Documents with a failed check go into a review queue, with the problem field highlighted and the original document alongside: whoever corrects it shouldn't have to search.
3. Blocking documents — a total that doesn't add up, a possible duplicate — never proceed, not even when there's a rush.

The review queue should be read as an indicator. If it keeps growing around the same type of document or the same vendor, the solution is almost never to improve the extraction: it's to ask for that document in a different format.

Preserving provenance

For every piece of data written into a system, you must be able to trace back where it came from. It isn't bureaucracy: it's the condition for being able to dispute, correct, and understand it.

• The original document, kept and linked to the data.
• The point in the document the value was extracted from, so it can be found without rereading everything.
• If the value was corrected by hand: by whom, when, and what the previous value was.
• Which checks were run and with what outcome.

Without these four pieces of information, when a number turns out to be wrong a year from now, you won't know whether the error is in the reading, the correction, or the original document.

What this guide doesn't cover

Here we cover assisted extraction: what can be read from a document and how it's verified. There's no configuration that guarantees total accuracy, and no extracted data should enter the books without the checks described here. The case of a system that doesn't just read but writes or modifies data in a business management system is a different problem, one of permissions and allowed actions, with its own guide; the one-time transfer of data from spreadsheets or an old system also follows its own method.