3 min read
Accuracy is the wrong number to ask for
When people evaluate a document-extraction tool they ask how accurate it is, and get an answer like 95%. It sounds decisive and tells you almost nothing useful.
The reason is simple: if you cannot identify which fields are in the failing 5%, you have to check all of them. A system that is 95% accurate and opaque generates exactly as much review work as one that is 60% accurate and opaque. The accuracy number only becomes valuable when paired with a second number telling you where to look.
What per-field confidence actually means
Rather than emitting a document's worth of values and a single overall score, the system emits a score per extracted field. The invoice total comes with its own confidence; so does the supplier name, so does each line item.
That changes the work fundamentally. Instead of reviewing every document, a reviewer sees only the fields that fell below a threshold. On a typical batch that is a small fraction of the total, and it is the fraction that actually needed a human.
The score should reflect real signals, not a number the model was asked to invent: whether the value was found where the schema expected it, whether OCR was confident about the characters, whether the value parses as the type it should be, whether cross-field arithmetic agrees.
Why template and regex approaches break
The traditional approach defines a template per document layout: the total is in this region, the date matches this pattern. It works until the next supplier sends a differently shaped invoice, and then it fails — often silently, extracting the wrong number from the right position.
Schema-driven extraction inverts this. You describe what you want (an invoice has a total, a date, a supplier, line items) rather than where it sits, and the system finds it wherever it appears. New layouts stop being breaking changes. OCR fallback covers the scanned documents where there is no text layer at all.
Choosing the review threshold
The threshold is a business decision, not a technical one, and it is set by the cost of being wrong.
For a field where an error is embarrassing but recoverable, a low threshold is fine — review only what the system is genuinely unsure about. For a field that moves money or triggers a contractual obligation, set it high enough that a reviewer sees anything remotely uncertain, and accept the extra review load as the price of not wiring an error into your accounts.
The mistake is picking one threshold for the whole document. Different fields carry different consequences, and the threshold should follow the consequence.
Human-in-the-loop is a design, not an admission
Teams sometimes treat a review step as evidence the automation failed. It is the opposite: it is what makes the automation safe to deploy at all.
The useful framing is throughput per reviewer, not percentage automated. A system that routes 8% of fields to a person and gets the other 92% right without supervision has removed most of the work while keeping a human accountable for the cases that need judgement. A system that claims 100% automation has simply moved the errors somewhere you will find them later.
What to check before you buy or build
Whether you are evaluating a vendor or scoping a build, these questions separate a demo from a system.
- Does it score every field, or only the document as a whole?
- What is the score derived from — real signals, or the model's own assertion of confidence?
- Can thresholds differ per field, so high-consequence values get more scrutiny?
- What happens to a low-confidence field: is there a review queue, or does it silently pass through?
- Can it fall back to OCR when a PDF has no text layer?
- Can it run on your own infrastructure if the documents cannot leave it?