How to keep hallucinations out of a document pipeline: grounding, citations, confidence thresholds

When a document AI confidently states something that is not in the source, that is a hallucination, and in a document pipeline it is the failure mode that matters most. The fix is not a better prompt. It is structural. Keep the model grounded in retrieved source text, force it to cite where each answer came from, attach a confidence signal to every extracted field, and route anything below your threshold to a person. Together these four controls turn a model that can invent answers into a system that either reports what the document says or admits it does not know. None of them are exotic. They are the difference between a demo that reads well and a pipeline you can put in front of an auditor.

What causes a document AI to hallucinate?

A language model predicts plausible text. Left to answer from memory, it will fill gaps with something that sounds right rather than something that is in your document. In a pipeline this shows up in predictable places: a total the model inferred instead of read, a date that matches the wrong invoice, a supplier name that is close but wrong, a field reported as present when the page was blank. The model is not lying. It was never shown the constraint that its answer must come from the source in front of it. Most production techniques exist to impose exactly that constraint.

How does retrieval grounding reduce hallucinations?

Grounding means the model only answers from text you have actually given it, not from its training. Instead of asking the model what the total is on an invoice, you retrieve the relevant passages from the parsed document and ask it to answer using only those passages. If the answer is not in the supplied text, the correct output is "not found", not a guess.

In practice this means a few things working together:

1. Parse before you prompt. Extract the document into clean, structured text with layout preserved, so the model reads real content rather than a blurry image.
2. Retrieve the relevant span. For a long document, pull the section that should contain the field and pass that, rather than dumping the whole file and hoping.
3. Constrain the instruction. Tell the model plainly that it must answer only from the provided text and return an explicit empty value when the field is absent.
4. Pin to the schema. Ask for structured output with named fields, so there is nowhere to add commentary or invent extras.

Grounding alone removes a large share of invented answers. The model can still misread, which is why the next two controls exist. We rely on this approach heavily when building Pileform, our AI bookkeeping tool, because a fabricated number in someone's accounts is not a cosmetic error.

Why should every answer carry a citation?

A citation ties each extracted value back to the exact place in the source it came from: a page, a line, a bounding box, a character span. This does two jobs at once.

First, it makes the answer checkable. A reviewer, or an automated rule, can jump straight to the spot on the page and confirm the value in seconds instead of reading the whole document. Second, it raises the cost of hallucination for the model itself. When the output format requires a location for every field, an answer with no supporting span is a visible defect you can detect and reject automatically. A value that cannot point to where it lives in the document is treated as not found.

The practical rule is simple. If a field has no citation, it does not get written to the record. That single constraint catches a category of confident but unsupported answers before they ever reach your books.

How do confidence thresholds and human review work together?

Confidence thresholds decide which answers the system trusts on its own and which it escalates. Every extracted field carries a score. You set a cut-off. Above it, the value flows straight through. Below it, the value is held and sent to a person, with the original document and the proposed answer side by side. This is the human-in-the-loop step, and it is what makes the whole pipeline safe to automate.

Confidence	What the system does	Who acts
High	Accept and post the value automatically	No one
Borderline	Flag for quick confirmation	Reviewer glances, approves or corrects
Low or no citation	Hold the record, do not post	Reviewer reads the source and decides

The point is not to remove people. It is to spend their attention only where it changes the outcome. A good pipeline is confident about the clear majority of fields and honest about the rest. The smudged scan, the unfamiliar layout, the figure that does not reconcile, these are exactly where a human eye earns its keep, and surfacing them clearly beats pretending the model is always right. Over time the corrections people make become test cases, so the thresholds get sharper rather than drifting. This is the same operating model behind serious Document AI work, and the wider context is covered in our guide to Document AI for enterprise.

The practical takeaway

Treat hallucination as a system property, not a prompt you can perfect. Ground the model in retrieved source text, require a citation for every field, score confidence on each value, and route the low-confidence ones to a person. If a value cannot say where it came from, it does not get written. Build the pipeline so its honest answer is "I am not sure, check this one", and you get an assistant that earns trust instead of one you have to second-guess.