Language models such as Dokumentli® now read documents with high precision, so extracting values from a document is rarely the real challenge. The problem is the question of what happens to the value that was read before it reaches accounting, the ERP system or the archive, and who answers for it when it turns out to be wrong.
This article sets out which functions a validation layer between model and target system has to cover, which metrics it has to produce in order to stay controllable, and what a company should expect to spend if it builds one itself. It is written for teams that already run a language model in production or are preparing to. The implementation in the Parashift platform serves as a reference, not as a recommendation as such.
Starting point
A language model returns an answer for every document, including those that are incomplete, were fed in crooked, or belong to a type nobody in the process expected.
That answer is almost always cleanly formatted and plausible in substance. Whether it is correct cannot be read off it, because the model does not distinguish between a value it is certain about and one it has guessed unless that distinction is measured separately.
An example from a test on our playground. On a delivery note the printed quantity of 60 units had been struck through and corrected by hand to 48, while the printed line total still referred to the original 60. The model picked up the handwritten correction and returned 48, and it did so with high confidence because the digit was clearly legible.
The value was therefore read correctly, and the document still did not add up, because the line item contradicted the stated total. That contradiction appears in no confidence score, since confidence says something about the legibility of the digit and nothing about the consistency of the document.
In a research task such cases are acceptable, in an accounts payable invoice that enters a payment run they are not, because a difference of twelve units there leads to an incorrect posting rather than to an unfortunate turn of phrase.
Every pipeline that runs in production therefore needs an instance that receives each model output, assesses it against defined criteria and then either releases it for further processing or escalates it for review.
Requirements
The first requirement is a confidence score per field. A single score across the whole document is of no use, because a document consists of fields with very different legibility, and a cleanly printed invoice header would receive the same rating as a handwritten correction in the line item below it.
The second requirement is thresholds that can be set separately per field and per tenant, because an invoice amount calls for different protection than a contact name, and because an insurer treats its documents differently from a logistics provider. Where the threshold sits is a commercial decision about the ratio of review effort to error cost, not a technical constant.
The third requirement is rules that reach beyond the individual field, that is format and mandatory field checks as well as total checks, duplicate checks, vendor matching and reconciliation against purchase order, goods receipt and master data. A model can read the stated invoice total entirely correctly and still deliver a result whose line items do not add up to that total, and those cases only surface if somebody does the arithmetic.
The fourth requirement is routing that forwards everything which passes thresholds and rules without intervention and places everything else in a validation interface in which only the flagged fields are up for decision. If the entire document has to be re entered at that point, the time saved through automation is spent again.
The fifth requirement is an audit log that records, for every field, which value was read, with which confidence, at which location in the document, against which specification version and which rules it was checked, by whom it was corrected where applicable, and when it was released. Without that record the process can be evidenced neither to an auditor nor to a customer.
Implementation in the Parashift platform
The Parashift platform covers these five points as standard functions, so that they are configured rather than developed.
Confidence scores are produced per field, and the corresponding thresholds can be set per field and per tenant, which lets confidence based routing separate straight through processing from review cases.
The validation rules cover format, mandatory fields, totals, duplicates, vendors and reconciliation against connected systems, and whatever does not pass them lands in an embedded validation interface, operated either by the customer or by Parashift under an agreed turnaround time.
Approval workflows represent multi stage controls, where agent in the loop and human in the loop are two configurations of the same mechanism and can be set differently per document type.
OneTouchLearning® feeds corrections back as learning, the audit trail records specification version, location and processing history, and «bring your own model» allows any model to be connected provided it conforms to the OpenAI API specification.
The platform is built API first throughout, is multi tenant capable and comes with more than 30 connectors to the common target and source systems.
Further automation with agents on the document
The validation layer is the precondition for the next automation step rather than a substitute for it, because once confidence, rules and audit log are in place, agents can be deployed directly on the document without losing control over the result.
Such an agent can look up a missing entry in the source system when a rule fails, classify a discrepancy between invoice and purchase order and assign it to a tolerance, or request the missing pages when the document type is unclear, instead of passing the case into manual review untouched.
What matters is that every one of these agents works against the same thresholds and writes into the same audit log as the extraction itself, because the boundary of straight through processing can only be moved upwards on that condition without traceability suffering.
Without a validation layer this step makes no sense, because an agent without thresholds and without an audit log is a second system producing plausible answers, and therefore multiplies the original problem instead of solving it.
Build versus existing platform
The decision between building and buying is rarely made on the first version, because confidence logic with thresholds and a simple review screen can be built in a few weeks and, in experience, works well for the first document type.
The effort arises in everything that comes afterwards, namely tenant separation, roles and permissions, audit proof logging, resumption after an abort, bulk corrections, connectors to every further target system, operation of the validation interface across the full working day, and feeding corrections back into the model.
What emerges here is not a feature but a product of its own with ongoing maintenance, and whoever builds it ties development capacity permanently to a part of the value chain in which they do not differ from the competition.
Building in house is justified, however, where the business rules are so specific that no standard configuration can represent them, and that constellation does occur, though it should be evidenced before the decision is locked in.
Metrics for operations
The straight through processing rate shows which share of documents passes without manual intervention and is the figure against which the commercial effect of the whole pipeline is measured.
The error rate after release shows whether the thresholds sit too low, and can only be established if samples are drawn from the automatically released volume rather than from the reviewed one.
The review rate per field shows which fields actually cause the manual effort, and in practice regularly reveals that a handful of fields accounts for the greater part of the processing time.
The turnaround time in the review step shows whether the interface and the organisation behind it are sized for the volumes, and it is the first figure to tip when volume rises.
Without these four figures no threshold can be justified, with them its setting becomes a trade off between review effort and error cost that can be expressed in numbers and defended to the management.
Changing the model
Models are replaced at short intervals, the validation logic on the other hand remains the same for years, which is why thresholds, rules and audit log belong in the application and not in the model call.
Whoever observes this can exchange one model for another without rebuilding the process, and the comparison of two models becomes a measurement on the same metrics rather than a project with a schedule of its own.
Limits
The layer reliably detects what violates a stored rule, and it does not detect what is plausible and wrong at the same time, which is why a correctly read amount in the wrong position stays undetected as long as no rule addresses that position.
A high confidence score means that the model is certain, not that it is right, and in practice those two statements diverge more often than the number suggests.
Thresholds shift the ratio between missed errors and unnecessarily reviewed documents, but they eliminate neither, and whoever sets them decides on a distribution rather than on its abolition.
The strongest objection
Against everything stated so far it can be objected that thresholds and rules slow down automation precisely where it is meant to take effect, and that a process with forty validation rules ends up employing more people than the manual process it was supposed to replace.
The objection holds, and it holds above all in the pilot phase, in which nobody knows which fields are actually error prone and in which every additional rule rests on an assumption rather than on a measurement.
What follows from this, however, is not to do without the validation layer but to start it empty, that is with confidence and audit log but without an elaborate rule set, and to add rules only where measurement shows a recurring error. Whoever proceeds the other way round builds a rule set against errors that often do not exist in their documents.
Getting started
Start with one document type and a manageable number of fields, and deliberately set the thresholds high at first, so that a large share runs into manual review and you see where the errors actually are.
Measure for two to four weeks, then lower the thresholds where review found nothing and sharpen the rules where it found something.
Take on the next document type only once the first runs stably, because otherwise it is no longer possible to establish which change produced which effect.
Conclusion
The use of language models in document processing rarely fails on extraction and regularly fails on the question of what should happen to the result before it is written into a system of record.
The validation layer decides whether a pipeline is fit for production, and it is at the same time the part most frequently planned too late in projects, because it does not hurt during the pilot.
Whoever defines it early, that is settles confidence, rules, routing and audit log before the first document type goes live, obtains a pipeline whose quality is measurable and whose model is exchangeable, and at the same time creates the basis for deploying agents close to the document.
