Work · Production
Multi-model extraction pipeline
A document extraction pipeline that compares model outputs field by field, checks the results, and keeps the reviewer’s decision alongside the source material.
A document extraction pipeline that compares model outputs field by field, checks the results, and keeps the reviewer’s decision alongside the source material.
Scanned documents arriving at a steady rate, each one needing a structured record extracted into a downstream system. The existing process was manual: someone opened the scan, typed the fields into a form, and moved on. Fifteen minutes per scan. Reviewers spent much of their time transcribing fields.
The design moved transcription into the pipeline and gave reviewers a way to inspect uncertain fields and trace a value back to its source.
upload → render → segment → extract → consensus → validate → research → persist ↑ ↑ agreement score address/format rules data layering: Raw (model outputs) → Suggested (consensus) → Final (human-confirmed) └── no silent overwrites. every layer is auditable ──┘
Each stage of the pipeline is a separately addressable Azure Function inside a Durable orchestration. A scan goes in; a structured record comes out; every intermediate state is persisted. The orchestration is the audit trail.
One model is one opinion. The interesting failures aren't refusals or hallucinations — they're plausible-but-wrong values that pass eyeballing. Field-level consensus across multiple extraction models gives you an agreement score per field. Disagreement identifies fields to review. Agreement is one input to the decision; models can make the same mistake, so validation and field-specific review rules still matter.
The comparison identifies which fields differ, so the reviewer can inspect those values against the scan.
Raw, Suggested, Final. Raw is what each model returned. Suggested is consensus. Final is what a human confirmed (or auto-confirmed under threshold). Each layer is preserved on disk. Nothing is silently overwritten.
This is the layer that turns "the AI is making decisions" into "the AI is making suggestions you can review and override." It's also what made the compliance conversation tractable — every record has a complete provenance chain.
LLMs are not the right tool for "is this a valid US address." A Smarty / USPS lookup is. The pipeline runs deterministic validators after extraction — address verification, format rules, range checks — and pushes failures back into the review queue. A failed check sends the field back for review.
Every human correction produces a structured event: which field, what the model said, what the human changed it to, and why. Those events feed few-shot examples into subsequent extractions. Those examples can inform later prompts. Whether they improve accuracy requires evaluation on held-out documents.
Auto-apply gates exist because review time is the constraint, not extraction accuracy. The threshold is configurable per field — high-stakes fields (amounts, identifiers) require higher confidence than low-stakes ones (categorization, optional notes). The threshold is exposed in the admin UI; it's not buried in a config file.
The pipeline runs general-purpose models with structured prompts. Fine-tuning was considered and skipped — the consensus + validator + correction loop got accuracy past the threshold without it, and a fine-tuned model would have locked the architecture to a specific provider. The pipeline is provider-agnostic and can swap models per stage.
When something goes wrong — a model returns malformed JSON, a validator times out, a stage crashes — the orchestration captures the failure as a first-class event. The record is preserved in its last good state with the failure attached. Persisted stage state helps operators inspect the failure and decide where to resume. Recovery still depends on the storage and retry paths working as intended.