1When is building your own extraction models in-house the better choice?
Three cumulative conditions make in-house development defensible and a vendor's contribution marginal: a frozen document scope, low volumes, a permanent in-house MLOps team in place. Otherwise, two trade-offs must be budgeted. According to the IEEE Computer Society and ISO/IEC/IEEE 14764:2022, maintenance accounts for 60 to 80 % of a software product's life-cycle cost: for conventional software, a total cost roughly three times the development. For an AI system, budget double that, around six times: an extrapolation foreign to the standard, drawn from the data, validation and monitoring infrastructure around the model code (D. Sculley et al., NeurIPS 2015). Gartner (July 2024) estimates at least 30 % of generative AI projects are abandoned after the proof of concept; the RAND Corporation (RR-A2680-1, 2024) attributes such failures to mostly organisational causes. In life insurance the first rarely holds: new distributors, regulatory change, channel drift, portfolio transfers move the scope by construction.
2Is a general-purpose LLM called over an API enough to extract data from documents?
Yes for a proof of concept: a general-purpose model, well prompted with an imposed JSON schema, is enough to qualify the need on a limited sample. Production is the breaking point: according to Gartner (July 2024), at least 30% of generative AI projects are abandoned after proof of concept. Four things break. Output conformity: the share of responses that do not conform to the schema varies widely from one model to another; require it measured on your own versioned test set, because it sets the real automation rate as much as average accuracy does. Business accuracy: the gap between a general-purpose and a specialised model is measured at your site, same set, same metric. Cost at volume: per-token billing follows real annual volume, which a sample does not reveal. Finally, version dependency: a model called remotely evolves and gets deprecated without you, replaying prompt and schema acceptance at every change — recurring maintenance a proof of concept never surfaces.
3What multi-year budget should you set aside for an extraction AI system in production?
About six times the initial development cost: that is a deduction from the IEEE Computer Society finding standardised in ISO/IEC/IEEE 14764:2022, not a published figure. According to the IEEE Computer Society, maintenance accounts for 60 to 80 % of a software lifecycle cost; by complement, initial development accounts for only 20 to 40 %, i.e. a cost of ownership of about three times that development for conventional software. Doubling that multiple for a machine-learning system rests on the qualitative finding of D. Sculley et al., Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015 (Google), which puts forward no multiple of its own: model code is only a small fraction of the system, the bulk being the data, validation and monitoring infrastructure, plus re-annotation at every drift. The reasoning holds for a vendor platform too: only the distribution of the burden changes.
4Why do so many document AI projects never get past the POC stage?
The causes are organisational before they are technical: the RAND Corporation (RR-A2680-1, 2024) reports a high failure rate for AI projects, driven primarily by organisational rather than technical factors, and Gartner (July 2024) estimates that at least 30% of generative AI projects are abandoned after the proof of concept. The explanation is an old one: according to Sculley et al. (Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015, Google), model code is only a tiny fraction of a production machine learning system, the bulk being data, validation and monitoring infrastructure — precisely what a POC does not contain. Going to production means assembling eight operating items: annotation and re-annotation, an MLOps platform, drift detection and per-field confidence scoring, human correction, business orchestration, compliance, stack upgrades, reversibility and on-call support. This holds for every option, vendor platforms included: a successful POC says nothing about those eight.
5Which recurring cost items should you have quoted in a document AI project?
Eight operating cost lines to price on top of the build: initial annotation, re-annotation at every drift; an MLOps platform (model registry, versioned datasets, continuous integration); drift detection, a per-field confidence score; human correction, error recovery; business orchestration; compliance (SIEM, secrets, access rights, security testing); version upgrades; reversibility, documentation, skills transfer, on-call duty. The IEEE Computer Society puts maintenance at 60-80% of a software system's cost of ownership (ISO/IEC/IEEE 14764:2022): initial development accounts for only 20-40% of it, a total cost of roughly three times the initial development. For a learning system, double that multiple — on the order of six times the build: a deduction, not an IEEE measurement. Sculley et al. (NeurIPS 2015): model code is only a minimal fraction of such a system, the bulk being the data, validation and monitoring infrastructure. Design and price on both sides: with a vendor platform, infrastructure, integration, configuration, acceptance testing and operation remain your responsibility.
6How do you compare the accuracy rates advertised by two vendors?
An accuracy rate means nothing without its protocol: which test set, which field scope, which definition of an error, clean documents or a real production flow, measured before or after human correction. An average also hides its distribution: a rare but decisive field weighs little in a volume-weighted average, hence the field-by-field breakdown. Above all, average accuracy says nothing about what happens when the model is wrong, and that is where the real cost sits. Three capabilities are tested separately: knowing that it does not know, through a per-field confidence score that routes a doubtful file rather than letting it through; detecting the error by means other than accuracy, through consistency checks across documents and conformity to the expected schema — an unusable output being only one kind of error, though the kind that blocks a transaction; and correcting in the flow, at field level, without leaving the tool. To bring two offers onto the same basis: a versioned test set that you own, the same metric and the same corpus for every candidate, a per-field confidence score. No brochure figure, whoever publishes it, counts before that test.
7What in-house skills does running a document AI day to day require?
A standard operations team can run a vendor platform deployed inside your own perimeter; an in-house build asks for more, unless three conditions are met: a fixed document scope, low volumes, and a permanent in-house MLOps team already in place. D. Sculley et al. (Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015, Google) established that model code is only a tiny fraction of a production machine learning system: the bulk is data, validation and monitoring infrastructure. The skill to assemble is therefore not building a model but sustaining eight operating functions, each to be designed and costed: initial annotation and re-annotation at every drift; an MLOps platform (model registry, versioned datasets, continuous integration); drift detection and per-field confidence scoring; human correction and error recovery; business orchestration; compliance; stack upgrades; reversibility, documentation, skills transfer and on-call duty.
8Can AI read document bundles where every distributor has its own form?
Yes for reading, no for validation. Document extraction engines can read the pieces of a subscription pack — application form, mandate, ID document, bank details, supporting evidence. But a vendor catalogue covers document types, never every individual form variant: in life insurance there are as many variants as there are distributors, products and regulatory vintages, and every new form has to be tested. What decides the amount of rework, as much as average accuracy, is the share of outputs that do not match the expected schema: it varies widely from one model to another; ask for it measured on your own test set, not on a public benchmark. Document matching, completeness and consistency checks remain human. No published automation rate transfers from one pack to another: always ask on what perimeter it was measured.
9Do you still have to review every file once extraction is automated?
No, but the exit from full manual review has to be engineered: as long as case handlers re-check every field, automation produces no gain, and that question decides the return on investment. A per-field confidence score routes files — direct validation, targeted review, full rework — instead of reading everything again, with a human correction interface absorbing the cases sent back. Routing is only reliable if outputs respect the expected schema: the non-conformity rate varies widely from one model to another and weighs as much as average accuracy on the automation rate — require it, measured on your own test set. Full audit logs and the traceability of every decision make the reduction defensible to internal control. No target rate can be published: it depends on the corpus and on the accepted level of risk. Human by design remain document reconciliation, completeness and consistency checks, and doubtful cases.
10How do you design a pilot that lets you choose between two bids?
Design the pilot so that it continues into production, not so that it demonstrates: according to Gartner (July 2024), at least 30% of generative AI projects are abandoned after proof of concept. Supply a corpus you choose yourself, including the cases that make a model drift: phone photos, degraded scans, electronic stamps, formats from new distributors. Require the same versioned test set and the same metric from every candidate, and write down the success and stop criteria before you begin. A rehearsed demo proves nothing, whoever is giving it: ask for live processing of documents the vendor has not seen. A "free" POC costs business time: selecting the files, producing reference annotations, arbitrating. Check pilot-to-production continuity: a platform already in operation can be configured against your real processes during scoping; custom development means specifying, building, delivering, and only then testing.
11What is drift in extraction models, and how do you measure it?
Real and measurable — provided the measuring instrument stays in your hands. In banking and insurance, drift has four sources: new distributors or insurers, regulatory change, channel drift (phone photos, degraded scans, electronic stamps), and portfolio transfers. What makes it serious is not its frequency, which depends on your own portfolio and cannot be quantified in the abstract, but its silence: a drifting model keeps answering without raising any error. Three instruments should therefore stay on your side, and they serve equally to check a vendor's claims and an in-house team's: a versioned test set you own and extend, a per-field confidence score, and a human-correction rate tracked over time. According to the IEEE Computer Society (ISO/IEC/IEEE 14764:2022), maintenance accounts for 60 to 80% of a software's lifecycle cost: that line item exists whether you build or buy.
12What reversibility guarantees should a vendor contract provide for?
Three things protect you, and they belong in the contract: an open-source foundation, your data and configuration on your own infrastructure, model weights escrowed with a trusted third party. The rest — continuity commitments, declared intentions — reassures without protecting. The limit: those weights are worth something only if someone can run them and take over eight operating cost centres that have to be priced, from initial annotation and an MLOps platform through to compliance and on-call duty. Maintenance accounts for 60% to 80% of a software system's lifecycle cost, according to the IEEE Computer Society (ISO/IEC/IEEE 14764:2022): taking on the weights means taking on that share. Demand named deliverables: weight formats and versions, operating and retraining documentation, versioned test sets, data and configuration schemas, a handback procedure, escrow release triggers. Then check the three conditions for building in-house: a fixed document scope, low volumes, a permanent in-house MLOps team.
13How do you justify a decision based on an automated extraction to a regulator?
A field-by-field log is enough, provided it is complete: an explanation of the reasoning does not exist — no statistical model genuinely exposes its own, and a compact model is no exception. What can be demonstrated: traceability, reproducibility, accountability. For each field: model and rule-set version, source document and the region the value came from, confidence score, timestamp, and the identity of the human validator where one intervened — in complete audit logs where any alteration is detectable, under role-based access control and strong authentication. On the operations side, a model registry and versioned test sets make it possible to replay the decision with the version then in force. Sculley et al. (NeurIPS 2015) note that the model code is only a minimal fraction of a production machine-learning system: it is the data, validation and monitoring infrastructure that carries the record — GDPR, DORA and AI Act included.
14What infrastructure do you need to run document extraction in your own datacentre?
For a specialised model, the order of magnitude is a single GPU server, not a compute farm: ask for the reference configuration — card, memory, sustained throughput — validated on your documents. A workable CPU fallback costs throughput, not quality, enough to start before the committee meets. Compute follows size: ask for the parameter count, compare it with a general-purpose LLM's. At comparable compute, on your document packs, it is domain specialisation that decides, not size: settle it with a comparative test, same versioned set, same metric for all. That leaves the VMware plumbing: pass-through or vGPU, drivers, host sizing. The GPU is not the bulk of the matter: Sculley et al. (NeurIPS 2015, Google) show that model code is only a tiny fraction of a production machine-learning system, one dominated by data, validation and monitoring infrastructure.