Part two of a three-part series on how Athenium built AI-powered quality assurance.
Part one of this series ended with a finding that surprised some of the carriers we shared it with. Nine foundation models, 18 months of carrier proofs-of-concept, more than 2,500 findings, and the model selection work, as rigorous and necessary as it was, turned out not to be the hardest part of the build.
Athenium tested nine models across several levels of granularity precisely because the wrong choice at the wrong granularity degrades everything downstream, and we re-run that evaluation as frontier models advance. But a well-chosen model sitting on top of a weak data pipeline and infrastructure produces inconsistent, unexplainable output. A best-in-class QA function cannot use that.
This is about the rest of the system: the data infrastructure, the ingestion pipeline, the problems that broke early versions of it, and how we solved each one.
What carriers asked for, and what it implied
Through years of discovery and proof-of-concept work, four requirements came back from carriers with near-perfect consistency:
- Make QA workflows more efficient.
- Give leadership direct, high-level visibility into the entire book of business.
- Catch issues early enough that the front line can course-correct in real time.
- Start with something that delivers reliable value now and then expands.
Not one of those is a model requirement. All four are infrastructure requirements. “Visibility into the entire book” means ingesting everything, not a 2% sample. “Course-correct in real time” means auditing open files on triggers, not closed files on a quarterly cycle. “Reliable value now” means output consistent enough to trend over time; which, as we’ll get to, is a versioning discipline before it is an AI capability.
Athenium built its next-generation QA solution to those four requirements. Here is what stood in the way, and what we did about it.
Problem one: the data doesn’t arrive clean, and it never will
More than 90% of enterprise data in insurance is unstructured — adjuster notes, broker submissions, litigation demand packages, medical records, inspection reports, etc. One widely cited industry estimate puts claims data specifically at roughly 97% unstructured. AutoRek’s 2026 research found insurers running an average of 17 data sources feeding premium processes alone.
The tempting shortcut is to specify a clean input format and ask the carrier to conform to it. That shortcut fails immediately, because the documents carrying the QA signals are precisely the messy ones.
What we built: Ingestion that accepts carrier data in its native formats — Excel, CSV, Word, PDF, Outlook messages, images. Every document then runs a common processing pipeline: parsing, OCR and extraction; structured reformatting; classification; then chunking and storage for retrieval-augmented generation.
That sequence sounds routine but is where most builds quietly lose their accuracy. Salesforce’s engineering team published a useful accounting of this in 2026: a retrieval pipeline scoring above 90% on standard text benchmarks dropped to roughly 46% when evaluated against complex, customer-representative enterprise documents. Rebuilding parsing, chunking, enrichment, embeddings, and retrieval brought it back above 90%. This is a staged climb from 65.1% at baseline to 84.4% after parsing improvements to 86.8% after enriched indexing. Same models throughout. The entire gain came from the infrastructure.
Insurance documents are the textbook case of that “complex, customer-representative” category: tables breaking across pages, scanned attachments, threaded email chains, inconsistent SOV formats.
What this means for carriers: No data cleanup project before value starts. No IT-led reformatting effort standing between your QA team and its first audit. You send files as your systems already produce them.
Problem two: Carrier standards are real, but most of them aren’t written down
A general insurance QA knowledge base is table stakes. Every carrier defines “adequate investigation” differently, and the definition shifts by line of business, jurisdiction, and coverage type. In most organizations, those guidelines are partially undocumented and scattered across teams and systems. You cannot prompt your way around this.
What we built: A layered carrier-specific knowledge base drawing on four sources: industry standards; accumulated best practices from Athenium's QA subject-matter expert team; your own guides and documentation; and carrier-specific knowledge derived from your QA results. A built-in feedback loop lets vetted reviewer and admin corrections update those guidelines continuously, so alignment with your standards compounds rather than decays.
That fourth source is why this is hard to replicate on any timeline. Athenium has spent two decades accumulating experience with millions of scored, structured, human-reviewed insurance files across hundreds of carriers. That is not a dataset assembled in standing up an AI-only QA tool.
What it means for carriers: The system encodes how your organization defines quality — and it gets better at it every time a reviewer disagrees with it. Institutional knowledge that currently walks out the door with retiring adjusters becomes an appreciating asset.
Problem three: one QA program is not one use case
This is the objection we hear most: our use case is specific, so we’d need something purpose-built.
One wants open-claim feedback for exception-based supervision. Another is targeting FNOL intake and payment accuracy specifically. A third wants a quality score across the entire book. Several wanted all three, in sequence, as their program matured.
That pattern is the argument against building something purpose-built. A single-use tool solves this quarter's use case and becomes a constraint on next year's. Meanwhile, EXL’s 2026 U.S. Enterprise AI Study found 92% of insurers describe their data as a challenge to AI success, with insurance citing data silos as its top barrier more often than any other industry — which is what happens when each new need gets its own bespoke build.
What we built: A configuration layer rather than a fixed workflow. You control which files are audited and when, the granularity of assessment, which phases and questions are reviewed, and the pool of root causes available for issue identification. For open-file analysis, reviews fire on carrier-defined triggers — at 2, 5, 30 and 60 days from open, on reassignment, on reserve adjustment, on close. Part one established that granularity is a model selection variable, not a UX preference; it's also a configuration variable, and the platform selects accordingly.
What it means for carriers: One infrastructure investment, many QA programs. Start with closed-file scoring across the full book to establish a baseline, add real-time open-file supervision for a problem segment, stand up a targeted FNOL audit when a leakage study points there — without a new procurement cycle, a new integration, or a new model evaluation each time. Your QA function stops being limited by what it was scoped to do eighteen months ago and starts adapting at the speed the business actually moves.
Problem four: quality trends are worthless if the measurement moves
Large language models are non-deterministic, and providers update hosted models without always publishing changelogs. The failure mode isn’t a crash. It’s slight wrongness, consistently, across a large volume of files or a silent regression that surfaces as a quality trend that isn’t real.
For a QA function, that’s disqualifying. When your Q3 score moves two points, you need to know whether the book changed or the workflow did.
What we built: Every workflow component is versioned, and every change is validated against a control set of your files serving as ground truth before it goes live. Control-set metrics validate model, guideline, and workflow changes pre-release; normalize output so purely workflow-driven variation doesn’t masquerade as a quality change; and identify which guidelines perform reliably and which need work.
The regulatory case for this is now explicit. As of April 2026, 25 jurisdictions had adopted the NAIC Model Bulletin on the Use of AI Systems by Insurers, with California, Colorado, New York and Texas issuing insurance-specific AI guidance of their own. The bulletin expects a written AI Systems Program with senior-management accountability, model validation and retesting, data governance, and documentation that an insurer can produce in a market-conduct examination. It holds the insurer accountable for third-party AI — so the vendor question is not whether they’re accountable, but whether they can hand you the evidence.
What it means for carriers: Trend lines you can take to an operations committee and defend. Model and workflow changes that arrive with validation records attached. And when an examiner asks how the system was tested, the documentation already exists.
Problem five: findings aren’t outcomes
As our carriers know, all the findings in the world don’t matter unless they produce action — handler coaching, process improvement, or remediation before an issue becomes a bad outcome.
What we built: AI audit results feed a full analytics platform where QA teams trend results, compare segments, and drill to individual file findings. Because AI findings share a common issue and root-cause taxonomy with your existing manual QA findings, the two analyze together rather than in parallel silos. Findings on open files surface directly to front-line managers, with integrations to existing systems in development so no one has to hop between platforms to do their job.
What it means for carriers: QA moves from a retrospective scorecard to an operating input. Managers see what’s wrong on files they can still influence. Leadership sees the whole book instead of a sample of it — and the 3.6x lift on surfacing low-scoring files from part one becomes a coaching pipeline rather than a statistic.
The ROI is clear
Part one cited MIT’s finding that externally partnered AI deployments succeeded roughly 67% of the time against 33% for internal builds. Everything above is why.
An in-house equivalent means encrypted ingestion, a document pipeline that survives real carrier documents, a carrier-specific knowledge base with a working feedback loop, a configuration layer flexible enough to serve use cases you haven’t scoped yet, a versioning and control-set regime robust enough for a market-conduct exam, an analytics platform, and single-tenant infrastructure with enterprise access controls. Industry estimates consistently put data preparation alone at 30–50% of an AI project’s budget and 60–80% of its effort — before anything goes into production that a reviewer can use. AM Best’s April 2026 survey of more than 150 carriers and MGAs found only about one in five rating themselves at an advanced stage of AI implementation, and Capgemini’s 2026 World Property and Casualty Insurance Report found 60% still in exploration or proof-of-concept.
Athenium is ready today. Validated. In production.
Next in this series: we introduce the solution by name and show what 100% coverage looks like in the hands of a QA team.
To learn more about Athenium’s QA solutions, or to schedule a consultation, click the button below. We look forward to connecting with you soon!
Athenium has been a trusted quality assurance partner to the insurance industry’s leading carriers for more than twenty years. These carriers are improving consistency, accelerating performance improvement, and gaining actionable insights with Athenium’s purpose-built Quality Assurance solutions. Learn how you can too at: https://www.athenium.com/quality/
Sources: Collibra/AXA XL analysis of unstructured data in insurance; CLM, “Mining for Information Gold in Unstructured Claims Data”; AutoRek 2026 Insurance Report; EXL, 3rd Annual U.S. Enterprise AI Study (2026); AM Best, “Artificial Intelligence Appears to Be Ready, But Most Insurers Are Not” (April 2026); Capgemini, World Property and Casualty Insurance Report 2026; Salesforce Engineering, enterprise RAG accuracy benchmarks (2026); NAIC Model Bulletin on the Use of Artificial Intelligence Systems by Insurers, state adoption tracking as of April 2026; IBM, Cost of a Data Breach Report 2026; MIT NANDA, “The GenAI Divide: State of AI in Business 2025”; industry AI project cost benchmarks.