Finding the right model was never the hard part of building an AI-native QA solution at scale.
Part one of a three-part series on how Athenium built AI-powered quality assurance.
The industry is past the question of whether to use AI. Celent's annual survey of North American P&C carriers tracked generative AI in production jumping from 8% in 2023 to 28% in 2024 to 44% in 2025. Roughly 80% of insurers are either experimenting or planning to within two years, and claims and underwriting account for the majority of disclosed use cases.
Yet MIT's widely cited 2025 study of 300 enterprise AI deployments found that 95% of pilots produced no measurable P&L impact. The researchers were explicit that model quality didn't cause the failures. Everyone had access to the same models.
We wanted to understand that gap before we built a solution on top of it. So we ran a foundation model assessment against real carrier data — not a vendor demo, not a sandbox. What follows is the accounting of what that took.
The leaderboard is not a work plan
The instinct is to open a benchmark table and pick the top model. That told us almost nothing.
Public benchmarks are saturated: every frontier model now scores above 88% on MMLU. Worse, they don't transfer. Research on the CLEAR framework documented a 37% gap between lab benchmark scores and real-world deployment performance. Epoch AI found that simply changing the scaffolding around a model — the software operating it, not the model itself — swung agentic benchmark scores by 11 to 15 percentage points. Same model. Different plumbing. Wildly different result.
Here is the more fundamental challenge: no public benchmark contains a loss run, a claim note written at 4:45 p.m. by an adjuster with 60 open files, or a commercial property submission where the SOV contradicts the inspection report. A model that reasons brilliantly about graduate physics has told you nothing about whether it can spot a late reserve.
So we built our own evaluation instead.
What we tested
We ran proof-of-concept engagements with five different carriers, spanning both claims and underwriting.
We evaluated nine separate foundation models from six providers: Claude Haiku, Claude Sonnet 4.6, Claude Opus 4.7, Kimi K2, Kimi K2.5, Nemotron, GPT OSS, GLM-5, and Nova 2 Lite. Frontier and open-weight. Premium and cheap. We didn't assume the most expensive model would win, and it didn't always.
We tested dozens of distinct workflows across four levels of granularity: whole-file assessment, phase-level assessment, individual question assessment, and root-cause assessment. This turned out to be one of the most important findings of the entire exercise. Granularity is not a UX preference; it is a model selection variable. The model that performed best at answering a single discrete question was not the model that performed best at reasoning across an entire file's timeline. If you evaluate one model at one level of granularity, you have measured one cell in a very large matrix.
We ran this against thousands of claims and policies, incorporating loss runs, transaction histories, claim notes, and thousands of documents in Word, Excel, PNG, PDF, and HTML.
What we found
Across the POCs, the system flagged over 2,500 issues across more than 120 assessment questions, including novel issues nobody had specifically targeted the models to find.
The results that mattered most to carrier stakeholders:
- 2.1x lift on issue density. AI-flagged files contained an average of 2.1 times as many reviewer-identified issues as the average file assessed in the traditional QA process.
- 3.6x lift on finding the bad files. AI selection surfaced 3.6 times as many low-scoring files as random selection.
- 85–95% alignment with human reviewers.
- Over 80% precision on numerous individual questions, with significant lift over baseline on many others.
Put that against the economics of traditional QA. Industry estimates put claims leakage at roughly 5–10% of total claim payments. A QA team sampling 5% of closed files, by definition, is not looking at 95% of it. On a $1B claims book, moving leakage two points is $20M. The 3.6x number is not a technical curiosity. It is a direct multiplier on how much of that a review team can actually reach.
The number nobody budgets for: 20+ human-driven iterations
We completed more than 20 full feedback iterations, incorporating both carrier and internal SME input, and each round changed three things: the data processing, the prompting workflow, and the carrier-specific knowledge base.
That last one is the part you can't shortcut. Two carriers writing the same line of business do not define “adequate investigation” the same way. Their guidelines differ, their thresholds differ, their tolerance for judgment calls differs. Encoding is not prompt engineering; it is a structured, iterative, SME-intensive knowledge engineering project—repeated per carrier.
By the end, our honest assessment was this: the model was never the hard part. The hard part was data preparation, workflow design, granularity architecture, knowledge base construction, and the evaluation harness that let us distinguish real improvement from noise. Choosing a model is a decision you make in an afternoon. Everything upstream and downstream of it is the actual project.
Which brings us to the uncomfortable math
The same MIT research found something carriers should sit with: externally partnered AI deployments succeeded roughly 67% of the time. Internal builds succeeded 33% of the time.
Building this in-house means standing up an evaluation harness, generating labeled ground truth from scarce SME hours, re-running the whole assessment every time a new frontier model ships, and owning model validation and governance documentation for a regulated function — all before you have shipped anything a reviewer can use.
Athenium has spent two decades accumulating the expertise that makes this tractable: experience with hundreds of carriers, millions of scored, structured, human-reviewed insurance files, and partnering to build best-in-class QA programs. That is the ground truth.
Next in this series: if the model was never the hard part, what was? Data requirements, the ingestion pipeline, and the parts that broke.
To learn more about Athenium's QA solutions, or to schedule a consultation, click the button below. We look forward to connecting with you soon!
Athenium has been a trusted quality assurance partner to the insurance industry's leading carriers for more than twenty years. These carriers are improving consistency, accelerating performance improvement, and gaining actionable insights with Athenium's purpose-built Quality Assurance solutions. Learn how you can too at: https://www.athenium.com/quality/
Sources: Celent, 3rd Annual GenAI-oneers in P&C Insurance; MIT NANDA, “The GenAI Divide: State of AI in Business 2025”; Earnix, “Insurance 2026: The Race to Reinvent”; Epoch AI benchmark reproducibility research; CLEAR framework evaluation research; industry claims leakage benchmarks.