Council Post: Picking The Best LLM Is The Wrong Question For Carriers

Marcin Nowak, board member at Decerto, has 20+ years in insurance, focusing on automation, technology impact and software solutions.

getty

​Nearly every conversation I have with insurance executives about AI lands on the same question: Which model should we use? It's a reasonable question, but it's the wrong one. The capability gap between leading models narrows every quarter while the decisions that actually determine success sit elsewhere: where the model runs and how your work reaches it.

Regulators have noticed, too. The NAIC's AI Model Bulletin, adopted by 25 U.S. jurisdictions (and four more that regulate insurer AI under their own frameworks), requires insurers to govern not only their own AI systems but also their third-party AI arrangements. That governance work starts with deployment, not with model selection.

Data Privacy: A Contract Problem, Not A Technology Problem

There's a common myth that you should never send personal data to a public LLM. The real issue is the terms and controls governing the service. Consumer accounts generally don't provide the same contractual protections as enterprise offerings. Enterprise agreements can include commitments that customer data won't be used to train foundation models, that the customer retains ownership of its data and (depending on the product and region) controls over data residency and processing.​​

A self-hosted model helps remove this concern. You control where it runs and who has access to it. As Holland & Knight's May 2025 analysis of the NAIC bulletin points out, an insurer's AI program must be designed to prevent regulatory violations, including in systems it buys rather than builds. Either path can pass that test. The difference is how much of the evidence sits in your hands.

Why You Probably Don't Need A Frontier Model​

To be fair to the frontier models, they're genuinely better at open-ended work. You also can't host them yourself because their weights aren't licensed for it. But do you need one?

In my experience, frontier models earn their price when every task is new, like in software engineering. Insurance operations are different. Intake, document extraction, coverage checks and claims correspondence repeat in recognizable patterns all day long.

That repeatability is an architectural advantage. The key is the harness: the software layer around the model that turns a general-purpose model into a controlled workflow. The model answers questions. The harness determines which questions get asked, in what order, with what data and under what rules, and what happens next.​​

In a claims process, a document arrives, the harness classifies it and pulls the policy data the model needs, then asks a series of small questions, one at a time: "Is this a repair invoice?" "Does it match the reported damage?" "Which clause applies?" Every answer is checked against rules and confidence thresholds, and anything below the bar goes to a human adjuster.

Two design choices carry the whole construction. The first is task decomposition. The smaller and more precisely defined the task, the smaller the model that can handle it reliably. The second is evaluation. Each task type gets a golden set, examples with known correct answers, so you measure the model on your own work rather than on a public benchmark.

I've seen the harness matter more than the model it wraps. Deloitte's "2026 Global Insurance Outlook" points in a similar direction: The insurers best positioned for success will be those that act decisively on their models, tools and strategies, not necessarily those that adopt the latest technology fastest. That is also what makes self-hosting increasingly realistic: A well-designed harness can enable a smaller, more affordable model to handle workflows that might otherwise require a frontier model.​

The Stability Problem Nobody Prices In​

​Public LLM services aren't necessarily stable enough for every production workload. Providers may update model versions, change serving infrastructure or retire older versions, and a model that behaved well on Tuesday can behave differently on Thursday. If your agreement lacks sufficient capacity guarantees, you can also encounter the familiar 429 error: The model is too busy; come back later.​

Silent change matters because you always tune the harness to a specific model. Prompts, task decomposition and validation thresholds are calibrated against its behavior, so when the model changes, the tuning has to be redone. In a claims automation project, we added canary tests for exactly this reason. A subset of the golden set runs against the model every day, and when the results drift, we alert our users before the drift reaches production decisions. I'd recommend that practice to anyone running AI in a regulated process. And once your tasks are small and measured, swapping one model for another is days of retuning, not a rebuild, so the choice stays reversible.

With a self-hosted LLM, nobody changes your model without your consent. The version you validated is the version that runs.

The Economics Follow The Volume

Which option is cheaper? As with most infrastructure decisions, the answer depends on your workload. At low call volumes, paying per token on a public endpoint often wins because you carry no idle infrastructure. Self-hosting means providing the compute yourself. Smaller models can run on consumer hardware, but production workloads often require more capable GPUs, and dedicated or reserved rented capacity can cost money whether you use it or not.​

The threshold isn't a fixed number, but the shape of the calculation always is: monthly hardware cost divided by the tokens you actually process. Run that division on last quarter's real volume rather than on the volume in your roadmap because the roadmap number is what pushes carriers into hardware they never fill. It's your call volume, not a vendor's benchmark, that tells you which side of the threshold you're on.

The Question Worth Asking

I don't have benchmark data for every line of business, and honest people land on different sides of it. The point is to make it a decision, weighing privacy, capability, stability and cost against your actual workload rather than against a default inherited from a vendor pitch.

So, here's the question worth asking in your next leadership meeting: If your provider silently updated your model tomorrow, how would you find out? If the answer is that your users would tell you, then the choice between public and self-hosted isn't your first problem. Your first problem is the harness.​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?