Council Post: Why AI Agents Fail In Production And What The Execution Gap Means

Asen Lei, Founder & CEO of Actionbook, building AI agents for browser automation.

getty

During an AI demo, the room nods after seeing an agent log into a dashboard, pull a report and file a ticket autonomously.

However, many leaders have also seen the challenges that show up three weeks into production. The button may move after the front-end team ships a redesign, and then the same agent that worked before clicks on nothing, and nobody notices until the missed tickets pile up.

I've spent a decade building infrastructure that other developers depend on. I led the first internal serverless platform at one of the world's largest consumer internet companies—a system that grew to handle billions of requests a month.

Today, I run a company building browser execution infrastructure for AI agents.

Across these experiences, I've noticed that the headlines about new technology don't show what it actually looks like on the ground.

For example, many leaders are quoting figures like Gartner's stat that over 40% of agentic AI projects are projected to be canceled by the end of 2027, driven by rising costs and unclear business value. Nearly two-thirds of enterprises have experimented with agents, yet fewer than 10% have scaled them to tangible value.

Most commentary reads these figures as governance and strategy failures. This may be partly true, but from where I sit, what actually stalls these projects is that even when the model knows what to do, the system can't reliably do it.

​Why The Execution Layer Breaks

​When an LLM-native agent operates a website, it typically re-reads the live page structure (the DOM) at every step: tens of thousands of nodes, auto-generated class names, markup that shifts with every A/B test.

During this process, the model must guess which element is "the submit button," then guess again on the next step.

However, given the nature of AI, every guess is probabilistic, and probabilistic errors compound. An agent that is 95% accurate per step completes a 20-step task a little more than a third of the time. This is not hypothetical: In Mind2Web benchmark testing on real-world websites, models that scored respectably on individual steps completed only a small fraction of full tasks end to end.

There's a cost dimension, too. Feeding raw page structure to a model at every step consumes enormous token volume. You are paying the model to rediscover the same page, over and over, on every run.​

​Pre-Process The Web Before The Model Ever Sees It​

When teams see hallucinated clicks, the natural conclusion is that the model is the problem. Based on what I've seen, the main problem is the interface. ​

My team and others in the industry have been attempting to address this issue based on a simple architectural principle: Convert dynamic web structures into standardized, machine-readable action maps before inference is invoked.

The mapping happens once, deterministically. At runtime, the model chooses among verified actions instead of parsing raw markup. This separates two jobs that LLM-native pipelines wrongly fuse: understanding the page, which belongs to infrastructure, and deciding what to do, which belongs to the model.​

We validated this model on two public web-automation benchmarks, running the full task set of each: Mind2Web (2,350 tasks across 137 websites), which I cited above, and WebVoyager (643 tasks across 15 live sites). Across these ~3,000 tasks, measured against the same models operating on raw page content, this design lifted task success rates above 85%, against baselines closer to 50%, while cutting token consumption by roughly 95%.​

​What Makes This Harder Than It Sounds

I won't pretend this is free. Action maps are inventory, and the web changes daily, so maps go stale and need continuous regeneration and verification. That is ongoing operational work, not a one-time build.

Coverage is never complete, either. Long-tail sites and one-off tasks still require exploratory parsing, which means running a hybrid system, and the routing logic between the two modes is its own engineering problem.

Some teams will reasonably decide that an imperfect agent plus human review is cheaper than maintaining an execution layer for a workflow that runs twice a month. For high-volume workflows, the math will likely flip fast, though.

Where To Start

​Based on this experience, there are three things I'd do before launching, or relaunching, an agent initiative this quarter.

First, instrument failures at the step level and separate parsing failures from reasoning failures. Most teams log "task failed" and conclude the model isn't smart enough. Check if the logs say otherwise, as they usually do.

Second, apply an 80/20 rule to your web surface. Build or adopt deterministic action coverage for the handful of interfaces that carry most of your task volume, and let exploratory mode handle the tail.

Third, budget tokens per completed task, not per call. A cheap model that retries five times costs more than a structured execution layer that succeeds once.

Models will keep getting smarter, but intelligence was never the bottleneck in production. The next phase of automation will be decided where infrastructure shifts are always decided: at the unglamorous layer that turns intent into action. ​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?