When should an AI agent act, abstain or escalate?

A team puts an agent into a workflow and discovers that the useful question has no answer anywhere in its evaluation. Not how often it is right — that number is easy to get. The question is: on which of these tasks will it be wrong, and will it say so before it acts?

This note is about that gap, and what can honestly be built across it.

What a score cannot tell you

The International AI Safety Report 2026, the scientific assessment prepared for policymakers, reports gold-medal performance on International Mathematical Olympiad questions and, in the same bullet, that "Performance nevertheless remains ‘jagged’, with leading systems still failing at some seemingly simple tasks". Its own definition is plain: "The capabilities of these systems are ‘jagged’". It names where the unevenness shows — work that runs over many steps, false statements, reasoning about the physical world, and a decline in languages and cultural contexts less common in the training data. On agents it is specific: across most complex professional roles they are still an addition to people, not a substitute, because they stay unreliable once a task involves many steps or is more unusual than the common case.

The same report names the consequence for anyone buying or deploying a system. It calls the distance between test results and real use an "evaluation gap": pre-deployment results, it says, are not always strongly predictive of real-world capability or risk.

A benchmark score tells you how often a system is right. It does not tell you where it is wrong, and that is the number a team deploying an agent actually needs.

The phrase “jagged technological frontier” came from a 2023 field experiment, and it is worth being exact about what that study did. It put 758 consultants into two non-overlapping halves and gave them GPT-4 as it stood at the end of April 2023 — no reasoning mode, no tools, no scaffold. Inside what the authors called the frontier, across 18 consulting tasks, the AI group completed 12.2% more tasks, 25.1% faster, with quality rated more than 40% higher. Outside it, on one task built to sit there, 84.5% of the control group gave the right recommendation against 60% and 70% in the two AI conditions, and the AI groups finished faster while being wrong. What the study did not do: observe anyone on both sides of that line, test more than one model version, or print a limitations section.

One finding in it matters more than the headline and is rarely quoted. Human graders rated the AI users' recommendations higher in quality — and the paper reports that this "increased, regardless of whether the underlying recommendation was correct or not". The failure arrived better presented than the success.

The unevenness is measured, and so is the case against it

The phenomenon does not rest on that one experiment. A separate field experiment with 2,234 participants, producing 11,024 advertisements, found human–AI teams better on text and human–human teams better on images. On a symbolic variant of GSM8K, performance declined against GSM8K itself, and in almost all models it declined further when the numbers in a question were changed than when only the names were; adding one clause that looks relevant but does not affect the answer dropped performance by up to 65% across all the state-of-the-art models tested.

The honest version of this argument has to carry the other side, which is strong. A re-analysis of METR's long-task data finds that most of the apparent shift of gains towards harder problems is reproduced by a single model of uniformly rising ability, and is therefore "largely explained by ceiling effects rather than a qualitative change in capability". Some of the jaggedness people point at is broken benchmark items rather than model behaviour. Averages hide the unevenness they are built from, and which benchmark you pick changes the frontier you measure. The unevenness also runs the other way. Because different models fail on different items, one number understates what a collection of them could reach: across 21 models and 16 benchmarks, an oracle choosing the best model for each query cut the error rate by 54% against each benchmark's own best model at matched cost, and by 82% once it could also pick the best of several runs. We read that as an upper bound rather than an available option — no deployment has the oracle — but it measures how much a single score hides.

We think both readings survive together, and that what they leave is enough. A score is a rate, not a location. One frontier metric reports the task length at which a fitted curve predicts a model will succeed half the time — and its own authors work an example: across the tasks in the band around that length, one agent solved roughly a third of them every time and failed roughly a third every time. That is a location, not a rate, and it is still not your location: none of it tells a team which of its tasks sits on the wrong side.

Ask the system: its confidence is not what you think

If the failures cannot be located from outside, the obvious move is to ask the system. Three measured problems with that.

Calibration is not discrimination. A model can be well calibrated in aggregate and still be poor at separating its own right answers from its wrong ones, which is the property a gate actually needs.

Alignment training degrades it. On a subset of MMLU, scored by the model's token probability over the four answer letters, OpenAI's own report on GPT-4 gives an expected calibration error of 0.007 for the pre-trained checkpoint and 0.074 for the same model after reinforcement learning from human feedback — roughly tenfold — and states that "The post-training hurts calibration significantly." The effect replicates on open-weight models.

And the literature disagrees with itself about which signal to use. Spoken confidence beats token probabilities in one body of work and loses to it in another. The disagreement resolves, and the resolution is the useful part: it depends on the task and on the metric you score with. One paper measuring both calls it "challenging for LLMs to accurately express their internal confidence in natural language". Another argues a stated confidence is "better understood as a behaviour-facing readout of an internal commit-readiness state" than as a probability of being right.

In our experience the practical consequence is unglamorous: whichever signal you choose, you have to measure it on your own task, because the published answer flips with the task and with the metric.

On a long run, confidence arrives late

For agents this gets worse in a specific, measured way.

Across three deep-research benchmarks and five open-weight models from 27B to 1T parameters, confidence stated at the final step separates failed runs from successful ones well: a mean area under the ROC curve of 0.85 over the fifteen model-and-benchmark pairs. At half of trajectory progress no uncertainty signal exceeds a mean AUROC of 0.60, and none passes 0.70 even at 90%.

Confidence stated at the end of a long run separates success from failure well on deep-research benchmarks. Halfway through, when the information would still be worth something, it barely separates them at all.

One caveat from a separate experiment makes this honest, and a weaker account would omit it. Part of what reads as foresight is position in the run. In a 7B agent trained to report its confidence across three household, text-game and shopping environments, a predictor given only how deep the run was and how long the output, and never the confidence token, reached AUROC 0.787, 0.855 and 0.799 — matching or beating the trained confidence on two of the three. Some of what looks like an agent knowing it is in trouble is an agent knowing how far in it is. The same paper runs the control that limits this: residualising the confidence on depth and length still leaves AUROC 0.626, 0.677 and 0.811 for the trained signal against near chance for an untrained one, so a depth-independent component does survive. The deep-research paper runs a task-difficulty control and no depth control, so that reading is untested there.

So a mid-run confidence reading is weak in most measured settings, and stronger for coding agents in one study — intermediate AUROC of 0.655 to 0.781 across three models, which its authors suggest may follow from an agent that changes state being unable to quietly abandon a direction. Measure it on your own task, and never use it as the only gate.

A model checking itself almost never helps

A model asked to check its own work, with nothing outside it to check against, almost always gets worse rather than better. That is measured, repeatedly, and the mechanism is understood.

With no external feedback, two rounds of self-correction lowered GPT-4's accuracy on GSM8K from 95.5 to 91.5 to 89.0, and GPT-3.5's CommonSenseQA accuracy from 75.8 to 38.1 after one round, recovering only to 41.8 after a second; the authors report that across their settings "the accuracies of all models drop across all benchmarks". The earlier, encouraging results had used a ground-truth label to decide when to stop improving, and the gains go when the label does. The loop was being steered by the answer.

Why it cannot work from inside: a model cannot reliably locate the error in a chain of reasoning. On a benchmark that asks a model to name the step where a chain of reasoning first goes wrong, GPT-4's best accuracy was 52.87%, and four other models scored between 14.78% and 48.33%; the paper's conclusion is that "state-of-the-art LLMs currently cannot find mistakes reliably". Critiques invent their own evidence — one study found a verifier's objections "riddled with non-existent edges" in graph problems.

What does work is external. Replacing a model's self-critique with a sound verifier took Game of 24 from 3% to 38%, graph colouring from 2% to 38% and Blocksworld from 55% to 87% — and a fourth domain only from 0% to 10%; the authors report "significant performance gains with sound external verification". The verifier, not the critique, earns that: re-prompting the same question behind it, with no critique at all, reached 42%, 44% and 72% on those three and 14% on the fourth, at 25 samples a question. A smaller trained detector beat prompting a much larger model on four of five tasks, and its authors say it still falls short of the accuracy a backtracking loop would need.

There is a design rule in this, and it is cheap to follow: a check must not see the draft it is checking. One verification method states it directly — "verifying questions should not attend to the original baseline response".

What agents do when they are wrong

Single-attempt scores hide the thing that matters. τ-bench introduced pass^k, the probability that an agent succeeds on all k independent attempts at the same task, as "a new metric (pass^k) to evaluate the reliability of agent behavior". On its retail domain GPT-4o scored 61.2% at one attempt and under 25% at eight. Both are averages across the suite, not one task's rate: what collapses is the share of the work an agent can be relied on to repeat.

Then the behaviour. False success is a mismatch between what an agent says it did and what the environment shows: 45% to 48% of failures in two τ²-bench domains, 3% in a third, where an independent simulator can check the state, and 75.8% of failures on AppWorld among architectures that emit an explicit completion signal. The authors decline the causal reading that third domain invites — 15 cases, offered as a hypothesis — so treat it as a lead and not a mechanism. Training a model to reason does not protect it: on τ²-bench the rate runs from 13% to 79% across the systems measured, and the highest there is a reasoning-trained model. One study of computer-use agents found systems that "fail to verify task-critical properties and correct errors they have already noticed".

They also do not ask. The same benchmark reports agents that "proceed under missing evidence when they should ask the user" — on one task, in its weaker trajectories, while a better run on that same task used the ask-the-user action twice. The action is there; what varies is whether a run reaches for it. And they spend almost nothing on putting things right: the share of the action budget going to recovery and repair "stay below 7% across systems", under 2% on recovery for Claude Opus 4.7.

Some of this is the environment rather than the model. Most steps can be correct and the task still fail, and some environments contain states from which no recovery is possible — measured in procedurally generated games, where it is a property of one of the three environments studied. Joining those two findings is our own reading, not a result: an agent with no recovery budget in an environment with no undo is a system whose first mistake is its last.

Abstention is not scored, so it is not learned

This is not an oversight in the models. It is what they are rewarded for.

Score a system on whether it answered correctly and nothing else, and declining can never be its best move. It is being taught that a confident guess beats an honest refusal. The result is provable rather than observed: for any beliefs the system holds about the right answer, some answer is always strictly better than declining, because "I don't know" scores exactly what a wrong answer scores. The paper that proves it also surveys the benchmarks in use, finds the pattern across them, and puts its fix in the scoring rather than in another benchmark. Retrieval does not remove the incentive. Instructing a model to decline when unsure does not produce reliable declining either.

Training the penalty in does work. One method reports overall error rates falling from 15% to below 2% by rewarding hesitation — a 1.7B model on logic puzzles. On a non-synthetic benchmark in the same work, at a different penalty, the fall is smaller: a 4B model on MATH, 11.4% to 2.75%, with a third of the questions then left unanswered. But it can collapse, and the collapse is measured, not theoretical: on a held-out set of 500 prompts the model demonstrably still solved, the median probability of answering fell from 1.000 to 0.008 or below within ten optimiser steps in three runs out of three, while forced-answer correctness on those same prompts held between 0.94 and 0.97. The system had not got worse at the task. It had stopped answering.

That is the shape of the risk in training abstention: not that it fails to work, but that the same pressure that produces an honest refusal can produce a system that refuses everything, and an accuracy metric looks fine while the answer rate falls two orders of magnitude.

What to build, and what to tell the buyer

Keep the decision outside the model. Abstention trained in gives you one operating point per checkpoint: across TruthRL's three separately trained reward variants, "coverage spans less than 5 percentage points in 8/9 settings". A threshold outside the model traces the whole risk–coverage curve from a single checkpoint, and you can move it after launch without retraining.

Make the decline machine-readable: by default it is not. Anthropic's documentation separates the kinds of refusal and notes that a model's own refusal arrives as an ordinary text response, where only a streaming-classifier refusal carries a structured stop reason. A decline is a success at the transport layer, not an error — as the same documentation warns, "monitoring built only on error rates won't surface it". Structured outputs do not help here: a schema "ensures the model will always generate responses that adhere to your supplied JSON Schema", which guarantees shape and says nothing about truth. A deployed grounding check puts the number in the operator's hands and documents the edge case plainly: "A threshold of 1 is invalid as that will block all content."

Every signal in the first column is measured, and its source is above. The thresholds and the arrangements are ours, recommended from that failure data rather than from a result: no source we read tests a stop-on-mismatch rule, and where one varies a step budget it does so as an experimental setting — 150, 300 and 500 steps in the computer-use study — and not as a control that halts a run and calls a person.

Then tell the buyer where it stops. NIST's risk management framework asks, at MEASURE 2.6, that a system be demonstrated able to fail safely, naming the case where it is made to operate beyond its knowledge limits. Documenting those limits is MAP 2.2. MEASURE 3.2 concedes that a metric may not yet exist, and does not leave it there: where one is not available, it asks that risk-tracking approaches be considered. Appendix C sets out the arrangements between a person and a system as a spectrum, among issues the framework marks for further research rather than as provisions, so deferral to a person is on that spectrum and not prescribed by it. In the EU this is law for high-risk systems: Article 15 requires that levels of accuracy and the relevant metrics "shall be declared in the accompanying instructions of use".

Two neighbouring decisions are not this one, and we have written about each separately: permissions decide whether a call may happen at all, and the honest form of a gap is how it is shown once the system has one. Neither belongs to the model either.

If you are putting an agent into something that matters, the first question is not how accurate it is. It is which tasks it must not attempt, who finds out when it stops, and what you have agreed to tell the person paying for it.