The overlooked pareto line

Cost per task is on every leaderboard but turns and wall-clock are often overlooked. Turns and wall-clock are a cost paid twice, once by the model's context and once by yours.

This started as a comment I was drafting on a Hacker News thread about a Qwen3.8 release, and it got long enough that it clearly wanted to be something else. The thread was the usual shape: strong open-weight model, impressive scores, a lot of people reporting that it does the job. All of that is true, and I am genuinely impressed, even enthusiastic about the speed of innovation and competition coming out of Chinese labs. However I think there is often one overlooked metric in these discussions.

I wish there were more results in the style of DeepSWE that show where a model sits on the trade-off between how many turns it takes and how capable it is, especially at the lower end of the size range, and especially for the Chinese open-weight models, which is where that trade-off is doing the most work.

Go looking and you find that cost-per-task has become standard, and that is genuinely good. ARC Prize plots score against dollars spent per task as a real Pareto frontier. Aider's polyglot leaderboard puts total cost next to percent correct. Terminal-Bench reports accuracy and cost_usd. SWE-bench carries a cost field per entry and links full trajectories as artifacts.

So the field has agreed that capability alone is a bad headline and that efficiency belongs on the chart. It just picked dollars as the efficiency axis.

Turns are a different quantity, and almost nobody reports them. Terminal-Bench's columns are accuracy, agent, org, cost and date, with no steps and no wall-clock. SWE-bench links trajectories but does not rank on their length. The closest thing to a published turn metric I can find is internal: DeepSWE evaluates with a cap of 100 max environment steps, and their training figures plot average response length against average environment steps over the run, showing a model that learned to think less per step and reason more across steps. That is a genuinely interesting curve, and it is a training diagnostic rather than a comparative benchmark.

The one group treating time as a first-class axis is METR, whose 50%-task-completion time horizon asks how long a task a human would take is, at the point where the model succeeds half the time. Their headline, the horizon doubling roughly every seven months since 2019, is the most useful single number in the field. But note the axis: that is human task duration, an estimate of difficulty. It is a measure of how hard the job was, not how long the agent spent on it.

The argument I care about is this one. A lot of people praise models that take two to five times as long to get the same work done, and that praise is not undue, because the work does get done, the weights are open, and the bill is a fraction. But hidden inside that 2–5× multiplier on turns and tokens is a wall-clock cost, and wall-clock is where context rot lives. For both parties.

The model side of this is now well documented. Chroma's Context Rot report evaluated 18 models and found that performance degrades as input length grows even on trivially simple tasks, and that models do not use their context uniformly. A longer trajectory means more tool output, more failed attempts and more of its own prior reasoning in the window. So a model that runs three times as long spends its final stretch in a degraded state that its own earlier work created.

The human side is the part I think is badly underrated, because it never shows up in any measurement at all.

When you start a task, you are holding a large amount of context that exists nowhere else. The requirements are written down, so those are safe. The rest is not: the half-formed suspicion that the bug is really in the caller, the three edge cases you thought of while reading the ticket, the awareness that the module you are about to touch is the one someone complained about last month, the better API shape you noticed but did not write down because you were going to get to it in a minute.

That context in your head has a half-life, and the half-life is measured in wall-clock minutes rather than in token cost or even turns.

I keep good notes. I have knowledge graphs, a memory directory, per-project instructions, a handoff skill for long-running work. All of it helps and none of it catches everything. There are always ideas that were alive in my head at turn one and gone by the time the work comes back.

When the work comes back in twenty turns or 15 minutes, we might have a decent survival rate. When it comes back in sixty to a hundred (hours maybe), which for me is often long enough to have context-switched to something else and back, a real fraction of those memories will fade. This is especially true for members of the ADHD community like myself whose physiological make-up exacerbates this exact issue.

That is a cost invisible and under-represented in many benchmarks. The capability that degraded there is mine.

I want to be fair about the size of the effect, because it is easy to read the above as “slow models are bad” and that is not the claim.

A model that takes 2–5× the turns of a frontier model is still doing, generously, something like a twentieth of what the same task would have cost a junior developer in 2020, and that comparison should include the parts people forget. The code review round trips. The two video calls to agree on an approach. The day of ramp-up on an unfamiliar module. Measured against that baseline, every model in the conversation is astonishingly fast, and arguing about a 3× difference between them is a luxury.

It is also a luxury that matters, for a reason that has nothing to do with capability.

The competitive picture changes once the whole field is using LLMs. If your competitor ships with an agent and you ship with an agent, the model's raw capability stops being your edge, because they have access to the same capability. What differentiates you is iteration rate: how many idea-to-implementation cycles you complete in a week, and how much of the original intent survives each one.

• Faster cycles mean more shots on goal against the same calendar. This is the obvious one.
• Faster cycles mean the engineer is still holding the unwritten context when the diff lands, so review is sharper and the follow-up idea actually gets tried instead of forgotten.
• Faster cycles mean less compaction. Fewer turns is less to summarize, and a summarized window has already thrown away detail that a shorter run would still have had.

None of that is captured by pass@1 on a benchmark suite, and none of it is captured by cost per task either. A model that is cheaper per task and slower per task is a different product from a model that is more expensive and faster, and right now we have a good number for the first half of that sentence and no number at all for the second.

Concretely, and I do not think any of this is hard, because the trajectory artifacts already exist:

• Median turns to resolution, reported alongside pass rate, on a fixed agent scaffold. Same harness, same tools, same retry policy, so that the comparison lands on the models rather than on the scaffolds.
• Wall-clock to resolution at a stated concurrency, which is the number a person waiting at a terminal experiences and which no leaderboard currently reports.
• The distribution, not the mean. A model with a good median and a long tail of hundred-turn flails is a much worse daily driver than its average suggests, and the mean hides exactly that.
• Turns plotted against capability as a frontier, the way ARC Prize plots cost. Then “this small model is on the Pareto line” becomes a checkable statement rather than a vibe.

The open-weight releases would look good on this chart, by the way. That is rather the point. Right now their strongest argument is being made in forum comments and personal anecdote instead of on an axis. DeepSWE showed the community is willing to publish step-level detail when it has it. I would like the leaderboards to start asking for it.