AI & AGENTS / ENGINEERING NOTE
When is more AI reasoning worth the cost?
Adding more model calls can improve an answer. Whether that improvement is useful depends on what it costs, how long it takes, and whether the workflow finishes.
By Lance Ventura· Experiment dated In a Looper experiment, we compared five approaches on 50 GPQA Diamond questions: two single-model controls and three strategies that make multiple model calls. The strongest result improved accuracy from 74% to 88%. It also cost roughly 25 times as much and took about 28 times as long as the stronger single-model control.
That tradeoff is the starting point for this note. The experiment gives us a reason to investigate additional reasoning for tasks where a better answer could justify the wait. It does not establish a default for every request.
What we compared
Looper is a reasoning component: it accepts an input, coordinates model calls, and returns an answer. It does not itself use tools or maintain a long-running agent plan.
- Single-model controls: one stronger model and one cheaper model, used as reference points.
- Confidence escalation: start with a cheaper model and escalate when its self-reported confidence falls below a threshold.
- ReMoM: a multi-stage process with five initial drafts, three refinements, and a final synthesis.
- Fusion: a multi-judge approach that combines multiple assessments.
The brief reports greedy decoding at temperature zero. Multiple-call strategies used a model pool anchored on the stronger control. The figures below describe this particular setup.
The results, including failures
| Approach | Accuracy | Cost / question | Latency* | Terminal errors |
|---|---|---|---|---|
| Stronger control | 74% (37/50) | ~$0.0073 | ~9 sec | 0/50 |
| Cheaper control | 58% (29/50) | ~$0.0060 | ~9 sec | 2/50 |
| Confidence escalation | 74% (37/50) | ~$0.031 | ~50 sec | 0/50 |
| ReMoM | 88% (44/50) | ~$0.181 | ~250 sec | 0/50 |
| Fusion | 70% (35/50) | ~$0.072 | ~250 sec | 12/50 |
*Latency values are approximate as reported; the brief does not identify them as means, medians, or percentiles. Dollar costs are reported estimates for the experiment, not current pricing. Correct-answer counts are derived from the reported percentages and sample size.
Seven more correct answers have a price
ReMoM answered 44 of 50 questions correctly, compared with 37 for the stronger control: an improvement of 14 percentage points, or seven additional correct answers. Its reported cost across 50 questions was $9.037, compared with $0.366 for the control.
At roughly 250 seconds per question, this approach is a candidate for further testing in asynchronous work. Whether it pays off depends on the task’s value and tolerance for delay. We have not measured that return on real product workflows in this experiment.
Reliability changes the conclusion
Fusion answered 35 of its 38 completed requests correctly, approximately 92%. But 12 of the 50 requests timed out. Across all requests, its accuracy was 70%, below the stronger control’s 74%.
The completed-request figure helps diagnose answer quality; it does not describe what every user receives. Increasing timeouts or adding fallbacks may improve completion, but their effects on accuracy, cost, and latency need to be measured in another run.
Escalation did not save money in this run
Confidence escalation matched the stronger control’s 74% accuracy while costing about 4.2 times as much and taking about 5.6 times as long, using the reported totals and approximate timings.
A workload containing more easy requests could produce a different outcome. This evaluation does not demonstrate that saving. Self-reported confidence and the escalation threshold need to be tested against the workload where the system would actually run.
Where this fits inside an agent
An agent could call a reasoning component when a difficult decision warrants extra computation. Work that does not depend on the answer could continue while that call runs; dependent steps would still need to wait.
That is an architectural possibility, not a result established by this benchmark. Tool selection, recovery from failed actions, and completion of longer tasks require separate agent evaluations. Better question-answer accuracy alone does not prove a better agent.
What we still need to establish
The source brief does not provide enough detail to reproduce the experiment independently. A fuller evaluation should document question selection, prompts, exact model and provider configurations, answer extraction and regrading rules, and whether retries are included in costs.
It should also include paired item-level outcomes, latency distributions, and a held-out evaluation. The brief describes multiple benchmark iterations without establishing whether the same questions informed tuning. With only 50 questions and no item-level results here, we do not claim statistical significance or generalization to other workloads.
Any change to the model pool or synthesis model should be evaluated again before carrying these results into a deployment decision.
The next experiment
The useful next step is to test routing on a representative mix of tasks, with explicit budgets for time and cost. We would compare a strong single-model baseline with escalation and multi-stage reasoning, count failed requests, and measure whether the extra work improves the final outcome.
For now, this experiment shows an accuracy improvement for one multi-stage configuration on a small question set, alongside a substantial increase in cost and latency. Those tradeoffs deserve the same attention as the accuracy number.