AI & AGENTS / RESEARCH DEMO
Odette: understanding agent spend
An expensive coding session can be worthwhile. Odette investigates what happened, what the agent delivered, and whether the evidence explains the cost.
Cost and token totals tell us how much an agent used. Understanding whether that spend was useful takes more context: the objective, the work performed, and the outcome. Odette is an audit and evaluation harness for agentic coding sessions across shared LLM subscriptions, built around that distinction.
Filter first, investigate later
Running an LLM judge over every session would add cost before we know where investigation is useful. Odette starts with deterministic detectors to flag a small fraction of unusual sessions—roughly the top 1–5% described in the brief—for closer review.
Sessions are compared within model and task-type cohorts. Different models have different pricing, and debugging can have a different token profile from feature work. Each cohort needs at least 30 turns before the detectors emit signals.
- Robust z-score: uses the median and median absolute deviation to identify unusually large deviations.
- Percentile thresholds: identify costs above the cohort’s 95th or 99th percentile.
- Context size: examines cache-read and cache-creation tokens relative to output tokens, helping surface large context reads for very small responses.
A flag is a reason to investigate. It does not, on its own, establish that the work was wasteful or unsuccessful.
From a cost anomaly to an explanation
The Phase 2 design turns a flagged session into a reviewable account of the work. Each stage saves its output so an interruption does not discard earlier progress.
- Reconstruct. Assemble a chronological trace from messages, tool calls, edits, and linked child sessions. Record whether the evidence is full, partial, or limited to LLM exchanges.
- Compress. Reduce the trace before model evaluation. The brief reports roughly 62% token reduction across 25 real Hermes sessions; that is a compression measurement, not a demonstrated improvement in judging accuracy.
- Extract. Identify distinct objectives and outcomes as work items, with evidence pointers. Investigation, editing, and testing for one objective stay together.
- Describe. Label each work item by task, product, and technical area, preserving context about intent and outcome.
- Judge. Combine the detector signals, trace, work items, and costs into an assessment: justified, unjustified, or undetermined when evidence is insufficient.
The design marks judgments as evidence-limited whenever the trace is less than full. Keeping that limitation visible helps reviewers decide how much weight to give a verdict.
Inside the demo
This summary follows the supplied demo notes; it is not a word-for-word transcript.
The dashboard begins with audit-period and model-cohort filters, alongside session counts, flagged sessions, total cost, and average cost per turn. Cost-versus-context plots, anomaly rankings, and detector breakdowns help a reviewer find sessions worth opening.
The anomaly list organizes flagged sessions by status. Opening a session brings up verdict chips, a headline summary, and actionable review notes. Work items separate the objectives that contributed to the cost, making the session easier to assess than a single aggregate total.
Reviewers can inspect the compressed trace and detector signals, then open the full transcript for supporting evidence. The workflow connects an initial anomaly to an explanation that a person can check.
What we want to learn next
The demo notes identify prompt refinement and evaluation as the next step, using Langfuse to version prompts and score outputs against real flagged sessions. The brief also calls for integration, compressor calibration, and an end-to-end run on a flagged session.
Broader comparisons across models, work types, and developers depend on accumulating sufficient evaluated data. Anomaly counts need to be interpreted against session volume, task context, and evidence coverage before they can support decisions.
The research question is whether a focused evaluation pipeline can make agent spending easier to explain and act on. The materials describe the architecture and review experience; they do not yet quantify savings or the reliability of the judge’s assessments.