The final answer is the least interesting output of an agent run.
It is the only output most teams store. That choice makes every later question harder: why did it pick that tool, why did it stop, why did it cost four times more than yesterday, and why does it work for one user and not another?
Record the run, not just the result
A useful trace captures, per step:
- the model input (or a redacted, reproducible version of it)
- the model output, including tool calls it requested
- each tool call’s arguments and result, including errors
- latency and token counts per step
- the stopping reason, especially when a budget was hit
This is not verbose logging. It is a structured record that a program can read and a human can scan.
Evaluation starts with traces
Once runs are stored, evaluation stops being a vibe. You can sample real traces, label them, and build a regression set from actual failures instead of invented ones. You can also compare two prompts or two models on the same inputs, because the inputs are already captured.
Teams that skip this step end up arguing about anecdotes. Teams that store traces end up with a queue of concrete bugs.
Replay is the payoff
The most valuable property of a trace is that it can be replayed. Fix a tool, re-run the recorded inputs, and compare outcomes. Without replay, every change is a guess validated by a demo.
Store less than you think you need to keep, but store the structure. A trace with redacted content and accurate step boundaries beats a full transcript that cannot be parsed.
Instrument first. The evaluation framework can wait; the raw material cannot.