We already score the final answer. Why more?

Because a correct last message can hide a forbidden tool call, a leaked field, or a loop that cost you the budget. Process failures are the new production incidents.

Does this replace RAG faithfulness?

No. Agents that retrieve still need faithfulness and groundedness on what they say. Trajectory evaluation sits on top of that, not instead of it.

Yes, on replayed traces and held-out tasks. Online sampling of live trajectories is the continuous-evaluation counterpart.

Solutions

Agent & trajectory evaluation

Agents fail in the process: the wrong tool, the wrong argument, no recovery, a bad hand-off. A final-answer score will bless a trajectory that should never have been allowed to finish.

For teams shipping multi-step agents, tool-using assistants, or multi-agent coordination into production — especially where an intermediate mistake is already a policy or safety event.

Score the path

We evaluate tool selection, argument correctness, use of retrieved context, and what happens after an error. Task completion is necessary and not sufficient. Intermediate state — whether the agent knew it was lost — is part of the rubric.

Traces are the unit of work. A last-message corpus will not show a forbidden call that was later overwritten by a polite summary. If you cannot replay the path, you cannot score it.

Judges that can see a trace

Agent-as-judge designs are used when a single-turn LLM-judge cannot see the process. They are calibrated like any other judge. Humans remain on the trajectories that would be indefensible to wave through.

A process judge can still be fluent and wrong. We hold it to a human sample of traces, including recoveries and refusals, not only the successes a demo would pick.

Coordination is a failure surface

Multi-agent systems add hand-offs. We score whether the next agent received what it needed, whether ownership was clear, and whether the system stopped when it should have asked a person.

Escalation is a dimension. An agent that “finishes” by guessing instead of handing to a human has failed the threshold even if the answer happens to be right.

Dimensions on a trajectory

Not every agent needs every row. The failure-mode workshop chooses.

Tool selection

The right tool, at the right step

Arguments

Parameters match the schema and the intent

Recovery

Behaviour after a tool error or empty retrieval

Completion

The task actually finished, on the evidence

Hand-off

State and ownership across agents or humans

Dimension	What is scored
Tool selection	The right tool, at the right step
Arguments	Parameters match the schema and the intent
Recovery	Behaviour after a tool error or empty retrieval
Completion	The task actually finished, on the evidence
Hand-off	State and ownership across agents or humans

Engagement

Typical shape: Framework + Pilot focused on one agent family, then retainer, typically 6–10 weeks for a first trajectory ledger. Duration is typical, not contractual. Formats and deliverables are written on the evaluation ledger.

Questions

We already score the final answer. Why more?: Because a correct last message can hide a forbidden tool call, a leaked field, or a loop that cost you the budget. Process failures are the new production incidents.
Does this replace RAG faithfulness?: No. Agents that retrieve still need faithfulness and groundedness on what they say. Trajectory evaluation sits on top of that, not instead of it.
Can this run in CI?: Yes, on replayed traces and held-out tasks. Online sampling of live trajectories is the continuous-evaluation counterpart.