Is this observability?

Observability tracks latency, cost, and traces. Continuous evaluation scores output quality and behaviour against a written rubric. You need both. One does not replace the other.

Do we need this before the first audit?

Usually no. A sprint produces a first ledger. Continuous evaluation is how that ledger stays true once traffic and models move.

What is the deliverable on a retainer?

A living scorecard, drift notes, and the changes to sets, judges, and thresholds those notes require. The team should be able to run the cadence without us; the retainer is how it expands.

Solutions

Continuous evaluation in production

Offline evaluation is necessary and not sufficient. Models, prompts, retrieval, and traffic move. Continuous evaluation is the same ledger, re-run — so the threshold does not quietly travel.

For organisations that have a first golden set and a gate, and now need production sampling, drift detection, and a path from a quality signal to rollback.

Sample the live system

We design sampling that is cheap enough to run and representative enough to trust — including the slices that matter. Online scoring reuses the same dimensions as the offline ledger. A second, prettier dashboard is not continuity.

Production is a different distribution. The golden set stays the reference; the live sample is how you learn that the reference is no longer the job. Both stay in the scorecard.

Drift is a decision

Quality drift, behavioural drift, and judge drift are distinguished. An alert without an owner is décor. Signals connect to incident response: investigate, hold, roll back the prompt, the model, or the retrieval.

A quiet move in one slice — one language, one product line, one tool — is the usual incident. Continuity that only watches the global average will miss it.

Cost, latency, and judgment together

A model that is cheaper and slower to fail is not an improvement. Joint monitoring keeps the trade-off visible, so a cost optimisation cannot silently spend the threshold.

The retainer is how this stays a practice: new use cases, new models, the same dimensions. The first ledger is a beginning. Drift is the rest of the work.

What stays alive

The living scorecard. Same dimensions as the offline rubric.

Production sample

The distribution you actually serve

Slice report

Where quality moved, not only that it moved

Drift alert

A change large enough to own

Rollback path

Prompt, model, index — who may revert it

Signal	What it is for
Production sample	The distribution you actually serve
Slice report	Where quality moved, not only that it moved
Drift alert	A change large enough to own
Rollback path	Prompt, model, index — who may revert it

Engagement

Typical shape: Retainer / Continuous Evaluation, ongoing, typically in 3–12 month cadences after a first ledger exists. Duration is typical, not contractual. Formats and deliverables are written on the evaluation ledger.

Questions

Is this observability?: Observability tracks latency, cost, and traces. Continuous evaluation scores output quality and behaviour against a written rubric. You need both. One does not replace the other.
Do we need this before the first audit?: Usually no. A sprint produces a first ledger. Continuous evaluation is how that ledger stays true once traffic and models move.
What is the deliverable on a retainer?: A living scorecard, drift notes, and the changes to sets, judges, and thresholds those notes require. The team should be able to run the cadence without us; the retainer is how it expands.