Golden datasets & challenge sets

We sample from logs or historical cases under the data rules you already have. Client data stays in the agreed environment. Anonymisation is part of the artifact, not an afterthought. Every item carries provenance: where it came from, who labeled it, which version of the rubric applied.

The set is sliced. A single average hides the slice that fails. Challenge cases — rare, high-risk, adversarial — are constructed when production does not yet contain them, then validated by a human. Synthetic is a supplement, never the whole ledger.

A pipeline audit without a set is a reading of whatever happened to be in the room. The dataset is how that reading becomes repeatable when the model, the prompt, or the index moves.

Ground truth is a claim. We treat it as one: guidelines, more than one annotator where the stake requires it, agreement measured, disagreements adjudicated. The dataset card records that process so a later auditor is not asked to take the score on faith.

Where the label is preference rather than truth — two fluent answers, one better — the set still carries the guideline version and the pair. Rankings without a protocol are taste.

When the product, the policy, or the model changes, the set changes with it — or it is frozen and named. Access control belongs with the rest of your evaluation assets. A golden set that cannot be reproduced is a demo.

Retention follows your existing records policy. In regulated settings the evaluation set is itself an artifact that may need to be produced later. We write that down at the start, not after a request from audit.

Minimum fields. Further columns follow the rubric.

Typical shape: Framework Design + Pilot, or Full Build, typically inside a 6–10 week pilot; longer when several use cases share a library. Duration is typical, not contractual. Formats and deliverables are written on the evaluation ledger.

Can you evaluate without our production data?

We can start from a held-out sample you already hold, or from carefully validated synthetics. A ledger built only on public benchmarks will not survive contact with your traffic.

Do you take data out of our environment?

Not without explicit approval. The default is to work inside your VPC or workspace. Evaluation artifacts in regulated settings are themselves controlled records.

How large does a golden set need to be?

Large enough to cover the slices that can fail, small enough to relabel when the rubric moves. Size is a function of risk and diversity, not a round number on a slide.