Evaluation and Drift in Production
How do you know it is right before launch, and six months later when the corpus and the model have both changed - expert-built golden sets, a regression suite on a daily cadence, and online signals like correction and escalation rate.
Last updated:
How do you know the system is right before launch, and six months later when the corpus and the model have both changed? This is the question that decides the round, because it separates people who have shipped a demo from people who have operated one. We use an LLM as a judge is not an answer. The answer names who writes the golden set, what runs on every change, what runs daily, which online signal moves before a customer complains, and who gets paged when it does.
“Golden set built by domain experts, not engineers. Engineers write the questions the system already handles. Then: a regression suite on every change, drift monitors daily, and online signals - correction rate, escalation rate, reopen rate - because the offline set is frozen and the world is not.”
Clarifying Questions (Ask These First ~5 min)
| Question | Why it matters |
|---|---|
| Who owns the definition of correct - engineers or domain experts? | Engineers writing the golden set is the most common fatal mistake |
| Is there one right answer, or a range of acceptable ones? | Decides between exact match, rubric scoring and pairwise preference |
| What changes underneath you - corpus, model, prompts, or all three? | Each needs its own trigger, monitor and response |
| What does a wrong answer cost in this domain? | Sets the sampling rate for human review |
| May you log inputs and outputs at all? | In regulated settings the eval set may have to be synthetic or stay on-prem |
| Does the product already produce implicit signals - edits, reopens, clicks? | Free labels usually already exist and go unused |
| Who is paged when quality drops? | Quality without a named owner decays quietly |
Architecture (Draw This)
Say this as you draw it
Key Components (30-second pitch each)
- 1The golden set is built by domain experts - Engineers write the questions they already know the system handles. Lawyers, clinicians or support leads produce the questions users actually ask, and their disagreements expose ambiguous policy before launch. Version it in git and treat a change to it as a reviewed change.
- 2Evaluate retrieval separately from generation - Most quality failures are retrieval failures, and they are cheap to measure as recall at k against a labelled chunk. Mixing the two hides the cause and turns every regression into a prompt-tuning exercise that cannot succeed.
- 3Deterministic scorers first, judges only where necessary - Schema validity, citation presence, refusal when unsupported, forbidden phrases and exact match cost nothing and never drift. Use an LLM judge for the genuinely subjective remainder, validate it against a few hundred human labels, and re-validate whenever the judge model changes.
- 4Online signals see the real distribution - Correction rate - did the human edit it - plus escalation rate, reopen rate and abandonment move before any offline metric, because the offline set is frozen. In assisted workflows, edit distance is the highest-value signal you get for free.
- 5Drift has three separate sources - Input drift as users ask new things, corpus drift as documents change underneath the index, and model drift when the provider updates. Each needs its own monitor and its own response: expand the golden set, reindex and re-run retrieval cases, or pin the version and run the full suite.
- 6Pin and log every version - Model version, prompt version, index version and retriever config recorded on every request. Without them a regression cannot be attributed to a cause, and attribution is most of the work during an incident.
What Separates a Strong Answer
- 1Version the golden set - A strong candidate versions the golden set in git and treats changes to it as reviewed changes, ensuring traceability and accountability for every update.
- 2Use deterministic scorers first - They prioritize deterministic scorers for objective measures, reserving LLM judges for subjective cases, which reduces drift and maintains consistent scoring.
- 3Deploy canary releases - They implement canary releases for a small percentage of traffic to detect issues early, minimizing risk and ensuring stability before full deployment.
- 4Monitor implicit signals - They track implicit signals like edit distance and abandonment rates, which provide early warnings of issues that offline metrics might not catch.
What Runs When
| Cadence | What runs | Gate |
|---|---|---|
| Every change (CI) | Golden set: retrieval recall plus end-to-end scorers | Blocks the merge on regression beyond threshold |
| Every deploy | Canary on a small traffic slice, paired comparison | Automatic rollback on a significant drop |
| Daily | Drift monitors: inputs, retrieval scores, refusal rate, cost, latency | Alert and triage |
| Weekly | Stratified human review of sampled production traffic | New failure cases enter the golden set |
| Monthly | Golden-set review with domain experts; judge re-validation | Keeps the benchmark representative |
| On any provider model update | Full suite against the new version before switching | Never auto-upgrade a pinned model |
Signals and What They Actually Tell You
| Signal | Reads as | Watch out for |
|---|---|---|
| Thumbs up and down | Explicit, but rare and skewed to the angry | Response rates near one percent - never trend it alone |
| Human edit distance | The best cheap quality proxy in assisted workflows | It falls when reviewers get lazy, not only when quality improves |
| Escalation rate | End-to-end failure the user actually felt | Confounded by staffing levels and queue times |
| Reopen rate | The answer looked right and was not | Lags by days, so never use it as your fastest alarm |
| Retrieval score distribution | Corpus or query drift, early | Also moves on reindexing - annotate deploys on the chart |
| Refusal or abstain rate | A model, threshold or corpus change | A rising abstain rate can be correct behaviour, not a fault |
3 Biggest Risks
- 1Golden-set rot - The set stops resembling real traffic, so it passes green while users suffer. Mitigated by a monthly review against sampled production queries and by feeding every triaged failure back in as a new case.
- 2Judge drift - The judge model is updated and every historical score becomes incomparable, turning your quality trend into noise. Mitigated by pinning the judge version, recording it beside each score, and re-validating agreement against human labels on every change.
- 3No named owner - Dashboards exist, alerts fire into a channel, and nobody is accountable for the number. Mitigated by putting quality metrics on the same on-call rotation as availability, with a threshold that actually pages someone.
Google Stack: Vertex AI Evaluation Service for rubric and pairwise scoring → golden set versioned in git and run from Cloud Build → BigQuery for request logs, scores and drift queries → Looker dashboards → Cloud Monitoring alerts on abstain and escalation rate
Azure Stack: Azure AI Foundry evaluations (groundedness, relevance, custom evaluators) → suite run in Azure Pipelines as a merge gate → Application Insights for online signals → Azure Monitor workbooks for drift → Data Explorer for distribution comparisons
AWS Stack: Bedrock model evaluation plus custom scorers in SageMaker → suite run in CodeBuild as a merge gate → CloudWatch and Athena over request logs → QuickSight for the quality dashboard → EventBridge alerts routed to the on-call rotation
Other Options: Ragas or DeepEval for retrieval and faithfulness metrics, Promptfoo for prompt regression in CI, LangSmith or Langfuse for tracing and online signals, and a plain versioned JSONL golden set in git - which is the part that matters most and the part no tool can give you.
Frequently asked questions
How big should the golden set be?
Two to five hundred well-chosen cases beats five thousand scraped ones. Stratify by intent and difficulty, include the known hard cases, and deliberately include out-of-scope questions the system should refuse. Grow it from production failures rather than from imagination.
Can you trust an LLM as a judge?
For coarse, rubric-scored judgements with a validated prompt, yes - it correlates well enough to detect regressions. Validate against human labels, report the agreement rate, pin the judge model, and never present its score as an absolute quality number to a customer. It compares two versions of your own system.
The provider deprecates your model. What happens?
Because the version was pinned and the suite exists, this is a scheduled task rather than an emergency: run the suite on the new version, diff per case, fix the prompts where behaviour changed, canary, switch. Without the suite it is a rewrite with no way to know whether it worked.
How do you evaluate open-ended output with no ground truth?
Pairwise preference against the current production version, rubric scoring on the attributes you care about - coverage, faithfulness, format - and faithfulness checks against the source. Relative comparison is tractable where absolute scoring is not.
What do you do when the corpus changes underneath you?
Reindex on a schedule, keep index versions, and re-run the retrieval suite after every reindex. Write corpus-dependent cases whose expected answer is a chunk or clause identifier rather than free text, so a document rewrite does not invalidate the case.
Six months in, everything looks fine. What would you check?
Three things: whether the golden set still resembles this month's traffic, whether the abstain and escalation rates have shifted, and whether the model and judge are still the versions the baseline was measured against. In practice two of those three are usually wrong.