Calibrated Evaluation Loop for AI Outputs (Criteria Drift)
When you need this method
Your product ships AI outputs and nobody in the building can say with confidence whether they are good. You change a prompt, a model, or a knowledge source, and you learn what that did from complaints. The usual remedy, putting a model in the judge's seat, only moves the problem: that judge inherits the very weaknesses of the technology it is meant to assess, and nobody has checked whether its verdicts line up with those of your domain experts. Writing the criteria down in a workshop beforehand fails for a structural reason: what counts as good only becomes visible on actual outputs.
Approach
- 1Collect execution traces from real usage in one place and make them accessible enough that looking at them costs seconds, not half an hour of clicking.
- 2Read a sample of outputs by hand and write down the failure modes that actually occur. The criteria are born here, not earlier.
- 3For each failure mode, generate one or more candidate checks, either as a hard assertion in code or as a grader prompt to a model.
- 4Have the same sample graded by people with domain judgement and measure two quantities per check: what share of the bad outputs it catches, and how often it fires falsely on good ones.
- 5Keep only the checks that stay within a false-alarm threshold set in advance. Discard the rest, however plausible their wording.
- 6Sort the surviving checks by cost: level 1 is cheap assertions on every change, level 2 is human and model grading on samples at a fixed cadence, level 3 is an A/B test on live users and only pays off in a mature product.
- 7Extend the criteria catalogue on every pass, because each new review surfaces failure modes nobody knew about before.
Typical application
A typical B2B SaaS in the HR space has a language model draft job postings and candidate summaries. The team fixes its quality criteria in a workshop, turns them into a grader prompt, and reads a pass rate above ninety percent from it while support complaints keep rising. On the second attempt, two domain experts instead read one hundred and fifty real outputs and find four failure modes that appear in no workshop minutes, among them invented benefits that occur nowhere in the source material and mandatory details quietly dropped. Each failure mode gets a check: two can be expressed as cheap assertions in code, two need a grader prompt. Measured against the human grades, three of them clear the threshold and now run on every change. The fourth, a tone check, catches a lot but also fires constantly on good outputs, so it is discarded rather than carried forward as a pleasant-looking metric.
Limits and counter-indications
The method costs recurring grading time from people with domain judgement. Where nobody can be freed up for that on a regular basis, it stops after the first round and the catalogue goes stale. In its evidenced form it works on binary judgements, meaning acceptable or not acceptable. Where quality is a gradient, or where the graders disagree among themselves, agreement cannot be measured this way. The evidence base is thin: a qualitative study with nine experienced developers, an offline comparison on two pipelines, and the post-deployment phase explicitly out of scope. Criteria drift also means the catalogue is never finished; anyone who needs a fixed acceptance criterion for a contract will not get one here. The model-based judge brings biases of its own, among them position and verbosity effects and a tendency to favour its own outputs, and the frequently quoted agreement of over eighty percent comes from open-ended comparison of chat answers, not from a domain pipeline. Level 3 requires volume: below a certain amount of usage, differences are not detectable in an A/B test.
How to measure impact
Track two numbers per check against the human-graded sample, namely the share of bad outputs it catches and its false-alarm rate on good outputs. Also record how many candidate checks were discarded per round, and measure the time from an output being produced to the first human look at it.
Related methods
Tools for this
Sources
- 1.Shankar, Zamfirescu-Pereira, Hartmann, Parameswaran, Arawjo: Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences, UIST 2024 (arXiv:2404.12272) (opens in a new tab) · ACM, Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (Volltext über arXiv) · 2024-04-18 · academic and scholarly literature · describes the methodCarries the method in all its steps: generate criteria and candidate checks, implement them as Python functions or grader prompts, have a subset of outputs graded by people, and from that select the implementation that best matches the human grades. Agreement is defined as the harmonic mean of coverage and the complement of the false failure rate, with a user-set threshold on false failures. It also carries the criteria drift finding: people need criteria in order to grade outputs, yet they derive their criteria from grading, which is why evaluation criteria cannot be fully fixed in advance. Boundary: the evidence is a qualitative study with nine industry practitioners plus an offline comparison on two pipelines. The work operates on binary judgements, participants had little time to iterate, and the post-deployment phase is explicitly out of scope. No claim of improved product quality is made.
- 2.Hamel Husain: Your AI Product Needs Evals (opens in a new tab) · hamel.dev (Eigenveröffentlichung des Autors) · 2024-03-29 · practitioner source · describes the methodCarries the three-level cost ordering and the operating practice around it: level 1 is fast, cheap assertions that run on every change, level 2 is human and model grading on logged traces with ongoing monitoring of whether the grading model agrees with the humans, and level 3 is A/B testing on users, worthwhile only in a mature product. It also carries the demand to remove all friction from looking at data and to build purpose-made viewing tools rather than relying on generic platforms. Boundary: this is a field report from work on individual products without controlled measurement. The level ordering is a cost heuristic rather than a demonstrated result, and the magnitudes cited are the author's own observations.
- 3.Zheng, Chiang, Sheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 Datasets and Benchmarks Track (arXiv:2306.05685) (opens in a new tab) · NeurIPS 2023 Datasets and Benchmarks Track (Volltext über arXiv) · 2023-06-09 · academic and scholarly literature · limits the methodBounds the use of a model as judge and thereby justifies the calibration requirement. The work documents position bias across the compared answers, a preference for longer answers, a tendency to favour a model's own outputs, and limited reasoning ability. At the same time, strong judges reach over eighty percent agreement with human preferences on this task, matching the level of agreement between humans. Boundary: the measurement is open-ended pairwise comparison of chat answers in two purpose-built benchmarks. The figure holds for that task and does not transfer to arbitrary domain criteria in a product pipeline; it is precisely not a substitute for calibrating each case.
Origin: Shankar et al. (UC Berkeley) / Husain