What the judge does instead of judging

Using a model to grade another model's output is now the default way to run an eval. It is cheap, it scales, and it produces a number. The step almost nobody does is the one that makes the number mean anything: score the same answers yourself, blind, and find out where you and the judge disagree.

I did that. Forty questions in my own field, three models answering them, five criteria, and then eighteen answers scored by hand against the same rubric the judge was given.

The judge turned out to be very good, right up to the point where the task stopped being a check and started being a judgement. There, it did something specific and measurable instead.

The setup

The golden set is forty questions about infrastructure, incidents and observability, written before any model saw them. Each carries two lists: the claims that make an answer correct, and the confident wrong turns an answer must not make. The second list is the point. A model that states something plausible and wrong in its own words reads as clean to any string matching, and those are the answers that send someone debugging in the wrong direction for an hour.

Five criteria, each with every level anchored:

• coverage, four levels, how many of the listed claims the answer actually makes
• no_forbidden_claim, a gate, whether it asserts one of the wrong turns
• accuracy, whether anything else it says is false
• answers_the_question, whether it delivers what was asked
• economy, whether everything in it earns its place

The judge never sees the reference answer, only the question and the claims. It never sees which model wrote what. Neither did I: the labelling sheet has no model names in it, and the mapping lives in a separate file that the scoring page never reads and I did not open until I had finished.

Then the two sets of scores get joined on the answer both of us saw.

Where it agrees, and where it stops

Filled is where we scored identically. The outline is where we were never more than one band apart.

Both numbers matter and quoting either alone is a way of selling the result rather than reporting it. A four level scale where two raters are never more than one band apart is behaving well even at 39% exact. A two level gate at 39% exact would be broken.

Read top to bottom, there is a cliff. The criteria where the rubric points at a written list and asks whether something is present come in at 94% and 83%. The criterion that asks for a graded judgement about how much of a list is covered comes in at 39%.

That could just mean coverage is harder. What makes it more than that is what the judge substituted.

It reaches for length, in both directions

Same eighteen answers, same rubric, same moment. The person barely moves.

On coverage, my scores correlate with how long the answer was at +0.42. The judge's correlate at +0.79. On economy, which is explicitly about padding, mine correlate at -0.11, which is close to nothing, and the judge's at -0.59.

So the judge rewards length on one criterion and punishes it on another, while a person scoring the same text does neither strongly. That is not a judge with a length bias in the usual sense of a thumb on the scale. It is a judge using length as a proxy for a property it cannot otherwise assess, and picking the direction that plausibly fits each criterion.

My own +0.42 on coverage is worth sitting with, because it is not zero. Longer answers genuinely can make more of the listed claims, and that is exactly why the judge's number alone would prove nothing. The human score is the control. Without it, +0.79 is just a number that could mean the answers really were better when longer.

The disagreements say the same thing without any statistics. Every large gap on a long answer runs one way:

268w  inc-blameless-postmortem     me 2   judge 3
236w  net-dns-cutover              me 1   judge 3
208w  net-502-vs-503               me 1   judge 3
200w  obs-metric-cardinality       me 2   judge 3
122w  cloud-multi-az-vs-region     me 1   judge 3

Three of those are two full bands apart on the same text. Where I said the answer made a minority of the claims and missed one that mattered, the judge said it made all of them. Every disagreement in the other direction is on a short answer: 71 words, 35 words, 27 words.

The one that actually matters

The gate is the criterion that cannot be traded off. An answer that confidently states one of the wrong turns is not eighty percent right because the rest was fluent.

We agreed on 16 of 17. The one we did not was a question about S3 consistency where an answer asserted a forbidden claim, I failed it, and the judge passed it.

I re-ran the judge from scratch after revising the rubric, on a different day, and it missed the same answer again. So it is reproducible rather than a sampling artefact: a false negative on the only criterion where a false negative is expensive, at a rate of about one in seventeen.

That single result is worth more than the agreement percentages. A judge that is 94% accurate on the gate sounds excellent until you say it the other way round, which is that roughly one confidently wrong answer in twenty gets through with a clean bill of health.

Three things I got wrong building this

An anchor has to be a procedure, not an adjective. My first attempt at economy was anchored on "a substantial part of it does no work" and "noticeable filler". I could not apply that consistently to my own rubric. It now asks the rater to mark the sentences they would cut and band by how much that removes: a third or more, well under a third, nothing. Same for answers_the_question, which now asks you to name what the question wanted before you go looking for it. A rater who cannot repeat their own scores caps agreement before the judge has said anything.

The eval set has to be inside the expert's domain. I wrote forty questions across five themes and two of them were subjects I have not worked in properly. Trying to score those produced hesitation rather than data. Restricting the sheet to the three areas I can judge fast cut it from thirty answers to eighteen and made the remaining eighteen worth having. "Agreement on questions where the rater is a domain expert" is a defensible claim; agreement across a set where the rater was guessing on a third of it is not, and afterwards you cannot tell a guess from a real disagreement.

A refusal is not a match. The sheet lets a rater decline a criterion rather than guess, and those are dropped from the comparison rather than counted as agreement. Counting them would inflate the figure using precisely the cases the rater found hardest, which is backwards. The rate of refusals is also a measurement of the rubric rather than the rater: a criterion nobody can apply is a badly written criterion, and it shows up there before it shows up as poor agreement.

What would overturn this

One number is missing, and it is the one that decides whether the headline holds.

I have not yet scored the same answers twice. Until I do, "the judge disagrees with me 61% of the time on coverage" has an innocent explanation sitting right next to it: that coverage is simply hard to apply twice, and I would disagree with myself at a similar rate. If that turns out to be true, the judge is fine and my scale is the problem.

That is the next thing I will run, cold and shuffled, and I will publish the number either way. It is the honest ceiling on what any judge could score against me, and almost no published eval reports it, which is most of why it is worth having.

Eighteen answers, one rater, one sitting. This is a signal, not a settled result. But it is a signal produced by actually checking, and the checking is the part that keeps getting skipped.