Repeated AI Eval Runs Are Not Independent Test Cases

Running twenty questions ten times each gives you 200 executions, but it does not give you 200 independent examples of user needs. Repeats help measure consistency on familiar cases. New cases help measure coverage. An AI evaluation report should show both counts before it shows a confidence interval.

This Wednesday lesson extends building a golden set that can explain a regression. Once the fixtures are stable, the next question is what the experiment can support. A larger result file can conceal a very small experiment.

The counts and acceptance rules below are synthetic teaching examples, not measured model performance or employer data.

Find the unit behind the row

Consider a document assistant. A test case contains a question, a document snapshot, and an expected behavior. The harness sends that case through retrieval and generation, then grades the answer. Running the same case again creates another observation inside that case.

For this exercise, define the target as average success across equally weighted, independently selected questions from a specified task population. That definition matters: it determines what receives a vote. A production-traffic-weighted score would answer a different question and require defensible traffic weights.

There can also be larger groups. Ten questions derived from one policy document may share the same ambiguity or retrieval defect. If the intended claim concerns performance on new documents, the document is a candidate sampling unit, with questions nested inside it.

NIST illustrates the distinction between short-term repetitions and variation across days in its two-level measurement design. The QA application is an analogy: record the levels that can vary instead of flattening every observation into an interchangeable row.

A synthetic score with two different meanings

Suppose twenty distinct questions each receive ten fresh executions. Fifteen questions pass all ten times; five fail all ten times.

The execution pass rate is 75%. The equally weighted mean of question pass rates is also 75%. Matching point estimates do not make their uncertainty calculations interchangeable.

As a deliberately simplified illustration, the independent-Bernoulli standard-error formula is sqrt(p * (1 - p) / n). Substituting 0.75 and 200 gives about 3.1 percentage points. Using twenty independent question outcomes gives about 9.7 points. In this constructed example, every repeat follows its question exactly, so treating the 200 rows as independent exaggerates the breadth of evidence.

Those are illustrative standard errors, not validated release intervals. In a real suite, questions can have intermediate pass rates, shared documents, and selection bias. Changing the denominator alone does not solve all of those problems.

Compare versions on the same cases

To compare a baseline with a candidate, run both on the same frozen cases with the same repeat budget. Preserve every planned outcome, including timeouts and invalid responses. Define their treatment before execution; silently dropping them changes the experiment.

For each question, calculate the candidate pass rate minus the baseline pass rate. Then average those question-level differences. This makes an improvement on a hard question visible alongside a regression on an easy one.

The principle follows NIST guidance on paired observations: corresponding measurements stay paired, and analysis uses their differences. Pair by case identity, not by whichever two rows happen to be adjacent in a file.

I recommend randomizing or balancing version order within execution blocks and recording service conditions. Running every baseline request in the morning and every candidate request during an outage would mix the version comparison with a time effect. A shared random seed is useful metadata, but does not establish that two different systems made equivalent random choices.

Keep the grouping in the uncertainty estimate

For a simple case-level analysis, first produce one baseline mean and one candidate mean per question. If questions are the independent sampling units, resample question indices while keeping each baseline/candidate pair together, then recompute the average difference.

SciPy documents paired bootstrap resampling: enabling the paired option applies the same sampled indices to the supplied arrays. Supplying case-level arrays preserves this pairing. Supplying 200 flattened execution rows does not magically recover the missing question groups.

This simple analysis targets variation across sampled cases using their observed means. If repeat-level uncertainty is material, use an analysis designed for the nested experiment. If questions share documents, consider resampling documents with their questions intact. Few independent groups, a curated convenience sample, or degenerate outcomes limit what any interval can establish. Document those limits beside the score.

Test the evaluation pipeline itself

Before using the report as a release gate, try these controls on synthetic data:

1. Copy every raw result row. The importer should reject duplicate execution IDs, not report twice as much evidence.
2. Shuffle the result order. Case pairing and the final score should remain unchanged.
3. Give one case many extra repeats. An equally weighted case score should retain its weighting; the planned-budget check should flag the mismatch.
4. Remove candidate results for one case. The comparison should report an incomplete pair instead of quietly dropping the case.
5. Attach several questions to one document. Verify that a document-level analysis preserves that group.

Keep case ID, document ID, execution ID, version, repeat index, outcome, and grader version in the exported record. Report unique cases, independent groups, completed runs, missing outcomes, and the weighting rule together.

Choose the release threshold before looking at the comparison, and review critical regressions separately from the average. Repeating a familiar question can reveal instability. It cannot substitute for asking a question the suite has never covered.