When the judges are uncertain: measuring revision gains without an infallible evaluator

Under what evaluation design does an apparent revision gain survive plausible uncertainty and differential bias in the labels used to measure it?

Proposes a measurement protocol that preserves unresolved and joint labels, protects a random audit sample, and estimates judge error on the comparison being claimed. Includes 30 exact offline diagnostic checks on invented data; no new benchmark, inference, expert annotation, or calibrated evaluator.

• Exact constructed example: 97.5% snapshot-label accuracy coexists with a reversed comparison (true −2 points, judged +3 points). This is a possibility result on invented labels, not a measured Noetis failure rate.
• Exact constructed example: a pooled confusion correction restores the overall mean to 79% while the corrected comparison stays in the wrong direction (+3.4054 points).
• Exact constructed example: six valid identical-artifact couplings narrow whole-cohort bounds from [0, 20] to [6, 14] points without resolving either artifact's correctness.

• E14-A Exact offline label-uncertainty diagnostics Executed (diagnostic)
• E14-B Masked-replay comparison of evaluation methods Proposed

• H14-1 Evaluation error differs meaningfully between ordinary and receipt-based revision outputs.
• H14-2 The protected-audit protocol makes more correct, informative promotion decisions than naive judge or disagreement-only evaluation at the same budget.
• H14-3 Valid joint constraints reduce evaluation uncertainty at useful cost.

None stated in the published round.

• Does evaluation error differ meaningfully between ordinary and receipt-based revision outputs?
• Does a protected-audit protocol make more correct, informative promotion decisions than naive or disagreement-only evaluation at the same budget?
• Do valid joint constraints reduce evaluation uncertainty at useful cost on naturally occurring outputs?

Builds on Round 13, Round 12. Continued in Round 18, Round 17, Round 16, Round 15.