Grading the grader: does my AI-rewrite checker actually work?

Why test it

In PrinciPAL, an on-device model rewrites a principal's quick notes about staff. Those notes become personnel records. The only thing between the model and the record is a validator that rejects rewrites that add numbers, invent names, flip perspective or drift in length.

I built that validator, and it had unit tests. But unit tests confirm that the checks I wrote do what I intended. They don't tell me what I didn't think to check. For that I needed a rubric that defines faithfulness independently of the code, and cases graded against it.

The rubric

One question: would a principal be comfortable signing this rewrite as an accurate record of their note? A rewrite must pass all six criteria:

The hard part of any rubric is the edge cases, so I wrote down the rulings too:

• Precision counts as addition. "a couple of parents" → "2 parents" fails R1.
• Expanding shorthand doesn't. "thurs" → "Thursday" and "third grade" → "3rd grade" pass.
• Keeping a hedge passes. "like a hundred essays" → "about 100 essays" passes; dropping the hedge would fail R5.

Test set design

I graded every case against the rubric before running the validator. The development set covers what the validator was built to catch, faithful rewrites it must let through, and two categories I suspected it would get wrong:

Baseline results

I copied the validator's checking code exactly (confirmed with a diff) and ran all 36 cases through it:

The overall accuracy (58%) isn't a meaningful real-world number, because the set deliberately concentrates on suspected weak spots. The per-category results are what matter.

What it missed

A dropped "not" reverses the record

A suspicion hardens into an accusation

The most dangerous one: actor and victim swapped

A false alarm: correct expansion rejected

The v2 fix

I prototyped v2 as a wrapper around v1, so nothing v1 already caught could regress. It adds two layers:

1 · Canonicalize before comparing

Both texts get the same shorthand expansion before v1 runs: day and month abbreviations, ordinals before words like "grade" and "period", "a hundred" and dotted acronyms. "thurs" and "Thursday" now compare as the same word.

2 · Five cheap meaning checks

• Negation count changed → reject (catches "did not" → "did")
• Future markers vanished → reject (catches "I'll meet" → "Met")
• Hedges vanished → reject (catches "may have taken" → "took")
• New causal word ("because", "due to") → reject (catches invented reasons)
• New judgment word ("excellent", "impressive") → reject (catches added praise)

I deliberately did not try to catch swapped actors, reversed feelings or invented events with keyword rules. Any rule I could write for those would be overfit to my own examples.

On the development set, v2 still misses those three plus two instructive ones. An invented name at the start of a sentence slips through, because v1 skips sentence-initial capitals, which are usually ordinary words. And a full swap ("she drafted it and sent it to me" → "I drafted it and sent it to her") passes, because v1 detects swaps by counting pronouns, and a clean swap leaves the counts balanced.

Held-out test

A fix designed by studying failures can simply memorize them. So after freezing v2, I wrote 16 new cases with different people, wording and situations, graded them, and ran both versions:

v2 held up: it caught every negation, tense, hedge, reason and judgment change in the new set, rejected no faithful rewrites, and regressed nothing v1 already handled.

The three it still missed are exactly the three classes I predicted. They are a reversed stance ("skeptical" → "supports"), a swapped actor ("a parent yelled at the secretary" → "the secretary yelled at a parent"), and an invented detail ("several classes were slow to exit").

What rules can't do

Keyword rules can catch changes to which words appear. They can't catch changes to who did what to whom. My recommendations for production:

1. Ship v2's rules. They're cheap, deterministic, and removed every false rejection in both sets.
2. Add a focused semantic check for the remaining classes: ask the on-device model yes/no rubric questions ("Does the rewrite change who performed the action?") rather than "is this faithful?", since narrow questions are easier to evaluate and audit.
3. Keep a human in the loop for actor changes. Swapped roles are the costliest error in a personnel record. When the people mentioned in a sentence are reordered, show the principal a side-by-side and require confirmation.
4. Grow the test set from real outputs, graded by a second rater, and track agreement between raters.

Limitations

• Constructed rewrites. The rewrites were hand-written to mimic model errors, not sampled from the model, so they measure the validator, not how often the model makes each error.
• Small sample. 52 cases is enough to expose failure classes, not to estimate rates precisely.
• One rater. I wrote and graded every case, including the held-out set, so an independent rater would make the result stronger.
• Brittle keywords. "since" can be temporal and "may" can be a month; I expect some false rejections at scale.

Reproduce it

Everything is in evals/principal-faithfulness:

• RUBRIC.md: the full rubric and edge-case rulings
• cases.json and cases-holdout.json: every case with its verdict, criteria and rationale
• Validator.swift (v1, copied exactly from the app) and ValidatorV2.swift
• results.json: per-case outcomes with rejection reasons

What this shows for AI training roles

• Writing a rubric that defines quality separately from the system being tested, with documented edge-case rulings
• Designing test sets that target failure classes, not just happy paths
• Weighing both error types: dangerous misses vs. costly false alarms
• Guarding against overfitting with a held-out set
• Being honest about what the numbers do and don't mean