# PrinciPAL Faithfulness Eval

Evaluates the check PrinciPAL runs on every AI rewrite of a principal's note, against a human-written rubric.

| File | What it is |
|---|---|
| `RUBRIC.md` | Six-criterion faithfulness rubric + edge-case rulings |
| `cases.json` | 36 development cases, each graded (accept/reject) with failing criteria and rationale |
| `cases-holdout.json` | 16 cases written after v2 was frozen |
| `Validator.swift` | v1: the app's checking code, copied exactly from `PrinciPAL/ios/PrinciPAL/Capture/NoteRefiner.swift` |
| `ValidatorV2.swift` | v2 prototype: canonicalization + five meaning checks, wrapping v1 |
| `main.swift` | Runs both validators on both sets and scores them |
| `results.json` | Per-case outcomes and rejection reasons |

## Run

```bash
swiftc -O Validator.swift ValidatorV2.swift main.swift -o eval
./eval "$PWD"
```

## Results (Sept 29, 2026)

| | Unfaithful caught | Faithful wrongly rejected |
|---|---|---|
| v1 · development | 11 / 21 | 5 / 15 |
| v2 · development | 16 / 21 | 0 / 15 |
| v1 · held-out | 0 / 8 | 3 / 8 |
| v2 · held-out | 5 / 8 | 0 / 8 |

Still missed by v2: reversed stance, swapped actor/target, invented events, names at the start of a sentence, and clean pronoun swaps. These need a semantic check; see the case study page.

The v1 check logic was verified identical to the app source (whitespace/comments aside) with `diff`. The PrinciPAL app itself was not modified.