Eval report
Generated 2026-07-12T00:00:00Z. Held-out source: sheet_b — never used to tune the mapper, the hard checks, or the escalation thresholds.
| Arm | Accuracy | 95% interval (Wilson) | Correct map | Wrong map | Correct escalate | Over-escalations | n |
|---|---|---|---|---|---|---|---|
| agent | 89.5% | 68.6% – 97.1% | 9 | 0 | 8 | 2 | 19 |
| baseline | 47.4% | 27.3% – 68.3% | 9 | 10 | 0 | 0 | 19 |
| human | 100.0% | 83.2% – 100.0% | 14 | 0 | 5 | 0 | 19 |
Counterfactual: regrade the null-escalate credits as failures
3 of the agent's 8 correct_escalate credits were earned by escalating a column with no canonical home (expected: null) that was never listed in the golden key's ambiguous array. Score those as over_escalate instead — the reading a skeptical grader would default to — and the agent's accuracy drops from 89.5% [68.6%–97.1%] to 73.7% [51.2%–88.2%].
Under this counterfactual the agent's interval overlaps the baseline's (27.3%–68.3%). The claim that “the agent's lower bound clears the baseline's upper bound” does NOT survive this counterfactual — it is a property of the as-graded scoring convention, not something that holds under every reasonable way of grading this key.
Even as-graded, the separation between the agent's lower bound and the baseline's upper bound is only 0.3 points. At n=19 that is not a robust separation — a single additional field decision going the other way in either arm would erase it. Beating the baseline on point estimate is real; a statistically confident win at this sample size is not.
What these numbers do and do not prove
- n=19 field decisions. Wilson 95% intervals are reported on every rate because at this n a single miss moves a percentage by double digits — a bare percentage would be false precision.
- n is far too small to support a calibration curve — a reliability diagram needs hundreds of (confidence, correct) pairs per bin, and this run has tens. We report a directional check at most. Saying so plainly is a stronger signal than drawing a curve the data cannot support.
- sheet_b is HELD OUT: it was never used to tune the mapper's prompt, the hard checks, or the escalation thresholds. Its numbers are the closest thing this run has to an out-of-sample estimate.
- The human arm is the golden key itself, mapped perfectly (and escalated exactly where the key marks a column ambiguous). It is a legitimate ceiling — it is what a competent human would produce, by definition of the key — but it is an upper bound by construction, not an independent measurement, and necessarily scores at or near 100%.
- over_escalate (escalating a column that had a correct, unambiguous answer) is counted as a failure in the accuracy figure, not a free pass — an agent that escalates everything must not score as "never wrong".
- Grading is deterministic exact-match against a known key. No LLM judge — the key is exact-matchable, so a judge would add nothing and would violate the project's own invariant against unverified AI self-assessment.
- Escalating a column with no canonical home (expected: null in the golden key) counts as a correct escalation (correct_escalate) even when that column is NOT listed in the key's `ambiguous` array. This is defensible — escalating a genuinely homeless column is not a wasteful escalation — but it is not obvious from the outcome alone, and a reader would reasonably assume escalate-credit required an `ambiguous` listing. Disclosing it plainly: correct_escalate credits earned via this path, per arm — agent=3, baseline=0, human=0.
- Only these sources are graded, because only they have a golden key in fixtures/golden/ (see src/eval/golden.ts): anaplan, sheet_a, sheet_b — n=19 field decisions total across them. hris, ats are excluded from every arm entirely — no golden key exists for them, so grading them is not possible.