Shay
|
9288688640
|
feat: ADR-0119.3 — gsm8k_math lane runner (Phase 5.3) (#145)
Composes the Phases 1-4 pipeline (parser → solver → verifier →
realizer) into a per-case scoring decision: correct / wrong /
refused.
Outcome categorization (ADR-0114a Obligation #4):
parser ParseError → refused
solver SolveError → refused
verifier verdict failed → wrong
realizer error → wrong
answer/unit mismatch → wrong
all match → correct
`wrong == 0` is the load-bearing gate. The lane's overall_pass
holds only if wrong == 0 AND correct + refused == total.
Initial measurement on the Phase 5.2 corpus:
dev (50) : 50 correct, 0 wrong, 0 refused, overall_pass=True
public (150) : 150 correct, 0 wrong, 0 refused, overall_pass=True
Every correct case carries a trace_hash (64-char SHA-256) and
realized prose — full audit trail per case, consumable by ADR-0119.4
(frontier comparison), ADR-0119.6 (depth curve), and ADR-0120
(eventual expert-tier gate).
Tests: 13/13 green; 443 total green across runner + realizer +
solver + verifier; 67/67 smoke green.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
|
2026-05-22 17:37:54 -07:00 |
|