Integrates en_units_v1 (#164) + en_numerics_v1 (#163) into the
ADR-0126 candidate-graph parser. Loader merge (re-exports from
numerics_loader.py give single import path), pack-aware unit
canonicalization (handles irregular plurals like feet/children
via lookup_unit), indefinite-quantifier refusal (ADR-0128.4 —
'some'/'many' emit no candidates, preserving wrong==0), and
widened initial-possession shapes:
- <Entity> has N <unit> [of <substance>] (ADR-0127 substance qualifier)
- There are N <unit> [in <place>] (implicit-subject shape)
Plus: pack-backed cardinal grounding in math_roundtrip._value_grounds
(widens word-number coverage from hard-coded 0-12 to full numerics
pack cardinal table + compound rule). Op-pattern trailing prep
alternation gains of/for/with for substance qualifiers.
REGRESSION: 1050/1050 tests green across math + ADR-0126 + ADR-0127
ratification + ADR-0128 ratification + runner.
EMPIRICAL RESULT (the Path-B trigger ADR-0126/0127/0128 named):
correct = 0/50 wrong = 0/50 refused = 50/50
on evals/gsm8k_math/train_sample/v1/cases.jsonl
Per ADR-0127's exit criterion (correct >= 10/50, wrong == 0):
**MISSED** — the full deterministic design (candidate-graph
topology + units pack + numerics pack + pack-aware parser) does
not move the GSM8K-math lane. This is the real Path-B trigger.
WHAT WORKS (synthetic verification, 6/6 cases solve end-to-end):
- 'Jan has 5 apples. Jan buys 3 apples. ...' -> 8
- 'Sam has 10 feet of rope. Sam uses 3 feet of rope. ...' -> 7
- 'There are 5 kids in camp. ...' -> 5
- 'Sam has 10 children. Sam loses 2 children. ...' -> 8
- (money + time-dimension variants pass)
WHY GSM8K STAYS AT ZERO: real GSM8K problems carry compound
linguistic structure (pronouns across statements, possessives,
subordinate clauses, multi-word entities, multi-step inference)
that no amount of pack vocabulary addresses. Per-sentence parse
rate improved measurably on simple shapes; joint problem-level
pass rate stayed at zero because every real problem contains at
least one sentence the parser still cannot handle.
Full results + Path-B recommendation in
docs/decisions/ADR-0127-0128-RESULTS.md. The substrate
(architecture + packs) stays load-bearing in main; the math
expert promotion path retargets to a benchmark where exact
recall and determinism are the discriminators (proposed
ADR-0131).
Diagnostic from ADR-0126's first train-sample run (0/0/50): every
refusal happens at the first statement of each problem, and every
refused first statement fails on the unit-of-measurement construction,
not on the operation grammar. Adding more verb regexes is the per-axis
treadmill that produced 4 zero-lift ADRs. Units form a finite, externally
well-defined ontology (NIST SI tables, currency, English container nouns)
that is semantic substrate the candidate-graph parser was designed to
consume.
Scope:
- en_units_v1 pack: dimensions, units (<=60), containers, rate connectors
- conversions.jsonl: directed weighted graph of within-dimension unit pairs
- 3 new initial-possession shapes + rate-declaration extractor in the
candidate parser
- Round-trip filter gains optional pack-typed-unit check
- Solver gains dimensional canonicalization helper (shortest path through
conversion graph); fired edges join SolutionTrace.steps for replay
- Pack ratification invariants: round-trip identity, per-dimension
connectivity, path consistency, canonical unit per dimension
Wire the same train-sample exit criterion as ADR-0126 (correct >=10/50,
wrong==0). If passed -> sealed holdout. If still missed -> Path B
trigger is REAL (full deterministic design with units substrate failed),
demote GSM8K, re-target math expert promotion.
Also commits the empirical evidence: train_sample/v1/runner.py swapped
_score_one -> _score_one_candidate_graph; report.json baseline 0/0/50
confirming the candidate-graph topology refuses cleanly without units
substrate.
P3 — generate/math_candidate_graph.py:
Branch enumeration over per-sentence candidate choices (Cartesian
product, cap=64). Per-sentence ambiguity tiebreaker via most-grounded-
slots-wins (transfer beats subtract when 'to Tom' grounds). Decision
rule: 0 admissible -> refuse; 1 -> emit; >=2 same answer -> emit;
>=2 different answers -> refuse (preserves wrong==0 on genuine
ambiguity). End-to-end parse_and_solve(text) -> CandidateGraphResult.
Question extractor added to math_candidate_parser.py (CandidateUnknown,
total + entity question shapes mirroring math_parser).
22 new tests. Permissive verbs ('bought', 'ate', 'bakes') now produce
correct answers via the candidate-graph path; ambiguous 'gives to Tom'
resolves to transfer reading (Tom gets the apples) deterministically.
P4 — evals/gsm8k_math/runner.py:
New sibling function _score_one_candidate_graph(case) -> CaseOutcome.
Identical shape to _score_one; swaps parse_problem for parse_and_solve;
preserves verifier/realizer/expected-answer stages. Callers (e.g.
PR #160's train_sample/v1/runner.py) substitute the new function in
one line to evaluate the candidate-graph topology.
9 new wiring tests. Three groups:
- No regression: cases legacy solves, new also solves.
- Lift: cases legacy refuses, new solves (the architectural payoff).
- Wrong==0: out-of-grammar refuses, never wrong.
Regression: 714/714 existing math + runner tests still green.
ADR-0126 total: 74/74 tests green across P1+P2+P3+P4.
Wraps existing math pipeline (parser -> solver -> verifier) against
PR #159's 50-case train sample. Emits deterministic report.json with
per-case verdicts. CLI exit code reflects exit criterion
(correct >= 10 AND wrong == 0).
Baseline against current parser: 0 correct / 0 wrong / 50 refused.
This baseline is the inner-loop gradient signal for ADR-0126's
candidate-graph parser (in flight on feat/adr-0126-candidate-graph).
Registers tests/test_adr_0126_train_sample_runner.py under
'core test --suite math' so the wrong == 0 invariant becomes a hard
CI gate per ADR-0114a Obligation #4 (refuse rather than confabulate).
Depends on PR #159 (gemini/adr-0126-train-sample). Rebase onto main
after #159 lands.
The 1,319 GSM8K test cases are now sealed at
evals/gsm8k_math/holdouts/v1/cases.jsonl.age, age-encrypted to the
ADR-0119.1 recipient. Plaintext never touched disk in the working
tree; only ciphertext is committed.
First honest CORE-vs-real-GSM8K measurement
cases_total: 1319
correct: 0
wrong: 0 ← ADR-0114a Obligation #4 holds against external corpus
refused: 1319
overall_pass: True
Zero confabulation. Parser refuses what it can't grammar-handle; the
"wrong == 0" discipline survives the move from CORE-original cases
to a real public benchmark. The 0/1319 correct rate is the truthful
gap that ADR-0120's threshold work will quantify.
What landed
scripts/seal_gsm8k_test.py
- Loads GSM8K via datasets.load_dataset("openai/gsm8k", "main")
- Strips worked-solution prose; extracts final-answer integer/float
after "####" (handles "2,125" → 2125 thousands-separator)
- Reads recipient from docs/holdout_recipients.txt (single repo key
per ADR-0119.1)
- Encrypts via pyrage; writes only ciphertext
- Refuses to overwrite test path with train-derived seal
evals/gsm8k_math/runner.py
- Empty expected_unit (sentinel) skips unit-comparison; grades on
answer value alone. Required because GSM8K answers carry no unit
structurally. wrong-zero discipline preserved.
tests/test_adr_0119_7_sealed_gsm8k.py — 6 invariants:
1. sealed file present + age-formatted
2. no plaintext companion files (sibling-leak guard)
3. decrypted JSONL matches documented schema
4. runner against decrypted suite produces wrong==0
5. tests skip (not fail) when CORE_HOLDOUT_KEY unset
6. case ids match "gsm8k-test-NNNN" pattern
Defensive gitignore: plaintext patterns under
evals/gsm8k_math/holdouts/v1/ are explicitly excluded.
ADR-0114a obligation roll-up
10/10 discharged for the gsm8k_math lane:
#1 ✓ sealed-holdout (fab_control + GSM8K test)
#2..#10 ✓ as before
Phase 5 status: 5.1..5.7 done; 5.8 in flight (PR #149). After 5.8
merges, ADR-0120 (first expert promotion contract) becomes
feasible.
Test plan
- pytest tests/test_adr_0119_7_sealed_gsm8k.py with CORE_HOLDOUT_KEY → 6/6
- pytest without CORE_HOLDOUT_KEY → 3 pass + 3 skip
- core test --suite smoke -q → 67/67
- CLAIMS.md regenerated (no diff)
- HF token NEVER in repo (saved at ~/.cache/huggingface/token, mode 600)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Audit follow-ups from #145/#146 merge review. Five small fixes; no
behavior change on the green path, but failure modes are now explicit
rather than silent.
ADR-0119.6 depth_curve.py
- Add DepthCurveError typed exception
- Raise on case_id missing from lane_report (was: silent → "refused")
- Raise on depth >= 9 (was: silent new bucket key)
- Two new tests pin both refusals
- Removed stale sys.path hack at module top
ADR-0119.4 frontier-baseline tests
- Assert comparison_v1.json's core_measurement reports wrong == 0
(the load-bearing differentiator named in the disclaimer; a
tampered file with wrong > 0 was previously syntactically valid
and would have passed all old assertions)
- Assert frontier citations are dated 2023 or later (freshness
guard; older citations should be refreshed before ADR-0120
gates anything for `expert` promotion)
Tests
- tests/test_adr_0119_6_depth_curve.py: 7 → 9
- tests/test_adr_0119_4_frontier_baseline.py: 5 → 7
- 29/29 across runner + depth-curve + frontier suites; 67/67 smoke
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
