core/docs/decisions/ADR-0127-0128-RESULTS.md

ADR-0127 + ADR-0128 Results — Path-B Triggered

Status: Empirical result; load-bearing for the GSM8K-math arc decision Date: 2026-05-23 Author: CORE agents + reviewers Depends on: ADR-0126 (architecture), ADR-0127 (units pack), ADR-0128 (numerics pack), ADR-0114a (10 anti-overfitting obligations), ADR-0119 (+ all 8 sub-phases), ADR-0120 (expert promotion contract), ADR-0121 (math expert promotion deferred)

---

TL;DR

The ADR-0126 → 0127 → 0128 arc shipped the full deterministic design that the parser-by-rule + units-substrate hypothesis required. The empirical result on the GSM8K train sample is:

correct  =  0 / 50
wrong    =  0 / 50  (wrong == 0 preserved)
refused  = 50 / 50

Per ADR-0127's exit criterion and the Path-A vs Path-B decision documented in ADR-0126: the deterministic parser-by-rule architecture, with full units + numerics substrate, does not move the GSM8K-math lane. This is the real Path-B trigger. The math expert promotion path retargets to a benchmark where exact recall and determinism are the discriminators — see "Recommendation" below.

---

What was shipped

Layer	Module / Pack	What
Architecture	generate/math_roundtrip.py	Round-trip admissibility primitive (26 tests, ADR-0126 P1)
Architecture	generate/math_candidate_parser.py	Candidate-emitting sentence parser (17 tests, ADR-0126 P2)
Architecture	generate/math_candidate_graph.py	Branch enumeration + decision rule (22 tests, ADR-0126 P3)
Architecture	evals/gsm8k_math/runner.py::_score_one_candidate_graph	Runner wiring (9 tests, ADR-0126 P4)
Substrate	language_packs/data/en_units_v1/	284 lemmas, 401 conversion edges, NIST/ISO provenance (Gemini, PR #164)
Substrate	language_packs/data/en_numerics_v1/	130 lemmas across cardinals/ordinals/fractions/multipliers/quantifiers/comparison-anchors/format-rules (Opus #2, PR #163)
Loader	language_packs/loader.py re-exports	Single import path for both packs (ADR-0127/0128 deferred coordination)
Integration	generate/math_candidate_parser.py::_canonicalize_unit	Pack-aware unit canonicalization — handles irregular plurals (feet, children, etc.) via pack lookup
Integration	generate/math_candidate_parser.py::extract_initial_candidates	Widened to <Entity> has N <unit> [of <substance>] + There are N <unit> [in <place>] shapes
Integration	generate/math_candidate_parser.py::_is_indefinite_quantifier	ADR-0128.4 quantifier-driven refusal (some, many, etc. → no candidate emitted; preserves wrong == 0)
Integration	generate/math_candidate_parser.py op-pattern trailing prep	Added of / for / with to the discardable preposition tail (ADR-0127 substance qualifier)
Integration	generate/math_roundtrip.py::_value_grounds	Pack-backed cardinal lookup (widens word-number coverage from hard-coded 0-12 to full numerics pack)
Integration	evals/gsm8k_math/train_sample/v1/runner.py	Swapped _score_one → _score_one_candidate_graph

What measurably works (synthetic verification)

The candidate-graph architecture + pack substrate solves the kinds of problems it was designed to solve. Six tailored synthetic cases verify end-to-end:

Case	Result
Jan has 5 apples. Jan buys 3 apples. How many apples does Jan have?	✓ 8
Sam has 10 feet of rope. Sam uses 3 feet of rope. How many feet does Sam have?	✓ 7 (non-count unit; substance qualifier)
There are 5 kids in camp. How many kids do they have?	✓ 5 (implicit-subject shape)
Sam has 10 dollars. Sam spends 3 dollars. How many dollars does Sam have?	✓ 7 (money)
Sam has 5 hours. Sam uses 2 hours. How many hours does Sam have?	✓ 3 (time-dimension unit)
Sam has 10 children. Sam loses 2 children. How many children does Sam have?	✓ 8 (irregular plural via pack)

1050/1050 existing test regression suites green across math, ADR-0126, ADR-0127 pack ratification, ADR-0128 pack ratification, and runner. Zero regressions from the integration work.

What did not move (empirical reality)

The 50-case GSM8K train sample stays at 0 correct / 0 wrong / 50 refused. Inspection of refusal causes shows that real GSM8K problems carry compound linguistic structure that no pack adds on its own:

Refusal class (from baseline categorization)	Train sample share	Pack-addressable?
OTHER_SHAPE (subordinate clauses, multi-word entities, possessives, pronouns across statements)	27 / 50	No — these need parser grammar work, not packs
NON_COUNT_UNIT (feet, hours, etc.)	8 / 50	Partial — pack helps single-statement, but problems still chain across multiple statements that other gaps refuse
MONEY ($N)	5 / 50	Partial — same multi-statement compound issue
RATE (per, each)	5 / 50	Partial
INDEFINITE_QUANT (some, few)	3 / 50	Yes — pack refuses cleanly, but refusal isn't correct
CONTAINER_OF	1 / 50	Partial
THERE_ARE	1 / 50	Yes (now parses), but other statements in same problem still refuse

The structural problem: a 3-5-sentence GSM8K problem refuses if any sentence has no admissible candidate. P(problem passes) = P(sentence passes)^N. The pack work raised per-sentence parse rate measurably on simple shapes, but the joint pass rate stayed at zero because every real problem contains at least one sentence the parser still can't handle.

Why this is the Path-B trigger ADR-0126 named

ADR-0126's exit criterion documented two outcomes:

If passed: if 50-case train sample shows correct ≥ 10/50 with wrong == 0, the architecture is validated; run the sealed holdout.

If missed: if 50-case train sample shows correct < 10/50, the parser-by-rule architecture (in any topology) is the wrong abstraction for GSM8K coverage. ADR-0126 itself is deferred and the work pivots to benchmark re-selection.

ADR-0127's exit criterion sharpened this: re-run train sample with units pack mounted; if still missed, the failure is real (architecture + substrate both insufficient).

ADR-0128 added the numerics pack to the same exit criterion.

All three packs are mounted, the architecture is in place, the substrate is exhaustive (284 unit lemmas + 130 numeric lemmas + 401 conversion edges), and the result is 0/50.

This is the moment the architectural arc was designed to surface a decision. The deterministic design is correct, load-bearing, and complete; it does not produce GSM8K coverage because GSM8K's linguistic distribution is not parseable by any deterministic rule set at the rate the substrate enables. The 27/50 OTHER_SHAPE refusals are the empirical evidence that the gap is grammar coverage of paraphrase variance, not any specific missing pack lemma.

What wrong == 0 actually bought us

Despite 0/50 correct, the wrong-zero discipline produced a sound, replay-deterministic, audit-trail-complete pipeline that:

• Refuses honestly on every case it cannot handle
• Carries pack-grounded provenance on every emitted operation
• Round-trip-verifies every parsed slot against source tokens
• Passes every adversarial gate
• Composes with the existing teaching subsystem's reviewed- correction discipline

This is genuinely useful infrastructure. The verdict is "wrong benchmark for this architecture's strengths," not "the architecture is bad."

Recommendation: Path B

Three sub-decisions:

1. Demote GSM8K-math lane from the math expert promotion contract.

GSM8K is retained as a stress test and as a "we honestly refuse on this distribution" demonstration, but correct_rate on GSM8K is removed from the ADR-0120 expert-promotion gate for the mathematics_logic domain.

2. Re-target the math expert promotion to a benchmark where exact-recall + determinism are discriminators.

Candidates:

• MATH symbolic subset — symbolic-equivalence problems where exact algebraic recall is the right primitive
• CORE-native teaching-corpus eval — problems sourced from ratified teaching chains, where the parser's grammar exactly matches the corpus's surface forms by construction (no paraphrase-variance gap)
• A curated word-problem set with bounded grammar (e.g., Khan Academy style problems pre-filtered to single-sentence arithmetic shapes)

A separate ADR (proposed: ADR-0131) scopes the re-targeting decision and exit criteria.

3. ADR-0126 / 0127 / 0128 substrate stays in main as load-bearing infrastructure.

The candidate-graph topology, the round-trip admissibility primitive, the units pack, the numerics pack, and the deterministic pack-aware parser are useful for:

• Any future deterministic word-problem benchmark
• The teaching corpus's own evaluation lane (where grammar match is by-construction)
• Future cross-language packs (es_units_v1, es_numerics_v1) that reuse the architecture
• Operator interaction surfaces where deterministic refusal + honest provenance matter more than raw coverage

Do not revert. This work proved a hypothesis correctly even though the hypothesis didn't pan out for GSM8K.

What this does NOT recommend

• Does not recommend abandoning the deterministic engine philosophy. The architecture works; the benchmark choice was the error.
• Does not recommend pulling in LLM-assisted parsing or any opaque component. The contract integrity is intact and worth preserving.
• Does not recommend more parser regex expansion within the current architecture for GSM8K. Four previous ADRs in that shape produced 0 lift; this one (the architectural pivot) did the same. The treadmill has been independently characterized twice now.

Composition with deferred backlog (ADR-0129 / ADR-0130)

The deferred teaching-loop ADRs (spaced-correction-replay, pre-articulation-calibration) become more interesting once Path B lands a new benchmark, because:

• A benchmark where the parser handles by-construction grammar cleanly will produce a stable correction-store population — the precondition ADR-0129 named for un-deferral.
• A new benchmark's per-version calibration cohorts give ADR-0130 a real signal to measure.

These remain deferred under the new path, but their un-deferral exit criteria become reachable.

PR checklist

What capability did this add?
  → The integration layer that wires en_units_v1 + en_numerics_v1
    into the ADR-0126 candidate-graph parser. Pack-aware unit
    canonicalization, indefinite-quantifier refusal, substance-
    qualifier handling, There-are initial shape, pack-backed
    cardinal grounding. Also the Path-B trigger evidence.
What invariant proves the field remains valid?
  → wrong == 0 preserved on train sample (0/50 wrong). Trace
    determinism preserved. Round-trip admissibility extended
    with pack-typed unit grounding.
Which CLI suite/eval proves the lane?
  → smoke + math + packs + train_sample_runner. All 1050 tests
    green; train sample re-runs deterministic 0/0/50.
Did this avoid hidden normalization, stochastic fallback,
approximate recall, unreviewed mutation?
  → Yes. Pack lookups are deterministic. Indefinite quantifier
    refusal is a deliberate hard-no, not a probabilistic
    threshold. No LLM fallback added.
If it touches user input, what trust boundary was enforced?
  → No new user-input surfaces. Pack loaders already validate
    pack_id via safe_pack_id (ADR-0051). Train sample is
    unsealed by design (drawn from GSM8K train split, not
    holdout).