Docs-only PR. Follow-up to ADR-0127-0128-RESULTS' Path-B trigger.
Replaces ADR-0120's GSM8K-coverage requirement for the
mathematics_logic expert promotion with a composite gate over
three architecture-aligned benchmarks:
Benchmark 1: Symbolic equivalence (~500 algebra cases, public
+ sealed split). Primary discriminator — CGA exact-recall.
Pass: correct_rate >= 0.95, wrong == 0.
Benchmark 2: CORE-native teaching-corpus eval. Internal
consistency lane sourced from ratified packs (en_arithmetic_v1
+ en_units_v1 + en_numerics_v1). Grammar matches by
construction (no paraphrase-variance gap).
Pass: correct_rate >= 0.95, trace_hash byte-equality, wrong == 0.
Benchmark 3: Bounded-grammar word-problem set (~150 cases,
pre-filtered to ADR-0126/0127/0128 parser scope). Demonstrates
end-to-end pipeline determinism on the in-scope distribution.
Includes adversarial out-of-grammar probes.
Pass: correct_rate >= 0.95, wrong == 0.
Three benchmarks (not one) prevent overfitting per ADR-0114a
and force the math expert to demonstrate three distinct
architectural properties (algebraic correctness, internal
consistency, end-to-end pipeline determinism).
GSM8K stays as a stress-test lane the math expert RUNS but is
NOT gated on. Refusal counts reported in expert-claims artifact
as honest disclosure.
Other 12 ADR-0120 expert-promotion checks UNCHANGED.
Status: Proposed. The contract revision (removing GSM8K from
the gate) is a follow-up implementation PR after sub-phases
0131.1-0131.6 land the new benchmark substrate.
Integrates en_units_v1 (#164) + en_numerics_v1 (#163) into the
ADR-0126 candidate-graph parser. Loader merge (re-exports from
numerics_loader.py give single import path), pack-aware unit
canonicalization (handles irregular plurals like feet/children
via lookup_unit), indefinite-quantifier refusal (ADR-0128.4 —
'some'/'many' emit no candidates, preserving wrong==0), and
widened initial-possession shapes:
- <Entity> has N <unit> [of <substance>] (ADR-0127 substance qualifier)
- There are N <unit> [in <place>] (implicit-subject shape)
Plus: pack-backed cardinal grounding in math_roundtrip._value_grounds
(widens word-number coverage from hard-coded 0-12 to full numerics
pack cardinal table + compound rule). Op-pattern trailing prep
alternation gains of/for/with for substance qualifiers.
REGRESSION: 1050/1050 tests green across math + ADR-0126 + ADR-0127
ratification + ADR-0128 ratification + runner.
EMPIRICAL RESULT (the Path-B trigger ADR-0126/0127/0128 named):
correct = 0/50 wrong = 0/50 refused = 50/50
on evals/gsm8k_math/train_sample/v1/cases.jsonl
Per ADR-0127's exit criterion (correct >= 10/50, wrong == 0):
**MISSED** — the full deterministic design (candidate-graph
topology + units pack + numerics pack + pack-aware parser) does
not move the GSM8K-math lane. This is the real Path-B trigger.
WHAT WORKS (synthetic verification, 6/6 cases solve end-to-end):
- 'Jan has 5 apples. Jan buys 3 apples. ...' -> 8
- 'Sam has 10 feet of rope. Sam uses 3 feet of rope. ...' -> 7
- 'There are 5 kids in camp. ...' -> 5
- 'Sam has 10 children. Sam loses 2 children. ...' -> 8
- (money + time-dimension variants pass)
WHY GSM8K STAYS AT ZERO: real GSM8K problems carry compound
linguistic structure (pronouns across statements, possessives,
subordinate clauses, multi-word entities, multi-step inference)
that no amount of pack vocabulary addresses. Per-sentence parse
rate improved measurably on simple shapes; joint problem-level
pass rate stayed at zero because every real problem contains at
least one sentence the parser still cannot handle.
Full results + Path-B recommendation in
docs/decisions/ADR-0127-0128-RESULTS.md. The substrate
(architecture + packs) stays load-bearing in main; the math
expert promotion path retargets to a benchmark where exact
recall and determinism are the discriminators (proposed
ADR-0131).
Exhaustive English linguistic-form ontology for quantities:
cardinals (0..20 + tens + magnitudes + compound rule), ordinals
(1st..31st + decade/magnitude forms), named fractions (1/2..1/10
+ sixteenth/thirty-second) + symbol forms (½ ¼ ¾ ⅓ ⅔ ⅛ ⅜ ⅝ ⅞),
multipliers (double/triple/twice/half), quantifiers with
semantic_type (indefinite triggers refusal at parse time —
preserves wrong==0), comparison anchors migrated for
ratifiability, number-format regexes with positive/negative
corpora.
Loader API in language_packs/numerics_loader.py (sibling module
to be merged into main loader after Gemini's ADR-0127 loader
lands, to avoid concurrent merge conflict).
Ratification invariants gated: cardinal/ordinal/fraction
exhaustiveness, quantifier semantic-type closed set, format-regex
test corpora (10+ positive/negative per format, ambiguity
refused), manifest checksums = SHA-256 of bytes-on-disk,
self-sealing mastery report.
Cross-references en_units_v1 (Gemini ADR-0127): fraction symbols
authoritative here; en_units_v1 symbol-affix table will point to
these entries.
No parser changes (deferred to 0128.3-0128.6). No train-sample
re-run (joint exit gate with ADR-0127 runs after both packs land).
Total: 130 lexicon entries across 7 kinds.
Lanes: smoke 67/0/0, packs 6/0/0, ADR-0128 suite 243/0/0.
Operator-supplied review of 'Beyond Traditional Pedagogy' triggered
a literature confirmation pass and a structural cross-walk against
CORE's teaching loop. Three artifacts:
1. ADR-0129 (DEFERRED) — Spaced reviewed-correction replay.
Maps onto retrieval-with-spacing literature (most robust finding
in cognitive psychology). Deterministic re-run of past
corrections at fixed cadence to verify they still produce
intended outcomes; failures emit operator-visible events
(no auto-correction). Deferred pending GSM8K-math Path-A/B
resolution + observed incident triggering un-deferral criteria.
2. ADR-0130 (DEFERRED) — Pre-articulation calibration logging.
Maps onto metacognitive prediction-outcome literature. Logs
CORE's pre-correction prediction; emits gap event on
correction acceptance. Provides empirical signal for 'is CORE
actually getting better' across pack-version cohorts. Deferred
pending same conditions as ADR-0129; the two compose if
un-deferred.
3. SESSION-2026-05-23 session note. Documents the review process:
literature confirmation pass (productive failure overstated,
retrieval transfer weaker than claimed, embodied cognition
replication crisis), missed frameworks (worked-example effect,
expertise reversal, CLT, deliberate practice, Bloom's 2-sigma),
structural cross-walk to CORE architecture (12 mappings), and
the rationale for ADRs 0129 + 0130 over alternative ports
(productive failure rejected as inverse of wrong==0; pre-testing
same; embodied learning N/A).
No code changes. Docs-only PR; lands independently of in-flight
ADR-0126 / 0127 / 0128 substrate chain.
Diagnostic from ADR-0126's first train-sample run (0/0/50): every
refusal happens at the first statement of each problem, and every
refused first statement fails on the unit-of-measurement construction,
not on the operation grammar. Adding more verb regexes is the per-axis
treadmill that produced 4 zero-lift ADRs. Units form a finite, externally
well-defined ontology (NIST SI tables, currency, English container nouns)
that is semantic substrate the candidate-graph parser was designed to
consume.
Scope:
- en_units_v1 pack: dimensions, units (<=60), containers, rate connectors
- conversions.jsonl: directed weighted graph of within-dimension unit pairs
- 3 new initial-possession shapes + rate-declaration extractor in the
candidate parser
- Round-trip filter gains optional pack-typed-unit check
- Solver gains dimensional canonicalization helper (shortest path through
conversion graph); fired edges join SolutionTrace.steps for replay
- Pack ratification invariants: round-trip identity, per-dimension
connectivity, path consistency, canonical unit per dimension
Wire the same train-sample exit criterion as ADR-0126 (correct >=10/50,
wrong==0). If passed -> sealed holdout. If still missed -> Path B
trigger is REAL (full deterministic design with units substrate failed),
demote GSM8K, re-target math expert promotion.
Also commits the empirical evidence: train_sample/v1/runner.py swapped
_score_one -> _score_one_candidate_graph; report.json baseline 0/0/50
confirming the candidate-graph topology refuses cleanly without units
substrate.
P3 — generate/math_candidate_graph.py:
Branch enumeration over per-sentence candidate choices (Cartesian
product, cap=64). Per-sentence ambiguity tiebreaker via most-grounded-
slots-wins (transfer beats subtract when 'to Tom' grounds). Decision
rule: 0 admissible -> refuse; 1 -> emit; >=2 same answer -> emit;
>=2 different answers -> refuse (preserves wrong==0 on genuine
ambiguity). End-to-end parse_and_solve(text) -> CandidateGraphResult.
Question extractor added to math_candidate_parser.py (CandidateUnknown,
total + entity question shapes mirroring math_parser).
22 new tests. Permissive verbs ('bought', 'ate', 'bakes') now produce
correct answers via the candidate-graph path; ambiguous 'gives to Tom'
resolves to transfer reading (Tom gets the apples) deterministically.
P4 — evals/gsm8k_math/runner.py:
New sibling function _score_one_candidate_graph(case) -> CaseOutcome.
Identical shape to _score_one; swaps parse_problem for parse_and_solve;
preserves verifier/realizer/expected-answer stages. Callers (e.g.
PR #160's train_sample/v1/runner.py) substitute the new function in
one line to evaluate the candidate-graph topology.
9 new wiring tests. Three groups:
- No regression: cases legacy solves, new also solves.
- Lift: cases legacy refuses, new solves (the architectural payoff).
- Wrong==0: out-of-grammar refuses, never wrong.
Regression: 714/714 existing math + runner tests still green.
ADR-0126 total: 74/74 tests green across P1+P2+P3+P4.
Sibling to math_parser.py — pure candidate-extraction functions that
emit list[CandidateOperation] per sentence without mutating any state.
State threading defers to P3 (per-branch graph assembly).
Topology change vs legacy:
- No first-match-wins; every verb-kind regex runs independently.
- Ambiguous verbs ('gives', 'returns') emit multiple candidates;
P1's round-trip filter + P3's decision rule resolve.
- Out-of-grammar sentences return [], NOT ParseError. Empty list
is the deterministic 'no candidate' signal.
Permissive verb tables (imported from math_roundtrip.KIND_TO_VERBS)
mean past-tense and production verbs ('bought', 'ate', 'bakes')
that the legacy parser refused are now admissible — the round-trip
filter is the safety mechanism, not regex narrowness.
P2 scope (canonical Subject-verb-Value-Unit-[to-Target] shape only):
- extract_initial_candidates(sentence) for 'X has N units'
- extract_operation_candidates(sentence) for add/subtract/transfer
Out of scope (deferred to later sub-phases):
- Pronoun resolution / unit inheritance (needs per-branch state)
- Multiply / divide / rate / comparison (same machinery, more matchers)
Regression: existing math suite 701/701 green. Zero changes to
math_parser.py, math_solver.py, math_verifier.py, math_realizer.py.
The wrong-answer firewall for the candidate-graph parser topology.
A CandidateOperation carries an Operation plus source-span provenance
for every content slot the parser claimed (verb, value, unit, actor,
transfer target, comparison reference). roundtrip_admissible() checks
each slot grounds in the source span AND the matched verb is
registered for the claimed kind.
Two consequences:
- A regex that mis-reads 'loses' as add fails (loses not in ADD_VERBS).
- A regex that hallucinates a number/unit not in source fails to ground.
KIND_TO_VERBS is the new single source of truth for {kind -> verbs};
P2 will refactor math_parser to consume it. Verb tables are
permissive by design (much wider than current narrow regex tables)
because the filter rejects wrong candidates downstream — narrowness
is no longer the safety mechanism.
Deterministic: pure byte/regex containment. No randomness, no
learning, no approximation. Preserves wrong==0, trace_hash byte-
equality, replay determinism.
Architectural pivot from per-axis grammar expansion (ADR-0122/0123/0123a/
0123b, all 0/1319 sealed lift) to candidate-graph topology with round-trip
admissibility filter. Converts compound-gap failure from multiplicative
p^k to parallel 1-(1-p)^k.
Adds new invariant: round-trip admissibility (op slots must reconstruct
to byte-equal source sentence under whitespace/case normalization).
Preserves wrong==0, trace_hash byte-equality, pack-binding, replay
determinism, no stochastic sampling.
Exit criterion: 50-case GSM8K train-split sample (unsealed) must show
correct >=10/50 with wrong==0 before any sealed-holdout run. If miss,
escalate to Path B (benchmark re-selection).
Supersedes ADR-0123b (never opened as PR).
Wraps existing math pipeline (parser -> solver -> verifier) against
PR #159's 50-case train sample. Emits deterministic report.json with
per-case verdicts. CLI exit code reflects exit criterion
(correct >= 10 AND wrong == 0).
Baseline against current parser: 0 correct / 0 wrong / 50 refused.
This baseline is the inner-loop gradient signal for ADR-0126's
candidate-graph parser (in flight on feat/adr-0126-candidate-graph).
Registers tests/test_adr_0126_train_sample_runner.py under
'core test --suite math' so the wrong == 0 invariant becomes a hard
CI gate per ADR-0114a Obligation #4 (refuse rather than confabulate).
Depends on PR #159 (gemini/adr-0126-train-sample). Rebase onto main
after #159 lands.
Closes 5 of 8 surface-form gaps Gemini identified in Task 5 on the
99 comparison-bearing sentences my ADR-0123 substrate currently refuses
in the sealed holdout. Pure regex / parser-state work — no graph,
solver, verifier, or pack changes; preserves wrong==0 discipline.
Expansions (in safety order)
- Group 8 (verb): comparison verbs widened from {has} to {has, have,
had, gets, get, got, takes, take, took, buys, buy, bought}.
"lost"/"won" excluded — they semantically invert direction.
- Group 3 (word-form numbers): _WORD_NUMBERS {one..twelve} accepted
wherever digit values are. _parse_compare_number helper centralizes
the dispatch.
- Group 4 (ellipsis / implicit unit): unit slot made optional in
multiplicative patterns (solver already infers unit from reference
state); added "twice|N times as much", "twice|N times the
number/amount of <unit>" variants.
- Group 1 (subjects / references): actor/reference slots widened from
bare proper noun to {proper noun, "the <noun>" collective,
pronoun}. Pronouns resolve via state.last_singular_subject; missing
prior subject raises ParseError (no silent emission with empty
actor). New _resolve_compare_entity helper canonicalizes "The boys"
/ "the boys" to the same entity string.
- Initial-possession + question patterns widened symmetrically so
"the X" subjects round-trip end-to-end:
- _INITIAL_HAS_RE accepts "the <noun>" subject + has/have +
digit-or-word value
- _Q_ENTITY_RE accepts "the <noun>" entity + do/does auxiliary
- _Q_TOTAL_RE now tried first (specificity-ordered) so "do they
have" doesn't get greedily matched as entity="they"
Deferred (per Gemini Task 5c recommendation)
- Group 2 (age): needs new "years_old" attribute model
- Group 5 (nested): needs compound y = mx + c solver operation
- Group 7 (currency): low volume (2 cases), defer
- Group 6 (compound multi-clause "and" split): scoped out of this
PR to keep the wrong==0 risk profile tight; safer to land after
Gemini Task 6 confirms current expansion doesn't introduce
misparses on the sealed set
Test coverage
- 507 existing math + ADR-0122 + ADR-0123 tests pass (no regressions)
- 16 ad-hoc smoke cases pass (3 baseline + 3 Group 8 + 3 Group 3 +
3 Group 4 + 3 Group 1 + 2 refusal guards + 1 rate cross-check)
- smoke suite 67/67, algebra suite 82/82 green
Expected sealed lift
- Gemini Task 5 catalog projected ~65/90 strict-comparison-only
cases unblocked by the 5 included groups (71/99 comparison-bearing
sentences). Empirical sealed measurement pending Gemini Task 6;
PR will be updated with the actual correct/wrong/refused bucket
counts once measured.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ADR-0123-parser-comparison-phrasing as the **surface increment** on
PR #155's substrate (commit c9bd5d4). Closes the last architectural
gap in the comparison-phrasing class: before this commit, the
substrate's solver evaluated comparison problems successfully but
realize() crashed with `unknown operation_kind 'compare_additive'`
when asked for show-your-work prose.
Substrate (PR #155) already shipped:
- `Comparison` typed graph operand
- `compare_additive` / `compare_multiplicative` operation kinds
- parser patterns for the four canonical surfaces
(N more / N fewer / twice / N times / half)
- solver + verifier wiring + pack lemmas
(en-arith-006 compare_additive, en-arith-007 compare_multiplicative)
This surface adds:
- `_compare_additive_sentence(step)` rendering `direction='more'|'fewer'`
- `_compare_multiplicative_sentence(step, entity_units)` rendering
`direction='times'|'fraction'`
- two new branches in `_step_sentence` dispatch
- `_step_sentence` signature widened with optional `entity_units` map
(derived once-per-trace in `realize()` from `graph_initial_state`)
- ADR-0123-parser-comparison-phrasing.md (~15 invariants, substrate
+ surface decomposition rationale, multi-construction barrier
inheritance)
- 26 invariants pinned across canonical surfaces, plurality
independence, byte-determinism, refusal discipline, and
backwards-compatibility with the pre-comparison realizer templates
End-to-end pipeline now operates on all four canonical comparison
shapes:
parse_problem(
"Alice has 5 apples. Bob has 3 more apples than Alice. "
"How many apples does Bob have?"
) -> solve() -> realize().as_prose() ->
"Alice has 5 apples. Bob has 3 more apples than Alice, giving Bob
a total of 8 apples. Bob has 8 apples."
Measurement (this PR):
- 26/28 direct ADR-0123 tests pass; 2 skipped (CORE_HOLDOUT_KEY)
- `core eval cognition` byte-identical: 100/100/100/100
- ADR-0118 stepped-realizer templates re-render byte-identically
- Substrate measurements continue to hold
Honest non-result: sealed `correct_rate` stays at 0/1319. The
realizer cannot create matches the parser refuses; the multi-
construction barrier the substrate ADR documented holds at the
surface layer too. Cumulative lift signal expected only after the
3rd/4th foundational class lands (per ADR-0121's revised
sequencing). `wrong == 0` holds by construction — realizer only
renders successful traces.
Pre-existing failure noted (not introduced by this PR):
`tests/test_adr_0085_gloss_aware_cause.py::test_flag_off_metrics_byte_identical`
fails on substrate base (c9bd5d4) without these changes — an
ADR-0085 cognition baseline drift unrelated to the realizer.
Second parser-expansion ADR after ADR-0122 rate/per-unit. Adds the
comparison algebra substrate (Comparison dataclass + compare_additive /
compare_multiplicative operation kinds + parser patterns + solver /
verifier / pack lemmas) mirroring the substrate-only / lift-deferred
pattern ADR-0122 established.
Substrate
- Comparison(reference_actor, delta: Quantity|None, factor: float|None,
direction: Literal[more,fewer,times,fraction]) frozen dataclass with
direction-discriminated delta/factor enforcement and self-reference
refusal at the Operation boundary
- compare_additive + compare_multiplicative operation kinds admitted in
VALID_OPERATION_KINDS; Operation.operand widened to Quantity|Comparison
with kind-discriminated type enforcement; entity-set validation extended
to cover Comparison.reference_actor
- Parser: _COMPARE_ADDITIVE_RE (more/fewer/less), _COMPARE_TWICE_RE,
_COMPARE_N_TIMES_RE, _COMPARE_HALF_RE happy-path patterns + 5
refusal patterns (ambiguous 'N times more', age comparisons,
combined-with-aggregation, nested additive+multiplicative); inserted
before _try_initial so leading 'has <N>' shape is not greedily
consumed as initial possession with unit='more'/'fewer'
- Solver: _apply_compare_additive (refuses on missing reference state,
overwrite, negative result); _apply_compare_multiplicative (refuses
on missing reference, ambiguous multi-unit reference, overwrite);
unit comes from delta.unit (additive) or reference's unique unit
(multiplicative)
- Verifier: _verify_compare_additive_step + _verify_compare_multiplicative_step
byte-equal replay; tamper-detects after_value, direction, factor
- Pack: en-arith-006 compare_additive + en-arith-007 compare_multiplicative
lemmas + glosses; SHA-256 checksums refreshed; manifest 1.0.0 -> 1.1.0;
provenance tagged adr-0123:comparison_extension:2026-05-23
Measurement (honest; from Gemini empirical sealed run on parallel surface
branch with this substrate)
- Sealed GSM8K correct_rate: 0/1319 (substrate matches zero real cases
alone). Validates the ADR-0122 multi-construction barrier prediction:
comparison constructions in GSM8K rarely appear alone — they bind with
rate (ADR-0124), percentage (ADR-0125), aggregation (ADR-0126), or
conditional ('if') clauses. First lift signal requires composition.
- Sealed GSM8K wrong: 0 (load-bearing positive claim; ADR-0114a
Obligation #4 preserved across all 1,319 sealed problems)
- Regression safety: 0 — all 913 non-comparison cases continue to
refuse exactly as before (refused_parser), no greedy consumption by
the new comparison patterns
Surface-form catalog (from Gemini Task 2 survey, see ADR doc) covers
6 primary forms across Groups A/B/C; Groups D (age), E (combined with
aggregation), F (nested additive+multiplicative) refused as out-of-scope
with typed ParseError naming the missing companion ADR.
Branch isolation
- Landed via dedicated worktree (feat/adr-0123-substrate from origin/main)
after a file-race on the shared umbrella branch. Companion surface +
scaffolding (realizer, ADR doc, tests, README) lands separately as
feat/adr-0123-surface; orchestrator merges both into the umbrella
feat/adr-0123-comparison-phrasing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Documents the Phase 5 GSM8K-math substrate completion across 7 narrative docs.
All 8 sub-phases of ADR-0119 (5.1 through 5.8) have landed on main; ADR-0114a's
10 anti-overfitting proof obligations are all discharged for the gsm8k_math lane.
Key facts surfaced in each doc:
- CORE-original public split: 150/150 correct, 0 wrong, 0 refused
- Real GSM8K test (sealed holdout): 0 correct, 0 wrong, 1319 refused
- Adversarial suite: 38 cases x 12 families, 0 wrong
- Depth curve: flat at 1.0 across depths 1-8 on public split
- Frontier baselines: Claude 3.5 Sonnet 96.4%, GPT-4 92.0%, Gemini 1.5 Pro 90.8%
- New lane shape gsm8k_capability_shape in LANE_SHAPE_REGISTRY
- New operational pack en_arithmetic_v1 (5 lemmas)
- ADR-0120 (first expert promotion contract) is the next gate
Docs updated: docs/PROGRESS.md, docs/capability_roadmap.md, docs/runtime_contracts.md,
docs/Whitepaper.md (§XIII), docs/Yellowpaper.md (gsm8k_capability_shape formal spec),
README.md, docs/decisions/README.md (current frontier).
No code changes. No new ADRs.
First worked attempt at promoting a domain under the ADR-0120
expert promotion contract. The contract refuses honestly.
Gate evaluation against live state:
ADR-0114a obligations: 10 of 10 pass
ADR-0120 contract-level gates:
audit_passed_holds ✓
correct_rate (public) ✓ 150/150 = 1.0
correct_rate (sealed) ✗ 0/1319 = 0.0 < 0.60 floor
signed_expert_claim ✗ (no entry, downstream of correct_rate)
Decision: mathematics_logic NOT promoted; stays at audit-passed.
Substantive blocker: parser grammar covers 0/1319 of real GSM8K.
What this proves
- The contract is genuinely falsifiable. ADR-0120 §"Threshold
rationale" deliberately set the floor above current measurement
so the first attempt would defer honestly. Same load-bearing
pattern as ADR-0107 → ADR-0110 for audit-passed.
- Wrong-zero discipline holds against real GSM8K (the load-
bearing positive claim). CORE refuses every problem outside
its grammar without confabulating on a single one.
What unlocks the promotion
Multi-ADR parser-expansion arc lifting sealed-GSM8K correct_rate
from 0.0 to ≥ 0.60. Each construction class (rate/comparison/
percentage/time-modal/etc.) ships as its own scoped ADR with:
- parser+solver+verifier+realizer extensions
- re-measurement on sealed holdout
- ADR-0118a OOD re-measurement (no surface-feature regression)
- ADR-0125 perturbation re-measurement (no invariance regression)
- ADR-0119.5 adversarial re-measurement (no new misparses)
Honest-fitting discipline: every lift is graded on the anti-
overfit obligations BEFORE the correct_rate change counts.
Tests: 6/6 with CORE_HOLDOUT_KEY; 4/6 + 2 skipped without (matches
ADR-0119.7 seal discipline).
This deferral demonstrates the expert tier's promotion machinery
is load-bearing — the gate has refused at least once before any
domain reaches it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Defines the `expert` ledger tier (sixth status above `audit-passed`)
and the composition gate that governs every future promotion.
The gate composes ALL TEN ADR-0114a obligations plus three contract-
level checks:
- audit_passed predicate must hold
- correct_rate >= 0.60 on public AND sealed holdout splits
- signed expert_claims entry whose digest reproduces byte-equal
Numeric thresholds (load-bearing choices documented with rationale):
- correct_rate floor: 0.60 ("advanced" — above weak open-source LLMs,
forces real architecture work, raisable later)
- depth-curve flatness ε: 0.05 (accuracy(N) ≥ accuracy(d1) · 0.95^(N-1))
- drift tolerance from promotion: ±0.02
Documents the post-ADR-0120 sequence in Open Candidate Directions:
Phase 1: ADR-0121 first worked math promotion (likely deferral)
Phase 2: parser-expansion arc to lift sealed-GSM8K correct_rate
Phase 3: math expert promotion succeeds
Phase 4: second domain = symbolic_logic (60-70% of math substrate cost;
ProofWriter / PrOntoQA benchmark; same machinery class)
Phase 5: third domain = high-stakes refusal-centric (medical or legal;
wedge-sharpener; needs two prior successes first)
Phase 7: open candidate — multi-reviewer threshold signing for expert
No code lands with this ADR. Implementation ships under ADR-0120a
to keep the contract change reviewable independently.
The first promotion attempt (ADR-0121) will likely defer on the
correct_rate gate — current measurement is 0/1319 on real GSM8K.
That deferral IS the contract working as designed; same load-
bearing pattern ADR-0107 → ADR-0110 demonstrated for audit-passed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The 1,319 GSM8K test cases are now sealed at
evals/gsm8k_math/holdouts/v1/cases.jsonl.age, age-encrypted to the
ADR-0119.1 recipient. Plaintext never touched disk in the working
tree; only ciphertext is committed.
First honest CORE-vs-real-GSM8K measurement
cases_total: 1319
correct: 0
wrong: 0 ← ADR-0114a Obligation #4 holds against external corpus
refused: 1319
overall_pass: True
Zero confabulation. Parser refuses what it can't grammar-handle; the
"wrong == 0" discipline survives the move from CORE-original cases
to a real public benchmark. The 0/1319 correct rate is the truthful
gap that ADR-0120's threshold work will quantify.
What landed
scripts/seal_gsm8k_test.py
- Loads GSM8K via datasets.load_dataset("openai/gsm8k", "main")
- Strips worked-solution prose; extracts final-answer integer/float
after "####" (handles "2,125" → 2125 thousands-separator)
- Reads recipient from docs/holdout_recipients.txt (single repo key
per ADR-0119.1)
- Encrypts via pyrage; writes only ciphertext
- Refuses to overwrite test path with train-derived seal
evals/gsm8k_math/runner.py
- Empty expected_unit (sentinel) skips unit-comparison; grades on
answer value alone. Required because GSM8K answers carry no unit
structurally. wrong-zero discipline preserved.
tests/test_adr_0119_7_sealed_gsm8k.py — 6 invariants:
1. sealed file present + age-formatted
2. no plaintext companion files (sibling-leak guard)
3. decrypted JSONL matches documented schema
4. runner against decrypted suite produces wrong==0
5. tests skip (not fail) when CORE_HOLDOUT_KEY unset
6. case ids match "gsm8k-test-NNNN" pattern
Defensive gitignore: plaintext patterns under
evals/gsm8k_math/holdouts/v1/ are explicitly excluded.
ADR-0114a obligation roll-up
10/10 discharged for the gsm8k_math lane:
#1 ✓ sealed-holdout (fab_control + GSM8K test)
#2..#10 ✓ as before
Phase 5 status: 5.1..5.7 done; 5.8 in flight (PR #149). After 5.8
merges, ADR-0120 (first expert promotion contract) becomes
feasible.
Test plan
- pytest tests/test_adr_0119_7_sealed_gsm8k.py with CORE_HOLDOUT_KEY → 6/6
- pytest without CORE_HOLDOUT_KEY → 3 pass + 3 skip
- core test --suite smoke -q → 67/67
- CLAIMS.md regenerated (no diff)
- HF token NEVER in repo (saved at ~/.cache/huggingface/token, mode 600)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Audit follow-ups from #145/#146 merge review. Five small fixes; no
behavior change on the green path, but failure modes are now explicit
rather than silent.
ADR-0119.6 depth_curve.py
- Add DepthCurveError typed exception
- Raise on case_id missing from lane_report (was: silent → "refused")
- Raise on depth >= 9 (was: silent new bucket key)
- Two new tests pin both refusals
- Removed stale sys.path hack at module top
ADR-0119.4 frontier-baseline tests
- Assert comparison_v1.json's core_measurement reports wrong == 0
(the load-bearing differentiator named in the disclaimer; a
tampered file with wrong > 0 was previously syntactically valid
and would have passed all old assertions)
- Assert frontier citations are dated 2023 or later (freshness
guard; older citations should be refreshed before ADR-0120
gates anything for `expert` promotion)
Tests
- tests/test_adr_0119_6_depth_curve.py: 7 → 9
- tests/test_adr_0119_4_frontier_baseline.py: 5 → 7
- 29/29 across runner + depth-curve + frontier suites; 67/67 smoke
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
