Implements the external auditor ADR-0114a Obligation #10 requires:
"Every SolutionTrace.steps[*].pack_lemma_id resolves to a real
lexicon entry in the domain's operator pack." The solver enforces
this at solve time; this PR audits it from outside.
New module core/capability/pack_provenance.py:
- _load_lexicon_lemmas(): independent re-read of pack lexicon
- _parse_lemma_id(): <pack_id>:<lemma> shape parser
- validate_lane(): re-runs candidate-graph pipeline on a B-lane's
cases, walks every solver step, validates pack_lemma_id parses
AND resolves to a lexicon entry. Per-case + per-lane verdict.
- emit_provenance_report(): deterministic artifact emission.
CLI: core capability pack-provenance (added to core/cli.py).
Writes evals/obligation_10_pack_provenance/<lane_id>.json.
Empirical verdict on current main (post-PR #186):
lane: B3_bounded_grammar
cases_total: 50
cases_validated: 25 (every expected-correct B3 case)
cases_skipped_unsolved: 25 (refusal-expected probes — by design)
cases_violated: 0
obligation_10_passed: True
5 distinct lemma_ids observed (add, subtract, transfer,
compare_additive, compare_multiplicative) — all resolve to
en_arithmetic_v1. The other 3 op kinds (multiply, divide,
apply_rate) ratify-at-solve-time via _resolve_pack_lemmas so the
obligation holds for them too if a future case exercises them.
Honest scope-limit: B3 only. B1 (symbolic equivalence) and B2
(teaching corpus) equivalents deferred to separate sub-ADRs —
B1 needs reframing (algebra normalization chain, not arithmetic
steps); B2 can use this same auditor signature once corpus
solver-trace exercise is confirmed case-by-case.
Composition with ADR-0131.4: orthogonal. Composite gate verdict
+ obligation #10 verdict + 4 other obligation auditors (when
they land) + reviewer signature → full ADR-0120 wire-up.
Trust boundary: read-only access to pack lexicon + B3 cases;
single deterministic write to artifact path. No dynamic imports,
no shell passthrough, no network. Pure deterministic auditor.
Tests: 19/19 in tests/test_adr_0114a_10_pack_provenance.py
covering lemma-id parser (well-formed + malformed), lexicon loader
(real pack + every failure mode), lane validator (passes on real
B3 + refuses on missing pack/cases + skips refusal-expected cases
without false violation), determinism (report identical across
calls + artifact byte-equal).
Cognitive capability: extend bounded grammar to admit acquisition/action
verbs (buys, bought, collected, saved, saved-up, makes, sells) as
operation-kind entries, and pure-possession verbs (had, started, started-with)
as initial-possession anchors.
What invariant proves correctness:
- wrong == 0 across all G1 curated cases (20/20) and GSM8K probe (0 wrong/50).
- versor_condition and field invariants untouched — no algebra-path changes.
- Round-trip filter (math_roundtrip.roundtrip_admissible) unchanged.
Which CLI suite / eval proves the lane:
pytest tests/test_adr_0131_G1_verb_classes.py — 15/15 pass
pytest tests/test_adr_0126_runner_wiring.py — 9/9 pass (3 regressions fixed)
pytest tests/test_adr_0131_{1,3}_*lane.py — 17/17 pass
pytest tests/test_adr_0131_G_gsm8k_coverage_probe.py — 8/8 pass
pytest tests/test_gsm8k_math_runner.py — 11/11 pass
Key architectural change:
Acquisition verbs that also appear in ADD_VERBS/SUBTRACT_VERBS were
previously listed in _INITIAL_HAS_RE, causing branch-disagreement refusals
when a canonical 'has' initial preceded an acquisition sentence for the
same entity. Fix: narrow _INITIAL_HAS_RE to pure-possession anchors only
(has/have/had/started); acquisition verbs remain exclusively in KIND_TO_VERBS.
The solver's default-from-zero means 'Sam buys 5 apples. How many does
Sam have?' resolves as 0+5=5 without any initial-possession candidate.
Optional verb particle (up/down/out/...) added to _op_pattern to handle
'saved up N', 'picked up N' etc.
No changes to binding graph, solver, verifier, or versor/CGA algebra.
No stochastic generation, approximate recall, or hidden normalization.
Trust boundaries unaffected — no new dynamic imports or user-input paths.
Implements ADR-0131's revision of the ADR-0120 expert-promotion
contract for mathematics_logic: replaces the single-benchmark
GSM8K-coverage check with a composite B1+B2+B3 requirement.
New module core/capability/composite_math_gate.py:
- evaluate_composite_math_gate(): pure function over already-
committed B-lane reports; handles heterogeneous report shapes
(B1/B2 counts vs B3 metrics); applies pinned thresholds
(correct_rate >= 0.95 AND wrong == 0); composes verdicts.
- Reproducible SHA-256 claim_digest over canonical evidence bundle.
- GSM8K honest-disclosure (admission/wrong/refused/substrate)
embedded in artifact but never gates per ADR-0131.
CLI: core capability math-expert-gate (added to core/cli.py).
Writes evals/math_expert_claims/v1/expert_claims_math_v1.json.
Empirical verdict on current main (post-PR #182/#183/#184/#185):
composite_gate_passed: True
B1_public: 185/185 wrong=0 rate=1.0000
B1_sealed: 14/14 wrong=0 rate=1.0000
B2_teaching_corpus: 40/40 wrong=0 rate=1.0000
B3_bounded_grammar: 50/50 wrong=0 rate=1.0000
GSM8K disclosure: 0/50 admission, wrong=0, substrate=candidate_graph
The math expert is gate-passing under ADR-0131's revised composite
contract. The architectural bet ADR-0131 placed has paid off.
Honest scope-limit: this implements only the ADR-0131-specific
revision (composite benchmark portion). The full ADR-0120 10-
obligation contract still requires substrate for 5 missing
obligations (OOD ratio, perturbation, depth curve, adversarial,
operation-provenance-via-pack). Those are sequencing-wise *after*
ADR-0131.4, not bundled. Reviewer signature via ADR-0092 registry
is also reserved.
Trust boundary: read-only access to 5 committed lane reports;
single deterministic write to the artifact path. No dynamic
imports, no recomputation of lane verdicts.
Tests: 12/12 in tests/test_adr_0131_4_composite_math_gate.py
covering threshold pinning, heterogeneous shape handling, gate
logic (passing + every failure mode), GSM8K honest disclosure
(never gates), determinism (claim_digest + artifact byte-equality),
and a snapshot test confirming current main satisfies the gate.
ADR-0131.4 module note: the parent ADR-0131 plan named
formation/ratify.py + formation/promote.py as the wire-up site —
that was a misidentification (those govern teaching-example
SPECULATIVE→COHERENT bridging per ADR-0021, not domain-tier
promotion). Correct site is core/capability/, where audit-passed
gate already lives.
Four axes deferred from ADR-0131.G.3 (PR #183):
1. Fractions end-to-end: new _INITIAL_FRACTION_OF_RE extractor handles
`N/M of [a/an] <unit>` shape; _resolve_value already handles N/M arithmetic.
2. Multi-currency: _MONEY_SYMBOL widened to six symbols; _CURRENCY_SYMBOLS table
+ _resolve_currency dispatcher; ¢/€/¥/₱ wired end-to-end. £/pound sterling
deferred to G.3.2 (question extractor's single-token unit slot cannot parse
two-word surface "pounds sterling").
3. Multi-token cardinals: dedicated _MULTI_WORD_CARDINAL_RE extractor (approach a)
delegates to parse_compound_cardinal; avoids greedy unit-slot boundary ambiguity
from widening _VALUE.
4. Word-num-adjective: optional adjective group added to _INITIAL_HAS_RE and
_MULTI_WORD_CARDINAL_RE; closed adjective list identical to _CONJ_OBJECT_RE.
Also fixes six pre-existing G4 type bugs where _resolve_value() result was used
directly as a numeric operand (TypeError: _ResolvedValue is not a number).
Axis lane v1_1: 20/20 solved_correct, 0 wrong, 8/8 refusals, overall_pass=True.
GSM8K probe: 0/50 admission_rate unchanged, admitted_wrong=0 (safety rail intact).
42/42 new tests pass; parent v1 lane (26/26) unaffected.
Highest-risk axis of the ADR-0131.G capability iteration: within-
sentence multi-clause composition. Four extractors land in the
candidate-emitting parser; no graph-side or solver changes.
Parser extension (generate/math_candidate_parser.py)
- _conj_subject_each_candidates: '<A> and [his/her/their <kin>] <B>
each <verb> <N> <unit>' → 2 CandidateInitial (one per actor).
- _conj_object_candidates: '<E> has <N1> <unit1> and <N2> <unit2>' →
2 CandidateInitial for the same entity; same-unit conjuncts refuse
(would silently collide under solver overwrite-on-collision).
- _embedded_quantifier_candidates: '<E> has <N> <container> with <M>
<unit> in each [<container>]' → 1 derived CandidateInitial
(value=N*M).
- _embedded_quantifier_candidates (conj branch): '... <N1> <C> with
<M1> <U> in each ... and <N2> <C> with <M2> <U> in each ...' → 1
SUM CandidateInitial (value=N1*M1+N2*M2); mixed-unit refuses.
- CandidateInitial anchor whitelist widened to include
saved/earned/got/received/bought/made/paid (and inflections) —
narrow widening needed for the conjoined-subject-each shape.
Closed-set discipline
- Distributive 'each' only — 'each ... together/altogether' refuses.
- Two-way conjunction only — 3-way refuses by non-match.
- Cross-sentence coreference stays refused (within-sentence axis).
- Ambiguous 'each' scope refuses (container2 must agree).
Curated axis lane (32 cases)
- evals/math_capability_axes/G4_multi_clause/v1/cases.jsonl:
conj_subject_each ×6, conj_object ×6, embedded_quantifier ×6,
conj_embedded ×6, refusal ×8.
- evals/math_capability_axes/G4_multi_clause/v1/runner.py +
report.json: deterministic; wrong==0 gate; byte-equal across runs.
Tests (26 new)
- tests/test_adr_0131_G4_multi_clause.py: per-shape emission,
refusal probes (parametric), distributive-only policy,
cross-sentence refusal, runner byte-equality, GSM8K-probe gate.
GSM8K-probe gate (chosen: multi-clause refusals ↓)
- evals/gsm8k_math/train_sample/v1/report.json (candidate-graph
probe): multi-clause statement-refusal count 2 → 1. Case 0042
('Ella has 4 bags with 20 apples in each bag and six bags with 25
apples in each bag.') moves from statement-clause refusal to
question-layer refusal. Case 0026 ('Aaron and his brother Carson
each saved up $40') stays refused on the '$' value slot
(deferred to G.3 numeric-literals axis).
- evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json
(legacy probe): refreshed, byte-identical (legacy parser
untouched).
B3 + candidate-graph + GSM8K probe lanes all pass (95/95
regression). wrong==0 preserved everywhere — load-bearing for the
highest-risk axis.
Zero behavior delta on the main baseline (both substrates produce
0/50 admission today) — but every subsequent ADR-0131.G.<n> iteration
now produces attributable admission deltas on the probe, instead of
silently extending a parser layer the probe wasn't measuring.
Background: ADR-0131.G's probe consulted run_lane → _score_one →
parse_problem (legacy first-match-wins parser, pre-ADR-0126). Every
G.<n> iteration extends the candidate-graph parser via
_score_one_candidate_graph → parse_and_solve. The mismatch was
discovered during G.3 development and explicitly reserved as this
follow-up.
Changes:
- run_coverage_probe.py: switch import to _score_one_candidate_graph;
new private _score_lane aggregator mirrors run_lane's output shape
via per-case scoring; report root adds "substrate": "candidate_graph"
for audit trail.
- train_sample_coverage_report.json: regenerated. All metrics
byte-identical to prior baseline (0/50 admission, wrong=0).
refused_reasons_top text differs (candidate_graph: prefix instead
of parser:) — expected and part of the substrate audit-trail shift.
Discipline: separate small PR per ADR-0131.G's "expansion that only
moves admission must be a standalone PR" principle. Substrate swap
attributable; future G.<n> deltas attributable.
Evidence:
- python3 -m evals.gsm8k_math.train_sample.v1.run_coverage_probe
→ admission 0/50, wrong=0, safety_rail_intact=True, exit 0
- pytest tests/test_adr_0131_G_gsm8k_coverage_probe.py
→ 8/8 pass in 0.18s (no test edits needed; tests pin invariants
not numbers)
- No changes to runner.py, no changes to any G.<n> work in flight.
Effect on in-flight iterations: each G.<n> PR (G.1 Gemini / G.2 #182 /
G.3 #183 / G.4 Opus#2) rebases after this lands and refreshes its
committed train_sample_coverage_report.json with the new substrate's
numbers. Rebase is mechanical.
First capability-axis iteration after ADR-0131.G baseline. Extends the
candidate-graph parser's <value> slot to recognize:
- Money symbol literals: $N and $N.NN (1-2 decimals); $N.NNN refused
- Money word forms: N dollars / N cents
- Hyphenated multi-word cardinals: twenty-five, ninety-nine, ...
All money values normalize to integer cents, unit 'cents' — pack-aligned
with en_units_v1's canonical_unit='cent' for the money dimension.
en_numerics_v1's parse_compound_cardinal handles hyphenated cardinals.
Parser changes (generate/):
- math_candidate_parser.py: _VALUE alternation widened; _resolve_value
refactored to return _ResolvedValue|None carrying optional unit
override; _INITIAL_HAS_RE unit slot made optional; dollar/dollars →
cents normalization at candidate build.
- math_roundtrip.py: new _unit_grounds helper (money-aware); _value_grounds
widened for the three new literal shapes; roundtrip_admissible uses
_unit_grounds for the unit check.
- math_candidate_graph.py: _initial_admissible and _question_admissible
use _unit_grounds.
New axis lane (evals/math_capability_axes/G3_numerics/v1/):
- 26 curated cases (20 positive across 4 classes + 6 refusal probes)
- runner.py wraps _score_one_candidate_graph; byte-equal report.json
- 20/20 positive solved correct; 6/6 refusal probes refused typed;
solved_wrong == 0; overall_pass == True
Tests: 27/27 in 0.19s. 420 existing candidate-parser/math-parser/pack
tests still green. GSM8K probe safety rail (admitted_wrong == 0)
preserved.
Honest scope-limit (documented in ADR): admission_rate on the GSM8K
probe stays at 0/50 because (a) the probe currently consults the legacy
parser path, not the candidate-graph pipeline G.3 extends, and (b) most
money-bearing GSM8K cases fail first on verb (G.1) or multi-clause (G.4)
shape, not on the money literal. The axis lane is the load-bearing
measurement for this iteration. Reserved follow-up: a small probe-
infra ADR to switch run_coverage_probe.py to the candidate-graph
pipeline.
Out of scope, deferred to G.3.1: fractions end-to-end (resolver supports
N/M but no axis cases), multi-currency (¢ € £ ¥ ₱), space-separated
multi-word cardinals (one hundred), word-number-adjective compositions
(five full boxes).
Wire compare_additive / compare_multiplicative extractors into the
candidate-emitting sentence parser, closing the deferred phase flagged
at generate/math_candidate_parser.py:30.
Capability axis: comparatives (additive + multiplicative)
- generate/math_candidate_parser.py: new _compare_additive_candidates,
_compare_multiplicative_candidates, _compare_nested_candidates
emitting CandidateOperation records keyed to the four
Comparison.direction literals registered in ADR-0123.
- Closed-set anchor alternation; 'less' admitted as surface synonym of
'fewer'; reference slot widened to admit "the number/amount of <unit>"
for nested forms.
- Nested 'A has N more <unit> than M times <REF>' emits two flat
candidates (additive + multiplicative); binding-graph picks the
admissible composition or refuses (no solver stub).
Curated axis lane (24 cases)
- evals/math_capability_axes/G2_comparatives/v1/cases.jsonl:
8 additive / 8 multiplicative / 3 nested / 5 refusal
- evals/math_capability_axes/G2_comparatives/v1/runner.py +
report.json: deterministic, wrong==0 gate, byte-equal across runs.
Tests (21 new)
- tests/test_adr_0131_G2_comparatives.py: per-direction at-least-one
passing, nested-both-emitted, closed-set refusal, runner
byte-equality, GSM8K-probe gate (comparative-clause refusals
strictly decrease).
GSM8K-probe gate (chosen: comparative-clause refusals ↓)
- evals/gsm8k_math/train_sample/v1/report.json (candidate-graph
probe): comparative-clause refusal count 2 → 1 (case 0009 'Jen has
10 more ducks than four times the number of chickens' moves from
statement-clause refusal to question-layer refusal). admitted_wrong
remains 0; admission_rate unchanged (downstream composition is a
follow-up ADR).
- evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json
(legacy probe): refreshed, byte-identical (legacy parser untouched).
B3 + candidate-graph + GSM8K probe lanes all pass (90/90). Direction
vocab stays closed to {more, fewer, times, fraction}; wrong==0
preserved everywhere.
ADR-0131 deferred GSM8K because it rewards paraphrase flexibility,
which is the deterministic engine's structural weakness. This ADR
re-engages it on architecture-aligned terms: as a *coverage probe*
of the bounded grammar + binding graph, not a promotion gate.
The framing pinned by this ADR:
GSM8K is not a target. The model's capability is the target.
GSM8K passing is the symptom of capability, not the goal of
the work.
Wrong mindset (rejected by ADR's iteration discipline):
"Find templates that admit more GSM8K cases."
Right mindset (load-bearing):
"Extend the model's NL-to-typed-graph capability along
principled axes (verb classes, comparative structures, numeric
forms, multi-clause grammar). GSM8K admission rises as a
side effect alongside every other word-problem corpus."
Baseline pinned by this commit:
admission_rate: 0/50 = 0.0%
admitted_wrong: 0 (gate intact, safety rail bulletproof)
refused: 50/50 = 100.0%
Every refusal is a typed parser error citing the specific clause
that did not match a template. Zero crashes, zero confabulations
— refusal-first works perfectly at admission rate zero.
What's in this PR:
- ``docs/decisions/ADR-0131.G-gsm8k-coverage-probe.md``: the ADR.
Cites parents (ADR-0131, -0115/-0116/-0117, -0131.3, -0132..-0135).
Documents the capability-first iteration discipline that every
subsequent ADR-0131.G.<n> must follow:
1. Name a single capability axis the iteration extends
2. Add B3-style curated coverage cases (capability proves
itself OUTSIDE GSM8K)
3. Re-run both B3 lane + GSM8K probe; B3 must not regress
4. Reject any expansion that only moves GSM8K admission
- ``evals/gsm8k_math/train_sample/v1/run_coverage_probe.py``:
pure-adapter wrapper around the existing run_lane. Emits a
deterministic train_sample_coverage_report.json with metrics,
per-case outcomes, and the top refused-reason families (the
work queue for capability extension).
- ``evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json``:
the baseline report. Diff-able artifact every future iteration
moves.
- ``tests/test_adr_0131_G_gsm8k_coverage_probe.py``: 8 contract
tests pinning the safety rail (admitted_wrong == 0), typed
refusal invariant (every refused case has non-empty reason),
closed outcome vocabulary, deterministic replay, committed-
report matches fresh-run.
The promotion-gate composite (B1 + B2 + B3) is unaffected.
ADR-0131.4 still consumes those three. The GSM8K probe is
empirical context for honest external claims, not a gate.
* feat(ADR-0131.1.F): frontier-baseline comparison harness for B1
Adapts the ADR-0119.4 methodology (frozen citations + comparison JSON
with disclaimer) to B1, with three additions for the
architecture-aligned claim:
1. A provider-agnostic live head-to-head runner. Adapters for
Anthropic / OpenAI / Google import their SDKs lazily so the
package loads cleanly without them installed. Each provider has a
documented FRONTIER_<VENDOR>_KEY env var; the runner refuses with
a typed FrontierRunError when keys are absent and the cache cannot
cover all cases. Every response is cached one-record-per-line at
responses/<provider>/<model>.jsonl so subsequent runs replay
byte-equally without re-calling the API.
2. A conservative free-text-to-closed-vocab verdict parser. Ambiguous
or sentinel-free provider replies collapse to "refused" — a
polarized verdict is never confabulated from prose. Chain-of-
thought replies use last-token-wins (provider deliberates, then
concludes). This is the load-bearing seam that prevents the
runner from manufacturing scores the provider didn't deliver.
3. Architecture-aligned comparison metrics. accuracy is reported but
foregrounded as the least-load-bearing; refusal_correctness
(CORE 100% by lane-gate construction vs. frontier confabulation
rate) and determinism (CORE byte-equal vs. frontier variance) are
the differentiators.
Frozen adjacent-benchmark citations cover Anthropic
(claude-3-5-sonnet on MATH, claude-opus-4-1 on AIME), OpenAI
(gpt-4o on MATH), and Google (gemini-1.5-pro on MATH). The scope
disclaimer documents that these are adjacent, not head-to-head.
Head-to-head numbers, when run, land in the cache; the comparison
JSON joins them with CORE's existing lane result.
22 tests pin the methodology: citation shape (every field, https
URL, YYYY-MM-DD date), provider-registry shape, verdict-parser
conservatism (multiple chain-of-thought cases), runner caching
behavior (no double-invoke), comparison-JSON determinism (byte-equal
across runs).
No live API call at test time. The harness gates real runs behind
explicit env vars + CLI invocation.
Composes with ADR-0131.1 (B1 v1), ADR-0131.1.B (v1.B hardening,
#169), ADR-0131.1.S (sealed holdout, #173).
* feat(ADR-0131.1.F): live head-to-head — anthropic/claude-sonnet-4-6
First real frontier baseline on the full B1.B 185-case set
(curated + generated). Cached one-record-per-line at
responses/anthropic/claude-sonnet-4-6.jsonl. Re-runs replay from
disk; no further API calls.
Headline (after scoring fix):
CORE 185/185 = 100.0% accuracy
3/3 = 100.0% refusal_correctness
deterministic (byte-equal across runs)
anthropic/claude-sonnet-4-6 182/185 = 98.4% accuracy
1/3 = 33.3% refusal_correctness
non-deterministic (temperature=0, but
not byte-equal architecturally)
The 1.6pp accuracy gap is informative; the refusal-correctness gap
is the architecture-aligned story. Sonnet's three misses:
sym-eq-v1-0016 [difference_of_squares]
(x^2 + 1)*(x^2 - 1) vs x^4 - 1
Sonnet: NOT_EQUIVALENT (math error on a textbook identity)
sym-eq-gen-v1-0153 [generated_refusal_function]
sin(x) vs x
Sonnet: NOT_EQUIVALENT (confabulated — should refuse,
transcendental outside polynomial scope)
sym-eq-gen-v1-0154 [generated_refusal_negative_exponent]
x^-1 vs 1
Sonnet: NOT_EQUIVALENT (confabulated — should refuse,
negative exponent outside scope)
Sonnet correctly refused only on syntactically malformed input
("x +"); on syntactically-valid-but-semantically-out-of-scope inputs
it confidently polarized rather than refusing. CORE refuses both
classes with typed reasons.
Scoring fix: comparison.py now composes curated + generated cases
(mirroring runner.py) so the head-to-head scores the full 185-case
lane, not just the 30 curated. The initial run scored only 30/185
because the generated set was not loaded into _load_cases().
22/22 frontier-methodology tests still pass.
* feat(ADR-0131.1.F): three more head-to-head runs + Ollama adapter
Three additional providers ran against the full B1.B 185-case set,
joining the prior claude-sonnet-4-6 result:
CORE 185/185 = 100.0% acc | 3/3 = 100% refusal | 33 ms
claude-sonnet-4-6 182/185 = 98.4% acc | 1/3 = 33.3% refusal | 294 s
claude-opus-4-7 178/185 = 96.2% acc | 1/3 = 33.3% refusal | 309 s
gpt-5 134/185 = 72.4% acc | 1/3 = 33.3% refusal | 1153 s
qwen3:8b (M1 local, partial) 91/91 = 100.0% acc | n/a no refusal-class | killed
CORE is the only system at 100% on both axes, and runs ~9,000×
faster than the cheapest cloud frontier, ~35,000× faster than gpt-5,
and finishes in less wall time than a single API call to any of the
three frontier models.
Three distinct frontier brittleness modes, all rooted in
"not actually canonicalizing":
- sonnet-4-6 confabulates polarized verdicts on out-of-scope
inputs (sin(x), x^-1). Misses one in-scope difference-of-squares
identity (x^2+1)*(x^2-1) vs x^4-1.
- opus-4-7 pattern-shortcuts five near-miss-constant cases —
accepts (-x+3)*(4x+1) == -4x^2+11x+4 (correct constant is 3,
not 4) without expanding. Same two out-of-scope confabulations
as sonnet.
- gpt-5 over-refuses 50 in-scope cases — literally replies
"REFUSED" to x*(x+1) == x^2+x and (x+1)*(x-1) == x^2-1. Same
two out-of-scope confabulations as sonnet/opus.
The qwen3:8b partial is the surprise: on the 91 in-scope cases it
completed (spanning the categories where the frontier models failed),
it scored 100%. Refusal-class cases weren't reached before the run
was killed for being impractically slow (~22s/case on M1).
Changes in this commit:
- frontier_runner.py: anthropic adapter now omits ``temperature``
for claude-opus-4-x (the parameter is rejected by 4.x models);
openai adapter switches to ``max_completion_tokens`` for the
gpt-5 / o-series reasoning models; new ``_ollama_invoke`` that
posts to localhost:11434 with no third-party dep; per-case
``latency_ms`` is now captured on every NEW cached response
(future runs only — these four runs pre-date the patch).
- comparison.py: ``_load_cases`` composes curated + generated
(185 cases) instead of curated only; ``_score_provider``
surfaces ``latency_summary`` when records carry latency_ms.
- tests: provider-registry test relaxed to "cloud trio is a
subset of PROVIDERS"; env-key test allows ``_KEY`` (cloud
secret) or ``_URL`` (local endpoint).
* feat(evals): add deterministic symbolic equivalence generated corpus
* feat(evals): add symbolic equivalence replay helpers
* feat(evals): load generated symbolic equivalence corpus
* feat(evals): emit symbolic equivalence replay manifest
* feat(symbolic): support multivariable integer polynomials
* feat(symbolic): support exact rational polynomial coefficients
* feat(symbolic): align equivalence API with multivariable normalization
* test(ADR-0131.1.B): reconcile v1 expectations to v1.B scope expansion
The v1.B refactor (univariate int → sparse multivariable Fraction) deliberately
admits multivariable polynomials and constant-denominator division. The v1
dataset and tests pinned the old refusal behavior, so the lane runner reported
wrong=4 and 10 unit tests failed.
Reconcile:
- cases.jsonl: flip sym-eq-v1-0029 ('x+y' vs 'x+1') and sym-eq-v1-0030
('x/2' vs 'x') from expected=refused to expected=not_equivalent; rename
categories to multivariable_distinct / constant_denominator_distinct;
extend provenance with adr-0131.1b:scope-expanded.
- generated_cases.py: split _refusal_cases into scope_expanded (admits)
and templates (still refused); the first two adversarial cases move to
the scope-expanded list with expected=not_equivalent.
- test_math_symbolic_normalizer.py: replace test_undefined_variable and
test_unknown_operator_division with positive scope-expansion tests +
symbolic-denominator refusal; rewrite TestPolynomialInvariants for the
new terms/variables constructor (Polynomial(terms={...}, variables=(...)))
with float-rejection and zero-coef-collapse invariants.
- test_math_symbolic_equivalence.py: TestRefused.test_empty_left reason
string matches new normalizer error; flip multivariable + constant-
denominator cases to NOT_EQUIVALENT; add symbolic-denominator-refused
case; relax canonical_a assertion in test_a_normalizes_b_refuses (engine
now zeroes both on either-side refusal).
- report.json + manifest.json: regenerated; lane PASS 185/185 wrong=0.
Lane invariants reaffirmed by the new tests: wrong==0, refusal-first for
truly out-of-scope inputs (symbolic denominator, transcendental, malformed,
negative exponent), determinism via byte-equal report.
ADR-0131 Benchmark 1 substrate — the primary discriminator for the
mathematics_logic expert promotion under the architecture-aligned
benchmark composite proposed in ADR-0131.
WHAT LANDED:
generate/math_symbolic_normalizer.py
Deterministic univariate polynomial normalizer. Scope: single
variable, integer coefficients, +/-/*/** operators, parens, no
division, no transcendentals. Pipeline: tokenize -> recursive-
descent parse -> expand-and-collect -> canonical string. Refusal
is first-class via SymbolicError; out-of-scope inputs refuse
rather than guess (preserves wrong == 0).
generate/math_symbolic_equivalence.py
check_equivalence(a, b) -> EquivalenceVerdict
Returns EQUIVALENT / NOT_EQUIVALENT / REFUSED with canonical
strings + reason. Compares byte-equal canonical forms.
evals/math_symbolic_equivalence/v1/
cases.jsonl — 30 hand-curated cases across 18 algebraic
identity categories + 2 out-of-scope refusals.
Coverage: commutative, distributive, square +
cube of binomial, difference of squares, FOIL,
collect like terms, zero cancellation, factoring,
exponent combination, unary negation.
runner.py — CLI entry point. Loads cases, builds report,
writes JSON, exits 0/1 on gate pass/fail.
README.md — methodology, scope, dataset categorization,
exit criterion, baseline result.
tests/
test_math_symbolic_normalizer.py — 44 tests covering parser,
algebra primitives,
canonical-form invariants,
and every refusal path.
test_math_symbolic_equivalence.py — 16 tests on the public
check_equivalence API.
test_adr_0131_1_symbolic_equivalence_lane.py
— 8 tests gating the lane:
dataset integrity, exit
criterion, wrong == 0,
determinism (byte-equal
report across runs).
EMPIRICAL RESULT (the lane PASSED):
correct = 30 / 30 (100.0%)
wrong = 0 / 30 (wrong == 0 invariant satisfied)
refused = 0 / 30 (refusals all matched expected)
correct_rate = 1.00
exit_criterion: PASSED (>= 0.95 required)
CONTRAST WITH ADR-0127-0128 GSM8K TRAIN-SAMPLE RESULT (0/0/50):
This is the first benchmark on the mathematics_logic lane where
the architecture's structural strengths fully express. The result
is the empirical inverse of the GSM8K result — and that's
exactly the architecture-benchmark fit ADR-0131 was written to
re-target toward.
REGRESSION: 1033/1033 existing tests green across math + ADR-0126
+ pack ratification + runner. Zero regressions.
SCOPE DISCIPLINE (per ADR-0131.1 v1 plan):
v1 deliberately narrow (univariate, integer, polynomial). Future
ADR-0131.1.B expansions documented in README: multi-variable,
rationals, larger dataset (~500), sealed holdout per ADR-0119.7
pattern.
PARALLEL WORK (per ADR-0131 plan to run all 3 sub-phases concurrently):
- ADR-0131.2: CORE-native teaching-corpus eval (separate PR)
- ADR-0131.3: bounded-grammar word-problem set (separate PR)
These are independent of ADR-0131.1; no shared files, no
cross-PR coordination required beyond final composite gate.
Integrates en_units_v1 (#164) + en_numerics_v1 (#163) into the
ADR-0126 candidate-graph parser. Loader merge (re-exports from
numerics_loader.py give single import path), pack-aware unit
canonicalization (handles irregular plurals like feet/children
via lookup_unit), indefinite-quantifier refusal (ADR-0128.4 —
'some'/'many' emit no candidates, preserving wrong==0), and
widened initial-possession shapes:
- <Entity> has N <unit> [of <substance>] (ADR-0127 substance qualifier)
- There are N <unit> [in <place>] (implicit-subject shape)
Plus: pack-backed cardinal grounding in math_roundtrip._value_grounds
(widens word-number coverage from hard-coded 0-12 to full numerics
pack cardinal table + compound rule). Op-pattern trailing prep
alternation gains of/for/with for substance qualifiers.
REGRESSION: 1050/1050 tests green across math + ADR-0126 + ADR-0127
ratification + ADR-0128 ratification + runner.
EMPIRICAL RESULT (the Path-B trigger ADR-0126/0127/0128 named):
correct = 0/50 wrong = 0/50 refused = 50/50
on evals/gsm8k_math/train_sample/v1/cases.jsonl
Per ADR-0127's exit criterion (correct >= 10/50, wrong == 0):
**MISSED** — the full deterministic design (candidate-graph
topology + units pack + numerics pack + pack-aware parser) does
not move the GSM8K-math lane. This is the real Path-B trigger.
WHAT WORKS (synthetic verification, 6/6 cases solve end-to-end):
- 'Jan has 5 apples. Jan buys 3 apples. ...' -> 8
- 'Sam has 10 feet of rope. Sam uses 3 feet of rope. ...' -> 7
- 'There are 5 kids in camp. ...' -> 5
- 'Sam has 10 children. Sam loses 2 children. ...' -> 8
- (money + time-dimension variants pass)
WHY GSM8K STAYS AT ZERO: real GSM8K problems carry compound
linguistic structure (pronouns across statements, possessives,
subordinate clauses, multi-word entities, multi-step inference)
that no amount of pack vocabulary addresses. Per-sentence parse
rate improved measurably on simple shapes; joint problem-level
pass rate stayed at zero because every real problem contains at
least one sentence the parser still cannot handle.
Full results + Path-B recommendation in
docs/decisions/ADR-0127-0128-RESULTS.md. The substrate
(architecture + packs) stays load-bearing in main; the math
expert promotion path retargets to a benchmark where exact
recall and determinism are the discriminators (proposed
ADR-0131).
Diagnostic from ADR-0126's first train-sample run (0/0/50): every
refusal happens at the first statement of each problem, and every
refused first statement fails on the unit-of-measurement construction,
not on the operation grammar. Adding more verb regexes is the per-axis
treadmill that produced 4 zero-lift ADRs. Units form a finite, externally
well-defined ontology (NIST SI tables, currency, English container nouns)
that is semantic substrate the candidate-graph parser was designed to
consume.
Scope:
- en_units_v1 pack: dimensions, units (<=60), containers, rate connectors
- conversions.jsonl: directed weighted graph of within-dimension unit pairs
- 3 new initial-possession shapes + rate-declaration extractor in the
candidate parser
- Round-trip filter gains optional pack-typed-unit check
- Solver gains dimensional canonicalization helper (shortest path through
conversion graph); fired edges join SolutionTrace.steps for replay
- Pack ratification invariants: round-trip identity, per-dimension
connectivity, path consistency, canonical unit per dimension
Wire the same train-sample exit criterion as ADR-0126 (correct >=10/50,
wrong==0). If passed -> sealed holdout. If still missed -> Path B
trigger is REAL (full deterministic design with units substrate failed),
demote GSM8K, re-target math expert promotion.
Also commits the empirical evidence: train_sample/v1/runner.py swapped
_score_one -> _score_one_candidate_graph; report.json baseline 0/0/50
confirming the candidate-graph topology refuses cleanly without units
substrate.
P3 — generate/math_candidate_graph.py:
Branch enumeration over per-sentence candidate choices (Cartesian
product, cap=64). Per-sentence ambiguity tiebreaker via most-grounded-
slots-wins (transfer beats subtract when 'to Tom' grounds). Decision
rule: 0 admissible -> refuse; 1 -> emit; >=2 same answer -> emit;
>=2 different answers -> refuse (preserves wrong==0 on genuine
ambiguity). End-to-end parse_and_solve(text) -> CandidateGraphResult.
Question extractor added to math_candidate_parser.py (CandidateUnknown,
total + entity question shapes mirroring math_parser).
22 new tests. Permissive verbs ('bought', 'ate', 'bakes') now produce
correct answers via the candidate-graph path; ambiguous 'gives to Tom'
resolves to transfer reading (Tom gets the apples) deterministically.
P4 — evals/gsm8k_math/runner.py:
New sibling function _score_one_candidate_graph(case) -> CaseOutcome.
Identical shape to _score_one; swaps parse_problem for parse_and_solve;
preserves verifier/realizer/expected-answer stages. Callers (e.g.
PR #160's train_sample/v1/runner.py) substitute the new function in
one line to evaluate the candidate-graph topology.
9 new wiring tests. Three groups:
- No regression: cases legacy solves, new also solves.
- Lift: cases legacy refuses, new solves (the architectural payoff).
- Wrong==0: out-of-grammar refuses, never wrong.
Regression: 714/714 existing math + runner tests still green.
ADR-0126 total: 74/74 tests green across P1+P2+P3+P4.
Wraps existing math pipeline (parser -> solver -> verifier) against
PR #159's 50-case train sample. Emits deterministic report.json with
per-case verdicts. CLI exit code reflects exit criterion
(correct >= 10 AND wrong == 0).
Baseline against current parser: 0 correct / 0 wrong / 50 refused.
This baseline is the inner-loop gradient signal for ADR-0126's
candidate-graph parser (in flight on feat/adr-0126-candidate-graph).
Registers tests/test_adr_0126_train_sample_runner.py under
'core test --suite math' so the wrong == 0 invariant becomes a hard
CI gate per ADR-0114a Obligation #4 (refuse rather than confabulate).
Depends on PR #159 (gemini/adr-0126-train-sample). Rebase onto main
after #159 lands.
The 1,319 GSM8K test cases are now sealed at
evals/gsm8k_math/holdouts/v1/cases.jsonl.age, age-encrypted to the
ADR-0119.1 recipient. Plaintext never touched disk in the working
tree; only ciphertext is committed.
First honest CORE-vs-real-GSM8K measurement
cases_total: 1319
correct: 0
wrong: 0 ← ADR-0114a Obligation #4 holds against external corpus
refused: 1319
overall_pass: True
Zero confabulation. Parser refuses what it can't grammar-handle; the
"wrong == 0" discipline survives the move from CORE-original cases
to a real public benchmark. The 0/1319 correct rate is the truthful
gap that ADR-0120's threshold work will quantify.
What landed
scripts/seal_gsm8k_test.py
- Loads GSM8K via datasets.load_dataset("openai/gsm8k", "main")
- Strips worked-solution prose; extracts final-answer integer/float
after "####" (handles "2,125" → 2125 thousands-separator)
- Reads recipient from docs/holdout_recipients.txt (single repo key
per ADR-0119.1)
- Encrypts via pyrage; writes only ciphertext
- Refuses to overwrite test path with train-derived seal
evals/gsm8k_math/runner.py
- Empty expected_unit (sentinel) skips unit-comparison; grades on
answer value alone. Required because GSM8K answers carry no unit
structurally. wrong-zero discipline preserved.
tests/test_adr_0119_7_sealed_gsm8k.py — 6 invariants:
1. sealed file present + age-formatted
2. no plaintext companion files (sibling-leak guard)
3. decrypted JSONL matches documented schema
4. runner against decrypted suite produces wrong==0
5. tests skip (not fail) when CORE_HOLDOUT_KEY unset
6. case ids match "gsm8k-test-NNNN" pattern
Defensive gitignore: plaintext patterns under
evals/gsm8k_math/holdouts/v1/ are explicitly excluded.
ADR-0114a obligation roll-up
10/10 discharged for the gsm8k_math lane:
#1 ✓ sealed-holdout (fab_control + GSM8K test)
#2..#10 ✓ as before
Phase 5 status: 5.1..5.7 done; 5.8 in flight (PR #149). After 5.8
merges, ADR-0120 (first expert promotion contract) becomes
feasible.
Test plan
- pytest tests/test_adr_0119_7_sealed_gsm8k.py with CORE_HOLDOUT_KEY → 6/6
- pytest without CORE_HOLDOUT_KEY → 3 pass + 3 skip
- core test --suite smoke -q → 67/67
- CLAIMS.md regenerated (no diff)
- HF token NEVER in repo (saved at ~/.cache/huggingface/token, mode 600)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Audit follow-ups from #145/#146 merge review. Five small fixes; no
behavior change on the green path, but failure modes are now explicit
rather than silent.
ADR-0119.6 depth_curve.py
- Add DepthCurveError typed exception
- Raise on case_id missing from lane_report (was: silent → "refused")
- Raise on depth >= 9 (was: silent new bucket key)
- Two new tests pin both refusals
- Removed stale sys.path hack at module top
ADR-0119.4 frontier-baseline tests
- Assert comparison_v1.json's core_measurement reports wrong == 0
(the load-bearing differentiator named in the disclaimer; a
tampered file with wrong > 0 was previously syntactically valid
and would have passed all old assertions)
- Assert frontier citations are dated 2023 or later (freshness
guard; older citations should be refreshed before ADR-0120
gates anything for `expert` promotion)
Tests
- tests/test_adr_0119_6_depth_curve.py: 7 → 9
- tests/test_adr_0119_4_frontier_baseline.py: 5 → 7
- 29/29 across runner + depth-curve + frontier suites; 67/67 smoke
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First Phase of ADR-0114's expert-capability roadmap. Decomposed into four
sub-phases so each lands as its own auditable step:
1.1 schema + 5 seed cases + invariants ← this commit
1.2 45 more dev-set cases ← delegated (Codex)
1.3 the parser itself ← exit: ≥0.90 on dev set
1.4 runtime binding ← if non-trivial
What landed
- generate/math_problem_graph.py — typed dataclasses (Quantity,
InitialPossession, Operation, Unknown, MathProblemGraph) + frozen
validation + canonical_bytes() byte-deterministic serialization +
graph_from_dict roundtrip.
- evals/gsm8k_parser_dev/cases.jsonl — 5 seed cases (gpd-001..005)
covering single-add, single-subtract, multi-step, two-entity
transfer, and multi-entity sum constructions. Every case carries a
ground_truth_graph and the documented patterns it exercises.
- evals/gsm8k_parser_dev/README.md — authoring contract: schema,
pattern registry, canonicalization rules, Phase 1.1 scope boundary,
hand-solving rubric, distribution target for the remaining 45
cases. This is the spec Phase 1.2 authors work against.
- tests/test_math_problem_graph.py — 26 cases pinning four invariants:
round-trip byte equality, canonical_bytes() determinism, schema
rejection of malformed graphs, and ground_truth_graph ↔
expected_answer agreement (a hand-solver inside the test module
falsifies mis-authored cases).
Why this is sticky
The Phase 1.1 schema is load-bearing for Phase 1.2 (the 45 authored
cases will be written against it) AND Phase 1.3 (the parser will be
graded byte-equal against ground-truth graphs in this schema). Changing
the schema after Phase 1.2 lands requires an amendment ADR + rewriting
authored cases. The schema choices here are intentionally conservative.
Tests: 26/26 new; 67/67 smoke green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Second worked promotion exercising the ADR-0106 + ADR-0109 contract
on a domain distinct from mathematics_logic. No contract change.
Evidence:
- foundational_physics_ood: accuracy=1.0 (117/117 public, 39/39 holdout)
- inference_closure: all_pass_rate=1.0 (shared with math, distinct digest via domain_id)
- fabrication_control: refused=n, fabricated=0 across all classes (shared)
Signed claim digest: a104cad136f3219df05dc7ce6a78437c02f7b5827cd3cdce568db3acda6a43ed
Bridge landed: cases_plaintext.jsonl dev-mode fallback for
foundational_physics_ood (matches ADR-0105 convention; analogous to the
math/inference bridges in ADR-0110). One small file, not a contract change.
Tests:
- tests/test_adr_0111_physics_expert_demo.py — 4 invariants, 6 cases
- tests/test_adr_0110_math_expert_demo.py — relaxed "only math promoted"
to "math stays promoted" (load-bearing for ADR-0110 is persistence)
- tests/test_capability_reports.py — physics row now expert-demo
Retires the "first promotion was math-specific" objection: the bridges
ADR-0110 landed were correctly scoped, and the contract holds across
two distinct domains using shared lane infrastructure with distinct
digests.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Between 2026-05-17 and 2026-05-22 the inference_closure lane regressed
from all_pass_rate=1.0 to 0.4 on public. Root cause: the
_DECLARATIVE_RELATION_RE branch in generate/intent.py runs ahead of the
_RULES loop and swallowed sentences beginning with 'Actually' into the
subject phrase, routing them to VERIFICATION. The lane's premise emit
path is gated on CORRECTION intent, so PackMutationProposal records
stopped being emitted for any non-'is' relation (precedes / grounds /
causes / reveals). Only the four transitive_is cases passed because
'is' is not in the declarative-relation verb list.
Fix: _CORRECTION_CUE_PREFIX_RE guard. When the text begins with a
correction cue ('Actually', 'Incorrect, ', 'No, ', 'Correction'), the
declarative-match branch is skipped and the sentence falls through to
the _RULES CORRECTION rule. Plain declarative-relation assertions still
route to VERIFICATION unchanged.
Lane on 2026-05-22 post-fix:
dev/v1: all_pass_rate=1.0, overall_pass=True (5 cases)
public/v1: all_pass_rate=1.0, overall_pass=True (20 cases)
- tests/test_correction_cue_prefix_routing.py pins both halves of the
guard (10 new tests).
- evals/inference_closure/gaps.md documents the regression + fix in a
new section, preserving the 2026-05-17 resolution narrative.
- evals/inference_closure/results/ now carries canonical v1_dev and
v1_public reports (the lane had no checked-in results before; ADR-0110
will reference these).
This unblocks the second of ADR-0107's two named blockers. ADR-0110
(math expert-demo re-attempt) now becomes feasible once the math
domain's three lanes have signed-and-digested evidence.
