Cognitive capability: extend bounded grammar to admit acquisition/action
verbs (buys, bought, collected, saved, saved-up, makes, sells) as
operation-kind entries, and pure-possession verbs (had, started, started-with)
as initial-possession anchors.
What invariant proves correctness:
- wrong == 0 across all G1 curated cases (20/20) and GSM8K probe (0 wrong/50).
- versor_condition and field invariants untouched — no algebra-path changes.
- Round-trip filter (math_roundtrip.roundtrip_admissible) unchanged.
Which CLI suite / eval proves the lane:
pytest tests/test_adr_0131_G1_verb_classes.py — 15/15 pass
pytest tests/test_adr_0126_runner_wiring.py — 9/9 pass (3 regressions fixed)
pytest tests/test_adr_0131_{1,3}_*lane.py — 17/17 pass
pytest tests/test_adr_0131_G_gsm8k_coverage_probe.py — 8/8 pass
pytest tests/test_gsm8k_math_runner.py — 11/11 pass
Key architectural change:
Acquisition verbs that also appear in ADD_VERBS/SUBTRACT_VERBS were
previously listed in _INITIAL_HAS_RE, causing branch-disagreement refusals
when a canonical 'has' initial preceded an acquisition sentence for the
same entity. Fix: narrow _INITIAL_HAS_RE to pure-possession anchors only
(has/have/had/started); acquisition verbs remain exclusively in KIND_TO_VERBS.
The solver's default-from-zero means 'Sam buys 5 apples. How many does
Sam have?' resolves as 0+5=5 without any initial-possession candidate.
Optional verb particle (up/down/out/...) added to _op_pattern to handle
'saved up N', 'picked up N' etc.
No changes to binding graph, solver, verifier, or versor/CGA algebra.
No stochastic generation, approximate recall, or hidden normalization.
Trust boundaries unaffected — no new dynamic imports or user-input paths.
Implements ADR-0131's revision of the ADR-0120 expert-promotion
contract for mathematics_logic: replaces the single-benchmark
GSM8K-coverage check with a composite B1+B2+B3 requirement.
New module core/capability/composite_math_gate.py:
- evaluate_composite_math_gate(): pure function over already-
committed B-lane reports; handles heterogeneous report shapes
(B1/B2 counts vs B3 metrics); applies pinned thresholds
(correct_rate >= 0.95 AND wrong == 0); composes verdicts.
- Reproducible SHA-256 claim_digest over canonical evidence bundle.
- GSM8K honest-disclosure (admission/wrong/refused/substrate)
embedded in artifact but never gates per ADR-0131.
CLI: core capability math-expert-gate (added to core/cli.py).
Writes evals/math_expert_claims/v1/expert_claims_math_v1.json.
Empirical verdict on current main (post-PR #182/#183/#184/#185):
composite_gate_passed: True
B1_public: 185/185 wrong=0 rate=1.0000
B1_sealed: 14/14 wrong=0 rate=1.0000
B2_teaching_corpus: 40/40 wrong=0 rate=1.0000
B3_bounded_grammar: 50/50 wrong=0 rate=1.0000
GSM8K disclosure: 0/50 admission, wrong=0, substrate=candidate_graph
The math expert is gate-passing under ADR-0131's revised composite
contract. The architectural bet ADR-0131 placed has paid off.
Honest scope-limit: this implements only the ADR-0131-specific
revision (composite benchmark portion). The full ADR-0120 10-
obligation contract still requires substrate for 5 missing
obligations (OOD ratio, perturbation, depth curve, adversarial,
operation-provenance-via-pack). Those are sequencing-wise *after*
ADR-0131.4, not bundled. Reviewer signature via ADR-0092 registry
is also reserved.
Trust boundary: read-only access to 5 committed lane reports;
single deterministic write to the artifact path. No dynamic
imports, no recomputation of lane verdicts.
Tests: 12/12 in tests/test_adr_0131_4_composite_math_gate.py
covering threshold pinning, heterogeneous shape handling, gate
logic (passing + every failure mode), GSM8K honest disclosure
(never gates), determinism (claim_digest + artifact byte-equality),
and a snapshot test confirming current main satisfies the gate.
ADR-0131.4 module note: the parent ADR-0131 plan named
formation/ratify.py + formation/promote.py as the wire-up site —
that was a misidentification (those govern teaching-example
SPECULATIVE→COHERENT bridging per ADR-0021, not domain-tier
promotion). Correct site is core/capability/, where audit-passed
gate already lives.
Four axes deferred from ADR-0131.G.3 (PR #183):
1. Fractions end-to-end: new _INITIAL_FRACTION_OF_RE extractor handles
`N/M of [a/an] <unit>` shape; _resolve_value already handles N/M arithmetic.
2. Multi-currency: _MONEY_SYMBOL widened to six symbols; _CURRENCY_SYMBOLS table
+ _resolve_currency dispatcher; ¢/€/¥/₱ wired end-to-end. £/pound sterling
deferred to G.3.2 (question extractor's single-token unit slot cannot parse
two-word surface "pounds sterling").
3. Multi-token cardinals: dedicated _MULTI_WORD_CARDINAL_RE extractor (approach a)
delegates to parse_compound_cardinal; avoids greedy unit-slot boundary ambiguity
from widening _VALUE.
4. Word-num-adjective: optional adjective group added to _INITIAL_HAS_RE and
_MULTI_WORD_CARDINAL_RE; closed adjective list identical to _CONJ_OBJECT_RE.
Also fixes six pre-existing G4 type bugs where _resolve_value() result was used
directly as a numeric operand (TypeError: _ResolvedValue is not a number).
Axis lane v1_1: 20/20 solved_correct, 0 wrong, 8/8 refusals, overall_pass=True.
GSM8K probe: 0/50 admission_rate unchanged, admitted_wrong=0 (safety rail intact).
42/42 new tests pass; parent v1 lane (26/26) unaffected.
Highest-risk axis of the ADR-0131.G capability iteration: within-
sentence multi-clause composition. Four extractors land in the
candidate-emitting parser; no graph-side or solver changes.
Parser extension (generate/math_candidate_parser.py)
- _conj_subject_each_candidates: '<A> and [his/her/their <kin>] <B>
each <verb> <N> <unit>' → 2 CandidateInitial (one per actor).
- _conj_object_candidates: '<E> has <N1> <unit1> and <N2> <unit2>' →
2 CandidateInitial for the same entity; same-unit conjuncts refuse
(would silently collide under solver overwrite-on-collision).
- _embedded_quantifier_candidates: '<E> has <N> <container> with <M>
<unit> in each [<container>]' → 1 derived CandidateInitial
(value=N*M).
- _embedded_quantifier_candidates (conj branch): '... <N1> <C> with
<M1> <U> in each ... and <N2> <C> with <M2> <U> in each ...' → 1
SUM CandidateInitial (value=N1*M1+N2*M2); mixed-unit refuses.
- CandidateInitial anchor whitelist widened to include
saved/earned/got/received/bought/made/paid (and inflections) —
narrow widening needed for the conjoined-subject-each shape.
Closed-set discipline
- Distributive 'each' only — 'each ... together/altogether' refuses.
- Two-way conjunction only — 3-way refuses by non-match.
- Cross-sentence coreference stays refused (within-sentence axis).
- Ambiguous 'each' scope refuses (container2 must agree).
Curated axis lane (32 cases)
- evals/math_capability_axes/G4_multi_clause/v1/cases.jsonl:
conj_subject_each ×6, conj_object ×6, embedded_quantifier ×6,
conj_embedded ×6, refusal ×8.
- evals/math_capability_axes/G4_multi_clause/v1/runner.py +
report.json: deterministic; wrong==0 gate; byte-equal across runs.
Tests (26 new)
- tests/test_adr_0131_G4_multi_clause.py: per-shape emission,
refusal probes (parametric), distributive-only policy,
cross-sentence refusal, runner byte-equality, GSM8K-probe gate.
GSM8K-probe gate (chosen: multi-clause refusals ↓)
- evals/gsm8k_math/train_sample/v1/report.json (candidate-graph
probe): multi-clause statement-refusal count 2 → 1. Case 0042
('Ella has 4 bags with 20 apples in each bag and six bags with 25
apples in each bag.') moves from statement-clause refusal to
question-layer refusal. Case 0026 ('Aaron and his brother Carson
each saved up $40') stays refused on the '$' value slot
(deferred to G.3 numeric-literals axis).
- evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json
(legacy probe): refreshed, byte-identical (legacy parser
untouched).
B3 + candidate-graph + GSM8K probe lanes all pass (95/95
regression). wrong==0 preserved everywhere — load-bearing for the
highest-risk axis.
Zero behavior delta on the main baseline (both substrates produce
0/50 admission today) — but every subsequent ADR-0131.G.<n> iteration
now produces attributable admission deltas on the probe, instead of
silently extending a parser layer the probe wasn't measuring.
Background: ADR-0131.G's probe consulted run_lane → _score_one →
parse_problem (legacy first-match-wins parser, pre-ADR-0126). Every
G.<n> iteration extends the candidate-graph parser via
_score_one_candidate_graph → parse_and_solve. The mismatch was
discovered during G.3 development and explicitly reserved as this
follow-up.
Changes:
- run_coverage_probe.py: switch import to _score_one_candidate_graph;
new private _score_lane aggregator mirrors run_lane's output shape
via per-case scoring; report root adds "substrate": "candidate_graph"
for audit trail.
- train_sample_coverage_report.json: regenerated. All metrics
byte-identical to prior baseline (0/50 admission, wrong=0).
refused_reasons_top text differs (candidate_graph: prefix instead
of parser:) — expected and part of the substrate audit-trail shift.
Discipline: separate small PR per ADR-0131.G's "expansion that only
moves admission must be a standalone PR" principle. Substrate swap
attributable; future G.<n> deltas attributable.
Evidence:
- python3 -m evals.gsm8k_math.train_sample.v1.run_coverage_probe
→ admission 0/50, wrong=0, safety_rail_intact=True, exit 0
- pytest tests/test_adr_0131_G_gsm8k_coverage_probe.py
→ 8/8 pass in 0.18s (no test edits needed; tests pin invariants
not numbers)
- No changes to runner.py, no changes to any G.<n> work in flight.
Effect on in-flight iterations: each G.<n> PR (G.1 Gemini / G.2 #182 /
G.3 #183 / G.4 Opus#2) rebases after this lands and refreshes its
committed train_sample_coverage_report.json with the new substrate's
numbers. Rebase is mechanical.
First capability-axis iteration after ADR-0131.G baseline. Extends the
candidate-graph parser's <value> slot to recognize:
- Money symbol literals: $N and $N.NN (1-2 decimals); $N.NNN refused
- Money word forms: N dollars / N cents
- Hyphenated multi-word cardinals: twenty-five, ninety-nine, ...
All money values normalize to integer cents, unit 'cents' — pack-aligned
with en_units_v1's canonical_unit='cent' for the money dimension.
en_numerics_v1's parse_compound_cardinal handles hyphenated cardinals.
Parser changes (generate/):
- math_candidate_parser.py: _VALUE alternation widened; _resolve_value
refactored to return _ResolvedValue|None carrying optional unit
override; _INITIAL_HAS_RE unit slot made optional; dollar/dollars →
cents normalization at candidate build.
- math_roundtrip.py: new _unit_grounds helper (money-aware); _value_grounds
widened for the three new literal shapes; roundtrip_admissible uses
_unit_grounds for the unit check.
- math_candidate_graph.py: _initial_admissible and _question_admissible
use _unit_grounds.
New axis lane (evals/math_capability_axes/G3_numerics/v1/):
- 26 curated cases (20 positive across 4 classes + 6 refusal probes)
- runner.py wraps _score_one_candidate_graph; byte-equal report.json
- 20/20 positive solved correct; 6/6 refusal probes refused typed;
solved_wrong == 0; overall_pass == True
Tests: 27/27 in 0.19s. 420 existing candidate-parser/math-parser/pack
tests still green. GSM8K probe safety rail (admitted_wrong == 0)
preserved.
Honest scope-limit (documented in ADR): admission_rate on the GSM8K
probe stays at 0/50 because (a) the probe currently consults the legacy
parser path, not the candidate-graph pipeline G.3 extends, and (b) most
money-bearing GSM8K cases fail first on verb (G.1) or multi-clause (G.4)
shape, not on the money literal. The axis lane is the load-bearing
measurement for this iteration. Reserved follow-up: a small probe-
infra ADR to switch run_coverage_probe.py to the candidate-graph
pipeline.
Out of scope, deferred to G.3.1: fractions end-to-end (resolver supports
N/M but no axis cases), multi-currency (¢ € £ ¥ ₱), space-separated
multi-word cardinals (one hundred), word-number-adjective compositions
(five full boxes).
Wire compare_additive / compare_multiplicative extractors into the
candidate-emitting sentence parser, closing the deferred phase flagged
at generate/math_candidate_parser.py:30.
Capability axis: comparatives (additive + multiplicative)
- generate/math_candidate_parser.py: new _compare_additive_candidates,
_compare_multiplicative_candidates, _compare_nested_candidates
emitting CandidateOperation records keyed to the four
Comparison.direction literals registered in ADR-0123.
- Closed-set anchor alternation; 'less' admitted as surface synonym of
'fewer'; reference slot widened to admit "the number/amount of <unit>"
for nested forms.
- Nested 'A has N more <unit> than M times <REF>' emits two flat
candidates (additive + multiplicative); binding-graph picks the
admissible composition or refuses (no solver stub).
Curated axis lane (24 cases)
- evals/math_capability_axes/G2_comparatives/v1/cases.jsonl:
8 additive / 8 multiplicative / 3 nested / 5 refusal
- evals/math_capability_axes/G2_comparatives/v1/runner.py +
report.json: deterministic, wrong==0 gate, byte-equal across runs.
Tests (21 new)
- tests/test_adr_0131_G2_comparatives.py: per-direction at-least-one
passing, nested-both-emitted, closed-set refusal, runner
byte-equality, GSM8K-probe gate (comparative-clause refusals
strictly decrease).
GSM8K-probe gate (chosen: comparative-clause refusals ↓)
- evals/gsm8k_math/train_sample/v1/report.json (candidate-graph
probe): comparative-clause refusal count 2 → 1 (case 0009 'Jen has
10 more ducks than four times the number of chickens' moves from
statement-clause refusal to question-layer refusal). admitted_wrong
remains 0; admission_rate unchanged (downstream composition is a
follow-up ADR).
- evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json
(legacy probe): refreshed, byte-identical (legacy parser untouched).
B3 + candidate-graph + GSM8K probe lanes all pass (90/90). Direction
vocab stays closed to {more, fewer, times, fraction}; wrong==0
preserved everywhere.
ADR-0131 deferred GSM8K because it rewards paraphrase flexibility,
which is the deterministic engine's structural weakness. This ADR
re-engages it on architecture-aligned terms: as a *coverage probe*
of the bounded grammar + binding graph, not a promotion gate.
The framing pinned by this ADR:
GSM8K is not a target. The model's capability is the target.
GSM8K passing is the symptom of capability, not the goal of
the work.
Wrong mindset (rejected by ADR's iteration discipline):
"Find templates that admit more GSM8K cases."
Right mindset (load-bearing):
"Extend the model's NL-to-typed-graph capability along
principled axes (verb classes, comparative structures, numeric
forms, multi-clause grammar). GSM8K admission rises as a
side effect alongside every other word-problem corpus."
Baseline pinned by this commit:
admission_rate: 0/50 = 0.0%
admitted_wrong: 0 (gate intact, safety rail bulletproof)
refused: 50/50 = 100.0%
Every refusal is a typed parser error citing the specific clause
that did not match a template. Zero crashes, zero confabulations
— refusal-first works perfectly at admission rate zero.
What's in this PR:
- ``docs/decisions/ADR-0131.G-gsm8k-coverage-probe.md``: the ADR.
Cites parents (ADR-0131, -0115/-0116/-0117, -0131.3, -0132..-0135).
Documents the capability-first iteration discipline that every
subsequent ADR-0131.G.<n> must follow:
1. Name a single capability axis the iteration extends
2. Add B3-style curated coverage cases (capability proves
itself OUTSIDE GSM8K)
3. Re-run both B3 lane + GSM8K probe; B3 must not regress
4. Reject any expansion that only moves GSM8K admission
- ``evals/gsm8k_math/train_sample/v1/run_coverage_probe.py``:
pure-adapter wrapper around the existing run_lane. Emits a
deterministic train_sample_coverage_report.json with metrics,
per-case outcomes, and the top refused-reason families (the
work queue for capability extension).
- ``evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json``:
the baseline report. Diff-able artifact every future iteration
moves.
- ``tests/test_adr_0131_G_gsm8k_coverage_probe.py``: 8 contract
tests pinning the safety rail (admitted_wrong == 0), typed
refusal invariant (every refused case has non-empty reason),
closed outcome vocabulary, deterministic replay, committed-
report matches fresh-run.
The promotion-gate composite (B1 + B2 + B3) is unaffected.
ADR-0131.4 still consumes those three. The GSM8K probe is
empirical context for honest external claims, not a gate.
* feat(ADR-0131.1.F): frontier-baseline comparison harness for B1
Adapts the ADR-0119.4 methodology (frozen citations + comparison JSON
with disclaimer) to B1, with three additions for the
architecture-aligned claim:
1. A provider-agnostic live head-to-head runner. Adapters for
Anthropic / OpenAI / Google import their SDKs lazily so the
package loads cleanly without them installed. Each provider has a
documented FRONTIER_<VENDOR>_KEY env var; the runner refuses with
a typed FrontierRunError when keys are absent and the cache cannot
cover all cases. Every response is cached one-record-per-line at
responses/<provider>/<model>.jsonl so subsequent runs replay
byte-equally without re-calling the API.
2. A conservative free-text-to-closed-vocab verdict parser. Ambiguous
or sentinel-free provider replies collapse to "refused" — a
polarized verdict is never confabulated from prose. Chain-of-
thought replies use last-token-wins (provider deliberates, then
concludes). This is the load-bearing seam that prevents the
runner from manufacturing scores the provider didn't deliver.
3. Architecture-aligned comparison metrics. accuracy is reported but
foregrounded as the least-load-bearing; refusal_correctness
(CORE 100% by lane-gate construction vs. frontier confabulation
rate) and determinism (CORE byte-equal vs. frontier variance) are
the differentiators.
Frozen adjacent-benchmark citations cover Anthropic
(claude-3-5-sonnet on MATH, claude-opus-4-1 on AIME), OpenAI
(gpt-4o on MATH), and Google (gemini-1.5-pro on MATH). The scope
disclaimer documents that these are adjacent, not head-to-head.
Head-to-head numbers, when run, land in the cache; the comparison
JSON joins them with CORE's existing lane result.
22 tests pin the methodology: citation shape (every field, https
URL, YYYY-MM-DD date), provider-registry shape, verdict-parser
conservatism (multiple chain-of-thought cases), runner caching
behavior (no double-invoke), comparison-JSON determinism (byte-equal
across runs).
No live API call at test time. The harness gates real runs behind
explicit env vars + CLI invocation.
Composes with ADR-0131.1 (B1 v1), ADR-0131.1.B (v1.B hardening,
#169), ADR-0131.1.S (sealed holdout, #173).
* feat(ADR-0131.1.F): live head-to-head — anthropic/claude-sonnet-4-6
First real frontier baseline on the full B1.B 185-case set
(curated + generated). Cached one-record-per-line at
responses/anthropic/claude-sonnet-4-6.jsonl. Re-runs replay from
disk; no further API calls.
Headline (after scoring fix):
CORE 185/185 = 100.0% accuracy
3/3 = 100.0% refusal_correctness
deterministic (byte-equal across runs)
anthropic/claude-sonnet-4-6 182/185 = 98.4% accuracy
1/3 = 33.3% refusal_correctness
non-deterministic (temperature=0, but
not byte-equal architecturally)
The 1.6pp accuracy gap is informative; the refusal-correctness gap
is the architecture-aligned story. Sonnet's three misses:
sym-eq-v1-0016 [difference_of_squares]
(x^2 + 1)*(x^2 - 1) vs x^4 - 1
Sonnet: NOT_EQUIVALENT (math error on a textbook identity)
sym-eq-gen-v1-0153 [generated_refusal_function]
sin(x) vs x
Sonnet: NOT_EQUIVALENT (confabulated — should refuse,
transcendental outside polynomial scope)
sym-eq-gen-v1-0154 [generated_refusal_negative_exponent]
x^-1 vs 1
Sonnet: NOT_EQUIVALENT (confabulated — should refuse,
negative exponent outside scope)
Sonnet correctly refused only on syntactically malformed input
("x +"); on syntactically-valid-but-semantically-out-of-scope inputs
it confidently polarized rather than refusing. CORE refuses both
classes with typed reasons.
Scoring fix: comparison.py now composes curated + generated cases
(mirroring runner.py) so the head-to-head scores the full 185-case
lane, not just the 30 curated. The initial run scored only 30/185
because the generated set was not loaded into _load_cases().
22/22 frontier-methodology tests still pass.
* feat(ADR-0131.1.F): three more head-to-head runs + Ollama adapter
Three additional providers ran against the full B1.B 185-case set,
joining the prior claude-sonnet-4-6 result:
CORE 185/185 = 100.0% acc | 3/3 = 100% refusal | 33 ms
claude-sonnet-4-6 182/185 = 98.4% acc | 1/3 = 33.3% refusal | 294 s
claude-opus-4-7 178/185 = 96.2% acc | 1/3 = 33.3% refusal | 309 s
gpt-5 134/185 = 72.4% acc | 1/3 = 33.3% refusal | 1153 s
qwen3:8b (M1 local, partial) 91/91 = 100.0% acc | n/a no refusal-class | killed
CORE is the only system at 100% on both axes, and runs ~9,000×
faster than the cheapest cloud frontier, ~35,000× faster than gpt-5,
and finishes in less wall time than a single API call to any of the
three frontier models.
Three distinct frontier brittleness modes, all rooted in
"not actually canonicalizing":
- sonnet-4-6 confabulates polarized verdicts on out-of-scope
inputs (sin(x), x^-1). Misses one in-scope difference-of-squares
identity (x^2+1)*(x^2-1) vs x^4-1.
- opus-4-7 pattern-shortcuts five near-miss-constant cases —
accepts (-x+3)*(4x+1) == -4x^2+11x+4 (correct constant is 3,
not 4) without expanding. Same two out-of-scope confabulations
as sonnet.
- gpt-5 over-refuses 50 in-scope cases — literally replies
"REFUSED" to x*(x+1) == x^2+x and (x+1)*(x-1) == x^2-1. Same
two out-of-scope confabulations as sonnet/opus.
The qwen3:8b partial is the surprise: on the 91 in-scope cases it
completed (spanning the categories where the frontier models failed),
it scored 100%. Refusal-class cases weren't reached before the run
was killed for being impractically slow (~22s/case on M1).
Changes in this commit:
- frontier_runner.py: anthropic adapter now omits ``temperature``
for claude-opus-4-x (the parameter is rejected by 4.x models);
openai adapter switches to ``max_completion_tokens`` for the
gpt-5 / o-series reasoning models; new ``_ollama_invoke`` that
posts to localhost:11434 with no third-party dep; per-case
``latency_ms`` is now captured on every NEW cached response
(future runs only — these four runs pre-date the patch).
- comparison.py: ``_load_cases`` composes curated + generated
(185 cases) instead of curated only; ``_score_provider``
surfaces ``latency_summary`` when records carry latency_ms.
- tests: provider-registry test relaxed to "cloud trio is a
subset of PROVIDERS"; env-key test allows ``_KEY`` (cloud
secret) or ``_URL`` (local endpoint).
Refines BoundUnknown from "the symbol whose value the solver determines"
to "the symbol at a specific temporal/state index with a specific
question-form". Two new required fields on BoundUnknown — state_index
(initial/terminal/Operation(operation_index)) and question_form
(count/rate/total/difference/ratio/identity) — populated by the new
pure-function resolver in generate/binding_graph/question_target.py.
The adapter (ADR-0133) now delegates Unknown -> BoundUnknown construction
to bound_unknown_from_math_problem_graph. No runtime wiring, no solver
invocation. Phase 5 (bounded-grammar / B3 integration) remains deferred.
Refusal-first via the new QuestionTargetError (sibling of AdapterError /
AdmissibilityError). Closed reason vocab: not_a_math_problem_graph,
unknown_entity_not_in_entities, apply_rate_unit_mismatch,
unmappable_question_form. Closed precedence rule on question_form
documented in ADR-0135 (compare_multiplicative > compare_additive >
apply_rate{numerator|denominator unit-match} > count); ambiguity refuses.
SemanticSymbolicBindingGraph.__post_init__ gains a cross-collection
guard: Operation(operation_index) must satisfy operation_index <
len(equations). canonical_string emission widened to include
state=... form=... tokens (hash differs from Phase 3 main by design —
not a regression; byte-equal across runs preserved).
Parents: ADR-0132 / ADR-0133 / ADR-0134.
Tests: +70 new (45 unit in test_binding_graph_question_target.py +
25 integration in test_binding_graph_adapter_question_target.py); 5
Phase 1+3 BoundUnknown fixtures migrated. Total binding-graph lane
295/1 pass (1 pre-existing test_symbol_binding_uses_slots failure on
Python 3.14, unrelated to Phase 4 — exists on origin/main). Pyright
clean on new and modified files. No edits to algebra/, chat/, core/,
or runtime hot path. Field invariant untouched.
Wires deterministic, refusal-first dimensional analysis into the
binding-graph adapter. Every BoundEquation emitted by
bind_math_problem_graph now carries either admissibility_status='admitted'
+ populated unit_proof or admissibility_status='refused' + typed
refusal_reason. No silent coercion; no invented units; no solver.
Adds:
- generate/binding_graph/units.py — pure unit algebra over a 6-dim
integer exponent vector (length, time, mass, money, count,
temperature). Closed vocabulary loaded once from en_units_v1
(ADR-0127) and memoized; composite "<num>_per_<denom>" resolved
recursively; conservative depluralization; refusal-first.
- generate/binding_graph/admissibility.py — check_admissibility with
per-operation-kind dispatch over the closed 8-string vocab, typed
AdmissibilityError (closed reason set), frozen UnitProof.
- ADR-0134 documenting the contract, invariants, and Phase 4-5
deferrals.
Adapter changes are surgical: synthesizes operand-literal symbols where
the verifier needs them (op<NNN>__multiplicand / __divisor / __rate),
then stamps each equation via check_admissibility. Input/output types
unchanged; bind_math_problem_graph still byte-equal across runs.
Tests: 226 total in the binding-graph lane (110 Phase 1+2 still pass; 47
units + 40 admissibility + 29 adapter-units new). Pyright clean on all
new files. No runtime wiring outside generate/binding_graph/.
Phase 4 (question-target binding) and Phase 5 (B3 / bounded grammar)
remain deferred per the brief.
Docs-only coordination artifact for the ADR-0131 post-GSM8K pivot.
Adds an implementation plan that defines the composite proof corridor,
claim boundaries, phase sequencing, and non-negotiable acceptance gates.
Docs-only PR. Follow-up to ADR-0127-0128-RESULTS' Path-B trigger.
Replaces ADR-0120's GSM8K-coverage requirement for the
mathematics_logic expert promotion with a composite gate over
three architecture-aligned benchmarks:
Benchmark 1: Symbolic equivalence (~500 algebra cases, public
+ sealed split). Primary discriminator — CGA exact-recall.
Pass: correct_rate >= 0.95, wrong == 0.
Benchmark 2: CORE-native teaching-corpus eval. Internal
consistency lane sourced from ratified packs (en_arithmetic_v1
+ en_units_v1 + en_numerics_v1). Grammar matches by
construction (no paraphrase-variance gap).
Pass: correct_rate >= 0.95, trace_hash byte-equality, wrong == 0.
Benchmark 3: Bounded-grammar word-problem set (~150 cases,
pre-filtered to ADR-0126/0127/0128 parser scope). Demonstrates
end-to-end pipeline determinism on the in-scope distribution.
Includes adversarial out-of-grammar probes.
Pass: correct_rate >= 0.95, wrong == 0.
Three benchmarks (not one) prevent overfitting per ADR-0114a
and force the math expert to demonstrate three distinct
architectural properties (algebraic correctness, internal
consistency, end-to-end pipeline determinism).
GSM8K stays as a stress-test lane the math expert RUNS but is
NOT gated on. Refusal counts reported in expert-claims artifact
as honest disclosure.
Other 12 ADR-0120 expert-promotion checks UNCHANGED.
Status: Proposed. The contract revision (removing GSM8K from
the gate) is a follow-up implementation PR after sub-phases
0131.1-0131.6 land the new benchmark substrate.
Integrates en_units_v1 (#164) + en_numerics_v1 (#163) into the
ADR-0126 candidate-graph parser. Loader merge (re-exports from
numerics_loader.py give single import path), pack-aware unit
canonicalization (handles irregular plurals like feet/children
via lookup_unit), indefinite-quantifier refusal (ADR-0128.4 —
'some'/'many' emit no candidates, preserving wrong==0), and
widened initial-possession shapes:
- <Entity> has N <unit> [of <substance>] (ADR-0127 substance qualifier)
- There are N <unit> [in <place>] (implicit-subject shape)
Plus: pack-backed cardinal grounding in math_roundtrip._value_grounds
(widens word-number coverage from hard-coded 0-12 to full numerics
pack cardinal table + compound rule). Op-pattern trailing prep
alternation gains of/for/with for substance qualifiers.
REGRESSION: 1050/1050 tests green across math + ADR-0126 + ADR-0127
ratification + ADR-0128 ratification + runner.
EMPIRICAL RESULT (the Path-B trigger ADR-0126/0127/0128 named):
correct = 0/50 wrong = 0/50 refused = 50/50
on evals/gsm8k_math/train_sample/v1/cases.jsonl
Per ADR-0127's exit criterion (correct >= 10/50, wrong == 0):
**MISSED** — the full deterministic design (candidate-graph
topology + units pack + numerics pack + pack-aware parser) does
not move the GSM8K-math lane. This is the real Path-B trigger.
WHAT WORKS (synthetic verification, 6/6 cases solve end-to-end):
- 'Jan has 5 apples. Jan buys 3 apples. ...' -> 8
- 'Sam has 10 feet of rope. Sam uses 3 feet of rope. ...' -> 7
- 'There are 5 kids in camp. ...' -> 5
- 'Sam has 10 children. Sam loses 2 children. ...' -> 8
- (money + time-dimension variants pass)
WHY GSM8K STAYS AT ZERO: real GSM8K problems carry compound
linguistic structure (pronouns across statements, possessives,
subordinate clauses, multi-word entities, multi-step inference)
that no amount of pack vocabulary addresses. Per-sentence parse
rate improved measurably on simple shapes; joint problem-level
pass rate stayed at zero because every real problem contains at
least one sentence the parser still cannot handle.
Full results + Path-B recommendation in
docs/decisions/ADR-0127-0128-RESULTS.md. The substrate
(architecture + packs) stays load-bearing in main; the math
expert promotion path retargets to a benchmark where exact
recall and determinism are the discriminators (proposed
ADR-0131).
Operator-supplied review of 'Beyond Traditional Pedagogy' triggered
a literature confirmation pass and a structural cross-walk against
CORE's teaching loop. Three artifacts:
1. ADR-0129 (DEFERRED) — Spaced reviewed-correction replay.
Maps onto retrieval-with-spacing literature (most robust finding
in cognitive psychology). Deterministic re-run of past
corrections at fixed cadence to verify they still produce
intended outcomes; failures emit operator-visible events
(no auto-correction). Deferred pending GSM8K-math Path-A/B
resolution + observed incident triggering un-deferral criteria.
2. ADR-0130 (DEFERRED) — Pre-articulation calibration logging.
Maps onto metacognitive prediction-outcome literature. Logs
CORE's pre-correction prediction; emits gap event on
correction acceptance. Provides empirical signal for 'is CORE
actually getting better' across pack-version cohorts. Deferred
pending same conditions as ADR-0129; the two compose if
un-deferred.
3. SESSION-2026-05-23 session note. Documents the review process:
literature confirmation pass (productive failure overstated,
retrieval transfer weaker than claimed, embodied cognition
replication crisis), missed frameworks (worked-example effect,
expertise reversal, CLT, deliberate practice, Bloom's 2-sigma),
structural cross-walk to CORE architecture (12 mappings), and
the rationale for ADRs 0129 + 0130 over alternative ports
(productive failure rejected as inverse of wrong==0; pre-testing
same; embodied learning N/A).
No code changes. Docs-only PR; lands independently of in-flight
ADR-0126 / 0127 / 0128 substrate chain.
Diagnostic from ADR-0126's first train-sample run (0/0/50): every
refusal happens at the first statement of each problem, and every
refused first statement fails on the unit-of-measurement construction,
not on the operation grammar. Adding more verb regexes is the per-axis
treadmill that produced 4 zero-lift ADRs. Units form a finite, externally
well-defined ontology (NIST SI tables, currency, English container nouns)
that is semantic substrate the candidate-graph parser was designed to
consume.
Scope:
- en_units_v1 pack: dimensions, units (<=60), containers, rate connectors
- conversions.jsonl: directed weighted graph of within-dimension unit pairs
- 3 new initial-possession shapes + rate-declaration extractor in the
candidate parser
- Round-trip filter gains optional pack-typed-unit check
- Solver gains dimensional canonicalization helper (shortest path through
conversion graph); fired edges join SolutionTrace.steps for replay
- Pack ratification invariants: round-trip identity, per-dimension
connectivity, path consistency, canonical unit per dimension
Wire the same train-sample exit criterion as ADR-0126 (correct >=10/50,
wrong==0). If passed -> sealed holdout. If still missed -> Path B
trigger is REAL (full deterministic design with units substrate failed),
demote GSM8K, re-target math expert promotion.
Also commits the empirical evidence: train_sample/v1/runner.py swapped
_score_one -> _score_one_candidate_graph; report.json baseline 0/0/50
confirming the candidate-graph topology refuses cleanly without units
substrate.
Architectural pivot from per-axis grammar expansion (ADR-0122/0123/0123a/
0123b, all 0/1319 sealed lift) to candidate-graph topology with round-trip
admissibility filter. Converts compound-gap failure from multiplicative
p^k to parallel 1-(1-p)^k.
Adds new invariant: round-trip admissibility (op slots must reconstruct
to byte-equal source sentence under whitespace/case normalization).
Preserves wrong==0, trace_hash byte-equality, pack-binding, replay
determinism, no stochastic sampling.
Exit criterion: 50-case GSM8K train-split sample (unsealed) must show
correct >=10/50 with wrong==0 before any sealed-holdout run. If miss,
escalate to Path B (benchmark re-selection).
Supersedes ADR-0123b (never opened as PR).
ADR-0123-parser-comparison-phrasing as the **surface increment** on
PR #155's substrate (commit c9bd5d4). Closes the last architectural
gap in the comparison-phrasing class: before this commit, the
substrate's solver evaluated comparison problems successfully but
realize() crashed with `unknown operation_kind 'compare_additive'`
when asked for show-your-work prose.
Substrate (PR #155) already shipped:
- `Comparison` typed graph operand
- `compare_additive` / `compare_multiplicative` operation kinds
- parser patterns for the four canonical surfaces
(N more / N fewer / twice / N times / half)
- solver + verifier wiring + pack lemmas
(en-arith-006 compare_additive, en-arith-007 compare_multiplicative)
This surface adds:
- `_compare_additive_sentence(step)` rendering `direction='more'|'fewer'`
- `_compare_multiplicative_sentence(step, entity_units)` rendering
`direction='times'|'fraction'`
- two new branches in `_step_sentence` dispatch
- `_step_sentence` signature widened with optional `entity_units` map
(derived once-per-trace in `realize()` from `graph_initial_state`)
- ADR-0123-parser-comparison-phrasing.md (~15 invariants, substrate
+ surface decomposition rationale, multi-construction barrier
inheritance)
- 26 invariants pinned across canonical surfaces, plurality
independence, byte-determinism, refusal discipline, and
backwards-compatibility with the pre-comparison realizer templates
End-to-end pipeline now operates on all four canonical comparison
shapes:
parse_problem(
"Alice has 5 apples. Bob has 3 more apples than Alice. "
"How many apples does Bob have?"
) -> solve() -> realize().as_prose() ->
"Alice has 5 apples. Bob has 3 more apples than Alice, giving Bob
a total of 8 apples. Bob has 8 apples."
Measurement (this PR):
- 26/28 direct ADR-0123 tests pass; 2 skipped (CORE_HOLDOUT_KEY)
- `core eval cognition` byte-identical: 100/100/100/100
- ADR-0118 stepped-realizer templates re-render byte-identically
- Substrate measurements continue to hold
Honest non-result: sealed `correct_rate` stays at 0/1319. The
realizer cannot create matches the parser refuses; the multi-
construction barrier the substrate ADR documented holds at the
surface layer too. Cumulative lift signal expected only after the
3rd/4th foundational class lands (per ADR-0121's revised
sequencing). `wrong == 0` holds by construction — realizer only
renders successful traces.
Pre-existing failure noted (not introduced by this PR):
`tests/test_adr_0085_gloss_aware_cause.py::test_flag_off_metrics_byte_identical`
fails on substrate base (c9bd5d4) without these changes — an
ADR-0085 cognition baseline drift unrelated to the realizer.
Documents the Phase 5 GSM8K-math substrate completion across 7 narrative docs.
All 8 sub-phases of ADR-0119 (5.1 through 5.8) have landed on main; ADR-0114a's
10 anti-overfitting proof obligations are all discharged for the gsm8k_math lane.
Key facts surfaced in each doc:
- CORE-original public split: 150/150 correct, 0 wrong, 0 refused
- Real GSM8K test (sealed holdout): 0 correct, 0 wrong, 1319 refused
- Adversarial suite: 38 cases x 12 families, 0 wrong
- Depth curve: flat at 1.0 across depths 1-8 on public split
- Frontier baselines: Claude 3.5 Sonnet 96.4%, GPT-4 92.0%, Gemini 1.5 Pro 90.8%
- New lane shape gsm8k_capability_shape in LANE_SHAPE_REGISTRY
- New operational pack en_arithmetic_v1 (5 lemmas)
- ADR-0120 (first expert promotion contract) is the next gate
Docs updated: docs/PROGRESS.md, docs/capability_roadmap.md, docs/runtime_contracts.md,
docs/Whitepaper.md (§XIII), docs/Yellowpaper.md (gsm8k_capability_shape formal spec),
README.md, docs/decisions/README.md (current frontier).
No code changes. No new ADRs.
First worked attempt at promoting a domain under the ADR-0120
expert promotion contract. The contract refuses honestly.
Gate evaluation against live state:
ADR-0114a obligations: 10 of 10 pass
ADR-0120 contract-level gates:
audit_passed_holds ✓
correct_rate (public) ✓ 150/150 = 1.0
correct_rate (sealed) ✗ 0/1319 = 0.0 < 0.60 floor
signed_expert_claim ✗ (no entry, downstream of correct_rate)
Decision: mathematics_logic NOT promoted; stays at audit-passed.
Substantive blocker: parser grammar covers 0/1319 of real GSM8K.
What this proves
- The contract is genuinely falsifiable. ADR-0120 §"Threshold
rationale" deliberately set the floor above current measurement
so the first attempt would defer honestly. Same load-bearing
pattern as ADR-0107 → ADR-0110 for audit-passed.
- Wrong-zero discipline holds against real GSM8K (the load-
bearing positive claim). CORE refuses every problem outside
its grammar without confabulating on a single one.
What unlocks the promotion
Multi-ADR parser-expansion arc lifting sealed-GSM8K correct_rate
from 0.0 to ≥ 0.60. Each construction class (rate/comparison/
percentage/time-modal/etc.) ships as its own scoped ADR with:
- parser+solver+verifier+realizer extensions
- re-measurement on sealed holdout
- ADR-0118a OOD re-measurement (no surface-feature regression)
- ADR-0125 perturbation re-measurement (no invariance regression)
- ADR-0119.5 adversarial re-measurement (no new misparses)
Honest-fitting discipline: every lift is graded on the anti-
overfit obligations BEFORE the correct_rate change counts.
Tests: 6/6 with CORE_HOLDOUT_KEY; 4/6 + 2 skipped without (matches
ADR-0119.7 seal discipline).
This deferral demonstrates the expert tier's promotion machinery
is load-bearing — the gate has refused at least once before any
domain reaches it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Defines the `expert` ledger tier (sixth status above `audit-passed`)
and the composition gate that governs every future promotion.
The gate composes ALL TEN ADR-0114a obligations plus three contract-
level checks:
- audit_passed predicate must hold
- correct_rate >= 0.60 on public AND sealed holdout splits
- signed expert_claims entry whose digest reproduces byte-equal
Numeric thresholds (load-bearing choices documented with rationale):
- correct_rate floor: 0.60 ("advanced" — above weak open-source LLMs,
forces real architecture work, raisable later)
- depth-curve flatness ε: 0.05 (accuracy(N) ≥ accuracy(d1) · 0.95^(N-1))
- drift tolerance from promotion: ±0.02
Documents the post-ADR-0120 sequence in Open Candidate Directions:
Phase 1: ADR-0121 first worked math promotion (likely deferral)
Phase 2: parser-expansion arc to lift sealed-GSM8K correct_rate
Phase 3: math expert promotion succeeds
Phase 4: second domain = symbolic_logic (60-70% of math substrate cost;
ProofWriter / PrOntoQA benchmark; same machinery class)
Phase 5: third domain = high-stakes refusal-centric (medical or legal;
wedge-sharpener; needs two prior successes first)
Phase 7: open candidate — multi-reviewer threshold signing for expert
No code lands with this ADR. Implementation ships under ADR-0120a
to keep the contract change reviewable independently.
The first promotion attempt (ADR-0121) will likely defer on the
correct_rate gate — current measurement is 0/1319 on real GSM8K.
That deferral IS the contract working as designed; same load-
bearing pattern ADR-0107 → ADR-0110 demonstrated for audit-passed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The 1,319 GSM8K test cases are now sealed at
evals/gsm8k_math/holdouts/v1/cases.jsonl.age, age-encrypted to the
ADR-0119.1 recipient. Plaintext never touched disk in the working
tree; only ciphertext is committed.
First honest CORE-vs-real-GSM8K measurement
cases_total: 1319
correct: 0
wrong: 0 ← ADR-0114a Obligation #4 holds against external corpus
refused: 1319
overall_pass: True
Zero confabulation. Parser refuses what it can't grammar-handle; the
"wrong == 0" discipline survives the move from CORE-original cases
to a real public benchmark. The 0/1319 correct rate is the truthful
gap that ADR-0120's threshold work will quantify.
What landed
scripts/seal_gsm8k_test.py
- Loads GSM8K via datasets.load_dataset("openai/gsm8k", "main")
- Strips worked-solution prose; extracts final-answer integer/float
after "####" (handles "2,125" → 2125 thousands-separator)
- Reads recipient from docs/holdout_recipients.txt (single repo key
per ADR-0119.1)
- Encrypts via pyrage; writes only ciphertext
- Refuses to overwrite test path with train-derived seal
evals/gsm8k_math/runner.py
- Empty expected_unit (sentinel) skips unit-comparison; grades on
answer value alone. Required because GSM8K answers carry no unit
structurally. wrong-zero discipline preserved.
tests/test_adr_0119_7_sealed_gsm8k.py — 6 invariants:
1. sealed file present + age-formatted
2. no plaintext companion files (sibling-leak guard)
3. decrypted JSONL matches documented schema
4. runner against decrypted suite produces wrong==0
5. tests skip (not fail) when CORE_HOLDOUT_KEY unset
6. case ids match "gsm8k-test-NNNN" pattern
Defensive gitignore: plaintext patterns under
evals/gsm8k_math/holdouts/v1/ are explicitly excluded.
ADR-0114a obligation roll-up
10/10 discharged for the gsm8k_math lane:
#1 ✓ sealed-holdout (fab_control + GSM8K test)
#2..#10 ✓ as before
Phase 5 status: 5.1..5.7 done; 5.8 in flight (PR #149). After 5.8
merges, ADR-0120 (first expert promotion contract) becomes
feasible.
Test plan
- pytest tests/test_adr_0119_7_sealed_gsm8k.py with CORE_HOLDOUT_KEY → 6/6
- pytest without CORE_HOLDOUT_KEY → 3 pass + 3 skip
- core test --suite smoke -q → 67/67
- CLAIMS.md regenerated (no diff)
- HF token NEVER in repo (saved at ~/.cache/huggingface/token, mode 600)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Umbrella ADR for Phase 5 of ADR-0114. Decomposes the eval-lane work
into eight sub-phases (5.1..5.8) that ship under their own ADRs.
Sub-phase scope:
5.1 — sealed-holdout encryption (one lane, fabrication_control)
5.2 — CORE-original GSM8K-style corpus (dev + public, 200 cases)
5.3 — lane runner (parser → solver → verifier → realizer)
5.4 — frontier-baseline comparison (Obligation #7)
5.5 — adversarial generation; misparse rate zero (Obligation #8)
5.6 — depth-curve measurement harness (Obligation #6, measurement)
5.7 — sealed GSM8K test as the holdout (Obligation #1, lane-side)
5.8 — overall lane gate; new gsm8k_capability_shape
Roadmap only — no code lands with this ADR. Each sub-phase ships
independently and discharges a specific ADR-0114a obligation.
Critical invariant: dev + public splits are CORE-original; the real
GSM8K test set enters only via the encrypted holdout under 5.7.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 4 of the ADR-0114 GSM8K-math roadmap. Consumes a SolutionTrace
and emits one sentence per step plus setup + answer sentences. Pure
function; same trace → byte-equal RealizedTrace.
What landed
generate/math_realizer.py
- realize(initial_state, trace) -> RealizedTrace
- Frozen RealizedTrace dataclass with canonical_bytes() + as_prose()
- Per-kind sentence rules (add / subtract / transfer / multiply×2 /
multiply×3 / multiply-general / divide)
- Singular/plural surface rule matches parser canonicalization
- Typed RealizerError on unrecognized step kinds
tests/test_math_realizer.py — 60 cases pinning five invariants:
1. All 50 dev-set cases realize without error
2. Determinism: byte-equal RealizedTrace across two calls
3. Setup sentence count == initial_state count
4. Step sentence count == operation count
5. Answer sentence contains the resolved value + unit
ADR-0114a obligation discharge update
ADR-0118 hardens determinism (#9) across a third layer (realizer)
and makes #3 / #10 human-inspectable via the prose surface. No
obligation is directly newly discharged by ADR-0118; it's substrate
for ADR-0119 GSM8K eval lane.
Round-trippability of the prose through the parser is explicitly
out of scope for this phase. The trace is the verifiable artifact
(ADR-0117); the prose is human-readable documentation.
Tests: 60 new realizer cases; 546 total green across realizer +
parser + solver + verifier + OOD; 67/67 smoke green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 3 of the ADR-0114 expert-capability roadmap. Re-applies every
step of a SolutionTrace from the input graph's initial state and
asserts byte-equal reproduction of answer_value. Pure function; same
(graph, trace) → byte-equal VerifierVerdict.
Why this is distinct from the solver
ADR-0116's solver enforces correctness at construction. ADR-0117's
verifier is a SECOND, INDEPENDENT implementation that re-derives
every value the trace claims. The verifier does NOT call solve(). It
re-implements the operation semantics from ADR-0116 directly inside
_verify_step. If the solver had a bug or was tampered with after the
fact, the verifier catches it.
Six checks per verdict (named, ordered, audit-logged):
1. graph_canonical_hash_matches
2. pack_id_matches
3. pack_lemmas_resolve
4. step_pack_lemma_ids_match_bindings
5. step_replay_matches_before_after
6. answer_value_reproduces
Seven named tamper classes all caught:
- mutated before_value / after_value / operand of any step
- mutated pack_lemma_id of any step
- mutated graph_canonical_hash
- mutated answer_value
- mutated pack_id
- mutated target_before / target_after of transfer step
ADR-0114a obligation update
#3 Replay-equal trace — now discharged at VERIFIER FIDELITY
(was solver-only under ADR-0116). A third party with only
(graph, trace, pack) can reproduce the answer byte-equal.
Five of ten obligations now load-bearing: #3, #4, #9, #10 plus
in-flight #2 (Codex's ADR-0118a OOD generator).
Tests: 62/62 verifier suite green; 67/67 smoke green; existing
solver + parser + schema suites unaffected.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 2 of the ADR-0114 expert-capability roadmap. Consumes the
MathProblemGraph from Phase 1 and emits a SolutionTrace — ordered
operation applications ending at a numeric answer, byte-deterministic
across runs, with each step's operation bound to a pack-resolved
lemma identifier.
What landed
generate/math_solver.py
- solve(graph) -> SolutionTrace; pure function, no I/O, no globals
- SolutionStep dataclass with before/after values per step (for
verifier replay; ADR-0117 hardens)
- SolutionTrace with canonical_bytes() byte-deterministic JSON
- SolveError typed refusal: missing pack, division by zero,
unknown-references-nothing
language_packs/data/en_arithmetic_v1/
- 5 operator lemmas: add / subtract / multiply / divide / transfer
- role=operational_base (vocabulary-only; no domain claim)
- SHA-256-anchored lexicon + glosses; manifest carries
provenance=adr-0116:operator_seed:2026-05-22
tests/test_math_solver.py — 109 cases pinning five invariants:
1. Phase 2 exit criterion: ≥ 0.80 on parser-correct dev set
(current: 50/50 = 1.00)
2. Determinism: two solves produce byte-equal trace
3. Trace replay reproduces answer_value (verifier rehearsal)
4. Typed refusal on under-determined inputs
5. Every step.pack_lemma_id resolves to a real lexicon entry
in en_arithmetic_v1
ADR-0114a obligation discharge
Four of ten anti-overfitting obligations now have load-bearing
implementations in code:
#3 replay-equal trace — discharged (solver-layer)
#4 typed refusal — discharged (solver-layer)
#9 determinism — discharged (solver-layer)
#10 operation provenance via pack — DISCHARGED IN FULL
Removing the en_arithmetic_v1 pack now breaks every solve loudly.
The "operations bind to concepts, not hardcoded strings" claim is
architecturally true, not rhetorical.
Tests: 109/109 green on solver suite; 67/67 smoke suite green;
parser + schema suites still green from prior phases.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Documentation-only amendment to ADR-0114. Locks in the 10-point
falsifiable proof framework that any future `expert` ledger-tier
promotion (ADR-0120+) MUST require.
The obligations:
1. Sealed-holdout discipline
2. OOD surface variation ≥ 0.95 of public
3. Every correct answer ships with replay-equal trace
4. Refusal is first-class; misparse rate zero; zero `wrong` answers
5. Reasoning-isolation perturbation suite (invariance + predictable change)
6. Compositional-depth curve flat within documented ε
7. Frontier-baseline comparison on identical items, published
8. Adversarial generation; misparse rate zero
9. Determinism across release boundaries
10. Operation provenance via the pack (not hardcoded strings)
Each obligation is load-bearing and falsifiable: a domain that
cannot satisfy any one stays at `audit-passed`. ADR-0114a binds
ADR-0116..ADR-0120 to carry the obligations into implementation;
ADR-0120 finally invokes all ten as hard gates.
The audit-passed tier (ADR-0106/0109/0113) is unaffected. The two
tiers measure orthogonal properties: audit-passed verifies CORE
claim-shape compliance (transformer-unreachable invariants); expert
verifies capability with anti-overfitting proof.
No code change. Pure forward contract for the next phase of work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First Phase of ADR-0114's expert-capability roadmap. Decomposed into four
sub-phases so each lands as its own auditable step:
1.1 schema + 5 seed cases + invariants ← this commit
1.2 45 more dev-set cases ← delegated (Codex)
1.3 the parser itself ← exit: ≥0.90 on dev set
1.4 runtime binding ← if non-trivial
What landed
- generate/math_problem_graph.py — typed dataclasses (Quantity,
InitialPossession, Operation, Unknown, MathProblemGraph) + frozen
validation + canonical_bytes() byte-deterministic serialization +
graph_from_dict roundtrip.
- evals/gsm8k_parser_dev/cases.jsonl — 5 seed cases (gpd-001..005)
covering single-add, single-subtract, multi-step, two-entity
transfer, and multi-entity sum constructions. Every case carries a
ground_truth_graph and the documented patterns it exercises.
- evals/gsm8k_parser_dev/README.md — authoring contract: schema,
pattern registry, canonicalization rules, Phase 1.1 scope boundary,
hand-solving rubric, distribution target for the remaining 45
cases. This is the spec Phase 1.2 authors work against.
- tests/test_math_problem_graph.py — 26 cases pinning four invariants:
round-trip byte equality, canonical_bytes() determinism, schema
rejection of malformed graphs, and ground_truth_graph ↔
expected_answer agreement (a hand-solver inside the test module
falsifies mis-authored cases).
Why this is sticky
The Phase 1.1 schema is load-bearing for Phase 1.2 (the 45 authored
cases will be written against it) AND Phase 1.3 (the parser will be
graded byte-equal against ground-truth graphs in this schema). Changing
the schema after Phase 1.2 lands requires an amendment ADR + rewriting
authored cases. The schema choices here are intentionally conservative.
Tests: 26/26 new; 67/67 smoke green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The word "expert" in the previous status name implied raw-capability parity
with frontier LLMs on the same benchmark — which the gate does NOT verify.
What the gate actually verifies is CORE *claim-shape compliance*:
* signed digest (replay-reproducible from on-disk lane results)
* replay determinism (same inputs → byte-equal trace_hash)
* typed refusal (fabrication refused, not paraphrased)
* exact recall (no ANN, no cosine, no attention bottleneck)
* grounding-source provenance
These are claim shapes a transformer LLM cannot structurally produce
regardless of raw accuracy. A frontier LLM might score higher on the
same benchmark but cannot pass this contract.
Rename scope (semantics only, per ADR-0113):
status string "expert-demo" → "audit-passed"
predicate key predicates.expert_demo → predicates.audit_passed
reason key expert_demo_reason → audit_passed_reason
YAML key expert_demo_claims → audit_passed_claims
CLI command core demo expert → core demo audit-passed
output dir evals/expert_demos/ → evals/audit_passed/
artifact filenames expert_demo.{json,html} → audit_passed.{json,html}
HTML title CORE Expert-Demo: X → CORE Audit-Passed: X
Internal Python identifiers (module/file/function/class names like
`expert_demo.py`, `evaluate_expert_demo`, `ExpertDemoClaim`,
`expert_demo_claim_for`) are deliberately kept to minimize churn. ADR
file titles (ADR-0106..0112) preserved as historical record.
`expert` namespace reserved for ADR-0114+: an actual capability tier
above `audit-passed` backed by a public benchmark with a stated
threshold. ADR-0114 proposes the first such target — GSM8K-math —
laying out a falsifiable 7-phase arc (parser → solver → verifier →
stepped-realizer → eval lane → first `expert` ledger tier promotion).
Tests: 184 directly-affected tests green (140 capability/expert-demo
suite + 34 demo/audit-tour + 10 correction-cue). Smoke suite 67/67.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes the asymmetry between the `expert-demo` ledger status (audit
artifact only) and the actual `core demo` surface (runnable
walkthroughs producing HTML + JSON). Until this commit the word
"demo" in `expert-demo` was aspirational; now it corresponds to
something a reader can open.
What it does
- Reads the signed expert_demo_claims entry from docs/reviewers.yaml
- Loads latest on-disk result files for each attached lane × split
- Re-derives the evidence-bundle digest and asserts byte-for-byte
match against the signed claim_digest — this is the load-bearing
audit step, now exercised at two independent enforcement points
(ledger gate + showcase)
- Runs each lane's metrics through the ADR-0109 lane-shape registry
and surfaces the verdict
- Picks the first three cases from each split verbatim (deterministic
by file order) and renders them as HTML for inspection
- Emits expert_demo.json (canonical bytes, deterministic) + expert_demo.html
Surface
core demo expert --domain mathematics_logic
core demo expert --domain physics
# → evals/expert_demos/<domain>/latest/expert_demo.{json,html}
Read-only by construction: cannot mutate docs/reviewers.yaml or any
lane result file. Tested. Unpromoted domains raise ValueError —
no silent fallback, no "preview" mode that fakes a showcase.
Generated artifacts are gitignored — the inputs they derive from are
already committed, so duplicating the renders would just churn the
tree.
Tests: 16 new cases pinning all five ADR-0112 invariants. Smoke suite
still 67/67 green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ADR-0111 landed via PR #123 (merge commit 4b5e6b7). The structural docs
(README, ADR index, reviewers.yaml, CLAIMS.md) were updated in that PR.
The narrative docs that reference the worked-promotion arc by ordinal
("first promotion / one demonstrated domain") needed a follow-up sweep:
- docs/PROGRESS.md — ADR-0100 footnote, ADR-0111 row, ledger table,
Open-within-Phase-6 changed from "second" to "third" promotion
- docs/runtime_contracts.md — narrative line on contract demonstration
- docs/Whitepaper.md §XIII — chain range, demonstration paragraph,
"two domains demonstrated" line, full-chain pointer
- docs/capability_roadmap.md — Phase 6 worked-promotion narrative step 5
and exit-criteria checklist (5th item now ☑, 6th open as "third")
docs/Yellowpaper.md is unchanged: its expert-demo section is the formal
contract spec, domain-agnostic. docs/eval_methodology.md unchanged for
the same reason. docs/sessions/SESSION-2026-05-22-contract-layer-arc.md
is left as the historical snapshot of the original session.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Second worked promotion exercising the ADR-0106 + ADR-0109 contract
on a domain distinct from mathematics_logic. No contract change.
Evidence:
- foundational_physics_ood: accuracy=1.0 (117/117 public, 39/39 holdout)
- inference_closure: all_pass_rate=1.0 (shared with math, distinct digest via domain_id)
- fabrication_control: refused=n, fabricated=0 across all classes (shared)
Signed claim digest: a104cad136f3219df05dc7ce6a78437c02f7b5827cd3cdce568db3acda6a43ed
Bridge landed: cases_plaintext.jsonl dev-mode fallback for
foundational_physics_ood (matches ADR-0105 convention; analogous to the
math/inference bridges in ADR-0110). One small file, not a contract change.
Tests:
- tests/test_adr_0111_physics_expert_demo.py — 4 invariants, 6 cases
- tests/test_adr_0110_math_expert_demo.py — relaxed "only math promoted"
to "math stays promoted" (load-bearing for ADR-0110 is persistence)
- tests/test_capability_reports.py — physics row now expert-demo
Retires the "first promotion was math-specific" objection: the bridges
ADR-0110 landed were correctly scoped, and the contract holds across
two distinct domains using shared lane infrastructure with distinct
digests.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes the documentation audit. All four lower-priority surfaces from
the earlier scan now reflect the post-ADR-0110 reality.
New:
- docs/sessions/SESSION-2026-05-22-contract-layer-arc.md: full
narrative of the ADR-0103 -> ADR-0110 arc, including the
refused/amended/succeeded narrative, infrastructure bridges, and
ledger state at session close. Pairs with the 2026-05-21
articulation-arc session log.
Extended (additive, no rewrites):
- docs/runtime_contracts.md: new 'Expert-Demo Promotion Contract
(ADR-0106 + ADR-0109)' section. Surfaces ledger_report() shape,
reviewer-yaml schema, lane-shape registry, replay invariant,
fail-closed registry behavior, and trust boundary.
- docs/eval_methodology.md: new 'Lane-shape registry (ADR-0109)'
section. Documents what to do when adding a new eval lane:
pick a shape or amend ADR-0109. Notes the holdout-runner gating
(cases_plaintext.jsonl vs sealed). Header bumped to 2026-05-22.
- docs/Whitepaper.md: new section XIII 'Evidence-Governed Domain
Layer' between XII (Forward Semantic Control) and the
Extensions/closing. Narrates the contract chain at the
philosophical / external-reader level. The original §XIII
becomes §XIV.
- docs/Yellowpaper.md: new section XII 'Ratification Contract
(ADR-0091 + ADR-0106 + ADR-0109)'. Formal specification of the
nine predicates, the promotion predicate (with explicit logical
form), the digest function, and the lane-shape registry table.
The original §XII becomes §XIII.
No code, no tests, no contract changes.
PROGRESS.md was last updated mid-Phase-5 (~ADR-0021). The pack-layer
chain (ADR-0027..0045), forward-graph+surface composer chain
(ADR-0046..0089), and evidence-governed domain layer (ADR-0091..0110)
were all absent.
capability_roadmap.md was last updated 2026-05-17 and predated the
expert-demo promotion contract entirely.
Added to both:
- Phase 6 — Evidence-Governed Domain Layer section covering ADR-0091
contract chain through ADR-0110 expert-demo arc, including the
refused-amended-succeeded narrative and the current ledger state
(math at expert-demo, four others at reasoning-capable).
- Backfill subsections for ADR-0027..0045 (pack-layer chain) and
ADR-0046..0089 (forward-graph + surface-composer + register +
anchor-lens + contemplation-loop work) marked retrospective.
capability_roadmap.md also gains a '2026-05-22' status update at the
top alongside the existing 2026-05-17 entry. The full Phase 0-5
exit-criteria rewrite remains queued separately.
No code, no tests, no claims changes.
- README.md: fix broken evals/CLAIMS.md link to root CLAIMS.md;
add 'Evidence-Governed Domain Layer' section pointing to the
ADR-0091/0096/0106/0109 contract chain and current ledger state
(math at expert-demo, four other domains at reasoning-capable).
- docs/decisions/README.md: extend the 'fully accepted' slate from
0091..0105 to 0091..0110 with ADR-0106 through 0110 entries; extend
the 'Evidence-governed domain chain' chain-notes section to reflect
the expert-demo arc (refused-amended-succeeded narrative).
Replaces the cognition-shape-uniform threshold dispatch in
core/capability/expert_demo.py with an explicit LANE_SHAPE_REGISTRY
mapping 8 ratified lane ids to 5 shapes:
cognition -> cognition_shape
elementary_math_ood -> accuracy_shape
foundational_physics_ood -> accuracy_shape
symbolic_logic -> symbolic_logic_shape
hebrew_fluency -> accuracy_shape
koine_greek_fluency -> accuracy_shape
inference_closure -> inference_shape
fabrication_control -> refusal_shape
Each shape has a documented threshold checker. Unknown lane ids
fail-closed with a named reason. ADR-0106 \xc2\xa71.1/\xc2\xa71.3/\xc2\xa71.4/\xc2\xa71.5
unchanged; only \xc2\xa71.2 (threshold rules) dispatches by shape.
tests/test_lane_shape_thresholds.py pins all four ADR-0109 invariants
plus dead-shape and threshold-value gates (13 new tests).
tests/test_expert_demo_contract.py fixtures updated to provide
shape-appropriate metrics (no semantic change to those tests; same
12 cases still pin the ADR-0106 contract).
ADR-0109 status: Proposed -> Accepted. README sequencing updated
(ADR-0110 now only blocked by inference_closure, not by metric-shape
amendment).
Ledger: all five domains remain reasoning-capable, expert_demo=false.
Amends ADR-0106 \xc2\xa71.2 to dispatch threshold rules by lane shape rather
than imposing cognition-pack-shape metrics uniformly. ADR-0107
surfaced that every non-cognition lane was failing the gate by
absence-of-key, not by substance.
Status: Proposed. Ships five shapes covering every lane currently
attached to a ratified pack: cognition_shape, accuracy_shape,
inference_shape, refusal_shape, symbolic_logic_shape. Four invariants
pinned. Unknown lanes fail closed; new shapes require ADR amendment.
\xc2\xa71.1 (reasoning-capable prereq), \xc2\xa71.3 (signature scoping), \xc2\xa71.4
(domain-aware), \xc2\xa71.5 (replay byte-equality) all preserved. ADR-0106
status remains Accepted.
The ADR-0106 contract correctly refused promotion. ADR-0107 records the
deferral and reserves two follow-up ADRs:
- ADR-0109 (lane-shape-aware threshold amendment): ADR-0106 \xc2\xa71.2
prescribes cognition-pack-shape metrics uniformly, but math /
physics / systems / hebrew-greek lanes carry native shapes
(accuracy, passed_rate, all_pass_rate). Prerequisite for any future
expert-demo promotion.
- ADR-0110 (math re-attempt): conditional on ADR-0109 landing and
inference_closure substantively passing (currently all_pass_rate=0.4
on public).
tests/test_adr_0107_deferral.py pins adr_0107_no_silent_promotion: math
stays at reasoning-capable, has no expert_demo_claims entry, and the
ledger row carries a named refusal reason.
No change to core/capability/expert_demo.py or reporting.py -- the
contract is honored, not amended. README sequencing updated to reflect
ADR-0107 acceptance and the new ADR-0109/0110 prerequisites.
Closes ADR-0106 acceptance evidence:
- ExpertDemoClaim dataclass + additive expert_demo_claims block on
ReviewerRegistry (schema_version stays at 1; backward-compatible).
- New core/capability/expert_demo.py with derive_evidence_digest,
evaluate_expert_demo, collect_domain_lanes, materialise_lane_results.
- core/capability/reporting.py: replaces the cognition-lane-only
predicate (previous lines 418-433) with a domain-aware,
reviewer-signed gate; ledger rows now also carry
expert_demo_reason for operator legibility. Reviewer registry is
fail-closed: an unloadable registry yields zero claims, so a broken
registry never silently grants expert_demo=true.
- tests/test_expert_demo_contract.py covers all three ADR-0106
invariants: requires_signature, domain_aware, replay_byte_equality;
plus threshold + production-ledger-untouched gates. 12 new tests.
- tests/test_reviewer_registry.py extended with TestExpertDemoClaimsSchema
covering omitted block, valid parse, unknown signer rejection,
malformed digest rejection, duplicate domain rejection. 5 new tests.
- README index row + table preface updated to note expert_demo is
contract-gated. Frontier list trimmed (ADR-0106 has landed).
- ADR-0106 Status flipped Proposed -> Accepted.
No domain row's expert_demo field flips by this PR -- only the contract
changes. Promotion of any ratified domain requires a follow-up ADR
(ADR-0107 reserved for mathematics_logic) plus a signed claim.
- ADR-0108 Status: Proposed -> Accepted
- README index row updated to Accepted
- 'Current frontier' rewritten with the ranked Proposed-ADR list mandated
by ADR-0108 \xc2\xa7Decision; removes the now-false 'No ADR currently sits in
a "Proposed but unimplemented" state' sentence
- Open candidate directions (no-ADR-yet) section retained for the
multi-reviewer governance frontier item from ADR-0105
Makes the post-ADR-0105 sequencing of ADR-0080 / 0084 / 0087 / 0106
explicit, durable, and revisable. Status: Proposed. No content of the
four sequenced ADRs is modified — sequencing is meta, not content.
Defines a domain-aware, reviewer-signed expert_demo promotion gate to
replace the current cognition-lane-only predicate in
core/capability/reporting.py:418. Status: Proposed. This ADR does not
promote any domain — it defines the contract that a follow-up ADR (likely
mathematics_logic as ADR-0107) will consume as the first worked
promotion.
Sibling reconciliation PR to #104. The four ADRs explicitly called out as
the 'current implementation frontier' in PR #104 are already implemented
to the same evidence bar as the eight ADRs that PR accepted:
- ADR-0094: teaching/source.py + proposal schema widening + migration
script; tests/test_proposal_source.py green
- ADR-0095: teaching/from_miner.py + miner_loop_closure lane;
SHA-pinned in scripts/verify_lane_shas.py; tests/test_miner_proposals.py
green
- ADR-0098: core/demos/contract.py + adapter surface + demo_composition
lane; SHA-pinned; tests/test_demo_composition.py green
- ADR-0099: core/demos/showcase.py + public_demo lane;
SHA-pinned; tests/test_public_showcase.py green
Three of four lanes are SHA-pinned in CI (a stricter bar than several
already-accepted ADRs). Local pytest run: 85/85 passed across the four
tests/test_*.py files in 17s.
Also refreshes docs/decisions/README.md:
- flips the four table rows to Accepted (2026-05-22)
- rewrites the 'Current frontier' section now that no ADR-0091..0102
entry is unimplemented
- enumerates candidate next directions (curriculum proposals,
language-specific holdout splits, expert-demo ratification)
Docs-only change; no runtime code touched.
Closes the load-bearing gap blocking every reasoning-capable claim
under ADR-0091: docs/reviewers.yaml was previously `reviewers: []` and
unparsed. Now schema-validated at v1, with a bootstrap shay-j entry
self-sealed via provenance.
- new core.capability.reviewers module: frozen Reviewer/ReviewerRegistry
dataclasses, strict load_reviewer_registry parser, ReviewerRegistryError
- enforces ADR-0092 schema rules: schema_version==1, no unknown
top-level keys, no unknown reviewer fields, role∈{primary,domain},
primary must claim ["*"], domain must NOT claim "*", review_scope
subset of {pack,proposal,chain,eval}, no duplicate reviewer_ids
- can_review(reviewer_id, domain_id, scope) helper implements
ADR-0092 rules 2-4 for downstream use by ADR-0093 validator
- docs/reviewers.yaml updated to v1 schema with shay-j bootstrap
- ledger_report() evidence_counts now exposes structured
reviewer_registry status (valid, schema_version, reviewer_count,
reviewer_ids, error) alongside the legacy reviewers_present bool
- new evals/reviewer_registry/ lane: 6 cases (2 positive + 4 negative)
covering empty-registry, wrong-version, domain-wildcard rejection,
and unknown-field rejection
- runner emits deterministic JSON report; two runs produce byte-identical
output (sha256 verified)
- 26 unit tests in tests/test_reviewer_registry.py
- capability ledger test extended to assert new reviewer_registry block
- smoke suite green (67/67); lane passes 6/6
The pre-existing test_flag_report_tracks_default_off_flags failure is
unrelated (discourse_planner flag default) and not introduced here.
Eight load-bearing ADRs closing the loop from contemplation Phase 5 through
a public showcase demo. Each one is small and evidence-bearing; together
they sequence the next arc without duplicating existing substrate.
- 0092 Reviewer Registry v1 — populates docs/reviewers.yaml schema;
unblocks all reasoning-capable claims under ADR-0091.
- 0093 Domain Pack Contract v1 Implementation — wires ADR-0091's five
follow-up items (parser, dry-run validator, chain registry, eval lane
refs, reviewer resolution) so manifest fields actually gate status.
- 0094 Proposal Source Provenance — sealed ProposalSource type widening
proposal schemas ahead of 0095.
- 0095 Miner-Sourced Teaching Proposals — closes the contemplation
loop: articulation_quality / contradiction_detection / frontier_compare
miners emit PackMutationProposal candidates routed through the single
reviewed teaching path; identity-pack defense at construction, not
review; replay-equivalence pre-gate.
- 0096 Fabrication-Control Eval Lane — first negative-control measure;
three case classes (phantom endpoint, cross-pack non-bridge, sibling
collapse) with frozen thresholds (fabrication_rate ≤ 0.01).
- 0097 Mathematics-Logic Reasoning-Capable Ratification — first
domain claim under ADR-0091; chain corpus + eval lanes already
exist, this is the formal contract ratification.
- 0098 Demo Composition Contract — DemoCommand protocol so demos can
be composed without reimplementation; deterministic JSON, no global
state mutation, declared output paths only.
- 0099 Public Showcase Demo — composes four scenes (determinism /
honest unknown / reviewed learning / multi-hop+trace) under 30s;
pure composition enforced by grep gate; JSON byte-equality CI-pinned.
Landing order: 0092 → 0094 → 0095 → 0093 → 0096 → 0097 → 0098 → 0099.
Deliberately not included: curriculum compiler, formation course
runner, calculator operators, response-mode taxonomy expansion,
learning-scale 10k harness. Each is deferred with a documented reason.
Phase 5 landed in commit 327047c (articulation-quality miner +
runtime sink wiring + full end-to-end loop tests). Extending the
session note in-place per its own append-only convention so the
single document covers the complete arc Phases 1–5.
Sections updated
----------------
* §0 executive summary
- commit count: 9 → 11
- phases shipped: 4 → 5 (new Phase 5 row in the deliverables
table)
- observation surfaces table grows two rows for
``attach_articulation_sink`` and
``mine_articulation_observations``
- test artifacts: 6 → 8 files, 64 → 82 cases
* §3.5 NEW — full architectural walkthrough of Phase 5
- Why the loop closes (links the user's
"memory confidence scoring" intuition to ADR-0080's
doctrine-aligned realisation: reviewable evidence, not
autonomous mutation)
- File-by-file delta (chat/articulation_telemetry.py + miner
+ runtime wiring)
- Three v1 mining rules (recurring_predicate_monotony /
recurring_planner_gap / low_average_predicate_diversity)
- Loop diagram showing live + offline halves
- Recorded demo output from the commit message
- Doctrine pin table mapping each constraint to its test
* §4 pipeline diagram extended to show Phase 5 sink + offline
miner branches
* §5 verification table gains four new rows for Phase 5 claims
(full-loop emission, byte-equal finding IDs across two e2e
runs, JSONL round-trip identity, opt-in gating)
* §5.6 suite totals updated:
- Contemplation subsuite: 35/35 → 53/53 (Phase 3+4+5)
- New row for Phase 5 articulation-quality e2e (7/7)
* §6 case study — added the "After Phase 5" trace and the
closing one-line story across all five phases for one prompt
("What is truth, and why does it matter?")
* §7 architecture surfaces table grows a row for
chat/articulation_telemetry.py and adds the miner to the
contemplation subsystem row
* §9 inverted from "what would close the loop" (future work) to
"SHIPPED — here's what it now unlocks":
- production sink + retention policy
- additional aggregation rules
- CLI hook
- review-loop wiring back into PackMutationProposal
* §10 reference index grows new lines for
chat/articulation_telemetry.py,
core/contemplation/miners/articulation_quality.py,
tests/test_articulation_quality_miner.py, and
tests/test_articulation_quality_e2e.py
* Footer updated: notes the arc is complete; future arcs should
start a new session-notes file and cross-link rather than
rewriting this one.
The session-notes file is now 1100+ lines — the complete frozen
reference for the articulation arc that took CORE from one-sentence
pack-grounded surfaces to a full live-reasoning + offline-mining
+ reviewable-proposals feedback loop, all doctrine-aligned, all in
one session.
Thorough why/when/where/how reference for the four-phase articulation
arc shipped this session plus the pre-arc classifier/Rust cleanup
that made it possible. Designed as the load-bearing entry point for
future contributors, case studies, capability audits, and
architectural reviews.
Sections
--------
0. Executive summary — what was achieved (9 commits, 64 new tests,
user-visible before/after, doctrine evidence)
1. Why this work happened (visible gap, doctrine constraint, what
was already wired vs. missing)
2. Pre-arc cleanup — RECALL trigger / CORRECTION x2 / Rust FFI
3. Phase 1 — discourse planner default ON + fast-path
Phase 2 — reflective rendering (subject pronominalization)
Phase 3 — live plan contemplation pre-flight
Phase 4 — per-plan articulation telemetry metrics
4. The pipeline today (diagram + before/after table)
5. Verification — every claim and the test that holds it.
Five sub-tables: doctrine claims, quality claims, back-compat /
null-lift claims, ADR-0072 structural invariants,
`core bench --suite all` performance, suite-level totals.
6. Case study — the compound prompt as a story across all four
phases
7. Architecture surfaces touched (which file got what change in
which phase)
8. What was deliberately NOT built (and why — connective rotation,
generalised pronoun selection, plan revision, Phase 2.5,
Rust algorithmic optimisation)
9. Phase 5 — what would close the user-intuited "live reasoning
→ memory confidence" loop, doctrine-aligned
10. Reference index — modules, flags, tests, ADR cross-refs
Per the user's request: captures the why/when/where/how thoroughly
so this work is recoverable for future reference, case studies, and
building on top. Append-only convention: future sessions extending
this arc should add new sections below rather than rewriting — the
history is itself evidence of the doctrine working in practice.
Brief 1 scopes the PR #97 follow-up: flip `definitional_layer` to false on
the three new domain seed packs (en_mathematics_logic_v1 / en_physics_v1 /
en_systems_software_v1) so the ADR-0084 closure verifier stops flagging
them as malformed. They're governed by ADR-0091 `domain_contract_version`,
which is a different layer; the integration test's allowlist already
excludes them, so the flag-flip aligns the script with the test's
clearly-intended scope.
Brief 2 scopes ADR-0073 L1.1 content phase II: extend the grc/he anchor-lens
substrate with 8 cross-language lemma pairs (νοῦς ↔ בינה / διάνοια /
καρδία ↔ לב / ψυχή ↔ נפש / ἔλεος / εἰρήνη / δικαιοσύνη / ἅγιος ↔ קדוש)
authored against the L1.1 file shapes, alignment edge weights, and
en-collapse annotation discipline. Anchor-lens-tour seam claims and
cognition-eval byte-identity under unanchored default remain the gates.
Both briefs are bounded to content authoring; neither touches code.
Closes audit Findings 6 (within-turn recall not batched) and 7
(probe-ingest / commit-ingest dual field) as a single PR — the two
are architecturally entangled and resolve together.
Pre-fix flow in ``ChatRuntime.chat()``:
1. ``probe_ingest(filtered)`` → ``probe_state.F``
2. Gate check on ``probe_state.F``
3. If gate fires: ``commit_ingest`` + stub response
4. Otherwise: ``commit_ingest`` + drive bias → ``field_state.F``
5. Walk runs on ``field_state.F``
The gate observes one manifold position; the walk navigates a
slightly different one (drive bias applied between them). Honest
refusal decisions and walk outputs are made on different fields —
the audit's named coherence gap.
This PR ships a flag-gated unified-ingest path following the
codebase's standard substantive-change pattern (ADR-0046 /
ADR-0062 / ADR-0085 / ADR-0088 / ADR-0089):
``RuntimeConfig.unified_ingest: bool = False`` (default).
When ``True``:
1. ``commit_ingest(filtered)`` runs first.
2. Drive bias applied immediately.
3. Gate observes ``committed.F``.
4. If gate fires: stub response (turn has already committed —
intentional semantic change documented in ADR-0090).
5. Otherwise: walk runs on the same ``committed.F`` the gate
decided against — no second ``commit_ingest`` call.
6. ``probe_ingest`` is not called on this path.
When ``False`` (default): historical behavior is preserved
bit-for-bit; ``probe_ingest`` still runs first.
ADR-0090 documents:
* Phase 1 (this PR): unified-ingest substrate.
* Phase 2 (separate PR, after Phase 1 validates): batched recall
— pass the gate's ``direct_hits`` into ``generate()`` as a
``prebuilt_first_recall`` so the walk's first step does not
re-call ``vault.recall()`` on the same field. Single recall
call eliminated per turn.
* Out of scope: ``recall_batch`` for per-step walk recalls
(each step's query depends on the previous step's field
state; not batchable without changing walk geometry).
Validation:
* 5 new tests in ``tests/test_unified_ingest_null_lift.py``:
- flag defaults to ``False`` on ``DEFAULT_CONFIG``
- flag-off surface + trace_hash + vault_hits byte-identical
- flag-on does not call ``probe_ingest`` (verified via spy)
- flag-on produces well-formed surface + trace_hash
- flag-off still calls ``probe_ingest`` (historical guard)
* ``core eval cognition`` byte-identical across all three splits:
public 100/100/91.7/100, dev 100/100/78.6/100, holdout
100/100/83.3/100.
* ``core test --suite cognition`` 120/0/1, ``smoke`` 67/0,
``runtime`` 19/0.
Comb-pass status after this PR:
* Item 4 (graph topo) ✓ #92
* Item 5 (realizer node_map) ✓ #91
* Item 6 (batch recall) ✓ ADR-0090 substrate (this PR); Phase 2
optimization is queued
* Item 7 (probe/commit dual ingest) ✓ ADR-0090 (this PR)
* Item 8 (dead defensiveness sweep) ✓ #91
* Item 9 (local imports) ✓ #91
* Item 11 (dead ``_fold_compose_into_surface``) ✓ #91
* Item 13 (``_serialize_*`` fold) ✓ #91
* Item 15 (GenerationResult tuple/list) ⊘ false positive
* Item 16 (subject normalization consistency) ✓ #93
* Item 17 (redundant ``^`` anchors) ✓ #94
* Tier 5 minor (``_BE_FORMS`` hoist, walrus, reverse-iter) ✓ #94
Comprehensive survey of every pack in the tree across five layers
(primitives → language → teaching → policy → selection axes → style)
with per-pack stats, cross-pack lemma overlap, teaching-chain graph
topology, and a ranked top-leverage-gaps list.
Key findings:
* **Layer 4 (selection axes) is the densest by far** — 24 packs
(17 anchor lenses + 7 registers). Greek and Hebrew lens
families are at parity (8 each).
* **Layer 2 (teaching corpora) is the load-bearing thin layer** —
only 41 reviewed chains across 18 subjects, with just 2 intent
shapes covered (CAUSE / VERIFICATION) and 7 connectives total.
No COMPARISON / PROCEDURE / CORRECTION chains exist; those
intents route through pack composers only.
* **Layer 1 EN is solid** — 354 lemmas across 12 mounted packs
with 93% gloss coverage; 24 ungloss'd lemmas remain.
* **Cross-language is structurally present but content-light** —
grc / he micro-packs ship and mount; the bigger
``*_cognition_v1`` siblings ratify but are not in the default
mount; no glosses exist on any non-EN pack.
* **Three drafted ethics packs are unratified** (legal /
research / engineering) — the highest-leverage low-effort gap.
* **Rhetorical-style axis ships substrate only** — one
null-lift pack; ADR-0087 consumer phase pending.
Top-leverage gaps ordered by ratio of unblocked value to effort
are enumerated in §7 of the doc itself. No code lands here.
Two ADRs that unblock the remaining items from the 2026-05-20 audit
that could not ship as direct PRs.
ADR-0088 — Realizer-Grounded Authority (Finding 2 retry)
========================================================
The first-response audit remedy (wire ``ground_graph`` between
``runtime.chat`` and ``realize_semantic``) was empirically attempted
on ``fix/ground-graph-wiring`` and reverted: the grounded realizer's
template output (e.g. ``"Light is a visible medium that reveal
truth"``) is grammatically and stylistically weaker than the runtime
path's ADR-0085-polished pack-grounded surface, so the realizer wins
the surface resolver (PR #76) and the user-visible surface regresses
on 23 byte-identical tests + ``register_invariant_grounding``.
ADR-0088 reframes Finding 2 as a two-phase rollout:
* Phase A (no behavior change) — realizer fluency parity.
Templates consult the same gloss source ADR-0085 wired into the
CAUSE composer, emit 3sg verb agreement, and carry the same
pack-provenance tag the runtime path emits. Byte-identical
today because the realizer is still gated by
``_is_useful_surface``.
* Phase B (substantive) — ground the graph and let the realizer
compete. Surfaces change exactly once, with a per-case
re-baseline justified by a "fluency ≥ pre-fix runtime surface"
invariant.
The audit's final-draft remedy (hot-path short-circuit only) is
explicitly rejected — pure perf cleanup, no metric lift since
``core eval cognition`` is already at 100% groundedness.
ADR-0089 — Compound-Intent Pipeline Dispatch (Finding 4)
========================================================
``classify_compound_intent`` is implemented but never reaches
``CognitiveTurnPipeline.run()``. Compound inputs like *"What is X
and how does it relate to Y?"* silently drop the second clause.
Naive multi-node dispatch breaks every downstream stage:
PropositionGraph (one root), plan_articulation (single-root),
realize_semantic (one target), surface resolver (one surface per
turn), compute_trace_hash (one intent_tag + articulation_surface),
teaching loop (one correction-source proposal), register / anchor-
lens telemetry (one variant per turn).
ADR-0089 proposes a three-phase rollout:
* Phase C1 (no behavior change) — call ``classify_compound_intent``
in step 1b, record dropped clauses on
``CognitiveTurnResult.dropped_compound_clauses`` for
observability while routing the dominant clause through the
existing single-intent path.
* Phase C2 (opt-in substantive) — flag-gated multi-node graph
dispatch with new ``CompoundEdge`` / ``ConjunctionRelation``,
widened ``compute_trace_hash`` carrying a
``compound_clauses_hash``, and a ``multi_clause_surface`` field
on the resolver. Flag-off preserves byte-identity.
* Phase C3 — telemetry alignment + demo + docs.
Each phase is independently shippable and preserves the existing
null-lift / byte-identity invariants register and anchor-lens
established (ADR-0072, ADR-0073d) as the project's pattern for
substantive runtime behavior changes.
Both ADRs are Proposed; ratification follows the existing pack /
ADR review process. No code lands in this commit.
Follow-up brief for the cheaper dev agent. Scoped tight: ~18 concrete
row-by-row edits across two patterns the v1 pass (PR #73) deferred or
missed.
Pattern A — 3sg present-tense agreement after relative pronouns
what/who/that/which followed by a bare-form verb where the implied
subject is singular. 10 candidates identified up front by repo scan:
- en_core_causation_v1/effect (with judgment note)
- en_core_cognition_v1/beginning, creation, definition, evidence,
light, reason, symbol
- en_core_meta_v1/example, mind
Brief explicitly clarifies that bare verbs after modals (`can`,
`may`, `should`) are CORRECT and must NOT be changed — flagging
the "who can know and do" case the v1 agent did right.
Pattern B — plural agreement after quantifier / preposition
between/among/of/two/three/many + count-noun-in-singular. 5
confident edits + 3 borderline cases with judgment guidance:
- en_core_attitude_v1/broad
- en_core_cognition_v1/context, order, style
- en_core_spatial_v1/between
Borderline guidance distinguishes count vs mass nouns: `reason`
in "group of reason" is count (apply fix); `reason` in "because
of reason" is mass (leave alone).
Same hard rules as v1 brief:
- no code edits
- definitional_atoms / predicates_invited / pos / lemma /
definition_version must not change
- Greek/Hebrew packs and primitives pack untouched
- closure verifier must exit 0
- cognition eval must stay byte-identical
- draft PR, human review before merge
Estimated effort: ~18 one-character edits. Whole pass should take an
order of magnitude less time than v1 because the candidate rows are
enumerated in the brief itself rather than discovered via heuristic
scan.
Why a v2 brief rather than amending v1's PR:
Plural patterns were not on v1's explicit pattern list (the v1
brief only named verb agreement, missing articles, missing
infinitives, missing copulas). 3sg agreement was named but
required a different tool than the agent had at hand. Scoping
v2 to exactly the rows known to need fixing is cheaper than
re-running the v1 heuristic scan with an expanded ruleset.
Provenance tag for the v2 pass:
adr-0085-style-v2:reviewed:2026-05-22
