Load-bearing strategic record: what the GSM8K chase got wrong (overfit
50-case ruler hiding a sealed wrong=0 breach), the held-out measurement
that exposed real capability=0% / composer 17% wrong, why GSM8K is the
wrong terrain for a deterministic verifiable engine, and the deductive-
logic path now (Phase 1 done: 500/500 held-out wrong=0, independent-
oracle verified) with phases 2-4 and the anti-recurrence disciplines.
The first SIZEABLE, honestly-verified reasoning capability — built on CORE's
own terrain (exact, verifiable, deterministic), not GSM8K's stochastic terrain.
THE OPERATOR (generate/proof_chain/entail.py, ADR-0206):
- evaluate_entailment(premises, query) -> entailed | refuted | unknown | refused.
- The multi-hop inference operator evals/symbolic_logic/gaps.md said did not exist
("no operator that takes A->B, B->C and returns A->C") and ADR-0205 deferred.
- Built on the ADR-0201 ROBDD canonicalizer: premises |= Q iff (AND P) -> Q is a
tautology. SOUND AND COMPLETE for propositional logic, not single-step.
- wrong=0 is structural: an exact tautology check refuses (LogicError) on
malformed / out-of-decidable-regime (quantified/predicate) input, never guesses.
THE HONEST METRIC (evals/deductive_logic/):
- holdout v1 (500 cases): 500 correct / 0 WRONG, incl. 227 non-trivial deductions
(117 entailed + 110 refuted). dev (200): 200/0.
- Gold from an INDEPENDENT truth-table oracle (oracle.py) sharing zero code with
the engine. 8,000-case fuzz across two independent decision procedures:
0 disagreements. This is the soundness evidence the GSM8K composer could never
produce (it could not separate its 2 right from its 87 wrong answers).
- contract.md states the load-bearing honesty boundary: PROPOSITIONAL ONLY, and
given-formulas (NL->logic grounding is a separate later layer, kept out of scope
so we do not re-step on the GSM8K natural-language rake).
TESTS (17, all green): classic inference shapes (MP, multi-hop chain, modus
tollens, disjunctive/hypothetical syllogism, conjunctive rules, genuine unknown),
refusal boundary (inconsistent / quantified / predicate / malformed), and a
deterministic engine-vs-oracle fuzz cross-check.
Pure new module — does NOT touch serving. Smoke 73 passed; invariants 40 passed.
Four genuine build-and-measure attempts on held-out, all falsified:
- composer 87 wrong, narrow reader 61 wrong, strict sum-reader 2 wrong (100%),
candidate-graph 0 admissible. Even 'exactly 2 numbers + total cue' cases are
multi-step in disguise. No sound committing strategy exists in this substrate.
All three GSM8K paths exhausted on the held-out 500:
- candidate-graph (sound filter): 0 admissible candidates built (498/500 build nothing)
- composer (open): 87 wrong
- narrow reader (open): 61 wrong
The sound path is wrong=0 only because it constructs no reading to commit. No sound
lift exists in the current substrate; confirmed empirically, three ways.
Built and measured two committing readers on held-out (not just diagnosed):
- resolve_pooled composer: 2 correct / 87 wrong (17%)
- maximally-narrow forced reader: 0 correct / 61 wrong (100% when it fires)
Shallow committing CANNOT be sound on real GSM8K (measured). The impasse:
- candidate-graph path is SOUND (wrong=0) but FROZEN by ADR-0207 §4, covers 0 real
- composer is OPEN (ADR-0207 §5) but UNSOUND (17% wrong, no separating gate)
ADR-0207's premise (composer is the wrong=0-safe path) is FALSIFIED by held-out data;
'feed the composer' is a path to more confabulation. Needs a follow-up ADR.
The result that should have been visible for weeks. Measured on the new holdout_dev
lane (500 real held-out cases) + sealed test:
- real capability 0% (train_sample's 4 correct generalize to 0/500 held-out)
- resolve_pooled (the composition substrate) commits 17.4% WRONG on held-out
- NO structural gate (step/pool/op) separates its 2 correct from 87 wrong -> the
composer cannot tell right readings from wrong; gating it is overfitting
- the only wrong=0 policy on real GSM8K is refusal (current serving 0/0/500)
Real lift needs a verifiable reading mechanism, not composer tuning. Non-serving (docs).
The 2026-06-04 sealed-breach post-mortem proved the 50-case train_sample has ZERO
predictive validity (its 4 "correct" are overfit; they hid a 5-wrong sealed breach).
This adds the instrument we never had: 500 real GSM8K cases CORE was NOT built on —
the train split minus the 50 train_sample, deterministic sha256(question) sort.
Same scorer as train_sample + the sealed lane, so the three are directly comparable:
train_sample(50): 4/0/46 holdout_dev(500): 0/0/500 sealed test(1319): 0/0/1319
Real GSM8K capability is 0%. The 4 train "correct" generalize to NOT ONE of 500
held-out cases. wrong=0 holds (refuses, never confabulates).
- evals/gsm8k_math/holdout_dev/v1/: cases.jsonl (500), runner, report (0/0/500), README.
- tests/test_holdout_dev_lane.py: floor (wrong==0, forever) + baseline snapshot (0/500).
Discipline: iterate here (open, large enough to resist trivial overfit); the sealed
test stays the final arbiter. wrong=0 is the floor; correct rising is the goal;
"refuse everything" is the FAILING baseline to beat, not a pass. Non-serving (eval only).
The FIRST real sealed measurement (operator-decrypted 1,319 held-out GSM8K)
found `0 correct / 5 WRONG` — a wrong=0 breach hidden for weeks because the
working metric was the 50-case train sample the bridges were tuned to. Bisection
isolated it to the product_bridge serving promotion (ADR-0195).
- generate/math_candidate_graph.py: REMOVE both serving promotion bridges
(product_bridge + goal_residual/ADR-0207 §5 step 2). Serving = main-graph-only.
Restores sealed 0/0/1319 (verified by bisect: disabling product_bridge -> 0 wrong).
Production modules remain in generate/derivation/; only serving promotion is
unwired, until a gate is proven wrong=0 on the SEALED set (never the train sample).
- Honest numbers everywhere: train_sample 7/43/0 -> 4/46/0 (the bridges' "correct"
was train-overfit). report.json + coverage probe regenerated. 7 ADR test lanes
de-pinned from the inflated count. corpus: cv-0005 (R4) reverts to refuse; cv-0020
(a "baseline control" that solved ONLY via product_bridge) reclassified.
- docs/claims_ledger.md: dated wrong=0-breach-and-remediation note + the rule:
the train_sample number had ZERO predictive validity for the exam; never the score.
- docs/analysis/gsm8k-lift-program-strategy: the program to actually move the 1,319.
NOTE the exit gate stays `correct>=10 AND wrong==0` — refusing-everything is an
explicit FAIL, not a wrong=0 pass; serving still commits (main graph). Verified:
broad regression 848 passed, smoke 73 passed.
Authority-ratified per the ADR-0207 convention (Accepted (ratified DATE)):
- ADR-0181 (audio), ADR-0197 (vision), ADR-0208 (environmental loop),
ADR-0209 (sensorimotor afferent) -> Accepted (ratified 2026-06-03).
All four are implemented + test-backed (0208/0209 falsifiability deeply
verified in the lookback; 0181/0197 backed by CRDT-merge/compiler/mount/
eval-gate suites, all green in the 182-test sweep).
- ADR-0198 deliberately NOT ratified: partially-implemented spike (Gap A +
fail-closed efferent gate landed; \xc2\xa73 verdict-lowering + motor decoder
deferred behind a dedicated motor governance ADR).
CLAUDE.md Work Sequencing now records the sensorium/modalities arc as a
sanctioned parallel track that does not displace the near-term sequence and
is disjoint from the GSM8K serving path. Markdown only; no test asserts ADR
status or the work-sequencing text.
Wires the R4 goal-residual production to serving via
resolve_promotable_goal_residual (math_candidate_graph.py, mirroring
product_bridge). cv-0005 / train_sample 0037 now solves on serving as
goal - Σprogress = 10 - 3 - 4 = 3. First Phase-5b composition lift on serving.
wrong=0 preserved on every runnable surface:
- train_sample 6/44/0 -> 7/43/0 (0037 added; 6 prior correct intact; wrong=0).
- Fires on 2/455 visible GSM8K cases, both correct, ZERO wrong.
- Gain-goal divergence firewall proves it reads the GOAL, not a possession.
- smoke 73, math+invariants 53, derivation/pool/practice 341, corpus, all green.
Lockstep updates (the ratified metric move, 6/44/0 -> 7/43/0):
- report.json; 7 ADR test lanes that pinned 6/44/0; corpus cv-0005 baseline
fields + snapshot (4/18 -> 5/17) + contract; plan-doc cv-0018 control fix.
⚠ SEALED MEASUREMENT REQUIRED — NOT DONE. The sealed 1,319 (encrypted, not
CI-reproducible) is the real bar (ADR-0207 §6) and was NOT re-measured. The
operator/CI must decrypt+run it and confirm sealed wrong==0; if wrong>0, revert
the resolve_promotable_goal_residual block (isolated). See
docs/handoff/sealed-measurement-obligation-2026-06-04.md.
DefaultEfferentGate is a capability/shape pre-filter only; it does not
lower decoded actions into safety/ethics pack verdicts (ADR-0198 §3 /
§1.2 Gap B). ModalityRegistry.decode/decode_batch now refuse fail-closed
any emission through a gate whose enforces_action_verdicts is False,
unless an explicit allow_unverified_efferent sandbox opt-in is set.
A real motor decoder thus cannot emit through the capability-only gate;
the §3 verdict-lowering gate must be built and installed first. No
production caller of the decode path exists today, so this closes the
latent hazard before a motor decoder makes it load-bearing. Adds two
falsifiable tests (fail-closed refusal; verdict-enforcing gate allowed).
Disjoint from the GSM8K serving path.
ADR-0181/0197/0208/0209 were marked Proposed despite test-backed
afferent implementations merged to main; flip to Accepted with
verifiable Implementation lines (paths + PRs + falsifiable tests).
ADR-0198's 'no implementation' status was stale (Gap A protocol
change + baseline efferent gate landed in #541); correct it and
document the deferred §3 safety/ethics verdict-lowering as an
explicit blocking obligation before any motor decoder mounts.
First composition lift-target built end-to-end: cv-0005 (train_sample 0037)
now resolves in the sealed pool as goal - Σprogress = 10 - 3 - 4 = 3.
- generate/derivation/goal_residual.py: new R4 production. Reads a GOAL anchor
(goal-intent lexeme) + a residual question, subtracts each same-referent
progress quantity (progress reduces the residual regardless of world-polarity).
Gated by the unchanged self-verification gate.
- wrong=0 firewall (test_reads_goal_not_possession): on a gain goal the
goal-residual (20-5-6=9) DIVERGES from possession-accumulation (20+5+6=31);
the production gives 9 and is all-subtract -> it reads the goal, not the
possession. This is the coincidental-correctness trap cv-0005 alone hides
(10-3-4 == 10-(3+4)).
- pool.py: goal_residual added to pooled_candidates (sealed). Verified: fires on
exactly one train_sample case (0037, correct), zero new pool wrong-commits
(the 8 are pre-existing, gated off serving by product_bridge).
Does NOT move the serving metric: train_sample stays 6/44/0 byte-identical
(serving = candidate-graph; product_bridge promotes only pure products, never a
subtract chain). The serving promotion gate for goal-residual is the next,
separately-gated step (needs the sealed 1,319 verdict). Smoke 73, math 4,
derivation/pool/practice 196, corpus, completeness-guard all green.
Three reviewed, non-serving analysis/handoff docs for the COMPOSITION lever.
Read-only; no serving code; train_sample 6/44/0 untouched.
- analysis/wiring-promotion-gate-brief: product_bridge is the only serving
derivation tendril (two-token target whitelist); resolve_pooled refuses all
ten R1/R4/R5/R6 positives and wrong-commits 0016 -> 510. Wiring is trivial;
the gate is the work.
- analysis/composition-wall-execution-plan: per-case stage taxonomy (A/B/C/D)
for all 15 corpus positives, reproduced live; R4-first scope (new
goal-residual production, verified by code-read of accumulate.py, NOT a
target tweak); promotion gate generalizes product_bridge, never disagreement
alone; gates inherit ADR-0207 §6 (corpus + sealed 1,319).
- handoff/stage-c-composition-investigation: dispatch task for the execution
lane to pin the per-case bail for the production-wrong shapes (the real
grouping/op-order wall), read-only, in-tree.
Reading the actual gate logic (product_bridge/pool/verify) corrected §5's
sequencing optimism, before merge:
- The safe wire is already done (product_bridge at :530); its safety is a
two-token target whitelist (money+make/earn, weight+total/move) hardcoded to
0003/0021 — it generalizes nothing.
- Verified live (2026-06-03): the general composer resolve_pooled refuses ALL
ten R1/R4/R5/R6 corpus positives (R1/R4 build zero candidates; R5 only wrong
ones) -> wiring yields +0 correct; and it wrong-commits 0016 -> 510, so wiring
it wholesale is a live wrong=0 regression the disagreement rule does not catch.
So WIRING and COMPOSITION are the same task: composer production per shape, each
behind a structural promotion gate (extract_target + target_units, not
disagreement alone); the wire is the trivial last step. Core ratification
(freeze, design-of-record, gates, sealed-set measurement) unchanged.
The composition_validation/v1 corpus shipped as data + a prose contract
with no executing test -- decoration until a test can fail (CLAUDE.md,
Schema-Defined Proof Obligations). Add the load-bearing gate:
- forever-invariants: wrong=0 firewall (admit => ==gold; null-gold =>
refuse), baseline-control regression net, permanent-refusal permanence,
and frozen baseline-field/tree match for the non-positive rows.
- current snapshot: aggregate 4 solve / 16 refuse / 0 wrong -- the single
assertion a Phase 5b slice updates when it flips a positive.
Future positives (5b-* gates) are checked by the firewall only, so a
refuse->solve flip at 5b stays green without rewriting frozen rows.
Verified: 44 passed; falsified (corrupting a control's gold fails the
firewall + control + snapshot tests); completeness guard still 21 passed.
Phase 5a (retire the inert GSM8K scoring-path comprehension-reader
dispatch, -1,038 LOC) landed in 3fd3172, an ancestor of HEAD 8327c6b,
but composition-capability-scope.md still listed it as remaining work --
drift that mis-scoped a downstream brief. Re-anchor the canonical doc to
the tree:
- §1: the lifecycle.py GSM8K-scoring dispatch row now reads "retired in
Phase 5a (3fd3172)" instead of "inert ... retirable in 5a".
- §2: Phase 5a marked SHIPPED with a runway-status banner; the §9
completeness-gate precondition marked LANDED (PR #534, e1bcdf6).
- §4: sequencing restructured -- items 0 (§9 precondition), 1 (gating
analysis), 3 (5a) are DONE; the live runway starts at item 2 (the
<=20-case validation sub-corpus), then Phase 5b.
- §9: status banner noting the recommended guard landed (#534); the
"RED on main" statements are the pre-landing measurement, kept for
provenance.
Docs-only. No code/eval/serving change. Baseline re-verified at HEAD:
train_sample 6/44/0 (all 44 refusals at branches_enumerated=0),
completeness-guard suite 21 passed.
CLAUDE.md's GSM8K substrate header asserted "serving stays `3/47/0` until
ratified". That figure is two ratified metric-moves stale: the official
candidate-graph serving metric (evals/gsm8k_math/train_sample/v1/report.json,
scored by `_score_one_candidate_graph` in train_sample/v1/runner.py:88) moved
3/47/0 → 4/46/0 (#488, ADR-0189/0189a) → 6/44/0 (#500, ADR-0195), each PR
re-baselining report.json.
Verified on main 7e98ad3: committed report.json == live `build_report()` ==
{correct: 6, refused: 44, wrong: 0}. wrong==0 preserved.
The stale header was the source of recurring 3/47/0 confusion across sessions.
Historical ADRs (0164/0166/0174/0175/0176/0177/0178/0179) that say "serving
stays 3/47/0" are point-in-time decision records and are intentionally left
unchanged.
