# The pivot to deductive logic — what went wrong, and the path now This is the load-bearing record of a strategic correction. It documents (1) the GSM8K capability chase and exactly how it misled us for weeks, (2) the measurement that exposed the truth, (3) why the terrain was wrong for CORE, and (4) the deductive-logic direction we are now on, with the plan and the disciplines that keep us honest. Read this before resuming capability work. --- ## 1. What we were doing, and what went wrong **The goal:** lift CORE's GSM8K (grade-school math word problem) accuracy off zero while holding the prime directive `wrong = 0` (refuse rather than confabulate). **The mistake — a ruler that could not measure.** Development was scored on `train_sample` — the **50 cases the grammar was hand-built against**. Tuning against those 50 produced "lift" with **no predictive validity**: it was overfit to the exact problems being optimised. Worse, it hid a real defect. **The breach.** The first real **sealed** measurement (the full 1,319-case GSM8K test split, decrypted and run by the operator, never read by Claude) showed serving was **`0 correct / 5 WRONG / 1314 refused`** — a live `wrong = 0` **breach**. The five wrong answers were `product_bridge` promoting the derivation composer's confabulations to serving. This had been invisible *for weeks* behind the 50-case ruler, which reported a clean `wrong = 0` the whole time. **The cost.** Time and tokens were spent making the system *worse* (wiring an unsound reader into serving) while believing it was improving, and the work was shown to outside professionals as progress. That is the failure this document exists to prevent recurring. **Immediate remediation (done):** the unsound committing bridges (`product_bridge`, `goal_residual`) were disabled; sealed re-confirmed `0 / 0 / 1319` — back to the honest refusal floor with zero confabulation. Rule recorded in `docs/claims_ledger.md`: never re-enable a committing bridge without sealed-proven `wrong = 0`. --- ## 2. The measurement that exposed the truth We built the instrument we never had: a **held-out dev lane** — 500 real GSM8K problems CORE was **not** built against (`evals/gsm8k_math/holdout_dev/`). One scorer, three sets, same day: | Set | correct | wrong | what it is | |---|---:|---:|---| | `train_sample` (50) | 4 | 0 | the data CORE was **built on** — overfit, no predictive validity | | `holdout_dev` (500) | 5† | 0 | real GSM8K **never built-against** — the composer scored **0**; the 5 are the narrow R1 reconstruction | | sealed test (1,319) | 0 | 0 | the final exam | † Committed `evals/gsm8k_math/holdout_dev/v1/report.json` reads **5 correct / 0 wrong / 495 refused**. All 5 come from the narrow, verifier-gated **R1 comparative-total reconstruction** (a typed reader, not the composer), which landed the same day; the composer (`resolve_pooled`) generalised to **0** of 500. The "0%" below is the composer's number — the subject of this post-mortem. (The original snapshot read `0/500`; it predated the R1 landing and is corrected here to match the committed artifact.) **The composer's real GSM8K capability was 0%.** Its four train "correct" generalised to *zero* of 500 held-out cases (the 5 committed held-out correct are the separate R1 reconstruction, above). Serving held `wrong = 0` only because it refused everything real. **The deeper finding — the composer is *unsound*, not just narrow.** The derivation composer (`resolve_pooled`), the reader the whole composition program was built around as "`wrong=0`-safe", measured on held-out: **2 correct / 87 WRONG / 411 refused — 17% confabulation.** And no gate fixes it: characterising all 89 commits, *step count, pool size, and op-sequence do not separate the 2 right from the 87 wrong.* The composer **cannot distinguish its correct readings from its wrong ones** — so no principled gate admits the right while refusing the wrong. (Full analysis: `docs/analysis/real-gsm8k-capability-measurement-2026-06-04.md`.) Four committing-reader lift attempts were built and **empirically falsified** on held-out (not just argued): the composer (17% wrong), a maximally-narrow forced reader (100% wrong when it fired), a strict sum-reader (100% wrong), and the sound candidate-graph filter (built 0 admissible candidates — sound but constructs nothing). **Shallow committing cannot be sound on real GSM8K**, because GSM8K problems that look like "two numbers and a cue word" are almost always multi-step problems where the shallow reading is wrong every time. --- ## 3. Why the terrain was wrong for CORE GSM8K is **deliberately engineered** to require 2–8 step natural-language reasoning — built to defeat exactly the pattern-matching we kept attempting. CORE is a **deterministic, verifiable** engine; its strength is exact, inspectable, replayable reasoning where a conclusion can be *checked*. GSM8K is the benchmark built *for* stochastic multi-step LLM reasoning. **We were arm-wrestling a machine on its home court.** This is not a new realisation buried in the loss — it is CORE's own stated doctrine: *"Next primitives = checkable-conclusion domains (logic / constraint / proof)."* The GSM8K chase had drifted away from it. --- ## 4. The path now: deductive logic (checkable-conclusion terrain) We pivoted the "sizeable numbers" goal to **deductive logic**, where the answer is *verifiable by construction* — so capability can show real correct numbers, not just a refusal floor, and `wrong = 0` is structural rather than aspirational. **What this reuses (already paid for, previously zero-consumer):** the ADR-0201 ROBDD propositional canonicalizer, the binding-graph DAG (ADR-0132), the proof_chain model / builder / single-step modus ponens (ADR-0204/0205). ### Phase 1 — DONE (this PR) A **sound and complete propositional entailment operator** (`generate/proof_chain/entail.py`, ADR-0206) — the multi-hop inference operator `evals/symbolic_logic/gaps.md` said did not exist and ADR-0205 deferred: > premises ⊨ query iff `(⋀ premises) → query` is a tautology — an exact ROBDD check. Outcomes: `entailed` / `refuted` / `unknown` / `refused` (inconsistent or out-of-decidable-regime). `wrong = 0` is structural; quantified/predicate input refuses by design (propositional regime only). **Measured, honestly:** | set | n | correct | wrong | non-trivial (entailed+refuted) | |---|---:|---:|---:|---:| | dev | 200 | 200 | **0** | 74 | | **holdout v1** | 500 | **500** | **0** | **227** | | fuzz (seed 424242, engine vs **independent** oracle) | 2,796 | 2,796 | **0** | 1,241 | The gold is computed by an **independent truth-table oracle** (`evals/deductive_logic/oracle.py`) sharing **no code** with the ROBDD engine. The committed gate (`test_engine_matches_independent_oracle_fuzz`) runs a deterministic **3,000-case** fuzz of which **2,796** are in-regime/definite, with **zero** engine-vs-oracle disagreements (a complementary 4,000-case fuzz exercises inconsistent-premise refusal) — the soundness evidence the GSM8K composer could never produce. INV-25 (`tests/test_architectural_invariants.py`) ratifies that independence structurally. **Honest caveat (state it plainly):** propositional validity is *decidable*, so 100% is the expected ceiling for a correct implementation. The value is not "solved something unsolved" — it is (a) CORE's first real, non-zero, **independently-verified** capability (not a refusal floor), (b) the honest foundation the harder layers build on, (c) it is on CORE's terrain. External framing: *"CORE performs sound, complete, deterministic deductive inference — multi-hop entailment and refutation — verified `wrong = 0` against an independent oracle on held-out data."* ### Phases 2–4 — the plan 2. **Scale + recognizable benchmark.** Harder/deeper held-out sets; a grounded **RuleTaker / ProofWriter** mirror (finite-entity rules ground to propositional) → a number on a *published, recognizable* benchmark, same `wrong = 0` discipline. 3. **NL→logic grounding.** Read English logic problems into formulas. This is where it becomes externally impressive — and where the GSM8K natural-language rake lives, so it is measured the same disciplined way (held-out + independent gold) and refuses what it cannot ground. 4. **Serving.** Wire the operator into `chat/runtime` so CORE answers deductive queries to users, gated by the same `wrong = 0` discipline (and, for any serving promotion, the sealed-confirmation rule from §1). --- ## 5. Disciplines carried forward (the anti-recurrence rules) 1. **Never score capability on data the system was built against.** Every capability claim is measured on held-out data + confirmed on a sealed/independent arbiter. No `train_sample`-class ruler is ever the score again. 2. **Gold must be independent of the system under test.** A held-out lane uses an independent oracle (or sealed operator-run gold), never the engine's own output. 3. **`wrong = 0` is a floor, not a result.** "Refuses everything" is the *failing* baseline. The capability claim is *coverage with* `wrong = 0`. 4. **No serving promotion without sealed/independent-proven `wrong = 0`.** The product_bridge breach is the standing reason. 5. **Falsify lift attempts empirically, on held-out, before trusting them.** Build the reader, measure it on data it was not tuned to, and report what it actually does — including the failures. 6. **Prefer checkable-conclusion terrain.** Where the answer is verifiable, capability is honest and `wrong = 0` is structural. That is where CORE competes. --- ## Reproduce ```bash PYTHONPATH=. .venv/bin/python -m evals.deductive_logic.runner # dev + holdout; exits 1 if wrong>0 PYTHONPATH=. .venv/bin/python -m pytest tests/test_deductive_logic_entail.py -q # GSM8K honest baseline (context for the pivot): PYTHONPATH=. .venv/bin/python -m evals.gsm8k_math.holdout_dev.v1.runner # the held-out instrument ```