Commit graph

1161 commits

Author SHA1 Message Date
Shay
bbf75ff195 docs(l10): reconcile grounding ADR to registry (ADR-0210) + flag gold consistency
Rename adr-012-l10-grounding.md -> ADR-0210-l10-grounding-pack.md: the
requested number collided with ADR-0012-core-ingest-governance-layer.md and
did not follow the ADR-NNNN convention. 0210 is the next free number.

Record the one latent soundness item: l10-adv-003 (false) vs l10-adv-008
(refuse) take different stances on an unsatisfied guard; the discriminating
principle (same-sort negative -> false; cross-sort mismatch -> refuse) must
be encoded by the future symbolic runner before the fixtures become a live
wrong=0 oracle. Pack + fixtures remain inert and byte-identical.
2026-06-05 10:01:17 -07:00
Shay
6eccdb41ea docs(l10): propose finite grounding ADR 2026-06-05 09:08:42 -07:00
Shay
51c6852b0c test(l10): add independent-gold adversarial logic fixtures 2026-06-05 09:08:23 -07:00
Shay
fb3ca87fd0 docs(l10): describe finite grounding pack 2026-06-05 09:07:55 -07:00
Shay
543e6c04f5 feat(l10): add finite grounding pack manifest 2026-06-05 09:07:42 -07:00
Shay
21aa8b765f feat(l10): add finite grounding lexicon 2026-06-05 09:07:13 -07:00
Shay
cd45103da8
Merge pull request #559 from AssetOverflow/docs/independent-comprehension-gate
docs(research): define independent comprehension gate + field wedge
2026-06-05 08:07:59 -07:00
Shay
d47741f5df
Merge pull request #562 from AssetOverflow/feat/cga-incidence-primitives
algebra: incidence algebra — graded_wedge, dual, meet + honest outer_product
2026-06-04 21:52:20 -07:00
Shay
ed42a83e7a
Merge pull request #561 from AssetOverflow/feat/relational-grounding
Deductive grounding: binary relations + multi-variable rules (finite propositional)
2026-06-04 21:47:10 -07:00
Shay
de645055ea feat(algebra): incidence algebra — graded_wedge, dual, meet + honest outer_product
Adds the correct grade-raising "wire" the field substrate was missing — so cga_inner
can operate on RELATIONS among entities (lines/planes/incidence), not just pairwise
point distance. Built only from existing Cl(4,1) primitives (geometric_product,
grade_project) + the pseudoscalar; no normalization, no approximation, versor_condition
path untouched (flats are null-cone wedges, not unit versors).

- outer_product: DOCSTRING-ONLY honesty fix (behavior byte-identical, every caller
  unchanged). It is the commutator 0.5*(XY-YX) = the wedge ONLY for grade-1 vectors;
  for higher grades it is the Lie bracket, NOT the wedge, and does NOT build a k-blade
  by repetition. Existing callers consume it as an opaque cga_inner-reduced feature
  (none read it by grade), so the relabel is safe. Points to graded_wedge for the real
  exterior product.
- graded_wedge(X,Y) = <XY>_{grade(X)+grade(Y)} — the true wedge; agrees with
  outer_product on grade-1, differs above (pinned by test). Builds lines/planes.
- is_incident(point, flat): EXACT zero-test (point^flat == 0, no float tolerance to
  admit — near-incident is refused, per wrong=0). Exact at scale in f64.
- dual(X) = X*I5^{-1} (I5^2=-1 confirmed); involutive up to sign.
- meet(A,B) = dual(dual(A)^dual(B)): correct for spanning operands (two planes -> their
  line, incidence verified). HONEST ENVELOPE: degenerates for non-spanning operands
  (coplanar lines) — returns the ZERO multivector (detectable, documented, tested),
  never a silent wrong value. The general coplanar intersection needs the join-relative
  meet, deliberately NOT faked here.

Green: smoke 87, algebra 82, incidence 8, outer_product consumers + invariants 109;
zero regressions (outer_product behavior unchanged).
2026-06-04 21:43:35 -07:00
Shay
5f274b75b7 feat(deductive): binary relations + multi-variable grounding (finite propositional)
Extends evals/deductive_logic/grounding.py from unary predicates / single-var rules to
binary relations + multi-variable universal rules, still by FINITE PROPOSITIONAL
grounding into the regime the ROBDD engine + the independent truth-table oracle both
decide. wrong==0 stays structural. This is the real capability step a RuleTaker/
ProofWriter-style mirror needs (the unary fragment alone is trivial).

- atom_n lowers pred(a,b) -> pred__a__b; arity-1 is byte-identical to the old atom, so
  the live unary panel lowers unchanged (proven by an exact-string back-compat test).
- multi-variable universal rules ground over n^k assignments — transitivity now decides.
- range-restriction: a rule with a head variable unbound in the body refuses (unsafe_rule)
  — it grounds soundly but is outside the clean regime real benchmarks use.
- typed refusals: arity>=3/functions, explicit quantifiers, variable-free rules, bounds.

Honest ceilings (documented in docs/analysis/relational-grounding-extension-2026-06-04.md):
- THE binding constraint is the GOLD, not the grammar: the truth-table oracle is
  O(2^atoms), so grounding refuses above MAX_GROUND_ATOMS=20 => binary problems cap at
  ~4 entities/predicate. A real lift needs a 2nd genuinely-independent sub-enumeration
  oracle (not built).
- OPEN-WORLD only: RuleTaker/ProofWriter's main splits are closed-world + NAF; a future
  adapter MUST refuse CWA/NAF (mapping CWA "False"->"refuted" is a wrong=0 breach).
- arity <= 2, function-free.

Validated: held-out differential fuzz (400 random binary problems, oracle-golded) = 0
engine/oracle mismatches; unary back-compat byte-identical; INV-25b reproducibility green;
deductive lane wrong=0 16/16; smoke 87.
2026-06-04 20:17:33 -07:00
Shay
5c2f005e96
Merge pull request #560 from AssetOverflow/feat/field-reasoner-wedge
Field-reasoner wedge: f64 foundation, lineage firewall (INV-27), and the honest C3 ablation verdict
2026-06-04 20:00:29 -07:00
Shay
5c77c9eece feat(field-wedge): ablation verdict — field is decoration on additive (C3) (Phase W.2)
The falsifiable experiment's measurements #2 (ablation) and #3 (diversity). Builds the
competent, code-disjoint SYMBOLIC reader (the control arm AND the C3 capability path)
and the ablation instrument that runs both readers through the real
verify_tier2_agreement gate.

VERDICT: C3 — the field is decoration on this domain (a sanctioned, honest negative):
- field_wrong_commits = []  (wrong=0 holds; the per-step drift guard refuses bad ints)
- field_caught_symbolic_errors = []  (the field caught ZERO symbolic errors)
- per-class diversity = 0 everywhere (both readers agree and are both correct)
- the only admitted-set change is the field LOSING coverage at the precision ceiling.

Insight: on forward-substitutable relations, geometric translation IS arithmetic
addition, so there is no metric over-determination for the field to exploit — field and
symbol are common-mode (Knight-Leveson), not a genuine second derivation. This is the
deductive finding's twin: logic was combinatorial (field can't earn it), additive is
arithmetically trivial (field adds nothing). The field needs metric-nontrivial AND
arithmetically-hard structure to earn a reasoning role — dedicated research, not
near-term. Field-as-reasoner is NOT earned; no field vote enters any serving path; the
field stays a servant. Capability path = symbolic (C3), not shipped here.

- generate/relational_symbolic_reader.py: competent independent reader (pure int).
- evals/relational_metric/ablation.py: the reusable decoration instrument.
- docs/analysis/field-wedge-ablation-result-2026-06-04.md: the recorded verdict.

All prior artifacts STAY (field reader = real wrong=0 read demo + 3rd panel domain).
Green: full wedge suite 104; 53 architectural invariants.
2026-06-04 19:44:22 -07:00
Shay
145d797196 feat(field-wedge): geometric field reader — relational-metric lane wrong=0 (Phase W.1)
Measurement #1 of the field-reasoner falsifiable experiment: does the CL(4,1) field,
given an honest metric encoding, read forward-substitutable quantitative-relational
problems from TEXT with wrong==0? It does — 14/15 correct, 0 wrong, 1 refused
(precision ceiling), scored against an independent arithmetic oracle.

- generate/relational_field_reader.py: reads problem text into conformal points on
  the e1 number line; additive/part-whole relations are conformal TRANSLATOR versors
  (versor_apply(T_delta, embed[x]) == embed[x+delta], exact); the answer reads back
  by projective dehomogenization. Refusal-first: fences multiplicative/ratio (the
  sign/orientation-blind cases), the precision ceiling, non-forward-substitutable
  references, negatives. A per-step exactness self-check turns any f64 translator
  drift into a refusal (precision_drift) — it NEVER commits a wrong integer. Its
  parser is an independent reimplementation importing no generate.derivation/math_*.
- evals/relational_metric/: independent arithmetic oracle (computes gold from the
  STRUCTURE, shares no code with the reader), 15-case fixture, and a runner that
  enforces gold integrity + wrong==0.
- INV-25: relational_metric registered in INDEPENDENT_GOLD_LANES (oracle proven
  code-disjoint from the field reader and the algebra engine). The independently
  golded panel is now three domains: deductive, dimensional, relational-metric.

Green: smoke 87, 53 architectural invariants, 16 new tests; deductive + dimensional
lanes unperturbed (wrong=0).
2026-06-04 19:34:43 -07:00
Shay
568face63e feat(reasoning): field-wedge foundation — f64 embedding, lineage firewall, INV-27
Phase 0 of the field-reasoner wedge — net hardening regardless of the
experiment's outcome.

- algebra/cga.py: embed_point gains a dtype kwarg (f32 default byte-unchanged;
  cl41.geometric_product already preserves f64) + read_scalar_e1 projective
  dehomogenization read-back (weight-invariant; correct for dilations, where a
  raw distance-from-origin is wrong) + EMBED_EXACT_MAX pinned magnitude ceiling.
  f32 silently collapsed integer coordinates past ~1e4.
- core/reasoning/evidence.py: verify_tier2_agreement now keys independence on a
  reader_lineage pathway token (refuses SAME_READER_LINEAGE), replacing the
  label-only len(set(signatures))<2 check a single reader could satisfy by
  relabeling. reader_lineage is excluded from canonical serialization, so the
  entailment trace_hash is unchanged.
- tests INV-27: transitive reader-disjointness over TIER2_READER_PATHWAYS makes
  the lineage check load-bearing (distinct lineage => proven import-disjoint
  pathway). The two seeded readers share zero transitive first-party modules.

Green: smoke 87, algebra 82, cognition 121, 53 architectural invariants,
reasoning/deductive/r1 50; 16 new f64-exactness tests; zero regressions.
2026-06-04 19:22:16 -07:00
Shay
991be784fa docs: field-reasoner wedge design + falsification dossier
Source-grounded design of the field<->symbol coherence-gate wedge, corrected by
an 11-agent adversarial review into a falsifiable experiment with a sanctioned
negative outcome (field stays a servant -> C3 two code-disjoint symbolic
readings). Records the substrate ledger (the CL(4,1) field has one exact strength
relevant to reasoning -- the conformal distance metric -- and no solver/incidence
machinery), the corrected design, the three adversarial verdicts, the falsifiable
experiment (ablation + per-class diversity), and the phased path.
2026-06-04 19:22:16 -07:00
Shay
b53f6e2465 docs(research): select quantitative field reasoner wedge 2026-06-04 17:23:57 -07:00
Shay
83429f5135 docs(research): define independent comprehension agreement gate 2026-06-04 17:23:20 -07:00
Shay
388a0a6d80
Merge pull request #558 from AssetOverflow/docs/session-2026-06-04-universal-structure
docs: session record (2026-06-04) + runway PR-1/PR-2 shipped
2026-06-04 16:57:50 -07:00
Shay
f97fcaf453 docs: session record (2026-06-04) + mark runway PR-1/PR-2 shipped
Session journey: the deductive pivot (verified, not asserted), INV-25 independent-
gold ratification, the universal-structure + field<->symbol coherence-gate synthesis,
and the autonomous build of the foundation + 3-domain anti-overfit panel (#554-557).
Records the two field-as-reasoner findings (logic is combinatorial; independence must
live in the reading) that defer the geometric wedge to dedicated research.

Marks the deductive-logic runway's PR-1 (finite-entity grounding) and PR-2 (oracle
parity) as shipped via #556.
2026-06-04 16:54:03 -07:00
Shay
4c6290f773
Merge pull request #557 from AssetOverflow/feat/dimensional-reasoning-domain
Dimensional-reasoning lane — 3rd diversity-panel domain
2026-06-04 16:48:55 -07:00
Shay
c0d7ec48db
Merge pull request #556 from AssetOverflow/feat/phase2-finite-entity-grounding
Phase 2 — finite-entity grounding compiler (2nd diversity-panel domain)
2026-06-04 16:48:40 -07:00
Shay
96c1d4bcee feat: dimensional-reasoning lane — 3rd diversity-panel domain
The first non-GSM8K consumer of the binding-graph interlingua's unit algebra as a
load-bearing reasoner: given two units and an operation, decide the result's
dimension. SUT = generate.binding_graph.units; gold = evals/dimensional/oracle.py,
a genuinely INDEPENDENT dimensional reasoner (own unit->base-exponent table, own
exponent arithmetic, own canonical-string renderer; shares no code with the SUT).

12 cases (area / speed / wage / mass-density / dimensionless / 2 refused) gated by
SUT == oracle == gold (wrong=0). Registered in INV-25's INDEPENDENT_GOLD_LANES,
proving the independent-gold discipline generalizes to a SECOND oracle.

This is the 3rd structurally-distinct golded domain (logic / grounding / dimensional)
— the anti-overfit >=2-domain panel is now real, and the interlingua is load-bearing
beyond GSM8K.
2026-06-04 16:38:56 -07:00
Shay
3e2a52870d feat: Phase 2 — finite-entity grounding compiler + Phase 1.5 finding
The first comprehension->structure compiler: evals/deductive_logic/grounding.py
lowers a typed finite-entity problem (finite entities + unary predicates +
single-variable universal rules) into the propositional regime the ADR-0206
entailment operator decides, refusal-first with a closed typed reason vocabulary
(unsafe_symbol / unknown_entity / unsupported_predicate_arity / unsupported_
quantifier / malformed_case / empty_case) and collision-safe atom slugging.

finite_entity/v1/cases.jsonl: 8 cases with INDEPENDENT (oracle-derived) gold
(4 entailed, 2 unknown, 1 refuted, 1 refused) — chained rules, conjunctive
bodies, negative heads, inconsistent premises. 20 tests gate engine==oracle==gold.

This is the second diversity-panel domain (distinct comprehension, same checkable
substrate) — the universal-structure thesis validated on a different problem shape,
with the anti-overfit >=2-domain discipline now live.

Phase 1.5 finding recorded: a clean geometric/algebraic propositional decoder
agrees 716/716 with the oracle but is O(2^n) (enumeration-class), so logic is the
wrong first domain for field-as-reasoner; the wedge redirects to quantitative-
relational structure where the field is the natural non-redundant decoder.
2026-06-04 16:32:03 -07:00
Shay
4d964020a9
Merge pull request #555 from AssetOverflow/feat/phase1-universal-structure
Phase 1 — canonize binding-graph interlingua + INV-26 neutrality
2026-06-04 16:29:56 -07:00
Shay
8d620d6257 feat: Phase 1 — canonize binding-graph interlingua + INV-26 neutrality
Declares SemanticSymbolicBindingGraph the universal problem-structure interlingua
(the corpus callosum where the geometric field and symbolic ROBDD decodings meet
and must agree). INV-26 enforces that neutrality structurally: 26a no binding-graph
module imports field/algebra/eval/vault/chat/core/sensorium; 26b the core imports
no domain reader (only allowlisted bridges adapter/question_target may); 26c proven
non-vacuous (flags pipeline.py's field import + the adapter's domain import).

Amends the plan doc with the structurally-diverse checkable panel (logic/grounding/
dimensional/execution/constraint, each with independent gold) and the 'a capability
change must move >=2 structurally-distinct domains or it is suspected overfitting'
rule, woven in from Phase 2 — the anti-overfit instrument the train_sample breach
proved we need.

Validated: architectural invariants 49 + binding_graph model = 118 passed.
2026-06-04 16:20:12 -07:00
Shay
0932c0e92e
Merge pull request #554 from AssetOverflow/claude/modest-franklin-85067f
Ratify independent-gold invariant (INV-25) + SHA-pin deductive lane
2026-06-04 16:05:50 -07:00
Shay
0ac725ccfd docs: plan — universal structure + field<->symbol coherence gate
Planning-only design doc for the comprehension->structure->solve->verify spine:
converge reading onto the binding-graph interlingua; field comprehends, structure
bridges, symbol verifies; their AGREEMENT is the wrong=0 gate and the genuine
second derivation that unblocks t2_precision. Field-as-reasoner must EARN each
domain against independent gold (Phase 1.5 wedge: field-decides-entailment vs
ROBDD oracle); never asserted unfalsifiably. INV-25 is Phase 0.
2026-06-04 15:56:28 -07:00
Shay
a447dce5d1 feat: ratify independent-gold invariant (INV-25) + SHA-pin deductive lane
INV-25 makes the GSM8K lesson structural: no capability claim is valid unless
its gold is computed by a procedure sharing no code with the system under test.
Three meaningfully-failing checks (proven able to fail): 25a oracle imports no
SUT module (AST); 25b every committed deductive gold is reproduced by the
independent oracle AND matched by the engine; 25c an unsound engine disagrees
on committed cases.

SHA-pin the deductive lane (deductive_logic_v1, dev+holdout+external 716/716,
wrong=0, refused=0) via a deterministic --report writer + run_as_module (the
lane dir's local generate.py shadows the package in script mode) + CLAIMS regen.

Fix drift the review surfaced: contract.md + pivot doc claimed an 8,000/7,340
fuzz and an 'external mirror' -> corrected to the real gated 3,000-case fuzz
(2,796 definite) and 'hand-authored, NOT a published-benchmark mirror'; pivot
doc GSM8K holdout 0->5 (all 5 are R1 reconstruction; composer scored 0).

Validated: smoke 78, deductive 23, proof 29, invariants 44, lane-sha+claims 9.
2026-06-04 15:56:28 -07:00
Shay
4215f1b4cf
Merge pull request #553 from AssetOverflow/codex/tier2-evidence-spine
feat: add tier2 reasoning evidence spine
2026-06-04 14:21:05 -07:00
Shay
134a91c48a feat: add tier2 reasoning evidence spine 2026-06-04 14:12:18 -07:00
Shay
efbd646ba1
Merge pull request #552 from AssetOverflow/codex/gsm8k-r1-reconstruction
feat: add gsm8k r1 reconstruction
2026-06-04 13:59:07 -07:00
Shay
7155f5ab34 feat: add gsm8k r1 reconstruction 2026-06-04 13:25:11 -07:00
Shay
0cc2618f5c
Merge pull request #551 from AssetOverflow/codex/deductive-phase2-benchmark-runway
docs: define deductive logic phase 2 benchmark runway
2026-06-04 13:24:55 -07:00
Shay
d4a07f23b1 docs: define deductive logic phase 2 benchmark runway 2026-06-04 09:33:23 -07:00
Shay
fc20d01832
Merge pull request #550 from AssetOverflow/codex/deductive-capability-foundation
feat: add deductive proof evidence gates
2026-06-04 08:47:24 -07:00
Shay
3389cc68c3 feat: add deductive proof evidence gates 2026-06-04 08:37:51 -07:00
Shay
cc885157a5
Merge pull request #549 from AssetOverflow/feat/deductive-logic-entailment
feat: sound+complete propositional entailment operator + deductive-logic lane (the pivot)
2026-06-04 08:11:36 -07:00
Shay
602a76221a docs: record the GSM8K->deductive-logic pivot, post-mortem, and plan
Load-bearing strategic record: what the GSM8K chase got wrong (overfit
50-case ruler hiding a sealed wrong=0 breach), the held-out measurement
that exposed real capability=0% / composer 17% wrong, why GSM8K is the
wrong terrain for a deterministic verifiable engine, and the deductive-
logic path now (Phase 1 done: 500/500 held-out wrong=0, independent-
oracle verified) with phases 2-4 and the anti-recurrence disciplines.
2026-06-04 07:51:36 -07:00
Shay
48827f281a feat: sound+complete propositional entailment operator + deductive-logic lane
The first SIZEABLE, honestly-verified reasoning capability — built on CORE's
own terrain (exact, verifiable, deterministic), not GSM8K's stochastic terrain.

THE OPERATOR (generate/proof_chain/entail.py, ADR-0206):
- evaluate_entailment(premises, query) -> entailed | refuted | unknown | refused.
- The multi-hop inference operator evals/symbolic_logic/gaps.md said did not exist
  ("no operator that takes A->B, B->C and returns A->C") and ADR-0205 deferred.
- Built on the ADR-0201 ROBDD canonicalizer: premises |= Q iff (AND P) -> Q is a
  tautology. SOUND AND COMPLETE for propositional logic, not single-step.
- wrong=0 is structural: an exact tautology check refuses (LogicError) on
  malformed / out-of-decidable-regime (quantified/predicate) input, never guesses.

THE HONEST METRIC (evals/deductive_logic/):
- holdout v1 (500 cases): 500 correct / 0 WRONG, incl. 227 non-trivial deductions
  (117 entailed + 110 refuted). dev (200): 200/0.
- Gold from an INDEPENDENT truth-table oracle (oracle.py) sharing zero code with
  the engine. 8,000-case fuzz across two independent decision procedures:
  0 disagreements. This is the soundness evidence the GSM8K composer could never
  produce (it could not separate its 2 right from its 87 wrong answers).
- contract.md states the load-bearing honesty boundary: PROPOSITIONAL ONLY, and
  given-formulas (NL->logic grounding is a separate later layer, kept out of scope
  so we do not re-step on the GSM8K natural-language rake).

TESTS (17, all green): classic inference shapes (MP, multi-hop chain, modus
tollens, disjunctive/hypothetical syllogism, conjunctive rules, genuine unknown),
refusal boundary (inconsistent / quantified / predicate / malformed), and a
deterministic engine-vs-oracle fuzz cross-check.

Pure new module — does NOT touch serving. Smoke 73 passed; invariants 40 passed.
2026-06-04 07:47:01 -07:00
Shay
f301f3a928
Merge pull request #548 from AssetOverflow/feat/holdout-dev-set
feat(eval): held-out dev lane — honest iteration metric (real GSM8K capability = 0/500)
2026-06-04 07:14:58 -07:00
Shay
f56a0cfdba
Merge pull request #546 from AssetOverflow/docs/reconcile-current-state-2026-06-03
docs: reconcile current-state claims after GSM8K + sensorium progress
2026-06-04 07:14:39 -07:00
Shay
d4a002e626 docs: fourth lift attempt falsified (strict sum-reader 0/2) — exhaustive
Four genuine build-and-measure attempts on held-out, all falsified:
- composer 87 wrong, narrow reader 61 wrong, strict sum-reader 2 wrong (100%),
  candidate-graph 0 admissible. Even 'exactly 2 numbers + total cue' cases are
  multi-step in disguise. No sound committing strategy exists in this substrate.
2026-06-04 02:52:24 -07:00
Shay
c31bc82621 docs: third leg — candidate-graph builds 0 admissible candidates on held-out
All three GSM8K paths exhausted on the held-out 500:
- candidate-graph (sound filter): 0 admissible candidates built (498/500 build nothing)
- composer (open): 87 wrong
- narrow reader (open): 61 wrong
The sound path is wrong=0 only because it constructs no reading to commit. No sound
lift exists in the current substrate; confirmed empirically, three ways.
2026-06-04 02:49:00 -07:00
Shay
92e9631467 docs: two lift attempts falsified + the ADR-0207 architectural impasse
Built and measured two committing readers on held-out (not just diagnosed):
- resolve_pooled composer: 2 correct / 87 wrong (17%)
- maximally-narrow forced reader: 0 correct / 61 wrong (100% when it fires)

Shallow committing CANNOT be sound on real GSM8K (measured). The impasse:
- candidate-graph path is SOUND (wrong=0) but FROZEN by ADR-0207 §4, covers 0 real
- composer is OPEN (ADR-0207 §5) but UNSOUND (17% wrong, no separating gate)
ADR-0207's premise (composer is the wrong=0-safe path) is FALSIFIED by held-out data;
'feed the composer' is a path to more confabulation. Needs a follow-up ADR.
2026-06-04 02:47:03 -07:00
Shay
c2e74fe2fd docs: the honest real-GSM8K capability measurement (0% sound; composer 17% wrong on held-out)
The result that should have been visible for weeks. Measured on the new holdout_dev
lane (500 real held-out cases) + sealed test:
- real capability 0% (train_sample's 4 correct generalize to 0/500 held-out)
- resolve_pooled (the composition substrate) commits 17.4% WRONG on held-out
- NO structural gate (step/pool/op) separates its 2 correct from 87 wrong -> the
  composer cannot tell right readings from wrong; gating it is overfitting
- the only wrong=0 policy on real GSM8K is refusal (current serving 0/0/500)

Real lift needs a verifiable reading mechanism, not composer tuning. Non-serving (docs).
2026-06-04 02:42:14 -07:00
Shay
d81084ffe3 feat(eval): held-out dev lane — the honest iteration metric (real capability = 0)
The 2026-06-04 sealed-breach post-mortem proved the 50-case train_sample has ZERO
predictive validity (its 4 "correct" are overfit; they hid a 5-wrong sealed breach).
This adds the instrument we never had: 500 real GSM8K cases CORE was NOT built on —
the train split minus the 50 train_sample, deterministic sha256(question) sort.

Same scorer as train_sample + the sealed lane, so the three are directly comparable:
  train_sample(50): 4/0/46   holdout_dev(500): 0/0/500   sealed test(1319): 0/0/1319

Real GSM8K capability is 0%. The 4 train "correct" generalize to NOT ONE of 500
held-out cases. wrong=0 holds (refuses, never confabulates).

- evals/gsm8k_math/holdout_dev/v1/: cases.jsonl (500), runner, report (0/0/500), README.
- tests/test_holdout_dev_lane.py: floor (wrong==0, forever) + baseline snapshot (0/500).

Discipline: iterate here (open, large enough to resist trivial overfit); the sealed
test stays the final arbiter. wrong=0 is the floor; correct rising is the goal;
"refuse everything" is the FAILING baseline to beat, not a pass. Non-serving (eval only).
2026-06-04 02:30:42 -07:00
Shay
431699791c
Merge pull request #547 from AssetOverflow/fix/disable-unsound-serving-bridges
fix(gsm8k): disable unsound serving bridges — restore sealed wrong=0 (0/5 → 0/0)
2026-06-04 02:25:49 -07:00
Shay
763c46d2f4 fix(gsm8k): disable unsound serving bridges — restore sealed wrong=0 (0/5 -> 0/0)
The FIRST real sealed measurement (operator-decrypted 1,319 held-out GSM8K)
found `0 correct / 5 WRONG` — a wrong=0 breach hidden for weeks because the
working metric was the 50-case train sample the bridges were tuned to. Bisection
isolated it to the product_bridge serving promotion (ADR-0195).

- generate/math_candidate_graph.py: REMOVE both serving promotion bridges
  (product_bridge + goal_residual/ADR-0207 §5 step 2). Serving = main-graph-only.
  Restores sealed 0/0/1319 (verified by bisect: disabling product_bridge -> 0 wrong).
  Production modules remain in generate/derivation/; only serving promotion is
  unwired, until a gate is proven wrong=0 on the SEALED set (never the train sample).
- Honest numbers everywhere: train_sample 7/43/0 -> 4/46/0 (the bridges' "correct"
  was train-overfit). report.json + coverage probe regenerated. 7 ADR test lanes
  de-pinned from the inflated count. corpus: cv-0005 (R4) reverts to refuse; cv-0020
  (a "baseline control" that solved ONLY via product_bridge) reclassified.
- docs/claims_ledger.md: dated wrong=0-breach-and-remediation note + the rule:
  the train_sample number had ZERO predictive validity for the exam; never the score.
- docs/analysis/gsm8k-lift-program-strategy: the program to actually move the 1,319.

NOTE the exit gate stays `correct>=10 AND wrong==0` — refusing-everything is an
explicit FAIL, not a wrong=0 pass; serving still commits (main graph). Verified:
broad regression 848 passed, smoke 73 passed.
2026-06-04 01:55:05 -07:00
Shay
399720817f docs: reconcile ADR-0207 execution status 2026-06-03 22:45:07 -07:00