From 94bf1be1bcc7ebf7c78e031719ab1ad424d37fa0 Mon Sep 17 00:00:00 2001 From: Shay Date: Wed, 3 Jun 2026 19:42:47 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20ratify=20ADR-0207=20=E2=80=94=20GSM8K?= =?UTF-8?q?=20comprehension/composition=20substrate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Consolidating ratification of the GSM8K design of record. Ratify the built comprehension/derivation substrate, freeze the serving regex recognizer/ injector path to lexemes + refusal-only, pin Phase 5b execution to WIRING -> COMPOSITION -> LEXICON. - ADR-0207: new consolidating decision (Accepted, ratified 2026-06-03). Supersedes ADR-0163 §Phase B-E + ADR-0136 regex sentence-template prescriptions. Freeze + wrong=0 gates (22-case corpus + sealed 1,319). - ADR-0164/0165/0174/0178/0179: -> Accepted (ratified by ADR-0207, 2026-06-03). 0164 keeps its implementation clause (Phase 1+2 shipped; remainder per §5) so Accepted != fully built. - composition_validation/v1: 20 -> 22 cases (2nd R4/R5 positives, dataset-sourced golds), +contract invariants 6-7, +dataset-gold test. Baseline 4/18/0; 47 passed. - docs/analysis: extraction-richness audit (read-only) reconciling ADR-0179 to the tree (EX-1/2/4/5/6 landed; EX-3 deferred). Non-serving (evals/docs/tests only). train_sample 6/44/0 unchanged; no-ref times hazard stays refused. GB3b/0136 untouched. --- .../extraction-richness-audit-2026-06-03.md | 166 ++++++++++++++ ...R-0164-incremental-comprehension-reader.md | 2 +- docs/decisions/ADR-0165-regex-scope-rule.md | 2 +- .../ADR-0174-held-hypothesis-comprehension.md | 2 +- .../ADR-0178-compositional-structure.md | 2 +- .../decisions/ADR-0179-extraction-richness.md | 2 +- .../ADR-0207-gsm8k-substrate-ratification.md | 206 ++++++++++++++++++ .../composition_validation/v1/cases.jsonl | 2 + .../composition_validation/v1/contract.md | 21 +- tests/test_composition_validation_corpus.py | 51 ++++- 10 files changed, 442 insertions(+), 14 deletions(-) create mode 100644 docs/analysis/extraction-richness-audit-2026-06-03.md create mode 100644 docs/decisions/ADR-0207-gsm8k-substrate-ratification.md diff --git a/docs/analysis/extraction-richness-audit-2026-06-03.md b/docs/analysis/extraction-richness-audit-2026-06-03.md new file mode 100644 index 00000000..96e429f6 --- /dev/null +++ b/docs/analysis/extraction-richness-audit-2026-06-03.md @@ -0,0 +1,166 @@ + + +# Extraction-Richness Audit — ADR-0179 named cases, reconciled to the shipped tree + +**Status:** Analysis only. No serving, eval, or code edits. Every claim below +reproduced live against `origin/main` @ `2cb0922` via read-only +`extract_quantities` / derivation-composer / `parse_and_solve` probes. + +**Scope.** Open [ADR-0179 — Extraction Richness](../decisions/ADR-0179-extraction-richness.md), +read the exact cases its §Context table names as starved by the thin +`(\d+(?:\.\d+)?)\s+([a-z]+)` extractor, and verify for each — *against the tree* — +whether it actually refuses at the **extraction** layer (`generate/derivation/extract.py`) +or **downstream** (`clauses` / `compose` / `multistep` / `search` / `verify`). Then map +the minimal productions the **derivation/** composer still needs. Per the brief, this is +the audit feeding ADR-0207's "ADR-0179 extraction richness" execution lever; it touches +no code. + +> **Reader-disjointness caveat (load-bearing).** ADR-0179 enriches the **sealed, +> practice-only derivation reader** (`generate/derivation/*`). `chat/` does not import it, +> so none of this moves the serving metric. The serving `6/44/0` comes from the **disjoint +> candidate-graph reader** (`generate/math_candidate_graph.py`). A case can be +> serving-correct and derivation-starved at once (0003, 0024 below). This audit is about +> the derivation reader, as Task 3 specifies. + +--- + +## 1. The headline: ADR-0179's §Context table is stale + +ADR-0179 (dated 2026-05-28, when serving was `3/47/0`) diagnoses a thin extractor. The +tree has moved. `generate/derivation/extract.py`'s own module docstring + live probes +confirm the following are **already integrated**: + +| ADR-0179 sub-phase | In tree? | Evidence (live) | +|---|---|---| +| **EX-1** word-numbers (`three` → 3.0, hyphen compounds) | **LANDED** | `_WORD_QTY_RE`; `0024` extracts `(3.0,'times','three')` | +| **EX-2** currency/decimal + **bare-decimal grounding** | **LANDED** | `math_roundtrip.py:362` "ADR-0179 EX-2"; `_value_grounds("0.75")` → `True`; `0003` extracts `(0.75,'','0.75')` | +| **EX-4** list-unit inheritance (trailing unit) | **LANDED** | `_LIST_WITH_TRAILING_UNIT_RE` | +| **EX-5** sentence-final / unit-less number | **LANDED** | `_FINAL_NUMBER_RE` | +| **EX-6** hyphen-bonded number-unit (`25-foot`) | **LANDED** | `_HYPHEN_QTY_RE` (ADR-0163-F2) | +| Unit hygiene (function-word filter: `$0.75 each`→blank, not `each`) | **LANDED** | `_NON_UNIT_WORDS`; `0003`/`0024` blank the adjunct unit | +| **EX-3** multi-word units (`jumping jacks`, `stop signs`) | **DEFERRED** | `TestEX3StillDeferred` pins two documented traps (below) | + +So **EX-3 (multi-word units) is the only open extraction production.** It was deferred +deliberately, not overlooked — see §3. ADR-0207 should cite ADR-0179 with this +reconciliation, not the stale §Context framing. + +--- + +## 2. Per-case verification — extraction layer vs downstream + +Live extraction (`extract_quantities`) and end-to-end derivation composer +(`search_multiplicative` / `compose_sequential` / `search_chain`) at HEAD: + +### 0003 — `$0.75` erasers (gold 864) — **extraction CLOSED; ADR-0179's diagnosis is stale** +- Extracts `(48,'boxes')`, `(24,'erasers')`, `(0.75,'')` — the decimal **grounds** (EX-2). +- Derivation `search_chain` → **864.0 = gold.** The case the ADR says "refuses" now **solves** + in derivation. (Serving also correct, fast-path.) +- **Residual: none.** This case is a positive proof EX-2 landed and works end-to-end. + +### 0024 — Sidney jumping jacks (gold 438) — **purely extraction-blocked (EX-3 + head inherit)** +- Extracts `(20,'jumping')`, `(36,'')`, `(40,'')`, `(50,'')`, `(3.0,'times','three')`. + EX-1 caught `three`; but `"jumping jacks"` → `"jumping"` (multi-word truncation), and the + list states its unit **once at the head** (`"20 jumping jacks on Monday, 36 on Tuesday…"`), + so the trailing-unit EX-4 rule does not fire. Units are non-uniform (`jumping` vs `''`). +- Derivation: all composers → **None** (GB-2 same-unit list-sum never fires). +- **Residual: extraction** — a *head*-unit list-inheritance production + a bounded multi-word + unit. (Serving solves 0024 independently; this gap is derivation-only.) + +### 0016 — Rudolph stop signs (gold 2) — **both layers** +- Extracts `(2,'')`, `(5,'miles')`, `(3,'')`, `(17,'stop')` — `"stop signs"` → `"stop"` + (EX-3 truncation), and `"2 more than 5"` / `"3 less than 17"` are extracted as **separate + bare numbers**, not composed operands. +- Derivation `search_chain` → 510.0 (≠ gold 2); the self-verify gate refuses it (no wrong + commit — wrong=0 holds). The raw 510 is a non-committed candidate. +- **Residual: extraction (EX-3 `stop signs`) AND downstream** — word-arithmetic derived + operand (`N more/less than M`) and a `"per X"` division question target. Extraction alone + will not solve 0016. + +### 0033 — Rachel's father's age (gold 60) — **purely downstream** +- Extracts `(12,'years')`, `(7.0,'times','7')`, `(5,'years')`, `(25,'years')` — **complete.** + `"Rachel is 12 years old"` grounded cleanly (the `years old` postmodifier did not corrupt + it: only the first content word `years` is taken). +- Derivation: all composers → **None**. The block is the multi-step age chain + (grandfather = 7×12 = 84; mother = ½·84 = 42; father = 42+5 = 47) plus the **temporal + offset** `"when she is 25"` (Rachel +13 → father 60). +- **Residual: downstream only** (compose/multistep). Extraction needs nothing here. + +### Summary table + +| Case | Serving | Derivation @ HEAD | Block locus | Genuine residual | +|---|---|---|---|---| +| 0003 | correct | **solves 864** | — | none (EX-2 closed it) | +| 0024 | correct (438) | refuses | **extraction** | EX-3 multi-word + head-unit inheritance | +| 0016 | refused | gate-refused (raw 510) | **both** | EX-3 + word-arithmetic operand + division target | +| 0033 | refused | refuses | **downstream** | multi-step chain + temporal offset | + +**Of the four cases ADR-0179 names as extraction-starved, exactly one (0024) is purely an +extraction problem today.** 0003 is already solved, 0033 is purely downstream, 0016 needs +both. This is the single most important correction the audit surfaces: extraction richness +is largely *shipped*; the remaining derivation coverage is now mostly a **composition** +problem, not an extraction one. + +--- + +## 3. Why EX-3 was deferred (the two traps — preserved, not re-litigated) + +`extract.py`'s docstring + `TestEX3StillDeferred` (`tests/test_adr_0179_extract.py`) pin two +failure modes that defeat the tightest lookahead rule the ADR-0165 lexeme constraint admits: + +1. **Connective-crossing.** A greedy lowercase unit span eats the connective: + `"6 apples and 4 apples"` → unit `"apples and"`, regressing GB-2 same-unit detection. +2. **Postmodifier-adjective tails.** Even `digit + word + word + (?=terminator)` fires on + `"25 years old?"` → `"years old"` instead of `"years"`, regressing + `test_adr_0176_ms1_question_target`. Endemic across 0006/0033 and MS2 chain tests. + +Any EX-3 proposal must clear both without regressing those pinned tests. + +--- + +## 4. Minimal productions proposed (analysis only — no code) + +Ordered by leverage / safety. All are lexeme-level (ADR-0165), refuse-preferring +(over-extraction can only cost a *refusal* through the self-verify gate, never a wrong +commit), and each is independently shippable behind the existing derivation gate. + +### P1 — Head-unit list inheritance (unblocks 0024; sidesteps both EX-3 traps) +Symmetric to the landed trailing-unit EX-4, but for the unit stated **once at the list +head**: `" , , … "` → attach +`` to the unit-blank members `n1…nk`. Bound the head `` on the +right by a **closed adjunct stop set** `{on, at, in, per, each, of, and}` — so +`"20 jumping jacks on Monday"` yields unit `"jumping jacks"`, stopping cleanly before `on`. +This makes 0024 a uniform-unit list → GB-2 list-sum fires, **without** general multi-word +capture, so neither §3 trap is touched. Highest leverage, lowest risk. + +### P2 — Bounded 2-word content unit with a closed measurement-postmodifier stop list +The general `"stop signs"` shape (0016 extraction half). Allow exactly two content words +**only when** the second is not in a closed stop list of measurement postmodifiers +`{old, tall, long, wide, deep, high, away, apart, ago, …}` **and** not a connective. This is +the production the Track-C brief judged "too open-ended"; the audit's position is that the +stop set is *enumerable and closed* if scoped to measurement postmodifiers, and must ship +with `TestEX3StillDeferred` converted to a positive/negative matrix (the two traps become +must-refuse rows). Medium leverage, medium risk — gate it behind P1. + +### P3 — (explicitly **not** extraction — flagged, out of Task-3 scope) +0016's `"N more/less than M"` word-arithmetic operand and `"per X"` division question +target, and 0033's multi-step age chain + temporal `"when she is 25"` offset, are +**composition** concerns for `compose.py` / `multistep.py` / `search.py`, not `extract.py`. +The audit names them so they are not mistaken for extraction work: feeding richer quantities +will not solve 0016/0033 without these downstream productions. + +--- + +## 5. Bottom line for ADR-0207 + +- ADR-0179's "extraction is too thin" framing is **mostly resolved in-tree** (EX-1/2/4/5/6 + + unit hygiene landed). Cite the reconciled state, not the stale §Context table. +- The one open extraction lever is **EX-3**, and its highest-value, lowest-risk slice is + **P1 (head-unit list inheritance)** — it unblocks 0024 in derivation with no EX-3-trap + exposure. +- The broader Phase-5b coverage wall is now **composition** (P3), not extraction. ADR-0207's + two diagnosed levers (ADR-0179 extraction richness + ADR-0164 lexicon expansion) should be + read with this correction: extraction richness is near-complete; the residual lift is in + the comprehension/derivation composer, gated by self-verification (wrong=0 preserved). + +**Invariants observed by this audit:** read-only; train_sample reproduced live `6/44/0`; +no serving code touched; the no-ref `×` hazard remains refused (untouched). diff --git a/docs/decisions/ADR-0164-incremental-comprehension-reader.md b/docs/decisions/ADR-0164-incremental-comprehension-reader.md index 941336e7..25ed7d92 100644 --- a/docs/decisions/ADR-0164-incremental-comprehension-reader.md +++ b/docs/decisions/ADR-0164-incremental-comprehension-reader.md @@ -1,6 +1,6 @@ # ADR-0164 — Incremental Comprehension Reader (replaces regex sentence-template parsing) -**Status:** Partially implemented (Phase 1 + 2 shipped; eval delta pending lexicon expansion) +**Status:** Accepted (ratified by ADR-0207, 2026-06-03) — Phase 1+2 shipped; remainder per ADR-0207 §5 **Date:** 2026-05-26 **Author:** Shay **Anchor:** [[thesis-decoding-not-generating]] diff --git a/docs/decisions/ADR-0165-regex-scope-rule.md b/docs/decisions/ADR-0165-regex-scope-rule.md index 8ec6e26c..977cc6f5 100644 --- a/docs/decisions/ADR-0165-regex-scope-rule.md +++ b/docs/decisions/ADR-0165-regex-scope-rule.md @@ -1,6 +1,6 @@ # ADR-0165 — Regex Scope Rule: Lexemes Only, Never Grammar -**Status:** Proposed +**Status:** Accepted (ratified by ADR-0207, 2026-06-03) **Date:** 2026-05-26 **Author:** Shay **Anchor:** [[thesis-decoding-not-generating]] diff --git a/docs/decisions/ADR-0174-held-hypothesis-comprehension.md b/docs/decisions/ADR-0174-held-hypothesis-comprehension.md index 737c7a6c..37749c30 100644 --- a/docs/decisions/ADR-0174-held-hypothesis-comprehension.md +++ b/docs/decisions/ADR-0174-held-hypothesis-comprehension.md @@ -1,6 +1,6 @@ # ADR-0174 — Held-Hypothesis Comprehension with Lookback and In-Loop Contemplation -**Status:** Proposed +**Status:** Accepted (ratified by ADR-0207, 2026-06-03) **Date:** 2026-05-28 **Author:** Shay **Anchor:** [[thesis-decoding-not-generating]] diff --git a/docs/decisions/ADR-0178-compositional-structure.md b/docs/decisions/ADR-0178-compositional-structure.md index 0053a360..836587e8 100644 --- a/docs/decisions/ADR-0178-compositional-structure.md +++ b/docs/decisions/ADR-0178-compositional-structure.md @@ -1,6 +1,6 @@ # ADR-0178 — Compositional Structure: Comprehension-Guided Multi-Step Derivation (Gap B) -**Status:** Proposed +**Status:** Accepted (ratified by ADR-0207, 2026-06-03) **Date:** 2026-05-28 **Author:** Shay **Anchor:** [[thesis-decoding-not-generating]] diff --git a/docs/decisions/ADR-0179-extraction-richness.md b/docs/decisions/ADR-0179-extraction-richness.md index 5122f035..2c7940da 100644 --- a/docs/decisions/ADR-0179-extraction-richness.md +++ b/docs/decisions/ADR-0179-extraction-richness.md @@ -1,6 +1,6 @@ # ADR-0179 — Extraction Richness: feeding the comprehension composer real quantities -**Status:** Proposed +**Status:** Accepted (ratified by ADR-0207, 2026-06-03) **Date:** 2026-05-28 **Author:** Shay **Anchor:** [[thesis-decoding-not-generating]] diff --git a/docs/decisions/ADR-0207-gsm8k-substrate-ratification.md b/docs/decisions/ADR-0207-gsm8k-substrate-ratification.md new file mode 100644 index 00000000..d89424e6 --- /dev/null +++ b/docs/decisions/ADR-0207-gsm8k-substrate-ratification.md @@ -0,0 +1,206 @@ +# ADR-0207 — GSM8K Comprehension/Composition Substrate: Ratify · Freeze · Execute + +**Status:** Accepted (ratified 2026-06-03) +**Date:** 2026-06-03 +**Author:** Shay +**Anchor:** [[thesis-decoding-not-generating]] +**Consolidates (does not replace):** ADR-0164, ADR-0165, ADR-0174, +ADR-0178 (Compositional Structure), ADR-0179. +**Supersedes:** ADR-0163 §Phase B–E *prescription* and ADR-0136's regex +sentence-template prescriptions (refusal taxonomies preserved as evidence). + +--- + +## 1. This is not a new design + +It is the ratification of a design that already exists and is already built, plus the +closing of the fallback path that keeps re-opening it. It ends a process loop, not an +architectural gap: + +> a structured substrate gets proposed → never ratified or finished → the old regex path +> stays live in parallel → someone sees overfitting on the live regex path and proposes a +> "new" structured substrate that already exists → repeat. + +**If anyone (human or agent) proposes a "semantic operation grammar compiler," a "new IR," +or a "new reader" for GSM8K after this ADR: it is redundant with the five ADRs consolidated +here. Redirect to this document.** The next correct artifact on this surface is *execution +against the gates* (§6), not another design. + +## 2. Verified findings (origin/main `2cb0922`) + +Every claim below was reproduced read-only against the tree on 2026-06-03. + +**252 ADRs.** The relevant design is on the books, not missing. Actual status lines (cited as +they stand — these are *not* "design of record" labels until this ADR is ratified): + +| ADR | Title (verbatim) | Status (verbatim) | +|---|---|---| +| **0164** | Incremental Comprehension Reader (replaces regex sentence-template parsing) | *Partially implemented (Phase 1 + 2 shipped; eval delta pending lexicon expansion)* | +| **0165** | Regex Scope Rule: Lexemes Only, Never Grammar | *Proposed* | +| **0174** | Held-Hypothesis Comprehension with Lookback and In-Loop Contemplation | *Proposed* | +| **0178** | Compositional Structure: Comprehension-Guided Multi-Step Derivation (Gap B) | *Proposed* | +| **0179** | Extraction Richness | *Proposed* | +| 0136 | Statement Layer Corridor | *Active* — regex sentence-template prescription superseded by 0164 | + +- **ADR-0164** explicitly *"replaces regex sentence-template parsing"* and supersedes + ADR-0163 §Phase B–E + ADR-0136 (+ the .S.1–.S.4 sub-family). Verified verbatim. +- **ADR-0174** *deprecates* the per-category injector dispatch table at + `generate/recognizer_anchor_inject.py:233` **as the runtime admission path** — *"the + injectors become hypothesis-emitters within the held-hypothesis reader; they no longer + route admission via category lookup."* Verified verbatim. + +**The machinery is built, not theoretical.** All present and importable at HEAD: +`generate/comprehension/{lifecycle,state,lookback,constraint_propagation,contemplate,composition_registry}.py` +and `generate/derivation/{multistep,compose,accumulate,clauses,extract,pool}.py`. +Held-hypothesis primitives (`lookback.reevaluate`, `contemplate`) are invoked live inside +`generate/math_candidate_graph.py` (ADR-0174 Phase 3 lookback block, ~line 685–743). + +**Topology is fragmented into three coexisting pieces:** +1. **The regex recognizer/injector path** (`generate/recognizer_match.py` + + `generate/recognizer_anchor_inject.py`) — the **live serving admission path** + (imported by `math_candidate_graph.py` at lines 663, 685, 1004) and the overfitting vector. +2. **The held-hypothesis primitives** — bolted into the candidate-graph flow (above). +3. **The `derivation/` composer** — built and *already enriched* (see §3), but **disjoint + from serving**: `chat/` imports zero derivation modules, and the serving candidate-graph + path imports **exactly one** derivation symbol — + `generate.derivation.product_bridge.resolve_promotable_product` + (`math_candidate_graph.py:530`, the guarded product bridge behind the current lift). The + composer proper (`extract`/`clauses`/`compose`/`accumulate`/`multistep`/`search`/`verify`) + is **not reached from serving**. (`cue_precision/`, `proof_chain/`, and `derivation.model` + type imports are inert w.r.t. serving.) + +The reader-*as-admission-dispatch* was found inert and **retired in Phase 5a** (`3fd3172`, +ancestor of the prior HEAD `8327c6b`, −1,038 LOC); the primitives survived. So ADR-0174's +intended topology (**reader = the brain; injectors = emitters**) **was never realized — it +fragmented.** + +**Empirical signature of the unfinished state:** train_sample **6/44/0** (reproduced live); +sealed holdout **0/0/1319** (cited per `docs/claims_ledger.md` / ADR-0119.7 ciphertext — not +CI-reproducible). The live regex path solves a few train surface forms and transfers to +~nothing unseen. Safe failure (refuses, never wrong), but narrow-and-non-transferring. + +**Root cause of the wasted weeks is process, not architecture:** five `Proposed`/partial ADRs +never ratified or driven to completion, with the deprecated fallback left live. + +## 3. RATIFY — the design of record + +Upon ratification of this ADR, adopt the following as the single ratified GSM8K design (and +move ADR-0164/0165/0174/0178/0179 out of `Proposed`/limbo, this ADR serving as the +consolidating decision that cites them): + +> **Doctrine (decoding-not-generating).** GSM8K capability comes from a structured +> comprehension reader that builds typed hypotheses and a composer that assembles them into +> multi-step derivations. Regex recognizes **lexemes only** (currency, fractions, numerics) — +> **never** sentence/grammar structure (ADR-0165). The composer's incomplete productions +> **refuse structurally**, preserving `wrong=0`. + +ADR-0163 §Phase B–E and ADR-0136's regex sentence-template prescriptions are **superseded** +(their refusal taxonomies are kept as evidence; the regex *prescription* is dead — already +recorded in ADR-0164's "Supersedes in part" block). + +## 4. FREEZE — close the fallback + +The regex recognizer/injector path on the **serving candidate-graph** (`recognizer_match.py` ++ `recognizer_anchor_inject.py`) is **lexeme-recognition + refusal-only.** + +- **Permitted:** refusal-only `wrong=0` guards — the no-reference ` times` completeness + tripwire (composition-capability-scope §9; `tests/test_candidate_graph_completeness_guard.py`) + is the model: it *removes* wrong answers, it cannot overfit. +- **Forbidden:** any new *positive* capability as a bespoke recognizer/injector branch. + Positive capability flows only through the comprehension/composition machinery (§5). + +This is consistent with ADR-0174's own deprecation (the per-category injector dispatch is no +longer the admission path) and is the line that, left open, re-creates the loop. + +## 5. CONVERGE + EXECUTE — feed the built machinery (no new architecture) + +The convergence target is ADR-0174's intent: the structured machinery is the brain; regex is +demoted to lexemes + refusal. Realize it by **connecting and feeding what is already built**, +not rewriting it. + +> **Extraction is not the open lever (tree-verified, 2026-06-03).** An earlier draft named +> *extraction richness* (ADR-0179) as primary. That is **stale**: EX-1/EX-2/EX-4/EX-5/EX-6 +> + unit-hygiene have **landed** in `generate/derivation/extract.py`; only EX-3 (multi-word +> units) is deliberately deferred with documented traps (`TestEX3StillDeferred` pins them), +> and ADR-0179's §Context "thin extractor" table no longer matches the code — e.g. case 0003 +> now solves 864 end-to-end through the derivation composer. See +> [extraction-richness-audit-2026-06-03](../analysis/extraction-richness-audit-2026-06-03.md). + +Execute these three levers, in order: + +1. **WIRING — the crux.** Connect the disjoint `generate/derivation/` reader (where the + enriched extractor *and* the ADR-0178 composer already live) into the **serving + candidate-graph path**, so its output counts toward the live metric. The derivation + machinery is built and enriched but disjoint; until it feeds serving, none of its richness + moves 6/44/0. `product_bridge` (§2) is the lone existing tendril and the proof the wiring + pattern works. **This is wiring, not new design**, and it is the central task — not a + preliminary. +2. **COMPOSITION (ADR-0178) — the actual wall.** Complete the derivation composer's + multi-step assembly: which quantities group, via which ops, in what order (the R4/R5/R6 + lever). This is the genuine research risk; extraction already feeds it. +3. **LEXICON (ADR-0164) — category/lexicon expansion.** So the serving reader stops emitting + `ReaderRefusal` on the first unknown token (the "eval delta pending lexicon expansion" + ADR-0164 itself flagged). + +**EX-3 multi-word units = narrow mop-up only**, addressed minimally if at all — e.g. +head-unit list inheritance for the 0024-class, sidestepping the deferred EX-3 traps +(see the audit's P1). **Not a lever.** + +Positive capability = **wiring + composer enrichment + shared lexicon** (composable, +generalizing). Never a per-shape branch. + +## 6. Acceptance gates (`wrong=0` is the governor) + +Every step must hold all of: + +1. train_sample stays **`wrong=0`** (6/44/0 or better — `correct` may only rise). +2. The no-reference ` times` hazard (composition-capability-scope §9) stays **refused**. +3. No partial graph admits while any source quantity is unbound (the completeness leg). +4. Progress is measured on the composition-typed **validation sub-corpus** + ([`evals/gsm8k_math/composition_validation/v1/`](../../evals/gsm8k_math/composition_validation/v1/cases.jsonl), + now **22 cases**) *and* on the **sealed 1,319** — the sealed number is the real bar. A + train_sample gain that does not move held-out is overfitting in a tidier wrapper and does + **not** count. +5. Each gain lands as machinery enrichment (§5), audited to confirm it is not a new + recognizer/injector branch (the §4 freeze). + +**Phase 5b is gated on the 22-case composition corpus AND the sealed 1,319 together.** + +## 7. What this explicitly is NOT + +Not a new grammar compiler. Not a new IR. Not a new reader. Not a new ADR-numbered +architecture beyond this consolidation. It is: ratify the five that exist, freeze the +fallback, feed the built machinery, measure on held-out. Any proposal to build a "new" +structured front-end for GSM8K is this, restated — point it here (§1). + +## 8. Why this ends the loop + +The loop had three drivers: **propose** (Proposed-limbo), **leave the fallback live** (an +overfitting vector to rediscover), and **re-propose** (perpetual design instead of +finish-to-instrument). This ADR removes all three at once — **ratify** (§3), **freeze** (§4), +**finish-to-instrument** (§5–§6, measured on the sealed set, not on more proposals). + +## 9. Documentation-hygiene flags (for the operator, non-blocking) + +- **Duplicate ADR-0178 number.** Two files carry `ADR-0178`: + `ADR-0178-compositional-structure.md` (the Gap-B composer this ADR consolidates) and + `ADR-0178-GB3b-referent-accumulation-scope.md` (a scope-only sub-phase). They should be + renumbered/renamed for unambiguous citation. +- **ADR-0179 §Context drift.** Its "thin extractor" table predates the landed + EX-1/2/4/5/6 + unit-hygiene work; reconcile it to the tree (the audit in §5 is the + reconciliation) when 0179 is moved out of `Proposed`. + +## 10. Cross-references + +- **Consolidates:** [ADR-0164](./ADR-0164-incremental-comprehension-reader.md), + [ADR-0165](./ADR-0165-regex-scope-rule.md), + [ADR-0174](./ADR-0174-held-hypothesis-comprehension.md), + [ADR-0178 Compositional Structure](./ADR-0178-compositional-structure.md), + [ADR-0179](./ADR-0179-extraction-richness.md). +- **Supersedes:** ADR-0163 §Phase B–E prescription, ADR-0136 regex sentence-templates. +- **Instruments:** the 22-case composition validation corpus + (`evals/gsm8k_math/composition_validation/v1/`) and the sealed holdout + (`evals/gsm8k_math/holdouts/v1/cases.jsonl.age`, `0/0/1319`). +- **Evidence:** [composition-capability-scope](../analysis/composition-capability-scope.md), + [extraction-richness-audit-2026-06-03](../analysis/extraction-richness-audit-2026-06-03.md). +- **Thesis:** [[thesis-decoding-not-generating]]. diff --git a/evals/gsm8k_math/composition_validation/v1/cases.jsonl b/evals/gsm8k_math/composition_validation/v1/cases.jsonl index 11756b69..484c34fc 100644 --- a/evals/gsm8k_math/composition_validation/v1/cases.jsonl +++ b/evals/gsm8k_math/composition_validation/v1/cases.jsonl @@ -18,3 +18,5 @@ {"baseline_answer":28,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0018","composition":"compare_mult","gate":"baseline","gold":28,"note":"Referenced multiplicative comparison already solves; protects against over-broad no-reference refusal.","question":"Tom has 7 apples. Jerry has 3 times as many apples as Tom. How many apples do they have together?","source":"guard:n_times_with_reference:3_as_many_as_tom","target_verdict":"solve"} {"baseline_answer":1200,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0019","composition":"additive","gate":"baseline","gold":1200,"note":"Stated-sum contrastive control for the derived-symbol Nicole/Cindy form.","question":"Nicole has 400 cards. Cindy has 800 cards. How many cards do they have together?","source":"analysis:composition-capability-scope:R1_contrastive_control","target_verdict":"solve"} {"baseline_answer":450,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0020","composition":"product_additive","gate":"baseline","gold":450,"note":"Existing guarded product bridge correct case; protects current 6/44/0 lift.","question":"John is lifting weights. He bench presses 15 pounds for 10 reps and does 3 sets. How much total weight does he move?","source":"gsm8k_train_sample:0021","target_verdict":"solve"} +{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0021","composition":"R4","gate":"5b-R4","gold":4,"note":"Residual-to-target give-away: needed-to-give = start - desired remainder, minus already given (Jane 5, James 5+2).","question":"Martha has 20 apples. She decided to split them among her friends. Jane got 5 apples from her, and James got 2 more than Jane. How many more apples would Martha need to give away to be left with only 4 of them?","source":"gsm8k_train_sample:0035","target_verdict":"solve"} +{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0022","composition":"R5","gate":"5b-R5","gold":38,"note":"Multi-step duration: train leg = 2x subway leg (scalar of prior stage), then sum of the three legs.","question":"Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride, and then bikes the remaining distance for 8 hours. What's the total time he takes to reach the Bronx from Manhattan?","source":"gsm8k_train_sample:0015","target_verdict":"solve"} diff --git a/evals/gsm8k_math/composition_validation/v1/contract.md b/evals/gsm8k_math/composition_validation/v1/contract.md index 0e163711..fee81d31 100644 --- a/evals/gsm8k_math/composition_validation/v1/contract.md +++ b/evals/gsm8k_math/composition_validation/v1/contract.md @@ -7,7 +7,8 @@ coverage may rise only while `wrong == 0` remains true. ## Scope -- 20 cases total. +- 22 cases total (4 baseline controls, 7 permanent hard-negatives, 11 future + positives). - Positive cases use `target_verdict = solve` and a non-baseline `gate`; they refuse today and should flip only when that gate's capability slice lands. - Hard-negative cases use `target_verdict = refuse` and `gate = permanent`; they @@ -46,15 +47,27 @@ Each JSONL row has these fields: reviewed ADR explicitly changes its decidable regime. 5. Progress metric: count rows where `target_verdict = solve` and the current answer equals `gold`; this starts at the baseline controls and may increase - only when invariants 1-4 still pass. + only when invariants 1-4 still pass. The solved-positive count is monotone + non-decreasing across Phase 5b slices. +6. Dataset-sourced golds: every `gsm8k_train_sample:*` row's `gold` equals that + case's `answer_numeric` in `evals/gsm8k_math/train_sample/v1/cases.jsonl`, + verbatim. Golds are never hand-computed — a wrong gold is a `wrong=0` hazard + in the instrument itself. (`guard:*` / `analysis:*` rows are seeded probes, + not dataset cases, and carry their stated gold.) +7. Baseline fields are diagnostic, not asserted targets: + `baseline_verdict` / `baseline_answer` / `baseline_branches_enumerated` + record the tree state at creation for drift detection on non-positive rows. + They are not a pass/fail target for positive rows — a 5b flip updates them in + lockstep with the snapshot assertion. ## Baseline At creation on `origin/main` lineage after PR #534 plus the docs-only runway -correction, the intended baseline is: +correction, and extended with the second R4/R5 positives (cv-0021/cv-0022), the +intended baseline is: - 4 solve -- 16 refuse +- 18 refuse - 0 wrong The corpus deliberately includes both future positives and permanent diff --git a/tests/test_composition_validation_corpus.py b/tests/test_composition_validation_corpus.py index b55617a3..633fe83c 100644 --- a/tests/test_composition_validation_corpus.py +++ b/tests/test_composition_validation_corpus.py @@ -51,10 +51,10 @@ _REQUIRED_FIELDS = frozenset( } ) -_EXPECTED_TOTAL = 20 +_EXPECTED_TOTAL = 22 _EXPECTED_BASELINE_CONTROLS = 4 _EXPECTED_PERMANENT = 7 -_EXPECTED_FUTURE_POSITIVE = 9 +_EXPECTED_FUTURE_POSITIVE = 11 def _load_cases() -> list[dict]: @@ -77,7 +77,7 @@ def _num_eq(a, b) -> bool: def test_corpus_structure() -> None: - """20 unique, fully-specified rows with the expected gate partition.""" + """22 unique, fully-specified rows with the expected gate partition.""" assert len(_CASES) == _EXPECTED_TOTAL assert len(set(_IDS)) == _EXPECTED_TOTAL, "duplicate case_id" for c in _CASES: @@ -155,7 +155,7 @@ def test_frozen_baseline_fields_match_tree(case: dict) -> None: def test_current_baseline_snapshot() -> None: - """Current aggregate is 4 solve / 16 refuse / 0 wrong. + """Current aggregate is 4 solve / 18 refuse / 0 wrong. This is the single assertion a Phase 5b slice updates when it flips a positive (refuse -> solve); the forever-invariants above do not change. @@ -170,8 +170,49 @@ def test_current_baseline_snapshot() -> None: else: refuse += 1 assert wrong == 0 - assert (solve, refuse) == (4, 16), ( + assert (solve, refuse) == (4, 18), ( f"snapshot moved to {solve} solve / {refuse} refuse — if a Phase 5b " f"slice landed, update this expectation and the affected rows' " f"baseline fields in lockstep" ) + + +_TRAIN_SAMPLE_PATH = ( + _REPO_ROOT / "evals" / "gsm8k_math" / "train_sample" / "v1" / "cases.jsonl" +) + + +def _load_train_sample_answers() -> dict[str, object]: + answers: dict[str, object] = {} + for line in _TRAIN_SAMPLE_PATH.read_text(encoding="utf-8").splitlines(): + if not line.strip(): + continue + rec = json.loads(line) + answers[rec["case_id"]] = rec["answer_numeric"] + return answers + + +def test_dataset_golds_match_answer_numeric() -> None: + """Contract invariant 6: every ``gsm8k_train_sample:*`` row's ``gold`` equals + that case's ``answer_numeric`` verbatim. A hand-computed (and thus possibly + wrong) gold would be a ``wrong=0`` hazard inside the measurement instrument + itself, so this is a real proof obligation, not decoration. + + ``guard:*`` / ``analysis:*`` rows are seeded probes, not dataset cases, and + are intentionally excluded. + """ + answers = _load_train_sample_answers() + checked = 0 + for case in _CASES: + source = case["source"] + if not source.startswith("gsm8k_train_sample:"): + continue + idx = source.split(":", 1)[1] + ts_id = f"gsm8k-train-sample-v1-{idx}" + assert ts_id in answers, f"{case['case_id']}: unknown train_sample source {source}" + assert _num_eq(case["gold"], answers[ts_id]), ( + f"{case['case_id']}: gold {case['gold']!r} != dataset answer_numeric " + f"{answers[ts_id]!r} for {source} — hand-computed gold is a wrong=0 hazard" + ) + checked += 1 + assert checked >= 8, f"expected ≥8 dataset-sourced rows, checked {checked}"