core/docs/analysis/extraction-richness-audit-2026-06-03.md
Shay 94bf1be1bc docs: ratify ADR-0207 — GSM8K comprehension/composition substrate
Consolidating ratification of the GSM8K design of record. Ratify the built
comprehension/derivation substrate, freeze the serving regex recognizer/
injector path to lexemes + refusal-only, pin Phase 5b execution to
WIRING -> COMPOSITION -> LEXICON.

- ADR-0207: new consolidating decision (Accepted, ratified 2026-06-03).
  Supersedes ADR-0163 §Phase B-E + ADR-0136 regex sentence-template
  prescriptions. Freeze + wrong=0 gates (22-case corpus + sealed 1,319).
- ADR-0164/0165/0174/0178/0179: -> Accepted (ratified by ADR-0207,
  2026-06-03). 0164 keeps its implementation clause (Phase 1+2 shipped;
  remainder per §5) so Accepted != fully built.
- composition_validation/v1: 20 -> 22 cases (2nd R4/R5 positives,
  dataset-sourced golds), +contract invariants 6-7, +dataset-gold test.
  Baseline 4/18/0; 47 passed.
- docs/analysis: extraction-richness audit (read-only) reconciling
  ADR-0179 to the tree (EX-1/2/4/5/6 landed; EX-3 deferred).

Non-serving (evals/docs/tests only). train_sample 6/44/0 unchanged;
no-ref <N> times hazard stays refused. GB3b/0136 untouched.
2026-06-03 19:42:47 -07:00

10 KiB
Raw Permalink Blame History

Extraction-Richness Audit — ADR-0179 named cases, reconciled to the shipped tree

Status: Analysis only. No serving, eval, or code edits. Every claim below reproduced live against origin/main @ 2cb0922 via read-only extract_quantities / derivation-composer / parse_and_solve probes.

Scope. Open ADR-0179 — Extraction Richness, read the exact cases its §Context table names as starved by the thin (\d+(?:\.\d+)?)\s+([a-z]+) extractor, and verify for each — against the tree — whether it actually refuses at the extraction layer (generate/derivation/extract.py) or downstream (clauses / compose / multistep / search / verify). Then map the minimal productions the derivation/ composer still needs. Per the brief, this is the audit feeding ADR-0207's "ADR-0179 extraction richness" execution lever; it touches no code.

Reader-disjointness caveat (load-bearing). ADR-0179 enriches the sealed, practice-only derivation reader (generate/derivation/*). chat/ does not import it, so none of this moves the serving metric. The serving 6/44/0 comes from the disjoint candidate-graph reader (generate/math_candidate_graph.py). A case can be serving-correct and derivation-starved at once (0003, 0024 below). This audit is about the derivation reader, as Task 3 specifies.


1. The headline: ADR-0179's §Context table is stale

ADR-0179 (dated 2026-05-28, when serving was 3/47/0) diagnoses a thin extractor. The tree has moved. generate/derivation/extract.py's own module docstring + live probes confirm the following are already integrated:

ADR-0179 sub-phase In tree? Evidence (live)
EX-1 word-numbers (three → 3.0, hyphen compounds) LANDED _WORD_QTY_RE; 0024 extracts (3.0,'times','three')
EX-2 currency/decimal + bare-decimal grounding LANDED math_roundtrip.py:362 "ADR-0179 EX-2"; _value_grounds("0.75")True; 0003 extracts (0.75,'','0.75')
EX-4 list-unit inheritance (trailing unit) LANDED _LIST_WITH_TRAILING_UNIT_RE
EX-5 sentence-final / unit-less number LANDED _FINAL_NUMBER_RE
EX-6 hyphen-bonded number-unit (25-foot) LANDED _HYPHEN_QTY_RE (ADR-0163-F2)
Unit hygiene (function-word filter: $0.75 each→blank, not each) LANDED _NON_UNIT_WORDS; 0003/0024 blank the adjunct unit
EX-3 multi-word units (jumping jacks, stop signs) DEFERRED TestEX3StillDeferred pins two documented traps (below)

So EX-3 (multi-word units) is the only open extraction production. It was deferred deliberately, not overlooked — see §3. ADR-0207 should cite ADR-0179 with this reconciliation, not the stale §Context framing.


2. Per-case verification — extraction layer vs downstream

Live extraction (extract_quantities) and end-to-end derivation composer (search_multiplicative / compose_sequential / search_chain) at HEAD:

0003 — $0.75 erasers (gold 864) — extraction CLOSED; ADR-0179's diagnosis is stale

  • Extracts (48,'boxes'), (24,'erasers'), (0.75,'') — the decimal grounds (EX-2).
  • Derivation search_chain864.0 = gold. The case the ADR says "refuses" now solves in derivation. (Serving also correct, fast-path.)
  • Residual: none. This case is a positive proof EX-2 landed and works end-to-end.

0024 — Sidney jumping jacks (gold 438) — purely extraction-blocked (EX-3 + head inherit)

  • Extracts (20,'jumping'), (36,''), (40,''), (50,''), (3.0,'times','three'). EX-1 caught three; but "jumping jacks""jumping" (multi-word truncation), and the list states its unit once at the head ("20 jumping jacks on Monday, 36 on Tuesday…"), so the trailing-unit EX-4 rule does not fire. Units are non-uniform (jumping vs '').
  • Derivation: all composers → None (GB-2 same-unit list-sum never fires).
  • Residual: extraction — a head-unit list-inheritance production + a bounded multi-word unit. (Serving solves 0024 independently; this gap is derivation-only.)

0016 — Rudolph stop signs (gold 2) — both layers

  • Extracts (2,''), (5,'miles'), (3,''), (17,'stop')"stop signs""stop" (EX-3 truncation), and "2 more than 5" / "3 less than 17" are extracted as separate bare numbers, not composed operands.
  • Derivation search_chain → 510.0 (≠ gold 2); the self-verify gate refuses it (no wrong commit — wrong=0 holds). The raw 510 is a non-committed candidate.
  • Residual: extraction (EX-3 stop signs) AND downstream — word-arithmetic derived operand (N more/less than M) and a "per X" division question target. Extraction alone will not solve 0016.

0033 — Rachel's father's age (gold 60) — purely downstream

  • Extracts (12,'years'), (7.0,'times','7'), (5,'years'), (25,'years')complete. "Rachel is 12 years old" grounded cleanly (the years old postmodifier did not corrupt it: only the first content word years is taken).
  • Derivation: all composers → None. The block is the multi-step age chain (grandfather = 7×12 = 84; mother = ½·84 = 42; father = 42+5 = 47) plus the temporal offset "when she is 25" (Rachel +13 → father 60).
  • Residual: downstream only (compose/multistep). Extraction needs nothing here.

Summary table

Case Serving Derivation @ HEAD Block locus Genuine residual
0003 correct solves 864 none (EX-2 closed it)
0024 correct (438) refuses extraction EX-3 multi-word + head-unit inheritance
0016 refused gate-refused (raw 510) both EX-3 + word-arithmetic operand + division target
0033 refused refuses downstream multi-step chain + temporal offset

Of the four cases ADR-0179 names as extraction-starved, exactly one (0024) is purely an extraction problem today. 0003 is already solved, 0033 is purely downstream, 0016 needs both. This is the single most important correction the audit surfaces: extraction richness is largely shipped; the remaining derivation coverage is now mostly a composition problem, not an extraction one.


3. Why EX-3 was deferred (the two traps — preserved, not re-litigated)

extract.py's docstring + TestEX3StillDeferred (tests/test_adr_0179_extract.py) pin two failure modes that defeat the tightest lookahead rule the ADR-0165 lexeme constraint admits:

  1. Connective-crossing. A greedy lowercase unit span eats the connective: "6 apples and 4 apples" → unit "apples and", regressing GB-2 same-unit detection.
  2. Postmodifier-adjective tails. Even digit + word + word + (?=terminator) fires on "25 years old?""years old" instead of "years", regressing test_adr_0176_ms1_question_target. Endemic across 0006/0033 and MS2 chain tests.

Any EX-3 proposal must clear both without regressing those pinned tests.


4. Minimal productions proposed (analysis only — no code)

Ordered by leverage / safety. All are lexeme-level (ADR-0165), refuse-preferring (over-extraction can only cost a refusal through the self-verify gate, never a wrong commit), and each is independently shippable behind the existing derivation gate.

P1 — Head-unit list inheritance (unblocks 0024; sidesteps both EX-3 traps)

Symmetric to the landed trailing-unit EX-4, but for the unit stated once at the list head: "<n0> <UNIT-PHRASE> <adjunct>, <n1> <adjunct>, … <nk> <adjunct>" → attach <UNIT-PHRASE> to the unit-blank members n1…nk. Bound the head <UNIT-PHRASE> on the right by a closed adjunct stop set {on, at, in, per, each, of, and} — so "20 jumping jacks on Monday" yields unit "jumping jacks", stopping cleanly before on. This makes 0024 a uniform-unit list → GB-2 list-sum fires, without general multi-word capture, so neither §3 trap is touched. Highest leverage, lowest risk.

P2 — Bounded 2-word content unit with a closed measurement-postmodifier stop list

The general "stop signs" shape (0016 extraction half). Allow exactly two content words only when the second is not in a closed stop list of measurement postmodifiers {old, tall, long, wide, deep, high, away, apart, ago, …} and not a connective. This is the production the Track-C brief judged "too open-ended"; the audit's position is that the stop set is enumerable and closed if scoped to measurement postmodifiers, and must ship with TestEX3StillDeferred converted to a positive/negative matrix (the two traps become must-refuse rows). Medium leverage, medium risk — gate it behind P1.

P3 — (explicitly not extraction — flagged, out of Task-3 scope)

0016's "N more/less than M" word-arithmetic operand and "per X" division question target, and 0033's multi-step age chain + temporal "when she is 25" offset, are composition concerns for compose.py / multistep.py / search.py, not extract.py. The audit names them so they are not mistaken for extraction work: feeding richer quantities will not solve 0016/0033 without these downstream productions.


5. Bottom line for ADR-0207

  • ADR-0179's "extraction is too thin" framing is mostly resolved in-tree (EX-1/2/4/5/6 + unit hygiene landed). Cite the reconciled state, not the stale §Context table.
  • The one open extraction lever is EX-3, and its highest-value, lowest-risk slice is P1 (head-unit list inheritance) — it unblocks 0024 in derivation with no EX-3-trap exposure.
  • The broader Phase-5b coverage wall is now composition (P3), not extraction. ADR-0207's two diagnosed levers (ADR-0179 extraction richness + ADR-0164 lexicon expansion) should be read with this correction: extraction richness is near-complete; the residual lift is in the comprehension/derivation composer, gated by self-verification (wrong=0 preserved).

Invariants observed by this audit: read-only; train_sample reproduced live 6/44/0; no serving code touched; the no-ref <N>× hazard remains refused (untouched).