core/docs/decisions/ADR-0136-statement-layer-corridor.md
Shay e705f27d2e
docs(ADR-0164,0165): incremental comprehension reader + regex scope rule (#317)
Replace the regex sentence-template front-end of the math admissibility
layer with an incremental compositional reader. Lock the architectural
boundary that regex is permitted only at the lexeme level, never as
sentence-structure templates.

ADR-0164 (Proposed) — Incremental Comprehension Reader. Word-by-word
state accumulation over a closed set of semantic categories, with the
operational lexicon living as a pack-shaped data artifact under
language_packs/data/en_core_math_v1/. Reader output type matches the
existing regex parser's output, so the binding-graph admissibility
(ADR-0132/0133/0134/0135), the solver (ADR-0116), and the verifier
(ADR-0117) stay unchanged. wrong=0 is preserved by construction —
the reader produces inputs to the existing admissibility gate, not a
bypass around it. Phased coexistence with the regex layer during
transition; regex sentence templates removed in Phase 3.

ADR-0165 (Proposed) — Regex Scope Rule. Structural invariant: regex
matches one piece of orthographic material with a closed rule
(currency literal, fraction literal, percentage, time-amount, closed
unit-noun sets), never a sentence shape. Lexeme-primitive registry is
closed and grown through the same contemplation -> proposal -> HITL
review corridor that grows vocabulary (ADR-0150 / 0152 / 0155 / 0161).
The engine acquires new recognition tools through reviewed teaching,
not through operator edits to parser code.

ADR-0163's diagnosis (front-end is the bottleneck) is reaffirmed.
Its Phase B-E prescription (regex DerivedRecognizers via
recognizer_match.py) is partially superseded by ADR-0164. ADR-0136
and its S-family (S.1 / S.2 / S.3 / S.4) have the same disposition:
regex sentence-template prescription superseded; empirical refusal
taxonomies and closed-set vocabulary preserved as lexicon seed.
The HITL corridor architecture is preserved; what flows through it
changes from regex recognizers to lexicon entries, categories, and
lexeme primitives.

Session log SESSION-2026-05-26-comprehension-reader.md captures the
narrative of how this decision emerged from the post-D.2 train-sample
baseline review (correct=3 refused=47 wrong=0, 34/47 refusals at the
question gate).

No runtime code changes. ADRs only.
2026-05-26 19:23:05 -07:00

6.3 KiB
Raw Blame History

ADR-0136 — Statement-Layer Corridor: Graduated GSM8K Admission via Parser Extension

Status: Active — Regex sentence-template prescription superseded by ADR-0164 (2026-05-26). Empirical taxonomies preserved. Date: 2026-05-23 Author: CORE agents + reviewers Parent: ADR-0131.G Superseded in part by: ADR-0164 — Incremental Comprehension Reader, ADR-0165 — Regex Scope Rule


Amendment 2026-05-26 — Regex prescription superseded

This corridor's production mechanism — adding regex sentence-template patterns to generate/math_candidate_parser.py per S-stage sub-ADR — is superseded by ADR-0164's incremental comprehension reader. The closed-set vocabulary collected by each S-stage (verb lists, mass-noun lists, name lists, unit-noun lists) is preserved as seed input to the new operational lexicon (ADR-0164 §Decision §1). The empirical refusal taxonomies (refusal_taxonomy_v*.json) are preserved as input evidence for category and primitive development. The regex patterns themselves are scheduled for removal during ADR-0164 Phase 3.

Per ADR-0165 — Regex Scope Rule, regex remains permitted at the lexeme-primitive level (currency literal, fraction literal, etc.) and forbidden at the sentence-structure level.


Depends on: ADR-0115, ADR-0116, ADR-0117


Context

ADR-0131.G pinned the GSM8K coverage probe at 0/50 admission and established admitted_wrong == 0 as a standing architectural invariant. The G.x capability axes (G.1G.5) have landed on main; as of ADR-0131.5 (probe retirement), the GSM8K probe is no longer a per-iteration gate — it activates only when a new iteration claims Δadmission_rate ≥ 0.02.

A taxonomy pass over all 50 refused cases (S.0) produced the following breakdown:

Primary barrier Count Notes
context_filler 23 Narrative scene-setters; parser's refusal is correct
compound_statement 5 Two operations in one sentence
rate/capacity/price class 4 Direct targets of S.1
distributive_multiply 1 (+5 secondary) N bags × M items each
diverse long-tail 17 Age anchors, goal statements, multi-step chains, etc.

The 23 context-filler cases will not be addressed by statement-layer work alone — they need semantic classification of scene-setting sentences, which is a separate architectural concern. The safety rail stands: if a sentence cannot parse into a numeric initial-state candidate, the problem is refused.

The G.x micro-extension approach has been retired. Each G.x axis targeted one regex shape in isolation; the return per axis was tiny and the axes were not composable into a coherent parser narrative. The Statement-Layer Corridor replaces G.x with a phased taxonomy-driven program that:

  1. Classifies all refusals before writing any code.
  2. Targets the highest-signal unlockable barriers first.
  3. Ships each phase as a self-contained unit with a curated axis lane, wrong == 0 gate, and honest GSM8K delta.
  4. Never attempts to bypass the context-filler safety rail.

Decision

Organize parser extension work as a corridor of phases rooted at ADR-0136:

Phase ADR Scope Primary barrier targeted
S.0 (this doc) Taxonomy — deterministic classification of 50 refused cases
S.1 ADR-0136.S.1 Rate/event statements — capacity-rate + earnings-rate shapes rate class (≤4 cases)
S.2 ADR-0136.S.2 Temporal statements — time anchors, duration expressions time/age long-tail
S.3 ADR-0136.S.3 Compound statements — two operations in one sentence compound_statement (5 cases)
S.4 ADR-0136.S.4 Coreference — pronoun + ellipsis resolution across sentences cross-sentence barriers

Phases S.2S.4 are deferred; scope and sequencing will be revisited after S.1 lands.


Taxonomy (S.0)

Stored at: evals/gsm8k_math/train_sample/v1/refusal_taxonomy.json

Schema v1. Each record carries:

  • case_id — GSM8K case identifier
  • primary_barrier — the single barrier that causes refusal even if all others were resolved
  • secondary_barriers — additional barriers present in the problem
  • notes — free-text rationale

Key finding. The case with the shallowest barrier is gsm8k-0014:

Bob can shuck 10 oysters in 5 minutes.
How many oysters can he shuck in 2 hours?

This is a single-statement capacity-rate problem with a pronoun question. It is the proof case for S.1: it must admit with answer == 240.0.


Invariants

These invariants are non-negotiable across all corridor phases:

  • admitted_wrong == 0 — no GSM8K case is admitted with a wrong answer.
  • Context-filler safety rail — sentences without parseable numeric initial state are refused; no soft-fail or skip-and-continue.
  • Honest delta — each phase's PR body states the exact pre/post GSM8K admission count; no rounding, no "approximately".
  • No solver/graph/verifier changes — rate-path short-circuits and new extractors live in math_candidate_parser.py and math_candidate_graph.py only.

Consequences

  • The G.x capability axis namespace is closed. New axes use the S.x naming convention.
  • Each S.x phase ships its own curated axis lane at evals/math_capability_axes/S<N>_<name>/v1/cases.jsonl.
  • The GSM8K probe re-activates when a phase claims ≥1 new admission (Δadmission_rate ≥ 0.02); otherwise it stays retired per ADR-0131.5.
  • S.1 is the only phase that can honestly claim unlocking the context-filler-gated cases once sentence-semantic classification lands (out of S.x scope).

Deferred

  • Context-filler gated problems (23 cases) — requires semantic classification of narrative scene-setting sentences; architecturally separate from statement parsing.
  • Conditional branching (if she works more than 8 hours) — needs branching semantics in the solver.
  • Percentage/interest rates — needs decimal arithmetic extension.
  • Multi-statement rate problems (duration in separate sentence from capacity) — needs coreference, addressed in S.4.