60043973b0
Extend the comprehension reader from question-only scope to whole- problem scope. Phase 1 (Brief 8 / #326) implemented question_frame; this brief implements initial_state_frame, operation_frame, and descriptive_frame, plus finalize() projection into a strict ADR-0115 MathProblemGraph. Architecturally correct under ADR-0164.3; not yet productive on GSM8K train_sample. Below-floor measurement documented; specific bottlenecks tabled for Phase 2.1 follow-up. What landed - Frame-opener dispatch in lifecycle.py for the three new statement frames, plus rule handlers (_rule_op_*, _rule_preframe_*, _rule_descriptive_*). - finalize(state) -> MathProblemGraph | ReaderRefusal: pure projection with closure checks (entity registry non-empty, unknown target bound, every op/initial references a known entity, Decimal precision projects losslessly). - _classify extended to 3-tuple (category, surface, decimal_value) with possessive strip retry. Brief 8.2's sentence-initial lookup-first + gender-skip preserved AND extended to mid-sentence (gender is enrichment everywhere, never admission). - Whole-problem coexistence dispatch in math_candidate_graph.py (config.comprehension_reader_questions=True): reader attempts the whole problem; on any ReaderRefusal falls through to existing regex parser. All-or-nothing per the brief. - Lexicon expansion (carried into renamed proper_noun_gender_* files): +2 accumulation_verb (adopt, invest), +2 currency_unit_noun (dollar, cent), +6 capacity_verb (fill, lift, play, work, finish, drive), +5 female names (allison, brooke, jan, marion, sidney), +14 male names (bart, fernando, georgie, jake, jed, jeremie, jose, orlando, rex, rudolph, steve, troy, xavier, yun), +numerous count_unit_noun, drain_token, time_unit_noun. - ADR-0164.4-phase2-statement-frame-reader.md — the architectural rationale and acceptance contract. Measurement (reader_phase2_delta.json): flag-OFF: correct=3 refused=47 wrong=0 flag-ON: correct=3 refused=47 wrong=0 delta: 0/0/0 Below the brief's floor of correct >= 4. Architecture is sound — the reader admits cases as graphs when the structure resolves, refuses cleanly otherwise, preserves wrong=0 across both flag states. Bottleneck table (from per-case attribution): count refusal_class dominant cause ----- ---------------------- ------------------------------------ 18 incomplete_operation multi-quantity ops; no-quantity op 11 unknown_word "hundred", "presently", "one-hour", non-math verbs (compound numerics, lexicon gaps) 6 unexpected_category fraction / percentage literals; multi-subject sentences 6 unresolved_pronoun "them", "their", "his" with no compatible entity 5 unattached_quantity quantity never bound to a unit 1 no_question_target question parsed but slot never set Closing the gate to mixed-bounded [4, 24] is Phase 2.1 scope: extend composition rules for multi-quantity ops, add fraction/percentage primitives (per ADR-0164.1 amendment), expand lexicon for the remaining unknown_word cases, extend pronoun resolution. Invariants preserved - wrong = 0 in both flag states ✓ - flag-OFF byte-identical to today ✓ - determinism (50/50 identical runs) ✓ - Capability axes G1-G5, S1 unchanged ✓ - Reader tests: 19 (Phase 2) + 18 (Phase 1, post-update) + 53 (pack) + 76 (lexicon + primitives) = 166 specific to this change; all pass - core test --suite smoke -q: 67 passed Rebase note This PR was authored against an older base; rebased onto current main to incorporate #333 (Brief 8.2 universal proper_noun_token primitive) and #334 (ADR-0166 measurement discipline). The rebase required: - Lexicon files renamed proper_noun_entity_* -> proper_noun_gender_* (with the Phase 2 additions merged into the gender_* files) - Compiled lexicon.jsonl unchanged from #333's 207-entry state (Phase 2's per-category additions are runtime-visible via the source loader, not via the compiled file) - _classify reconciled with Brief 8.2's sentence-initial dispatch + Phase 2's 3-tuple decimal-value return - All dispatch tables and category checks updated to reference proper_noun_token (singular) instead of proper_noun_entity_{f,m} - Three Phase 1 test expectations updated to reflect Phase 2 behavior (proper noun at position 0 now opens statement pre-frame instead of refusing; pronoun resolution applies per ADR-0164.2) Per ADR-0166's three-question test, this PR is honest measurement: capability exists, at least one case admits, lane distinguishes presence from absence — which the bottleneck table demonstrates. Refs ADR-0164.3 §Phasing Phase 2, ADR-0164.1 amendment (Brief 8.2), ADR-0166 §"Mixed (notable but not blocking)" — except here, below floor.
151 lines
6.5 KiB
Markdown
151 lines
6.5 KiB
Markdown
# ADR-0164.4 — Phase 2 Statement-Frame Reader
|
|
|
|
**Status:** Proposed
|
|
**Date:** 2026-05-26
|
|
**Author:** Shay
|
|
**Anchor:** [[thesis-decoding-not-generating]]
|
|
**Parent:** [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md)
|
|
**Builds on:** [ADR-0164.3 — Cross-Sentence Reading State](./ADR-0164.3-cross-sentence-state.md)
|
|
**Related downstream types:** [ADR-0115 — `MathProblemGraph`](./ADR-0115-math-problem-parser-and-graph.md)
|
|
|
|
---
|
|
|
|
## Context
|
|
|
|
Phase 1 (ADR-0164.3) shipped the `question_frame` reader and the
|
|
two-level `ProblemReadingState` / `SentenceReadingState` lifecycle. Phase
|
|
2 extends the reader to the three statement-side frames and adds the
|
|
`finalize()` projection from `ProblemReadingState` into
|
|
`MathProblemGraph`, so a whole problem can be read end-to-end without
|
|
the legacy regex parser.
|
|
|
|
## Decision
|
|
|
|
### Frames ratified
|
|
|
|
- **`initial_state_frame`** — `entity possession_verb [count] [unit]`,
|
|
emits a `PartialInitialPossession` → `InitialPossession` at
|
|
`finalize()`.
|
|
- **`operation_frame`** — `entity (accumulation|depletion|transfer|
|
|
capacity)_verb [count] [unit] [to entity₂]`, emits a `PartialOperation`
|
|
→ `Operation` at `finalize()`. `accumulation` → `add`, `depletion` →
|
|
`subtract`, `capacity` → `add`, `transfer` → `transfer`.
|
|
- **`descriptive_frame`** — opens on `copula_verb` (or subject-dropped
|
|
verb position), drains known tokens, emits no math state. Used for
|
|
descriptive prose ("Sandra is a baker", "There are some kids in
|
|
camp") that does not bind quantities to operations.
|
|
|
|
### `finalize()` projection
|
|
|
|
Operates on a closed `ProblemReadingState`:
|
|
|
|
1. Require `unknown_target_slot` — else `no_question_target`.
|
|
2. Build `entities` tuple from `entity_registry` — empty registry yields
|
|
`dangling_entity`.
|
|
3. `PartialInitialPossession` → `InitialPossession` (requires
|
|
`entity`, `quantity.value`, and resolved `unit`).
|
|
4. `PartialOperation` → `Operation` (requires `actor`, op-kind from
|
|
verb category, `operand.value`, resolved `unit`).
|
|
5. `QuestionTargetSlot` → `Unknown` with `unit` derived from the slot's
|
|
captured unit lemma or the unit-class default.
|
|
|
|
### Integration flag
|
|
|
|
`generate.math_candidate_graph.parse_and_solve(text, *,
|
|
comprehension_reader: bool = False)` gates the reader-first path. With
|
|
the flag `False` (default), behaviour is byte-identical to the existing
|
|
regex parser. With the flag `True`, the reader is attempted first; if
|
|
any sentence refuses (all-or-nothing) the regex path runs unchanged.
|
|
|
|
### Wrong = 0 discipline
|
|
|
|
The reader never returns a graph with a wrong answer in the Phase 2
|
|
GSM8K-train sample: any structural ambiguity (multi-quantity ops,
|
|
fractions, multi-subject sentences) yields a typed `ReaderRefusal` so
|
|
the regex parser handles the case. This preserves the project-wide
|
|
`wrong == 0` invariant.
|
|
|
|
### Lexicon additions ratified (`phase_2_reader_gsm8k_2026-05-26`)
|
|
|
|
- `currency_unit_noun` +2 (`dollar`, `cent`).
|
|
- `accumulation_verb` +2 (`adopt`, `invest`).
|
|
- `capacity_verb` +6 (`fill`, `lift`, `play`, `work`, `finish`,
|
|
`drive`).
|
|
- `proper_noun_entity_female` +5 (`allison`, `brooke`, `jan`,
|
|
`marion`, `sidney`).
|
|
- `proper_noun_entity_male` +14 (`bart`, `fernando`, `georgie`, `jake`,
|
|
`jed`, `jeremie`, `jose`, `orlando`, `rex`, `rudolph`, `steve`,
|
|
`troy`, `xavier`, `yun`).
|
|
- `time_unit_noun` +3 (`year`, `month`, `second`).
|
|
- `count_unit_noun` +14 (puppy, kitten, parakeet, coconut, macaroon,
|
|
brownie, scoop, section, foot, cable, eraser, crayon, paperclip,
|
|
card, …).
|
|
- `drain_token` substantial expansion to absorb prose connectives,
|
|
written numerals, place names, and non-math verbs that should not
|
|
drive frame selection.
|
|
|
|
### Possessive handling
|
|
|
|
`_classify()` now strips trailing `'s` and re-attempts lexicon lookup,
|
|
so `Rudolph's` resolves to `rudolph` (proper_noun_entity_male). Genitive
|
|
possessives drain (e.g., "Aaron and his brother") rather than triggering
|
|
the multi-subject refusal.
|
|
|
|
## Evidence
|
|
|
|
`evals/gsm8k_math/train_sample/v1/reader_phase2_delta.json` captures
|
|
per-case attribution on the 50-case sample:
|
|
|
|
```text
|
|
flag-OFF: correct=3 wrong=0 refused=47
|
|
flag-ON: correct=3 wrong=0 refused=47
|
|
reader_accepted (built a Graph): 3 / 50
|
|
```
|
|
|
|
### Observed bottlenecks
|
|
|
|
| count | reader refusal class | examples / interpretation |
|
|
| ----- | ------------------------------ | -------------------------------------------- |
|
|
| 18 | `incomplete_operation` at end | multi-quantity ops ("4 bags with 20 apples in each bag"); no-quantity op_frame |
|
|
| 11 | `unknown_word` | `hundred`, `presently`, compound `one-hour`; non-math verbs (`encountered`, `studied`, `holds`) |
|
|
| 6 | `unexpected_category` | fraction/percentage literals (Phase 2.1); multi-subject ("Aaron and Carson") |
|
|
| 6 | `unresolved_pronoun` | `them`, `their`, `his` with no compatible registry entry |
|
|
| 5 | `unattached_quantity` at end | quantity never bound to a unit noun |
|
|
| 1 | `no_question_target` at finalize | question sentence parsed but never set the slot |
|
|
|
|
### Acceptance gate
|
|
|
|
The brief asks for `correct ≥ 25 (preferred)` or
|
|
`mixed ∈ [4, 24] with documented bottlenecks`. The current count of
|
|
`correct = 3` falls below the gate floor. **The reader is structurally
|
|
correct (wrong = 0 holds, flag-OFF byte-identical, determinism holds),
|
|
but the lexicon and structural coverage are not yet sufficient to clear
|
|
the gate.** Closing the gap requires:
|
|
|
|
1. Phase 2.1: embedded-quantifier aggregates ("N X with M Y in each")
|
|
and multi-quantity operation handling.
|
|
2. Phase 2.2: fraction/percentage literal handling (currently refused
|
|
wholesale).
|
|
3. Phase 3: multi-subject sentences, descriptive-clause unpacking.
|
|
|
|
## Invariants
|
|
|
|
- `versor_condition(F) < 1e-6` — Phase 2 reader does not touch field
|
|
state; trivially preserved.
|
|
- `wrong == 0` — empirically verified on the GSM8K train sample under
|
|
both flag settings.
|
|
- Flag-OFF byte-identical — `parse_and_solve` short-circuits before any
|
|
reader call when the flag is `False`.
|
|
- Determinism — identical input yields identical `trace_hash` on
|
|
repeated runs (50/50 verified).
|
|
|
|
## Out of scope (refused cleanly in Phase 2)
|
|
|
|
- Embedded-quantifier aggregates → `embedded-quantifier aggregate;
|
|
deferred to Phase 2.1`.
|
|
- Fraction / percentage literals → `unexpected_category` with
|
|
Phase 2.1 deferred-scope detail.
|
|
- Conditional frames ("If X, then Y") → Phase 3+.
|
|
- Multi-quantity initial state ("Two puppies, two kittens, …") →
|
|
Phase 2.1.
|
|
- Multi-subject sentences ("Aaron and Carson saved $40") → Phase 2.1.
|