diff --git a/docs/decisions/ADR-0164.3-cross-sentence-state.md b/docs/decisions/ADR-0164.3-cross-sentence-state.md new file mode 100644 index 00000000..17161de9 --- /dev/null +++ b/docs/decisions/ADR-0164.3-cross-sentence-state.md @@ -0,0 +1,621 @@ +# ADR-0164.3 — Cross-Sentence Reading State + +**Status:** Proposed +**Date:** 2026-05-26 +**Author:** Shay +**Anchor:** [[thesis-decoding-not-generating]] +**Parent:** [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md) §Open question #4 +**Companions:** [ADR-0164.1 — Lexical Primitive Set Scope](./ADR-0164.1-lexical-primitive-scope.md), [ADR-0164.2 — Pronoun/Entity Resolution Policy](./ADR-0164.2-pronoun-entity-resolution.md), [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md) +**Related downstream types:** [ADR-0115 — `MathProblemGraph`](./ADR-0115-math-problem-parser-and-graph.md), [ADR-0135 — `BoundUnknown` resolver](./ADR-0135-binding-graph-question-target.md) + +--- + +## Context — why two levels + +ADR-0164 §Decision §2 sketches a single `ComprehensionState` that +accumulates entities, quantities, operations, the question target, and a +current expectation frame. That sketch is correct for one sentence, but +GSM8K problems are multi-sentence: pronouns refer back across sentence +boundaries, entities introduced in sentence 1 are mutated in sentence 3, +and the question typically lives in the final sentence and refers to +state built across all prior sentences. + +Two structural facts force a split: + +1. **Lifetime asymmetry.** Some state (entity registry, accumulated + initial possessions, accumulated operations, the unknown target) is + *problem-scoped* — it persists across sentence boundaries and + accumulates monotonically. Other state (the current expectation + frame, pending quantities waiting for unit attachment, the partial + frame being built) is *sentence-scoped* — it must reset cleanly at + sentence boundaries so a stray expectation from a prior sentence + doesn't bleed into the next one. + +2. **Refusal locality.** When the reader refuses, the refusal points + either at a sentence-internal failure (unexpected category, dangling + quantity at sentence end, unfinished frame) or at a problem-level + failure (unresolved pronoun, conflicting entity reference, no + question target by problem end). Conflating both into one state + smears the refusal vocabulary and makes the failure modes harder to + diagnose. + +Collapsing both into a single immutable record is possible but the +collapsing buys nothing and costs vocabulary clarity. Two-level keeps +each field's lifetime explicit and each refusal mode local to its +appropriate level. + +> **Naming note for [Brief 5](./ADR-0164-incremental-comprehension-reader.md#acceptance-criteria-for-this-adr-proposed--accepted).** +> ADR-0164 §Decision §2's sketched `ComprehensionState` is structurally +> the **inner** level under this ADR — what is named `SentenceReadingState` +> here. The Brief 5 PR (ComprehensionState skeleton) will produce both +> `ProblemReadingState` and `SentenceReadingState` to honor the +> two-level model documented in this ADR. The sketch is preserved as +> the inner-level field list; the outer level is new. + +--- + +## Decision — two-level state model + +### `ProblemReadingState` (outer, problem-scoped) + +Immutable record. Persists across sentence boundaries. Mutated only by +`end_sentence` returning a new `ProblemReadingState` that absorbs the +just-closed sentence's contribution. + +| Field | Type | Role | +|---|---|---| +| `entity_registry` | `tuple[EntityRef, ...]` | Ordered by introduction position. Once an entity enters, it stays. Order-of-introduction matches [ADR-0115 `MathProblemGraph.entities`](./ADR-0115-math-problem-parser-and-graph.md) doctrine. | +| `accumulated_initial_state` | `tuple[PartialInitialPossession, ...]` | Initial-state declarations closed at sentence end. Tuple order is order-of-introduction. | +| `accumulated_operations` | `tuple[PartialOperation, ...]` | Operation declarations closed at sentence end. Tuple order is order-of-introduction (story order — ADR-0115 doctrine). | +| `unknown_target_slot` | `QuestionTargetSlot \| None` | Set exactly once, by the sentence containing the question. Locked after setting. `None` until a question sentence completes. | +| `pronoun_resolution_history` | `tuple[PronounResolution, ...]` | Replay-deterministic log of every pronoun resolution made during reading. Per [ADR-0164.2](./ADR-0164.2-pronoun-entity-resolution.md). | +| `sentence_index` | `int` | 0-based counter of completed sentences. Increments only on `end_sentence`. | +| `source_text_offset` | `int` | Character offset into the source problem text at which the next sentence begins. Maintained for span linkage. | + +`PartialInitialPossession` and `PartialOperation` are precursors to the +ADR-0115 types `InitialPossession` and `Operation`. They are "partial" +only in the sense that they can hold `None` for fields that are +optional during construction (e.g. a transfer with a missing target). +Once committed to `accumulated_*`, every field is set. The +finalization step (see §Termination) projects them into the strict +ADR-0115 types. + +### `SentenceReadingState` (inner, sentence-scoped) + +Immutable record. Lifetime = one sentence. Created by `begin_sentence`, +mutated by `apply_word`, consumed by `end_sentence`. + +| Field | Type | Role | +|---|---|---| +| `frame` | `SentenceFrame \| None` | The kind of sentence under construction once enough words have been read to decide. Discriminator: `initial_state_frame`, `operation_frame`, `question_frame`, `descriptive_frame` (context-only, no quantitative contribution). `None` while the frame is still ambiguous (very early in the sentence). | +| `expectation` | `ExpectationFrame \| None` | Open expectation slot — what categories would legally close or advance the current frame. Replaced as the frame narrows. `None` means "any frame opener is welcome." | +| `pending_quantities` | `tuple[QuantityRef, ...]` | Numbers seen so far in this sentence that haven't been attached to an entity + unit. Drains as units land. Sentence ends with non-empty `pending_quantities` → refusal `unattached_quantity`. | +| `pending_entity_ref` | `EntityRef \| None` | The entity reference active in the current frame (typically the sentence subject). Set when a proper-noun entity or a resolved pronoun lands in subject position. | +| `pending_verb` | `VerbReference \| None` | The verb captured at frame-determining position, waiting for completion (operand, target). Set on verb landing; consumed when the operation closes. | +| `token_index` | `int` | 0-based position within the current sentence. Increments on every `apply_word`. | +| `lookback` | `tuple[AppliedCategory, ...]` | Bounded history (≤8 entries) of categories applied in this sentence with their positions. Enables recontextualization without unbounded backtracking. | +| `partial_frame_payload` | `FramePayload \| None` | The frame-kind-specific in-construction structure. For `initial_state_frame`: a `PartialInitialPossession` being built up. For `operation_frame`: a `PartialOperation`. For `question_frame`: a `QuestionTargetSlot` being built up. | + +`SentenceReadingState` has read access to `ProblemReadingState` via the +lifecycle API (passed in to `apply_word`); it cannot mutate the outer +state directly. Only `end_sentence` produces a new +`ProblemReadingState`. + +--- + +## Lifecycle API (signatures only — no implementation) + +The reader exposes three pure functions. All are deterministic: same +inputs → byte-equal outputs and byte-equal canonical hashes. + +```python +def begin_sentence( + problem_state: ProblemReadingState, + source_text_offset: int, +) -> SentenceReadingState: + """Open a fresh sentence-local state. + + Resets all sentence-scoped fields. Inherits no transient state from + prior sentences — the only access to prior context is the + immutable ``problem_state`` argument (used read-only by + ``apply_word`` for entity resolution). + + Pure / deterministic. No I/O. + """ + + +def apply_word( + sentence_state: SentenceReadingState, + problem_state: ProblemReadingState, + word: str, +) -> SentenceReadingState | ReaderRefusal: + """Advance one token. Returns new sentence state or typed refusal. + + Lookup order per ADR-0164 §Decision §3: + 1. Lexical primitive scan (ADR-0164.1, ADR-0165). + 2. Lexicon lookup (en_core_math_v1, per ADR-0164 §Decision §1). + 3. Expectation check. + 4. Update emit. + + ``problem_state`` is read-only. Pronoun resolution consults + ``problem_state.entity_registry`` via the rules in + [ADR-0164.2](./ADR-0164.2-pronoun-entity-resolution.md). Resolutions + are recorded to a private buffer that ``end_sentence`` later folds + into ``problem_state.pronoun_resolution_history``. + + Pure / deterministic. No I/O. + """ + + +def end_sentence( + sentence_state: SentenceReadingState, + problem_state: ProblemReadingState, +) -> ProblemReadingState | ReaderRefusal: + """Close the sentence, fold its contribution into the problem state. + + Finalization rules: + + - ``sentence_state.frame`` must be one of the legal frame + kinds. ``None`` at end-of-sentence → refusal + ``unfinished_frame`` (we read words and never decided what shape + the sentence was). + - ``sentence_state.pending_quantities`` must be empty. A non-empty + pending list → refusal ``unattached_quantity``. + - The ``partial_frame_payload`` is projected into a typed + ``PartialInitialPossession`` / ``PartialOperation`` / + ``QuestionTargetSlot`` and appended to the appropriate + ``problem_state`` tuple. + - Newly introduced ``EntityRef`` records are appended to + ``problem_state.entity_registry``. + - Pronoun resolutions recorded in the sentence state are appended + to ``problem_state.pronoun_resolution_history``. + - ``sentence_index`` increments by 1. + - ``source_text_offset`` advances past the closing punctuation. + + The returned ``ProblemReadingState`` is the input to the next + ``begin_sentence`` call (next sentence) or to the finalization + predicate (last sentence). + + Pure / deterministic. No I/O. + """ +``` + +### `ReaderRefusal` + +Typed refusal record. Carries one of a closed set of reasons plus +diagnostic detail. + +```python +@dataclass(frozen=True, slots=True) +class ReaderRefusal: + reason: str # member of READER_REFUSAL_REASONS + detail: str # short human annotation + sentence_index: int # which sentence the refusal occurred in + token_index: int # position within the sentence (0 if end_sentence-level) + token_text: str # the token in question, or "" for non-token refusals +``` + +``` +READER_REFUSAL_REASONS = frozenset({ + # apply_word — token-level + "unknown_word", # not in lexicon, no primitive matched + "unexpected_category", # category does not satisfy current expectation + "expectation_collision", # two frame openers would be legal; precedence undecided + "unresolved_pronoun", # pronoun has no matching entity in registry + "ambiguous_pronoun_referent", # multiple matching entities in registry + # end_sentence — sentence-level + "unfinished_frame", # frame never decided + "unattached_quantity", # quantity never bound to entity+unit + "incomplete_operation", # operation missing operand or target + # problem-level (raised by the finalization predicate, not apply/end) + "no_question_target", # problem ended with unknown_target_slot=None + "dangling_entity", # entity in registry has no initial possession + "graph_construction_failure", # MathProblemGraph constructor rejected the projection +}) +``` + +The vocabulary is closed and ADR-tracked. New reasons require an ADR +amendment. This mirrors the +[ADR-0134 admissibility reason discipline](./ADR-0134-binding-graph-admissibility.md). + +--- + +## What persists vs sentence-local — explicit table + +| Concern | Where it lives | Lifetime | +|---|---|---| +| Entity registry | `ProblemReadingState` | All sentences | +| Initial possessions accumulated | `ProblemReadingState` | All sentences | +| Operations accumulated | `ProblemReadingState` | All sentences | +| The unknown target | `ProblemReadingState` | Set once; locked | +| Pronoun resolution history | `ProblemReadingState` | All sentences | +| Sentence index counter | `ProblemReadingState` | Monotonic | +| Current frame kind | `SentenceReadingState` | One sentence | +| Current expectation | `SentenceReadingState` | One sentence; replaced as frame narrows | +| Pending quantities (un-unit'd) | `SentenceReadingState` | One sentence; drains as units land | +| Pending entity reference (subject) | `SentenceReadingState` | One sentence | +| Pending verb (operation under construction) | `SentenceReadingState` | One sentence | +| Token position counter | `SentenceReadingState` | Resets at sentence boundary | +| Recent-category lookback window | `SentenceReadingState` | One sentence (bounded ≤8) | +| Frame-payload-in-construction | `SentenceReadingState` | One sentence; projected at `end_sentence` | + +The rule, stated negatively: **no field is in both levels.** If a +field seems to want to live in both, it is split — typically into a +"pending" sentence-local version and a "committed" problem-level +tuple. + +--- + +## Canonical-bytes serialization + +Both state levels serialize to deterministic JSON via the same +discipline used by +[`MathProblemGraph.canonical_bytes`](./ADR-0115-math-problem-parser-and-graph.md) +and the existing +[`trace_hash`](../runtime_contracts.md) production. + +```python +def to_canonical_bytes(state: ProblemReadingState | SentenceReadingState) -> bytes: + """Sorted-keys, compact-separators JSON. Tuples → lists. + + Decimal values render as strings to preserve precision (the math + graph uses int|float per ADR-0115; the reader uses Decimal + internally until projection to the graph). Optional fields are + omitted from JSON when None (not serialized as 'null') to keep + the canonical form minimal and to prevent spurious differences + between states that differ only in which optional fields they + chose to set explicitly to None. + """ + + +def canonical_hash(state: ProblemReadingState | SentenceReadingState) -> str: + """sha256 hex digest of to_canonical_bytes(state). + + Same shape as ADR-0153 turn-event trace-hash. Identical state → + identical hash. This is the determinism gate enforced by the + Brief 5 test scaffold. + """ +``` + +Serialization rules (matching existing CORE discipline): + +1. **Sort keys at every level** (`json.dumps(..., sort_keys=True)`). +2. **Compact separators** (`separators=(",", ":")`). +3. **Tuple → list**. Tuples carry ordering meaning; that ordering is + preserved by JSON array ordering. The list-vs-tuple distinction + is lost on the wire (intentional — JSON has no tuples). +4. **Decimal → string**. Use `str(value)` not float coercion. This + preserves precision through partial states; the projection to + `MathProblemGraph` (which uses `int|float`) happens at finalization + and must check loss-of-precision explicitly. +5. **Frozen dataclasses → dict** of field-name → field-value pairs, + recursing through children. +6. **Enums and Literals → their string value**. +7. **Optional fields with `None` → omitted from dict** (rule 2 caveat; + this prevents `{"x": null}` vs `{}` from being different). + +### `ReaderRefusal` is serialized too + +A refusal is not state, but it must be replay-deterministic for trace +audit. `to_canonical_bytes(ReaderRefusal(...))` follows the same rules. +Two runs that produce the same refusal produce byte-equal refusal +records. + +--- + +## Worked example — gsm8k-train-sample-v1-0001 + +> "Tina makes $18.00 an hour. If she works more than 8 hours per +> shift, she is eligible for overtime, which is paid by your hourly +> wage + 1/2 your hourly wage. If she works 10 hours every day for 5 +> days, how much money does she make?" + +Expected outcome under Phase 1: the reader will admit sentence 1 +(rate statement) and refuse sentences 2 and 3 with `conditional_*` +reasons because conditional structure is not in Phase 1 scope. +**The example demonstrates the state model — not solver success.** +That is the right kind of demonstration: when the engine refuses, +the state at the point of refusal is what tells us *why* and at what +specific position, and that is what enables principled corpus growth. + +### Sentence 1 — "Tina makes $18.00 an hour." + +`begin_sentence(problem_state=∅, source_text_offset=0)` produces: + +``` +SentenceReadingState( + frame=None, + expectation=None, + pending_quantities=(), + pending_entity_ref=None, + pending_verb=None, + token_index=0, + lookback=(), + partial_frame_payload=None, +) +``` + +Word-by-word (Phase 1 lexicon + primitive set, illustrative): + +| pos | word | primitive / lexicon hit | state change | +|---|---|---|---| +| 0 | `Tina` | lexicon: `proper_noun_entity_female` | `pending_entity_ref = EntityRef("Tina", "female", 0)`; entity not yet in registry — staged for commit at sentence end. | +| 1 | `makes` | lexicon: `accumulation_verb` / `rate_emit_verb` | Frame narrows. `pending_verb = VerbReference("makes", "rate_emit", 1)`. `frame = operation_frame` (tentative). `expectation = "QUANTITY (currency) followed by 'an X'"`. | +| 2 | `$18.00` | primitive: `currency_literal` → `QuantityRef(Decimal("18.00"), "dollars", "currency", attached_to_entity=None, source_position=2)` | `pending_quantities = (Q$18.00,)`. Expectation advances to `"'an' or 'per' followed by time-unit"`. | +| 3 | `an` | lexicon: `per_unit_marker` (closed-set: an, per, every, each per time-unit) | Expectation narrows to `"time-unit-noun"`. Lookback records `per_unit_marker`. | +| 4 | `hour` | lexicon: `time_unit_noun` | Rate composition closes. `pending_quantities[0]` (the $18.00) becomes the numerator of a rate; `hour` is the denominator. `frame` confirmed = `operation_frame`. `partial_frame_payload = PartialOperation(actor="Tina", kind="apply_rate", operand=Rate(18.00, "dollars", "hour"))`. `pending_quantities` drains to `()`. | +| 5 | `.` | sentence terminator | | + +`end_sentence(...)` projects `partial_frame_payload` into a typed +`PartialOperation`, commits Tina to `entity_registry`, appends the +operation to `accumulated_operations`, increments `sentence_index`. + +``` +ProblemReadingState after sentence 1: + entity_registry = (EntityRef("Tina", "female", 0),) + accumulated_initial_state = () + accumulated_operations = ( + PartialOperation( + actor="Tina", kind="apply_rate", + operand=Rate(18.00, "dollars", "hour"), + target=None, + ), + ) + unknown_target_slot = None + pronoun_resolution_history = () + sentence_index = 1 +``` + +### Sentence 2 — "If she works more than 8 hours per shift, ..." + +`begin_sentence` produces a fresh `SentenceReadingState`. + +| pos | word | result | +|---|---|---| +| 0 | `If` | lexicon: `conditional_open` | `frame = conditional_frame` (Phase-1 NOT in scope). Reader refuses. | + +Refusal: + +``` +ReaderRefusal( + reason="unexpected_category", + detail="conditional_open at sentence_index=1, position=0; " + "conditional_frame is Phase-1 out-of-scope (ADR-0164 §Phasing)", + sentence_index=1, + token_index=0, + token_text="If", +) +``` + +This is the **correct** Phase 1 behavior. The state at the moment of +refusal is enough to file a typed teaching candidate for the +conditional-frame category. The refusal does not corrupt the +`ProblemReadingState` built from sentence 1 — sentence 2's failure +leaves the outer state at its post-sentence-1 value (refusals do not +commit; `end_sentence` is the only commit path). + +### Sentence 3 — "If she works 10 hours every day for 5 days, how much money does she make?" + +Same refusal mode as sentence 2 (`unexpected_category` on the leading +`If`). Phase 2 / Phase 3 work expands the conditional-frame +vocabulary. + +### Pronoun resolution annotation + +If sentence 2 or 3 *were* in scope (after Phase 2), the word "she" at +their leading positions would consult +`problem_state.entity_registry = (EntityRef("Tina", "female", 0),)` +under [ADR-0164.2](./ADR-0164.2-pronoun-entity-resolution.md)'s +gender-plus-recency rule. Exactly one matching entity exists → +`PronounResolution(pronoun="she", resolved_to="Tina", at_position=..., entity_source=sentence_0)` +is recorded. Zero matching → `unresolved_pronoun`; more than one → +`ambiguous_pronoun_referent`. The resolution is appended to +`pronoun_resolution_history` exactly when the sentence containing it +closes successfully. + +--- + +## Termination predicate + +A `ProblemReadingState` is valid for handoff to `MathProblemGraph` +construction when **all** of the following hold. The predicate is a +pure function: same input → same verdict. + +```python +def is_terminable(state: ProblemReadingState) -> bool: + return ( + state.entity_registry # ≥1 entity + and state.unknown_target_slot is not None # question target bound + and _every_op_references_known_entity(state) # closure check + and _every_initial_references_known_entity(state) # closure check + and _question_target_entity_resolvable(state) # bound vs registry + and _no_pending_unresolved_pronouns(state) # no dangling refs + and _partial_payloads_project_to_strict_types(state) # ADR-0115 typecheck + ) +``` + +Failure of any condition produces a typed problem-level refusal (see +`READER_REFUSAL_REASONS` problem-level group). + +`is_terminable(state)` true → the reader calls a finalizer that +constructs the strict ADR-0115 `MathProblemGraph`: + +```python +def finalize(state: ProblemReadingState) -> MathProblemGraph | ReaderRefusal: + """Project the partial state into a strict MathProblemGraph. + + Tuple field ordering on the graph mirrors order-of-introduction in + ``state.entity_registry`` and source-text order in + ``state.accumulated_*``. Decimal values must round-trip losslessly + through ``int | float`` (the ADR-0115 type); a precision loss + refuses with ``graph_construction_failure``. + """ +``` + +The output of `finalize` is the exact input shape the existing +binding-graph adapter (ADR-0133) and `BoundUnknown` resolver +(ADR-0135) consume today. The reader does not add a new downstream +contract; it replaces the old front-end's output emit. + +--- + +## Refusal modes — summary + +Three lifetime groupings (mirroring the API): + +**Token-level (raised by `apply_word`):** + +- `unknown_word` — token not in lexicon and no primitive matched. +- `unexpected_category` — category does not satisfy current + expectation and is not a legal frame opener at this position. +- `expectation_collision` — two frame openers would be legal at this + position. Should not occur given a complete precedence table + (defer to ADR-0164.1 / phase-1 measurement); a real occurrence is + evidence the precedence rule needs an ADR. +- `unresolved_pronoun` — pronoun has no matching entity per + ADR-0164.2. +- `ambiguous_pronoun_referent` — pronoun has multiple matching + entities per ADR-0164.2. + +**Sentence-level (raised by `end_sentence`):** + +- `unfinished_frame` — frame kind never resolved. +- `unattached_quantity` — at least one number was seen but never + attached to an entity + unit. +- `incomplete_operation` — operation frame closed with missing + operand or missing target. + +**Problem-level (raised by the finalization predicate):** + +- `no_question_target` — problem ended with `unknown_target_slot=None`. + The question sentence either was refused or never identified itself + as a question frame. +- `dangling_entity` — entity appears in registry but has no initial + possession and is not the subject of any operation. Probably a + pronoun mis-resolution upstream. +- `graph_construction_failure` — projection into strict ADR-0115 + types failed (precision loss, schema violation). The detail field + carries the ADR-0115 `MathGraphError` message. + +Each refusal records `sentence_index` and `token_index` to localize. +Refusals are themselves canonical-bytes-serializable (see §Canonical-bytes). + +--- + +## Interaction with other ADRs + +### ADR-0164 (parent) + +ADR-0164's `ComprehensionState` sketch (§Decision §2) is the +**inner-level** type under this ADR, renamed `SentenceReadingState`. +The outer-level `ProblemReadingState` is new. ADR-0164's §Phasing is +unchanged: Phase 1 builds the reader for question sentences, which +under the two-level model means Phase 1 implements `apply_word` rules +for the `question_frame` kind and a minimal subset of frame openers, +enough to admit the question-class refusals that block 34/47 +train_sample cases. + +### ADR-0164.1 (companion) + +Lexical primitives are consumed inside `apply_word` step 1 (primitive +scan). The primitive registry's emit-category is what the +expectation-check step compares against. No state-shape dependency +here; ADR-0164.1 specifies *what* primitives produce, ADR-0164.3 +specifies *how* the producer's output flows into the state machine. + +### ADR-0164.2 (companion) + +Pronoun resolution is the only place `apply_word` reads from +`problem_state` non-trivially. ADR-0164.2 specifies the rule; +ADR-0164.3 specifies the carrier (`pronoun_resolution_history` and +the per-sentence resolution buffer). The resolution buffer flushes +to the outer history only on successful `end_sentence`. + +### ADR-0115 (downstream) + +The `finalize` step produces a strict `MathProblemGraph`. +`MathProblemGraph`'s order-of-introduction invariant +(ADR-0115 docstring) is preserved by reading the outer state's +tuples in their stored order (which is order-of-introduction +because that's how they were appended). + +### ADR-0134 / ADR-0135 (downstream binding graph) + +Unchanged. The binding-graph adapter reads `MathProblemGraph`; we +produce `MathProblemGraph`; nothing else changes. + +### ADR-0165 (regex scope rule) + +`apply_word` step 1 consults the lexical primitive registry. Every +primitive in that registry must satisfy ADR-0165 (orthographic-shape +recognition only, never grammar). This ADR does not bypass that +rule; if anything it makes the boundary cleaner because primitive +hits and lexicon hits are explicitly distinct lookup steps. + +--- + +## Open implementation choices (intentionally not pinned here) + +These belong in Brief 5 (the `ComprehensionState` / state types +skeleton PR) or in follow-up sub-ADRs. Pinning them now would +over-commit. + +1. **`Decimal` precision setting.** `decimal.getcontext().prec` + default is fine for GSM8K (currency to 2 decimal places, quantity + counts as integers). Pin in Brief 5 with a `localcontext` block + if precision needs to be smaller for canonical bytes. +2. **`lookback` window size.** ADR-0164.3 says ≤8 entries. The exact + number can be tuned during Phase 1 measurement; 8 covers all + GSM8K question sentences observed at session time. +3. **Frame kind enumeration.** The four discriminator values + (`initial_state_frame`, `operation_frame`, `question_frame`, + `descriptive_frame`) are sufficient for Phase 1 + Phase 2 scope. + Phase 3 may add `conditional_frame` and `rate_emit_frame`; each + addition is an ADR. +4. **Source-span linkage.** This ADR includes `token_index` and + `source_text_offset` but does not specify a full source-span + record. The binding graph's `SourceSpanLink` (ADR-0132) is the + downstream consumer; whether the reader emits a `SourceSpanLink` + per partial-payload field or a single coarse span per + sentence-payload is a Brief 5 decision. + +--- + +## Acceptance criteria (Proposed → Accepted) + +This ADR moves to Accepted when: + +1. Brief 5 lands `generate/comprehension/state.py` exporting + `ProblemReadingState` and `SentenceReadingState` (renamed from + ADR-0164's `ComprehensionState` sketch per §Decision above) with + the field tables matching §Decision exactly. +2. The lifecycle API signatures land as Python stubs (no body) in + `generate/comprehension/lifecycle.py`. Phase 1 implements the + bodies. +3. `to_canonical_bytes` / `canonical_hash` implementation passes the + determinism gate in `tests/test_comprehension_state.py` (Brief 5's + test scaffold) on both state levels. +4. `READER_REFUSAL_REASONS` is materialized as a frozenset constant + matching the closed set above. + +--- + +## Cross-references + +- **Parent**: [ADR-0164](./ADR-0164-incremental-comprehension-reader.md). +- **Companion sub-ADRs**: + [ADR-0164.1 lexical primitive scope](./ADR-0164.1-lexical-primitive-scope.md), + [ADR-0164.2 pronoun/entity resolution](./ADR-0164.2-pronoun-entity-resolution.md). +- **Boundary invariant**: [ADR-0165](./ADR-0165-regex-scope-rule.md). +- **Downstream target type**: [ADR-0115](./ADR-0115-math-problem-parser-and-graph.md). +- **Downstream binding-graph consumers (unchanged)**: + [ADR-0132](./ADR-0132-binding-graph-data-model.md) / + [ADR-0133](./ADR-0133-binding-graph-adapter.md) / + [ADR-0134](./ADR-0134-binding-graph-admissibility.md) / + [ADR-0135](./ADR-0135-binding-graph-question-target.md). +- **Brief 5 (acceptance dependency)**: + [ADR-0164 §Acceptance criteria #1](./ADR-0164-incremental-comprehension-reader.md#acceptance-criteria-for-this-adr-proposed--accepted). +- **Anchor**: `[[thesis-decoding-not-generating]]` — the state model + is the form of *decoding in progress*. Each transition narrows + the space of possible meanings until the structure that the source + text already implied has surfaced fully.