core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

docs(ADR-0164.3): cross-sentence reading state (#320)

Browse source 
Two-level state model for the incremental comprehension reader:
ProblemReadingState (outer, problem-scoped) carries the entity registry,
accumulated initial possessions, accumulated operations, the unknown
target slot, and the pronoun resolution history. SentenceReadingState
(inner, sentence-scoped) carries the current frame, expectation,
pending quantities, pending entity reference, pending verb, lookback
window, and the partial frame payload under construction.

Lifecycle API (signatures only): begin_sentence, apply_word,
end_sentence. All three pure / deterministic / no I/O. apply_word
reads from problem_state for pronoun resolution per ADR-0164.2 but
does not mutate it; only end_sentence produces a new
ProblemReadingState that folds in the just-closed sentence's
contribution.

Closed READER_REFUSAL_REASONS vocabulary across three lifetime
groupings (token-level, sentence-level, problem-level), mirroring
ADR-0134's admissibility-reason discipline.

Canonical-bytes serialization for both state levels matches existing
trace_hash and MathProblemGraph.canonical_bytes discipline.
Sorted-keys JSON, compact separators, Decimal-as-string for
precision, optional-None fields omitted.

Worked example: gsm8k-train-sample-v1-0001. Sentence 1 ("Tina makes
$18.00 an hour.") admits as a rate apply_rate operation; sentences 2
and 3 refuse at the leading "If" with unexpected_category
(conditional_frame is Phase-1 out-of-scope). The example demonstrates
the state model — that even when the reader refuses, the state at
the moment of refusal is what makes the refusal honest, typed, and
file-able as a teaching candidate.

Termination predicate is_terminable + finalize specified pure: a
ProblemReadingState becomes a strict ADR-0115 MathProblemGraph only
when entity registry is non-empty, unknown_target_slot is bound,
every accumulated op/initial references a known entity, and every
partial payload projects losslessly into the strict types.

Naming reconciliation: ADR-0164's sketched ComprehensionState is the
inner level under this ADR (SentenceReadingState). Brief 5 will
produce both types.

No code. ADR doc only.

Refs ADR-0164 §Open question #4.
This commit is contained in:
Shay 2026-05-26 19:25:59 -07:00 committed by GitHub
parent e705f27d2e
commit 20f3a5d586
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 621 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

621
docs/decisions/ADR-0164.3-cross-sentence-state.md Normal file
View file

@ -0,0 +1,621 @@
# ADR-0164.3 — Cross-Sentence Reading State
**Status:** Proposed
**Date:** 2026-05-26
**Author:** Shay
**Anchor:** [[thesis-decoding-not-generating]]
**Parent:** [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md) §Open question #4
**Companions:** [ADR-0164.1 — Lexical Primitive Set Scope](./ADR-0164.1-lexical-primitive-scope.md), [ADR-0164.2 — Pronoun/Entity Resolution Policy](./ADR-0164.2-pronoun-entity-resolution.md), [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md)
**Related downstream types:** [ADR-0115 — `MathProblemGraph`](./ADR-0115-math-problem-parser-and-graph.md), [ADR-0135 — `BoundUnknown` resolver](./ADR-0135-binding-graph-question-target.md)
---
## Context — why two levels
ADR-0164 §Decision §2 sketches a single `ComprehensionState` that
accumulates entities, quantities, operations, the question target, and a
current expectation frame. That sketch is correct for one sentence, but
GSM8K problems are multi-sentence: pronouns refer back across sentence
boundaries, entities introduced in sentence 1 are mutated in sentence 3,
and the question typically lives in the final sentence and refers to
state built across all prior sentences.
Two structural facts force a split:
1. **Lifetime asymmetry.** Some state (entity registry, accumulated
initial possessions, accumulated operations, the unknown target) is
*problem-scoped* — it persists across sentence boundaries and
accumulates monotonically. Other state (the current expectation
frame, pending quantities waiting for unit attachment, the partial
frame being built) is *sentence-scoped* — it must reset cleanly at
sentence boundaries so a stray expectation from a prior sentence
doesn't bleed into the next one.
2. **Refusal locality.** When the reader refuses, the refusal points
either at a sentence-internal failure (unexpected category, dangling
quantity at sentence end, unfinished frame) or at a problem-level
failure (unresolved pronoun, conflicting entity reference, no
question target by problem end). Conflating both into one state
smears the refusal vocabulary and makes the failure modes harder to
diagnose.
Collapsing both into a single immutable record is possible but the
collapsing buys nothing and costs vocabulary clarity. Two-level keeps
each field's lifetime explicit and each refusal mode local to its
appropriate level.
> **Naming note for [Brief 5](./ADR-0164-incremental-comprehension-reader.md#acceptance-criteria-for-this-adr-proposed--accepted).**
> ADR-0164 §Decision §2's sketched `ComprehensionState` is structurally
> the **inner** level under this ADR — what is named `SentenceReadingState`
> here. The Brief 5 PR (ComprehensionState skeleton) will produce both
> `ProblemReadingState` and `SentenceReadingState` to honor the
> two-level model documented in this ADR. The sketch is preserved as
> the inner-level field list; the outer level is new.
---
## Decision — two-level state model
### `ProblemReadingState` (outer, problem-scoped)
Immutable record. Persists across sentence boundaries. Mutated only by
`end_sentence` returning a new `ProblemReadingState` that absorbs the
just-closed sentence's contribution.
| Field | Type | Role |
|---|---|---|
| `entity_registry` | `tuple[EntityRef, ...]` | Ordered by introduction position. Once an entity enters, it stays. Order-of-introduction matches [ADR-0115 `MathProblemGraph.entities`](./ADR-0115-math-problem-parser-and-graph.md) doctrine. |
| `accumulated_initial_state` | `tuple[PartialInitialPossession, ...]` | Initial-state declarations closed at sentence end. Tuple order is order-of-introduction. |
| `accumulated_operations` | `tuple[PartialOperation, ...]` | Operation declarations closed at sentence end. Tuple order is order-of-introduction (story order — ADR-0115 doctrine). |
| `unknown_target_slot` | `QuestionTargetSlot \| None` | Set exactly once, by the sentence containing the question. Locked after setting. `None` until a question sentence completes. |
| `pronoun_resolution_history` | `tuple[PronounResolution, ...]` | Replay-deterministic log of every pronoun resolution made during reading. Per [ADR-0164.2](./ADR-0164.2-pronoun-entity-resolution.md). |
| `sentence_index` | `int` | 0-based counter of completed sentences. Increments only on `end_sentence`. |
| `source_text_offset` | `int` | Character offset into the source problem text at which the next sentence begins. Maintained for span linkage. |
`PartialInitialPossession` and `PartialOperation` are precursors to the
ADR-0115 types `InitialPossession` and `Operation`. They are "partial"
only in the sense that they can hold `None` for fields that are
optional during construction (e.g. a transfer with a missing target).
Once committed to `accumulated_*`, every field is set. The
finalization step (see §Termination) projects them into the strict
ADR-0115 types.
### `SentenceReadingState` (inner, sentence-scoped)
Immutable record. Lifetime = one sentence. Created by `begin_sentence`,
mutated by `apply_word`, consumed by `end_sentence`.
| Field | Type | Role |
|---|---|---|
| `frame` | `SentenceFrame \| None` | The kind of sentence under construction once enough words have been read to decide. Discriminator: `initial_state_frame`, `operation_frame`, `question_frame`, `descriptive_frame` (context-only, no quantitative contribution). `None` while the frame is still ambiguous (very early in the sentence). |
| `expectation` | `ExpectationFrame \| None` | Open expectation slot — what categories would legally close or advance the current frame. Replaced as the frame narrows. `None` means "any frame opener is welcome." |
| `pending_quantities` | `tuple[QuantityRef, ...]` | Numbers seen so far in this sentence that haven't been attached to an entity + unit. Drains as units land. Sentence ends with non-empty `pending_quantities` → refusal `unattached_quantity`. |
| `pending_entity_ref` | `EntityRef \| None` | The entity reference active in the current frame (typically the sentence subject). Set when a proper-noun entity or a resolved pronoun lands in subject position. |
| `pending_verb` | `VerbReference \| None` | The verb captured at frame-determining position, waiting for completion (operand, target). Set on verb landing; consumed when the operation closes. |
| `token_index` | `int` | 0-based position within the current sentence. Increments on every `apply_word`. |
| `lookback` | `tuple[AppliedCategory, ...]` | Bounded history (≤8 entries) of categories applied in this sentence with their positions. Enables recontextualization without unbounded backtracking. |
| `partial_frame_payload` | `FramePayload \| None` | The frame-kind-specific in-construction structure. For `initial_state_frame`: a `PartialInitialPossession` being built up. For `operation_frame`: a `PartialOperation`. For `question_frame`: a `QuestionTargetSlot` being built up. |
`SentenceReadingState` has read access to `ProblemReadingState` via the
lifecycle API (passed in to `apply_word`); it cannot mutate the outer
state directly. Only `end_sentence` produces a new
`ProblemReadingState`.
---
## Lifecycle API (signatures only — no implementation)
The reader exposes three pure functions. All are deterministic: same
inputs → byte-equal outputs and byte-equal canonical hashes.
```python
def begin_sentence(
problem_state: ProblemReadingState,
source_text_offset: int,
) -> SentenceReadingState:
"""Open a fresh sentence-local state.
Resets all sentence-scoped fields. Inherits no transient state from
prior sentences — the only access to prior context is the
immutable ``problem_state`` argument (used read-only by
``apply_word`` for entity resolution).
Pure / deterministic. No I/O.
"""
def apply_word(
sentence_state: SentenceReadingState,
problem_state: ProblemReadingState,
word: str,
) -> SentenceReadingState | ReaderRefusal:
"""Advance one token. Returns new sentence state or typed refusal.
Lookup order per ADR-0164 §Decision §3:
1. Lexical primitive scan (ADR-0164.1, ADR-0165).
2. Lexicon lookup (en_core_math_v1, per ADR-0164 §Decision §1).
3. Expectation check.
4. Update emit.
``problem_state`` is read-only. Pronoun resolution consults
``problem_state.entity_registry`` via the rules in
[ADR-0164.2](./ADR-0164.2-pronoun-entity-resolution.md). Resolutions
are recorded to a private buffer that ``end_sentence`` later folds
into ``problem_state.pronoun_resolution_history``.
Pure / deterministic. No I/O.
"""
def end_sentence(
sentence_state: SentenceReadingState,
problem_state: ProblemReadingState,
) -> ProblemReadingState | ReaderRefusal:
"""Close the sentence, fold its contribution into the problem state.
Finalization rules:
- ``sentence_state.frame`` must be one of the legal frame
kinds. ``None`` at end-of-sentence → refusal
``unfinished_frame`` (we read words and never decided what shape
the sentence was).
- ``sentence_state.pending_quantities`` must be empty. A non-empty
pending list → refusal ``unattached_quantity``.
- The ``partial_frame_payload`` is projected into a typed
``PartialInitialPossession`` / ``PartialOperation`` /
``QuestionTargetSlot`` and appended to the appropriate
``problem_state`` tuple.
- Newly introduced ``EntityRef`` records are appended to
``problem_state.entity_registry``.
- Pronoun resolutions recorded in the sentence state are appended
to ``problem_state.pronoun_resolution_history``.
- ``sentence_index`` increments by 1.
- ``source_text_offset`` advances past the closing punctuation.
The returned ``ProblemReadingState`` is the input to the next
``begin_sentence`` call (next sentence) or to the finalization
predicate (last sentence).
Pure / deterministic. No I/O.
"""
```
### `ReaderRefusal`
Typed refusal record. Carries one of a closed set of reasons plus
diagnostic detail.
```python
@dataclass(frozen=True, slots=True)
class ReaderRefusal:
reason: str # member of READER_REFUSAL_REASONS
detail: str # short human annotation
sentence_index: int # which sentence the refusal occurred in
token_index: int # position within the sentence (0 if end_sentence-level)
token_text: str # the token in question, or "" for non-token refusals
```
```
READER_REFUSAL_REASONS = frozenset({
# apply_word — token-level
"unknown_word", # not in lexicon, no primitive matched
"unexpected_category", # category does not satisfy current expectation
"expectation_collision", # two frame openers would be legal; precedence undecided
"unresolved_pronoun", # pronoun has no matching entity in registry
"ambiguous_pronoun_referent", # multiple matching entities in registry
# end_sentence — sentence-level
"unfinished_frame", # frame never decided
"unattached_quantity", # quantity never bound to entity+unit
"incomplete_operation", # operation missing operand or target
# problem-level (raised by the finalization predicate, not apply/end)
"no_question_target", # problem ended with unknown_target_slot=None
"dangling_entity", # entity in registry has no initial possession
"graph_construction_failure", # MathProblemGraph constructor rejected the projection
})
```
The vocabulary is closed and ADR-tracked. New reasons require an ADR
amendment. This mirrors the
[ADR-0134 admissibility reason discipline](./ADR-0134-binding-graph-admissibility.md).
---
## What persists vs sentence-local — explicit table
| Concern | Where it lives | Lifetime |
|---|---|---|
| Entity registry | `ProblemReadingState` | All sentences |
| Initial possessions accumulated | `ProblemReadingState` | All sentences |
| Operations accumulated | `ProblemReadingState` | All sentences |
| The unknown target | `ProblemReadingState` | Set once; locked |
| Pronoun resolution history | `ProblemReadingState` | All sentences |
| Sentence index counter | `ProblemReadingState` | Monotonic |
| Current frame kind | `SentenceReadingState` | One sentence |
| Current expectation | `SentenceReadingState` | One sentence; replaced as frame narrows |
| Pending quantities (un-unit'd) | `SentenceReadingState` | One sentence; drains as units land |
| Pending entity reference (subject) | `SentenceReadingState` | One sentence |
| Pending verb (operation under construction) | `SentenceReadingState` | One sentence |
| Token position counter | `SentenceReadingState` | Resets at sentence boundary |
| Recent-category lookback window | `SentenceReadingState` | One sentence (bounded ≤8) |
| Frame-payload-in-construction | `SentenceReadingState` | One sentence; projected at `end_sentence` |
The rule, stated negatively: **no field is in both levels.** If a
field seems to want to live in both, it is split — typically into a
"pending" sentence-local version and a "committed" problem-level
tuple.
---
## Canonical-bytes serialization
Both state levels serialize to deterministic JSON via the same
discipline used by
[`MathProblemGraph.canonical_bytes`](./ADR-0115-math-problem-parser-and-graph.md)
and the existing
[`trace_hash`](../runtime_contracts.md) production.
```python
def to_canonical_bytes(state: ProblemReadingState | SentenceReadingState) -> bytes:
"""Sorted-keys, compact-separators JSON. Tuples → lists.
Decimal values render as strings to preserve precision (the math
graph uses int|float per ADR-0115; the reader uses Decimal
internally until projection to the graph). Optional fields are
omitted from JSON when None (not serialized as 'null') to keep
the canonical form minimal and to prevent spurious differences
between states that differ only in which optional fields they
chose to set explicitly to None.
"""
def canonical_hash(state: ProblemReadingState | SentenceReadingState) -> str:
"""sha256 hex digest of to_canonical_bytes(state).
Same shape as ADR-0153 turn-event trace-hash. Identical state →
identical hash. This is the determinism gate enforced by the
Brief 5 test scaffold.
"""
```
Serialization rules (matching existing CORE discipline):
1. **Sort keys at every level** (`json.dumps(..., sort_keys=True)`).
2. **Compact separators** (`separators=(",", ":")`).
3. **Tuple → list**. Tuples carry ordering meaning; that ordering is
preserved by JSON array ordering. The list-vs-tuple distinction
is lost on the wire (intentional — JSON has no tuples).
4. **Decimal → string**. Use `str(value)` not float coercion. This
preserves precision through partial states; the projection to
`MathProblemGraph` (which uses `int|float`) happens at finalization
and must check loss-of-precision explicitly.
5. **Frozen dataclasses → dict** of field-name → field-value pairs,
recursing through children.
6. **Enums and Literals → their string value**.
7. **Optional fields with `None` → omitted from dict** (rule 2 caveat;
this prevents `{"x": null}` vs `{}` from being different).
### `ReaderRefusal` is serialized too
A refusal is not state, but it must be replay-deterministic for trace
audit. `to_canonical_bytes(ReaderRefusal(...))` follows the same rules.
Two runs that produce the same refusal produce byte-equal refusal
records.
---
## Worked example — gsm8k-train-sample-v1-0001
> "Tina makes $18.00 an hour. If she works more than 8 hours per
> shift, she is eligible for overtime, which is paid by your hourly
> wage + 1/2 your hourly wage. If she works 10 hours every day for 5
> days, how much money does she make?"
Expected outcome under Phase 1: the reader will admit sentence 1
(rate statement) and refuse sentences 2 and 3 with `conditional_*`
reasons because conditional structure is not in Phase 1 scope.
**The example demonstrates the state model — not solver success.**
That is the right kind of demonstration: when the engine refuses,
the state at the point of refusal is what tells us *why* and at what
specific position, and that is what enables principled corpus growth.
### Sentence 1 — "Tina makes $18.00 an hour."
`begin_sentence(problem_state=∅, source_text_offset=0)` produces:
```
SentenceReadingState(
frame=None,
expectation=None,
pending_quantities=(),
pending_entity_ref=None,
pending_verb=None,
token_index=0,
lookback=(),
partial_frame_payload=None,
)
```
Word-by-word (Phase 1 lexicon + primitive set, illustrative):
| pos | word | primitive / lexicon hit | state change |
|---|---|---|---|
| 0 | `Tina` | lexicon: `proper_noun_entity_female` | `pending_entity_ref = EntityRef("Tina", "female", 0)`; entity not yet in registry — staged for commit at sentence end. |
| 1 | `makes` | lexicon: `accumulation_verb` / `rate_emit_verb` | Frame narrows. `pending_verb = VerbReference("makes", "rate_emit", 1)`. `frame = operation_frame` (tentative). `expectation = "QUANTITY (currency) followed by 'an X'"`. |
| 2 | `$18.00` | primitive: `currency_literal` → `QuantityRef(Decimal("18.00"), "dollars", "currency", attached_to_entity=None, source_position=2)` | `pending_quantities = (Q$18.00,)`. Expectation advances to `"'an' or 'per' followed by time-unit"`. |
| 3 | `an` | lexicon: `per_unit_marker` (closed-set: an, per, every, each per time-unit) | Expectation narrows to `"time-unit-noun"`. Lookback records `per_unit_marker`. |
| 4 | `hour` | lexicon: `time_unit_noun` | Rate composition closes. `pending_quantities[0]` (the $18.00) becomes the numerator of a rate; `hour` is the denominator. `frame` confirmed = `operation_frame`. `partial_frame_payload = PartialOperation(actor="Tina", kind="apply_rate", operand=Rate(18.00, "dollars", "hour"))`. `pending_quantities` drains to `()`. |
| 5 | `.` | sentence terminator | |
`end_sentence(...)` projects `partial_frame_payload` into a typed
`PartialOperation`, commits Tina to `entity_registry`, appends the
operation to `accumulated_operations`, increments `sentence_index`.
```
ProblemReadingState after sentence 1:
entity_registry = (EntityRef("Tina", "female", 0),)
accumulated_initial_state = ()
accumulated_operations = (
PartialOperation(
actor="Tina", kind="apply_rate",
operand=Rate(18.00, "dollars", "hour"),
target=None,
),
)
unknown_target_slot = None
pronoun_resolution_history = ()
sentence_index = 1
```
### Sentence 2 — "If she works more than 8 hours per shift, ..."
`begin_sentence` produces a fresh `SentenceReadingState`.
| pos | word | result |
|---|---|---|
| 0 | `If` | lexicon: `conditional_open` | `frame = conditional_frame` (Phase-1 NOT in scope). Reader refuses. |
Refusal:
```
ReaderRefusal(
reason="unexpected_category",
detail="conditional_open at sentence_index=1, position=0; "
"conditional_frame is Phase-1 out-of-scope (ADR-0164 §Phasing)",
sentence_index=1,
token_index=0,
token_text="If",
)
```
This is the **correct** Phase 1 behavior. The state at the moment of
refusal is enough to file a typed teaching candidate for the
conditional-frame category. The refusal does not corrupt the
`ProblemReadingState` built from sentence 1 — sentence 2's failure
leaves the outer state at its post-sentence-1 value (refusals do not
commit; `end_sentence` is the only commit path).
### Sentence 3 — "If she works 10 hours every day for 5 days, how much money does she make?"
Same refusal mode as sentence 2 (`unexpected_category` on the leading
`If`). Phase 2 / Phase 3 work expands the conditional-frame
vocabulary.
### Pronoun resolution annotation
If sentence 2 or 3 *were* in scope (after Phase 2), the word "she" at
their leading positions would consult
`problem_state.entity_registry = (EntityRef("Tina", "female", 0),)`
under [ADR-0164.2](./ADR-0164.2-pronoun-entity-resolution.md)'s
gender-plus-recency rule. Exactly one matching entity exists →
`PronounResolution(pronoun="she", resolved_to="Tina", at_position=..., entity_source=sentence_0)`
is recorded. Zero matching → `unresolved_pronoun`; more than one →
`ambiguous_pronoun_referent`. The resolution is appended to
`pronoun_resolution_history` exactly when the sentence containing it
closes successfully.
---
## Termination predicate
A `ProblemReadingState` is valid for handoff to `MathProblemGraph`
construction when **all** of the following hold. The predicate is a
pure function: same input → same verdict.
```python
def is_terminable(state: ProblemReadingState) -> bool:
return (
state.entity_registry # ≥1 entity
and state.unknown_target_slot is not None # question target bound
and _every_op_references_known_entity(state) # closure check
and _every_initial_references_known_entity(state) # closure check
and _question_target_entity_resolvable(state) # bound vs registry
and _no_pending_unresolved_pronouns(state) # no dangling refs
and _partial_payloads_project_to_strict_types(state) # ADR-0115 typecheck
)
```
Failure of any condition produces a typed problem-level refusal (see
`READER_REFUSAL_REASONS` problem-level group).
`is_terminable(state)` true → the reader calls a finalizer that
constructs the strict ADR-0115 `MathProblemGraph`:
```python
def finalize(state: ProblemReadingState) -> MathProblemGraph | ReaderRefusal:
"""Project the partial state into a strict MathProblemGraph.
Tuple field ordering on the graph mirrors order-of-introduction in
``state.entity_registry`` and source-text order in
``state.accumulated_*``. Decimal values must round-trip losslessly
through ``int | float`` (the ADR-0115 type); a precision loss
refuses with ``graph_construction_failure``.
"""
```
The output of `finalize` is the exact input shape the existing
binding-graph adapter (ADR-0133) and `BoundUnknown` resolver
(ADR-0135) consume today. The reader does not add a new downstream
contract; it replaces the old front-end's output emit.
---
## Refusal modes — summary
Three lifetime groupings (mirroring the API):
**Token-level (raised by `apply_word`):**
- `unknown_word` — token not in lexicon and no primitive matched.
- `unexpected_category` — category does not satisfy current
expectation and is not a legal frame opener at this position.
- `expectation_collision` — two frame openers would be legal at this
position. Should not occur given a complete precedence table
(defer to ADR-0164.1 / phase-1 measurement); a real occurrence is
evidence the precedence rule needs an ADR.
- `unresolved_pronoun` — pronoun has no matching entity per
ADR-0164.2.
- `ambiguous_pronoun_referent` — pronoun has multiple matching
entities per ADR-0164.2.
**Sentence-level (raised by `end_sentence`):**
- `unfinished_frame` — frame kind never resolved.
- `unattached_quantity` — at least one number was seen but never
attached to an entity + unit.
- `incomplete_operation` — operation frame closed with missing
operand or missing target.
**Problem-level (raised by the finalization predicate):**
- `no_question_target` — problem ended with `unknown_target_slot=None`.
The question sentence either was refused or never identified itself
as a question frame.
- `dangling_entity` — entity appears in registry but has no initial
possession and is not the subject of any operation. Probably a
pronoun mis-resolution upstream.
- `graph_construction_failure` — projection into strict ADR-0115
types failed (precision loss, schema violation). The detail field
carries the ADR-0115 `MathGraphError` message.
Each refusal records `sentence_index` and `token_index` to localize.
Refusals are themselves canonical-bytes-serializable (see §Canonical-bytes).
---
## Interaction with other ADRs
### ADR-0164 (parent)
ADR-0164's `ComprehensionState` sketch (§Decision §2) is the
**inner-level** type under this ADR, renamed `SentenceReadingState`.
The outer-level `ProblemReadingState` is new. ADR-0164's §Phasing is
unchanged: Phase 1 builds the reader for question sentences, which
under the two-level model means Phase 1 implements `apply_word` rules
for the `question_frame` kind and a minimal subset of frame openers,
enough to admit the question-class refusals that block 34/47
train_sample cases.
### ADR-0164.1 (companion)
Lexical primitives are consumed inside `apply_word` step 1 (primitive
scan). The primitive registry's emit-category is what the
expectation-check step compares against. No state-shape dependency
here; ADR-0164.1 specifies *what* primitives produce, ADR-0164.3
specifies *how* the producer's output flows into the state machine.
### ADR-0164.2 (companion)
Pronoun resolution is the only place `apply_word` reads from
`problem_state` non-trivially. ADR-0164.2 specifies the rule;
ADR-0164.3 specifies the carrier (`pronoun_resolution_history` and
the per-sentence resolution buffer). The resolution buffer flushes
to the outer history only on successful `end_sentence`.
### ADR-0115 (downstream)
The `finalize` step produces a strict `MathProblemGraph`.
`MathProblemGraph`'s order-of-introduction invariant
(ADR-0115 docstring) is preserved by reading the outer state's
tuples in their stored order (which is order-of-introduction
because that's how they were appended).
### ADR-0134 / ADR-0135 (downstream binding graph)
Unchanged. The binding-graph adapter reads `MathProblemGraph`; we
produce `MathProblemGraph`; nothing else changes.
### ADR-0165 (regex scope rule)
`apply_word` step 1 consults the lexical primitive registry. Every
primitive in that registry must satisfy ADR-0165 (orthographic-shape
recognition only, never grammar). This ADR does not bypass that
rule; if anything it makes the boundary cleaner because primitive
hits and lexicon hits are explicitly distinct lookup steps.
---
## Open implementation choices (intentionally not pinned here)
These belong in Brief 5 (the `ComprehensionState` / state types
skeleton PR) or in follow-up sub-ADRs. Pinning them now would
over-commit.
1. **`Decimal` precision setting.** `decimal.getcontext().prec`
default is fine for GSM8K (currency to 2 decimal places, quantity
counts as integers). Pin in Brief 5 with a `localcontext` block
if precision needs to be smaller for canonical bytes.
2. **`lookback` window size.** ADR-0164.3 says ≤8 entries. The exact
number can be tuned during Phase 1 measurement; 8 covers all
GSM8K question sentences observed at session time.
3. **Frame kind enumeration.** The four discriminator values
(`initial_state_frame`, `operation_frame`, `question_frame`,
`descriptive_frame`) are sufficient for Phase 1 + Phase 2 scope.
Phase 3 may add `conditional_frame` and `rate_emit_frame`; each
addition is an ADR.
4. **Source-span linkage.** This ADR includes `token_index` and
`source_text_offset` but does not specify a full source-span
record. The binding graph's `SourceSpanLink` (ADR-0132) is the
downstream consumer; whether the reader emits a `SourceSpanLink`
per partial-payload field or a single coarse span per
sentence-payload is a Brief 5 decision.
---
## Acceptance criteria (Proposed → Accepted)
This ADR moves to Accepted when:
1. Brief 5 lands `generate/comprehension/state.py` exporting
`ProblemReadingState` and `SentenceReadingState` (renamed from
ADR-0164's `ComprehensionState` sketch per §Decision above) with
the field tables matching §Decision exactly.
2. The lifecycle API signatures land as Python stubs (no body) in
`generate/comprehension/lifecycle.py`. Phase 1 implements the
bodies.
3. `to_canonical_bytes` / `canonical_hash` implementation passes the
determinism gate in `tests/test_comprehension_state.py` (Brief 5's
test scaffold) on both state levels.
4. `READER_REFUSAL_REASONS` is materialized as a frozenset constant
matching the closed set above.
---
## Cross-references
- **Parent**: [ADR-0164](./ADR-0164-incremental-comprehension-reader.md).
- **Companion sub-ADRs**:
[ADR-0164.1 lexical primitive scope](./ADR-0164.1-lexical-primitive-scope.md),
[ADR-0164.2 pronoun/entity resolution](./ADR-0164.2-pronoun-entity-resolution.md).
- **Boundary invariant**: [ADR-0165](./ADR-0165-regex-scope-rule.md).
- **Downstream target type**: [ADR-0115](./ADR-0115-math-problem-parser-and-graph.md).
- **Downstream binding-graph consumers (unchanged)**:
[ADR-0132](./ADR-0132-binding-graph-data-model.md) /
[ADR-0133](./ADR-0133-binding-graph-adapter.md) /
[ADR-0134](./ADR-0134-binding-graph-admissibility.md) /
[ADR-0135](./ADR-0135-binding-graph-question-target.md).
- **Brief 5 (acceptance dependency)**:
[ADR-0164 §Acceptance criteria #1](./ADR-0164-incremental-comprehension-reader.md#acceptance-criteria-for-this-adr-proposed--accepted).
- **Anchor**: `[[thesis-decoding-not-generating]]` — the state model
is the form of *decoding in progress*. Each transition narrows
the space of possible meanings until the structure that the source
text already implied has surfaced fully.