core/docs/decisions/ADR-0164.3-cross-sentence-state.md

ADR-0164.3 — Cross-Sentence Reading State

Status: Proposed Date: 2026-05-26 Author: Shay Anchor: thesis-decoding-not-generating Parent: ADR-0164 — Incremental Comprehension Reader §Open question #4 Companions: ADR-0164.1 — Lexical Primitive Set Scope, ADR-0164.2 — Pronoun/Entity Resolution Policy, ADR-0165 — Regex Scope Rule Related downstream types: ADR-0115 — MathProblemGraph, ADR-0135 — BoundUnknown resolver

---

Context — why two levels

ADR-0164 §Decision §2 sketches a single ComprehensionState that accumulates entities, quantities, operations, the question target, and a current expectation frame. That sketch is correct for one sentence, but GSM8K problems are multi-sentence: pronouns refer back across sentence boundaries, entities introduced in sentence 1 are mutated in sentence 3, and the question typically lives in the final sentence and refers to state built across all prior sentences.

Two structural facts force a split:

1. Lifetime asymmetry. Some state (entity registry, accumulated initial possessions, accumulated operations, the unknown target) is problem-scoped — it persists across sentence boundaries and accumulates monotonically. Other state (the current expectation frame, pending quantities waiting for unit attachment, the partial frame being built) is sentence-scoped — it must reset cleanly at sentence boundaries so a stray expectation from a prior sentence doesn't bleed into the next one.

2. Refusal locality. When the reader refuses, the refusal points either at a sentence-internal failure (unexpected category, dangling quantity at sentence end, unfinished frame) or at a problem-level failure (unresolved pronoun, conflicting entity reference, no question target by problem end). Conflating both into one state smears the refusal vocabulary and makes the failure modes harder to diagnose.

Collapsing both into a single immutable record is possible but the collapsing buys nothing and costs vocabulary clarity. Two-level keeps each field's lifetime explicit and each refusal mode local to its appropriate level.

Naming note for Brief 5. ADR-0164 §Decision §2's sketched ComprehensionState is structurally the inner level under this ADR — what is named SentenceReadingState here. The Brief 5 PR (ComprehensionState skeleton) will produce both ProblemReadingState and SentenceReadingState to honor the two-level model documented in this ADR. The sketch is preserved as the inner-level field list; the outer level is new.

---

Decision — two-level state model

ProblemReadingState (outer, problem-scoped)

Immutable record. Persists across sentence boundaries. Mutated only by end_sentence returning a new ProblemReadingState that absorbs the just-closed sentence's contribution.

Field	Type	Role
entity_registry	tuple[EntityRef, ...]	Ordered by introduction position. Once an entity enters, it stays. Order-of-introduction matches ADR-0115 MathProblemGraph.entities doctrine.
accumulated_initial_state	tuple[PartialInitialPossession, ...]	Initial-state declarations closed at sentence end. Tuple order is order-of-introduction.
accumulated_operations	tuple[PartialOperation, ...]	Operation declarations closed at sentence end. Tuple order is order-of-introduction (story order — ADR-0115 doctrine).
unknown_target_slot	QuestionTargetSlot | None	Set exactly once, by the sentence containing the question. Locked after setting. None until a question sentence completes.
pronoun_resolution_history	tuple[PronounResolution, ...]	Replay-deterministic log of every pronoun resolution made during reading. Per ADR-0164.2.
sentence_index	int	0-based counter of completed sentences. Increments only on end_sentence.
source_text_offset	int	Character offset into the source problem text at which the next sentence begins. Maintained for span linkage.

PartialInitialPossession and PartialOperation are precursors to the ADR-0115 types InitialPossession and Operation. They are "partial" only in the sense that they can hold None for fields that are optional during construction (e.g. a transfer with a missing target). Once committed to accumulated_*, every field is set. The finalization step (see §Termination) projects them into the strict ADR-0115 types.

SentenceReadingState (inner, sentence-scoped)

Immutable record. Lifetime = one sentence. Created by begin_sentence, mutated by apply_word, consumed by end_sentence.

Field	Type	Role
frame	SentenceFrame | None	The kind of sentence under construction once enough words have been read to decide. Discriminator: initial_state_frame, operation_frame, question_frame, descriptive_frame (context-only, no quantitative contribution). None while the frame is still ambiguous (very early in the sentence).
expectation	ExpectationFrame | None	Open expectation slot — what categories would legally close or advance the current frame. Replaced as the frame narrows. None means "any frame opener is welcome."
pending_quantities	tuple[QuantityRef, ...]	Numbers seen so far in this sentence that haven't been attached to an entity + unit. Drains as units land. Sentence ends with non-empty pending_quantities → refusal unattached_quantity.
pending_entity_ref	EntityRef | None	The entity reference active in the current frame (typically the sentence subject). Set when a proper-noun entity or a resolved pronoun lands in subject position.
pending_verb	VerbReference | None	The verb captured at frame-determining position, waiting for completion (operand, target). Set on verb landing; consumed when the operation closes.
token_index	int	0-based position within the current sentence. Increments on every apply_word.
lookback	tuple[AppliedCategory, ...]	Bounded history (≤8 entries) of categories applied in this sentence with their positions. Enables recontextualization without unbounded backtracking.
partial_frame_payload	FramePayload | None	The frame-kind-specific in-construction structure. For initial_state_frame: a PartialInitialPossession being built up. For operation_frame: a PartialOperation. For question_frame: a QuestionTargetSlot being built up.

SentenceReadingState has read access to ProblemReadingState via the lifecycle API (passed in to apply_word); it cannot mutate the outer state directly. Only end_sentence produces a new ProblemReadingState.

---

Lifecycle API (signatures only — no implementation)

The reader exposes three pure functions. All are deterministic: same inputs → byte-equal outputs and byte-equal canonical hashes.

def begin_sentence(
    problem_state: ProblemReadingState,
    source_text_offset: int,
) -> SentenceReadingState:
    """Open a fresh sentence-local state.

    Resets all sentence-scoped fields. Inherits no transient state from
    prior sentences — the only access to prior context is the
    immutable ``problem_state`` argument (used read-only by
    ``apply_word`` for entity resolution).

    Pure / deterministic. No I/O.
    """


def apply_word(
    sentence_state: SentenceReadingState,
    problem_state: ProblemReadingState,
    word: str,
) -> SentenceReadingState | ReaderRefusal:
    """Advance one token. Returns new sentence state or typed refusal.

    Lookup order per ADR-0164 §Decision §3:
      1. Lexical primitive scan (ADR-0164.1, ADR-0165).
      2. Lexicon lookup (en_core_math_v1, per ADR-0164 §Decision §1).
      3. Expectation check.
      4. Update emit.

    ``problem_state`` is read-only. Pronoun resolution consults
    ``problem_state.entity_registry`` via the rules in
    [ADR-0164.2](./ADR-0164.2-pronoun-entity-resolution.md). Resolutions
    are recorded to a private buffer that ``end_sentence`` later folds
    into ``problem_state.pronoun_resolution_history``.

    Pure / deterministic. No I/O.
    """


def end_sentence(
    sentence_state: SentenceReadingState,
    problem_state: ProblemReadingState,
) -> ProblemReadingState | ReaderRefusal:
    """Close the sentence, fold its contribution into the problem state.

    Finalization rules:

    - ``sentence_state.frame`` must be one of the legal frame
      kinds. ``None`` at end-of-sentence → refusal
      ``unfinished_frame`` (we read words and never decided what shape
      the sentence was).
    - ``sentence_state.pending_quantities`` must be empty. A non-empty
      pending list → refusal ``unattached_quantity``.
    - The ``partial_frame_payload`` is projected into a typed
      ``PartialInitialPossession`` / ``PartialOperation`` /
      ``QuestionTargetSlot`` and appended to the appropriate
      ``problem_state`` tuple.
    - Newly introduced ``EntityRef`` records are appended to
      ``problem_state.entity_registry``.
    - Pronoun resolutions recorded in the sentence state are appended
      to ``problem_state.pronoun_resolution_history``.
    - ``sentence_index`` increments by 1.
    - ``source_text_offset`` advances past the closing punctuation.

    The returned ``ProblemReadingState`` is the input to the next
    ``begin_sentence`` call (next sentence) or to the finalization
    predicate (last sentence).

    Pure / deterministic. No I/O.
    """

ReaderRefusal

Typed refusal record. Carries one of a closed set of reasons plus diagnostic detail.

@dataclass(frozen=True, slots=True)
class ReaderRefusal:
    reason: str            # member of READER_REFUSAL_REASONS
    detail: str            # short human annotation
    sentence_index: int    # which sentence the refusal occurred in
    token_index: int       # position within the sentence (0 if end_sentence-level)
    token_text: str        # the token in question, or "" for non-token refusals

READER_REFUSAL_REASONS = frozenset({
    # apply_word — token-level
    "unknown_word",            # not in lexicon, no primitive matched
    "unexpected_category",     # category does not satisfy current expectation
    "expectation_collision",   # two frame openers would be legal; precedence undecided
    "unresolved_pronoun",      # pronoun has no matching entity in registry
    "ambiguous_pronoun_referent",  # multiple matching entities in registry
    # end_sentence — sentence-level
    "unfinished_frame",        # frame never decided
    "unattached_quantity",     # quantity never bound to entity+unit
    "incomplete_operation",    # operation missing operand or target
    # problem-level (raised by the finalization predicate, not apply/end)
    "no_question_target",      # problem ended with unknown_target_slot=None
    "dangling_entity",         # entity in registry has no initial possession
    "graph_construction_failure",  # MathProblemGraph constructor rejected the projection
})

The vocabulary is closed and ADR-tracked. New reasons require an ADR amendment. This mirrors the ADR-0134 admissibility reason discipline.

---

What persists vs sentence-local — explicit table

Concern	Where it lives	Lifetime
Entity registry	ProblemReadingState	All sentences
Initial possessions accumulated	ProblemReadingState	All sentences
Operations accumulated	ProblemReadingState	All sentences
The unknown target	ProblemReadingState	Set once; locked
Pronoun resolution history	ProblemReadingState	All sentences
Sentence index counter	ProblemReadingState	Monotonic
Current frame kind	SentenceReadingState	One sentence
Current expectation	SentenceReadingState	One sentence; replaced as frame narrows
Pending quantities (un-unit'd)	SentenceReadingState	One sentence; drains as units land
Pending entity reference (subject)	SentenceReadingState	One sentence
Pending verb (operation under construction)	SentenceReadingState	One sentence
Token position counter	SentenceReadingState	Resets at sentence boundary
Recent-category lookback window	SentenceReadingState	One sentence (bounded ≤8)
Frame-payload-in-construction	SentenceReadingState	One sentence; projected at end_sentence

The rule, stated negatively: no field is in both levels. If a field seems to want to live in both, it is split — typically into a "pending" sentence-local version and a "committed" problem-level tuple.

---

Canonical-bytes serialization

Both state levels serialize to deterministic JSON via the same discipline used by MathProblemGraph.canonical_bytes and the existing trace_hash production.

def to_canonical_bytes(state: ProblemReadingState | SentenceReadingState) -> bytes:
    """Sorted-keys, compact-separators JSON. Tuples → lists.

    Decimal values render as strings to preserve precision (the math
    graph uses int|float per ADR-0115; the reader uses Decimal
    internally until projection to the graph). Optional fields are
    omitted from JSON when None (not serialized as 'null') to keep
    the canonical form minimal and to prevent spurious differences
    between states that differ only in which optional fields they
    chose to set explicitly to None.
    """


def canonical_hash(state: ProblemReadingState | SentenceReadingState) -> str:
    """sha256 hex digest of to_canonical_bytes(state).

    Same shape as ADR-0153 turn-event trace-hash. Identical state →
    identical hash. This is the determinism gate enforced by the
    Brief 5 test scaffold.
    """

Serialization rules (matching existing CORE discipline):

1. Sort keys at every level (json.dumps(..., sort_keys=True)).
2. Compact separators (separators=(",", ":")).
3. Tuple → list. Tuples carry ordering meaning; that ordering is preserved by JSON array ordering. The list-vs-tuple distinction is lost on the wire (intentional — JSON has no tuples).
4. Decimal → string. Use str(value) not float coercion. This preserves precision through partial states; the projection to MathProblemGraph (which uses int|float) happens at finalization and must check loss-of-precision explicitly.
5. Frozen dataclasses → dict of field-name → field-value pairs, recursing through children.
6. Enums and Literals → their string value.
7. Optional fields with None → omitted from dict (rule 2 caveat; this prevents {"x": null} vs {} from being different).

ReaderRefusal is serialized too

A refusal is not state, but it must be replay-deterministic for trace audit. to_canonical_bytes(ReaderRefusal(...)) follows the same rules. Two runs that produce the same refusal produce byte-equal refusal records.

---

Worked example — gsm8k-train-sample-v1-0001

"Tina makes $18.00 an hour. If she works more than 8 hours per shift, she is eligible for overtime, which is paid by your hourly wage + 1/2 your hourly wage. If she works 10 hours every day for 5 days, how much money does she make?"

Expected outcome under Phase 1: the reader will admit sentence 1 (rate statement) and refuse sentences 2 and 3 with conditional_* reasons because conditional structure is not in Phase 1 scope. The example demonstrates the state model — not solver success. That is the right kind of demonstration: when the engine refuses, the state at the point of refusal is what tells us why and at what specific position, and that is what enables principled corpus growth.

Sentence 1 — "Tina makes $18.00 an hour."

begin_sentence(problem_state=∅, source_text_offset=0) produces:

SentenceReadingState(
  frame=None,
  expectation=None,
  pending_quantities=(),
  pending_entity_ref=None,
  pending_verb=None,
  token_index=0,
  lookback=(),
  partial_frame_payload=None,
)

Word-by-word (Phase 1 lexicon + primitive set, illustrative):

pos	word	primitive / lexicon hit	state change
0	Tina	lexicon: proper_noun_entity_female	pending_entity_ref = EntityRef("Tina", "female", 0); entity not yet in registry — staged for commit at sentence end.
1	makes	lexicon: accumulation_verb / rate_emit_verb	Frame narrows. pending_verb = VerbReference("makes", "rate_emit", 1). frame = operation_frame (tentative). expectation = "QUANTITY (currency) followed by 'an X'".
2	$18.00	primitive: currency_literal → QuantityRef(Decimal("18.00"), "dollars", "currency", attached_to_entity=None, source_position=2)	pending_quantities = (Q$18.00,). Expectation advances to "'an' or 'per' followed by time-unit".
3	an	lexicon: per_unit_marker (closed-set: an, per, every, each per time-unit)	Expectation narrows to "time-unit-noun". Lookback records per_unit_marker.
4	hour	lexicon: time_unit_noun	Rate composition closes. pending_quantities[0] (the $18.00) becomes the numerator of a rate; hour is the denominator. frame confirmed = operation_frame. partial_frame_payload = PartialOperation(actor="Tina", kind="apply_rate", operand=Rate(18.00, "dollars", "hour")). pending_quantities drains to ().
5	.	sentence terminator

end_sentence(...) projects partial_frame_payload into a typed PartialOperation, commits Tina to entity_registry, appends the operation to accumulated_operations, increments sentence_index.

ProblemReadingState after sentence 1:
  entity_registry = (EntityRef("Tina", "female", 0),)
  accumulated_initial_state = ()
  accumulated_operations = (
    PartialOperation(
      actor="Tina", kind="apply_rate",
      operand=Rate(18.00, "dollars", "hour"),
      target=None,
    ),
  )
  unknown_target_slot = None
  pronoun_resolution_history = ()
  sentence_index = 1

Sentence 2 — "If she works more than 8 hours per shift, ..."

begin_sentence produces a fresh SentenceReadingState.

pos	word	result
0	If	lexicon: conditional_open

Refusal:

ReaderRefusal(
  reason="unexpected_category",
  detail="conditional_open at sentence_index=1, position=0; "
         "conditional_frame is Phase-1 out-of-scope (ADR-0164 §Phasing)",
  sentence_index=1,
  token_index=0,
  token_text="If",
)

This is the correct Phase 1 behavior. The state at the moment of refusal is enough to file a typed teaching candidate for the conditional-frame category. The refusal does not corrupt the ProblemReadingState built from sentence 1 — sentence 2's failure leaves the outer state at its post-sentence-1 value (refusals do not commit; end_sentence is the only commit path).

Sentence 3 — "If she works 10 hours every day for 5 days, how much money does she make?"

Same refusal mode as sentence 2 (unexpected_category on the leading If). Phase 2 / Phase 3 work expands the conditional-frame vocabulary.

Pronoun resolution annotation

If sentence 2 or 3 were in scope (after Phase 2), the word "she" at their leading positions would consult problem_state.entity_registry = (EntityRef("Tina", "female", 0),) under ADR-0164.2's gender-plus-recency rule. Exactly one matching entity exists → PronounResolution(pronoun="she", resolved_to="Tina", at_position=..., entity_source=sentence_0) is recorded. Zero matching → unresolved_pronoun; more than one → ambiguous_pronoun_referent. The resolution is appended to pronoun_resolution_history exactly when the sentence containing it closes successfully.

---

Termination predicate

A ProblemReadingState is valid for handoff to MathProblemGraph construction when all of the following hold. The predicate is a pure function: same input → same verdict.

def is_terminable(state: ProblemReadingState) -> bool:
    return (
        state.entity_registry                                      # ≥1 entity
        and state.unknown_target_slot is not None                  # question target bound
        and _every_op_references_known_entity(state)               # closure check
        and _every_initial_references_known_entity(state)          # closure check
        and _question_target_entity_resolvable(state)              # bound vs registry
        and _no_pending_unresolved_pronouns(state)                 # no dangling refs
        and _partial_payloads_project_to_strict_types(state)       # ADR-0115 typecheck
    )

Failure of any condition produces a typed problem-level refusal (see READER_REFUSAL_REASONS problem-level group).

is_terminable(state) true → the reader calls a finalizer that constructs the strict ADR-0115 MathProblemGraph:

def finalize(state: ProblemReadingState) -> MathProblemGraph | ReaderRefusal:
    """Project the partial state into a strict MathProblemGraph.

    Tuple field ordering on the graph mirrors order-of-introduction in
    ``state.entity_registry`` and source-text order in
    ``state.accumulated_*``. Decimal values must round-trip losslessly
    through ``int | float`` (the ADR-0115 type); a precision loss
    refuses with ``graph_construction_failure``.
    """

The output of finalize is the exact input shape the existing binding-graph adapter (ADR-0133) and BoundUnknown resolver (ADR-0135) consume today. The reader does not add a new downstream contract; it replaces the old front-end's output emit.

---

Refusal modes — summary

Three lifetime groupings (mirroring the API):

Token-level (raised by apply_word):

• unknown_word — token not in lexicon and no primitive matched.
• unexpected_category — category does not satisfy current expectation and is not a legal frame opener at this position.
• expectation_collision — two frame openers would be legal at this position. Should not occur given a complete precedence table (defer to ADR-0164.1 / phase-1 measurement); a real occurrence is evidence the precedence rule needs an ADR.
• unresolved_pronoun — pronoun has no matching entity per ADR-0164.2.
• ambiguous_pronoun_referent — pronoun has multiple matching entities per ADR-0164.2.

Sentence-level (raised by end_sentence):

• unfinished_frame — frame kind never resolved.
• unattached_quantity — at least one number was seen but never attached to an entity + unit.
• incomplete_operation — operation frame closed with missing operand or missing target.

Problem-level (raised by the finalization predicate):

• no_question_target — problem ended with unknown_target_slot=None. The question sentence either was refused or never identified itself as a question frame.
• dangling_entity — entity appears in registry but has no initial possession and is not the subject of any operation. Probably a pronoun mis-resolution upstream.
• graph_construction_failure — projection into strict ADR-0115 types failed (precision loss, schema violation). The detail field carries the ADR-0115 MathGraphError message.

Each refusal records sentence_index and token_index to localize. Refusals are themselves canonical-bytes-serializable (see §Canonical-bytes).

---

Interaction with other ADRs

ADR-0164 (parent)

ADR-0164's ComprehensionState sketch (§Decision §2) is the inner-level type under this ADR, renamed SentenceReadingState. The outer-level ProblemReadingState is new. ADR-0164's §Phasing is unchanged: Phase 1 builds the reader for question sentences, which under the two-level model means Phase 1 implements apply_word rules for the question_frame kind and a minimal subset of frame openers, enough to admit the question-class refusals that block 34/47 train_sample cases.

ADR-0164.1 (companion)

Lexical primitives are consumed inside apply_word step 1 (primitive scan). The primitive registry's emit-category is what the expectation-check step compares against. No state-shape dependency here; ADR-0164.1 specifies what primitives produce, ADR-0164.3 specifies how the producer's output flows into the state machine.

ADR-0164.2 (companion)

Pronoun resolution is the only place apply_word reads from problem_state non-trivially. ADR-0164.2 specifies the rule; ADR-0164.3 specifies the carrier (pronoun_resolution_history and the per-sentence resolution buffer). The resolution buffer flushes to the outer history only on successful end_sentence.

ADR-0115 (downstream)

The finalize step produces a strict MathProblemGraph. MathProblemGraph's order-of-introduction invariant (ADR-0115 docstring) is preserved by reading the outer state's tuples in their stored order (which is order-of-introduction because that's how they were appended).

ADR-0134 / ADR-0135 (downstream binding graph)

Unchanged. The binding-graph adapter reads MathProblemGraph; we produce MathProblemGraph; nothing else changes.

ADR-0165 (regex scope rule)

apply_word step 1 consults the lexical primitive registry. Every primitive in that registry must satisfy ADR-0165 (orthographic-shape recognition only, never grammar). This ADR does not bypass that rule; if anything it makes the boundary cleaner because primitive hits and lexicon hits are explicitly distinct lookup steps.

---

Open implementation choices (intentionally not pinned here)

These belong in Brief 5 (the ComprehensionState / state types skeleton PR) or in follow-up sub-ADRs. Pinning them now would over-commit.

1. Decimal precision setting. decimal.getcontext().prec default is fine for GSM8K (currency to 2 decimal places, quantity counts as integers). Pin in Brief 5 with a localcontext block if precision needs to be smaller for canonical bytes.
2. lookback window size. ADR-0164.3 says ≤8 entries. The exact number can be tuned during Phase 1 measurement; 8 covers all GSM8K question sentences observed at session time.
3. Frame kind enumeration. The four discriminator values (initial_state_frame, operation_frame, question_frame, descriptive_frame) are sufficient for Phase 1 + Phase 2 scope. Phase 3 may add conditional_frame and rate_emit_frame; each addition is an ADR.
4. Source-span linkage. This ADR includes token_index and source_text_offset but does not specify a full source-span record. The binding graph's SourceSpanLink (ADR-0132) is the downstream consumer; whether the reader emits a SourceSpanLink per partial-payload field or a single coarse span per sentence-payload is a Brief 5 decision.

---

Acceptance criteria (Proposed → Accepted)

This ADR moves to Accepted when:

1. Brief 5 lands generate/comprehension/state.py exporting ProblemReadingState and SentenceReadingState (renamed from ADR-0164's ComprehensionState sketch per §Decision above) with the field tables matching §Decision exactly.
2. The lifecycle API signatures land as Python stubs (no body) in generate/comprehension/lifecycle.py. Phase 1 implements the bodies.
3. to_canonical_bytes / canonical_hash implementation passes the determinism gate in tests/test_comprehension_state.py (Brief 5's test scaffold) on both state levels.
4. READER_REFUSAL_REASONS is materialized as a frozenset constant matching the closed set above.

---

Cross-references

• Parent: ADR-0164.
• Companion sub-ADRs: ADR-0164.1 lexical primitive scope, ADR-0164.2 pronoun/entity resolution.
• Boundary invariant: ADR-0165.
• Downstream target type: ADR-0115.
• Downstream binding-graph consumers (unchanged): ADR-0132 / ADR-0133 / ADR-0134 / ADR-0135.
• Brief 5 (acceptance dependency): ADR-0164 §Acceptance criteria #1.
• Anchor: [[thesis-decoding-not-generating]] — the state model is the form of decoding in progress. Each transition narrows the space of possible meanings until the structure that the source text already implied has surfaced fully.