core/docs/decisions/ADR-0174-held-hypothesis-comprehension.md

ADR-0174 — Held-Hypothesis Comprehension with Lookback and In-Loop Contemplation

Status: Proposed Date: 2026-05-28 Author: Shay Anchor: thesis-decoding-not-generating Parent: ADR-0164 — Incremental Comprehension Reader Companions: ADR-0165 — Regex Scope Rule, ADR-0163 — Path to GSM8K mastery, ADR-0150/0152/0155/0161 — Contemplation / HITL corridor, ADR-0170 — Injector Contract Widening, ADR-0172 — Math-Corpus Decomposition Extends: ADR-0164 (lexicon, categories, deterministic per-token reading — all preserved) Deprecates:

• The per-category injector dispatch table at generate/recognizer_anchor_inject.py:233 (ADR-0170 W2 and subsequent D.2.x extensions) as the runtime admission path. The injectors become hypothesis-emitters within the held-hypothesis reader; they no longer route admission via category lookup.
• The legacy regex parser at generate/math_parser.py as a parallel runtime path. It survives only as the offline measurement-comparison baseline (--legacy-parser flag per CLEANUP-C2 brief).

---

Context — what ADR-0164 shipped, why the score hasn't moved

ADR-0164 correctly diagnosed the regex sentence-template trap and prescribed the right substrate: an operational lexicon, a closed category set, a deterministic shift-reduce reader over categories. Phases 1 and 2 shipped. The reader is wired (generate/comprehension/lifecycle.py, generate/comprehension/state.py, 2,700 lines combined) and the flag is on in the current train_sample run (use_reader: true).

The eval is unchanged: correct=3, refused=47, wrong=0 on evals/gsm8k_math/train_sample/v1, byte-identical to the regex-only baseline. Reader-trace inspection confirms the reader fires on every problem and produces a ReaderRefusal on the first unknown-word or unexpected-category token, then falls through to the regex pipeline which also refuses.

Three causes, in priority order:

1. All-or-nothing refusal at the token level. apply_word(state, ps, token) returns state | ReaderRefusal. The first unknown word or unexpected category collapses the entire comprehension. There is no state representing "I do not yet know what role this token plays; I will hold the question open and continue reading." A natural-language reader without held hypotheses cannot read natural language.

2. No lookback re-evaluation. Even when apply_word succeeds, the commitment is final. The reader cannot undo a category assignment when a later token reveals the prior commitment was wrong. "She studied for 2 hours on Wednesday and three times as long on Thursday" — the temporal-aggregation reading of "for 2 hours" should be open until the second clause arrives; the reader currently locks it as a quantity at the first sentence boundary.

3. Recovery lives offline. When the reader refuses, the path to learning the missing word is: refusal → audit row → contemplation lane (offline) → workbench → HITL ratify → next session. This is correct for durable learning but wrong for the current problem. A reader that cannot ask "what would I need to know to finish this read" within the read itself cannot solve problems whose answer is one inference away.

The result: each accepted category, each new lexicon entry, each new D.2.x injector lifts the eval by 0–2 cases. The correct count rises monotonically and slowly; the architecture overfits to GSM8K sentence shapes one shape at a time. This is the library-of-founds trap the project thesis explicitly forbids.

---

Diagnosis — single-committed state cannot model partial understanding

A natural-language reader observes a token, hypothesizes its role given everything else, and revises that hypothesis as more tokens arrive. The hypothesis space narrows with reading. At sentence end (or problem end), the surviving hypothesis is admitted.

ADR-0164's reader is structurally single-committed: at every token, exactly one ProblemReadingState exists. Disambiguation must complete at the moment of token consumption, because there is nowhere to defer an interpretation. When deferral is impossible, the reader compensates with strict expectation-matching, and when expectation matching fails, the reader refuses.

The same problem manifests three ways:

• Ambiguous categories must commit immediately. "Tina makes" — makes could open an accumulation_verb frame, an aggregate_modifier (as in "makes total"), or a copular construction ("makes Tina happy"). The reader must pick one. Picking wrong means refusing later; picking right means refusing on the next ambiguous token.
• Unknown words refuse rather than narrow. A token absent from the lexicon emits ReaderRefusal(unknown_word, position). The legitimate alternative — "this is an unknown word at position N; my hypothesis set narrows to interpretations that don't depend on this word's category" — is not representable.
• End-of-sentence finalization is binary. finalize(state) → graph | Refusal. There is no "I have two complete hypotheses; let downstream constraints eliminate one." The held-hypothesis behavior already lives at the candidate-graph layer (Cartesian product over per-sentence choices + constraint elimination via _initial_admissible / roundtrip_admissible); it should live at the reader.

ADR-0164's apply_word(state, ps, token) → state | Refusal is the right shape but the wrong cardinality. Promote it to apply_word(state, ps, token) → state' where state'.open_hypotheses is a small ranked set, and refusal is what happens when the set becomes empty.

---

Decision — held-hypothesis comprehension

Replace single-committed ProblemReadingState with a held-hypothesis model in which the reader carries a small ranked set of open interpretations and applies three operators per token:

1. EMIT — the token extends every compatible hypothesis (existing reader behavior, but applied to each hypothesis in the set).
2. ELIMINATE — hypotheses whose constraints are violated by the new token are removed. Constraints are the existing admissibility predicates (_initial_admissible, roundtrip_admissible, unit-grounding, verb-kind whitelist) applied to the in-flight hypothesis, not just the finalized candidate.
3. HOLD — when the token does not commit a category but narrows the role for some hypotheses, those hypotheses survive at lower confidence; uncommitted hypotheses are retained.

At sentence end (and problem end), the surviving hypothesis set drives admission:

• |surviving| = 0 → refuse with the union of elimination reasons.
• |surviving| = 1 → admit (existing graph emission path, unchanged).
• |surviving| ≥ 2 → invoke in-loop contemplation to synthesize an elimination sub-question; on resolution, eliminate further; if still ≥ 2 at problem end, refuse with ambiguous: <N> surviving interpretations (preserves wrong = 0).

Four components

1. Held-hypothesis state (immutable, frozen-dataclass)

ProblemReadingState:
  entities:               tuple[EntityRef, ...]
  quantities:             tuple[QuantityRef, ...]
  open_hypotheses:        tuple[Hypothesis, ...]    # NEW — ranked, ≤ HYPOTHESIS_CAP
  pronoun_history:        tuple[PronounRef, ...]
  sentence_index:         int
  source_text_offset:     int
  unknown_held:           tuple[UnknownHeld, ...]   # NEW — tokens awaiting resolution

Hypothesis:
  candidate:              CandidateInitial | CandidateOperation | CandidateUnknown
  category_assignments:   tuple[CategoryAssignment, ...]   # per-token category trace
  constraint_state:       ConstraintState                  # what's been verified
  confidence_rank:        int                              # 0 = best; ties broken by appearance order
  unresolved:             tuple[UnresolvedSlot, ...]       # what we still need

UnknownHeld:
  token:                  str
  position:               int
  narrowed_categories:    frozenset[Category]              # hypotheses that survived this unknown

HYPOTHESIS_CAP is a small constant (3 or 4). The cap is not a heuristic limit on capability — it is a structural assertion that a coherent sentence has at most a few plausible parses. Exceeding the cap is a signal that the read has lost coherence; the reader refuses.

State remains immutable and canonical-bytes-serializable; the existing trace_hash deterministic-replay contract from ADR-0164 is preserved.

2. Lookback re-evaluation operator

A new operator on hypothesis state:

reevaluate(hypotheses, new_token, position) -> (refined_hypotheses, eliminations)

When a token arrives that resolves an earlier ambiguity, reevaluate walks the hypothesis set and:

• For each hypothesis, recomputes the category assignment of any prior token whose role depended on the now-resolved ambiguity.
• Records the recomputation in category_assignments (so the trace is auditable: "token N was originally accumulation_verb, reassigned to aggregate_modifier after token M=total").
• Eliminates hypotheses whose recomputed assignments violate existing constraints.

Lookback is bounded: it walks only back to the last token whose category was contested in the hypothesis set, never the whole sentence. The bound is structural — uncontested tokens contribute no recomputation work — not a heuristic cap.

3. Continuous constraint propagation

The candidate-graph layer's admissibility check already enforces grounding (roundtrip_admissible) and consistency (_initial_admissible). Today these run only on finalized candidates at the end of parse_and_solve. Move them inside the reader so they fire per-token:

• After every EMIT, run the in-flight constraint check against the partial hypothesis. A hypothesis whose partial state already violates a constraint cannot become valid by adding more tokens — eliminate immediately.
• The check is conservative: it only fires when the relevant slots are populated. An incomplete CandidateOperation with a verb but no value token does not trigger value-grounding; once the value token arrives, the check fires and eliminates the hypothesis if the value doesn't ground.

This is process of elimination during reading, not pattern-match-then-verify after reading. The set narrows token-by-token via real constraint failures, not via expectation-matching heuristics.

4. In-loop contemplation

When the surviving hypothesis set is ambiguous (|surviving| ≥ 2) or empty (|surviving| = 0 with held unknowns), the reader invokes contemplation inside the read, not as an offline batch job.

contemplate(state, residual) -> Resolution | None

Resolution:
  kind:            "eliminate" | "admit_unknown"
  target_hypothesis_id: int               # which hypothesis to refine
  sub_question:    str                    # the question the reader is asking itself
  source:          "vault" | "pack" | "audit_history"
  evidence:        tuple[EvidenceRef, ...]

The contemplation function consults — in this order — the vault (prior session memory), the active language packs (the cognition / relations / math packs), and the audit-history of prior reader refusals on the same word or shape. It returns Resolution only when the evidence is unambiguous; ambiguous evidence returns None and the reader refuses cleanly.

contemplate does not invoke an LLM, does not sample, does not normalize. It is a deterministic search over already-ratified evidence. The thesis discipline (thesis-decoding-not-generating) is enforced: the engine finds the resolution that already exists in its memory, or it refuses.

The in-loop call site is a single addition to finalize():

def finalize(ps: ProblemReadingState) -> MathProblemGraph | ReaderRefusal:
    survivors = ps.open_hypotheses
    if not survivors:
        return ReaderRefusal(union of elimination reasons)
    if len(survivors) == 1:
        return survivors[0].to_graph()
    # |survivors| ≥ 2
    resolution = contemplate(ps, residual=survivors)
    if resolution is None:
        return ReaderRefusal(f"ambiguous: {len(survivors)} surviving interpretations")
    return apply_resolution(ps, resolution).to_graph()

What this collapses

Three parallel parsing systems become one reader with hypothesis state:

Module	Today's role	After ADR-0174
generate/math_parser.py	Regex sentence-template parser (legacy)	Removed from runtime. Survives only as offline baseline behind --legacy-parser.
generate/math_candidate_graph.py::parse_and_solve	Recognizer-driven Cartesian-product parser	The held-hypothesis admission orchestrator. The Cartesian product, the per-sentence-choices accumulator, and the elimination passes move inside the reader; this module becomes the thin admission/solve dispatcher.
generate/comprehension/lifecycle.py	All-or-nothing reader (Phase 1+2 shipped, inert)	The reader. Hosts open_hypotheses, reevaluate, in-loop contemplate integration.
generate/recognizer_anchor_inject.py	Per-category injector dispatch table	Hypothesis emitters. The category-keyed lookup is removed; injectors become functions the reader calls to propose hypotheses from a recognized anchor. The category tag becomes a property of the emitted hypothesis, not a routing key.

Lines saved (approximate, conservative): ~1,800 from math_parser.py removal, ~400 from injector dispatch elimination, ~300 from duplicated per-sentence-choice scaffolding in math_candidate_graph.py. The reader grows by an estimated ~600 lines for hypothesis state + reevaluate + contemplate integration. Net: ~1,900 lines removed.

---

Constraints (non-negotiable)

1. wrong = 0 at every phase, every round, every split. Held hypotheses do not weaken admissibility — they relocate the elimination work earlier in the pipeline. The existing _initial_admissible, roundtrip_admissible, multi-branch-disagreement check, and the case-0050 hazard pin all remain in force. A hypothesis that survives the reader is admitted by the same predicates that admit a candidate today.

2. No hidden normalization, stochastic fallback, or best-guess. contemplate is deterministic search over ratified evidence; it returns Resolution | None, never a softmax distribution. Ambiguity that contemplation cannot resolve is a refusal, not a guess. HYPOTHESIS_CAP is a structural assertion about coherent reads, enforced by refusal — not a probability cutoff.

3. No regex sentence-templates. ADR-0165 is preserved verbatim. Regex remains allowed only at the lexeme level (currency literal, fraction literal, percentage literal, time-unit noun). Any regex matching across word combinations remains forbidden.

4. Lexicon and category set remain closed and ADR-tracked. ADR-0164's lexicon discipline carries forward unchanged. New categories or new composition rules continue to require an ADR. Held-hypothesis reading is a new use of the existing categories, not an expansion of the category set.

5. Deterministic replay. Identical input → byte-equal hypothesis trace. The trace_hash contract from CLAUDE.md §Runtime Surface Contract is extended to cover open_hypotheses serialization. The replay-equivalence gate (ADR-0172 W0-1) catches any non-determinism.

6. In-loop contemplation has the same trust boundary as offline contemplation. It reads from vault, packs, and audit history. It never mutates them in-session. Any ratification that needs to happen still rides the existing HITL corridor (ADR-0150/0152/0155/0161). The in-loop call is read-only; the offline call writes proposals.

---

What's deprecated, what's preserved

Deprecated by this ADR

• Per-category injector dispatch (generate/recognizer_anchor_inject.py:233 _INJECTORS). The category-keyed lookup table is removed. Injector functions (inject_discrete_count_statement, etc.) survive as hypothesis-emitter helpers called by the reader from per-token context, but they no longer route admission via category dispatch. The widening pattern (ADR-0170) is preserved as the type the emitter returns; the per-category dispatch as the admission mechanism is replaced.
• Legacy regex parser as a runtime path (generate/math_parser.py). Removed from the runtime pipeline; the file survives only to back the --legacy-parser baseline flag in evals/gsm8k_math/runner.py (CLEANUP-C2). Once Phase 3 of this ADR is met, the baseline comparison can be retired and the file deleted entirely.
• All-or-nothing ReaderRefusal on first unknown-word or unexpected-category in generate/comprehension/lifecycle.py. Replaced by UnknownHeld and hypothesis narrowing.

Preserved in full

• ADR-0164's operational lexicon and category set. The work that went into en_core_math_v1 lexicon, category definitions, and shift-reduce composition rules is the foundation this ADR builds on. Held hypotheses use exactly the same categories and the same lexicon entries.
• The binding graph and solver substrate (ADR-0116/0117/0132/0133/0134/0135). Downstream consumption is unchanged.
• The HITL corridor (ADR-0150/0152/0155/0161). Offline contemplation, proposal generation, ratification, and pack-mutation all preserved. In-loop contemplation is additive, not replacement.
• wrong = 0 doctrine and the replay-equivalence gate.
• The composition registry and frame registry consumers (ADR-0168/0169). These are constraint sources the held-hypothesis reader consults; they remain unchanged.
• The capability-axis lanes (G1–G5, S1) and the math contemplation corpus (ADR-0172). They continue to validate the downstream substrate and act as regression nets.

Untouched but adjacent

• The recognizer registry / matcher set (generate/recognizer_match.py, generate/recognizer_registry.py). Matchers continue to publish anchors; the difference is that anchors feed hypothesis emitters per-token rather than dispatch-table injectors. The HITL pathway that grows the recognizer registry (ADR-0163.C/D) is unchanged.
• The reader-question hybrid path (ADR-0164.2 / ADR-0164.3 pronoun + cross-sentence). These become first-class hypothesis-producers within the held-hypothesis state.

---

Phasing

Phase 1 — Held-hypothesis state primitive

Implement Hypothesis, UnknownHeld, and the open_hypotheses field on ProblemReadingState. Refactor apply_word to operate on the hypothesis set (single hypothesis → tuple of 1, behavior preserved). No new admission behavior; the change is structural.

Acceptance:

• All existing reader tests pass byte-identical (the single-hypothesis case is exactly today's behavior).
• test_reader_coexistence.py continues to assert wrong = 0 on the 50-case train_sample.
• trace_hash invariant verified on train_sample/v1.

Phase 2 — Continuous constraint propagation

Hoist _initial_admissible and roundtrip_admissible from the candidate-graph layer into per-token in-flight checks within the reader. Hypotheses violating constraints are eliminated immediately, not at end of parse_and_solve.

Acceptance:

• Train_sample score: no decrease (correct ≥ 3, wrong = 0 minimum).
• Some refusal reasons shift from no admissible candidate to early in-reader elimination (visible in reader_trace); the total count is preserved or improves.
• Holdout split unchanged within ±1 case.

Phase 3 — Lookback re-evaluation

Implement reevaluate operator and wire it into apply_word. Hypotheses can have prior category assignments refined when later tokens disambiguate.

Acceptance:

• The 21 currently-empty discrete_count_statement anchors (pronoun subject, compound clause) are revisited: when the question sentence resolves the pronoun, the held statement can be reevaluated. Target: ≥ 5 of these cases admit cleanly.
• correct ≥ 8 on train_sample, wrong = 0.

Phase 4 — In-loop contemplation

Wire contemplate into finalize(). The function consults vault + packs + audit history for deterministic resolution of ambiguous hypothesis sets.

Acceptance:

• correct ≥ 15 on train_sample, wrong = 0 (passes ADR-0163 Round-1 exit comfortably).
• In-loop contemplation events are observable in the reader trace and replay-equivalent (re-running the same input yields the same contemplation call sequence).
• No case where in-loop contemplation introduces a wrong answer that offline-only contemplation would not.

Phase 5 — Remove parallel parsers

Delete generate/math_parser.py's runtime invocation paths. Remove the per-category injector dispatch table; injectors become inlined hypothesis emitters. Collapse the duplicate per-sentence-choices scaffolding in math_candidate_graph.py.

Acceptance:

• correct ≥ 25 on train_sample, wrong = 0 (passes ADR-0163 Round-2 exit).
• Net line count reduction matches the estimate (~1,900 lines).
• The capability-axis lanes G1–G5, S1 remain at 100% wrong = 0.

Phase 6 — Scale

Per ADR-0163 §Phase F: public, dev, holdout. No changes to that scope from this ADR.

---

Acceptance criteria for this ADR (Proposed → Accepted)

This ADR moves to Accepted when:

1. Phase 1 acceptance is met: held-hypothesis state primitive lands, all existing tests pass, trace_hash invariant holds.
2. A prototype of the lookback reevaluate operator exists with at least one test demonstrating prior-token category reassignment under a later disambiguating token.
3. A prototype of in-loop contemplate exists with at least one test showing deterministic vault-consultation resolving an ambiguous hypothesis set on a curated case.
4. Capability-axis lanes G1–G5, S1 remain at 100% wrong = 0.
5. verify pinned lane SHAs continues to pass.
6. Cross-references to ADR-0164 (lexicon and category preservation), ADR-0165 (regex scope), and ADR-0172 (replay-equivalence) reviewed and confirmed unchanged.

---

Open questions (resolve before Phase 1 PR)

1. HYPOTHESIS_CAP value. Initial proposal: 4 (matches the candidate-graph layer's empirical observation that GSM8K sentences rarely admit more than 3 distinct readings before a constraint eliminates one). Should be set by measurement after Phase 1 lands the data collection.
2. Lookback walk bound. Whether to bound by token-distance (e.g., re-evaluate at most 10 tokens back) or by category-contest (re-evaluate only tokens whose category was contested in the hypothesis set). Proposal: category-contest, because it is structural and cannot mask a real ambiguity behind a numeric cap.
3. Contemplation evidence-ordering precedence. When vault, packs, and audit history all return candidate resolutions, the ordering matters for determinism. Proposal: vault > packs > audit history, mirroring the existing offline contemplation precedence in teaching/contemplation.py. Confirm against the audit-history schema before Phase 4.
4. Sub-ADR for hypothesis emitters. The per-category injector functions becoming hypothesis emitters may warrant its own sub-ADR (ADR-0174.1) given the surface area touched. Decide after Phase 1 lands and the call surface is concrete.
5. Eval set adequacy. The 50-case train_sample/v1 may be too narrow to validate the held-hypothesis approach; some of the architecture's value (lookback, in-loop contemplation) only manifests on multi-step problems with cross-sentence ambiguity. May need a curated 20-case sub-corpus exercising specifically these patterns before Phase 3 measurement.

---

Cross-references

• Parent: ADR-0164 — lexicon, categories, deterministic reader; this ADR extends without replacing.
• Constraint scope: ADR-0165 — preserved verbatim.
• Eval target: evals/gsm8k_math/train_sample/v1/report.json, evals/gsm8k_math/train_sample/v1/README.md (§ADR-0164 Reader — Zero-Delta Diagnosis).
• Substrate this builds on: generate/comprehension/lifecycle.py, generate/comprehension/state.py, generate/recognizer_anchor_inject.py, generate/math_candidate_graph.py, generate/math_roundtrip.py.
• Cleanup briefs touching the deprecated paths: docs/handoff/CLEANUP-C2-run-lane-migration.md, docs/handoff/CLEANUP-C4-compositions-compile.md.
• Thesis anchor: thesis-decoding-not-generating — every change in this ADR must pass the "teach the engine to find, not store another found thing" gate.
• HITL corridor preserved: ADR-0150 (contemplation), ADR-0152 (proposal), ADR-0155 (review), ADR-0161 (HITL queue), ADR-0172 (math-corpus decomposition).
• Anti-overfitting obligations: ADR-0114a — held-hypothesis reads are evaluated against the same obligations as the existing pipeline; perturbation, OOD ratio, depth curve, and adversarial axes all apply.