docs(adr-0174): held-hypothesis comprehension with lookback and in-loop contemplation (proposed)
Extends ADR-0164's incremental comprehension reader from single-committed state to held-hypothesis state, adding lookback re-evaluation and in-loop contemplation. Diagnoses why the ADR-0164 reader is wired but inert (all-or-nothing refusal at first unknown token / unexpected category). Architecture: apply_word produces ProblemReadingState.open_hypotheses (small ranked set, HYPOTHESIS_CAP=4 initial). Three operators per token: EMIT (extend compatible hypotheses), ELIMINATE (constraint violations remove hypotheses immediately), HOLD (uncommitted hypotheses survive at lower confidence). At finalize(), |survivors|=0 refuses, |survivors|=1 admits, |survivors|>=2 invokes in-loop contemplate() over vault + packs + audit history. Ambiguity contemplation cannot resolve refuses cleanly, preserving wrong=0. Collapses three parallel parsing systems into one held-hypothesis reader: removes regex parser runtime path (math_parser.py), per-category injector dispatch table (recognizer_anchor_inject._INJECTORS), and duplicate per-sentence-choices scaffolding. Net ~1,900 lines removed; reader grows by ~600 for hypothesis state + reevaluate + contemplate. Preserves ADR-0164 lexicon and category set, ADR-0165 regex scope, ADR-0150/0152/0155/0161 HITL corridor, binding graph + solver substrate, capability-axis lanes, replay-equivalence gate. Trust boundary for in-loop contemplation: read-only over vault/packs/audit; ratification still rides offline HITL. Status: Proposed. Six phases (no timelines) gated on wrong=0 and capability-axis 100% at each transition. Five open questions resolved before Phase 1 PR.
This commit is contained in:
parent
86d4e98d5c
commit
f90f0cf471
1 changed files with 294 additions and 0 deletions
294
docs/decisions/ADR-0174-held-hypothesis-comprehension.md
Normal file
294
docs/decisions/ADR-0174-held-hypothesis-comprehension.md
Normal file
|
|
@ -0,0 +1,294 @@
|
|||
# ADR-0174 — Held-Hypothesis Comprehension with Lookback and In-Loop Contemplation
|
||||
|
||||
**Status:** Proposed
|
||||
**Date:** 2026-05-28
|
||||
**Author:** Shay
|
||||
**Anchor:** [[thesis-decoding-not-generating]]
|
||||
**Parent:** [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md)
|
||||
**Companions:** [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md), [ADR-0163 — Path to GSM8K mastery](./ADR-0163-gsm8k-path-to-mastery.md), [ADR-0150/0152/0155/0161 — Contemplation / HITL corridor](./), [ADR-0170 — Injector Contract Widening](./ADR-0170-injector-contract-widening.md), [ADR-0172 — Math-Corpus Decomposition](./ADR-0172-math-corpus-decomposition-mechanism.md)
|
||||
**Extends:** [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) (lexicon, categories, deterministic per-token reading — all preserved)
|
||||
**Deprecates:**
|
||||
- The per-category injector dispatch table at `generate/recognizer_anchor_inject.py:233` (ADR-0170 W2 and subsequent D.2.x extensions) as the **runtime admission path**. The injectors become hypothesis-emitters within the held-hypothesis reader; they no longer route admission via category lookup.
|
||||
- The legacy regex parser at `generate/math_parser.py` as a **parallel runtime path**. It survives only as the offline measurement-comparison baseline (`--legacy-parser` flag per CLEANUP-C2 brief).
|
||||
|
||||
---
|
||||
|
||||
## Context — what ADR-0164 shipped, why the score hasn't moved
|
||||
|
||||
ADR-0164 correctly diagnosed the regex sentence-template trap and prescribed the right substrate: an operational lexicon, a closed category set, a deterministic shift-reduce reader over categories. Phases 1 and 2 shipped. The reader is wired (`generate/comprehension/lifecycle.py`, `generate/comprehension/state.py`, 2,700 lines combined) and the flag is on in the current train_sample run (`use_reader: true`).
|
||||
|
||||
The eval is unchanged: `correct=3, refused=47, wrong=0` on `evals/gsm8k_math/train_sample/v1`, byte-identical to the regex-only baseline. Reader-trace inspection confirms the reader fires on every problem and produces a `ReaderRefusal` on the first unknown-word or unexpected-category token, then falls through to the regex pipeline which also refuses.
|
||||
|
||||
Three causes, in priority order:
|
||||
|
||||
1. **All-or-nothing refusal at the token level.** `apply_word(state, ps, token)` returns `state | ReaderRefusal`. The first unknown word or unexpected category collapses the entire comprehension. There is no state representing "I do not yet know what role this token plays; I will hold the question open and continue reading." A natural-language reader without held hypotheses cannot read natural language.
|
||||
|
||||
2. **No lookback re-evaluation.** Even when `apply_word` succeeds, the commitment is final. The reader cannot undo a category assignment when a later token reveals the prior commitment was wrong. "She studied for 2 hours on Wednesday and three times as long on Thursday" — the temporal-aggregation reading of "for 2 hours" should be open until the second clause arrives; the reader currently locks it as a quantity at the first sentence boundary.
|
||||
|
||||
3. **Recovery lives offline.** When the reader refuses, the path to learning the missing word is: refusal → audit row → contemplation lane (offline) → workbench → HITL ratify → next session. This is correct for **durable** learning but wrong for **the current problem**. A reader that cannot ask "what would I need to know to finish this read" within the read itself cannot solve problems whose answer is one inference away.
|
||||
|
||||
The result: each accepted category, each new lexicon entry, each new D.2.x injector lifts the eval by 0–2 cases. The `correct` count rises monotonically and slowly; the architecture overfits to GSM8K sentence shapes one shape at a time. This is the **library-of-founds** trap the project thesis explicitly forbids.
|
||||
|
||||
---
|
||||
|
||||
## Diagnosis — single-committed state cannot model partial understanding
|
||||
|
||||
A natural-language reader observes a token, hypothesizes its role *given everything else*, and revises that hypothesis as more tokens arrive. The hypothesis space narrows with reading. At sentence end (or problem end), the surviving hypothesis is admitted.
|
||||
|
||||
ADR-0164's reader is structurally **single-committed**: at every token, exactly one `ProblemReadingState` exists. Disambiguation must complete at the moment of token consumption, because there is nowhere to *defer* an interpretation. When deferral is impossible, the reader compensates with strict expectation-matching, and when expectation matching fails, the reader refuses.
|
||||
|
||||
The same problem manifests three ways:
|
||||
|
||||
- **Ambiguous categories must commit immediately.** "Tina makes" — `makes` could open an `accumulation_verb` frame, an `aggregate_modifier` (as in "makes total"), or a copular construction ("makes Tina happy"). The reader must pick one. Picking wrong means refusing later; picking right means refusing on the next ambiguous token.
|
||||
- **Unknown words refuse rather than narrow.** A token absent from the lexicon emits `ReaderRefusal(unknown_word, position)`. The legitimate alternative — "this is an unknown word at position N; my hypothesis set narrows to interpretations that don't depend on this word's category" — is not representable.
|
||||
- **End-of-sentence finalization is binary.** `finalize(state) → graph | Refusal`. There is no "I have two complete hypotheses; let downstream constraints eliminate one." The held-hypothesis behavior already lives at the candidate-graph layer (Cartesian product over per-sentence choices + constraint elimination via `_initial_admissible` / `roundtrip_admissible`); it should live at the reader.
|
||||
|
||||
ADR-0164's `apply_word(state, ps, token) → state | Refusal` is the right *shape* but the wrong *cardinality*. Promote it to `apply_word(state, ps, token) → state'` where `state'.open_hypotheses` is a small ranked set, and refusal is what happens when the set becomes empty.
|
||||
|
||||
---
|
||||
|
||||
## Decision — held-hypothesis comprehension
|
||||
|
||||
Replace single-committed `ProblemReadingState` with a **held-hypothesis** model in which the reader carries a small ranked set of open interpretations and applies three operators per token:
|
||||
|
||||
1. **EMIT** — the token extends every compatible hypothesis (existing reader behavior, but applied to each hypothesis in the set).
|
||||
2. **ELIMINATE** — hypotheses whose constraints are violated by the new token are removed. Constraints are the existing admissibility predicates (`_initial_admissible`, `roundtrip_admissible`, unit-grounding, verb-kind whitelist) applied to the in-flight hypothesis, not just the finalized candidate.
|
||||
3. **HOLD** — when the token does not commit a category but narrows the role for some hypotheses, those hypotheses survive at lower confidence; uncommitted hypotheses are retained.
|
||||
|
||||
At sentence end (and problem end), the surviving hypothesis set drives admission:
|
||||
|
||||
- **|surviving| = 0** → refuse with the union of elimination reasons.
|
||||
- **|surviving| = 1** → admit (existing graph emission path, unchanged).
|
||||
- **|surviving| ≥ 2** → invoke **in-loop contemplation** to synthesize an elimination sub-question; on resolution, eliminate further; if still ≥ 2 at problem end, refuse with `ambiguous: <N> surviving interpretations` (preserves wrong = 0).
|
||||
|
||||
### Four components
|
||||
|
||||
#### 1. Held-hypothesis state (immutable, frozen-dataclass)
|
||||
|
||||
```text
|
||||
ProblemReadingState:
|
||||
entities: tuple[EntityRef, ...]
|
||||
quantities: tuple[QuantityRef, ...]
|
||||
open_hypotheses: tuple[Hypothesis, ...] # NEW — ranked, ≤ HYPOTHESIS_CAP
|
||||
pronoun_history: tuple[PronounRef, ...]
|
||||
sentence_index: int
|
||||
source_text_offset: int
|
||||
unknown_held: tuple[UnknownHeld, ...] # NEW — tokens awaiting resolution
|
||||
|
||||
Hypothesis:
|
||||
candidate: CandidateInitial | CandidateOperation | CandidateUnknown
|
||||
category_assignments: tuple[CategoryAssignment, ...] # per-token category trace
|
||||
constraint_state: ConstraintState # what's been verified
|
||||
confidence_rank: int # 0 = best; ties broken by appearance order
|
||||
unresolved: tuple[UnresolvedSlot, ...] # what we still need
|
||||
|
||||
UnknownHeld:
|
||||
token: str
|
||||
position: int
|
||||
narrowed_categories: frozenset[Category] # hypotheses that survived this unknown
|
||||
```
|
||||
|
||||
`HYPOTHESIS_CAP` is a small constant (3 or 4). The cap is not a heuristic limit on capability — it is a structural assertion that **a coherent sentence has at most a few plausible parses**. Exceeding the cap is a signal that the read has lost coherence; the reader refuses.
|
||||
|
||||
State remains immutable and canonical-bytes-serializable; the existing `trace_hash` deterministic-replay contract from ADR-0164 is preserved.
|
||||
|
||||
#### 2. Lookback re-evaluation operator
|
||||
|
||||
A new operator on hypothesis state:
|
||||
|
||||
```text
|
||||
reevaluate(hypotheses, new_token, position) -> (refined_hypotheses, eliminations)
|
||||
```
|
||||
|
||||
When a token arrives that resolves an earlier ambiguity, `reevaluate` walks the hypothesis set and:
|
||||
|
||||
- For each hypothesis, recomputes the category assignment of any *prior* token whose role depended on the now-resolved ambiguity.
|
||||
- Records the recomputation in `category_assignments` (so the trace is auditable: "token N was originally `accumulation_verb`, reassigned to `aggregate_modifier` after token M=`total`").
|
||||
- Eliminates hypotheses whose recomputed assignments violate existing constraints.
|
||||
|
||||
Lookback is **bounded**: it walks only back to the last token whose category was contested in the hypothesis set, never the whole sentence. The bound is structural — uncontested tokens contribute no recomputation work — not a heuristic cap.
|
||||
|
||||
#### 3. Continuous constraint propagation
|
||||
|
||||
The candidate-graph layer's admissibility check already enforces grounding (`roundtrip_admissible`) and consistency (`_initial_admissible`). Today these run only on **finalized** candidates at the end of `parse_and_solve`. Move them inside the reader so they fire per-token:
|
||||
|
||||
- After every `EMIT`, run the in-flight constraint check against the partial hypothesis. A hypothesis whose partial state already violates a constraint cannot become valid by adding more tokens — eliminate immediately.
|
||||
- The check is conservative: it only fires when the relevant slots are populated. An incomplete `CandidateOperation` with a verb but no value token does not trigger value-grounding; once the value token arrives, the check fires and eliminates the hypothesis if the value doesn't ground.
|
||||
|
||||
This is **process of elimination during reading**, not pattern-match-then-verify after reading. The set narrows token-by-token via real constraint failures, not via expectation-matching heuristics.
|
||||
|
||||
#### 4. In-loop contemplation
|
||||
|
||||
When the surviving hypothesis set is ambiguous (`|surviving| ≥ 2`) or empty (`|surviving| = 0` with held unknowns), the reader invokes contemplation **inside the read**, not as an offline batch job.
|
||||
|
||||
```text
|
||||
contemplate(state, residual) -> Resolution | None
|
||||
|
||||
Resolution:
|
||||
kind: "eliminate" | "admit_unknown"
|
||||
target_hypothesis_id: int # which hypothesis to refine
|
||||
sub_question: str # the question the reader is asking itself
|
||||
source: "vault" | "pack" | "audit_history"
|
||||
evidence: tuple[EvidenceRef, ...]
|
||||
```
|
||||
|
||||
The contemplation function consults — in this order — the vault (prior session memory), the active language packs (the cognition / relations / math packs), and the audit-history of prior reader refusals on the same word or shape. It returns `Resolution` only when the evidence is unambiguous; ambiguous evidence returns `None` and the reader refuses cleanly.
|
||||
|
||||
`contemplate` does not invoke an LLM, does not sample, does not normalize. It is a deterministic search over already-ratified evidence. The thesis discipline (`thesis-decoding-not-generating`) is enforced: the engine finds the resolution that already exists in its memory, or it refuses.
|
||||
|
||||
The in-loop call site is a single addition to `finalize()`:
|
||||
|
||||
```text
|
||||
def finalize(ps: ProblemReadingState) -> MathProblemGraph | ReaderRefusal:
|
||||
survivors = ps.open_hypotheses
|
||||
if not survivors:
|
||||
return ReaderRefusal(union of elimination reasons)
|
||||
if len(survivors) == 1:
|
||||
return survivors[0].to_graph()
|
||||
# |survivors| ≥ 2
|
||||
resolution = contemplate(ps, residual=survivors)
|
||||
if resolution is None:
|
||||
return ReaderRefusal(f"ambiguous: {len(survivors)} surviving interpretations")
|
||||
return apply_resolution(ps, resolution).to_graph()
|
||||
```
|
||||
|
||||
### What this collapses
|
||||
|
||||
Three parallel parsing systems become one reader with hypothesis state:
|
||||
|
||||
| Module | Today's role | After ADR-0174 |
|
||||
|---|---|---|
|
||||
| `generate/math_parser.py` | Regex sentence-template parser (legacy) | **Removed from runtime.** Survives only as offline baseline behind `--legacy-parser`. |
|
||||
| `generate/math_candidate_graph.py::parse_and_solve` | Recognizer-driven Cartesian-product parser | The held-hypothesis admission orchestrator. The Cartesian product, the per-sentence-choices accumulator, and the elimination passes move *inside* the reader; this module becomes the thin admission/solve dispatcher. |
|
||||
| `generate/comprehension/lifecycle.py` | All-or-nothing reader (Phase 1+2 shipped, inert) | **The reader.** Hosts `open_hypotheses`, `reevaluate`, in-loop `contemplate` integration. |
|
||||
| `generate/recognizer_anchor_inject.py` | Per-category injector dispatch table | **Hypothesis emitters.** The category-keyed lookup is removed; injectors become functions the reader calls to *propose* hypotheses from a recognized anchor. The category tag becomes a property of the emitted hypothesis, not a routing key. |
|
||||
|
||||
Lines saved (approximate, conservative): ~1,800 from `math_parser.py` removal, ~400 from injector dispatch elimination, ~300 from duplicated per-sentence-choice scaffolding in `math_candidate_graph.py`. The reader grows by an estimated ~600 lines for hypothesis state + `reevaluate` + `contemplate` integration. **Net: ~1,900 lines removed.**
|
||||
|
||||
---
|
||||
|
||||
## Constraints (non-negotiable)
|
||||
|
||||
1. **`wrong = 0` at every phase, every round, every split.** Held hypotheses do not weaken admissibility — they relocate the elimination work earlier in the pipeline. The existing `_initial_admissible`, `roundtrip_admissible`, multi-branch-disagreement check, and the case-0050 hazard pin all remain in force. A hypothesis that survives the reader is admitted by the same predicates that admit a candidate today.
|
||||
|
||||
2. **No hidden normalization, stochastic fallback, or best-guess.** `contemplate` is deterministic search over ratified evidence; it returns `Resolution | None`, never a softmax distribution. Ambiguity that contemplation cannot resolve is a refusal, not a guess. `HYPOTHESIS_CAP` is a structural assertion about coherent reads, enforced by refusal — not a probability cutoff.
|
||||
|
||||
3. **No regex sentence-templates.** ADR-0165 is preserved verbatim. Regex remains allowed only at the lexeme level (currency literal, fraction literal, percentage literal, time-unit noun). Any regex matching across word combinations remains forbidden.
|
||||
|
||||
4. **Lexicon and category set remain closed and ADR-tracked.** ADR-0164's lexicon discipline carries forward unchanged. New categories or new composition rules continue to require an ADR. Held-hypothesis reading is a new *use* of the existing categories, not an expansion of the category set.
|
||||
|
||||
5. **Deterministic replay.** Identical input → byte-equal hypothesis trace. The `trace_hash` contract from CLAUDE.md §Runtime Surface Contract is extended to cover `open_hypotheses` serialization. The replay-equivalence gate (ADR-0172 W0-1) catches any non-determinism.
|
||||
|
||||
6. **In-loop contemplation has the same trust boundary as offline contemplation.** It reads from vault, packs, and audit history. It never mutates them in-session. Any ratification that needs to happen still rides the existing HITL corridor (ADR-0150/0152/0155/0161). The in-loop call is read-only; the offline call writes proposals.
|
||||
|
||||
---
|
||||
|
||||
## What's deprecated, what's preserved
|
||||
|
||||
### Deprecated by this ADR
|
||||
|
||||
- **Per-category injector dispatch** (`generate/recognizer_anchor_inject.py:233 _INJECTORS`). The category-keyed lookup table is removed. Injector functions (`inject_discrete_count_statement`, etc.) survive as **hypothesis-emitter helpers** called by the reader from per-token context, but they no longer route admission via category dispatch. The widening pattern (ADR-0170) is preserved as the *type* the emitter returns; the per-category dispatch *as the admission mechanism* is replaced.
|
||||
- **Legacy regex parser as a runtime path** (`generate/math_parser.py`). Removed from the runtime pipeline; the file survives only to back the `--legacy-parser` baseline flag in `evals/gsm8k_math/runner.py` (CLEANUP-C2). Once Phase 3 of this ADR is met, the baseline comparison can be retired and the file deleted entirely.
|
||||
- **All-or-nothing `ReaderRefusal` on first unknown-word or unexpected-category** in `generate/comprehension/lifecycle.py`. Replaced by `UnknownHeld` and hypothesis narrowing.
|
||||
|
||||
### Preserved in full
|
||||
|
||||
- **ADR-0164's operational lexicon and category set.** The work that went into `en_core_math_v1` lexicon, category definitions, and shift-reduce composition rules is the foundation this ADR builds on. Held hypotheses use exactly the same categories and the same lexicon entries.
|
||||
- **The binding graph and solver substrate** (ADR-0116/0117/0132/0133/0134/0135). Downstream consumption is unchanged.
|
||||
- **The HITL corridor** (ADR-0150/0152/0155/0161). Offline contemplation, proposal generation, ratification, and pack-mutation all preserved. In-loop contemplation is additive, not replacement.
|
||||
- **`wrong = 0` doctrine** and the replay-equivalence gate.
|
||||
- **The composition registry and frame registry consumers** (ADR-0168/0169). These are constraint sources the held-hypothesis reader consults; they remain unchanged.
|
||||
- **The capability-axis lanes** (G1–G5, S1) and the math contemplation corpus (ADR-0172). They continue to validate the downstream substrate and act as regression nets.
|
||||
|
||||
### Untouched but adjacent
|
||||
|
||||
- **The recognizer registry / matcher set** (`generate/recognizer_match.py`, `generate/recognizer_registry.py`). Matchers continue to publish anchors; the difference is that anchors feed hypothesis emitters per-token rather than dispatch-table injectors. The HITL pathway that grows the recognizer registry (ADR-0163.C/D) is unchanged.
|
||||
- **The reader-question hybrid path** (ADR-0164.2 / ADR-0164.3 pronoun + cross-sentence). These become first-class hypothesis-producers within the held-hypothesis state.
|
||||
|
||||
---
|
||||
|
||||
## Phasing
|
||||
|
||||
### Phase 1 — Held-hypothesis state primitive
|
||||
|
||||
Implement `Hypothesis`, `UnknownHeld`, and the `open_hypotheses` field on `ProblemReadingState`. Refactor `apply_word` to operate on the hypothesis set (single hypothesis → tuple of 1, behavior preserved). No new admission behavior; the change is structural.
|
||||
|
||||
**Acceptance:**
|
||||
- All existing reader tests pass byte-identical (the single-hypothesis case is exactly today's behavior).
|
||||
- `test_reader_coexistence.py` continues to assert `wrong = 0` on the 50-case train_sample.
|
||||
- `trace_hash` invariant verified on `train_sample/v1`.
|
||||
|
||||
### Phase 2 — Continuous constraint propagation
|
||||
|
||||
Hoist `_initial_admissible` and `roundtrip_admissible` from the candidate-graph layer into per-token in-flight checks within the reader. Hypotheses violating constraints are eliminated immediately, not at end of `parse_and_solve`.
|
||||
|
||||
**Acceptance:**
|
||||
- Train_sample score: no decrease (`correct ≥ 3, wrong = 0` minimum).
|
||||
- Some refusal reasons shift from `no admissible candidate` to early in-reader elimination (visible in `reader_trace`); the total count is preserved or improves.
|
||||
- Holdout split unchanged within ±1 case.
|
||||
|
||||
### Phase 3 — Lookback re-evaluation
|
||||
|
||||
Implement `reevaluate` operator and wire it into `apply_word`. Hypotheses can have prior category assignments refined when later tokens disambiguate.
|
||||
|
||||
**Acceptance:**
|
||||
- The 21 currently-empty `discrete_count_statement` anchors (pronoun subject, compound clause) are revisited: when the question sentence resolves the pronoun, the held statement can be reevaluated. Target: ≥ 5 of these cases admit cleanly.
|
||||
- `correct ≥ 8` on train_sample, `wrong = 0`.
|
||||
|
||||
### Phase 4 — In-loop contemplation
|
||||
|
||||
Wire `contemplate` into `finalize()`. The function consults vault + packs + audit history for deterministic resolution of ambiguous hypothesis sets.
|
||||
|
||||
**Acceptance:**
|
||||
- `correct ≥ 15` on train_sample, `wrong = 0` (passes ADR-0163 Round-1 exit comfortably).
|
||||
- In-loop contemplation events are observable in the reader trace and replay-equivalent (re-running the same input yields the same contemplation call sequence).
|
||||
- No case where in-loop contemplation introduces a wrong answer that offline-only contemplation would not.
|
||||
|
||||
### Phase 5 — Remove parallel parsers
|
||||
|
||||
Delete `generate/math_parser.py`'s runtime invocation paths. Remove the per-category injector dispatch table; injectors become inlined hypothesis emitters. Collapse the duplicate per-sentence-choices scaffolding in `math_candidate_graph.py`.
|
||||
|
||||
**Acceptance:**
|
||||
- `correct ≥ 25` on train_sample, `wrong = 0` (passes ADR-0163 Round-2 exit).
|
||||
- Net line count reduction matches the estimate (~1,900 lines).
|
||||
- The capability-axis lanes G1–G5, S1 remain at 100% `wrong = 0`.
|
||||
|
||||
### Phase 6 — Scale
|
||||
|
||||
Per ADR-0163 §Phase F: public, dev, holdout. No changes to that scope from this ADR.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance criteria for this ADR (Proposed → Accepted)
|
||||
|
||||
This ADR moves to **Accepted** when:
|
||||
|
||||
1. Phase 1 acceptance is met: held-hypothesis state primitive lands, all existing tests pass, `trace_hash` invariant holds.
|
||||
2. A prototype of the lookback `reevaluate` operator exists with at least one test demonstrating prior-token category reassignment under a later disambiguating token.
|
||||
3. A prototype of in-loop `contemplate` exists with at least one test showing deterministic vault-consultation resolving an ambiguous hypothesis set on a curated case.
|
||||
4. Capability-axis lanes G1–G5, S1 remain at 100% `wrong = 0`.
|
||||
5. `verify pinned lane SHAs` continues to pass.
|
||||
6. Cross-references to ADR-0164 (lexicon and category preservation), ADR-0165 (regex scope), and ADR-0172 (replay-equivalence) reviewed and confirmed unchanged.
|
||||
|
||||
---
|
||||
|
||||
## Open questions (resolve before Phase 1 PR)
|
||||
|
||||
1. **HYPOTHESIS_CAP value.** Initial proposal: 4 (matches the candidate-graph layer's empirical observation that GSM8K sentences rarely admit more than 3 distinct readings before a constraint eliminates one). Should be set by measurement after Phase 1 lands the data collection.
|
||||
2. **Lookback walk bound.** Whether to bound by token-distance (e.g., re-evaluate at most 10 tokens back) or by category-contest (re-evaluate only tokens whose category was contested in the hypothesis set). Proposal: category-contest, because it is structural and cannot mask a real ambiguity behind a numeric cap.
|
||||
3. **Contemplation evidence-ordering precedence.** When vault, packs, and audit history all return candidate resolutions, the ordering matters for determinism. Proposal: vault > packs > audit history, mirroring the existing offline contemplation precedence in `teaching/contemplation.py`. Confirm against the audit-history schema before Phase 4.
|
||||
4. **Sub-ADR for hypothesis emitters.** The per-category injector functions becoming hypothesis emitters may warrant its own sub-ADR (ADR-0174.1) given the surface area touched. Decide after Phase 1 lands and the call surface is concrete.
|
||||
5. **Eval set adequacy.** The 50-case `train_sample/v1` may be too narrow to validate the held-hypothesis approach; some of the architecture's value (lookback, in-loop contemplation) only manifests on multi-step problems with cross-sentence ambiguity. May need a curated 20-case sub-corpus exercising specifically these patterns before Phase 3 measurement.
|
||||
|
||||
---
|
||||
|
||||
## Cross-references
|
||||
|
||||
- **Parent**: [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) — lexicon, categories, deterministic reader; this ADR extends without replacing.
|
||||
- **Constraint scope**: [ADR-0165](./ADR-0165-regex-scope-rule.md) — preserved verbatim.
|
||||
- **Eval target**: `evals/gsm8k_math/train_sample/v1/report.json`, `evals/gsm8k_math/train_sample/v1/README.md` (§ADR-0164 Reader — Zero-Delta Diagnosis).
|
||||
- **Substrate this builds on**: `generate/comprehension/lifecycle.py`, `generate/comprehension/state.py`, `generate/recognizer_anchor_inject.py`, `generate/math_candidate_graph.py`, `generate/math_roundtrip.py`.
|
||||
- **Cleanup briefs touching the deprecated paths**: `docs/handoff/CLEANUP-C2-run-lane-migration.md`, `docs/handoff/CLEANUP-C4-compositions-compile.md`.
|
||||
- **Thesis anchor**: [[thesis-decoding-not-generating]] — every change in this ADR must pass the "teach the engine to find, not store another found thing" gate.
|
||||
- **HITL corridor preserved**: ADR-0150 (contemplation), ADR-0152 (proposal), ADR-0155 (review), ADR-0161 (HITL queue), ADR-0172 (math-corpus decomposition).
|
||||
- **Anti-overfitting obligations**: ADR-0114a — held-hypothesis reads are evaluated against the same obligations as the existing pipeline; perturbation, OOD ratio, depth curve, and adversarial axes all apply.
|
||||
Loading…
Reference in a new issue