Self-verification strengthening (microscope-driven). The Phase 3b measurement
showed self-verification was necessary-but-not-sufficient: 9/13 self-verified
attempts were wrong. Inspecting them deterministically revealed most were
correct FIRST STEPS of multi-step problems that ignored numbers stated elsewhere.
Adds clause 5 to self_verifies: a derivation must account for every quantity the
problem states (problem quantities subset of used). Refuse-preferring: unused
quantities -> not self-verified. This catches the multi-step-incomplete attempts
the grounding/cue/unit clauses cannot (their operands ARE grounded).
Practice measurement: wrongs 9 -> 2 (4 correct / 2 wrong / 44 refused). The 2
survivors (0015, 0025) are COMPLETE but wrong due to missed WORD-quantities
('twice', 'her friends') -> the microscope points the next change at extraction.
Updated the disagreement test to use two complete derivations; added an
incomplete-refusal test. 32 tests pass; smoke green; serving untouched (sealed).
Solid: 4 invariants proven by failing-under-violation tests; 84 green; seal
verified. No live hazards (9 search wrongs are sealed eliminations).
Drift: (1) 3a gate is PARTIAL vs spec — round-trip + no-contradiction not wired;
3b's necessary-not-sufficient finding follows -> a self-verification-strengthening
phase must precede Phase 5. (2) class taxonomy: Phase 2 uses gold operation-class
not capability-axis. (3) minor defensive-branch test gaps (no risk).
ADR-0175 Phase 3b — the first live attempt generator. Runs only in the sealed
practice lane, only on cases the engine refused; every proposal is gated by the
Phase 3a self-verification gate.
generate/derivation/:
- extract.py: extract_quantities() — lexeme-level (number + unit word; ADR-0165).
- search.py: search_multiplicative() — one in-clause product candidate per
sentence with >=2 quantities + a present multiplicative cue; gated by
select_self_verified. Per-sentence scope + multi-candidate disagreement give
the uniqueness gate real teeth (two qualifying sentences -> refuse). The cue
set {each,every,for,per,times} is an explicit PROVISIONAL hypothesis the
practice loop refines, not a claimed-correct grammar.
evals/gsm8k_math/practice/v1/search_runner.py: search_augmented_scorer +
build_search_report — base scorer, then a practice-only attempt on refusals.
MEASUREMENT (the deliverable, per the breadth-of-impact test):
practice with search: correct=4 wrong=9 refused=37 (baseline 3/0/47)
- Flips +1 (0021, the clean in-clause aggregate) and its renumbered/reworded
variants (ADR-0114a perturbation guard) -> a real capability, not memorisation.
- 9 wrong attempts -> elimination records (§9), the learning signal. The naive
full-product cue model over-attempts; the eliminations are exactly the signal
that refines it.
HONEST FINDING: self-verification (grounding ∧ cue ∧ unit ∧ uniqueness) is
NECESSARY but NOT SUFFICIENT — 9/13 self-verified attempts were wrong vs gold.
The gap is cue PRECISION / which-quantities-compose (the knowledge axis), not
'can we multiply' (skill). This is why the search runs sealed: gold catches the
9, and case 0050 (canary) attempted-and-failed IN PRACTICE without touching
serving -> validates the seal.
Invariants: #1 seal (serving still 3/47/0; 0050 refuses in serving; no
generate/chat import of the lane), #3 determinism. Serving wrong=0 untouched.
Verified: 3a+3b 31/31; ruff clean; serving lane 4/4; smoke 67/67.
ADR-0175 Phase 3 splits wrong=0-first: build the gate (3a) and PROVE invariant #2
before the bounded search (3b) that could exploit gaps.
generate/derivation/:
- model.py: Quantity / Step / GroundedDerivation. A derivation is a left-fold over
text-sourced quantities; each Step carries its licensing cue (the lexeme the
search claims licenses the op).
- verify.py: self_verifies() — grounded operands ∧ grounded operation cues ∧ unit
consistency ∧ no divide-by-zero. Grounding REUSES the canonical primitives from
math_roundtrip (_tokens/_token_in/_value_grounds) so the gate cannot drift from
the round-trip contract. select_self_verified() adds the uniqueness rule:
unique self-verifying answer resolves; zero or disagreeing refuse (wrong=0).
INVARIANT #2 proven (TestInvariant2_NoSpuriousSelfVerification): the gate refuses
to self-verify a derivation that is not grounded+unit-consistent+unique even when
its value coincides with gold — the 20/5==4 class:
- invented operand not in text -> refused
- operation cue not in text -> refused (division not licensed by any present cue)
- value coincidence (20/5=4) with ungrounded op -> still refused
- add across units (pounds + reps) -> refused
- divide-by-zero -> refused
Plus uniqueness: disagreeing grounded derivations -> refuse; agreeing -> resolve.
Phase 3a is inert (nothing wires generate.derivation into serving). 3b is the
bounded search that produces derivations for this gate + measures the flip-curve
in the practice lane under perturbation.
Verified: 16/16; ruff clean; smoke 67/67; no serving import.
ADR-0175 Phase 2 — a NEW lane (evals/gsm8k_math/practice/v1/), separate from the
wrong=0-pinned serving runner which is NOT modified. Runs the 50 cases in
practice mode: scores correct/wrong/refused as practice metrics, feeds per-class
counts into the Phase 1 ledger, diagnoses every refusal (§8), emits an
elimination record per wrong.
- classify_operation: gold-derived primary op class {multiplicative,divisive,
additive} from <<a*b=c>> calc annotations (Tier-1 checkable in practice).
- diagnose_refusal (§8): skill_gap / knowledge_gap / genuine_ambiguity router.
- EliminationRecord (§9): wrong attempt gold caught -> pruning signal.
- PracticeReport: counts + per-class ledger + diagnoses + eliminations; as_dict.
- run_practice(cases, scorer=...): injectable scorer for tests; defaults to the
candidate-graph scorer (read-only — never alters serving).
Live result mirrors serving (3 correct / 0 wrong / 47 refused of 50) because the
engine still refuses rather than guesses — attempts/eliminations go live in
Phase 3. But the diagnosis is already actionable: 35 skill_gap / 12 knowledge_gap
/ 0 genuine_ambiguity — 74% of refusals are skill gaps (Phase 3's search target),
quantifying the skill-vs-knowledge split.
Invariants: #1 seal (serving still 3/47/0; no generate/chat import of the lane),
#3 determinism (report byte-identical across runs). Elimination + wrong-tolerance
paths unit-tested via injected scorer (no live wrongs yet).
Verified: Phase 1+2 53/53, serving train_sample tests 4/4 (seal), smoke 67/67,
ruff clean.
ADR-0175 Phase 1 — standalone, deterministic, zero serving change. Nothing in
the serving/eval path imports it.
core/reliability_gate/:
- floor.py: conservative_floor(s,k) — pinned one-sided Wilson lower bound over
COMMITTED trials. z=2.576, N_MIN=10; range [0,1) (never exactly 1.0); float64
rounded half-to-even to 1e-9 for cross-backend replay. Perfect record reduces
to k/(k+z²) (earned by volume).
- ledger.py: ClassTally — immutable per-class counts; reliability = commitment
precision (refusals excluded so coverage never penalizes reliability);
t2_precision over the anchor set; coverage tracked separately.
- ceilings.py: Action{PRACTICE,PROPOSE,SERVE} + Ceilings — human-set θ
(practice=0, propose=.85, serve=.99). Frozen; with_override returns a NEW
instance (no in-place self-authorization).
- gate.py: license_for() — deterministic gate, measured/required≥1 (≡ measured≥
required; required=0 ⟹ always). Pure; never mutates/emits ceilings.
34 tests, each ADR invariant exercised by a test that fails under its violation:
#3 determinism/replay (idempotent, pre-rounded, deterministic decisions),
#4 no self-authorization (frozen ceilings; gate never emits/mutates them),
#1 proxy (zero serving coupling). Plus the §4a worked examples (38 clean
commitments clear propose; one wrong in 40 drops below; serve needs ~657).
Verified: 34/34 pass; architectural invariants 40/40; smoke 67/67; ruff clean;
no serving/eval import of the package.
Lookback/cross-reference audit before any code. No hard blockers. Corrects 3
overclaims of reuse and records the audit:
- reliability ledger is NEW (calibration/ is a grid-search param tuner, not a ledger)
- 0174 eliminate/reevaluate/contemplate is reading-coupled -> 'repoint to solving'
needs generalization, not drop-in reuse
- teaching corridor evidence (MathReaderRefusalEvidence) is reader-refusal-coupled
-> solver-practice proposals need a new evidence type
Constraints recorded: wrong=0 pinned on serving lane by ~25 tests + train_sample
runner -> practice MUST be a separate lane (phasing updated). Alignments noted:
ADR-0165 (lexemes-not-grammar) fulfilled by thin-front-end/thick-solver split;
INV-07 governance = no-self-authorization pre-wired; MAX_TOTAL_BRANCHES precedent
for bounded deterministic search; seal exists.
Resolves Open Question #1. conservative_floor(s,k) = one-sided Wilson lower
bound over COMMITTED trials (k=correct+wrong; refusals excluded so coverage
never penalizes reliability). Constants: z=2.576 (single global pessimism
knob), N_min=10. Range [0,1) — never returns exactly 1.0. float64 rounded
half-to-even to 1e-9 for cross-backend replay. z (estimator skepticism) and
per-class theta (action's required reliability, human-set) are independent
dials; engine touches neither. Worked cost-to-clear table + asymmetry example
included.
Proposed ADR + session derivation doc capturing the 2026-05-28 design
discussion that took GSM8K Phase 5b from 'build another matcher' to a
self-calibrating problem solver.
Session doc (docs/sessions/SESSION-2026-05-28-...): the full journey —
problem (per-shape matchers can't compound; 79% need mul, 0% single-step),
dead-ends (brute-force spurious matches; 0021 is the only single-sentence
case and it's idiosyncratic), and the four pivots that converged on the
solution.
ADR-0175 (Proposed): the decision —
- two regimes: serving (wrong=0, unchanged) vs sealed practice
(attempt-and-eliminate; wrong is the learning signal)
- proof-carrying seal: practice never writes serving; ratification only
- deterministic attempt/refuse gate: reliability(C) / theta_required >= 1
(NOT RL; regimes collapse the reward side so only reliability is quantified)
- per-class calibration ledger of replayable COUNTS + conservative lower
bound; human-set theta ceilings raised only on evidence
- checkability ladder (gold > convergent self-verification > consistency-only),
privilege proportional to reversibility; provenance + gold tether against
correlated self-delusion
- diagnostic refusal routes skill vs knowledge vs ambiguity; three
compounding stores (vault/packs/pruning); self-proving acquisition narrows
human input without bypassing the gate
- five proof-obligation invariants (wrong=0 on serving, no spurious banking,
determinism, no self-authorization, retractability)
Supersedes the matcher-oriented ADR-0174 5b sub-phases; repoints the
0174 eliminate/reevaluate/contemplate substrate from reading to solving.
Open question: shape of conservative_floor + N_min.
Grounded in a ground-truth measurement of the 47 train_sample refusals
against GSM8K's own <<a*b=c>> calculator annotations:
- 37/47 (79%) use multiplication; 43/47 use mul-or-div; 0/47 single-step
- multiplication is the foundational general capability (breadth = the
anti-overfitting signal), but necessary-not-sufficient: no case flips
from an operation matcher alone
Solver already supports {add,subtract,transfer,multiply,divide,apply_rate,
compare_additive,compare_multiplicative}; the gap is the reader->injector->
Operation front-end (matcher extracts 0 anchors on real sentences).
Sub-phase sequence (biggest-chunk-first, measure-the-flip-gated):
- 5b.1 single-sentence multiplicative aggregate (cleanest proof, ~2-4)
- 5b.2 shallow 2-3 step composition (25/47, the real chunk)
- 5b.3 deep multi-step 4-7 (22/47) under held-hypothesis elimination
Generality guard: flipped cases must hold under ADR-0114a perturbation/OOD.
Explicitly NOT widening discrete_count injector (overfitting + wrong=0 hazard).
The recognizer/candidate-graph path is the single canonical reader.
Retires the flag-gated incremental-reader dispatch that admitted 0/50 on
train_sample and only added a dead fall-through:
- remove _try_comprehension_reader, _try_reader_for_question, _tokenize_sentence
and both dispatch blocks from generate/math_candidate_graph.py
- delete generate/comprehension/lifecycle_runtime_adapter.py (402 LOC,
used only by the question-reader dispatch)
- drop the comprehension_reader_questions config flag and the parse_and_solve
/ _score_one_candidate_graph config threading
- remove the --use-reader runner plumbing + flag-ON/OFF delta report from
the train_sample runner; refresh report.json (drops stale use_reader field
and a stale refusal-reason; verdicts unchanged at 3/47/0)
- remove the now-dead use_reader field from teaching/coverage.py
CoverageReport + the core teaching coverage CLI flag
- delete tests/test_reader_coexistence.py (flag-ON/OFF premise dissolved);
fix 3 ADR-0174 build_report calls and 2 subprocess invocations
lifecycle.py and audit.py are KEPT — they are load-bearing for the ADR-0172
math-contemplation teaching corridor (audit_problem -> teaching/math_*),
which a pre-deletion trace surfaced. The parent ADR's plan to delete
lifecycle.py was wrong; only its GSM8K scoring dispatch was inert.
Net -1,038 LOC (code + tests). Behavior-preserving:
- train_sample 3/47/0, byte-identical verdicts to pre-5a baseline
- determinism holds; smoke/packs/runtime/cognition/teaching lanes green
- contemplation corridor + lifecycle/audit tests pass
Pre-existing (NOT introduced here; reproduce on base with changes stashed):
5 out-of-curated-lane stale committed-artifact / stale-assertion failures
(test_math_evidence_e2e, test_adr_0126_runner_wiring, G3/coverage_probe
report-match, test_refusal_taxonomy_lane rebuild).
Pre-implementation investigation (lookback discipline) found the original
Phase 5 text inverted against shipped code:
- math_parser.py already out of runtime + candidate-graph scoring path
- lifecycle.py admits 0/50 (inert parallel parser, not the reader to promote)
- correct>=25 is a semantic gate structural collapse cannot meet
Decision (Invert + split): recognizer/candidate-graph path is the canonical
reader; lifecycle.py is retired. Phase 5a = structural retirement (net -LOC,
3/47/0 byte-identical, wrong=0). Phase 5b = semantic narrowness removal (the
real lift, own sub-phases, per-layer wrong=0 obligations).
ADR-0174 Phase 3b — emit N anchors for compound-clause discrete-count
sentences sharing one subject + one verb. Architectural substrate;
score on train_sample preserved at 3/47/0 (compound cases like 0027
admit past the recognizer-injection refusal but the rest of the
problem still has downstream complexity — fractions, percent — that
needs Phase 4 + solver work).
generate/comprehension/state.py:
HYPOTHESIS_CAP raised 4 → 8. Case 0040 emits 5 anchors; cap=8
gives headroom (7-item lists) without becoming permissive.
generate/recognizer_match.py:
_try_extract_compound_discrete_count_anchors() — new extractor
emitting tuple of anchors for compound sentences. Refusal-
preferring on:
- no conjunctive separator (single-anchor path)
- multiplicative/percent/fraction markers
- head verb not in whitelist
- any tail clause without grounded (count, observed_noun) pair
- exceeding HYPOTHESIS_CAP
- unaccounted digit in tail (wrong=0 hazard defense surfaced by
2026-05-28 implementation review: bogusnoun would silently fail
to produce anchor while leaving the digit unaccounted, admitting
partial state)
Wired into _match_discrete_count_statement dispatch as fallback when
single-anchor extraction fails.
tests/test_adr_0174_phase3b_compound_clause.py:
11 acceptance tests passing — pure conjunctive lists (proper-noun
+ pronoun-subject + single-actor antecedent), refusal-preferring
discipline (mixed-verb, multiplicative-tail, non-whitelisted-head,
partial-grounding all-or-nothing), HYPOTHESIS_CAP enforcement,
multi-actor pronoun defense preserved on compound, wrong=0 +
case-0050 canary.
tests/test_adr_0174_phase1_held_hypothesis_state.py:
Updated test_hypothesis_cap_is_four → test_hypothesis_cap_is_eight
with rationale for the raise.
Phase 3b implementation lookback review (per CLAUDE.md doctrine):
- Surfaced silent-partial-admission hazard in tail extraction;
fixed with digit-accounting check before commit
- Surfaced LATENT regex-path multi-actor pronoun hazard (not
introduced by Phase 3b; documented in test docstring with
cross-reference to project-adr-0174-multi-actor-pronoun-hazard
memory for follow-up)
- case 0040 ('He now has...') remains refused — 'now' adverb between
subject and verb defeats the existing canonical regex. Adverb-
stripping is separate scope (not Phase 3b).
Acceptance:
- 258/258 ADR-0174 + math_problem_graph tests pass
- Smoke 67/67, packs 141/141
- train_sample 3/47/0 preserved (wrong=0 held)
- Case 0027 'Malcolm has 240 followers on Instagram and 500 followers
on Facebook' now admits via the compound extractor — verified by
refusal moving to the next sentence (which has 'half' fraction)
All findings from the 2026-05-28 Phase 1-3a lookback review addressed
in one commit on the Phase 3a branch:
Wrong=0 hazard defense (the load-bearing fix):
- generate/math_candidate_graph.py: Phase 3a wiring now collects the
set of distinct proper-noun subjects seen in prior context. When
more than one exists, refuses with no_antecedent_ambiguous trace
event rather than guessing the most-recent (which was gender-blind
single-binding — wrong attribution in multi-actor problems).
- Refusals from the statement loop now preserve _statement_trace via
reader_trace in CandidateGraphResult (pre-existing latent issue:
Phase 2/3 trace events were dropped on early statement refusal).
- New tests assert: ambiguous case refuses with correct trace; single-
actor case still resolves normally.
Test coverage backfills (closes the 13 untested predicate-name gaps):
- TestCheckConstraintsInitialPredicateNames — 3 tests asserting the
exact predicate name on initial.value_grounds / initial.unit_grounds
/ initial.entity_grounds failure paths.
- TestCheckConstraintsOperationPredicateNames — 3 tests asserting
operation.verb_grounds / operation.value_grounds / operation.unit_grounds
failure-predicate-name parity.
- TestCheckConstraintsComposedInitialPath — 4 tests for the RAT-1
composed_initial path which was entirely untested in Phase 2
(parity manually verified during lookback review; now automated).
ADR amendment (honest doc vs impl drift):
- docs/decisions/ADR-0174-held-hypothesis-comprehension.md: appended
'Implementation Notes' section documenting:
- reevaluate signature differs from spec text (shipped is more
composable; treat as amended)
- Phase 2 wires per-candidate, not per-token (per-token is Phase 5)
- Lookback recompute is candidate-level, not token-level
- Hypothesis.constraint_state is never populated by Phase 2
- Multi-actor pronoun hazard defense rationale
- Honest LOC accounting: Phases 1-3a net +1,500 lines (Phase 5
delivers the projected net removal)
- Test coverage backfill summary
Cosmetic:
- lookback.py:297 unreachable raise — added # type: ignore[unreachable]
with comment explaining defensive future-proofing for Phase 3b.
Acceptance verified:
- 124/124 Phase 1+2+3a + reader tests pass (was 95/95 before backfills)
- Smoke 67/67, packs 141/141
- train_sample 3/47/0 preserved (wrong=0 invariant held)
- Multi-actor hazard live-tested: parse_and_solve refuses the
Alice/Bob/She case with no_antecedent_ambiguous trace event
See CLAUDE.md §Lookback Review Discipline and memory
feedback-lookback-review-discipline for the doctrine that surfaced
all of these issues at the right time.
ADR-0174 Phase 3a — substrate for held-hypothesis lookback.
Score unchanged at 3/47/0 (this PR is correctly-engineered
infrastructure; eval impact gated on ADR-0163.x recognizer expansion
documented in the follow-up brief).
Adds generate/comprehension/lookback.py:
- VALID_REFINEMENT_KINDS, VALID_UNRESOLVED_SLOTS — closed sets
contracted with reader_trace consumer
- PronounResolution refinement dataclass (pronoun + resolved_to +
evidence_source, all validated)
- Refinement Union (Phase 3b will widen with CompoundClauseExpansion)
- ReevaluateResult dataclass with admit/eliminate consistency
- reevaluate(hypothesis, refinement) operator — applies refinement,
re-runs check_constraints, returns refined Hypothesis or None.
- _rebuild_candidate_with_resolved_actor — rebuilds
CandidateOperation / CandidateInitial replacing the semantic actor
field (op.actor / initial.entity) while preserving matched_actor_token
/ matched_entity_token as the pronoun (so grounding still passes
against the held statement's source span).
Modifies generate/recognizer_match.py:
- _try_extract_discrete_count_anchor: pronoun-subject statements now
emit anchors with subject_role=<pronoun> + requires_pronoun_resolution
marker, rather than refusing at the _REFUSED_SUBJECT_TOKENS check.
The other narrowness layers (clause split, verb whitelist) still
refuse; only the pronoun layer changes.
Modifies generate/math_candidate_graph.py:
- After inject_from_match, when any parsed_anchor carries
requires_pronoun_resolution, the candidates are held as Hypothesis
objects with unresolved=('actor_pronoun',). The lookback path then
resolves via the existing _discourse_prior_subjects map and runs
PronounResolution refinements through reevaluate. Resolved
hypotheses flow into per_sentence_choices as if the regex parser
had produced them; unresolved hypotheses drop cleanly (refusal-
preferring). Emits 'lookback' JSON trace events with
outcome ∈ {admitted, eliminated, no_antecedent}.
Tests:
- tests/test_adr_0174_phase3_lookback.py — 17 acceptance tests
covering operator semantics on Operation/Initial, dataclass
invariants, closed-set constants, end-to-end wiring on synthetic
problems, and wrong=0 preservation on train_sample.
Phase 3.1 follow-up brief:
- docs/handoff/PHASE-3.1-FOLLOWUP-RECOGNIZER-EXPANSION.md documents
the empirical finding that the train_sample bottleneck is
verb-coverage (recognizer scope, ADR-0163.x) not lookback
(ADR-0174 scope). 11 verbs identified for HITL contemplation pass.
Recommends sequencing: Phase 3a now (substrate), ADR-0163.x verb
expansion next, Phase 3b after coverage matures.
Acceptance verified:
- 17/17 Phase 3a tests pass
- 95/95 existing tests pass (Phase 1 + Phase 2 + brief_11 + reader_phase2)
- Smoke 67/67, packs 141/141, lanes 8/8
- wrong=0 preserved, score unchanged 3/47/0 (intentional per brief)
Stacks on Phase 2 (PR #420). Rebases onto main after #416 + #420 land.
ADR-0174 Phase 2 — hoist _initial_admissible / roundtrip_admissible into
hypothesis-based constraint checks with structured elimination tracing.
Admission semantics are byte-equivalent to today; the change is structural.
Adds generate/comprehension/constraint_propagation.py:
- VALID_PREDICATE_NAMES: closed set of 17 sub-check names spanning
initial / composed_initial / operation admissibility predicates.
Adding new names requires an ADR amendment (structural contract with
reader_trace consumer).
- ConstraintResult dataclass: admitted bool + predicates_run trace +
elimination_reason. Validates admitted-vs-reason consistency.
- Elimination dataclass: confidence_rank + predicate + reason for one
hypothesis being eliminated. Serialisable as a reader_trace event.
- hypothesis_from_initial / hypothesis_from_operation: adapters wrapping
CandidateInitial / CandidateOperation as Phase-1 Hypothesis objects
with caller-supplied confidence_rank.
- _check_initial / _check_composed_initial / _check_operation:
decomposed sub-check implementations of the existing admissibility
predicates with first-failure short-circuit (matches current
semantics). Each sub-check populates predicates_run with (name, ok|
fail|skip) so the consumer sees exactly which predicate decided.
- check_constraints: dispatches on candidate type.
- eliminate_violating: bulk filter; returns (survivors, eliminations);
survivors are re-densified to satisfy ProblemReadingState's
open_hypotheses post_init invariant (dense-from-0 ranks);
eliminations carry the original confidence_rank for trace fidelity.
Wires into generate/math_candidate_graph.py at the recognizer
injection site (line 825+): replaces inline _initial_admissible /
roundtrip_admissible dispatch with eliminate_violating. Elimination
events become JSON entries in reader_trace with layer=
'constraint_propagation', phase=2, predicate, reason, sentence_index.
Phase 2 acceptance verified:
- 24/24 ADR-0174 Phase 2 tests pass (emission, parity with existing
predicates on 9 admit/reject cases, redensification, dataclass
invariants, integration).
- 71/71 existing reader + Phase 1 tests still pass.
- Smoke 67/67, packs 141/141, lanes 8/8.
- train_sample/v1 byte-identical across two runs with use_reader=True.
- Score preserved: correct=3 refused=47 wrong=0 — semantics identical
because the decomposed sub-checks short-circuit on the same predicates
the inline checks would have caught.
Trace-event behavior: today's injectors are conservative enough that
zero eliminations occur on train_sample/v1 (no false positives, no
mid-pipeline failures). The wiring is exercised by
test_phase2_event_shape_when_synthesized which proves the trace shape
on a synthetic CandidateInitial that fails initial.unit_grounds. When
Phase 3 begins emitting partial hypotheses from apply_word, the
elimination path will fire on real candidates and the trace will
populate.
Stacks on Phase 1 (feat/adr-0174-phase1-held-hypothesis-state, PR
#416). Merges cleanly into main after PR #416 lands.
Phase 3b (compound-clause held hypotheses) is prerequisite for Phase
4 (in-loop contemplate) to have anything to operate on. Combined
scope rather than separate briefs because:
- Phase 3b alone: 0 lift on train_sample (multi-actor defense
refuses, solver gaps prevent admission)
- Phase 3b + Phase 4: 2-4 case lift via gendered-pronoun resolution
- Phase 3b + 4 + solver multi-qty (separate ADR): 6-8 case lift
First concrete Phase 4 use case: gendered pronoun resolution via a
new en_core_names_v1 pack. Turns the multi-actor defense from
refuse-on-ambiguity into admit-via-evidence when an unambiguous
gendered name exists for one antecedent.
Architecture overview, Phase 3b extractor design, Phase 4
contemplate adapters (vault > packs > audit_history), wrong=0
hazard surfaces, sequencing (3b then 4 stacked), truth tests, and
deliberately-excluded scope (solver work, verb expansion, per-token
apply_word integration — all separate ADRs).
HYPOTHESIS_CAP raise from 4 to 8 needed for Phase 3b (case 0040
has 5 anchors). Documented in scope.
References: ADR-0174 (held-hypothesis comprehension), CLAUDE.md
§Lookback Review Discipline, memories for multi-actor pronoun
hazard and case-0050.
MathProblemGraph.__post_init__ now raises MathGraphError when two
InitialPossession entries share the same (entity, unit) key but
declare different quantity values.
Pre-fix behavior surfaced by 2026-05-28 ADR-0174 Phase 3 post-merge
diagnostic: math_solver.solve() line 207 used last-write-wins dict
assignment when consolidating initial state. Two contradictory
inputs would silently overwrite without trace:
'Sam has 5 marbles. Sam has 3 marbles. How many marbles does Sam have?'
→ returned 3.0 (wrong=0 violation: definite answer from
contradictory input)
Post-fix: same input refuses with 'no branch produced a solvable
graph' — refusal-preferring discipline as wrong=0 doctrine requires.
Identical duplicates (same value) are admitted as redundant (no
contradiction). Different units for same actor admitted. Different
actors for same unit admitted. Single-value cases (the dominant
real-world pattern) unchanged.
This is an extraction-layer hazard discovered while investigating
Phase 3b scope: Phase 3b compound-clause held hypotheses would
emit multiple CandidateInitial entries per sentence, exercising
exactly this consolidation path. Fixing the silent overwrite NOW
ensures Phase 3b admission doesn't silently produce wrong answers.
Acceptance:
- 4 new tests in TestContradictoryInitialPossessionsRefuse
- 165/165 test_math_problem_graph tests pass (was 161/161)
- Smoke 67/67, packs 141/141 unchanged
- train_sample 3/47/0 unchanged (no real case exercised the
overwrite — but the hazard was latent)
References: CLAUDE.md §Lookback Review Discipline (the doctrine
that surfaced this), CLAUDE.md §Non-Negotiable Field Invariant
(make illegal states difficult to represent).
Adds second mandatory hazard test to VE-A/B acceptance criteria:
multi-actor pronoun ambiguity must trigger no_antecedent_ambiguous
refusal per ADR-0174 Phase 3a defense.
Verb expansion widens the cases that reach Phase 3a lookback wiring;
without this test the multi-actor wrong=0 hazard could fire silently
in production. Surfaced by 2026-05-28 Phase 1-3a lookback review.
References: project-adr-0174-multi-actor-pronoun-hazard memory,
CLAUDE.md §Lookback Review Discipline.
Adds 'Lookback Review Discipline' section before 'Architectural Scan
Exclusions'. Multi-PR architectural work was accumulating latent
defects — three Phase 1-3a PRs shipped without a substrate audit
between phases, and a multi-actor pronoun wrong=0 hazard sat in
Phase 3a from the moment it was written until an explicit user-
requested review surfaced it.
Mandatory lookback triggers:
1. Before starting Phase N+1 of a multi-phase ADR
2. Before merging a stacked PR sequence into main
3. After any 3+ PR sequence on the same module/architectural surface
Review template covers: doc drift, test coverage gaps, parity gaps,
wrong=0 hazard surfaces, cross-PR consistency, honest LOC accounting.
Output is structured: solid / gaps / drift / hazards. Hazards require
fix-before-next-phase decision.
Cost: 20-40 minutes per 3-PR substrate. Skipping costs more (every PR
built on undetected hazard becomes implicated when it fires).
Operationalises the recommendation from PHASE-3.1-FOLLOWUP-RECOGNIZER-EXPANSION.md
into three independent dispatchable PRs:
- VE-A: acquisition widening (gain, earn, save, accumulate, acquire) — Opus
- VE-B: new depletion class (donate, give, lose, spend, eat) — Opus
- VE-C: refusal evidence for non-arithmetic verbs (instrumentation only) — Sonnet
Each PR includes:
- Hazard pinning: explicit case 0050 test must pass after widening
- Lift evidence: at least one train_sample case moves refused → correct
- Phase 3a substrate fires: the lifted case shows a 'lookback' trace event
- wrong=0 preserved across train_sample AND case 0050
Operator decisions needed before dispatch: which specific lemmas to
admit per class, whether to introduce the depletion class at all, and
whether to ship VE-C as evidence groundwork.
Verb classification rationale per lemma documented in the brief.
Hazard surfaces called out per lemma (delta-of-attribute for 'gained',
direction inversion for 'lost', monetary-vs-time ambiguity for 'saved').
No timelines; operator dispatches when ready.
86d4e98 (multi-word unit grounding fix) changed refusal reasons for two
cases without changing the 3/47/0 count:
- case 0019 (currency_amount): refusal moves from 'requires 3 vet
appointments which cost $400 each' to 'After the first appointment,
John paid $100 for pet insurance...' — first sentence now passes,
refusal moves to subsequent sentence
- case 0023 (Nicole/Pokemon cards): refusal moves from 'Nicole collected
400 Pokemon cards' (now passes via multi-word unit grounding) to
'Cindy collected twice as many, and Rex collected half of...'
Counts unchanged: correct=3 refused=47 wrong=0. Updates report.json to
match current behavior so subsequent eval runs are byte-deterministic
from the committed snapshot.
ADR-0174 Phase 1 — substrate only, no admission behavior change.
Adds to generate/comprehension/state.py:
- HYPOTHESIS_CAP (=4, structural assertion per ADR-0174 §Constraints)
- VALID_HYPOTHESIS_CONFIDENCE_RANKS (closed set, no probabilistic ranking)
- Hypothesis dataclass (frozen, slots) — candidate, category_assignments,
constraint_state, confidence_rank, unresolved. The 'candidate' field is
typed as object to avoid circular import on math_roundtrip /
math_candidate_graph candidate types; Phase 2 will pin canonical_bytes
contract over real candidates.
- UnknownHeld dataclass — token, position, narrowed_categories (frozenset).
Substrate for Phase 3 'hold instead of refuse' on unknown words; Phase 1
introduces only the type.
- ProblemReadingState.open_hypotheses + unknown_held fields, both default
to () (empty tuple). Defaults preserve today's single-committed behavior
exactly. Confidence-rank uniqueness + density-from-0 enforced at
__post_init__ as structural invariants.
- Canonical-bytes serializer extended to handle frozenset (sorted list).
Phase 1 acceptance verified:
- 29/29 ADR-0174 Phase 1 tests pass (construction, validation, cap
enforcement, canonical-bytes determinism, frozenset stability).
- 42/42 existing reader tests pass (test_brief_11_audit +
test_reader_phase2) — default-empty fields preserve byte-identity.
- Smoke 67/67, packs 141/141.
- train_sample/v1 byte-identical across two runs with use_reader=True.
- wrong=0 invariant held: 3/47/0 unchanged.
No apply_word body changes. The 'thread the hypothesis set' requirement
at Phase 1 is satisfied by field defaults that propagate through every
ProblemReadingState construction site in lifecycle.py without code edits.
Phase 2 (continuous constraint propagation) and Phase 3 (lookback
re-evaluation) will populate these fields with real hypothesis data and
wire the EMIT / ELIMINATE / HOLD operators.
Extends ADR-0164's incremental comprehension reader from single-committed
state to held-hypothesis state, adding lookback re-evaluation and in-loop
contemplation. Diagnoses why the ADR-0164 reader is wired but inert
(all-or-nothing refusal at first unknown token / unexpected category).
Architecture: apply_word produces ProblemReadingState.open_hypotheses
(small ranked set, HYPOTHESIS_CAP=4 initial). Three operators per token:
EMIT (extend compatible hypotheses), ELIMINATE (constraint violations
remove hypotheses immediately), HOLD (uncommitted hypotheses survive at
lower confidence). At finalize(), |survivors|=0 refuses, |survivors|=1
admits, |survivors|>=2 invokes in-loop contemplate() over vault + packs
+ audit history. Ambiguity contemplation cannot resolve refuses cleanly,
preserving wrong=0.
Collapses three parallel parsing systems into one held-hypothesis
reader: removes regex parser runtime path (math_parser.py),
per-category injector dispatch table (recognizer_anchor_inject._INJECTORS),
and duplicate per-sentence-choices scaffolding. Net ~1,900 lines
removed; reader grows by ~600 for hypothesis state + reevaluate +
contemplate.
Preserves ADR-0164 lexicon and category set, ADR-0165 regex scope,
ADR-0150/0152/0155/0161 HITL corridor, binding graph + solver
substrate, capability-axis lanes, replay-equivalence gate. Trust
boundary for in-loop contemplation: read-only over vault/packs/audit;
ratification still rides offline HITL.
Status: Proposed. Six phases (no timelines) gated on wrong=0 and
capability-axis 100% at each transition. Five open questions resolved
before Phase 1 PR.
_unit_grounds() previously refused multi-word units like 'Pokemon cards'
even when both component words appeared as tokens in the source span.
The function checked unit_token against the haystack as a single key,
but the tokenizer splits source into per-word tokens — 'Pokemon cards'
was never going to match.
Fix is conjunctive by design: every component word must appear in the
haystack. A missing component refuses, preserving wrong=0.
Truth-test: case 0023 (Nicole/Pokemon cards) previously refused with
'recognizer matched but produced no injection' on its first sentence.
After this fix, sentence 1 passes injection cleanly; the case now
refuses on sentence 2 (Cindy/Rex compositional clause) — a more
honest refusal reason that reflects the actual remaining gap.
Score unchanged at 3/47/0 (no overall lift; correctness win).
smoke 67/67, packs 141/141, lanes 8/8 all green.
Adds 'Architectural Scan Exclusions' section before the PR Checklist.
Agent worktrees under .claude/worktrees/ contain full source copies that
trip every structural invariant scan (vault writers, normalize_to_versor
callsites, recall sites). Without .claude/ in the exclusion set, smoke
failures look exactly like real architectural violations but point at
non-live files — silent killers that mask actual regressions.
Documents the two maintained exclusion sets in
test_architectural_invariants.py and requires new scans to add .claude
before the first commit. Never rely on worktree pruning.
Both INV-02/INV-21/INV-24 scan functions walked into .claude/worktrees/
and found vault recall/write callsites in the stale
step-3-submission-invariants checkout, producing 3 false failures.
Fix: add '.claude' to the os.walk exclusion set (INV-02) and to
EXCLUDED_DIRS (INV-21/INV-24). Defensive against any future worktree
that agents create under .claude/worktrees/.
Also pruned 58 stale worktree git-dir entries via git worktree prune
and removed the step-3-submission-invariants worktree directory.
Smoke suite: 67/67 passed.
C1: delete generate/math_versor_arithmetic.py and its 3 tests (ADR-0139
add-only arithmetic spike; no runtime consumers, no pipeline wiring,
follow-on lift paused per module docstring).
C3: gitignore engine_state runtime artifacts (manifest.json,
recognizers.jsonl, discovery_candidates.jsonl). Module code
(engine_state/__init__.py) remains tracked; generated checkpoint
files should not be.
C5: document reader zero-delta root cause in train_sample/v1/README.md.
Both Phase 2 (whole-problem) and Phase 1 (question-only) reader paths are
called but inert because all 47 refusals are statement-level NO_INJECTOR
gaps, not question-sentence gaps. Reader unblocks when injector coverage
expands (C2 work). report.json use_reader flag corrected to reflect last run.
C6: add deprecation header to generate/math_parser.py pointing at
generate.math_candidate_graph.parse_and_solve as the live path.
C2/C4 briefs: docs/handoff/CLEANUP-C2-run-lane-migration.md and
docs/handoff/CLEANUP-C4-compositions-compile.md added as operator
dispatch docs for the medium-scope wiring tasks.
The repo is public. The thesis is *decoding, not generating* with
wrong=0 as the load-bearing invariant. The demo any visitor can run
to see the loop turn end-to-end on the canonical pack:
git clone https://github.com/AssetOverflow/core
cd core && uv pip install -e .
core demo flywheel
Four falsifiable scenes:
1. RATIFY — apply_composition_claim writes source JSONL; RAT-1
auto-compile regenerates compositions.jsonl + bumps
manifest.composition_checksum
2. LOAD — composition_registry picks up the new entry on the
next runtime turn
3. SOLVE — "Lilibeth fills 6 baskets where each basket holds
50 strawberries. How many strawberries does Lilibeth
have?" admits via matcher → injector → admission →
candidate-graph and produces answer=300
4. HAZARD — case 0050 (wrong=0 canary) remains refused; no SAFE
composition category can convert it
All four scenes byte-deterministic. The canonical pack is read-only
throughout; the demo mutates only a synthetic test pack in a
tempfile.TemporaryDirectory. One-time recognizer seed is idempotent
(same content_digest each run → no duplicate proposal log entries).
Exit code 0 iff all scenes pass; --json for CI integration.
Also adds:
- README "Watch the flywheel turn — one command" section pointing
to the demo + the coverage CLI (per-shape histogram + hazard pin)
- ProposalLog entry for the multiplicative_aggregate recognizer
with extract_values=True (one-time operator seed)
Files:
- evals/flywheel_demo/run_tour.py (new) — the four-scene tour
- evals/flywheel_demo/__init__.py (new)
- core/cli.py — `flywheel` added to `core demo` choices + dispatch
- README.md — new "Quick Start" subsection
- teaching/proposals/proposals.jsonl — seeded recognizer
Three review fixes:
1. Security: validate lane/split/version against ^[a-z0-9_]+$ before
building the runner module name. The runner_args list is passed to
subprocess.run without shell=True (no shell injection possible),
but defense-in-depth blocks arbitrary token characters from
reaching Python's -m module loader. Bad input now errors at the
CLI boundary with a clear message.
2. Bug-risk: _classify_refusal docstring referenced a
no_admissible_candidate bucket that the implementation never
emitted. Aligned docstring with actual buckets
(no_admissible_question / no_admissible_statement). Also made all
matching consistently case-insensitive (was mixed — some checks
used raw reason, one used .lower()).
3. Bug-risk: fetch_committed_baseline wrote to
.git/coverage_baseline_tmp.json. Replaced with tempfile.mkstemp in
the system temp dir — avoids (a) failures in non-git worktrees
where .git is a file pointer, (b) concurrent-access collisions
between simultaneous operators.
Tests (+3 new):
- test_classify_refusal_is_case_insensitive
- test_classify_docstring_matches_implementation_buckets
- test_fetch_committed_baseline_uses_system_temp
All 16 coverage tests green. Verified the validation:
core teaching coverage --lane 'evil; rm -rf /'
→ ERROR: lane='evil; rm -rf /' must match ^[a-z0-9_]+$
