Resolves Open Question #1. conservative_floor(s,k) = one-sided Wilson lower
bound over COMMITTED trials (k=correct+wrong; refusals excluded so coverage
never penalizes reliability). Constants: z=2.576 (single global pessimism
knob), N_min=10. Range [0,1) — never returns exactly 1.0. float64 rounded
half-to-even to 1e-9 for cross-backend replay. z (estimator skepticism) and
per-class theta (action's required reliability, human-set) are independent
dials; engine touches neither. Worked cost-to-clear table + asymmetry example
included.
Proposed ADR + session derivation doc capturing the 2026-05-28 design
discussion that took GSM8K Phase 5b from 'build another matcher' to a
self-calibrating problem solver.
Session doc (docs/sessions/SESSION-2026-05-28-...): the full journey —
problem (per-shape matchers can't compound; 79% need mul, 0% single-step),
dead-ends (brute-force spurious matches; 0021 is the only single-sentence
case and it's idiosyncratic), and the four pivots that converged on the
solution.
ADR-0175 (Proposed): the decision —
- two regimes: serving (wrong=0, unchanged) vs sealed practice
(attempt-and-eliminate; wrong is the learning signal)
- proof-carrying seal: practice never writes serving; ratification only
- deterministic attempt/refuse gate: reliability(C) / theta_required >= 1
(NOT RL; regimes collapse the reward side so only reliability is quantified)
- per-class calibration ledger of replayable COUNTS + conservative lower
bound; human-set theta ceilings raised only on evidence
- checkability ladder (gold > convergent self-verification > consistency-only),
privilege proportional to reversibility; provenance + gold tether against
correlated self-delusion
- diagnostic refusal routes skill vs knowledge vs ambiguity; three
compounding stores (vault/packs/pruning); self-proving acquisition narrows
human input without bypassing the gate
- five proof-obligation invariants (wrong=0 on serving, no spurious banking,
determinism, no self-authorization, retractability)
Supersedes the matcher-oriented ADR-0174 5b sub-phases; repoints the
0174 eliminate/reevaluate/contemplate substrate from reading to solving.
Open question: shape of conservative_floor + N_min.
ADR-0174 Phase 3b — emit N anchors for compound-clause discrete-count
sentences sharing one subject + one verb. Architectural substrate;
score on train_sample preserved at 3/47/0 (compound cases like 0027
admit past the recognizer-injection refusal but the rest of the
problem still has downstream complexity — fractions, percent — that
needs Phase 4 + solver work).
generate/comprehension/state.py:
HYPOTHESIS_CAP raised 4 → 8. Case 0040 emits 5 anchors; cap=8
gives headroom (7-item lists) without becoming permissive.
generate/recognizer_match.py:
_try_extract_compound_discrete_count_anchors() — new extractor
emitting tuple of anchors for compound sentences. Refusal-
preferring on:
- no conjunctive separator (single-anchor path)
- multiplicative/percent/fraction markers
- head verb not in whitelist
- any tail clause without grounded (count, observed_noun) pair
- exceeding HYPOTHESIS_CAP
- unaccounted digit in tail (wrong=0 hazard defense surfaced by
2026-05-28 implementation review: bogusnoun would silently fail
to produce anchor while leaving the digit unaccounted, admitting
partial state)
Wired into _match_discrete_count_statement dispatch as fallback when
single-anchor extraction fails.
tests/test_adr_0174_phase3b_compound_clause.py:
11 acceptance tests passing — pure conjunctive lists (proper-noun
+ pronoun-subject + single-actor antecedent), refusal-preferring
discipline (mixed-verb, multiplicative-tail, non-whitelisted-head,
partial-grounding all-or-nothing), HYPOTHESIS_CAP enforcement,
multi-actor pronoun defense preserved on compound, wrong=0 +
case-0050 canary.
tests/test_adr_0174_phase1_held_hypothesis_state.py:
Updated test_hypothesis_cap_is_four → test_hypothesis_cap_is_eight
with rationale for the raise.
Phase 3b implementation lookback review (per CLAUDE.md doctrine):
- Surfaced silent-partial-admission hazard in tail extraction;
fixed with digit-accounting check before commit
- Surfaced LATENT regex-path multi-actor pronoun hazard (not
introduced by Phase 3b; documented in test docstring with
cross-reference to project-adr-0174-multi-actor-pronoun-hazard
memory for follow-up)
- case 0040 ('He now has...') remains refused — 'now' adverb between
subject and verb defeats the existing canonical regex. Adverb-
stripping is separate scope (not Phase 3b).
Acceptance:
- 258/258 ADR-0174 + math_problem_graph tests pass
- Smoke 67/67, packs 141/141
- train_sample 3/47/0 preserved (wrong=0 held)
- Case 0027 'Malcolm has 240 followers on Instagram and 500 followers
on Facebook' now admits via the compound extractor — verified by
refusal moving to the next sentence (which has 'half' fraction)
All findings from the 2026-05-28 Phase 1-3a lookback review addressed
in one commit on the Phase 3a branch:
Wrong=0 hazard defense (the load-bearing fix):
- generate/math_candidate_graph.py: Phase 3a wiring now collects the
set of distinct proper-noun subjects seen in prior context. When
more than one exists, refuses with no_antecedent_ambiguous trace
event rather than guessing the most-recent (which was gender-blind
single-binding — wrong attribution in multi-actor problems).
- Refusals from the statement loop now preserve _statement_trace via
reader_trace in CandidateGraphResult (pre-existing latent issue:
Phase 2/3 trace events were dropped on early statement refusal).
- New tests assert: ambiguous case refuses with correct trace; single-
actor case still resolves normally.
Test coverage backfills (closes the 13 untested predicate-name gaps):
- TestCheckConstraintsInitialPredicateNames — 3 tests asserting the
exact predicate name on initial.value_grounds / initial.unit_grounds
/ initial.entity_grounds failure paths.
- TestCheckConstraintsOperationPredicateNames — 3 tests asserting
operation.verb_grounds / operation.value_grounds / operation.unit_grounds
failure-predicate-name parity.
- TestCheckConstraintsComposedInitialPath — 4 tests for the RAT-1
composed_initial path which was entirely untested in Phase 2
(parity manually verified during lookback review; now automated).
ADR amendment (honest doc vs impl drift):
- docs/decisions/ADR-0174-held-hypothesis-comprehension.md: appended
'Implementation Notes' section documenting:
- reevaluate signature differs from spec text (shipped is more
composable; treat as amended)
- Phase 2 wires per-candidate, not per-token (per-token is Phase 5)
- Lookback recompute is candidate-level, not token-level
- Hypothesis.constraint_state is never populated by Phase 2
- Multi-actor pronoun hazard defense rationale
- Honest LOC accounting: Phases 1-3a net +1,500 lines (Phase 5
delivers the projected net removal)
- Test coverage backfill summary
Cosmetic:
- lookback.py:297 unreachable raise — added # type: ignore[unreachable]
with comment explaining defensive future-proofing for Phase 3b.
Acceptance verified:
- 124/124 Phase 1+2+3a + reader tests pass (was 95/95 before backfills)
- Smoke 67/67, packs 141/141
- train_sample 3/47/0 preserved (wrong=0 invariant held)
- Multi-actor hazard live-tested: parse_and_solve refuses the
Alice/Bob/She case with no_antecedent_ambiguous trace event
See CLAUDE.md §Lookback Review Discipline and memory
feedback-lookback-review-discipline for the doctrine that surfaced
all of these issues at the right time.
ADR-0174 Phase 3a — substrate for held-hypothesis lookback.
Score unchanged at 3/47/0 (this PR is correctly-engineered
infrastructure; eval impact gated on ADR-0163.x recognizer expansion
documented in the follow-up brief).
Adds generate/comprehension/lookback.py:
- VALID_REFINEMENT_KINDS, VALID_UNRESOLVED_SLOTS — closed sets
contracted with reader_trace consumer
- PronounResolution refinement dataclass (pronoun + resolved_to +
evidence_source, all validated)
- Refinement Union (Phase 3b will widen with CompoundClauseExpansion)
- ReevaluateResult dataclass with admit/eliminate consistency
- reevaluate(hypothesis, refinement) operator — applies refinement,
re-runs check_constraints, returns refined Hypothesis or None.
- _rebuild_candidate_with_resolved_actor — rebuilds
CandidateOperation / CandidateInitial replacing the semantic actor
field (op.actor / initial.entity) while preserving matched_actor_token
/ matched_entity_token as the pronoun (so grounding still passes
against the held statement's source span).
Modifies generate/recognizer_match.py:
- _try_extract_discrete_count_anchor: pronoun-subject statements now
emit anchors with subject_role=<pronoun> + requires_pronoun_resolution
marker, rather than refusing at the _REFUSED_SUBJECT_TOKENS check.
The other narrowness layers (clause split, verb whitelist) still
refuse; only the pronoun layer changes.
Modifies generate/math_candidate_graph.py:
- After inject_from_match, when any parsed_anchor carries
requires_pronoun_resolution, the candidates are held as Hypothesis
objects with unresolved=('actor_pronoun',). The lookback path then
resolves via the existing _discourse_prior_subjects map and runs
PronounResolution refinements through reevaluate. Resolved
hypotheses flow into per_sentence_choices as if the regex parser
had produced them; unresolved hypotheses drop cleanly (refusal-
preferring). Emits 'lookback' JSON trace events with
outcome ∈ {admitted, eliminated, no_antecedent}.
Tests:
- tests/test_adr_0174_phase3_lookback.py — 17 acceptance tests
covering operator semantics on Operation/Initial, dataclass
invariants, closed-set constants, end-to-end wiring on synthetic
problems, and wrong=0 preservation on train_sample.
Phase 3.1 follow-up brief:
- docs/handoff/PHASE-3.1-FOLLOWUP-RECOGNIZER-EXPANSION.md documents
the empirical finding that the train_sample bottleneck is
verb-coverage (recognizer scope, ADR-0163.x) not lookback
(ADR-0174 scope). 11 verbs identified for HITL contemplation pass.
Recommends sequencing: Phase 3a now (substrate), ADR-0163.x verb
expansion next, Phase 3b after coverage matures.
Acceptance verified:
- 17/17 Phase 3a tests pass
- 95/95 existing tests pass (Phase 1 + Phase 2 + brief_11 + reader_phase2)
- Smoke 67/67, packs 141/141, lanes 8/8
- wrong=0 preserved, score unchanged 3/47/0 (intentional per brief)
Stacks on Phase 2 (PR #420). Rebases onto main after #416 + #420 land.
ADR-0174 Phase 2 — hoist _initial_admissible / roundtrip_admissible into
hypothesis-based constraint checks with structured elimination tracing.
Admission semantics are byte-equivalent to today; the change is structural.
Adds generate/comprehension/constraint_propagation.py:
- VALID_PREDICATE_NAMES: closed set of 17 sub-check names spanning
initial / composed_initial / operation admissibility predicates.
Adding new names requires an ADR amendment (structural contract with
reader_trace consumer).
- ConstraintResult dataclass: admitted bool + predicates_run trace +
elimination_reason. Validates admitted-vs-reason consistency.
- Elimination dataclass: confidence_rank + predicate + reason for one
hypothesis being eliminated. Serialisable as a reader_trace event.
- hypothesis_from_initial / hypothesis_from_operation: adapters wrapping
CandidateInitial / CandidateOperation as Phase-1 Hypothesis objects
with caller-supplied confidence_rank.
- _check_initial / _check_composed_initial / _check_operation:
decomposed sub-check implementations of the existing admissibility
predicates with first-failure short-circuit (matches current
semantics). Each sub-check populates predicates_run with (name, ok|
fail|skip) so the consumer sees exactly which predicate decided.
- check_constraints: dispatches on candidate type.
- eliminate_violating: bulk filter; returns (survivors, eliminations);
survivors are re-densified to satisfy ProblemReadingState's
open_hypotheses post_init invariant (dense-from-0 ranks);
eliminations carry the original confidence_rank for trace fidelity.
Wires into generate/math_candidate_graph.py at the recognizer
injection site (line 825+): replaces inline _initial_admissible /
roundtrip_admissible dispatch with eliminate_violating. Elimination
events become JSON entries in reader_trace with layer=
'constraint_propagation', phase=2, predicate, reason, sentence_index.
Phase 2 acceptance verified:
- 24/24 ADR-0174 Phase 2 tests pass (emission, parity with existing
predicates on 9 admit/reject cases, redensification, dataclass
invariants, integration).
- 71/71 existing reader + Phase 1 tests still pass.
- Smoke 67/67, packs 141/141, lanes 8/8.
- train_sample/v1 byte-identical across two runs with use_reader=True.
- Score preserved: correct=3 refused=47 wrong=0 — semantics identical
because the decomposed sub-checks short-circuit on the same predicates
the inline checks would have caught.
Trace-event behavior: today's injectors are conservative enough that
zero eliminations occur on train_sample/v1 (no false positives, no
mid-pipeline failures). The wiring is exercised by
test_phase2_event_shape_when_synthesized which proves the trace shape
on a synthetic CandidateInitial that fails initial.unit_grounds. When
Phase 3 begins emitting partial hypotheses from apply_word, the
elimination path will fire on real candidates and the trace will
populate.
Stacks on Phase 1 (feat/adr-0174-phase1-held-hypothesis-state, PR
#416). Merges cleanly into main after PR #416 lands.
Phase 3b (compound-clause held hypotheses) is prerequisite for Phase
4 (in-loop contemplate) to have anything to operate on. Combined
scope rather than separate briefs because:
- Phase 3b alone: 0 lift on train_sample (multi-actor defense
refuses, solver gaps prevent admission)
- Phase 3b + Phase 4: 2-4 case lift via gendered-pronoun resolution
- Phase 3b + 4 + solver multi-qty (separate ADR): 6-8 case lift
First concrete Phase 4 use case: gendered pronoun resolution via a
new en_core_names_v1 pack. Turns the multi-actor defense from
refuse-on-ambiguity into admit-via-evidence when an unambiguous
gendered name exists for one antecedent.
Architecture overview, Phase 3b extractor design, Phase 4
contemplate adapters (vault > packs > audit_history), wrong=0
hazard surfaces, sequencing (3b then 4 stacked), truth tests, and
deliberately-excluded scope (solver work, verb expansion, per-token
apply_word integration — all separate ADRs).
HYPOTHESIS_CAP raise from 4 to 8 needed for Phase 3b (case 0040
has 5 anchors). Documented in scope.
References: ADR-0174 (held-hypothesis comprehension), CLAUDE.md
§Lookback Review Discipline, memories for multi-actor pronoun
hazard and case-0050.
MathProblemGraph.__post_init__ now raises MathGraphError when two
InitialPossession entries share the same (entity, unit) key but
declare different quantity values.
Pre-fix behavior surfaced by 2026-05-28 ADR-0174 Phase 3 post-merge
diagnostic: math_solver.solve() line 207 used last-write-wins dict
assignment when consolidating initial state. Two contradictory
inputs would silently overwrite without trace:
'Sam has 5 marbles. Sam has 3 marbles. How many marbles does Sam have?'
→ returned 3.0 (wrong=0 violation: definite answer from
contradictory input)
Post-fix: same input refuses with 'no branch produced a solvable
graph' — refusal-preferring discipline as wrong=0 doctrine requires.
Identical duplicates (same value) are admitted as redundant (no
contradiction). Different units for same actor admitted. Different
actors for same unit admitted. Single-value cases (the dominant
real-world pattern) unchanged.
This is an extraction-layer hazard discovered while investigating
Phase 3b scope: Phase 3b compound-clause held hypotheses would
emit multiple CandidateInitial entries per sentence, exercising
exactly this consolidation path. Fixing the silent overwrite NOW
ensures Phase 3b admission doesn't silently produce wrong answers.
Acceptance:
- 4 new tests in TestContradictoryInitialPossessionsRefuse
- 165/165 test_math_problem_graph tests pass (was 161/161)
- Smoke 67/67, packs 141/141 unchanged
- train_sample 3/47/0 unchanged (no real case exercised the
overwrite — but the hazard was latent)
References: CLAUDE.md §Lookback Review Discipline (the doctrine
that surfaced this), CLAUDE.md §Non-Negotiable Field Invariant
(make illegal states difficult to represent).
Adds second mandatory hazard test to VE-A/B acceptance criteria:
multi-actor pronoun ambiguity must trigger no_antecedent_ambiguous
refusal per ADR-0174 Phase 3a defense.
Verb expansion widens the cases that reach Phase 3a lookback wiring;
without this test the multi-actor wrong=0 hazard could fire silently
in production. Surfaced by 2026-05-28 Phase 1-3a lookback review.
References: project-adr-0174-multi-actor-pronoun-hazard memory,
CLAUDE.md §Lookback Review Discipline.
Adds 'Lookback Review Discipline' section before 'Architectural Scan
Exclusions'. Multi-PR architectural work was accumulating latent
defects — three Phase 1-3a PRs shipped without a substrate audit
between phases, and a multi-actor pronoun wrong=0 hazard sat in
Phase 3a from the moment it was written until an explicit user-
requested review surfaced it.
Mandatory lookback triggers:
1. Before starting Phase N+1 of a multi-phase ADR
2. Before merging a stacked PR sequence into main
3. After any 3+ PR sequence on the same module/architectural surface
Review template covers: doc drift, test coverage gaps, parity gaps,
wrong=0 hazard surfaces, cross-PR consistency, honest LOC accounting.
Output is structured: solid / gaps / drift / hazards. Hazards require
fix-before-next-phase decision.
Cost: 20-40 minutes per 3-PR substrate. Skipping costs more (every PR
built on undetected hazard becomes implicated when it fires).
Operationalises the recommendation from PHASE-3.1-FOLLOWUP-RECOGNIZER-EXPANSION.md
into three independent dispatchable PRs:
- VE-A: acquisition widening (gain, earn, save, accumulate, acquire) — Opus
- VE-B: new depletion class (donate, give, lose, spend, eat) — Opus
- VE-C: refusal evidence for non-arithmetic verbs (instrumentation only) — Sonnet
Each PR includes:
- Hazard pinning: explicit case 0050 test must pass after widening
- Lift evidence: at least one train_sample case moves refused → correct
- Phase 3a substrate fires: the lifted case shows a 'lookback' trace event
- wrong=0 preserved across train_sample AND case 0050
Operator decisions needed before dispatch: which specific lemmas to
admit per class, whether to introduce the depletion class at all, and
whether to ship VE-C as evidence groundwork.
Verb classification rationale per lemma documented in the brief.
Hazard surfaces called out per lemma (delta-of-attribute for 'gained',
direction inversion for 'lost', monetary-vs-time ambiguity for 'saved').
No timelines; operator dispatches when ready.
86d4e98 (multi-word unit grounding fix) changed refusal reasons for two
cases without changing the 3/47/0 count:
- case 0019 (currency_amount): refusal moves from 'requires 3 vet
appointments which cost $400 each' to 'After the first appointment,
John paid $100 for pet insurance...' — first sentence now passes,
refusal moves to subsequent sentence
- case 0023 (Nicole/Pokemon cards): refusal moves from 'Nicole collected
400 Pokemon cards' (now passes via multi-word unit grounding) to
'Cindy collected twice as many, and Rex collected half of...'
Counts unchanged: correct=3 refused=47 wrong=0. Updates report.json to
match current behavior so subsequent eval runs are byte-deterministic
from the committed snapshot.
ADR-0174 Phase 1 — substrate only, no admission behavior change.
Adds to generate/comprehension/state.py:
- HYPOTHESIS_CAP (=4, structural assertion per ADR-0174 §Constraints)
- VALID_HYPOTHESIS_CONFIDENCE_RANKS (closed set, no probabilistic ranking)
- Hypothesis dataclass (frozen, slots) — candidate, category_assignments,
constraint_state, confidence_rank, unresolved. The 'candidate' field is
typed as object to avoid circular import on math_roundtrip /
math_candidate_graph candidate types; Phase 2 will pin canonical_bytes
contract over real candidates.
- UnknownHeld dataclass — token, position, narrowed_categories (frozenset).
Substrate for Phase 3 'hold instead of refuse' on unknown words; Phase 1
introduces only the type.
- ProblemReadingState.open_hypotheses + unknown_held fields, both default
to () (empty tuple). Defaults preserve today's single-committed behavior
exactly. Confidence-rank uniqueness + density-from-0 enforced at
__post_init__ as structural invariants.
- Canonical-bytes serializer extended to handle frozenset (sorted list).
Phase 1 acceptance verified:
- 29/29 ADR-0174 Phase 1 tests pass (construction, validation, cap
enforcement, canonical-bytes determinism, frozenset stability).
- 42/42 existing reader tests pass (test_brief_11_audit +
test_reader_phase2) — default-empty fields preserve byte-identity.
- Smoke 67/67, packs 141/141.
- train_sample/v1 byte-identical across two runs with use_reader=True.
- wrong=0 invariant held: 3/47/0 unchanged.
No apply_word body changes. The 'thread the hypothesis set' requirement
at Phase 1 is satisfied by field defaults that propagate through every
ProblemReadingState construction site in lifecycle.py without code edits.
Phase 2 (continuous constraint propagation) and Phase 3 (lookback
re-evaluation) will populate these fields with real hypothesis data and
wire the EMIT / ELIMINATE / HOLD operators.
Extends ADR-0164's incremental comprehension reader from single-committed
state to held-hypothesis state, adding lookback re-evaluation and in-loop
contemplation. Diagnoses why the ADR-0164 reader is wired but inert
(all-or-nothing refusal at first unknown token / unexpected category).
Architecture: apply_word produces ProblemReadingState.open_hypotheses
(small ranked set, HYPOTHESIS_CAP=4 initial). Three operators per token:
EMIT (extend compatible hypotheses), ELIMINATE (constraint violations
remove hypotheses immediately), HOLD (uncommitted hypotheses survive at
lower confidence). At finalize(), |survivors|=0 refuses, |survivors|=1
admits, |survivors|>=2 invokes in-loop contemplate() over vault + packs
+ audit history. Ambiguity contemplation cannot resolve refuses cleanly,
preserving wrong=0.
Collapses three parallel parsing systems into one held-hypothesis
reader: removes regex parser runtime path (math_parser.py),
per-category injector dispatch table (recognizer_anchor_inject._INJECTORS),
and duplicate per-sentence-choices scaffolding. Net ~1,900 lines
removed; reader grows by ~600 for hypothesis state + reevaluate +
contemplate.
Preserves ADR-0164 lexicon and category set, ADR-0165 regex scope,
ADR-0150/0152/0155/0161 HITL corridor, binding graph + solver
substrate, capability-axis lanes, replay-equivalence gate. Trust
boundary for in-loop contemplation: read-only over vault/packs/audit;
ratification still rides offline HITL.
Status: Proposed. Six phases (no timelines) gated on wrong=0 and
capability-axis 100% at each transition. Five open questions resolved
before Phase 1 PR.
_unit_grounds() previously refused multi-word units like 'Pokemon cards'
even when both component words appeared as tokens in the source span.
The function checked unit_token against the haystack as a single key,
but the tokenizer splits source into per-word tokens — 'Pokemon cards'
was never going to match.
Fix is conjunctive by design: every component word must appear in the
haystack. A missing component refuses, preserving wrong=0.
Truth-test: case 0023 (Nicole/Pokemon cards) previously refused with
'recognizer matched but produced no injection' on its first sentence.
After this fix, sentence 1 passes injection cleanly; the case now
refuses on sentence 2 (Cindy/Rex compositional clause) — a more
honest refusal reason that reflects the actual remaining gap.
Score unchanged at 3/47/0 (no overall lift; correctness win).
smoke 67/67, packs 141/141, lanes 8/8 all green.
Adds 'Architectural Scan Exclusions' section before the PR Checklist.
Agent worktrees under .claude/worktrees/ contain full source copies that
trip every structural invariant scan (vault writers, normalize_to_versor
callsites, recall sites). Without .claude/ in the exclusion set, smoke
failures look exactly like real architectural violations but point at
non-live files — silent killers that mask actual regressions.
Documents the two maintained exclusion sets in
test_architectural_invariants.py and requires new scans to add .claude
before the first commit. Never rely on worktree pruning.
Both INV-02/INV-21/INV-24 scan functions walked into .claude/worktrees/
and found vault recall/write callsites in the stale
step-3-submission-invariants checkout, producing 3 false failures.
Fix: add '.claude' to the os.walk exclusion set (INV-02) and to
EXCLUDED_DIRS (INV-21/INV-24). Defensive against any future worktree
that agents create under .claude/worktrees/.
Also pruned 58 stale worktree git-dir entries via git worktree prune
and removed the step-3-submission-invariants worktree directory.
Smoke suite: 67/67 passed.
C1: delete generate/math_versor_arithmetic.py and its 3 tests (ADR-0139
add-only arithmetic spike; no runtime consumers, no pipeline wiring,
follow-on lift paused per module docstring).
C3: gitignore engine_state runtime artifacts (manifest.json,
recognizers.jsonl, discovery_candidates.jsonl). Module code
(engine_state/__init__.py) remains tracked; generated checkpoint
files should not be.
C5: document reader zero-delta root cause in train_sample/v1/README.md.
Both Phase 2 (whole-problem) and Phase 1 (question-only) reader paths are
called but inert because all 47 refusals are statement-level NO_INJECTOR
gaps, not question-sentence gaps. Reader unblocks when injector coverage
expands (C2 work). report.json use_reader flag corrected to reflect last run.
C6: add deprecation header to generate/math_parser.py pointing at
generate.math_candidate_graph.parse_and_solve as the live path.
C2/C4 briefs: docs/handoff/CLEANUP-C2-run-lane-migration.md and
docs/handoff/CLEANUP-C4-compositions-compile.md added as operator
dispatch docs for the medium-scope wiring tasks.
The repo is public. The thesis is *decoding, not generating* with
wrong=0 as the load-bearing invariant. The demo any visitor can run
to see the loop turn end-to-end on the canonical pack:
git clone https://github.com/AssetOverflow/core
cd core && uv pip install -e .
core demo flywheel
Four falsifiable scenes:
1. RATIFY — apply_composition_claim writes source JSONL; RAT-1
auto-compile regenerates compositions.jsonl + bumps
manifest.composition_checksum
2. LOAD — composition_registry picks up the new entry on the
next runtime turn
3. SOLVE — "Lilibeth fills 6 baskets where each basket holds
50 strawberries. How many strawberries does Lilibeth
have?" admits via matcher → injector → admission →
candidate-graph and produces answer=300
4. HAZARD — case 0050 (wrong=0 canary) remains refused; no SAFE
composition category can convert it
All four scenes byte-deterministic. The canonical pack is read-only
throughout; the demo mutates only a synthetic test pack in a
tempfile.TemporaryDirectory. One-time recognizer seed is idempotent
(same content_digest each run → no duplicate proposal log entries).
Exit code 0 iff all scenes pass; --json for CI integration.
Also adds:
- README "Watch the flywheel turn — one command" section pointing
to the demo + the coverage CLI (per-shape histogram + hazard pin)
- ProposalLog entry for the multiplicative_aggregate recognizer
with extract_values=True (one-time operator seed)
Files:
- evals/flywheel_demo/run_tour.py (new) — the four-scene tour
- evals/flywheel_demo/__init__.py (new)
- core/cli.py — `flywheel` added to `core demo` choices + dispatch
- README.md — new "Quick Start" subsection
- teaching/proposals/proposals.jsonl — seeded recognizer
Three review fixes:
1. Security: validate lane/split/version against ^[a-z0-9_]+$ before
building the runner module name. The runner_args list is passed to
subprocess.run without shell=True (no shell injection possible),
but defense-in-depth blocks arbitrary token characters from
reaching Python's -m module loader. Bad input now errors at the
CLI boundary with a clear message.
2. Bug-risk: _classify_refusal docstring referenced a
no_admissible_candidate bucket that the implementation never
emitted. Aligned docstring with actual buckets
(no_admissible_question / no_admissible_statement). Also made all
matching consistently case-insensitive (was mixed — some checks
used raw reason, one used .lower()).
3. Bug-risk: fetch_committed_baseline wrote to
.git/coverage_baseline_tmp.json. Replaced with tempfile.mkstemp in
the system temp dir — avoids (a) failures in non-git worktrees
where .git is a file pointer, (b) concurrent-access collisions
between simultaneous operators.
Tests (+3 new):
- test_classify_refusal_is_case_insensitive
- test_classify_docstring_matches_implementation_buckets
- test_fetch_committed_baseline_uses_system_temp
All 16 coverage tests green. Verified the validation:
core teaching coverage --lane 'evil; rm -rf /'
→ ERROR: lane='evil; rm -rf /' must match ^[a-z0-9_]+$
Brief D from PR #407. Closes the "flying blind on per-shape coverage"
gap identified in RAT-1's audit (finding 6).
After this PR, every operator can run a single command to see exactly
which refusal modes their work moved (or didn't), without re-eyeballing
report.json by hand.
Modules
-------
- teaching/coverage.py — pure aggregator:
- _classify_refusal — maps each per-case refusal reason to a
stable bucket (recognizer_empty_injection(<ShapeCategory>),
no_admissible_question, no_admissible_statement,
unexpected_question_count, other)
- build_coverage_report — reads a lane's report.json + emits a
CoverageReport with counts, refusal_taxonomy (sorted by count
desc), case_0050_verdict, optional delta vs baseline
- fetch_committed_baseline — uses `git show HEAD:<relpath>` to
pull the baseline report.json for delta computation
- core/cli.py:
- cmd_teaching_coverage — formats the report for terminal output
- core teaching coverage [--lane gsm8k_math] [--split train_sample]
[--version v1] [--use-reader] [--run] [--delta] [--json]
CLI output example
------------------
Lane: gsm8k_math/train_sample/v1 (use_reader=True)
Counts: correct=3 refused=47 wrong=0
Refusal taxonomy:
21 recognizer_empty_injection(discrete_count_statement)
6 no_admissible_statement
5 recognizer_empty_injection(multiplicative_aggregation)
4 no_admissible_question
4 recognizer_empty_injection(currency_amount)
3 recognizer_empty_injection(rate_with_currency)
2 recognizer_empty_injection(descriptive_setup_no_quantity)
2 recognizer_empty_injection(temporal_aggregation)
Wrong=0: ✓
Case 0050 hazard pin: refused ✓
Tests (13 new)
--------------
tests/test_teaching_coverage_cli.py — classification narrowness,
counts aggregation, case 0050 verdict capture, delta computation,
missing-baseline path, missing-report error, taxonomy sort order,
wrong=0 invariant visibility via as_dict.
Suite results
-------------
core test --suite teaching -q → 106 passed (93 → +13)
core test --suite runtime -q → 20 passed
core test --suite packs -q → 127 passed
core eval gsm8k_math --split public → 150/150, wrong=0
Note on Brief E (lexical auto-compile): the audit was WRONG. The
lexicon loader (generate/comprehension/lexicon.py::load_lexicon)
reads from the per-category source files directly; the compiled
lexicon.jsonl is only a manifest-checksum pin, not the source of
truth at runtime. apply_lexical_claim() writes a new entry → next
turn the loader sees it. Brief E is a non-issue; closing without a
code PR.
Verified by direct test: stage a clone of the math pack, write a
synthetic lemma to drain_token.jsonl, clear the lexicon cache, load
again → new entry present. So 3 of the 5 audit gaps closed (A, D,
E-as-correction); B and C remain as the next operator dispatch
targets.
Independent of PR #406 (RAT-1) and PR #408 (WAVE-A). Based on main.
Addresses 5 of 47 train_sample "recognizer matched but produced no
injection" refusals (the largest single failure-mode bucket
identified in RAT-1's audit).
Modules
-------
- generate/recognizer_match.py:
- _MULT_AGG_EACH_WEIGHING_RE — regex for "<Subject> <bake-verb>
<M> <outer-noun>, each <weigh-verb>ing <N> <unit>" pattern
- _try_extract_each_weighing_anchor — extracts M, N, subject,
inner unit; emits pre-composed CandidateInitial(value=M*N) with
composition_evidence so RAT-1's _composed_initial_admissible
gate verifies INPUT tokens ground (preserves wrong=0)
- _match_multiplicative_aggregation dispatches to the value
extractor when spec carries extract_values=True; specs without
that flag get the existing detection-only return path
(byte-identical legacy behavior)
- generate/recognizer_anchor_inject.py:
- inject_multiplicative_aggregation — new per-category injector;
narrow by anchor.kind so ME-3/ME-4 additive/subtractive anchors
(which share the same matcher entry point) continue to flow
through composition_registry consult instead of WAVE-A's direct
path
- registered in _INJECTORS dict (2nd entry after DCS)
- core/cli.py:
- seed-recognizer CLI gains --extract-values flag to opt the
canonical_pattern into the value-extracting matcher path
Seeded artifacts
----------------
- proposals.jsonl: rat1-seed-4dc30608fb783bc7 — multiplicative_
aggregation recognizer with anchor_kind=multiplicative_aggregate,
extract_values=True, observed_units covering ounces/strawberries/
questions/etc.
Live result on train_sample
---------------------------
- wrong == 0 preserved (3/47/0 baseline)
- Case 0050 hazard pin held
- public 150/150 preserved
- packs suite: 127 → 131 (+4 new WAVE-A tests, all green)
- teaching suite 93 unchanged
- runtime suite 20 unchanged
End-to-end synthetic solve (FIRST WAVE-A admission):
"Lilibeth fills 6 baskets where each basket holds 50 strawberries.
How many strawberries does Lilibeth have?" → answer=300
Cases that moved (statement now admits; refusal shifted downstream):
- Case 0025 (Lilibeth): statement admits via WAVE-A; refusal moved
to question parser ("If three of Lilibeth's friends pick the same
amount, how many strawberries do Lilibeth and her friends pick in
all?")
- Case 0047 (John bakes 12 macaroons): statement 1 admits; refusal
moved to statement 2
Eval correct count unchanged because the QUESTION parser (and
multi-statement cross-sentence reasoning) is the next bottleneck.
RAT-1's audit identified that gap; WAVE-A closes the injector half.
The remaining 3 multiplicative_aggregation refusals (0006, 0013,
0045) have different shape patterns the WAVE-A regex does not yet
cover; they're follow-up matcher extensions in the same architecture.
Tests
-----
- tests/test_wave_a_multiplicative_aggregation_injector.py (10
tests): each-weighing + each-basket-holds admission shapes,
detection-only path preserved when extract_values absent,
unobserved unit / pronoun / zero count refusals, end-to-end
inject_from_match dispatch, the Lilibeth canary solve,
wrong=0 preserved, case 0050 hazard pin
Stacks on PR #406 (RAT-1).
Adds surface_pattern, composition_category, and polarity to the
proposed_change_payload for composition_reclassification proposals so
operators can call apply_composition_claim() without field synthesis.
Dispatch by missing_operator:
- quantity_extraction → multiplicative_composition + bound(count) × bound(unit_cost)
- multi_quantity_composition → additive_composition + bound(qty_a) + bound(qty_b)
All other change kinds (matcher_extension, injector_sub_shape,
frame_reclassification) keep the existing evidence-aggregation payload.
Legacy fields (evidence_count, group_key, modal_sub_type) preserved.
Adds tests/test_contemplation_ratifiable_payload.py with 11 tests
including a round-trip from decompose_audit → apply_composition_claim.
Phase 1 and Phase 2 of the ADR-0164 reader are correctly implemented but
contribute zero eval admissions today. The bottleneck is lexicon coverage
(unknown verbs/nouns) and explicit Phase 2.1 scope gates (fractions), not
the all-or-nothing dispatch policy.
Produces COMPREHENSION-READER-AUDIT.md answering the five Brief C questions:
call trace, cognition-lane usage (none), bottleneck analysis, ADR promise
audit, and three falsifiable options (operationalize/relabel/retire).
Recommendation: relabel (status update) now; lexicon expansion next as the
highest-leverage first step toward actual eval lift.
Also updates ADR-0164 status from "Proposed" to "Partially implemented" with
a Current Status table and next-lift-path summary.
After RAT-1's architecture audit exposed five systematically
underbuilt components, A (the 4 missing injectors) is being taken by
this agent. The other four are parallel-safe and each gets its own
dispatch brief:
B (Opus) — contemplation produces ratifiable claims (medium)
C (Sonnet) — comprehension reader audit + decision (small→medium)
D (Sonnet) — core teaching coverage CLI (small)
E (Codex) — lexical ratification auto-compile (tiny)
Each brief carries: operator profile, branch name, base, dispatch
line, reads required FIRST, outcome inventory, hard requirements,
tests, truth test. Anti-regression invariants enumerated. All
parallel-safe — no shared file conflicts.
The dispatch DAG names ALL FIVE pieces with A in-flight as a peer.
The user's question — "shouldn't we be running it multiple times so
it can learn? or is that part broken?" — exposed that the math
teaching loop's `ratify → admit` closure had been structurally
broken at the connector between operator ratification and runtime
visibility. The handlers wrote source files (compositions/, frames/)
that the runtime loader never read because no compile step
regenerated the runtime artifacts.
This PR fixes the gap end-to-end AND fires the first live composition
admission on the canonical pack.
Modules
-------
- language_packs/compile_pack.py — unified compile step that
regenerates frames.jsonl + compositions.jsonl + updates
manifest.{frame,composition}_checksum atomically. Idempotent.
- teaching/math_composition_ratification.py — apply_composition_claim
now calls compile_pack at end of successful ratification. Closes
the source-file→runtime-artifact gap.
- teaching/math_frame_ratification.py — same auto-compile wire for
apply_frame_claim.
- generate/math_candidate_parser.py — CandidateInitial gains optional
composition_evidence Mapping field. When populated, signals the
candidate was produced by a registry-gated composition (ADR-0169);
the value/unit/entity are DERIVED arithmetic over grounded inputs.
- generate/math_candidate_graph.py — new _composed_initial_admissible
predicate that branches on composition_evidence. Wrong=0 preserved
by requiring each composition INPUT token (count, amount) to ground
in source_span literally; the derived value is admitted because the
arithmetic over grounded inputs is deterministic.
- generate/math_candidate_graph.py — discourse-level prior_subject
tracking: capture proper-noun subjects from ALL statement sentences
(including ADR-0136.S.0 context-filler sentences that get filtered
out before the candidate loop). Without this, "John adopts a dog"
(no numbers) is dropped and the cross-sentence subject resolver for
case 0019 sees prior_subject=None.
- generate/recognizer_match.py — all four composition matchers
(ME-1 currency-per-unit same-sentence, ME-2 cross-sentence, ME-3
additive, ME-4 subtractive) now populate composition_evidence in
CandidateInitial. Also added standalone " each " / " apiece " to
_PER_UNIT_TOKENS so currency_amount detection-only matcher refuses
per-item costs instead of swallowing them.
CLIs
----
- core teaching compile-pack — explicit operator surface for
regenerating runtime artifacts. JSON output for CI integration.
- core teaching seed-recognizer — operator surface for seeding a
RatifiedRecognizer entry in the proposal log for a given
(shape_category, anchor_kind). Writes created + transition(accepted)
events directly via ProposalLog._append.
Seeded artifacts (the actual loop closure)
------------------------------------------
- proposals.jsonl: new rat1-seed-48dd2673d6ad673d RatifiedRecognizer
entry for shape_category=rate_with_currency,
anchor_kind=currency_per_unit_composition.
- compositions/multiplicative_composition.jsonl: ratified
"bound(count) × bound(unit_cost)" affirms entry sourced from
case 0019 evidence.
- compositions.jsonl + manifest.composition_checksum: compiled
runtime artifact + manifest pin (RAT-1 auto-compile).
Live result on train_sample
---------------------------
- wrong == 0 preserved (3 correct / 47 refused / 0 wrong)
- Case 0050 hazard pin holds (refused)
- public split 150/150 preserved
- Case 0019 sentence 1 ("requires 3 vet appointments, which cost
$400 each") NOW ADMITS via composition. Previously refused with
"recognizer matched but produced no injection". The refusal moved
downstream to sentence 2 (a different currency_amount detection
bottleneck that is its own follow-up).
This is the first time a composition ratification on the canonical
pack actually reaches the runtime. The flywheel turned one
revolution.
Tests
-----
- tests/test_rat1_end_to_end_admission.py — 4 new live tests:
composition statement admits on isolated synthetic problem, case
0019 cross-sentence admission, wrong=0 preserved on train_sample,
case 0050 hazard pin.
- tests/test_consumption_empty_registry_no_op.py — refactored to use
isolated synthetic packs (the canonical pack may now carry ratified
entries).
- tests/test_math_{frame,composition}_ratification.py — updated
"manifest checksum unchanged" tests to "lexicon checksum
preserved" semantics: RAT-1 auto-compile may add the new optional
checksum fields; pre-existing lexicon checksum stays untouched.
Suite results: teaching 93, packs 131 (+4), runtime 20. All green.
