Reconciles ChatGPT's four independently-branched extraction PRs (#451/#452/
#453/#454) into one coherent generate/derivation/extract.py. They each rewrote
the same file + same new test off main, so they conflicted pairwise and needed
integration, not a merge.
Integrated (span-tracked, most-specific-pass-first so numbers are never double
counted):
- EX-1 word-numbers (#452): reuses WORD_NUMBERS; tens-one hyphen compounds;
factor-bearing half/third/quarter excluded.
- EX-4 list-unit inheritance (#451): bare numeric list with one trailing unit.
- EX-5 sentence-final numbers (#454): bare final number with empty unit.
Deferred: EX-3 multi-word units (#453). Its greedy lowercase span reads
"6 apples and 4 apples" as unit "apples and", regressing GB-2's
test_same_unit_list_sums, and still can't recover real multi-word units from
0024-class text ("jumping jacks on"). Needs a tighter rule; see
docs/handoff/AUDIT-ADR-0179-EX-RECONCILE.md.
Verification (sealed lane only; chat/ does not import this module):
- Serving frozen: lane-SHA 8/8 match, generate_claims --check OK -> 3/47/0
byte-identical, wrong=0 held.
- Sealed practice improved 4/2/44 -> 4/1/45: case 0025 flips wrong->refused.
EX-1 reads "three", so completeness sees a quantity the 6x50 chain omits and
refuses the spurious 300 (gold 1200) instead of committing it.
- No new test failures (3 pre-existing on main).
Also fixes stale test drift from EX-2 (#447): TestDecimalGroundingGapIsDeferred
asserted decimals still refuse, but #447 made $0.75-class resolve to 864.
Renamed to TestDecimalGroundingResolves and updated to assert the flip.
Honest scope note: EX-4 does NOT unblock real case 0024 (its PR test used a
fabricated bare-list paraphrase). TestRealCase0024StillBlocked pins the true
boundary.
EX-2 — the ONE shared-primitive (serving-path) touch of extraction richness.
_value_grounds already grounds the symbol form $N.NN (currency) and N/M (fraction);
a decimal written WITHOUT a symbol ('0.75') is never a single token (the tokenizer
splits on '.') so it failed to ground — refusing correct products like 0003
(48*24*0.75=864). Now a bare decimal 'N.M' grounds when both digit-runs appear,
symmetric with the $N.NN and N/M branches. Only returns True on a match; non-matching
decimals fall through (refuse).
wrong=0 (the load-bearing obligation, this being a shared/serving-path change):
- serving 3/47/0 BYTE-IDENTICAL (verified).
- round-trip + candidate-graph tests 51/51.
- lane-SHA gate (pinned eval lanes unchanged) — verified before merge.
Payoff: 0003 unblocked in sealed practice (search_chain -> 864 = gold; +1 flip).
Decimal cases that now ground but mis-compose become sealed eliminations (learning
signal); serving untouched. 6 EX-2 tests (decimal grounds/refuses, integer
unchanged, serving byte-identical, 0003 resolves).
GB-2 first increment (ADR-0178). compose_sequential() adds the structure the blunt
MS-3 shapes couldn't reach: a same-unit quantity LIST sums (additive cue), and any
stated comparative scales the sum (sum-then-scale, 0024-family). Op-per-step from
text structure (list => add; comparative => scale); operands are text quantities
(grounded) + comparative steps (cue-grounded) on the flat left-fold — no derived-
intermediate model needed (running value is the intermediate).
Deliberately narrow: same-unit lists only. A stated comparative is ALWAYS applied
(no bare-vs-scaled self-disagreement). A product base over the same list is added
WITHOUT a comparative tail purely as a disagreement-safety candidate -> a same-unit
list that also carries a mult cue (ambiguous) REFUSES. Product-of-all/cross-unit
products stay MS-3's job (avoids the product x comparative blowups a blunt all-bases
composer produced: 0024 -> 4.3M).
Clean-case capability proven: 8 tests (list-sum, sum-then-double/triple, mixed-units
refuse, ambiguous-disagreement refuse, determinism). Honest practice result: 3/2/45
— NO new flips (extraction wall: real cases like 0024 extract non-uniform units
'36 on' so they aren't seen as same-unit lists), 2 sealed eliminations (0037/0039:
list-sum was the wrong structure -> learning signal). Coverage gated by extraction
richness + cue precision, as predicted.
Sealed; serving untouched. Full derivation surface 53/53; ruff clean; smoke 67.
Continuation: richer relational ops (per/each->multiply, more/older->add), branch/
DAG (0033), and the extraction richness (uniform-unit extraction) that unblocks this
on real cases.
GB-1 — first slice of the comprehension-guided composer (ADR-0178). Reads the
problem one clause at a time and derives each clause's LOCAL contribution; GB-2
combines them across clauses.
generate/derivation/clauses.py:
- segment_clauses(text): sentence-level orthographic split (ADR-0165; not grammar).
- clause_local_results(text) -> tuple[ClauseResult]: per clause, 0 quantities =
context (hold), 1 = leaf (its value), >=2 = bounded local search (reuses MS-3
search_chain). Refuse-preferring: ambiguous multi-quantity clause -> unresolved
hold, not guessed.
Locality is the guidance that bounds the search + steers grouping. 9 GB-1 tests
(segmentation, leaf/context/local-product, ambiguous-holds, determinism,
per-clause structure of a multi-sentence problem). Full derivation surface 86/86;
ruff clean; smoke 67. Sealed; not wired into serving (ClauseResults ready for
GB-2 sequential combination).
MS-3 composes MS-1 (Target) + MS-2 (comparative chains + completeness) + the gate.
generate/derivation/multistep.py: search_chain(problem_text, target=None).
Shape-based, NOT blind enumeration: enumerates a small principled candidate set
(product-of-all if a multiplicative cue is present; sum-of-all if an aggregation
hint is present; each optionally + comparative scalars), each using all
quantities, routed through select_self_verified (grounding ∧ cue ∧ unit ∧
completeness ∧ uniqueness). Bounded (MAX_QUANTITIES, refuse-on-overflow) +
deterministic. Target supplies the aggregation cue + question quantities; target-
UNIT matching is deferred (answer_unit=start.unit is wrong for cross-unit products
-> a unit gate would over-refuse; documented).
Honest practice measurement (sealed lane): 4 correct / 9 wrong / 37 refused
(baseline 3/0/47). +1 flip is the unambiguous whole-problem product (0021); the 9
wrongs are product-of-all eliminations on multi-step problems (caught by gold,
the learning signal). Whole-problem shapes add no coverage beyond the unambiguous
product WITHOUT cue precision: when product and sum both self-verify they disagree
-> uniqueness refuses (safe-but-low-coverage by design). The lever remains cue
precision (the ADR-0175 learning loop).
Microscope finding: 0003-class flips (48*24*0.75=864=gold) are blocked by a
DECIMAL/currency grounding gap -- '$0.75' tokenizes to 0/75 so '0.75' is not
grounded by the shared round-trip primitive. Not a search bug; deferred
extraction-richness work (won't casually change the serving round-trip primitive).
A test documents the current refusal so the fix is detectable.
wrong=0: serving untouched (sealed); ambiguity + no-licensed-cue refuse; routes
through the proven gate. 8 MS-3 tests; full derivation surface 77/77; ruff clean;
smoke 67.
MS-2 of multi-step composition. Extends the derivation model so a chain mixes
text-quantity operands and COMPARATIVE-scalar operands (twice->x2, 'N times'->xN,
half->x0.5), self-verifying the whole chain with completeness over body+question
and question-target matching.
- model.py: Step gains comparative flag.
- comparatives.py: ComparativeScalar gains number_token (the '<N> times' number,
so completeness counts the consumed body quantity); comparative_step(cs) bridges
a scalar into a Step (operand grounded by cue, not a text value token).
- verify.py: self_verifies exempts comparative operands from value-grounding
(clause 1) — they are cue-grounded (clause 2); completeness (Counter) counts a
digit comparative's number_token as consuming the body quantity. Adds target_units
to select_self_verified: a chain whose answer_unit isn't the asked unit is dropped
(question-target match; empty target_units imposes no constraint).
Proves the multi-step shapes from the gold structures: 0024 (text sum then 'three
times' scale -> 438), 0033 father-chain (digit-comparative '7 times' + fixed 'half'
+ text add -> 47). Full 0033 DAG (quantity reuse + the question's 25) deferred.
25 MS-2 tests; full derivation surface 69/69 (3a/3b/comparatives/ms1/ms2); ruff
clean; smoke 67. Not wired into serving (model ready for MS-3 target-guided search).
MS-1 of multi-step composition. Turns the question into a Target = what the
problem asks for, the search's pruning signal + stopping criterion (MS-3).
Lexeme-level only (ADR-0165): the existing question parser returns nothing on
these GSM8K questions, and 0165 forbids new question-shape grammar regex. Three
robust signals:
- quantities: numbers stated IN the question (0033's 'when she is 25') via the
body's lexeme extractor — they participate in the derivation.
- aggregation: presence of an aggregation lexeme (total/altogether/combined/sum/
'in all'/'in total') — soft hint the final step is a sum.
- units: asked units resolved by INTERSECTION with the body's known units
(precise lexeme match, e.g. 'jumping'). Superordinates (weight<->pounds) are
NOT faked — deferred to a curated superordinate-units pack; until then the unit
signal is precise-but-incomplete and the search leans on completeness.
Refuse-preferring: empty target field is not an error, just a weaker prune.
generate/derivation/target.py: Target + extract_target(question, known_units=()).
12 MS-1 tests (question-quantity, aggregation, body-unit intersection,
superordinate-not-faked, determinism, frozen). Verified: derivation suite 57/57;
ruff clean; smoke 67. Not wired into serving (Target ready for MS-2/MS-3).
The curated, irreducible world-fact primitives multi-step composition needs
(ADR-0175 section 10: the engine can't derive 'twice = 2' from arithmetic). The
microscope flagged these via the 0015/0025/0024/0033 wrongs.
language_packs/data/en_core_comparatives_v1/: 9 closed-set multiplicative
comparatives (twice/double/triple/quadruple/half/quarter + inflections) -> scalar
ops. manifest.json with sha256 of the bytes on disk (CLAUDE.md pack rule).
Refusal-preferring: non-terminating/ambiguous comparatives (a third, several)
deliberately excluded; expansion via HITL corridor.
generate/derivation/comparatives.py: extract_comparative_scalars() ->
ComparativeScalar(op, scalar, span, cue). Fixed lexemes + the '<number> times'
pattern (digit or word-number via WORD_NUMBERS). Lexeme-level (ADR-0165);
deterministic (text-order); supplies only the SCALAR primitive — referent
binding is the multi-step search's job (ADR-0176).
14 tests incl. refusal-preferring discipline + pack integrity (manifest checksum
matches bytes on disk). Verified: derivation suite 45/45; ruff clean; smoke 67;
packs 141. Not wired into serving (data + extractor ready for ADR-0176 MS phases).
Self-verification strengthening (microscope-driven). The Phase 3b measurement
showed self-verification was necessary-but-not-sufficient: 9/13 self-verified
attempts were wrong. Inspecting them deterministically revealed most were
correct FIRST STEPS of multi-step problems that ignored numbers stated elsewhere.
Adds clause 5 to self_verifies: a derivation must account for every quantity the
problem states (problem quantities subset of used). Refuse-preferring: unused
quantities -> not self-verified. This catches the multi-step-incomplete attempts
the grounding/cue/unit clauses cannot (their operands ARE grounded).
Practice measurement: wrongs 9 -> 2 (4 correct / 2 wrong / 44 refused). The 2
survivors (0015, 0025) are COMPLETE but wrong due to missed WORD-quantities
('twice', 'her friends') -> the microscope points the next change at extraction.
Updated the disagreement test to use two complete derivations; added an
incomplete-refusal test. 32 tests pass; smoke green; serving untouched (sealed).
ADR-0175 Phase 3b — the first live attempt generator. Runs only in the sealed
practice lane, only on cases the engine refused; every proposal is gated by the
Phase 3a self-verification gate.
generate/derivation/:
- extract.py: extract_quantities() — lexeme-level (number + unit word; ADR-0165).
- search.py: search_multiplicative() — one in-clause product candidate per
sentence with >=2 quantities + a present multiplicative cue; gated by
select_self_verified. Per-sentence scope + multi-candidate disagreement give
the uniqueness gate real teeth (two qualifying sentences -> refuse). The cue
set {each,every,for,per,times} is an explicit PROVISIONAL hypothesis the
practice loop refines, not a claimed-correct grammar.
evals/gsm8k_math/practice/v1/search_runner.py: search_augmented_scorer +
build_search_report — base scorer, then a practice-only attempt on refusals.
MEASUREMENT (the deliverable, per the breadth-of-impact test):
practice with search: correct=4 wrong=9 refused=37 (baseline 3/0/47)
- Flips +1 (0021, the clean in-clause aggregate) and its renumbered/reworded
variants (ADR-0114a perturbation guard) -> a real capability, not memorisation.
- 9 wrong attempts -> elimination records (§9), the learning signal. The naive
full-product cue model over-attempts; the eliminations are exactly the signal
that refines it.
HONEST FINDING: self-verification (grounding ∧ cue ∧ unit ∧ uniqueness) is
NECESSARY but NOT SUFFICIENT — 9/13 self-verified attempts were wrong vs gold.
The gap is cue PRECISION / which-quantities-compose (the knowledge axis), not
'can we multiply' (skill). This is why the search runs sealed: gold catches the
9, and case 0050 (canary) attempted-and-failed IN PRACTICE without touching
serving -> validates the seal.
Invariants: #1 seal (serving still 3/47/0; 0050 refuses in serving; no
generate/chat import of the lane), #3 determinism. Serving wrong=0 untouched.
Verified: 3a+3b 31/31; ruff clean; serving lane 4/4; smoke 67/67.
ADR-0175 Phase 3 splits wrong=0-first: build the gate (3a) and PROVE invariant #2
before the bounded search (3b) that could exploit gaps.
generate/derivation/:
- model.py: Quantity / Step / GroundedDerivation. A derivation is a left-fold over
text-sourced quantities; each Step carries its licensing cue (the lexeme the
search claims licenses the op).
- verify.py: self_verifies() — grounded operands ∧ grounded operation cues ∧ unit
consistency ∧ no divide-by-zero. Grounding REUSES the canonical primitives from
math_roundtrip (_tokens/_token_in/_value_grounds) so the gate cannot drift from
the round-trip contract. select_self_verified() adds the uniqueness rule:
unique self-verifying answer resolves; zero or disagreeing refuse (wrong=0).
INVARIANT #2 proven (TestInvariant2_NoSpuriousSelfVerification): the gate refuses
to self-verify a derivation that is not grounded+unit-consistent+unique even when
its value coincides with gold — the 20/5==4 class:
- invented operand not in text -> refused
- operation cue not in text -> refused (division not licensed by any present cue)
- value coincidence (20/5=4) with ungrounded op -> still refused
- add across units (pounds + reps) -> refused
- divide-by-zero -> refused
Plus uniqueness: disagreeing grounded derivations -> refuse; agreeing -> resolve.
Phase 3a is inert (nothing wires generate.derivation into serving). 3b is the
bounded search that produces derivations for this gate + measures the flip-curve
in the practice lane under perturbation.
Verified: 16/16; ruff clean; smoke 67/67; no serving import.
ADR-0175 Phase 2 — a NEW lane (evals/gsm8k_math/practice/v1/), separate from the
wrong=0-pinned serving runner which is NOT modified. Runs the 50 cases in
practice mode: scores correct/wrong/refused as practice metrics, feeds per-class
counts into the Phase 1 ledger, diagnoses every refusal (§8), emits an
elimination record per wrong.
- classify_operation: gold-derived primary op class {multiplicative,divisive,
additive} from <<a*b=c>> calc annotations (Tier-1 checkable in practice).
- diagnose_refusal (§8): skill_gap / knowledge_gap / genuine_ambiguity router.
- EliminationRecord (§9): wrong attempt gold caught -> pruning signal.
- PracticeReport: counts + per-class ledger + diagnoses + eliminations; as_dict.
- run_practice(cases, scorer=...): injectable scorer for tests; defaults to the
candidate-graph scorer (read-only — never alters serving).
Live result mirrors serving (3 correct / 0 wrong / 47 refused of 50) because the
engine still refuses rather than guesses — attempts/eliminations go live in
Phase 3. But the diagnosis is already actionable: 35 skill_gap / 12 knowledge_gap
/ 0 genuine_ambiguity — 74% of refusals are skill gaps (Phase 3's search target),
quantifying the skill-vs-knowledge split.
Invariants: #1 seal (serving still 3/47/0; no generate/chat import of the lane),
#3 determinism (report byte-identical across runs). Elimination + wrong-tolerance
paths unit-tested via injected scorer (no live wrongs yet).
Verified: Phase 1+2 53/53, serving train_sample tests 4/4 (seal), smoke 67/67,
ruff clean.
ADR-0175 Phase 1 — standalone, deterministic, zero serving change. Nothing in
the serving/eval path imports it.
core/reliability_gate/:
- floor.py: conservative_floor(s,k) — pinned one-sided Wilson lower bound over
COMMITTED trials. z=2.576, N_MIN=10; range [0,1) (never exactly 1.0); float64
rounded half-to-even to 1e-9 for cross-backend replay. Perfect record reduces
to k/(k+z²) (earned by volume).
- ledger.py: ClassTally — immutable per-class counts; reliability = commitment
precision (refusals excluded so coverage never penalizes reliability);
t2_precision over the anchor set; coverage tracked separately.
- ceilings.py: Action{PRACTICE,PROPOSE,SERVE} + Ceilings — human-set θ
(practice=0, propose=.85, serve=.99). Frozen; with_override returns a NEW
instance (no in-place self-authorization).
- gate.py: license_for() — deterministic gate, measured/required≥1 (≡ measured≥
required; required=0 ⟹ always). Pure; never mutates/emits ceilings.
34 tests, each ADR invariant exercised by a test that fails under its violation:
#3 determinism/replay (idempotent, pre-rounded, deterministic decisions),
#4 no self-authorization (frozen ceilings; gate never emits/mutates them),
#1 proxy (zero serving coupling). Plus the §4a worked examples (38 clean
commitments clear propose; one wrong in 40 drops below; serve needs ~657).
Verified: 34/34 pass; architectural invariants 40/40; smoke 67/67; ruff clean;
no serving/eval import of the package.
The recognizer/candidate-graph path is the single canonical reader.
Retires the flag-gated incremental-reader dispatch that admitted 0/50 on
train_sample and only added a dead fall-through:
- remove _try_comprehension_reader, _try_reader_for_question, _tokenize_sentence
and both dispatch blocks from generate/math_candidate_graph.py
- delete generate/comprehension/lifecycle_runtime_adapter.py (402 LOC,
used only by the question-reader dispatch)
- drop the comprehension_reader_questions config flag and the parse_and_solve
/ _score_one_candidate_graph config threading
- remove the --use-reader runner plumbing + flag-ON/OFF delta report from
the train_sample runner; refresh report.json (drops stale use_reader field
and a stale refusal-reason; verdicts unchanged at 3/47/0)
- remove the now-dead use_reader field from teaching/coverage.py
CoverageReport + the core teaching coverage CLI flag
- delete tests/test_reader_coexistence.py (flag-ON/OFF premise dissolved);
fix 3 ADR-0174 build_report calls and 2 subprocess invocations
lifecycle.py and audit.py are KEPT — they are load-bearing for the ADR-0172
math-contemplation teaching corridor (audit_problem -> teaching/math_*),
which a pre-deletion trace surfaced. The parent ADR's plan to delete
lifecycle.py was wrong; only its GSM8K scoring dispatch was inert.
Net -1,038 LOC (code + tests). Behavior-preserving:
- train_sample 3/47/0, byte-identical verdicts to pre-5a baseline
- determinism holds; smoke/packs/runtime/cognition/teaching lanes green
- contemplation corridor + lifecycle/audit tests pass
Pre-existing (NOT introduced here; reproduce on base with changes stashed):
5 out-of-curated-lane stale committed-artifact / stale-assertion failures
(test_math_evidence_e2e, test_adr_0126_runner_wiring, G3/coverage_probe
report-match, test_refusal_taxonomy_lane rebuild).
ADR-0174 Phase 3b — emit N anchors for compound-clause discrete-count
sentences sharing one subject + one verb. Architectural substrate;
score on train_sample preserved at 3/47/0 (compound cases like 0027
admit past the recognizer-injection refusal but the rest of the
problem still has downstream complexity — fractions, percent — that
needs Phase 4 + solver work).
generate/comprehension/state.py:
HYPOTHESIS_CAP raised 4 → 8. Case 0040 emits 5 anchors; cap=8
gives headroom (7-item lists) without becoming permissive.
generate/recognizer_match.py:
_try_extract_compound_discrete_count_anchors() — new extractor
emitting tuple of anchors for compound sentences. Refusal-
preferring on:
- no conjunctive separator (single-anchor path)
- multiplicative/percent/fraction markers
- head verb not in whitelist
- any tail clause without grounded (count, observed_noun) pair
- exceeding HYPOTHESIS_CAP
- unaccounted digit in tail (wrong=0 hazard defense surfaced by
2026-05-28 implementation review: bogusnoun would silently fail
to produce anchor while leaving the digit unaccounted, admitting
partial state)
Wired into _match_discrete_count_statement dispatch as fallback when
single-anchor extraction fails.
tests/test_adr_0174_phase3b_compound_clause.py:
11 acceptance tests passing — pure conjunctive lists (proper-noun
+ pronoun-subject + single-actor antecedent), refusal-preferring
discipline (mixed-verb, multiplicative-tail, non-whitelisted-head,
partial-grounding all-or-nothing), HYPOTHESIS_CAP enforcement,
multi-actor pronoun defense preserved on compound, wrong=0 +
case-0050 canary.
tests/test_adr_0174_phase1_held_hypothesis_state.py:
Updated test_hypothesis_cap_is_four → test_hypothesis_cap_is_eight
with rationale for the raise.
Phase 3b implementation lookback review (per CLAUDE.md doctrine):
- Surfaced silent-partial-admission hazard in tail extraction;
fixed with digit-accounting check before commit
- Surfaced LATENT regex-path multi-actor pronoun hazard (not
introduced by Phase 3b; documented in test docstring with
cross-reference to project-adr-0174-multi-actor-pronoun-hazard
memory for follow-up)
- case 0040 ('He now has...') remains refused — 'now' adverb between
subject and verb defeats the existing canonical regex. Adverb-
stripping is separate scope (not Phase 3b).
Acceptance:
- 258/258 ADR-0174 + math_problem_graph tests pass
- Smoke 67/67, packs 141/141
- train_sample 3/47/0 preserved (wrong=0 held)
- Case 0027 'Malcolm has 240 followers on Instagram and 500 followers
on Facebook' now admits via the compound extractor — verified by
refusal moving to the next sentence (which has 'half' fraction)
All findings from the 2026-05-28 Phase 1-3a lookback review addressed
in one commit on the Phase 3a branch:
Wrong=0 hazard defense (the load-bearing fix):
- generate/math_candidate_graph.py: Phase 3a wiring now collects the
set of distinct proper-noun subjects seen in prior context. When
more than one exists, refuses with no_antecedent_ambiguous trace
event rather than guessing the most-recent (which was gender-blind
single-binding — wrong attribution in multi-actor problems).
- Refusals from the statement loop now preserve _statement_trace via
reader_trace in CandidateGraphResult (pre-existing latent issue:
Phase 2/3 trace events were dropped on early statement refusal).
- New tests assert: ambiguous case refuses with correct trace; single-
actor case still resolves normally.
Test coverage backfills (closes the 13 untested predicate-name gaps):
- TestCheckConstraintsInitialPredicateNames — 3 tests asserting the
exact predicate name on initial.value_grounds / initial.unit_grounds
/ initial.entity_grounds failure paths.
- TestCheckConstraintsOperationPredicateNames — 3 tests asserting
operation.verb_grounds / operation.value_grounds / operation.unit_grounds
failure-predicate-name parity.
- TestCheckConstraintsComposedInitialPath — 4 tests for the RAT-1
composed_initial path which was entirely untested in Phase 2
(parity manually verified during lookback review; now automated).
ADR amendment (honest doc vs impl drift):
- docs/decisions/ADR-0174-held-hypothesis-comprehension.md: appended
'Implementation Notes' section documenting:
- reevaluate signature differs from spec text (shipped is more
composable; treat as amended)
- Phase 2 wires per-candidate, not per-token (per-token is Phase 5)
- Lookback recompute is candidate-level, not token-level
- Hypothesis.constraint_state is never populated by Phase 2
- Multi-actor pronoun hazard defense rationale
- Honest LOC accounting: Phases 1-3a net +1,500 lines (Phase 5
delivers the projected net removal)
- Test coverage backfill summary
Cosmetic:
- lookback.py:297 unreachable raise — added # type: ignore[unreachable]
with comment explaining defensive future-proofing for Phase 3b.
Acceptance verified:
- 124/124 Phase 1+2+3a + reader tests pass (was 95/95 before backfills)
- Smoke 67/67, packs 141/141
- train_sample 3/47/0 preserved (wrong=0 invariant held)
- Multi-actor hazard live-tested: parse_and_solve refuses the
Alice/Bob/She case with no_antecedent_ambiguous trace event
See CLAUDE.md §Lookback Review Discipline and memory
feedback-lookback-review-discipline for the doctrine that surfaced
all of these issues at the right time.
ADR-0174 Phase 3a — substrate for held-hypothesis lookback.
Score unchanged at 3/47/0 (this PR is correctly-engineered
infrastructure; eval impact gated on ADR-0163.x recognizer expansion
documented in the follow-up brief).
Adds generate/comprehension/lookback.py:
- VALID_REFINEMENT_KINDS, VALID_UNRESOLVED_SLOTS — closed sets
contracted with reader_trace consumer
- PronounResolution refinement dataclass (pronoun + resolved_to +
evidence_source, all validated)
- Refinement Union (Phase 3b will widen with CompoundClauseExpansion)
- ReevaluateResult dataclass with admit/eliminate consistency
- reevaluate(hypothesis, refinement) operator — applies refinement,
re-runs check_constraints, returns refined Hypothesis or None.
- _rebuild_candidate_with_resolved_actor — rebuilds
CandidateOperation / CandidateInitial replacing the semantic actor
field (op.actor / initial.entity) while preserving matched_actor_token
/ matched_entity_token as the pronoun (so grounding still passes
against the held statement's source span).
Modifies generate/recognizer_match.py:
- _try_extract_discrete_count_anchor: pronoun-subject statements now
emit anchors with subject_role=<pronoun> + requires_pronoun_resolution
marker, rather than refusing at the _REFUSED_SUBJECT_TOKENS check.
The other narrowness layers (clause split, verb whitelist) still
refuse; only the pronoun layer changes.
Modifies generate/math_candidate_graph.py:
- After inject_from_match, when any parsed_anchor carries
requires_pronoun_resolution, the candidates are held as Hypothesis
objects with unresolved=('actor_pronoun',). The lookback path then
resolves via the existing _discourse_prior_subjects map and runs
PronounResolution refinements through reevaluate. Resolved
hypotheses flow into per_sentence_choices as if the regex parser
had produced them; unresolved hypotheses drop cleanly (refusal-
preferring). Emits 'lookback' JSON trace events with
outcome ∈ {admitted, eliminated, no_antecedent}.
Tests:
- tests/test_adr_0174_phase3_lookback.py — 17 acceptance tests
covering operator semantics on Operation/Initial, dataclass
invariants, closed-set constants, end-to-end wiring on synthetic
problems, and wrong=0 preservation on train_sample.
Phase 3.1 follow-up brief:
- docs/handoff/PHASE-3.1-FOLLOWUP-RECOGNIZER-EXPANSION.md documents
the empirical finding that the train_sample bottleneck is
verb-coverage (recognizer scope, ADR-0163.x) not lookback
(ADR-0174 scope). 11 verbs identified for HITL contemplation pass.
Recommends sequencing: Phase 3a now (substrate), ADR-0163.x verb
expansion next, Phase 3b after coverage matures.
Acceptance verified:
- 17/17 Phase 3a tests pass
- 95/95 existing tests pass (Phase 1 + Phase 2 + brief_11 + reader_phase2)
- Smoke 67/67, packs 141/141, lanes 8/8
- wrong=0 preserved, score unchanged 3/47/0 (intentional per brief)
Stacks on Phase 2 (PR #420). Rebases onto main after #416 + #420 land.
ADR-0174 Phase 2 — hoist _initial_admissible / roundtrip_admissible into
hypothesis-based constraint checks with structured elimination tracing.
Admission semantics are byte-equivalent to today; the change is structural.
Adds generate/comprehension/constraint_propagation.py:
- VALID_PREDICATE_NAMES: closed set of 17 sub-check names spanning
initial / composed_initial / operation admissibility predicates.
Adding new names requires an ADR amendment (structural contract with
reader_trace consumer).
- ConstraintResult dataclass: admitted bool + predicates_run trace +
elimination_reason. Validates admitted-vs-reason consistency.
- Elimination dataclass: confidence_rank + predicate + reason for one
hypothesis being eliminated. Serialisable as a reader_trace event.
- hypothesis_from_initial / hypothesis_from_operation: adapters wrapping
CandidateInitial / CandidateOperation as Phase-1 Hypothesis objects
with caller-supplied confidence_rank.
- _check_initial / _check_composed_initial / _check_operation:
decomposed sub-check implementations of the existing admissibility
predicates with first-failure short-circuit (matches current
semantics). Each sub-check populates predicates_run with (name, ok|
fail|skip) so the consumer sees exactly which predicate decided.
- check_constraints: dispatches on candidate type.
- eliminate_violating: bulk filter; returns (survivors, eliminations);
survivors are re-densified to satisfy ProblemReadingState's
open_hypotheses post_init invariant (dense-from-0 ranks);
eliminations carry the original confidence_rank for trace fidelity.
Wires into generate/math_candidate_graph.py at the recognizer
injection site (line 825+): replaces inline _initial_admissible /
roundtrip_admissible dispatch with eliminate_violating. Elimination
events become JSON entries in reader_trace with layer=
'constraint_propagation', phase=2, predicate, reason, sentence_index.
Phase 2 acceptance verified:
- 24/24 ADR-0174 Phase 2 tests pass (emission, parity with existing
predicates on 9 admit/reject cases, redensification, dataclass
invariants, integration).
- 71/71 existing reader + Phase 1 tests still pass.
- Smoke 67/67, packs 141/141, lanes 8/8.
- train_sample/v1 byte-identical across two runs with use_reader=True.
- Score preserved: correct=3 refused=47 wrong=0 — semantics identical
because the decomposed sub-checks short-circuit on the same predicates
the inline checks would have caught.
Trace-event behavior: today's injectors are conservative enough that
zero eliminations occur on train_sample/v1 (no false positives, no
mid-pipeline failures). The wiring is exercised by
test_phase2_event_shape_when_synthesized which proves the trace shape
on a synthetic CandidateInitial that fails initial.unit_grounds. When
Phase 3 begins emitting partial hypotheses from apply_word, the
elimination path will fire on real candidates and the trace will
populate.
Stacks on Phase 1 (feat/adr-0174-phase1-held-hypothesis-state, PR
#416). Merges cleanly into main after PR #416 lands.
MathProblemGraph.__post_init__ now raises MathGraphError when two
InitialPossession entries share the same (entity, unit) key but
declare different quantity values.
Pre-fix behavior surfaced by 2026-05-28 ADR-0174 Phase 3 post-merge
diagnostic: math_solver.solve() line 207 used last-write-wins dict
assignment when consolidating initial state. Two contradictory
inputs would silently overwrite without trace:
'Sam has 5 marbles. Sam has 3 marbles. How many marbles does Sam have?'
→ returned 3.0 (wrong=0 violation: definite answer from
contradictory input)
Post-fix: same input refuses with 'no branch produced a solvable
graph' — refusal-preferring discipline as wrong=0 doctrine requires.
Identical duplicates (same value) are admitted as redundant (no
contradiction). Different units for same actor admitted. Different
actors for same unit admitted. Single-value cases (the dominant
real-world pattern) unchanged.
This is an extraction-layer hazard discovered while investigating
Phase 3b scope: Phase 3b compound-clause held hypotheses would
emit multiple CandidateInitial entries per sentence, exercising
exactly this consolidation path. Fixing the silent overwrite NOW
ensures Phase 3b admission doesn't silently produce wrong answers.
Acceptance:
- 4 new tests in TestContradictoryInitialPossessionsRefuse
- 165/165 test_math_problem_graph tests pass (was 161/161)
- Smoke 67/67, packs 141/141 unchanged
- train_sample 3/47/0 unchanged (no real case exercised the
overwrite — but the hazard was latent)
References: CLAUDE.md §Lookback Review Discipline (the doctrine
that surfaced this), CLAUDE.md §Non-Negotiable Field Invariant
(make illegal states difficult to represent).
ADR-0174 Phase 1 — substrate only, no admission behavior change.
Adds to generate/comprehension/state.py:
- HYPOTHESIS_CAP (=4, structural assertion per ADR-0174 §Constraints)
- VALID_HYPOTHESIS_CONFIDENCE_RANKS (closed set, no probabilistic ranking)
- Hypothesis dataclass (frozen, slots) — candidate, category_assignments,
constraint_state, confidence_rank, unresolved. The 'candidate' field is
typed as object to avoid circular import on math_roundtrip /
math_candidate_graph candidate types; Phase 2 will pin canonical_bytes
contract over real candidates.
- UnknownHeld dataclass — token, position, narrowed_categories (frozenset).
Substrate for Phase 3 'hold instead of refuse' on unknown words; Phase 1
introduces only the type.
- ProblemReadingState.open_hypotheses + unknown_held fields, both default
to () (empty tuple). Defaults preserve today's single-committed behavior
exactly. Confidence-rank uniqueness + density-from-0 enforced at
__post_init__ as structural invariants.
- Canonical-bytes serializer extended to handle frozenset (sorted list).
Phase 1 acceptance verified:
- 29/29 ADR-0174 Phase 1 tests pass (construction, validation, cap
enforcement, canonical-bytes determinism, frozenset stability).
- 42/42 existing reader tests pass (test_brief_11_audit +
test_reader_phase2) — default-empty fields preserve byte-identity.
- Smoke 67/67, packs 141/141.
- train_sample/v1 byte-identical across two runs with use_reader=True.
- wrong=0 invariant held: 3/47/0 unchanged.
No apply_word body changes. The 'thread the hypothesis set' requirement
at Phase 1 is satisfied by field defaults that propagate through every
ProblemReadingState construction site in lifecycle.py without code edits.
Phase 2 (continuous constraint propagation) and Phase 3 (lookback
re-evaluation) will populate these fields with real hypothesis data and
wire the EMIT / ELIMINATE / HOLD operators.
_unit_grounds() previously refused multi-word units like 'Pokemon cards'
even when both component words appeared as tokens in the source span.
The function checked unit_token against the haystack as a single key,
but the tokenizer splits source into per-word tokens — 'Pokemon cards'
was never going to match.
Fix is conjunctive by design: every component word must appear in the
haystack. A missing component refuses, preserving wrong=0.
Truth-test: case 0023 (Nicole/Pokemon cards) previously refused with
'recognizer matched but produced no injection' on its first sentence.
After this fix, sentence 1 passes injection cleanly; the case now
refuses on sentence 2 (Cindy/Rex compositional clause) — a more
honest refusal reason that reflects the actual remaining gap.
Score unchanged at 3/47/0 (no overall lift; correctness win).
smoke 67/67, packs 141/141, lanes 8/8 all green.
Both INV-02/INV-21/INV-24 scan functions walked into .claude/worktrees/
and found vault recall/write callsites in the stale
step-3-submission-invariants checkout, producing 3 false failures.
Fix: add '.claude' to the os.walk exclusion set (INV-02) and to
EXCLUDED_DIRS (INV-21/INV-24). Defensive against any future worktree
that agents create under .claude/worktrees/.
Also pruned 58 stale worktree git-dir entries via git worktree prune
and removed the step-3-submission-invariants worktree directory.
Smoke suite: 67/67 passed.
C1: delete generate/math_versor_arithmetic.py and its 3 tests (ADR-0139
add-only arithmetic spike; no runtime consumers, no pipeline wiring,
follow-on lift paused per module docstring).
C3: gitignore engine_state runtime artifacts (manifest.json,
recognizers.jsonl, discovery_candidates.jsonl). Module code
(engine_state/__init__.py) remains tracked; generated checkpoint
files should not be.
C5: document reader zero-delta root cause in train_sample/v1/README.md.
Both Phase 2 (whole-problem) and Phase 1 (question-only) reader paths are
called but inert because all 47 refusals are statement-level NO_INJECTOR
gaps, not question-sentence gaps. Reader unblocks when injector coverage
expands (C2 work). report.json use_reader flag corrected to reflect last run.
C6: add deprecation header to generate/math_parser.py pointing at
generate.math_candidate_graph.parse_and_solve as the live path.
C2/C4 briefs: docs/handoff/CLEANUP-C2-run-lane-migration.md and
docs/handoff/CLEANUP-C4-compositions-compile.md added as operator
dispatch docs for the medium-scope wiring tasks.
Three review fixes:
1. Security: validate lane/split/version against ^[a-z0-9_]+$ before
building the runner module name. The runner_args list is passed to
subprocess.run without shell=True (no shell injection possible),
but defense-in-depth blocks arbitrary token characters from
reaching Python's -m module loader. Bad input now errors at the
CLI boundary with a clear message.
2. Bug-risk: _classify_refusal docstring referenced a
no_admissible_candidate bucket that the implementation never
emitted. Aligned docstring with actual buckets
(no_admissible_question / no_admissible_statement). Also made all
matching consistently case-insensitive (was mixed — some checks
used raw reason, one used .lower()).
3. Bug-risk: fetch_committed_baseline wrote to
.git/coverage_baseline_tmp.json. Replaced with tempfile.mkstemp in
the system temp dir — avoids (a) failures in non-git worktrees
where .git is a file pointer, (b) concurrent-access collisions
between simultaneous operators.
Tests (+3 new):
- test_classify_refusal_is_case_insensitive
- test_classify_docstring_matches_implementation_buckets
- test_fetch_committed_baseline_uses_system_temp
All 16 coverage tests green. Verified the validation:
core teaching coverage --lane 'evil; rm -rf /'
→ ERROR: lane='evil; rm -rf /' must match ^[a-z0-9_]+$
Brief D from PR #407. Closes the "flying blind on per-shape coverage"
gap identified in RAT-1's audit (finding 6).
After this PR, every operator can run a single command to see exactly
which refusal modes their work moved (or didn't), without re-eyeballing
report.json by hand.
Modules
-------
- teaching/coverage.py — pure aggregator:
- _classify_refusal — maps each per-case refusal reason to a
stable bucket (recognizer_empty_injection(<ShapeCategory>),
no_admissible_question, no_admissible_statement,
unexpected_question_count, other)
- build_coverage_report — reads a lane's report.json + emits a
CoverageReport with counts, refusal_taxonomy (sorted by count
desc), case_0050_verdict, optional delta vs baseline
- fetch_committed_baseline — uses `git show HEAD:<relpath>` to
pull the baseline report.json for delta computation
- core/cli.py:
- cmd_teaching_coverage — formats the report for terminal output
- core teaching coverage [--lane gsm8k_math] [--split train_sample]
[--version v1] [--use-reader] [--run] [--delta] [--json]
CLI output example
------------------
Lane: gsm8k_math/train_sample/v1 (use_reader=True)
Counts: correct=3 refused=47 wrong=0
Refusal taxonomy:
21 recognizer_empty_injection(discrete_count_statement)
6 no_admissible_statement
5 recognizer_empty_injection(multiplicative_aggregation)
4 no_admissible_question
4 recognizer_empty_injection(currency_amount)
3 recognizer_empty_injection(rate_with_currency)
2 recognizer_empty_injection(descriptive_setup_no_quantity)
2 recognizer_empty_injection(temporal_aggregation)
Wrong=0: ✓
Case 0050 hazard pin: refused ✓
Tests (13 new)
--------------
tests/test_teaching_coverage_cli.py — classification narrowness,
counts aggregation, case 0050 verdict capture, delta computation,
missing-baseline path, missing-report error, taxonomy sort order,
wrong=0 invariant visibility via as_dict.
Suite results
-------------
core test --suite teaching -q → 106 passed (93 → +13)
core test --suite runtime -q → 20 passed
core test --suite packs -q → 127 passed
core eval gsm8k_math --split public → 150/150, wrong=0
Note on Brief E (lexical auto-compile): the audit was WRONG. The
lexicon loader (generate/comprehension/lexicon.py::load_lexicon)
reads from the per-category source files directly; the compiled
lexicon.jsonl is only a manifest-checksum pin, not the source of
truth at runtime. apply_lexical_claim() writes a new entry → next
turn the loader sees it. Brief E is a non-issue; closing without a
code PR.
Verified by direct test: stage a clone of the math pack, write a
synthetic lemma to drain_token.jsonl, clear the lexicon cache, load
again → new entry present. So 3 of the 5 audit gaps closed (A, D,
E-as-correction); B and C remain as the next operator dispatch
targets.
Independent of PR #406 (RAT-1) and PR #408 (WAVE-A). Based on main.
Addresses 5 of 47 train_sample "recognizer matched but produced no
injection" refusals (the largest single failure-mode bucket
identified in RAT-1's audit).
Modules
-------
- generate/recognizer_match.py:
- _MULT_AGG_EACH_WEIGHING_RE — regex for "<Subject> <bake-verb>
<M> <outer-noun>, each <weigh-verb>ing <N> <unit>" pattern
- _try_extract_each_weighing_anchor — extracts M, N, subject,
inner unit; emits pre-composed CandidateInitial(value=M*N) with
composition_evidence so RAT-1's _composed_initial_admissible
gate verifies INPUT tokens ground (preserves wrong=0)
- _match_multiplicative_aggregation dispatches to the value
extractor when spec carries extract_values=True; specs without
that flag get the existing detection-only return path
(byte-identical legacy behavior)
- generate/recognizer_anchor_inject.py:
- inject_multiplicative_aggregation — new per-category injector;
narrow by anchor.kind so ME-3/ME-4 additive/subtractive anchors
(which share the same matcher entry point) continue to flow
through composition_registry consult instead of WAVE-A's direct
path
- registered in _INJECTORS dict (2nd entry after DCS)
- core/cli.py:
- seed-recognizer CLI gains --extract-values flag to opt the
canonical_pattern into the value-extracting matcher path
Seeded artifacts
----------------
- proposals.jsonl: rat1-seed-4dc30608fb783bc7 — multiplicative_
aggregation recognizer with anchor_kind=multiplicative_aggregate,
extract_values=True, observed_units covering ounces/strawberries/
questions/etc.
Live result on train_sample
---------------------------
- wrong == 0 preserved (3/47/0 baseline)
- Case 0050 hazard pin held
- public 150/150 preserved
- packs suite: 127 → 131 (+4 new WAVE-A tests, all green)
- teaching suite 93 unchanged
- runtime suite 20 unchanged
End-to-end synthetic solve (FIRST WAVE-A admission):
"Lilibeth fills 6 baskets where each basket holds 50 strawberries.
How many strawberries does Lilibeth have?" → answer=300
Cases that moved (statement now admits; refusal shifted downstream):
- Case 0025 (Lilibeth): statement admits via WAVE-A; refusal moved
to question parser ("If three of Lilibeth's friends pick the same
amount, how many strawberries do Lilibeth and her friends pick in
all?")
- Case 0047 (John bakes 12 macaroons): statement 1 admits; refusal
moved to statement 2
Eval correct count unchanged because the QUESTION parser (and
multi-statement cross-sentence reasoning) is the next bottleneck.
RAT-1's audit identified that gap; WAVE-A closes the injector half.
The remaining 3 multiplicative_aggregation refusals (0006, 0013,
0045) have different shape patterns the WAVE-A regex does not yet
cover; they're follow-up matcher extensions in the same architecture.
Tests
-----
- tests/test_wave_a_multiplicative_aggregation_injector.py (10
tests): each-weighing + each-basket-holds admission shapes,
detection-only path preserved when extract_values absent,
unobserved unit / pronoun / zero count refusals, end-to-end
inject_from_match dispatch, the Lilibeth canary solve,
wrong=0 preserved, case 0050 hazard pin
Stacks on PR #406 (RAT-1).
Adds surface_pattern, composition_category, and polarity to the
proposed_change_payload for composition_reclassification proposals so
operators can call apply_composition_claim() without field synthesis.
Dispatch by missing_operator:
- quantity_extraction → multiplicative_composition + bound(count) × bound(unit_cost)
- multi_quantity_composition → additive_composition + bound(qty_a) + bound(qty_b)
All other change kinds (matcher_extension, injector_sub_shape,
frame_reclassification) keep the existing evidence-aggregation payload.
Legacy fields (evidence_count, group_key, modal_sub_type) preserved.
Adds tests/test_contemplation_ratifiable_payload.py with 11 tests
including a round-trip from decompose_audit → apply_composition_claim.
The user's question — "shouldn't we be running it multiple times so
it can learn? or is that part broken?" — exposed that the math
teaching loop's `ratify → admit` closure had been structurally
broken at the connector between operator ratification and runtime
visibility. The handlers wrote source files (compositions/, frames/)
that the runtime loader never read because no compile step
regenerated the runtime artifacts.
This PR fixes the gap end-to-end AND fires the first live composition
admission on the canonical pack.
Modules
-------
- language_packs/compile_pack.py — unified compile step that
regenerates frames.jsonl + compositions.jsonl + updates
manifest.{frame,composition}_checksum atomically. Idempotent.
- teaching/math_composition_ratification.py — apply_composition_claim
now calls compile_pack at end of successful ratification. Closes
the source-file→runtime-artifact gap.
- teaching/math_frame_ratification.py — same auto-compile wire for
apply_frame_claim.
- generate/math_candidate_parser.py — CandidateInitial gains optional
composition_evidence Mapping field. When populated, signals the
candidate was produced by a registry-gated composition (ADR-0169);
the value/unit/entity are DERIVED arithmetic over grounded inputs.
- generate/math_candidate_graph.py — new _composed_initial_admissible
predicate that branches on composition_evidence. Wrong=0 preserved
by requiring each composition INPUT token (count, amount) to ground
in source_span literally; the derived value is admitted because the
arithmetic over grounded inputs is deterministic.
- generate/math_candidate_graph.py — discourse-level prior_subject
tracking: capture proper-noun subjects from ALL statement sentences
(including ADR-0136.S.0 context-filler sentences that get filtered
out before the candidate loop). Without this, "John adopts a dog"
(no numbers) is dropped and the cross-sentence subject resolver for
case 0019 sees prior_subject=None.
- generate/recognizer_match.py — all four composition matchers
(ME-1 currency-per-unit same-sentence, ME-2 cross-sentence, ME-3
additive, ME-4 subtractive) now populate composition_evidence in
CandidateInitial. Also added standalone " each " / " apiece " to
_PER_UNIT_TOKENS so currency_amount detection-only matcher refuses
per-item costs instead of swallowing them.
CLIs
----
- core teaching compile-pack — explicit operator surface for
regenerating runtime artifacts. JSON output for CI integration.
- core teaching seed-recognizer — operator surface for seeding a
RatifiedRecognizer entry in the proposal log for a given
(shape_category, anchor_kind). Writes created + transition(accepted)
events directly via ProposalLog._append.
Seeded artifacts (the actual loop closure)
------------------------------------------
- proposals.jsonl: new rat1-seed-48dd2673d6ad673d RatifiedRecognizer
entry for shape_category=rate_with_currency,
anchor_kind=currency_per_unit_composition.
- compositions/multiplicative_composition.jsonl: ratified
"bound(count) × bound(unit_cost)" affirms entry sourced from
case 0019 evidence.
- compositions.jsonl + manifest.composition_checksum: compiled
runtime artifact + manifest pin (RAT-1 auto-compile).
Live result on train_sample
---------------------------
- wrong == 0 preserved (3 correct / 47 refused / 0 wrong)
- Case 0050 hazard pin holds (refused)
- public split 150/150 preserved
- Case 0019 sentence 1 ("requires 3 vet appointments, which cost
$400 each") NOW ADMITS via composition. Previously refused with
"recognizer matched but produced no injection". The refusal moved
downstream to sentence 2 (a different currency_amount detection
bottleneck that is its own follow-up).
This is the first time a composition ratification on the canonical
pack actually reaches the runtime. The flywheel turned one
revolution.
Tests
-----
- tests/test_rat1_end_to_end_admission.py — 4 new live tests:
composition statement admits on isolated synthetic problem, case
0019 cross-sentence admission, wrong=0 preserved on train_sample,
case 0050 hazard pin.
- tests/test_consumption_empty_registry_no_op.py — refactored to use
isolated synthetic packs (the canonical pack may now carry ratified
entries).
- tests/test_math_{frame,composition}_ratification.py — updated
"manifest checksum unchanged" tests to "lexicon checksum
preserved" semantics: RAT-1 auto-compile may add the new optional
checksum fields; pre-existing lexicon checksum stays untouched.
Suite results: teaching 93, packs 131 (+4), runtime 20. All green.
Final PR of the matcher-extension wave. Ships:
1. tests/test_me5_all_categories_integration.py — 4 new tests:
- test_all_three_canaries_admit_through_full_pipeline: stages a
pack with all three SAFE_COMPOSITION_CATEGORIES entries +
ratifies, runs Maria/Sam/Tom canaries through matcher →
inject_from_match, asserts admission for all three
- test_partial_pack_only_admits_present_categories: refusal-
preferring when only one category is ratified
- test_all_safe_categories_have_extension_admission: pins that
SAFE_COMPOSITION_CATEGORIES is exactly the three covered
categories (breaks if future ADR widens without matcher)
- test_falsifies_uniformly_suppresses_across_categories:
polarity discipline holds across all three matchers
2. docs/handoff/ME1-ME5-MILESTONE.md — wave milestone doc:
- architecture diagram (audit → ratify → compile → load →
match → consult → admit)
- SAFE_COMPOSITION_CATEGORIES coverage matrix
- invariants preserved across the entire stack
- scope boundary (what does NOT fire yet — RAT-1 follow-up)
- recommended next dispatch
3. Test registration in core/cli.py packs suite.
Across the full ME-1..ME-5 stack:
- 5 stacked PRs (#400/#401/#402/#403/#404)
- 1 foundation PR (#398 — consumption wiring)
- 114 new tests, all green
- packs suite 127 passed
- core eval gsm8k_math --split public → 150/150, wrong=0
- All three SAFE_COMPOSITION_CATEGORIES have matcher extensions
Anti-regression invariants preserved across the entire stack:
- wrong == 0 on public split
- Case 0050 hazard pin (parametrized over all three categories)
- ADR-0166 — no new eval lanes
- ADR-0167 partition — no cognition imports
- ADR-0169 mutation boundary — registry is a gate, not arithmetic
- All matcher detection paths byte-identical
- engine_state/* never committed
- SAFE_COMPOSITION_CATEGORIES enforced at write AND load
- polarity falsifies honored uniformly
Live train_sample admission requires operator-seeded ratifications
(RAT-1 follow-up). Wiring is end-to-end correct, verified by ME-5
integration tests.
Memory: milestone-me1-me5-matcher-extensions-complete saved.
Stacks on PR #403 (base: feat/matcher-extension-subtractive).
Extends _match_multiplicative_aggregation with a new branch keyed on
anchor_kind="additive_quantity_composition". When a statement carries
"<Subject> <verb> <N> <unit> and <M> <unit>" (same unit) shape, emits
a pre-composed CandidateInitial(N+M, unit) and publishes
composition_shape="bound(qty_a) + bound(qty_b)".
Subject binding under Option A (refuse on pronoun / determiner / no
proper-noun head). Cross-sentence subject support (mirroring ME-2)
is deferred — not needed for the v1 ME-3 canaries.
Verb whitelist: lost / gained / earned / saved / made / paid / spent /
bought / sold / added / removed / received. Verbs that route through
CandidateInitial.matched_anchor's existing post-init whitelist;
unmapped verbs fall back to "had".
Unit normalization: rstrip 's' for plural matching (pounds vs pound).
Cross-unit composition refused — no conversion table in v1.
Tests (15 new, all green):
- same-unit admission with sum
- pronoun subject refuses
- determiner subject refuses
- cross-unit refuses
- unobserved unit refuses
- zero count refuses
- plural normalization
- unknown verb refuses
- multiplicative_aggregate detection path unaffected
- wrong anchor_kind refuses
- anchor audit fields complete
- source_span substring invariant
- no match returns None
- end-to-end admission via composition_registry
- end-to-end falsifies suppresses
Registered in core/cli.py "packs" suite. core test --suite packs -q →
106 passed (91 existing + 15 new).
Anti-regression invariants preserved:
- wrong == 0 on gsm8k_math public 150/150
- Case 0050 hazard pin holds
- ADR-0166 — no new eval lanes
- ADR-0167 partition — no cognition imports
- Original multiplicative_aggregate detection path byte-identical
- ME-1 currency-per-unit path unaffected
- ME-2 cross-sentence path unaffected
- engine_state/* not committed
Live train_sample admission requires the same operator workflow as
ME-2: a RatifiedRecognizer for the new anchor_kind + composition_registry
entry for "bound(qty_a) + bound(qty_b)" under additive_composition.
Without those, the wiring is correctly positioned but dormant — no
regression in the live eval.
Stacks on PR #401 (base: feat/matcher-extension-cross-sentence-subject).
Admits case 0019's composition sentence via prior_subject resolved
from upstream sentences. Stacks on PR #400 (ME-1).
Modules
-------
- generate/recognizer_match.py:
- _CROSS_SENTENCE_COMPOSITION_RE — regex for "requires N noun, which
cost(s) $X each" (no subject prefix)
- try_extract_cross_sentence_composition_anchor(statement, spec,
prior_subject) — refuses on None / empty / pronoun prior_subject;
publishes the same composition_shape + composed_initial payload as
ME-1, sourced via prior_subject
- extract_proper_noun_subject(statement) — head proper-noun extractor
used by callers to track running prior_subject; rejects determiners,
sentence-initial connectors (After/How/Every/...), and pronouns
- match() dispatcher gains keyword-only prior_subject parameter;
when a per-category matcher returns None for a RATE_WITH_CURRENCY
recognizer with currency_per_unit_composition anchor_kind AND
prior_subject is supplied, the cross-sentence helper is tried as
a fallback
- generate/math_candidate_graph.py:
- tracks _prior_subject across statement_sentences iteration
- passes prior_subject to recognizer_match.match()
- updates _prior_subject from each sentence's head proper-noun
Tests (19 new, all green)
-------------------------
- test_me2_cross_sentence_subject.py (15 tests)
- subject extraction narrowness (proper noun / determiner / connector
/ pronoun / non-string)
- cross-sentence helper happy path + refusals (None, empty, pronoun,
unobserved currency / per_unit, wrong anchor_kind, zero count,
multi-match)
- source_span substring invariant
- kind label "currency_per_unit_composition_cross_sentence"
- test_me2_case_0019_admits.py (4 tests)
- case_0019_admits_with_prior_subject_john — the truth test
- case_0019_refuses_without_prior_subject — ME-1 Option A still holds
- case_0019_refuses_with_pronoun_prior — refusal-preferring
- maria_same_sentence_unaffected_by_prior_subject — ME-1 path intact
Registered in core/cli.py "packs" suite.
Suite results
-------------
core test --suite packs -q → 91 passed (existing + ME-1's 21 + 19 new)
core test --suite runtime -q → 20 passed
core eval gsm8k_math --split public → 150/150, wrong=0
Scope boundary
--------------
The wiring is load-bearing AND tested end-to-end via synthetic
recognizer registry (test_case_0019_admits_with_prior_subject_john
proves the full chain match → inject → admit).
For the LIVE train_sample case 0019 admission, two ratifications must
also be seeded (operator workflow outside this PR's code scope):
1. A RatifiedRecognizer in the proposal log with shape_category=
RATE_WITH_CURRENCY and canonical_pattern carrying
anchor_kind="currency_per_unit_composition"
2. A composition_registry entry for "bound(count) × bound(unit_cost)"
under multiplicative_composition with polarity=affirms
With both ratifications in place, case 0019 admits via the wiring
this PR ships. Without them, the live train_sample run remains at
the 3/47 baseline (preserved; no regression).
Anti-regression invariants preserved
------------------------------------
- wrong == 0 on gsm8k_math public
- Case 0050 hazard pin holds (no _COMPOSITION_SUBJECT_BUY_RE or
_CROSS_SENTENCE_COMPOSITION_RE match on case 0050's sentences)
- ADR-0166 — no new eval lanes
- ADR-0167 partition — no cognition imports
- ME-1 Maria same-sentence path byte-identical (test pins)
- Existing currency_per_unit_rate path unaffected (test pins)
- prior_subject is keyword-only on match() (additive; old callers
unaffected)
- engine_state/* not committed
Stacks on PR #400 (base: feat/matcher-extension-currency-per-unit-composition).
Closes the consumption-half of the math teaching loop for two of three
sub-types per docs/handoff/CONSUMPTION-WIRING-DISPATCH-PACK.md (PR #397).
Companion to the doctrinal brief in PR #396.
Modules
-------
- language_packs/compile_frames.py — byte-deterministic compile of
frames/*.jsonl → frames.jsonl (sorted by (frame_category, surface_form))
- language_packs/compile_compositions.py — same shape for
compositions/*.jsonl → compositions.jsonl
- generate/comprehension/frame_registry.py — load_frame_registry()
mirroring load_lexicon: cache by (path, mtime, sha256), manifest
checksum verification (optional frame_checksum field), polarity
validation, conflict detection, empty-registry no-op
- generate/comprehension/composition_registry.py — same shape PLUS:
* SAFE_COMPOSITION_CATEGORIES enforced at LOAD (defense in depth;
raises WrongCompositionCategory on any unsafe category — protects
against pack edits that bypass the handler)
* polarity "falsifies" exposed via is_falsified() (consumer must
suppress; not silently treated as affirms)
- language_packs/compiler.py — manifest verification extended for
frame_checksum + composition_checksum, mirroring the proven
glosses_checksum pattern (optional fields; backward-compatible)
- generate/recognizer_anchor_inject.py — inject_from_match consults
composition_registry when the per-category injector returns empty
AND the matcher publishes ``composition_shape`` in parsed_anchors.
Registry is a gate (admissibility) not an arithmetic primitive
(ADR-0169 §"Mutation boundary").
Tests (38 new, all green)
-------------------------
tests/test_frame_registry_load.py (11 tests)
tests/test_composition_registry_load.py (11 tests)
tests/test_composition_consult_in_injector.py ( 6 tests)
tests/test_consumption_case_0050_hazard_pin.py( 3 tests, parametrized
over allowlist)
tests/test_consumption_empty_registry_no_op.py( 4 tests)
tests/test_consumption_partition.py ( 3 tests)
Registered in core/cli.py "packs" suite.
Suite results
-------------
core test --suite teaching -q → 93 passed
core test --suite runtime -q → 20 passed
core test --suite packs -q → 51 passed
core eval gsm8k_math --split public → 150/150, wrong=0
Truth-test rows (6-row binding table in dispatch pack):
#1 Case 0019 admits ............. PARTIAL — see Scope Boundary below
#2 Case 0050 stays refused ....... PASS
#3 train_sample 3/47 → ≥4/46 ..... PARTIAL — same as #1#4 wrong == 0 preserved .......... PASS
#5 public split 150/150 .......... PASS
#6 Empty-registry no-op .......... PASS
Scope Boundary (honest finding)
-------------------------------
Rows #1 and #3 (case 0019 admission) require a matcher extension that
publishes ``composition_shape`` + a pre-composed CandidateInitial in
parsed_anchors. The existing currency_amount / multiplicative_aggregation
matchers in generate/recognizer_match.py are detection-only (return
empty parsed_anchors). This PR ships the consumption infrastructure
correctly but the runtime path remains dormant until a follow-up PR
extends the matcher. The dispatch pack's truth test #1/#3 cannot fire
without that extension.
The wiring is positioned correctly: inject_from_match → consult
composition_registry → admit on affirms-with-payload, suppress on
falsifies, refuse on absence. A synthetic recognizer match with
populated composition_shape + composed_initial DOES admit through the
new path (covered by 6 tests in test_composition_consult_in_injector.py).
A follow-up brief naming the matcher-extension work is the
recommended next step.
Anti-regression invariants verified
-----------------------------------
- wrong == 0 on core eval gsm8k_math (public 150/150)
- case 0050 stays refused (parametrized over allowlist categories)
- ADR-0166 — no new eval lanes
- ADR-0167 partition — no cognition imports in any new module
- Empty-registry runtime byte-identical to today (no-op test)
- SAFE_COMPOSITION_CATEGORIES enforced at write AND load
- polarity semantics (affirms vs falsifies) honored
- engine_state/* never committed
Bundles three post-Tier-1 follow-ups into one PR (no scope change, no
new ADR — implementation tightening on the already-shipped corridor).
(1) Standalone JSONL self-containment
teaching/math_contemplation_proposal.py
+ to_jsonl_record() — emits proposal_id + full evidence_pointers
(nested dicts including audit_row) + full reasoning_trace.steps
+ from_jsonl_record() — inverse; goes through build_proposal()
so all invariants are re-validated; raises on proposal_id mismatch
canonical_bytes() UNCHANGED (still the content-hash function;
trace_id/proposal_id stability preserved)
core/cli.py W3 lane now writes to_jsonl_record() output instead of
canonical_bytes() — same compact-JSON encoding (sort_keys=True,
ensure_ascii=False, separators=(",", ":"))
workbench/readers.py loads via self-contained record fields directly;
decompose_audit() re-run removed. read_math_proposal() now reads
reasoning_trace.steps and evidence_pointers from the JSONL record.
(2) Widened change_kind heuristic dispatch
teaching/math_contemplation.py
+ _CHANGE_KIND_BY_PAIR table on (refusal_reason, missing_operator):
(unexpected_category, pre_frame_filler_sentence) → matcher_extension
(unexpected_category, multi_subject_sentence) → frame_reclassification
(unexpected_category, fraction_percentage_literal) → matcher_extension
(unexpected_category, descriptive_frame_question) → frame_reclassification
(unresolved_pronoun, pronoun_resolution) → matcher_extension
Single-key fallback (lexicon_entry/narrowness_violation/
frame_unrecognized) retained for completeness.
hypothesis-step justification text updated to reflect new table.
Result on audit_brief_11.json:
3 matcher_extension (was 0)
2 frame_reclassification (was 0)
3 injector_sub_shape (was 8)
0 vocabulary_addition (no unknown_word group ≥2 in train sample)
(3) shape_category structural gap
MathReaderRefusalEvidence does not carry shape_category, so the
proposal cannot derive it. All proposals continue to emit
ShapeCategory.UNCATEGORIZED with a structural-gap comment. No
invented values — handler dispatch decision (per ADR-0167-FOLLOWUPS
§1) drives ratification routing today, not shape_category.
Tests
+ W1: 5 new tests (to_jsonl_record self-containment, round-trip,
byte stability, proposal_id mismatch rejection, canonical_bytes
unchanged invariant)
+ W2: 3 new pair-dispatch tests + real-audit change_kind distribution
test + shape_category-uncategorized test
+ W3: 2 new tests (records are self-contained, round-trip via
from_jsonl_record); existing byte-comparison test updated to use
proposal_id ordering instead of canonical_bytes
+ W4: existing 6 tests updated to build JSONL via to_jsonl_record;
+ 1 new decoupling test that drops teaching.math_contemplation from
sys.modules and verifies the workbench still loads + serves detail
Verification
- core eval math-contemplation produces the expected 3/2/3 distribution
- core test --suite teaching -q → 33 passed
- core test --suite runtime -q → 20 passed
- All 57 ADR-0172 W1-W4 tests pass (49 existing + 8 new)
Determinism / invariants preserved
- canonical_bytes() byte-stable (test pins this)
- to_jsonl_record() byte-stable via sort_keys=True + no floats
- wrong=0 invariant: proposals stay evidence-only; no auto-apply
- ChangeKind Literal unchanged (4 values; no new ones invented)
Wires teaching/math_proposals/proposals.jsonl into the CORE Workbench
API (ADR-0160) alongside the existing cognition proposal queue:
workbench/schemas.py
- MathReasoningStep, MathProposalSummary, MathProposalDetail,
MathRatifyResult schemas
workbench/readers.py
- MATH_PROPOSALS_JSONL + _DEFAULT_MATH_AUDIT_PATH constants
- teaching/math_proposals added to ALLOWED_ARTIFACT_ROOTS
- _HANDLER_DISPATCH table (vocabulary_addition→LexicalClaim; all
others not yet implemented)
- list_math_proposals(), read_math_proposal(), ratify_math_proposal()
- read_math_proposal() re-runs decompose_audit() to recover full
4-step reasoning trace (canonical_bytes only carries trace_id)
- ratify_math_proposal() raises NotImplementedError with clear
"handler not yet implemented: {change_kind}" for unhandled kinds
workbench/api.py
- GET /math-proposals, GET /math-proposals/{id}
- POST /math-proposals/{id}/ratify → _math_ratify()
(vocabulary_addition→200/routed; unhandled→501 with loud message)
tests/test_adr_0172_w4_workbench_e2e.py — 6 tests:
1. loads from JSONL
2. renders domain:math badge (distinct from cognition /proposals)
3. ratify-vocabulary_addition routes to LexicalClaim (200)
4. ratify-matcher_extension fails loudly (501 "handler not yet
implemented")
5. all 4 trace steps visible in detail response
6. no cross-contamination between math and cognition queues
teaching + runtime suites green (28 + 20 passed).
Brief-gap note: canonical_bytes() excludes proposal_id and serialises
evidence pointers as hashes only. D1 loader derives proposal_id via
sha256(line_bytes) and re-runs decompose_audit() to recover full trace
for read_math_proposal(). This works but means the JSONL cannot be
loaded without the original audit file. If a future wave needs
standalone JSONL loading, C1 should emit a richer format.
Add decompose_audit(audit_path) to teaching/math_contemplation.py.
Groups audit_brief_11.json refusal rows by
(refusal_reason, missing_operator), emits one
MathReaderRefusalShapeProposal per group of >=2 rows, each carrying a
4-step ReasoningTrace (observation -> grouping -> hypothesis ->
conclusion).
Determinism:
- Group iteration sorted by (refusal_reason, missing_operator).
- Evidence per group sorted by case_id.
- Output tuple sorted by proposal_id.
- 10x rerun -> byte-identical proposals + trace_ids.
Pure read-only: audit file is not mutated, no proposals written to
disk, no chat/field/generate/algebra imports.
Tests (tests/test_adr_0172_w2_decomposer.py): real-audit emission,
determinism (10x), evidence floor, change-kind dispatch over all four
heuristic branches, four-step trace, case_id sort, proposal_id sort,
empty input -> empty tuple, unmapped operator skip, missing file ->
FileNotFoundError, no-mutation contract.
Added to core test --suite teaching.
New module `teaching/math_contemplation_proposal.py` defines the
`MathReaderRefusalShapeProposal` dataclass — the math-domain analog of
`TeachingChainProposal` for the Tier-1 contemplation corridor.
- `build_proposal` enforces all seven invariants: math domain, ShapeCategory
enum membership, ≥2 evidence pointers, valid ChangeKind Literal, JSON-
serializable payload, ≥40-char wrong_zero_assertion, and non-None
reasoning_trace with a non-empty trace_id.
- `canonical_bytes` / `compute_proposal_id` produce stable sha256-based IDs;
evidence reduced to evidence_hash, trace to trace_id for stability.
- `ReasoningTrace` imported under TYPE_CHECKING only (W0/A1 not yet merged);
duck-typed at runtime via trace_id attribute.
- 16 tests cover all eight brief obligations plus freeze and sensitivity checks.
- `core test --suite teaching -q` green (17 passed).
Schema-only module defining ReasoningStep / ReasoningTrace with
byte-identical canonical serialization and sha256 trace_id derivation.
Replay-equivalence is enforced by:
- sorted-key JSON, no whitespace, ensure_ascii=False, allow_nan=False
- recursive rejection of float values in payloads (replay hazard)
- step_index monotonicity from 0
- empty trace rejected
- Literal-checked step_kind across all eight Tier 1+2 kinds
No runtime hook. No import from chat/field/generate/algebra.
Downstream (W1 ShapeProposal, W2 decomposer) consume this schema.
Tests: 12 new, full teaching suite green (17 passed).
Second implementation PR of the ADR-0170 wave. Extends the DCS injector
to emit ``CandidateOperation(kind='add')`` for acquisition verbs
alongside the existing ``CandidateInitial`` emission for possession
verbs. Proves the W1 type-widening with real emission of both union
members.
## What changes
### `generate/recognizer_match.py`
- New `_ACQUISITION_VERBS` frozenset (12 verbs: collect/get/receive/buy
inflections). Each member is a subset of `ADD_VERBS` so the downstream
CandidateOperation post-init whitelist accepts the matched_verb token.
- Extractor now accepts either possession OR acquisition verbs and
records `anchor_kind` (`"possession"` | `"acquisition"`) plus
`verb_token` in the parsed anchor schema.
### `generate/recognizer_anchor_inject.py`
- `inject_discrete_count_statement` dispatches on `anchor_kind`:
- `"possession"` → `CandidateInitial` (existing behavior unchanged)
- `"acquisition"` → `CandidateOperation(add)` (new)
- New helper `_build_operation_from_discrete_count_acquisition`
constructs the operation. Operand uses `_resolve_count_value`;
matched_verb uses `_locate_token` for round-trip ground check.
- Return type uses `InjectorEmission` from W1.
### Tests
- `tests/test_adr_0170_w2_dcs_acquisition_verbs.py` (new) — 22 tests:
- Verb-set membership pins
- Acquisition ⊂ ADD_VERBS sanity check
- Possession + Acquisition disjoint
- Extractor records anchor_kind correctly
- Injector emits CandidateOperation for acquisition verbs
- Possession path still emits CandidateInitial unchanged
- Deliberate exclusions (gained / donated / saved) still refuse
- Case 0050 hazard pinned (does/contemplates not in either set)
- Determinism + roundtrip_admissible passes
- Updated `tests/test_adr_0163_d2_discrete_count_injection.py` to
reflect new anchor schema fields (anchor_kind, verb_token).
- Updated `tests/test_adr_0170_w1_injector_type_widening.py` —
the DCS injector now legitimately returns
`tuple[InjectorEmission, ...]` (not narrower).
## Deliberate exclusions
These verbs are NOT in `_ACQUISITION_VERBS` and the extractor refuses
them — preserving wrong=0:
- `gained / gains / gain` — delta-of-attribute (weight, age), not
acquisition. Admitting as add-operation would risk wrong>0 on
questions that ask total state.
- `donated / donates / donate` — SUBTRACT semantics (actor gives away).
- `saved / saves / save` — ambiguous (time vs money vs effort).
Widening this set is operator-reviewable per `feedback-wrong-zero-
hazard-case-0050` discipline.
## ADR-0131.G.1 branch-disagreement discipline preserved
The regex parser already emits `CandidateOperation(add)` for
acquisition verbs via `ADD_VERBS` for single-word units. The new DCS
injector path emits the same kind of operation for multi-word units
(where the regex parser fails). Collapsed-tie when both paths emit
identical operations on overlapping shapes; no disagreement.
## Test plan
- tests/test_adr_0170_w2_dcs_acquisition_verbs.py: 22 passed (new)
- tests/test_adr_0163_d2_discrete_count_injection.py: ~30 passed
(existing tests updated for new schema fields)
- tests/test_adr_0170_w1_injector_type_widening.py: 6 passed
- tests/test_recognizer_skip_wrong_zero.py + brief_11b + brief_11 +
candidate_graph_wiring + candidate_domain_partition: passed
- evals/gsm8k_math/train_sample/v1: counts=correct=3 refused=47 wrong=0
unchanged (case 0023 still has S2/S3 downstream blockers; W2's value
is infrastructure, not direct lift)
## Hard invariants
- `wrong == 0` preserved (case 0050 hazard pin + deliberate verb
exclusions + roundtrip_admissible gate)
- ADR-0166: no new eval lanes
- No teaching-store / pack mutation
- ADR-0131.G.1 branch-disagreement discipline preserved (acquisition →
operation, not initial)
- Five-layer wrong=0 safety net (ADR-0163.D.2) intact and extended
## W3 NOT in this PR — honest skip
Initial plan was to bundle W2 + W3 (A1 currency_amount injector).
Inspection of the 4 actual `currency_amount` GSM8K refusals showed
none match A1's narrow form (`<ProperNoun> earns|charges $<amount>`):
| Case | Statement | Reason narrow form doesn't fit |
|---|---|---|
| 0019 | "this requires 3 vet appointments, which cost $400 each" | anaphoric subject + multi-quantity |
| 0026 | "Aaron and his brother Carson each saved up $40" | multi-subject + "each" |
| 0028 | "It cost $100,000 to open initially" | pronoun subject |
| 0043 | "Her mother gave her an additional $4, and her father twice as much" | multi-clause + comparative + transfer |
Shipping W3 as-designed would have re-introduced the dead-code pattern
#373 just cleaned up. Skipped honestly; ADR-0172 Tier 1's decomposer
(the next wave) will surface category-shape mismatches like this
programmatically.
The G.2 test \`_comparative_clause_refusal_count\` reads \`report.json\`
and counts refusals whose reason quotes a statement clause containing
comparative anchors ("more/less than", "twice as many", etc.). After
#359's wrong=0 fix, the candidate-graph emits two refusal-reason
families that both quote a statement:
1. "no admissible candidate for statement: '...'" — parser-path
refusal (the comparative-parse-failure family this metric tracks).
2. "recognizer matched but produced no injection for statement:
'...'" — recognizer-path refusal; the quoted statement may
incidentally contain comparative anchors but the refusal cause is
the missing injector, NOT the comparative parse.
The pre-#359 counter only saw family (1) reasons; post-#359 it
over-counts whenever a recognizer-path refusal quotes a statement
containing comparative anchors. This was the test failure A2's PR
(#369) and the cleanup PR (#373) both surfaced.
## Fix
Filter the counter to exclude family (2) explicitly. Recognizer-path
refusals are tracked separately by the recognizer-wiring test suite;
they don't belong in the G.2 metric.
Result on current main:
- total statements with comparative anchors in refusal reasons: 2
- parser-path: 1 (case 0009, the legitimate G.2-tracked refusal)
- recognizer-path: 1 (filtered out — incidental anchor in #359-format reason)
- G.2 metric correctly reports 1 < baseline 2 → assertion passes
## Also: refresh report.json
The checked-in \`report.json\` was generated pre-#359 with the legacy
refusal-reason format. The runner now emits the new format on every
run; checking in the current output makes the baseline reproducible
and clears the CI friction that A2 originally flagged.
## Test plan
- tests/test_adr_0131_G2_comparatives.py: 25 passed (was 24 pass / 1 fail)
- tests/test_adr_0131_G4_multi_clause.py + G5_aggregate + S1_rate_events: 105 passed
- tests/test_brief_11b_audit_artifact + step2_lexicon + recognizer_skip + brief_11_audit + wiring + partition + adr_0163_d2: 89 passed
- Total: 219 passed
## Hard invariants
- No runtime change
- wrong=0 invariant preserved
- ADR-0166: no new eval lanes
- No teaching-store / pack mutation
First implementation PR of the ADR-0170 wave. Type-level widening only:
the recognizer-injector dispatch now returns
``tuple[InjectorEmission, ...]`` where
``InjectorEmission = CandidateInitial | CandidateOperation``.
The existing ``inject_discrete_count_statement`` continues to emit only
``CandidateInitial`` — the widening unlocks but does not exercise
operation emission. Subsequent W2-W5 PRs ship the per-injector emission
shapes:
- W2 — DCS-S1 acquisition verbs (CandidateOperation(add))
- W3 — A1 currency_amount (CandidateInitial reimplementation)
- W4 — A3 multiplicative_aggregation (CandidateInitial(product))
- W5 — A4 temporal_aggregation (deferred until apply_rate primitive)
## Changes
### `generate/recognizer_anchor_inject.py`
- New `InjectorEmission = Union[CandidateInitial, CandidateOperation]`
- `inject_from_match` return type widened to
`tuple[InjectorEmission, ...]`
- `__all__` exports `InjectorEmission`
- Documentation comment names ADR-0170 §"Implementation outline"
### `generate/math_candidate_graph.py` (admissibility dispatch)
The per-statement admission loop now dispatches admissibility on the
concrete candidate type:
if isinstance(c, CandidateInitial):
if _initial_admissible(c): admitted.append(c)
elif isinstance(c, CandidateOperation):
if roundtrip_admissible(c): admitted.append(c)
No new admission semantics — each type is gated by the predicate it was
already gated by elsewhere in the codebase. The dispatch unifies the
injector path with the parser path.
### `tests/test_adr_0170_w1_injector_type_widening.py` (new)
- Pin: `InjectorEmission` union members are exactly the two candidate types
- Pin: `inject_from_match` return type is widened
- Pin: `inject_discrete_count_statement` still emits CandidateInitial (W1
is type-level only)
- Hazard pin: case 0050 remains refused
- Hazard pin: unparseable-verb refusal path (#359) unchanged
- Anti-regression: canonical DCS narrow-form extraction still works
## Test plan
- tests/test_adr_0170_w1_injector_type_widening.py: 6 passed (new)
- tests/test_adr_0163_d2_discrete_count_injection.py: 21 passed
(existing D.2 v1 injector regression)
- tests/test_brief_11b_audit_artifact.py + step2_lexicon +
recognizer_skip_wrong_zero + brief_11_audit: 55 passed
- tests/test_candidate_graph_recognizer_wiring.py: 7 passed
- tests/test_candidate_domain_partition.py: 5 passed
- tests/test_adr_0131_G2_comparatives + G4 + G5 + S1_rate_events:
130 passed
- Total: 225 passed
- evals/gsm8k_math/train_sample/v1: counts=correct=3 refused=47 wrong=0
(unchanged; verified no behavioral regression)
## Hard invariants
- `wrong == 0` preserved (admissibility dispatch is type-aware but
semantically identical to the parser path's gating)
- ADR-0166: no new eval lanes
- No teaching-store / pack mutation
- Five-layer wrong=0 safety net (ADR-0163.D.2) intact
- Reader path unchanged
Three concrete cleanup items from the day's work, per the
cleanup-as-you-find memory principle.
## 1. Remove inject_rate_with_currency stub
PR #369 (A2 rate_with_currency) shipped a function that always returns
() with an extensive docstring documenting the Rate-not-in-SentenceChoice
schema gap. The function is dead at runtime — `_INJECTORS.get(category)`
returning None has the same downstream behavior as the function
returning (). The 16 tests pinned the empty-tuple return; the case-0050
hazard pin is duplicated in test_recognizer_skip_wrong_zero.py and
test_brief_11b_step2_lexicon.py.
The schema gap is now properly documented in ADR-0170 (PR #372). A
dispatch-table comment at the removal site retains the at-code pointer
to that ADR for anyone wiring a new injector.
Removed:
- `inject_rate_with_currency` function in generate/recognizer_anchor_inject.py
- Its `_INJECTORS` dispatch table entry
- Its `__all__` export
- tests/test_injector_rate_with_currency.py (371 lines, 16 tests)
## 2. Remove docs/handoff/GPT55-MOBILE-DISPATCH.md
Single-session travel-time scaffolding. The 5 tasks it named are
complete or superseded by ADR-0170's findings. Pure historical artifact.
## 3. Remove docs/handoff/WAVE-NEXT-INJECTORS.md
Superseded by docs/handoff/WAVE-NEXT-REVISED.md, which captures
everything load-bearing from the original brief in its A1–A4 findings
table. The "kept for history" justification didn't survive scrutiny:
the document was misframed (over-promised lift; misframed schema work
as injector work). Lessons captured in REVISED + ADR-0170.
Updated cross-references:
- WAVE-NEXT-REVISED.md: removed the "supersedes ... kept for history"
pointer; tightened cross-reference list
- ADR-0167-FOLLOWUPS.md §7: rewrote pointer to name ADR-0170 + REVISED
as the live plan rather than "the original is retained"
## Test plan
- 219 tests passed across G.2/G.4/G.5/S1/Brief 11/B1/B11A/wiring/partition/DCS-D.2
- evals/gsm8k_math/train_sample/v1/report.json untouched (regen
surfaces a separate stale-baseline test issue — out of cleanup scope)
- No runtime behavior change
## Net impact
- 5 files removed (~1200 lines)
- 1 file modified for explanatory comment (~30 lines)
- 2 doc files updated to remove dangling cross-references
- 0 behavioral change
The wrong=0 fix in #359 changed the candidate-graph's refusal-reason
format when a ratified recognizer matches but its v1 injector returns
():
- Pre-#359: silently drop the recognized statement and admit a partial
graph from the rest — a wrong>0 hazard analogous to case 0050.
- Post-#359: refuse explicitly with reason "recognizer matched but
produced no injection" naming the statement and recognizer category.
Three tests in `test_candidate_graph_recognizer_wiring.py` were written
against the pre-#359 silent-drop behavior:
1. `test_empty_registry_preserves_existing_refusal_reason` — asserted
the old "no admissible candidate" was the only valid format. Updated
to accept either the legacy format OR the new explicit-refusal
format.
2. `test_recognized_rate_statement_no_longer_triggers_per_statement_refusal`
— asserted that recognized statements should NOT cause a per-statement
refusal (encoding the silent-drop premise). Inverted to assert the
correct post-#359 behavior: recognized-but-uninjectable statements
refuse EXPLICITLY, and the statement IS named in the diagnostic.
Renamed to `_refuses_explicitly_post_wrong_zero_fix`.
3. `test_recognized_descriptive_statement_no_longer_triggers_per_statement_refusal`
— same inversion + rename.
Renames preserve the original sites for git-blame continuity while
making the post-#359 contract the documented behavior.
No runtime change. wrong=0 invariant preserved.
Test plan:
- tests/test_candidate_graph_recognizer_wiring.py: 7 passed (was 3 fail / 4 pass)
- tests/test_candidate_domain_partition.py: 5 passed (no cognition regression)
- tests/test_brief_11b_audit_artifact.py + step2_lexicon + recognizer_skip_wrong_zero + brief_11_audit: 55 passed
- Total: 62 passed
Wave-Next A2 brief outcome: the Rate type (ADR-0122) DOES structurally
model a per-unit rate, but it is not a member of the per-sentence
injector contract's SentenceChoice union (CandidateInitial |
CandidateOperation). The injector therefore returns () and documents
the schema gap inline plus in audit_brief_11.md.
Lift count: 0 (expected — the brief explicitly anticipates this
outcome when the schema decision is "no"). Documenting the gap is
the deliverable.
- generate/recognizer_anchor_inject.py: new inject_rate_with_currency
+ dispatch-table entry routing ShapeCategory.RATE_WITH_CURRENCY.
- tests/test_injector_rate_with_currency.py: 16 tests pinning schema
evidence, schema refusal, dispatch wiring, case 0050 hazard,
determinism, and wrong=0 invariant.
- evals/gsm8k_math/train_sample/v1/audit_brief_11.md: appended
Wave-Next A2 section documenting the schema decision, eval delta
(3/0/47 unchanged), case 0050 hazard verification, and the
CandidateRate follow-up sequencing.
Case 0050 hazard pin: sentence 0 ("Mark does a gig every other day
for 2 weeks.") carries no currency symbol — rate_with_currency
never matches it; case stays refused at sentence_index=0.
Adds 3 drain_token lemmas to en_core_math_v1 closing 2 of 3 remaining
lexicon_entry refusals from the prior wave:
- path (case 0049, new lemma)
- journey (case 0049 follow-on after path resolved)
- sees → alias of existing "see" lemma (case 0040)
The third remaining lexicon_entry refusal (case 0001, '+') is
deliberately NOT closed: '+' is an arithmetic operator literal, not a
lexical token. Adding it as drain_token would silently drop arithmetic
content from problems like "5 + 3 apples", a wrong=0 hazard. Documented
in the PR body and audit artifact.
Refusal taxonomy shift:
- unknown_word: 5 → 3 (-2)
- unresolved_pronoun: 3 → 4 (+1) — case 0049's pronoun barrier surfaced
- incomplete_operation: 20 → 21 (+1) — case 0049's quantity gap surfaced
Hard invariants:
- wrong == 0 (admitted=0, verified)
- case 0050 hazard pinned (refused at sentence_index=0)
- manifest checksum unchanged (per-category source file edit)
- no teaching-store mutation; no reader runtime change
## Summary
Two test failures on origin/main both trace to PR #315 (ADR-0163.D.2 —
discrete_count_statement recognizer + admissibility-intent chain). Earlier
runs treated them as "pre-existing unrelated" — they are not unrelated.
The first is a real wrong>0 hazard.
## Failure 1: silent admission via recognized-but-uninjectable statement
The ratified `discrete_count_statement` recognizer over-matches: ANY
sentence containing a number + noun resolves it, irrespective of the verb.
When `inject_from_match` returns `()` (the round-2 default for v1
categories without an injector), the old code path used `continue` to
silently drop the statement — and the solver then answered from whatever
initial state remained.
Reproduction:
parse_and_solve("Sam has 5 apples. Sam contemplates 3 apples. "
"How many apples does Sam have?")
→ is_admitted=True, answer=5.0 (silent admission of partial graph)
This is exactly the case-0050-class hazard wearing a different hat
(silently admitting an incomplete graph at the problem level).
ADR-0167 / Brief 11 §"correct-count greed" established the principle on
the reader path; this commit extends it to the recognizer path.
Fix: when a recognizer matches but produces no injection, REFUSE.
generate/math_candidate_graph.py:
- Replaced the skip-only `continue` with a CandidateGraphResult
refusal carrying the recognizer category in the reason.
tests/test_math_candidate_graph.py:
- test_unparseable_statement now accepts either the legacy
"no admissible candidate" reason or the new
"recognizer matched but produced no injection" reason.
Both legitimately refuse; what matters is is_admitted=False.
tests/test_recognizer_skip_wrong_zero.py (NEW):
- 5 regression tests pinning the wrong=0 invariant:
* 3 parametrized verbs unknown to both regex parser and reader
(contemplates / ponders / memorises) — must all refuse
* Nonsense token — must refuse
* Anti-regression: known initial + known operation still admits
## Failure 2: cognition audit drop-reason taxonomy
The audit test hardcoded `dropped.reason.startswith("superseded_by:")`
as the only valid drop-reason prefix. Commit da70919 (ADR-0163.D.2)
ratified an admissibility-intent chain that the audit categorizes with
reason `unsupported_intent:admissibility`, which fails this assertion.
Fix: tests/test_teaching_audit.py — expand the allowed-prefix set to
include `unsupported_intent:` with a written rationale. Future drop
classes extend the allowlist deliberately rather than silently
broadening the assertion to any non-empty reason.
## Surfaced regression: partition-test allowlist (ADR-0167 FOLLOWUPS §2)
This PR modifies three test files that the
test_existing_cognition_tests_untouched assertion would reject under
its named-allowlist scheme. Added the three test paths to the allowlist
as the tactical fix; the architectural fix (retire / move to CI / move
to CODEOWNERS) is queued in docs/handoff/ADR-0167-FOLLOWUPS.md §2.
## Test plan
uv run pytest tests/test_recognizer_skip_wrong_zero.py \
tests/test_math_candidate_graph.py \
tests/test_teaching_audit.py \
tests/test_candidate_domain_partition.py \
tests/test_math_evidence_e2e.py \
tests/test_math_evidence_schema.py \
tests/test_math_contemplation_adapter.py \
tests/test_math_claim_signature.py \
tests/test_math_lexical_ratification.py \
tests/test_brief_11b_audit_artifact.py \
tests/test_brief_11b_step2_lexicon.py \
tests/test_brief_11_audit.py
→ 152 passed
## Hard invariants
- wrong == 0 — restored on the recognizer path (was silently violated on main)
- ADR-0166 — no new eval lanes
- No teaching-store mutation, no pack mutation
- The reader path was already correct (it refused these cases); this fix
brings the regex/recognizer path back in line
Wave 3, closes the LexicalClaim slice of ADR-0167. After this PR the
math reader's refusal taxonomy is evidence, not terminus: lexical
refusals flow through audit row → typed evidence → dedup signature →
HITL ratification (W2-D) → pack write → next-audit-pass-resolves.
Deliverables
------------
- tests/test_math_evidence_e2e.py (new, 7 tests):
* test_full_pipeline_from_audit_to_evidence
* test_e2e_replay_equivalence
* test_lexical_ratification_advances_unknown_word_row (case 0040 'sees')
* test_e2e_determinism_across_processes
* test_cognition_teaching_corridor_unaffected
* test_evidence_dedup_via_claim_signature
* test_audit_artifact_round_trip_with_signatures
- evals/gsm8k_math/train_sample/v1/audit_brief_11.md: Post-W2 baseline
table + cognition regression line + case 0050 hazard status + pointer
to the new e2e regression module.
- tests/test_candidate_domain_partition.py: minimal allowlist patch to
test_existing_cognition_tests_untouched so that future ADR-0167 PRs
can add their own evidence test files without tripping a structurally
brittle hard-coded whitelist (W2-C partition risk; recorded in PR body).
Hard constraints held
---------------------
- wrong == 0: case 0050 hazard still refuses at sentence_index 0
after the tmpdir-pack 'sees' ratification; no admission introduced.
- Cognition regression: zero modifications to cognition test bodies;
only the W2-C whitelist assertion was loosened.
- Determinism: in-process and cross-process evidence_hash byte-identical.
- No real-pack mutation: a per-test digest fixture asserts
language_packs/data/en_core_math_v1/ is byte-identical before and
after each test.
Out of scope
------------
- Frame/Composition/Reference/Slot ratification handlers (follow-up ADRs).
- Workbench v1 wiring of math candidates (ADR-0167 §Q4).
- Auto-ratification — HITL only, forever.
- The two partition risks Gemini flagged in W2-C (cognition pack indexing,
replay-gate default) remain follow-up.
With this PR merged the engine can ratify math-domain lexical claims
from its own refusal evidence through the existing HITL teaching
corridor — the thesis claim of ADR-0167 becomes a concrete green test.
Adds `teaching/math_claim_signature.py` with `lexical_claim_signature()`:
sha256 hex of a normalised lexical token, collapsing two refusal cases on
the same surface token into one teaching-corpus candidate.
Normalisation pipeline (documented in module, breaking-change surface):
1. Lowercase surface
2. Strip string.punctuation from both ends (!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~)
3. Extract token from refusal_detail via r"no primitive or lexicon match for '([^']+)'"
4. Fallback: use stripped-lowercase surface if regex doesn't match
5. Canonical: "lexical:" + extracted_token
6. sha256 hex of UTF-8 bytes → 64-char lowercase hex
Also adds `teaching/math_contemplation.py` (W2-A adapter included as
union-merge; W2-A worktree was not yet dispatched):
- `audit_to_evidence()`: AuditRow iterable → MathReaderRefusalEvidence tuple
- `audit_problem_to_evidence()`: convenience wrapper for tests and W3-A
- Lexical evidence: claim_signature filled; evidence_hash recomputed to include it
- Non-lexical sub_types: claim_signature stays "" (deferred per ADR-0167 §Q1)
Real-data result on audit_brief_11.json:
- 14 distinct lexical tokens → 14 distinct signatures (no false collisions)
- No duplicate tokens in the 50-case sample; dedup logic verified deterministic
Wave 2, parallel with W2-C/D; depends on W1-A branch.
wrong=0 verified by passing regression suite.
Wave 2, parallel with W2-B/C/D. Implements the type-A→type-B converter
from AuditRow to MathReaderRefusalEvidence per ADR-0167 W2-A brief.
Deliverables:
- teaching/math_contemplation.py:
- audit_to_evidence(audit_rows): pure deterministic adapter, uses
SUB_TYPE_FOR_OPERATOR for subtype assignment, skips rows where
missing_operator is None, leaves claim_signature="" (W2-B will fill)
- audit_problem_to_evidence(problem_text, case_id): convenience wrapper
that runs the reader and adapts the output
- tests/test_math_contemplation_adapter.py: 8 tests covering
determinism, input-order preservation, sub-type mapping
exhaustiveness, distinct hashes across cases, empty input handling,
None-operator skip, and round-trip from problem text
Invariants:
- Deterministic across reruns (verified by determinism rerun)
- No I/O in adapter path
- Input order preserved (no internal sort)
- claim_signature == "" for all W2-A records (W2-B coordination)
Validation:
- tests/test_math_contemplation_adapter.py: 8 passed
- tests/test_math_evidence_schema.py: 11 passed (W1-A regression)
- tests/test_brief_11b_audit_artifact.py + step2_lexicon + brief_11_audit:
45 passed (regression)
- Determinism rerun: identical results
* feat(ADR-0167/W1-A): MathReaderRefusalEvidence schema + canonical-bytes
Foundation type for routing comprehension-reader refusals into the
teaching corridor. Frozen dataclass with sha256 evidence_hash computed
from deterministic canonical bytes (mirrors state.to_canonical_bytes
pattern). Includes SUB_TYPE_FOR_OPERATOR mapping table covering all 13
missing_operator values in the current audit artifact.
Wave 1 only — no runtime mutation, no teaching-store integration, no
admission path. Downstream W2-A/B/C/D type-import from this module.
* feat(ADR-0167/W2-C): domain discriminator + cross-domain audit
- Links to the audit doc: docs/handoff/ADR-0167-W2C-cross-domain-audit.md
- Inventory details: 5 construction sites, 8 consumption sites
- Verification: 0 cognition test files were modified; all tests are green
- Downstream partition work flagged: contemplation indexing (in teaching/contemplation.py) and replay gate (in teaching/proposals.py)
Foundation type for routing comprehension-reader refusals into the
teaching corridor. Frozen dataclass with sha256 evidence_hash computed
from deterministic canonical bytes (mirrors state.to_canonical_bytes
pattern). Includes SUB_TYPE_FOR_OPERATOR mapping table covering all 13
missing_operator values in the current audit artifact.
Wave 1 only — no runtime mutation, no teaching-store integration, no
admission path. Downstream W2-A/B/C/D type-import from this module.
## Summary
Lexicon-entry closure track per Brief 11D recommendation (Candidate A,
sub-PR 1). Adds 12 drain_token lemmas + 1 alias to `en_core_math_v1`.
`unknown_word` row strictly decreases: **11 → 5** (-6 cases moved past
the first-pass vocabulary gap). `wrong == 0` preserved. `correct` does
not move because admitted=0 (the unblocked cases now refuse at
downstream frames — real new work becoming visible, not regression, per
Brief 11 §Gate 1).
## Additions (all category=drain_token)
| Lemma | Surfaced from |
|-----------|----------------------------|
| along | case 0049 (3rd-wave) |
| animals | case 0040 (3rd-wave) |
| decrease | case 0005 |
| jacks | case 0024 (jumping jacks) |
| length | case 0006 (3rd-wave) |
| previous | case 0006 |
| reach | case 0015 |
| stray | case 0040 |
| too | case 0039 |
| uphill | case 0049 |
| which | case 0001 |
| your | case 0001 (3rd-wave) |
| weight → weights (alias) | case 0021 |
All classified as `drain_token` (the only category that cannot open a
frame and therefore cannot create wrong admissions per Brief 11
§"correct-count greed" doctrine). Reclassifying any as
accumulation/depletion/transfer verbs would risk wrong>0 by opening a
malformed operation_frame.
## wrong=0 verification
- `assert audit_problem(case_0050)` returns `ReaderRefusal` at
sentence_index 0 (pinned by `test_hazard_case_0050_remains_refused_pre_frame`)
- 50-case audit: `admitted=0, refused=50` (pinned by
`test_no_case_admits_after_lexicon_closure`)
- No reader runtime changes; pack-only mutation in a single
per-category source file
- Manifest checksum unchanged: source-file edit doesn't regenerate the
compiled `lexicon.jsonl`; loader reads per-category sources for
alias-aware entries (see `generate/comprehension/lexicon.py:127`)
## Test plan
- 11 new tests in `tests/test_brief_11b_step2_lexicon.py`:
- 4 pack-additions pinning (categories, provenance, aliases, sort order)
- 4 reader-effect / hazard tests (admitted=0, case 0050 refused,
unknown_word row strictly decreased, manifest checksum unchanged)
- 2 loader-integrity tests (new lemmas + aliases resolve through
`load_lexicon` → `lookup`)
- 12 existing tests in `tests/test_brief_11b_audit_artifact.py` pass
(taxonomy counts updated to post-step-2 values)
- 23 existing tests in `tests/test_brief_11_audit.py` pass
## Hard invariants preserved
- `wrong == 0` — no admissions, no frame-opener miscategorisation
- ADR-0166 — no new canonical eval lanes; existing
`gsm8k_math/train_sample/v1/` artifact updated in-place
- No teaching-store mutation; pack mutation is explicit, single-file,
reviewed
- Manifest checksum unchanged (compiled lexicon.jsonl byte-identical)
## Follow-up
- 3 lexicon_entry refusals remain (case 0001 '+', case 0040 'sees',
case 0049 'path'). Not addressed in this PR: '+' is an arithmetic
literal (would change semantics of drain), 'sees' and 'path' have
many other downstream barriers. Address with next-bottleneck PR.
- The 6 cases now refusing at later frames feed directly into Brief
11D Candidate A sub-PR 2 (which bottleneck class to attack next).
Per Brief 11B-step-2 §Hard constraints: no safe runtime/pack change lifts
any of the 8 pre_frame_filler_sentence cases without violating wrong=0.
This PR publishes the verb-classification analysis as documentation and
leaves the reader runtime and en_core_math_v1 pack unchanged.
Per-case classification:
- 0002 (splits): drain_token; honest blocker is compound_numeric_literal
- 0016 (traveled): drain_token; honest blocker is multi_quantity_composition
- 0025 (go/picking): drain_token; no quantity in sentence (true filler)
- 0028 (opens): drain_token; no quantity (true filler)
- 0030 (decides/go): drain_token; no quantity (true filler)
- 0035 (decided/split): drain_token; no quantity (true filler)
- 0036 (studying): drain_token; no quantity (true filler)
- 0050 (does): modal_aux; HAZARD — naive drain produces wrong>0
because next sentence admits Operation(mark, add, 3, songs)
while the answer requires frequency-by-duration aggregation
(every other day for 2 weeks); blocker is out of scope.
Post-skip simulation: even with the offending sentence elided, every
case still refuses on a downstream bottleneck (lexicon_entry,
pronoun_resolution, unit_binding, fraction_percentage_literal). Zero
lifts are available in Brief 11B-step-2 scope.
wrong=0 verification: no change to lifecycle.py / lexicon.py / audit.py /
en_core_math_v1/**; parent invariants from test_brief_11b_audit_artifact
continue to hold (admitted=0, refused=50, wrong_count=0).
Tests: 11 new tests in tests/test_brief_11b_step2_verb_classification.py
pinning the 8-case enumeration, post-skip refusal taxonomy per case,
hazard case 0050 remaining refused pre-frame, and the 50-case
admitted=0/refused=50/wrong=0 invariant.
## Summary
PR 11B in the Brief 11 sequence. Closes the missing-operator inference gap
left by 11A (#343) and ships the per-case audit artifact that Brief 11 §Gate 2
identifies as "the main Brief 11 artifact."
## Why this PR does NOT touch the reader runtime
The naive closure fix for `pre_frame_filler_sentence` (drain
`statement_terminator` at pre-frame) lifts 2 cases from refused → admitted
but creates a `wrong > 0` hazard on `gsm8k-train-sample-v1-0050`:
```
Mark does a gig every other day for 2 weeks. For each gig, he plays 3 songs.
... How many minutes did he play?
```
With the drain enabled, the reader admits `Operation(mark, add, 3, songs)`
with unknown unit `minute` and would project to a wrong answer. The stricter
variant (`pending_entity_ref is None` + no quantities) fires on 0 of the 11
candidate cases. Per Brief 11 §"Failure modes to avoid §1 — Correct-count
greed," this PR rejects both variants and routes the closure fix to a
follow-up that adds the required verb vocabulary or sentence-intent
classifier.
## Deliverables
- `generate/comprehension/audit.py` — three new missing-operator labels:
- `pre_frame_filler_sentence` (8 cases)
- `descriptive_frame_question` (2 cases)
- `question_frame_slot` (1 case)
Closes the 11-case `None`-operator gap left by 11A.
- `evals/gsm8k_math/train_sample/v1/audit_brief_11.json` — per-case audit
artifact pinned by tests.
- `evals/gsm8k_math/train_sample/v1/audit_brief_11.md` — narrative summary
including the rejected-fix design tension and ranked Brief 11B-step-2
backlog.
- `tests/test_brief_11b_audit_artifact.py` — 12 tests pinning the new labels,
the per-case artifact, the wrong=0 invariant, and the refusal taxonomy.
## Bottleneck taxonomy (after Brief 11B labelling)
| missing_operator | count | category |
|-------------------------------|------:|------------------------|
| quantity_extraction | 9 | incomplete_operation |
| lexicon_entry | 9 | unknown_word |
| multi_quantity_composition | 8 | incomplete_operation |
| pre_frame_filler_sentence | 8 | unexpected_category |
| pronoun_resolution | 3 | unresolved_pronoun |
| fraction_percentage_literal | 3 | unexpected_category |
| unit_binding | 3 | unattached_quantity |
| descriptive_frame_question | 2 | unexpected_category |
| (others, 1 each) | 5 | various |
## Test plan
- 12 new tests in `tests/test_brief_11b_audit_artifact.py` pass
- 23 existing 11A tests in `tests/test_brief_11_audit.py` pass
- No runtime changes; reader byte-identical to main
## Hard invariants preserved
- `wrong == 0` — no runtime change, no new admissions
- ADR-0166 — no new canonical eval lanes added; existing
`evals/gsm8k_math/train_sample/v1/` artifact set extended
- No teaching store / pack mutation
## Follow-up
- **11B-step-2** — verb-vocabulary expansion or sentence-intent classifier
for `pre_frame_filler_sentence` (8 cases). See audit_brief_11.md §"design
tension" for the rejected one-line variants and why they fail wrong=0.
- **11C** — existing-lane capability snapshot (still gated on 11B-step-2 or
another closure pass).
Extend the comprehension reader from question-only scope to whole-
problem scope. Phase 1 (Brief 8 / #326) implemented question_frame;
this brief implements initial_state_frame, operation_frame, and
descriptive_frame, plus finalize() projection into a strict
ADR-0115 MathProblemGraph.
Architecturally correct under ADR-0164.3; not yet productive on
GSM8K train_sample. Below-floor measurement documented; specific
bottlenecks tabled for Phase 2.1 follow-up.
What landed
- Frame-opener dispatch in lifecycle.py for the three new statement
frames, plus rule handlers (_rule_op_*, _rule_preframe_*,
_rule_descriptive_*).
- finalize(state) -> MathProblemGraph | ReaderRefusal: pure
projection with closure checks (entity registry non-empty,
unknown target bound, every op/initial references a known entity,
Decimal precision projects losslessly).
- _classify extended to 3-tuple (category, surface, decimal_value)
with possessive strip retry. Brief 8.2's sentence-initial
lookup-first + gender-skip preserved AND extended to mid-sentence
(gender is enrichment everywhere, never admission).
- Whole-problem coexistence dispatch in math_candidate_graph.py
(config.comprehension_reader_questions=True): reader attempts the
whole problem; on any ReaderRefusal falls through to existing
regex parser. All-or-nothing per the brief.
- Lexicon expansion (carried into renamed proper_noun_gender_*
files): +2 accumulation_verb (adopt, invest), +2 currency_unit_noun
(dollar, cent), +6 capacity_verb (fill, lift, play, work, finish,
drive), +5 female names (allison, brooke, jan, marion, sidney),
+14 male names (bart, fernando, georgie, jake, jed, jeremie, jose,
orlando, rex, rudolph, steve, troy, xavier, yun), +numerous
count_unit_noun, drain_token, time_unit_noun.
- ADR-0164.4-phase2-statement-frame-reader.md — the architectural
rationale and acceptance contract.
Measurement (reader_phase2_delta.json):
flag-OFF: correct=3 refused=47 wrong=0
flag-ON: correct=3 refused=47 wrong=0
delta: 0/0/0
Below the brief's floor of correct >= 4. Architecture is sound — the
reader admits cases as graphs when the structure resolves, refuses
cleanly otherwise, preserves wrong=0 across both flag states.
Bottleneck table (from per-case attribution):
count refusal_class dominant cause
----- ---------------------- ------------------------------------
18 incomplete_operation multi-quantity ops; no-quantity op
11 unknown_word "hundred", "presently", "one-hour",
non-math verbs (compound numerics,
lexicon gaps)
6 unexpected_category fraction / percentage literals;
multi-subject sentences
6 unresolved_pronoun "them", "their", "his" with no
compatible entity
5 unattached_quantity quantity never bound to a unit
1 no_question_target question parsed but slot never set
Closing the gate to mixed-bounded [4, 24] is Phase 2.1 scope: extend
composition rules for multi-quantity ops, add fraction/percentage
primitives (per ADR-0164.1 amendment), expand lexicon for the
remaining unknown_word cases, extend pronoun resolution.
Invariants preserved
- wrong = 0 in both flag states ✓
- flag-OFF byte-identical to today ✓
- determinism (50/50 identical runs) ✓
- Capability axes G1-G5, S1 unchanged ✓
- Reader tests: 19 (Phase 2) + 18 (Phase 1, post-update) + 53 (pack)
+ 76 (lexicon + primitives) = 166 specific to this change; all pass
- core test --suite smoke -q: 67 passed
Rebase note
This PR was authored against an older base; rebased onto current
main to incorporate #333 (Brief 8.2 universal proper_noun_token
primitive) and #334 (ADR-0166 measurement discipline). The rebase
required:
- Lexicon files renamed proper_noun_entity_* -> proper_noun_gender_*
(with the Phase 2 additions merged into the gender_* files)
- Compiled lexicon.jsonl unchanged from #333's 207-entry state
(Phase 2's per-category additions are runtime-visible via the
source loader, not via the compiled file)
- _classify reconciled with Brief 8.2's sentence-initial dispatch +
Phase 2's 3-tuple decimal-value return
- All dispatch tables and category checks updated to reference
proper_noun_token (singular) instead of proper_noun_entity_{f,m}
- Three Phase 1 test expectations updated to reflect Phase 2
behavior (proper noun at position 0 now opens statement pre-frame
instead of refusing; pronoun resolution applies per ADR-0164.2)
Per ADR-0166's three-question test, this PR is honest measurement:
capability exists, at least one case admits, lane distinguishes
presence from absence — which the bottleneck table demonstrates.
Refs ADR-0164.3 §Phasing Phase 2, ADR-0164.1 amendment (Brief 8.2),
ADR-0166 §"Mixed (notable but not blocking)" — except here, below
floor.
ADR-0164.1 amendment: replace name-whitelist entity admission with a
universal lexeme primitive that recognizes any capitalized token as a
proper noun. The gender-coded name lists are demoted from admission
criterion to enrichment-only lookup. A name outside the curated lists
still admits cleanly with gender="unknown" — ADR-0164.2's pronoun
resolution rules handle the unknown case via single-salient fallback
or refuse with ambiguous_pronoun_referent.
Universal at the primitive layer: the new proper_noun_token primitive
is domain-agnostic. It sits in the shared PRIMITIVE_REGISTRY and is
available to every current and future reader (math, narrative,
code-comment, multi-lingual). The math reader is its first consumer.
Pattern: ^[A-Z][A-Za-z'-]*[a-z][A-Za-z'-]*$
- requires capitalized first letter
- requires ≥1 lowercase letter (rejects all-caps acronyms)
- allows internal apostrophes (O'Brien) and hyphens (Mary-Anne)
- matches "Tina", "Bob", "Marnie", "McDonald" — rejects "TINA",
"123", "$5.00" (those go to their own primitives)
Sentence-initial lookup-first dispatch (lifecycle._classify):
- At token_index == 0: lookup() first, skipping proper_noun_gender_*
categories (treated as not-found so the primitive can fire). If
lookup misses, primitive scan picks up novel names. Inverts the
question from "is this a name?" to "is this a known common word?"
- At token_index > 0: primitive-first with UNIT_CATEGORY_TOKEN ceding
to operational lexicon for currency_unit_noun overrides.
Lexicon rename (per-category source files):
- proper_noun_entity_female.jsonl -> proper_noun_gender_female.jsonl
- proper_noun_entity_male.jsonl -> proper_noun_gender_male.jsonl
Compiled lexicon.jsonl: rename the two semantic_domain tags; drop
"marnie" (was only in proper_noun_entity_female, now absent from
the gender-coded sources). Net: 208 -> 207 entries. New manifest
checksum: 1fb9b0d790258736267d528e8e8a2436ce88b9ce690805fe2813ba077861ba2a
New helper gender_of_proper_noun(surface, lexicon) returns
Literal["female","male","neuter","unknown"] — pure enrichment lookup,
never gates admission.
Measurement (reader_phase1_plus_proper_noun_delta.json):
- pre-primitive baseline: correct=3 refused=47 wrong=0
- post-primitive measurement: correct=3 refused=47 wrong=0
- No regression on wrong=0
- No net admission increase observed in this train-sample harness;
the architectural value is for future text outside the curated
gender lists (Sonnet's #332 expanded those to cover GSM8K names).
Tests:
- test_lexeme_primitives.py: registry count 8 -> 9, proper_noun_token
fires + variants (Bob, Marnie, McDonald, O'Brien, Mary-Anne),
numeric/all-caps refusals, numeric-literal still wins overlap on "123"
- test_reader_question_frame.py: 5 new tests for sentence-initial
dispatch + unknown-gender pronoun resolution + novel-name admission
via primitive (Zelda)
- test_en_core_math_v1_pack.py: category counts updated; mutual-exclusion
between gender_female and gender_male preserved; total 208 -> 207
- test_lexicon.py: category list + lookup assertion updated to renamed
proper_noun_gender_female
- test_proper_noun_primitive_universality.py: new test module asserting
domain-agnostic property of the primitive
Validation:
- pack + lexicon + primitive tests: 147 passed
- reader + universality tests: 22 passed
- smoke lane: 67 passed
Closes the engine_state question by leaving those files untracked
(repo discipline: runtime artifacts never enter PRs).
Refs ADR-0164.1 amendment, ADR-0164.2 §EntityRegistry, ADR-0165
§Legitimate uses (the new primitive passes the three-question test).
Phase A — RuntimeConfig flag:
core/config.py: adds `comprehension_reader_questions: bool = False`
Default OFF preserves byte-identical behaviour with today.
Phase B — Hybrid wiring in candidate-graph path:
generate/math_candidate_graph.py:
- _try_reader_for_question() dispatches to the comprehension reader
BEFORE the regex question parser; refusal falls through to regex
- reader_trace: tuple[str, ...] field on CandidateGraphResult captures
JSON-encoded admit/fallthrough events for audit
generate/comprehension/lifecycle_runtime_adapter.py (new):
- build_problem_state_from_candidates(): converts regex-parser output
to ProblemReadingState for the reader's pronoun-resolution step
- invoke_reader_for_question(): tokenises sentence, drives lifecycle
- project_to_candidate_unknown(): QuestionTargetSlot → CandidateUnknown
- trace-event constructors for admit and fallthrough
Phase C — Capability-axis regression:
All existing tests pass with flag OFF and ON; zero new regressions.
Two pre-existing failures on main are unrelated to this PR.
Phase D — GSM8K train_sample measurement:
evals/gsm8k_math/train_sample/v1/runner.py: --use-reader flag triggers
baseline-off + reader-on runs and writes reader_phase1_delta.json
evals/gsm8k_math/train_sample/v1/reader_phase1_delta.json (new):
baseline-off: correct=3 refused=47 wrong=0
reader-on: correct=3 refused=47 wrong=0
delta: all zeros — Mixed result expected (Phase 2 scope)
wrong=0 invariant preserved in both modes.
Phase E — Coexistence tests:
tests/test_reader_coexistence.py (new): 13 tests covering
flag-OFF byte-identity, flag-ON determinism, wrong=0 invariant,
trace shape validation, Brief-8 target admission, and fallthrough
preservation for unknown-unit words.
Admission gate result: Mixed (correct=3, below the ≥10 bar).
All statement-side barriers remain in place; Phase 2 (reader for
statement sentences) is required to drive correct≥10. Documented in
reader_phase1_delta.json and train_sample/v1/runner.py docstring.
Adds the three lifecycle functions for the incremental compositional
reader per ADR-0164.3 §Lifecycle API:
- begin_sentence(problem_state, source_text_offset) -> SentenceReadingState
- apply_word(sentence_state, problem_state, word) -> SentenceReadingState | ReaderRefusal
- end_sentence(sentence_state, problem_state) -> ProblemReadingState | ReaderRefusal
Phase 1 scope is question sentences only. The update rules for the
question_frame live in a single readable table (_QUESTION_FRAME_RULES);
statement-side frames (initial_state_frame, operation_frame,
descriptive_frame) refuse with a Phase-2 diagnostic.
The five Brief-8 GSM8K target question sentences (0007, 0017, 0027,
0036, 0043) produce valid QuestionTargetSlot outputs end-to-end.
_interface_stubs.py provides a thin, functional surface for the
lexeme-primitive scanner (Brief 6) and lexicon loader (Brief 7) so
this PR does not block on them. The stub honours the en_core_math_v1
pack entries and adds a closed Phase-1 supplemental vocabulary marked
for fold-in to the pack once Briefs 6/7 land.
Tests cover determinism (byte-equal canonical bytes), the five GSM8K
target sentences with expected (entity, unit_class, kind) triples,
all token-level and sentence-level refusal modes, and lifecycle
invariants (registry preservation, sentence_index advance).
Stacked on feat/state-two-level-split (PR #323) per ADR-0164.3
§Naming — state types live in state.py.
Adds generate/comprehension/lexeme_primitives.py with the eight seed
primitives specified by ADR-0164.1:
decimal-currency-literal (priority 10)
currency-literal (priority 20)
percentage-literal (priority 30)
fraction-literal (priority 40)
time-amount-literal (priority 50)
ordinal-literal (priority 60)
mass-noun-token (priority 70)
numeric-literal (priority 100)
LexemePrimitive and LexemeMatch are frozen/slots dataclasses. scan()
runs primitives in priority order and returns the first hit wrapped in
a MappingProxyType over sorted-key extracted_values for canonical-bytes
stability. All patterns use explicit space characters ([ ]?, [- ]?) not
\s so the ADR-0165 compliance invariant holds.
55 tests cover: construction invariants, canonical fires (each
primitive on its own example), overlap precedence ($18.00, 1/2, 50%),
refusal on Tina/empty/verbs, determinism, sorted-key stability, and
the ADR-0165 compliance smoke test.
Ports the closed-set vocabulary from generate/math_candidate_parser.py and
generate/math_roundtrip.py into a new language pack en_core_math_v1, following
the manifest-checksum discipline of en_core_cognition_v1 and en_core_relations_v1.
208 lemmas across 11 semantic categories:
- accumulation_verb (17) — from ADD_VERBS + _COND_ADD_VERBS + _EARNINGS_VERBS
- depletion_verb (15) — from SUBTRACT_VERBS + _COND_SUBTRACT_VERBS
- transfer_verb (7) — from TRANSFER_VERBS; give/send/return removed from depletion
- currency_unit_noun (8) — from _MASS_NOUNS
- entity_pronoun (4) — from _Q_SUBJECT_PRONOUN
- proper_noun_entity_female (62) — from _FEMALE_NAMES
- proper_noun_entity_male (76) — from _MALE_NAMES
- possession_verb (1) — have/has/had collapsed to bare lemma
- capacity_verb (13) — from _CAPACITY_VERBS (pick/pack/make exclusive here)
- question_open (2) — how, what
- residual_modifier (3) — left, remaining, after (attested in _COND_OP_Q_RE)
Pack is NOT wired into any runtime path (ADR-0164 Phase 3).
Source constants in math_candidate_parser.py are unchanged.
Deferred categories documented in manifest.json `deferred` field.
53 contract tests cover: checksum, per-category counts, provenance,
mutual-exclusivity invariants (acc ∩ dep = ∅, acc ∩ cap = ∅, dep ∩ xfer = ∅),
and ≥2 semantic domains per compiled entry.
First PR plumbing recognizer parsed_anchors into the candidate-graph as
typed CandidateInitial primitives. Scope limited to discrete_count_statement;
other five round-2 categories route to the round-2 skip-only fallback until
follow-up D.2.x PRs.
Five-layer wrong=0 safety net:
1. Matcher narrowness — _try_extract_discrete_count_anchor refuses on any
ambiguity (multi-subject, pronoun subject, non-possession verb,
multi-count, clause-split, unobserved counted_noun, unobserved
count_kind).
2. Extraction correctness — refusal-preferring; populated parsed_anchors
only when ALL narrowness rules hold.
3. Injection correctness — _initial_admissible gates every constructed
CandidateInitial; failure to ground returns () (under-admit).
4. Replay gate — propose-time admissibility_replay_gate auto-rejects any
matcher change that would lift GSM8K wrong count.
5. Multi-branch decision rule — injected candidate disagreeing with
another branch triggers refuse path.
Re-baseline (GSM8K train_sample v1):
- Old (#309 alone): correct=3 refused=47 wrong=0
- New (#309 + D.2 v1): correct=3 refused=47 wrong=0
- Empirical lift in v1 = 0 cases; framework operational. No GSM8K
train_sample case has a discrete_count statement that simultaneously
meets all narrowness rules AND is missed by the existing parser.
Bottleneck moves to other recognizer categories (D.2.2+).
Validation:
- tests/test_adr_0163_d2_discrete_count_injection.py: 34 passed
- tests/test_recognizer_match.py + test_candidate_graph_recognizer_wiring
+ test_admissibility_replay_gate: 27 passed
- adr_0131_* (G1..G5 + S1 wrong=0 invariant): 222 passed / 2 pre-existing
report-comparison failures / 3 skipped — byte-identical to pre-D.2
- Solver code: unchanged
Operator caveat: round-1's ratified discrete_count_statement spec is
unchanged. Matcher behavior on the spec's canonical_pattern has been
extended from detection-only to populated parsed_anchors. Re-ratification
is not required; if policy requires it on matcher-behavior changes, the
registry digest provides byte-stable provenance.
The issue #300 regression test calls normalize_to_versor() directly
to verify its closure contract — identical justification to
test_versor_closure.py. Without the allowlist entry, INV-02 fails
in CI on every PR rebased on top of the #312 fix.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Adds two pre-gate checks to propose_from_candidate that fire after the
Step 2 capacity check and before the replay gate. No log entry is
written on either refusal — the append-only invariant holds.
Check order at function entry (ADR-0161 §3):
1. Capacity (Step 2) → RefusedAtCapacity
2. Duplicate → RefusedAsDuplicate
3. Dependent_on_pending → RefusedAsDependent
4. Replay gate → auto-reject on regression
New frozen dataclasses:
@dataclass(frozen=True, slots=True)
class RefusedAsDuplicate:
proposal_id: str
existing_state: str # covers all states: pending/accepted/rejected/withdrawn
reason: str = "duplicate"
@dataclass(frozen=True, slots=True)
class RefusedAsDependent:
candidate_id: str
dependent_on: tuple[str, ...] # pending proposal_ids that block
overlapping_lemmas: tuple[str, ...] # normalised lemmas that triggered
reason: str = "dependent_on_pending"
Lemma-overlap rule: case-insensitive exact-match on strip().lower().
Conservative — over-reject rather than admit-with-hidden-dependency.
False positives are recoverable (re-emit after blocker is ratified);
false negatives silently couple ratification choices.
CLI surfaces both outcomes in cmd_teaching_propose and
cmd_teaching_propose_from_exemplars (exit code 1).
Step 2 backpressure tests updated: made pre-populated candidates use
unique objects to avoid triggering the new dependency check, and
updated idempotency assertions to reflect the new RefusedAsDuplicate
return for re-submitted content.
Co-references: ADR-0161 §3, Step 1 PR #296, Step 2 PR #311,
ADR-0057, ADR-0151.
The bug: ingest.gate.inject raised RuntimeError("Injection produced
non-versor field") on a class of ordinary English token combinations
(declarative-with-quantity + transfer phrase + "How many" question).
Both observed condition values (1.02e-06, 2.12e-06) cleared
unitize_versor's `bad_residue` heuristic but landed just above the
gate's 1e-6 downstream check, crashing the engine on textbook word
problems like:
"Tom has 5 apples. He gives 2 to Sarah. How many does Tom have?"
Root cause: normalize_to_versor accepted the unitized candidate
without checking that it strictly satisfied the gate's
versor_condition < _RUNTIME_CLOSURE_TOLERANCE (1e-6) contract.
unitize_versor's internal tolerance is permissive for construction-
time inputs; the gate's downstream tolerance is stricter. When the
two diverged on certain token mixes, the candidate slipped through
and the gate's assert fired.
Fix: mirror the strict-closure pattern from _runtime_closed /
_close_applied_versor. If unitize_versor succeeds but the result
still fails the public versor_condition < _RUNTIME_CLOSURE_TOLERANCE
contract, project through the deterministic construction map
(_seed_to_rotor) instead of returning the drifted candidate.
Per CLAUDE.md: threshold stays at 1e-6 (Non-Negotiable Field
Invariant). Construction boundary is where drift is repaired.
The fix lives at the SINGLE allowed normalization site
(ingest/gate.py's only entry point into the algebra) without
loosening any invariant.
Tests added (11):
- versor_condition strictly satisfied on a range of seeded random
inputs (property test)
- 20-iteration synthetic-marginal probe exercises the construction-
fallback path
- The three issue-#300 bisected crash repros run end-to-end through
`core chat` and complete without raising the RuntimeError
- Threshold constant pinned (failing the test if anyone lowers
_RUNTIME_CLOSURE_TOLERANCE)
Validation:
- All 11 new tests pass
- 37 existing versor / ingest tests pass (test_versor_closure +
test_versor_*_rust_parity + test_core_ingest + test_unknown_token_ingest)
- Three pre-existing main failures (architectural_invariants
INV02 / INV21 / INV24) are unchanged by this PR — verified by
running them against origin/main directly before and after the
fix
- The three crashing prompts now produce clean grounded surfaces
through `core chat`
Closes issue #300.
Three new question shapes extracted from the GSM8K train_sample
post-Phase-D refusal taxonomy:
- Pattern A — "How much MASS_NOUN does ENTITY VERB ..." with narrow
whitelist (money, profit, interest, income, savings, cost, amount,
total). Extending the whitelist requires a separate ADR.
- Pattern B — "How many more UNIT does ENTITY VERB ..." (comparative).
Structurally detected (regex + comparative_marker field) but
emission is gated until the solver gains comparative semantics
(D.5 follow-up). Without solver-side handling, emission would
return the entity's current total (off by the missing delta) and
break wrong=0.
- Pattern C — "How many UNIT does PRONOUN VERB [to VERB2] ..." with
a closed-set action-verb whitelist.
Pronoun-entity resolution (Pattern C):
- Pure, deterministic function _resolve_pronoun_entity
- Refuses on ambiguity: >1 distinct female/male name in problem text
→ no candidate emitted (better refuse than admit-with-wrong-entity)
- "they" / "it" outside scope — refuses
- Closed-set ~50/~50 female/male name whitelists sourced from
GSM8K train_sample observation
Wrong=0 safety nets:
1. Regex narrowness (mass-noun whitelist, "more" anchor, closed verb set)
2. Pronoun resolver refuse-on-ambiguity
3. Pattern B emission gated until solver semantics catch up
CandidateUnknown.comparative_marker added with default False so
existing 200+ construction sites stay byte-identical.
Plumbing: extract_question_candidates / _filtered_question_choices /
parse_and_solve thread an optional problem_text through to the
pronoun resolver. No solver, recognizer-registry, matcher,
candidate-graph wiring, proposal log, or eval-harness changes.
Validation (all green on this branch):
pytest tests/test_adr_0163_d4_question_grammar.py -> 45 passed
pytest tests/test_adr_0163_d3_conditional_prefix.py -> green
pytest tests/test_math_candidate_parser.py -> green
pytest tests/test_math_candidate_graph.py -> green
pytest tests/test_candidate_graph_recognizer_wiring.py -> green
pytest tests/test_adr_0131_*.py -> green
331 passed, 3 skipped
python -m evals.math_capability_axes.G3_numerics.v1.runner -> overall_pass=True
solved=20 / wrong=0
python -m evals.gsm8k_math.train_sample.v1.runner -> correct=3
refused=47
wrong=0
GSM8K train_sample baseline:
Pre-D.4 (D.3 base): correct=3, refused=47, wrong=0
Post-D.4 (this PR): correct=3, refused=47, wrong=0
No lift on this base branch. Cases that Pattern A admits at the
question level (e.g. 0001 "how much money does she make") still
refuse at the statement layer because the round-2 exemplar-corpus
recognizers (PR #309) are not on this base. Refusal reasons
update from "no admissible candidate for question" to "no admissible
candidate for statement" / "no branch produced a solvable graph" —
expected. The grammar machinery is structurally ready: when
stacked on PR #309, the projected lift to correct=8-13 should
manifest.
Per-pattern coverage on the 38 question refusals (post-Phase-D
question shape categorization):
Pattern A — mass-noun ENTITY VERB: ≥4 evidenced cases
(0001, 0003, 0022, 0029)
Pattern B — comparative quantifier: ≥3 evidenced (0007, 0035, ...)
— detection only, no emission
Pattern C — pronoun + action verb: ≥1 in-scope (0011)
(0008 modal "be able to" + 0025
joint-subject deferred to D.5)
Cross-references: ADR-0163 (#294), Phase D.3 (#308 — base), round-1
ratification (#304), round-2 ratification (#309 — required for the
projected lift), session recap (#305).
Phase D made statement-level admission consult the ratified
recognizer registry (PR #302) but the same wiring at the
question-admissibility point was left for follow-up. Post-Phase-B
round-2 ratification, 38 of 47 still-refused GSM8K train_sample
cases now refuse on QUESTIONS (vs 7 pre-ratification) — the
architectural bottleneck has migrated downstream.
The biggest single still-refused question shape is
``nested_question_target`` (11 of 38 cases): ``If X, how many Y
does Z have?`` style. The existing ``_Q_ENTITY_RE`` regex only
matches ``How many UNIT does ENTITY have`` without a conditional
prefix.
D.3 adds a deterministic, pure prefix-strip step that runs ONLY
when the bare parser returns no candidates:
_filtered_question_choices:
candidates = existing parser
if empty AND sentence starts with "If X, ":
strip the prefix, upper-case the first letter
re-run the existing parser on the suffix
Tests pin: prefix-strip correctness on the 5 brief-mandated case
shapes, no false admissions when the suffix is still unparseable,
non-question pass-through unchanged, idempotency, no input
mutation, real-GSM8K-question parameterised coverage.
Empirical reality (verified by re-running the train_sample lane):
the strip operation succeeds deterministically on every
nested_question_target case, but the resulting suffix still hits
OTHER parser limitations (``how much`` mass nouns instead of
``how many`` units, modal verbs like ``will be able to``, pronoun
entities, additional clause prefixes). D.3 alone produces ZERO
additional case-level lift on the current parser regex. D.3 is
necessary-but-not-sufficient; the next layer (extending the
question grammar to mass nouns + non-"have" verbs + pronoun
entity resolution) is required for the conditional-question
cases to compose into correct answers.
That layer is a separate ADR — it touches grammar surface, not
admission wiring. This PR ships ONLY the wiring extension.
Validation:
- 43 new + existing tests passed: tests/test_adr_0163_d3_*,
tests/test_math_candidate_graph,
tests/test_candidate_graph_recognizer_wiring
- 222 capability-axis tests passed / 2 pre-existing main
failures / 3 skipped — G1..G5 + S1 wrong=0 byte-identical
- 67 smoke passed
wrong=0 invariant preserved by construction: recovered candidates
flow through the same _question_admissible gate as direct
candidates; no new admission paths bypass the structural check.
Scope: extends one function in generate/math_candidate_graph.py.
Does not modify the parser regexes, the solver, or the recognizer
registry.
Unblocks the four Phase B round-2 exemplar corpora (PR #306) so they
can flow through `core teaching propose-from-exemplars`. The corpora
were committed in #306 but Phase C's ingest validator + synthesizer
were hard-coded to round-1 categories; this PR closes that gap.
Extends three modules with the three new categories
(discrete_count_statement, multiplicative_aggregation, currency_amount):
- teaching/exemplar_ingest.py — per-category validator dispatch +
_SUPPORTED_CATEGORIES. The file-stem rule loosens from
exact ``<category>_v1`` to ``<category>_v<N>`` so the
temporal_aggregation v2 widening from #306 ingests.
- teaching/recognizer_synthesis.py — per-category synthesizers
following the same observed_*-set + coverage-histogram pattern as
round 1. Determinism, narrowness rule (narrower-not-broader),
rules-only — same discipline.
- generate/recognizer_match.py — per-category matchers shipped as
DETECTION-ONLY (return empty parsed_anchors). Consistent with
Phase D's current skip-only wiring (PR #302). Real value
extraction lands when Phase D.2 plumbs parsed_anchors into the
solver; until then, detection-only is the right shape and
preserves wrong=0 by construction.
graph_intent Literal expanded to include "count" and "amount".
Test updates:
- tests/test_exemplar_ingest.py: extend _ROUND_1 with _ROUND_2;
test_list_corpora_loads_every_round_1_file now asserts every
committed corpus (round 1 + round 2) loads.
- tests/test_recognizer_registry.py: rename + repair
test_live_proposal_log_has_phase_c_pending_proposals →
test_live_proposal_log_has_phase_c_proposals. The original
asserted state=="pending"; PR #304 ratified the three, so the
test now asserts state=="accepted" and registry length matches.
Pre-existing failure on main, fixed here.
Validation:
- 132 passed across exemplar_ingest, recognizer_synthesis,
recognizer_match, recognizer_registry, candidate_graph_wiring,
admissibility_exemplars, refusal_taxonomy_lane,
admissibility_replay_gate
- 222 capability-axis tests passed / 2 pre-existing main failures /
3 skipped — G1..G5 + S1 wrong=0 invariant intact
- 67 smoke passed
- End-to-end CLI sanity check: `core teaching propose-from-exemplars
teaching/admissibility_exemplars/discrete_count_statement_v1.jsonl
--log /tmp/test.jsonl` produced proposal_id 8c7645b4..., state
pending, replay_equivalent=True, wrong_count_delta=0
Empirical projection: of 47 still-refused GSM8K train_sample
statements, ~22 match the discrete_count_statement recognizer, ~2
match multiplicative_aggregation, plus 3 rate_with_currency + 3
temporal_aggregation + 18 descriptive_setup_no_quantity recognized
under the existing round-1 wiring. After operator ratifies round-2
proposals, the candidate-graph skip-only wiring will drop those
sentences from the math state and a meaningful lift is projected.
wrong=0 preserved at every level by Phase D's skip-only
construction.
Scope: enables the round-2 pipeline; does NOT ratify anything;
does NOT modify generate/math_candidate_graph.py. Operator runs
propose-from-exemplars + review --accept after merge.
Phase B round 2. Categorizing the post-#304 GSM8K train_sample's
still-refused 47 set surfaced three coherent sub-shapes in the previously
UNCATEGORIZED tail plus five ratified-but-narrowness-blocked temporal
cases; this PR ships the operator-authored exemplar seeds + Phase A
categorizer extension that prove the corridor scales beyond round 1.
Exemplar corpora (70 new exemplars across 4 files):
- discrete_count_statement_v1.jsonl (20)
- multiplicative_aggregation_v1.jsonl (20)
- currency_amount_v1.jsonl (20)
- temporal_aggregation_v2.jsonl (10, widening)
Each corpus carries ≥3 verbatim train-sample citations, ≥12 (≥5 for v2)
novel operator-authored statements, and ≥1–3 edge cases. Statements are
disjoint across all 7 round-1 + round-2 corpora; tests enforce.
Phase A categorizer (evals/refusal_taxonomy/shape_categories.py)
extends ShapeCategory with three new members and inserts their rule
predicates AFTER the existing more-specific categories:
- rate_with_currency before currency_amount
- multiplicative_aggregation before discrete_count_statement
Each new rule predicate cites ≥3 train_sample case_ids in its docstring
(ADR-0163 §Risks). No LLM, no embedding, no learned classifier.
Refusal-taxonomy histogram empirical signal (public 50 sample):
- pre-round-2: 14 UNCATEGORIZED (categorized_rate 0.72)
- post-round-2: 1 UNCATEGORIZED (categorized_rate 0.98)
The single residual is case 0044 ("10% simple interest" — percentage
without change verb), an honest tail outside the three round-2 shapes.
wrong=0 holds on capability axes G1..G5 + S1; no runtime code shipped.
Smoke suite green (67/67).
Cross-refs: ADR-0163, #297 (Phase A), #298 (Phase B round 1),
#301 (Phase C), #302 (Phase D), #304 (round-1 ratify), #305 (session
recap).
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* chore(ADR-0163.C): land three Phase C pending proposals in live log
Phase C (#301) shipped the CLI but its PR dry-run wrote to a tmp log
path. This commit moves the three Phase C proposals into the live
teaching/proposals/proposals.jsonl so the Phase B→C audit trail is
visible in the proposal log and the proposals are ready for the
operator to ratify after Phase D ships.
Proposals (all state=pending, kind="exemplar_corpus"):
- 59223f13722f906a1cf9b65d9b01c990 — descriptive_setup_no_quantity
- 46ce297f797ff16da12db5de422ca3c9 — rate_with_currency
- a3b892546977c5f0f64c578d6052adbd — temporal_aggregation
Produced by `core teaching propose-from-exemplars --all` against the
live Phase B corpora. No ratification (ADR-0161 §5 — only the repo
owner ratifies). The Phase D admissibility-replay gate confirmed
replay_equivalent=true, wrong_count_delta=0 for all three.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(ADR-0163.D): wire ratified RecognizerSpecs into math_candidate_graph admissibility surface
Phase D is the first PR to extend the math admission surface. The
audit (#294) said the gap was admission, not operators, algebra,
substrate, or packs. Phase A measured the refusal taxonomy. Phase B
authored seeds. Phase C synthesized recognizers. Phase D wires
those recognizers into generate/math_candidate_graph.py.
Modules
- generate/recognizer_registry.py — pure projection over the proposal
log. Only proposals with source.kind="exemplar_corpus" AND
review_state="accepted" enter the tuple. Sorted by
(review_date, proposal_id). In-process cache keyed on log
(mtime, sha256) — no filesystem cache (ADR-0161 §1). Malformed
accepted specs raise RegistryLoadError citing the offending
proposal_id; silent drops are forbidden.
- generate/recognizer_match.py — per-category rules-only matchers
(no LLM, no embedding, no learned classifier). Honors the Phase C
synthesizer's narrowness rule: out-of-corpus currency symbols,
window units, and per-unit values do NOT match. Three matchers:
_match_descriptive_setup_no_quantity (zero-quantity surface),
_match_temporal_aggregation (event_count_per_window with
observed_window_units/quantifiers honored), _match_rate_with_currency
(currency_per_unit_rate with observed currency/per-unit/amount-kind
honored).
- generate/math_candidate_graph.py — narrowest-edit guard at the
per-statement choice loop. Before the existing
"no admissible candidate for statement" refusal, consult the
ratified registry. Recognized statements are dropped from
per_sentence_choices (zero math state) so the Cartesian product is
identical to "this statement was never there." Empty registry is
a no-op — backward compatibility preserved byte-identically.
Downstream consumption of parsed_anchors (turning recognized
rate/temporal surfaces into solver state that produces concrete
answers) is Phase E follow-up.
Tests (32 new)
- tests/_phase_d_fixture.py — synthetic in-memory ratified registry
built from the three Phase C pending proposals' content. Per
ADR-0161 §5 the agent does NOT ratify the live log; the synthetic
registry round-trips the real RecognizerSpec bytes the operator
will ratify after Phase D ships.
- tests/test_recognizer_registry.py (9) — empty/pending/wrong-kind
filtering, sort order, malformed-spec rejection, cache hit +
invalidation, live-log Phase C audit check.
- tests/test_recognizer_match.py (14) — per-category positive cases,
narrowness (out-of-corpus surface forms rejected), no-LLM import
check.
- tests/test_candidate_graph_recognizer_wiring.py (7) — empty registry
preserves existing refusal; synthetic registry: recognized
statements no longer trigger per-statement refusal;
wrong_count_delta == 0 on GSM8K train_sample; capability axes G1..
G5+S1 wrong=0 unchanged; per-category admission counts on the
refused-set; unrecognized statements still refuse with the
existing reason.
- tests/test_phase_d_replay_evidence.py (2) — full admissibility
replay gate under synthetic registry: replay_equivalent=true,
wrong_count_delta=0, every capability axis wrong=0; each
ratified recognizer admits >= 1 train_sample statement (wiring
is consequential).
Per-category fixture-based admission counts (synthetic registry vs
GSM8K train_sample refused-set sentences):
- descriptive_setup_no_quantity: 40
- rate_with_currency: 2
- temporal_aggregation: 7
Narrowness-invariant negative case results (matcher correctly
returns None on out-of-corpus / load-bearing-math surfaces):
- rate_with_currency: "She paid $5 for the book." (no per-unit)
- temporal_aggregation: "On Saturday she went to the store." (single day token)
- descriptive_setup_no_quantity: "There are some kids in camp." (indefinite quantifier)
Candidates for Phase B round 2 (3 of 20 temporal seeds match the
spec's structural commitment but not my surface regex — author_notes
explicitly flagged these as schema-gap edge cases):
- ta-v1-0004 "Mark does a gig every other day for 2 weeks."
- ta-v1-0012 "Robin walks 4 dogs every other day around the park."
- ta-v1-0019 "The pump fills the tank with 80 gallons over 6 hours."
Three landed wirings DO NOT shift the GSM8K train_sample baseline
counts under fixture (correct=3, wrong=0, refused=47 unchanged) —
Phase D's narrow wiring is wrong=0 safe by construction; lift to
"correct" requires Phase E's downstream parser-side consumption of
parsed_anchors. Capability axes G1..G5+S1 wrong=0 unchanged.
Cross-refs: ADR-0163 (Phase D), ADR-0057 (proposal review),
ADR-0151 (auto-proposal), ADR-0161 §5 (ratification boundary),
Phase A PR #297, Phase B PR #298, Phase C PR #301.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Phase C is the first phase where operator-authored exemplar corpora
become engine-derived recognizer proposals automatically. The math
thesis ("decodes, not generates") manifests in the math lane here.
Modules
- teaching/exemplar_ingest.py — pure-function loader for Phase B
exemplar JSONLs. ExemplarCorpus carries a sha256 digest over its
canonical (sorted-by-exemplar_id, sort-keyed) bytes.
- teaching/recognizer_synthesis.py — per-category synthesizers
(_synthesize_descriptive_setup_no_quantity / _temporal_aggregation /
_rate_with_currency) distil a corpus into one RecognizerSpec.
Determinism: same corpus -> byte-identical spec. Narrowness: the
spec records only observed sub-shapes; an out-of-corpus currency
symbol or window unit does not match. Phase B author_notes surface
in canonical_pattern.unresolved_notes — never silently dropped.
- teaching/contemplation.py — contemplate_exemplar_corpus(corpus)
returns a DiscoveryCandidate whose proposed_chain encodes the
RecognizerSpec as a synthetic four-field chain plus the full
recognizer_spec submap. Evidence cites every exemplar's case_id.
- teaching/replay.py — run_admissibility_replay_gate(spec, *,
active_corpus_path=None) runs cognition + G1..G5+S1 + GSM8K
train_sample. In-process baseline cache keyed on the active
corpus digest. WRONG-COUNT INVARIANT: if a candidate run lifts
the GSM8K train_sample wrong count, gate returns
replay_equivalent=False with
regressed_metrics=["gsm8k_train_sample_wrong_count"].
- teaching/source.py — ProposalKind widened with "exemplar_corpus";
exhaustive-match docs + tests updated.
CLI
- core teaching propose-from-exemplars <path> [--all] [--review-date]
[--log] [--json]. Routes the candidate through the existing
propose_from_candidate path with the admissibility gate substituted
for the cognition-only run_replay_equivalence. Never auto-accepts;
proposals land as pending for operator review.
Tests (38 new)
- tests/test_exemplar_ingest.py (12) — load, digest stability,
malformed-record rejection, file-name binding, read-only purity.
- tests/test_recognizer_synthesis.py (16) — determinism, purity,
per-category subsumption, narrowness (out-of-corpus seeds rejected),
author_notes surfaced.
- tests/test_admissibility_replay_gate.py (6) — happy path, cache
hit/invalidation, WRONG-COUNT INVARIANT regression, capability-axis
regression, cognition regression.
- tests/test_propose_from_exemplars_cli.py (4) — single corpus, --all,
determinism, read-only snapshot.
Acceptance evidence (dry run)
- All three Phase B corpora produce replay_equivalent=true,
wrong_count_delta=0. Proposal IDs:
descriptive_setup_no_quantity: 59223f13722f906a1cf9b65d9b01c990
rate_with_currency: 46ce297f797ff16da12db5de422ca3c9
temporal_aggregation: a3b892546977c5f0f64c578d6052adbd
- G1..G5+S1 wrong=0 unchanged; GSM8K train_sample 3/47/0 unchanged.
- core test --suite smoke -q: 67 passed.
- uv run core eval refusal_taxonomy: case_digest
d030f826cb0f4088771d90c52c8be2ff75054ab27c7d47eae8dbfe1225b2eea1
unchanged.
Cross-refs: ADR-0163 (Phase C), ADR-0057 (gating discipline),
ADR-0151 (auto-proposal), ADR-0152 (learning-arc), ADR-0149/0154
(recognizer pipeline), ADR-0094 (ProposalSource), Phase A PR #297,
Phase B PR #298.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Round 1 of ADR-0163 Phase B: hand-author seed exemplars for the top three
refusal shape categories surfaced by the Phase A histogram. These corpora
are INPUT to the Phase C contemplation runner, which will derive
DerivedRecognizer proposals from them; this PR ships no recognizer logic,
no proposal logging, and no runtime change.
Per-category breakdown:
- descriptive_setup_no_quantity_v1.jsonl — 20 exemplars (5 train + 12 novel + 3 edge)
- temporal_aggregation_v1.jsonl — 20 exemplars (4 train + 13 novel + 3 edge)
- rate_with_currency_v1.jsonl — 20 exemplars (3 train + 14 novel + 3 edge)
Train-sample citations resolve against
evals/gsm8k_math/train_sample/v1/report.json (the 50-case sample only;
public/holdout/full splits NOT mined per ADR-0163 §Constraints).
Each file is sorted by exemplar_id, byte-canonical, and disjoint from the
others. Statements are surface-preserved verbatim from the train sample
where cited.
Validation:
- tests/test_admissibility_exemplars.py: 20/20 passed (schema, enum
binding, per-category quantity_anchor dispatch, cross-file disjointness,
>=3 train-sample citations per category, sort/byte-canonical determinism,
read-only import invariant)
- tests/test_adr_0131_*.py: 224 passed / 3 skipped — capability axes
G1..G5 + S1 remain wrong=0
- core test --suite smoke: 67 passed
- core eval refusal_taxonomy: case_digest unchanged
(d030f826cb0f4088771d90c52c8be2ff75054ab27c7d47eae8dbfe1225b2eea1)
- Phase A categorize() agrees with the file's category for all 60
statements (sanity check; not pinned in tests since the rules-only
categorizer is coarser than the recognizer Phase C will derive)
Author notes on quantity_anchor annotation calls flagged for operator
review are embedded in provenance.author_note where ambiguous (notably:
'in N minutes' / 'over N hours' window framings collapsed to
window_quantifier='per', 'every other day' approximated as 'every',
day-of-week labels not captured in the schema, 'for one X' / slash-form
per-unit framings, non-USD currencies, and discrete-occurrence per_unit
values like 'event' and 'session').
Refs: ADR-0163 §Phase B; depends on the Phase A lane shipped in #297.
Cross-refs: ADR-0057 (proposal review), ADR-0149/0154 (recognizer
pipeline), ADR-0161 (HITL queue), [[thesis-decoding-not-generating]].
* docs(math): ADR-0163 — path to GSM8K mastery via candidate-graph admissibility (proposed)
Audit reframes the math roadmap entirely.
State of main: every named math capability axis (G1..G5, S1) passes
at 100% with wrong=0 on its controlled lane. binding_graph,
math_versor_arithmetic, math_symbolic_equivalence, math_parser,
math_candidate_parser, math_solver, math_verifier, math_realizer,
math_problem_graph — all landed. The worktrees on disk are stale
forks.
State of GSM8K (50-case train sample): correct=0, refused=50, wrong=0.
Every refusal reason is identical: "candidate_graph: no admissible
candidate for statement: <STATEMENT>".
The reframe: the gap is NOT in operator algebra, NOT in binding graph
internals, NOT in symbolic equivalence. The gap is in
generate/math_candidate_graph.py — the admissibility surface that
turns a natural-language statement into a candidate the downstream
pipeline can consume. The capability axes pass at 100% because they
test statement shapes the candidate-graph already admits. GSM8K
refuses at 100% because its statements span shapes the candidate-graph
has never been taught.
Six-phase plan to lift GSM8K under the thesis "decodes, not generates":
A. Refusal taxonomy (measure before building)
B. Exemplar corpora per shape category (≤20 statements each, ≤3 per round)
C. Contemplation runner ingests exemplars; emits DerivedRecognizer
proposals
D. Operator ratifies through ADR-0161 HITL queue (no new surface)
E. Re-baseline GSM8K train sample. Round 1 exit: correct ≥ 10, wrong = 0.
Round 2: ≥ 25. Round 3: ≥ 35.
F. Scale to public/v1 (200 cases, target correct ≥ 100), then
holdout (measurement-only — never tune against).
Three non-negotiables:
- wrong = 0 at every phase. Auto-rejected by replay gate, not by
operator vigilance.
- No hand-rolled recognizers in generate/. Every recognizer lands
via contemplation → proposal → review corridor.
- Active corpus mutation only via accept_proposal.
Status: proposed. Implementation lands as three PRs starting with
Phase A scaffolding.
Scope discipline: docs-only. No code, no eval changes, no corpus
mutation.
* feat(ADR-0161.1): core teaching queue list|show — read-only queue projection
* fix(ADR-0161.1): restore gap-queue CLI + rename new commands to hitl-queue + R1..R5 refinements
ADR-0163 Phase A measurement. Reads the GSM8K train-sample refusal report
(50 cases, all refused on candidate-graph admissibility) and emits a
histogram of statement shapes. Read-only: no corpus, pack, or proposal
mutation; the categorizer is rules-only with no LLM, embedding, or
learned model.
Lane: evals/refusal_taxonomy/ (auto-discovered by evals.framework)
- shape_categories.py — ShapeCategory enum + deterministic categorizer
(9 ADR-mandated baseline categories + UNCATEGORIZED, first-match-wins)
- runner.py — pure run_lane(cases) -> LaneReport
- contract.md — purpose, doctrine, schema, ADR compatibility
- public/v1/cases.jsonl — 50 refused statements (sorted by case_id)
- v1/report.json — first run output (categorized_rate=72%)
CLI: core teaching refusal-taxonomy [--input PATH] [--json] [--save]
Accepts a cases JSONL or a raw GSM8K eval report.json directly.
Helper: scripts/build_refusal_taxonomy_cases.py rebuilds the v1 case set
from the GSM8K train-sample report deterministically.
Tests: tests/test_refusal_taxonomy_lane.py (21 passing) cover schema
integrity, lane auto-discovery, enum exhaustiveness, categorizer
determinism + purity + no-ML-imports, histogram correctness, replay
byte-identity, committed report match, helper extraction, and a
read-only invariant snapshot over teaching/, packs/, language_packs/data/.
v1 histogram (50-case sample):
17 descriptive_setup_no_quantity
14 uncategorized
4 temporal_aggregation
3 rate_with_currency
3 fractional_rate_of_change
3 indefinite_quantity
3 comparative_with_unit
2 nested_question_target
1 unit_partition
0 conditional_quantity
total=50 categorized_rate=72% uncategorized=28% (below 50% target)
Top three by count (Phase B candidates):
1. descriptive_setup_no_quantity (17)
2. temporal_aggregation (4)
3. tie at 3 — operator selects from {rate_with_currency,
fractional_rate_of_change, indefinite_quantity, comparative_with_unit}
Phase B is not started in this PR — the ADR explicitly requires the
operator to ratify the top-N selection before any exemplar corpus is
authored.
Invariants verified:
- tests/test_adr_0131_*.py: 224 passed, 0 wrong on G1..G5 + S1
- core test --suite smoke -q: 67 passed
- The refusal_taxonomy/__init__.py and runner do not import openai,
anthropic, transformers, torch, sklearn, sentence_transformers,
requests, or httpx — verified by test_categorizer_no_llm_or_ml_imports.
Cross-references: ADR-0163 (parent), ADR-0114a (capability obligations),
ADR-0149 (recognizer pipeline substrate that Phases C–E build on).
Refs: [[thesis-decoding-not-generating]] — the rules-only categorizer
honors the doctrine: the engine learns to find better shapes; this PR
does not stuff it with another found pattern.
Three follow-ups raised in the W-025 PR #286 review, completed together so
the lane reaches its full mastery-level contract.
1. ``core eval`` failure-printer is now gated on ``lane_name == "cognition"``.
Before this fix, every non-cognition lane that returned clean case_details
without ``intent_correct``/``versor_closure`` keys triggered a spurious
``failures (N): <case_id>: intent, versor=0.00e+00`` block at the end of
the human-readable output, even when every metric passed. This matched
the gating pattern already used for the workers preamble at the top of
``cmd_eval``.
2. EPILOG examples in ``core/cli.py`` now advertise
``core eval contemplation_quality`` and the ``--json --save`` form, so
the lane is discoverable from ``core --help`` and not only from
``core eval --list``.
3. Tightened the learning-arc demo's Scene 5 to thread the demo's
tempdir-scoped ``engine_state_dir`` into the second ``ChatRuntime``.
The previous default-constructed runtime checkpointed to the repo's
``engine_state/``, which contradicted ADR-0159's read-only claim.
ADR-0146/0150 still govern the runtime checkpoint path itself.
Tests:
- ``tests/test_contemplation_quality_lane.py`` (35 tests):
case-set integrity, lane discovery, ``evaluate_report`` purity over
well-formed / malformed / boundary-violating inputs, ``run_lane``
invocation-contract enforcement (single case, supported source enum),
and a read-only invariant snapshot on ``teaching/corpora``, ``packs/``,
and ``language_packs/data/``.
- ``tests/test_eval_cli_failure_printer.py`` (4 tests): pins the
cognition-only gating of the failure printer with stubbed
``evals.framework`` so the regression cannot return as a lane-blind
condition.
Validation:
uv run pytest tests/test_contemplation_quality_lane.py \
tests/test_eval_cli_failure_printer.py \
tests/test_learning_arc_demo.py -q # 50 passed
uv run core test --suite smoke -q # 67 passed
uv run core eval contemplation_quality # 9/9 passed, clean output
* feat(W-024): reboot_event audit trail entry (L10b.3, ADR-0158)
L10 scope §Sub-question 3: a reboot_event analog of TurnEvent, written
to the telemetry JSONL, lets future audit reconstruct when this engine
instance lost and regained its lifetime.
- serialize_reboot_event / format_reboot_event_jsonl in chat/telemetry.py
emit type="reboot" with restored_turn_count, stored/current revisions,
revision_matched, recognizers_count, candidates_count
- ChatRuntime._load_engine_state() buffers the JSONL line in
_pending_reboot_payload (str|None); ChatRuntime.attach_telemetry_sink()
flushes it exactly once when a sink is first attached
- Reboot event precedes all turn events in the session audit stream
- Pinned by 11 tests: serializer structure, determinism, revision_matched
logic, runtime integration (emit-once, no-checkpoint, no-load-state,
revision match, ordering)
Closes L10b: W-022 (atomic writes) + W-023 (revision warning) + W-024
together satisfy ADR-0146's atomic/observable/auditable checkpoint triad.
* fix(W-024): expose cached public git revision helper
* feat(W-022): ratify-proposal workflow_dispatch for mobile ratification
Adds .github/workflows/ratify-proposal.yml — a manually triggered
workflow that lets the operator ratify engine-authored proposals from
the GitHub mobile app without needing terminal access.
Inputs: proposal_id (required), review_date (default: today UTC),
operator_note (optional). Runs `core teaching review --accept`,
commits the updated corpus + proposal log to main, and posts a
job summary with the accepted chain_id.
Shared CONTEMPLATION_ENABLED kill switch disables the entire
learning-arc loop (contemplation + ratification) with one toggle.
ADR-0155 / ADR-0057
* feat(W-023): revision-mismatch warning on engine-state load (L10b.2, ADR-0157)
ADR-0146 §Risks line 127 specified that load_manifest() should compare
written_at_revision against the current git SHA and warn if they differ,
but never refuse to load (reboot is recovery, not control flow).
- EngineStateStore.load_manifest() emits RuntimeWarning when stored and
current revisions are both known and do not match
- Suppresses warning when either side is "unknown" (offline/packaged builds)
- Always returns the manifest; no state is cleared or rejected
- Pinned by 8 tests covering match, mismatch, unknown suppression, and
missing/empty manifest edge cases
ADR-0156 §Out of scope closes; L10b.3 (reboot_event audit entry, W-024) remains.
Shay