docs(cmb): lookback review + fix single-agent-attribution hygiene hazard (H3)

The mandatory lookback before CMB-d (3+ PRs on one surface; next-phase boundary). A 5-dimension
read-only fan-out audit over the 10-item checklist found the CMB-a/b/c substrate composes cleanly:
25 solid / 4 drift / 8 gap / 1 reader hazard. Doc:
docs/analysis/cmb-lookback-review-2026-06-08.md.

Fixes the one real hazard (H3): a query attributing the answer to a SINGLE agent ('how many words
does Alice type' with a distractor second rate) wrongly claimed the substantive
combine_mode_ambiguous — a hygiene over-claim that would corrupt the shared router in CMB-d. The
combined-query gate now excludes single-agent 'does <Agent>' attribution (_SINGLE_AGENT_QUERY) and
steps aside as not_combined_rate_shaped; the genuinely-combined forms ('do they' / 'are produced' /
'does it') still yield combine_mode_ambiguous. Pinned by gold cmb-16 + a reader test.

The review also resolves the CMB-d failure-family classification (input_shape:
not_combined_rate_shaped; must_remain_refused incl. rate_unit_mismatch — DECISIVELY, the reader has
no dimension representation so it cannot tell convertible from incompatible units; proposal_allowed:
combine_mode_ambiguous, missing_second_rate) and enumerates the live CMB-d registry preconditions
(not_combined_rate_shaped mapping, rate_unit_mismatch/non_integer_solution string collisions,
non_positive_net_rate family, Organ/ALL_REASONS extension). Provenance false-alarm (stale local main
checkout) noted + verified against origin/main.

gold 19/19 (6/5/8); reader 11/0/0 + 8 refused-correct; solver 6/0 + 5/0; 0 hygiene breaches; 123
tests; serving + R1/R2/R3 unchanged.
This commit is contained in:
Shay 2026-06-08 13:27:17 -07:00
parent 84d3456e61
commit a4d0c4ebf3
6 changed files with 178 additions and 5 deletions

View file

@ -0,0 +1,146 @@
# CMB lookback review (CMB-a/b/c) — before CMB-d router wiring
**As of:** `main` @ `84d3456e` (CMB-a #650 · CMB-b #653 · CMB-c #654 all merged).
**Trigger:** the mandatory lookback before wiring a multi-rung substrate into shared infrastructure
(CLAUDE.md *Lookback Review Discipline* — "before starting the next phase of a multi-phase ADR" and
"after any 3+ PR sequence on the same surface"). CMB-d wires CMB into the shared
`route_setup` + contemplation + failure-family registry, where any latent CMB-c classification
mistake stops being local.
**Method.** A 5-dimension read-only fan-out audit (one agent per dimension, each reading the source
and running live probes) over a 10-item checklist, plus a synthesis pass. 38 findings: **25 solid /
4 drift / 8 gap / 1 reader hazard** (+ several CMB-d preconditions that are wiring, not CMB-c bugs).
**Verdict: ready for CMB-d once (a) the one reader hazard is fixed [done in this PR] and (b) the
registry preconditions below land in the CMB-d PR itself.**
---
## Provenance note (a false alarm, resolved)
The synthesis pass raised a "live main is CMB-a only; reader/solver are unmerged" alarm. **This is a
stale-local-checkout artifact, not a real finding.** Merges land server-side via `gh`; the *local*
`main` checkout was never pulled, so the synthesis agent's `git` saw an old tree. Verified:
`git cat-file -e origin/main:generate/combined_rate_comprehension/reader.py` → present at
`84d3456e`; `solver.py` likewise. The per-dimension audit agents read the worktree (= the merged
code) and their findings stand. Discount the provenance hazard.
---
## Hazard (1) — fixed in this PR
**H3 · single-agent-attribution over-claims `combine_mode_ambiguous`.** Two same-unit rates with a
query that attributes the answer to **one** agent — *"Alice types 5 words per minute and Bob types 3
words per minute. How many words does **Alice** type in 3 minutes?"* — is a single-rate question
with a distractor rate, not a combined query. The reader claimed the substantive
`combine_mode_ambiguous`; on the shared router (CMB-d) that is a substantive-refusal-on-foreign-text
= a hygiene breach.
**Fix (this PR):** the combined-query gate now excludes a single-agent `does <Agent>` attribution
(`_SINGLE_AGENT_QUERY`), so single-agent text steps aside as `not_combined_rate_shaped`. The
genuinely-combined forms (`… do they …`, `… are produced`, `does it`) still yield
`combine_mode_ambiguous`. Pinned by gold fixture **`cmb-16`** + `test_single_agent_attribution_steps_aside`.
A reader that **steps aside** on ambiguity is safe (wrong=0 preserved); only over-reads to a wrong
answer or substantive-refusals-on-foreign-text are hazards. H3 was the only one. The
comparison-query, sequential-duration, decimal-rate, incidental-drain, and reciprocal step-asides
are all correctly handled (confirmed) and are **not** hazards.
---
## Solid (the substrate that holds)
- **Division of labor (model/reader/solver).** The reader **parses** every solver-boundary setup
(`eff ≤ 0`, non-integer) and never intercepts a solver-owned refusal — `non_positive_net_rate` and
`non_integer_solution` appear nowhere in `reader.py`. The model permits `effective_rate ≤ 0` by
construction; the solver owns it. `run_reader` 11/0/0 + (now) 8 refused-correct; `run_solver`
6/0 + 5/0. No reader-owned malformed setup reaches the solver (the unit-mismatch guard fires
before `RateUnit` construction; the model has one unit slot).
- **2×2 domain-entry grid.** All four cells correct; every substantive refusal is gated behind
positive CMB-shaped evidence (the N6 over-proposing invariant) — `three_or_more_rates` /
`rate_unit_mismatch` only after a cue; `combine_mode_ambiguous` only with a same-unit combined
query; `missing_second_rate` only with a cooperative cue + two conjoined agents.
- **Foreign-organ hygiene.** 0 substantive refusals across all 36 R1/R2/R3 gold (every foreign
fixture → `not_combined_rate_shaped`), and across fresh foreign probes (comparison questions, R1
arithmetic, R2 prose, R3 single-rate with incidental cue words).
- **Reader correctness (the 3 adversarial-round classes hold).** Difference subtraction is
semantic-role based (fill = minuend, drain = subtrahend) not text-order; sequential segments step
aside (incl. loose connectors); decimals don't partial-parse.
- **Off-serving.** No CMB file imports `generate.derivation` / `core.reliability_gate` (AST-checked).
- **Cross-rung composition.** The reader's setups round-trip through the oracle's
`combined_rate_setup_signature`; the solver consumes exactly the reader's output type; the closed
taxonomies are consistent across model/reader/solver/oracle.
## Drift / gaps (no current wrong=0 risk; most are CMB-d's to resolve)
- `empty` refusal (blank input) is not in the oracle `READER_REASONS` — unreachable from prose
routing; add only if a blank-input gold case is ever introduced.
- `_AGENT_CONJ` matches any `word and word` bigram (not just proper-noun pairs); `_LEADIN = 25` is an
undocumented bound. Both are conservative (a miss → safe step-aside), not hazards.
- The remaining items are **CMB-d preconditions** (below), not CMB-c defects.
---
## CMB-d failure-family classification (the decision)
The load-bearing output. Reasons map to families as:
| Family bucket | Reasons | Rationale |
|---|---|---|
| **input_shape** (no proposal) | `not_combined_rate_shaped` | the step-aside reason — the router-hygiene gate **requires** it be mapped, or every CMB foreign refusal fails the invariant |
| **must_remain_refused** (no proposal) | `rate_unit_mismatch`*, `three_or_more_rates`, `clock_interval_deferred`, `reciprocal_work_rate_deferred`, `non_positive_net_rate` (solver), `non_integer_solution` (solver) | correct wrong=0 boundaries / deferred-capability boundaries, not coverage gaps |
| **proposal_allowed** | `combine_mode_ambiguous`, `missing_second_rate` | genuine, semantically-coherent coverage gaps — fired only **after** positive CMB recognition, and **before** the solver, so neither can emit a wrong answer |
**\* `rate_unit_mismatch` stays `must_remain_refused` for v1 — decisively.** It fires for two
structurally different sub-cases under one code: (A) dimensionally-**incompatible** units
(`rooms/hour` vs `liters/minute` — no conversion exists) and (B) a duration-unit ≠ rate-denominator
case. The reader has **no dimension representation** (`units.py` compares only `(numerator,
denominator)` string equality) and **cannot** distinguish convertible (`gallons/min` vs
`gallons/hour`) from incompatible. Marking it `proposal_allowed` would emit a "try conversion"
proposal for the incompatible case = a wrong=0 hazard if ratified. **Do not** introduce a narrower
`rate_unit_mismatch_convertible` family until the reader gains a dimension registry **and** the two
sub-cases fire under distinct codes **and** a separate conversion gold lane exists.
**Note on `missing_second_rate`:** it is `must_remain_refused`-by-default *and* `proposal_allowed`
in the sense above — it is an *under-specified-input* boundary (no reader enhancement can infer the
missing rate), distinct in kind from the math-impossibility boundaries. The proposal it licenses is
"what is the second agent's rate?", not a rule-learning proposal.
---
## CMB-d risk list (preconditions, to land in the CMB-d PR itself)
These are **live registry** facts verified against `main`; each must be handled in the same PR that
adds `classify_cmb` to `route_setup`:
1. **`not_combined_rate_shaped` is unmapped** in `failure_family.py` (`family_for_reason → None`).
The moment CMB joins `test_router_organ_hygiene.py`, every CMB foreign refusal fails the
"must map to `input_shape`" assertion. → add it to the `input_shape` family's `refusal_reasons`.
2. **Shared `rate_unit_mismatch` string collision.** R3 emits the same string → maps to R3's
`unsupported_rate_duration` **growth** family (`proposal_allowed`, `owner=r3`). A CMB unit-mismatch
(a hard boundary) would be mis-routed to a wrong-type R3 proposal. → CMB must emit a **distinct**
string (e.g. `cmb_rate_unit_mismatch`) and register it `must_remain_refused`.
3. **`non_positive_net_rate` is unmapped** → `ContemplationResult.family` silently `None` on a CMB
solver refusal. → add a `cmb_non_positive_net` family (`must_remain_refused`).
4. **`non_integer_solution` owner mislabel.** The CMB solver shares R2's string → family `owner=r2`.
→ emit a CMB-distinct string or extend the owner to credit CMB.
5. **`Organ` Literal + `ALL_REASONS` lack CMB.** → add `r4_combined_rate` (name TBD) to the `Organ`
Literal and extend `ALL_REASONS` with every CMB reader+solver reason, or
`test_registry_covers_the_whole_refusal_surface` won't catch an unregistered CMB reason.
6. **Off-serving assertion.** Add the per-module AST off-serving test for `classify_cmb` (substring
scans false-negative on docstring mentions).
7. **Exactly-one-`setup_correct` contract.** Re-confirm `route_setup` still selects a unique organ
when CMB joins — CMB must not make any R1/R2/R3 problem ambiguous, nor claim one (the hygiene
audit shows it steps aside on all R1/R2/R3 gold, which protects this).
**CMB-d's claim stays narrow:** "CMB participates in routing + typed contemplation without
corrupting R1/R2/R3 or serving." No new capability (that was CMB-c).
---
## Net
The CMB-a/b/c substrate composes cleanly. One real reader hazard (H3) is fixed here with a ruler
fixture + test. The remaining items are CMB-d wiring preconditions, now enumerated with the family
classification resolved. Lane state after this PR: gold **19/19** (6/5/8), reader 11/0/0 + 8
refused-correct, solver 6/0 + 5/0, 0 hygiene breaches, serving + R1/R2/R3 unchanged.

View file

@ -16,3 +16,4 @@
{"id": "cmb-13-reciprocal-work-rate", "expect": "reader_refuses", "reader_reason": "reciprocal_work_rate_deferred", "text": "Anna can paint a house in 3 hours, and Ben can paint the same house in 6 hours. Working together, how many hours do they take?", "gold": null, "notes": "Durations to complete a whole job, NOT explicit per-unit rates — the classic reciprocal work-rate form 1/(1/3 + 1/6). A different representation (reciprocal rates, rational arithmetic). REFUSE (reciprocal_work_rate_deferred) — deferred."} {"id": "cmb-13-reciprocal-work-rate", "expect": "reader_refuses", "reader_reason": "reciprocal_work_rate_deferred", "text": "Anna can paint a house in 3 hours, and Ben can paint the same house in 6 hours. Working together, how many hours do they take?", "gold": null, "notes": "Durations to complete a whole job, NOT explicit per-unit rates — the classic reciprocal work-rate form 1/(1/3 + 1/6). A different representation (reciprocal rates, rational arithmetic). REFUSE (reciprocal_work_rate_deferred) — deferred."}
{"id": "cmb-14-clock-interval", "expect": "reader_refuses", "reader_reason": "clock_interval_deferred", "text": "A pipe fills a tank at 5 liters per minute and another fills it at 3 liters per minute, running together from 2 pm to 5 pm. How many liters are in the tank?", "gold": null, "notes": "Elapsed clock time (2 pm to 5 pm), not an explicit duration. REFUSE (clock_interval_deferred) — deferred (the CMB twin of R3's temporal_state)."} {"id": "cmb-14-clock-interval", "expect": "reader_refuses", "reader_reason": "clock_interval_deferred", "text": "A pipe fills a tank at 5 liters per minute and another fills it at 3 liters per minute, running together from 2 pm to 5 pm. How many liters are in the tank?", "gold": null, "notes": "Elapsed clock time (2 pm to 5 pm), not an explicit duration. REFUSE (clock_interval_deferred) — deferred (the CMB twin of R3's temporal_state)."}
{"id": "cmb-15-not-combined-shaped", "expect": "reader_refuses", "reader_reason": "not_combined_rate_shaped", "text": "A car travels 60 miles per hour for 3 hours. How many miles does it travel?", "gold": null, "notes": "A single explicit rate with no second rate and no combination cue — single-rate R3 territory, not CMB's. CMB must step aside, NOT claim a substantive boundary. REFUSE (not_combined_rate_shaped) -> maps to the input_shape family in CMB-d (router-organ-hygiene invariant)."} {"id": "cmb-15-not-combined-shaped", "expect": "reader_refuses", "reader_reason": "not_combined_rate_shaped", "text": "A car travels 60 miles per hour for 3 hours. How many miles does it travel?", "gold": null, "notes": "A single explicit rate with no second rate and no combination cue — single-rate R3 territory, not CMB's. CMB must step aside, NOT claim a substantive boundary. REFUSE (not_combined_rate_shaped) -> maps to the input_shape family in CMB-d (router-organ-hygiene invariant)."}
{"id": "cmb-16-single-agent-attribution", "expect": "reader_refuses", "reader_reason": "not_combined_rate_shaped", "text": "Alice types 5 words per minute and Bob types 3 words per minute. How many words does Alice type in 3 minutes?", "gold": null, "notes": "Two same-unit rates present, but the query attributes the answer to a SINGLE agent ('how many words does Alice type') -> a single-rate question with a distractor rate, NOT a combined query. CMB must step aside (not_combined_rate_shaped), never the substantive combine_mode_ambiguous. (CMB-lookback hazard H3: the combined-query gate must exclude single-agent 'does <Agent>' attribution; the combined cases use 'do they' / 'are produced' / 'does it'.)"}

View file

@ -54,6 +54,10 @@ _EFF_RATE_QUERY = re.compile(r"\bwhat\b[^.?!]*\b(?:net|combined)\b[^.?!]*\brate\
_CLOCK = re.compile(r"\b\d+\s*(?:am|pm)\b|\bo'?clock\b") _CLOCK = re.compile(r"\b\d+\s*(?:am|pm)\b|\bo'?clock\b")
# Two conjoined agents in the premise ("Anna and Ben …") — the strong signal for missing_second_rate. # Two conjoined agents in the premise ("Anna and Ben …") — the strong signal for missing_second_rate.
_AGENT_CONJ = re.compile(r"\b[a-z]+\s+and\s+[a-z]+\b") _AGENT_CONJ = re.compile(r"\b[a-z]+\s+and\s+[a-z]+\b")
# A query that attributes the answer to a SINGLE agent ("how many words does Alice type") rather
# than the combination ("how many … do they …" / "… are produced"). "does it/this/that" is the
# combined process, not a single agent, so it is excluded.
_SINGLE_AGENT_QUERY = re.compile(r"\bdoes\s+(?!it\b|this\b|that\b|they\b)[a-z]")
#: Whole-word fill / drain verbs. Detected PER RATE CLAUSE (in the clause's own lead-in), never #: Whole-word fill / drain verbs. Detected PER RATE CLAUSE (in the clause's own lead-in), never
#: globally — an incidental "the drain stays closed" or a "draining" that governs a different clause #: globally — an incidental "the drain stays closed" or a "draining" that governs a different clause
@ -159,7 +163,11 @@ def read_combined_rate_problem(text: str) -> CombinedRateProblem | Refusal:
if u0 == u1: if u0 == u1:
hm = _HOW_MANY.search(t) hm = _HOW_MANY.search(t)
asked0 = _singular(hm.group(1)) if hm else None asked0 = _singular(hm.group(1)) if hm else None
if asked0 in u0 or _EFF_RATE_QUERY.search(t, rate_spans[-1][1]) is not None: combined_q = asked0 in u0 or _EFF_RATE_QUERY.search(t, rate_spans[-1][1]) is not None
# A query attributing the answer to a SINGLE agent ("how many words does Alice type") is a
# single-rate question with a distractor rate, NOT a combined query -> step aside, never
# claim the substantive combine_mode_ambiguous on it (router hygiene).
if combined_q and _SINGLE_AGENT_QUERY.search(t, rate_spans[-1][1]) is None:
return Refusal("combine_mode_ambiguous", "two same-unit rates and a combined query but no mode cue") return Refusal("combine_mode_ambiguous", "two same-unit rates and a combined query but no mode cue")
return Refusal("not_combined_rate_shaped", "two rates that are not a clean combined-rate problem") return Refusal("not_combined_rate_shaped", "two rates that are not a clean combined-rate problem")
# A combination cue IS present — now substantive combined-rate refusals are legitimate. # A combination cue IS present — now substantive combined-rate refusals are legitimate.

View file

@ -43,8 +43,8 @@ def _solved() -> dict:
def test_run_validates_all_gold() -> None: def test_run_validates_all_gold() -> None:
r = run() r = run()
assert r["invalid"] == 0 and r["valid"] == r["total"] == 18 assert r["invalid"] == 0 and r["valid"] == r["total"] == 19
assert r["by_expect"] == {"solved": 6, "solver_refuses": 5, "reader_refuses": 7} assert r["by_expect"] == {"solved": 6, "solver_refuses": 5, "reader_refuses": 8}
def test_solved_grid_covers_every_mode_query_cell() -> None: def test_solved_grid_covers_every_mode_query_cell() -> None:

View file

@ -37,7 +37,7 @@ def test_reader_lane_is_wrong_zero_and_complete() -> None:
assert r["setup_wrong"] == 0 and r["reason_mismatch"] == 0 assert r["setup_wrong"] == 0 and r["reason_mismatch"] == 0
assert r["setup_refused"] == 0 assert r["setup_refused"] == 0
assert r["setup_correct"] == 11 # 6 solved + 5 solver_refuses (all parse) assert r["setup_correct"] == 11 # 6 solved + 5 solver_refuses (all parse)
assert r["refused_correct"] == 7 assert r["refused_correct"] == 8
def test_reads_every_well_formed_fixture_to_gold_signature() -> None: def test_reads_every_well_formed_fixture_to_gold_signature() -> None:
@ -312,6 +312,24 @@ def test_sequential_segments_with_loose_connector_step_aside() -> None:
assert isinstance(out, Refusal) and out.reason == "not_combined_rate_shaped" assert isinstance(out, Refusal) and out.reason == "not_combined_rate_shaped"
def test_single_agent_attribution_steps_aside() -> None:
# Two same-unit rates, but the query attributes the answer to ONE agent ("how many words does
# Alice type") -> single-rate question with a distractor, NOT a combined query. Must step aside,
# never claim the substantive combine_mode_ambiguous (CMB-lookback hazard H3). The genuinely
# combined cases ("are produced", "do they") still yield combine_mode_ambiguous.
for text in (
"Alice types 5 words per minute and Bob types 3 words per minute. How many words does Alice type in 3 minutes?",
"Pump A fills 5 liters per minute and pump B fills 3 liters per minute. How many liters does pump A add in 4 minutes?",
):
out = read_combined_rate_problem(text)
assert isinstance(out, Refusal) and out.reason == "not_combined_rate_shaped", text
# control: passive / combined attribution stays substantive
amb = read_combined_rate_problem(
"One machine runs at 3 widgets per minute. Another machine runs at 5 widgets per minute. How many widgets are produced in 4 minutes?"
)
assert isinstance(amb, Refusal) and amb.reason == "combine_mode_ambiguous"
def test_compared_rates_with_no_combined_query_step_aside() -> None: def test_compared_rates_with_no_combined_query_step_aside() -> None:
# Two same-unit rates that are COMPARED, not combined ("which is faster?" / "what is the # Two same-unit rates that are COMPARED, not combined ("which is faster?" / "what is the
# difference?"), are not a CMB problem -> step aside, never the substantive combine_mode_ambiguous. # difference?"), are not a CMB problem -> step aside, never the substantive combine_mode_ambiguous.

View file

@ -28,7 +28,7 @@ def test_solver_lane_is_wrong_zero_and_complete() -> None:
assert r["solved_wrong"] == 0 and r["refuse_wrong"] == 0 assert r["solved_wrong"] == 0 and r["refuse_wrong"] == 0
assert r["solved_correct"] == 6 assert r["solved_correct"] == 6
assert r["refuse_correct"] == 5 assert r["refuse_correct"] == 5
assert r["skipped_reader_refuses"] == 7 assert r["skipped_reader_refuses"] == 8
def test_solves_every_solved_fixture_to_gold() -> None: def test_solves_every_solved_fixture_to_gold() -> None: