docs(cmb): lookback review + fix single-agent-attribution hygiene hazard (H3)
The mandatory lookback before CMB-d (3+ PRs on one surface; next-phase boundary). A 5-dimension
read-only fan-out audit over the 10-item checklist found the CMB-a/b/c substrate composes cleanly:
25 solid / 4 drift / 8 gap / 1 reader hazard. Doc:
docs/analysis/cmb-lookback-review-2026-06-08.md.
Fixes the one real hazard (H3): a query attributing the answer to a SINGLE agent ('how many words
does Alice type' with a distractor second rate) wrongly claimed the substantive
combine_mode_ambiguous — a hygiene over-claim that would corrupt the shared router in CMB-d. The
combined-query gate now excludes single-agent 'does <Agent>' attribution (_SINGLE_AGENT_QUERY) and
steps aside as not_combined_rate_shaped; the genuinely-combined forms ('do they' / 'are produced' /
'does it') still yield combine_mode_ambiguous. Pinned by gold cmb-16 + a reader test.
The review also resolves the CMB-d failure-family classification (input_shape:
not_combined_rate_shaped; must_remain_refused incl. rate_unit_mismatch — DECISIVELY, the reader has
no dimension representation so it cannot tell convertible from incompatible units; proposal_allowed:
combine_mode_ambiguous, missing_second_rate) and enumerates the live CMB-d registry preconditions
(not_combined_rate_shaped mapping, rate_unit_mismatch/non_integer_solution string collisions,
non_positive_net_rate family, Organ/ALL_REASONS extension). Provenance false-alarm (stale local main
checkout) noted + verified against origin/main.
gold 19/19 (6/5/8); reader 11/0/0 + 8 refused-correct; solver 6/0 + 5/0; 0 hygiene breaches; 123
tests; serving + R1/R2/R3 unchanged.
This commit is contained in:
parent
84d3456e61
commit
a4d0c4ebf3
6 changed files with 178 additions and 5 deletions
146
docs/analysis/cmb-lookback-review-2026-06-08.md
Normal file
146
docs/analysis/cmb-lookback-review-2026-06-08.md
Normal file
|
|
@ -0,0 +1,146 @@
|
||||||
|
# CMB lookback review (CMB-a/b/c) — before CMB-d router wiring
|
||||||
|
|
||||||
|
**As of:** `main` @ `84d3456e` (CMB-a #650 · CMB-b #653 · CMB-c #654 all merged).
|
||||||
|
**Trigger:** the mandatory lookback before wiring a multi-rung substrate into shared infrastructure
|
||||||
|
(CLAUDE.md *Lookback Review Discipline* — "before starting the next phase of a multi-phase ADR" and
|
||||||
|
"after any 3+ PR sequence on the same surface"). CMB-d wires CMB into the shared
|
||||||
|
`route_setup` + contemplation + failure-family registry, where any latent CMB-c classification
|
||||||
|
mistake stops being local.
|
||||||
|
|
||||||
|
**Method.** A 5-dimension read-only fan-out audit (one agent per dimension, each reading the source
|
||||||
|
and running live probes) over a 10-item checklist, plus a synthesis pass. 38 findings: **25 solid /
|
||||||
|
4 drift / 8 gap / 1 reader hazard** (+ several CMB-d preconditions that are wiring, not CMB-c bugs).
|
||||||
|
|
||||||
|
**Verdict: ready for CMB-d once (a) the one reader hazard is fixed [done in this PR] and (b) the
|
||||||
|
registry preconditions below land in the CMB-d PR itself.**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Provenance note (a false alarm, resolved)
|
||||||
|
|
||||||
|
The synthesis pass raised a "live main is CMB-a only; reader/solver are unmerged" alarm. **This is a
|
||||||
|
stale-local-checkout artifact, not a real finding.** Merges land server-side via `gh`; the *local*
|
||||||
|
`main` checkout was never pulled, so the synthesis agent's `git` saw an old tree. Verified:
|
||||||
|
`git cat-file -e origin/main:generate/combined_rate_comprehension/reader.py` → present at
|
||||||
|
`84d3456e`; `solver.py` likewise. The per-dimension audit agents read the worktree (= the merged
|
||||||
|
code) and their findings stand. Discount the provenance hazard.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Hazard (1) — fixed in this PR
|
||||||
|
|
||||||
|
**H3 · single-agent-attribution over-claims `combine_mode_ambiguous`.** Two same-unit rates with a
|
||||||
|
query that attributes the answer to **one** agent — *"Alice types 5 words per minute and Bob types 3
|
||||||
|
words per minute. How many words does **Alice** type in 3 minutes?"* — is a single-rate question
|
||||||
|
with a distractor rate, not a combined query. The reader claimed the substantive
|
||||||
|
`combine_mode_ambiguous`; on the shared router (CMB-d) that is a substantive-refusal-on-foreign-text
|
||||||
|
= a hygiene breach.
|
||||||
|
|
||||||
|
**Fix (this PR):** the combined-query gate now excludes a single-agent `does <Agent>` attribution
|
||||||
|
(`_SINGLE_AGENT_QUERY`), so single-agent text steps aside as `not_combined_rate_shaped`. The
|
||||||
|
genuinely-combined forms (`… do they …`, `… are produced`, `does it`) still yield
|
||||||
|
`combine_mode_ambiguous`. Pinned by gold fixture **`cmb-16`** + `test_single_agent_attribution_steps_aside`.
|
||||||
|
|
||||||
|
A reader that **steps aside** on ambiguity is safe (wrong=0 preserved); only over-reads to a wrong
|
||||||
|
answer or substantive-refusals-on-foreign-text are hazards. H3 was the only one. The
|
||||||
|
comparison-query, sequential-duration, decimal-rate, incidental-drain, and reciprocal step-asides
|
||||||
|
are all correctly handled (confirmed) and are **not** hazards.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Solid (the substrate that holds)
|
||||||
|
|
||||||
|
- **Division of labor (model/reader/solver).** The reader **parses** every solver-boundary setup
|
||||||
|
(`eff ≤ 0`, non-integer) and never intercepts a solver-owned refusal — `non_positive_net_rate` and
|
||||||
|
`non_integer_solution` appear nowhere in `reader.py`. The model permits `effective_rate ≤ 0` by
|
||||||
|
construction; the solver owns it. `run_reader` 11/0/0 + (now) 8 refused-correct; `run_solver`
|
||||||
|
6/0 + 5/0. No reader-owned malformed setup reaches the solver (the unit-mismatch guard fires
|
||||||
|
before `RateUnit` construction; the model has one unit slot).
|
||||||
|
- **2×2 domain-entry grid.** All four cells correct; every substantive refusal is gated behind
|
||||||
|
positive CMB-shaped evidence (the N6 over-proposing invariant) — `three_or_more_rates` /
|
||||||
|
`rate_unit_mismatch` only after a cue; `combine_mode_ambiguous` only with a same-unit combined
|
||||||
|
query; `missing_second_rate` only with a cooperative cue + two conjoined agents.
|
||||||
|
- **Foreign-organ hygiene.** 0 substantive refusals across all 36 R1/R2/R3 gold (every foreign
|
||||||
|
fixture → `not_combined_rate_shaped`), and across fresh foreign probes (comparison questions, R1
|
||||||
|
arithmetic, R2 prose, R3 single-rate with incidental cue words).
|
||||||
|
- **Reader correctness (the 3 adversarial-round classes hold).** Difference subtraction is
|
||||||
|
semantic-role based (fill = minuend, drain = subtrahend) not text-order; sequential segments step
|
||||||
|
aside (incl. loose connectors); decimals don't partial-parse.
|
||||||
|
- **Off-serving.** No CMB file imports `generate.derivation` / `core.reliability_gate` (AST-checked).
|
||||||
|
- **Cross-rung composition.** The reader's setups round-trip through the oracle's
|
||||||
|
`combined_rate_setup_signature`; the solver consumes exactly the reader's output type; the closed
|
||||||
|
taxonomies are consistent across model/reader/solver/oracle.
|
||||||
|
|
||||||
|
## Drift / gaps (no current wrong=0 risk; most are CMB-d's to resolve)
|
||||||
|
|
||||||
|
- `empty` refusal (blank input) is not in the oracle `READER_REASONS` — unreachable from prose
|
||||||
|
routing; add only if a blank-input gold case is ever introduced.
|
||||||
|
- `_AGENT_CONJ` matches any `word and word` bigram (not just proper-noun pairs); `_LEADIN = 25` is an
|
||||||
|
undocumented bound. Both are conservative (a miss → safe step-aside), not hazards.
|
||||||
|
- The remaining items are **CMB-d preconditions** (below), not CMB-c defects.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CMB-d failure-family classification (the decision)
|
||||||
|
|
||||||
|
The load-bearing output. Reasons map to families as:
|
||||||
|
|
||||||
|
| Family bucket | Reasons | Rationale |
|
||||||
|
|---|---|---|
|
||||||
|
| **input_shape** (no proposal) | `not_combined_rate_shaped` | the step-aside reason — the router-hygiene gate **requires** it be mapped, or every CMB foreign refusal fails the invariant |
|
||||||
|
| **must_remain_refused** (no proposal) | `rate_unit_mismatch`*, `three_or_more_rates`, `clock_interval_deferred`, `reciprocal_work_rate_deferred`, `non_positive_net_rate` (solver), `non_integer_solution` (solver) | correct wrong=0 boundaries / deferred-capability boundaries, not coverage gaps |
|
||||||
|
| **proposal_allowed** | `combine_mode_ambiguous`, `missing_second_rate` | genuine, semantically-coherent coverage gaps — fired only **after** positive CMB recognition, and **before** the solver, so neither can emit a wrong answer |
|
||||||
|
|
||||||
|
**\* `rate_unit_mismatch` stays `must_remain_refused` for v1 — decisively.** It fires for two
|
||||||
|
structurally different sub-cases under one code: (A) dimensionally-**incompatible** units
|
||||||
|
(`rooms/hour` vs `liters/minute` — no conversion exists) and (B) a duration-unit ≠ rate-denominator
|
||||||
|
case. The reader has **no dimension representation** (`units.py` compares only `(numerator,
|
||||||
|
denominator)` string equality) and **cannot** distinguish convertible (`gallons/min` vs
|
||||||
|
`gallons/hour`) from incompatible. Marking it `proposal_allowed` would emit a "try conversion"
|
||||||
|
proposal for the incompatible case = a wrong=0 hazard if ratified. **Do not** introduce a narrower
|
||||||
|
`rate_unit_mismatch_convertible` family until the reader gains a dimension registry **and** the two
|
||||||
|
sub-cases fire under distinct codes **and** a separate conversion gold lane exists.
|
||||||
|
|
||||||
|
**Note on `missing_second_rate`:** it is `must_remain_refused`-by-default *and* `proposal_allowed`
|
||||||
|
in the sense above — it is an *under-specified-input* boundary (no reader enhancement can infer the
|
||||||
|
missing rate), distinct in kind from the math-impossibility boundaries. The proposal it licenses is
|
||||||
|
"what is the second agent's rate?", not a rule-learning proposal.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CMB-d risk list (preconditions, to land in the CMB-d PR itself)
|
||||||
|
|
||||||
|
These are **live registry** facts verified against `main`; each must be handled in the same PR that
|
||||||
|
adds `classify_cmb` to `route_setup`:
|
||||||
|
|
||||||
|
1. **`not_combined_rate_shaped` is unmapped** in `failure_family.py` (`family_for_reason → None`).
|
||||||
|
The moment CMB joins `test_router_organ_hygiene.py`, every CMB foreign refusal fails the
|
||||||
|
"must map to `input_shape`" assertion. → add it to the `input_shape` family's `refusal_reasons`.
|
||||||
|
2. **Shared `rate_unit_mismatch` string collision.** R3 emits the same string → maps to R3's
|
||||||
|
`unsupported_rate_duration` **growth** family (`proposal_allowed`, `owner=r3`). A CMB unit-mismatch
|
||||||
|
(a hard boundary) would be mis-routed to a wrong-type R3 proposal. → CMB must emit a **distinct**
|
||||||
|
string (e.g. `cmb_rate_unit_mismatch`) and register it `must_remain_refused`.
|
||||||
|
3. **`non_positive_net_rate` is unmapped** → `ContemplationResult.family` silently `None` on a CMB
|
||||||
|
solver refusal. → add a `cmb_non_positive_net` family (`must_remain_refused`).
|
||||||
|
4. **`non_integer_solution` owner mislabel.** The CMB solver shares R2's string → family `owner=r2`.
|
||||||
|
→ emit a CMB-distinct string or extend the owner to credit CMB.
|
||||||
|
5. **`Organ` Literal + `ALL_REASONS` lack CMB.** → add `r4_combined_rate` (name TBD) to the `Organ`
|
||||||
|
Literal and extend `ALL_REASONS` with every CMB reader+solver reason, or
|
||||||
|
`test_registry_covers_the_whole_refusal_surface` won't catch an unregistered CMB reason.
|
||||||
|
6. **Off-serving assertion.** Add the per-module AST off-serving test for `classify_cmb` (substring
|
||||||
|
scans false-negative on docstring mentions).
|
||||||
|
7. **Exactly-one-`setup_correct` contract.** Re-confirm `route_setup` still selects a unique organ
|
||||||
|
when CMB joins — CMB must not make any R1/R2/R3 problem ambiguous, nor claim one (the hygiene
|
||||||
|
audit shows it steps aside on all R1/R2/R3 gold, which protects this).
|
||||||
|
|
||||||
|
**CMB-d's claim stays narrow:** "CMB participates in routing + typed contemplation without
|
||||||
|
corrupting R1/R2/R3 or serving." No new capability (that was CMB-c).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Net
|
||||||
|
|
||||||
|
The CMB-a/b/c substrate composes cleanly. One real reader hazard (H3) is fixed here with a ruler
|
||||||
|
fixture + test. The remaining items are CMB-d wiring preconditions, now enumerated with the family
|
||||||
|
classification resolved. Lane state after this PR: gold **19/19** (6/5/8), reader 11/0/0 + 8
|
||||||
|
refused-correct, solver 6/0 + 5/0, 0 hygiene breaches, serving + R1/R2/R3 unchanged.
|
||||||
|
|
@ -16,3 +16,4 @@
|
||||||
{"id": "cmb-13-reciprocal-work-rate", "expect": "reader_refuses", "reader_reason": "reciprocal_work_rate_deferred", "text": "Anna can paint a house in 3 hours, and Ben can paint the same house in 6 hours. Working together, how many hours do they take?", "gold": null, "notes": "Durations to complete a whole job, NOT explicit per-unit rates — the classic reciprocal work-rate form 1/(1/3 + 1/6). A different representation (reciprocal rates, rational arithmetic). REFUSE (reciprocal_work_rate_deferred) — deferred."}
|
{"id": "cmb-13-reciprocal-work-rate", "expect": "reader_refuses", "reader_reason": "reciprocal_work_rate_deferred", "text": "Anna can paint a house in 3 hours, and Ben can paint the same house in 6 hours. Working together, how many hours do they take?", "gold": null, "notes": "Durations to complete a whole job, NOT explicit per-unit rates — the classic reciprocal work-rate form 1/(1/3 + 1/6). A different representation (reciprocal rates, rational arithmetic). REFUSE (reciprocal_work_rate_deferred) — deferred."}
|
||||||
{"id": "cmb-14-clock-interval", "expect": "reader_refuses", "reader_reason": "clock_interval_deferred", "text": "A pipe fills a tank at 5 liters per minute and another fills it at 3 liters per minute, running together from 2 pm to 5 pm. How many liters are in the tank?", "gold": null, "notes": "Elapsed clock time (2 pm to 5 pm), not an explicit duration. REFUSE (clock_interval_deferred) — deferred (the CMB twin of R3's temporal_state)."}
|
{"id": "cmb-14-clock-interval", "expect": "reader_refuses", "reader_reason": "clock_interval_deferred", "text": "A pipe fills a tank at 5 liters per minute and another fills it at 3 liters per minute, running together from 2 pm to 5 pm. How many liters are in the tank?", "gold": null, "notes": "Elapsed clock time (2 pm to 5 pm), not an explicit duration. REFUSE (clock_interval_deferred) — deferred (the CMB twin of R3's temporal_state)."}
|
||||||
{"id": "cmb-15-not-combined-shaped", "expect": "reader_refuses", "reader_reason": "not_combined_rate_shaped", "text": "A car travels 60 miles per hour for 3 hours. How many miles does it travel?", "gold": null, "notes": "A single explicit rate with no second rate and no combination cue — single-rate R3 territory, not CMB's. CMB must step aside, NOT claim a substantive boundary. REFUSE (not_combined_rate_shaped) -> maps to the input_shape family in CMB-d (router-organ-hygiene invariant)."}
|
{"id": "cmb-15-not-combined-shaped", "expect": "reader_refuses", "reader_reason": "not_combined_rate_shaped", "text": "A car travels 60 miles per hour for 3 hours. How many miles does it travel?", "gold": null, "notes": "A single explicit rate with no second rate and no combination cue — single-rate R3 territory, not CMB's. CMB must step aside, NOT claim a substantive boundary. REFUSE (not_combined_rate_shaped) -> maps to the input_shape family in CMB-d (router-organ-hygiene invariant)."}
|
||||||
|
{"id": "cmb-16-single-agent-attribution", "expect": "reader_refuses", "reader_reason": "not_combined_rate_shaped", "text": "Alice types 5 words per minute and Bob types 3 words per minute. How many words does Alice type in 3 minutes?", "gold": null, "notes": "Two same-unit rates present, but the query attributes the answer to a SINGLE agent ('how many words does Alice type') -> a single-rate question with a distractor rate, NOT a combined query. CMB must step aside (not_combined_rate_shaped), never the substantive combine_mode_ambiguous. (CMB-lookback hazard H3: the combined-query gate must exclude single-agent 'does <Agent>' attribution; the combined cases use 'do they' / 'are produced' / 'does it'.)"}
|
||||||
|
|
|
||||||
|
|
@ -54,6 +54,10 @@ _EFF_RATE_QUERY = re.compile(r"\bwhat\b[^.?!]*\b(?:net|combined)\b[^.?!]*\brate\
|
||||||
_CLOCK = re.compile(r"\b\d+\s*(?:am|pm)\b|\bo'?clock\b")
|
_CLOCK = re.compile(r"\b\d+\s*(?:am|pm)\b|\bo'?clock\b")
|
||||||
# Two conjoined agents in the premise ("Anna and Ben …") — the strong signal for missing_second_rate.
|
# Two conjoined agents in the premise ("Anna and Ben …") — the strong signal for missing_second_rate.
|
||||||
_AGENT_CONJ = re.compile(r"\b[a-z]+\s+and\s+[a-z]+\b")
|
_AGENT_CONJ = re.compile(r"\b[a-z]+\s+and\s+[a-z]+\b")
|
||||||
|
# A query that attributes the answer to a SINGLE agent ("how many words does Alice type") rather
|
||||||
|
# than the combination ("how many … do they …" / "… are produced"). "does it/this/that" is the
|
||||||
|
# combined process, not a single agent, so it is excluded.
|
||||||
|
_SINGLE_AGENT_QUERY = re.compile(r"\bdoes\s+(?!it\b|this\b|that\b|they\b)[a-z]")
|
||||||
|
|
||||||
#: Whole-word fill / drain verbs. Detected PER RATE CLAUSE (in the clause's own lead-in), never
|
#: Whole-word fill / drain verbs. Detected PER RATE CLAUSE (in the clause's own lead-in), never
|
||||||
#: globally — an incidental "the drain stays closed" or a "draining" that governs a different clause
|
#: globally — an incidental "the drain stays closed" or a "draining" that governs a different clause
|
||||||
|
|
@ -159,7 +163,11 @@ def read_combined_rate_problem(text: str) -> CombinedRateProblem | Refusal:
|
||||||
if u0 == u1:
|
if u0 == u1:
|
||||||
hm = _HOW_MANY.search(t)
|
hm = _HOW_MANY.search(t)
|
||||||
asked0 = _singular(hm.group(1)) if hm else None
|
asked0 = _singular(hm.group(1)) if hm else None
|
||||||
if asked0 in u0 or _EFF_RATE_QUERY.search(t, rate_spans[-1][1]) is not None:
|
combined_q = asked0 in u0 or _EFF_RATE_QUERY.search(t, rate_spans[-1][1]) is not None
|
||||||
|
# A query attributing the answer to a SINGLE agent ("how many words does Alice type") is a
|
||||||
|
# single-rate question with a distractor rate, NOT a combined query -> step aside, never
|
||||||
|
# claim the substantive combine_mode_ambiguous on it (router hygiene).
|
||||||
|
if combined_q and _SINGLE_AGENT_QUERY.search(t, rate_spans[-1][1]) is None:
|
||||||
return Refusal("combine_mode_ambiguous", "two same-unit rates and a combined query but no mode cue")
|
return Refusal("combine_mode_ambiguous", "two same-unit rates and a combined query but no mode cue")
|
||||||
return Refusal("not_combined_rate_shaped", "two rates that are not a clean combined-rate problem")
|
return Refusal("not_combined_rate_shaped", "two rates that are not a clean combined-rate problem")
|
||||||
# A combination cue IS present — now substantive combined-rate refusals are legitimate.
|
# A combination cue IS present — now substantive combined-rate refusals are legitimate.
|
||||||
|
|
|
||||||
|
|
@ -43,8 +43,8 @@ def _solved() -> dict:
|
||||||
|
|
||||||
def test_run_validates_all_gold() -> None:
|
def test_run_validates_all_gold() -> None:
|
||||||
r = run()
|
r = run()
|
||||||
assert r["invalid"] == 0 and r["valid"] == r["total"] == 18
|
assert r["invalid"] == 0 and r["valid"] == r["total"] == 19
|
||||||
assert r["by_expect"] == {"solved": 6, "solver_refuses": 5, "reader_refuses": 7}
|
assert r["by_expect"] == {"solved": 6, "solver_refuses": 5, "reader_refuses": 8}
|
||||||
|
|
||||||
|
|
||||||
def test_solved_grid_covers_every_mode_query_cell() -> None:
|
def test_solved_grid_covers_every_mode_query_cell() -> None:
|
||||||
|
|
|
||||||
|
|
@ -37,7 +37,7 @@ def test_reader_lane_is_wrong_zero_and_complete() -> None:
|
||||||
assert r["setup_wrong"] == 0 and r["reason_mismatch"] == 0
|
assert r["setup_wrong"] == 0 and r["reason_mismatch"] == 0
|
||||||
assert r["setup_refused"] == 0
|
assert r["setup_refused"] == 0
|
||||||
assert r["setup_correct"] == 11 # 6 solved + 5 solver_refuses (all parse)
|
assert r["setup_correct"] == 11 # 6 solved + 5 solver_refuses (all parse)
|
||||||
assert r["refused_correct"] == 7
|
assert r["refused_correct"] == 8
|
||||||
|
|
||||||
|
|
||||||
def test_reads_every_well_formed_fixture_to_gold_signature() -> None:
|
def test_reads_every_well_formed_fixture_to_gold_signature() -> None:
|
||||||
|
|
@ -312,6 +312,24 @@ def test_sequential_segments_with_loose_connector_step_aside() -> None:
|
||||||
assert isinstance(out, Refusal) and out.reason == "not_combined_rate_shaped"
|
assert isinstance(out, Refusal) and out.reason == "not_combined_rate_shaped"
|
||||||
|
|
||||||
|
|
||||||
|
def test_single_agent_attribution_steps_aside() -> None:
|
||||||
|
# Two same-unit rates, but the query attributes the answer to ONE agent ("how many words does
|
||||||
|
# Alice type") -> single-rate question with a distractor, NOT a combined query. Must step aside,
|
||||||
|
# never claim the substantive combine_mode_ambiguous (CMB-lookback hazard H3). The genuinely
|
||||||
|
# combined cases ("are produced", "do they") still yield combine_mode_ambiguous.
|
||||||
|
for text in (
|
||||||
|
"Alice types 5 words per minute and Bob types 3 words per minute. How many words does Alice type in 3 minutes?",
|
||||||
|
"Pump A fills 5 liters per minute and pump B fills 3 liters per minute. How many liters does pump A add in 4 minutes?",
|
||||||
|
):
|
||||||
|
out = read_combined_rate_problem(text)
|
||||||
|
assert isinstance(out, Refusal) and out.reason == "not_combined_rate_shaped", text
|
||||||
|
# control: passive / combined attribution stays substantive
|
||||||
|
amb = read_combined_rate_problem(
|
||||||
|
"One machine runs at 3 widgets per minute. Another machine runs at 5 widgets per minute. How many widgets are produced in 4 minutes?"
|
||||||
|
)
|
||||||
|
assert isinstance(amb, Refusal) and amb.reason == "combine_mode_ambiguous"
|
||||||
|
|
||||||
|
|
||||||
def test_compared_rates_with_no_combined_query_step_aside() -> None:
|
def test_compared_rates_with_no_combined_query_step_aside() -> None:
|
||||||
# Two same-unit rates that are COMPARED, not combined ("which is faster?" / "what is the
|
# Two same-unit rates that are COMPARED, not combined ("which is faster?" / "what is the
|
||||||
# difference?"), are not a CMB problem -> step aside, never the substantive combine_mode_ambiguous.
|
# difference?"), are not a CMB problem -> step aside, never the substantive combine_mode_ambiguous.
|
||||||
|
|
|
||||||
|
|
@ -28,7 +28,7 @@ def test_solver_lane_is_wrong_zero_and_complete() -> None:
|
||||||
assert r["solved_wrong"] == 0 and r["refuse_wrong"] == 0
|
assert r["solved_wrong"] == 0 and r["refuse_wrong"] == 0
|
||||||
assert r["solved_correct"] == 6
|
assert r["solved_correct"] == 6
|
||||||
assert r["refuse_correct"] == 5
|
assert r["refuse_correct"] == 5
|
||||||
assert r["skipped_reader_refuses"] == 7
|
assert r["skipped_reader_refuses"] == 8
|
||||||
|
|
||||||
|
|
||||||
def test_solves_every_solved_fixture_to_gold() -> None:
|
def test_solves_every_solved_fixture_to_gold() -> None:
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue