Merge pull request #534 from AssetOverflow/codex/ntimes-completeness-guard
Harden no-reference n-times comparative guard
This commit is contained in:
commit
8327c6bfa1
7 changed files with 877 additions and 0 deletions
49
HANDOFF-gpt55-2026-06-03.md
Normal file
49
HANDOFF-gpt55-2026-06-03.md
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
# HANDOFF gpt55 2026-06-03
|
||||
|
||||
Branch: `codex/ntimes-completeness-guard`
|
||||
|
||||
Final commit SHA: reported in the thread final after commit creation.
|
||||
|
||||
## File Status
|
||||
|
||||
| file | path | status | purpose | ready-to-merge |
|
||||
|---|---|---|---|---|
|
||||
| recognizer anchor injector | `generate/recognizer_anchor_inject.py` | FINAL | Refusal-only guard preventing no-reference `<N> times ...` comparative multipliers from being injected as `CandidateInitial(value=N, unit="times")`. | Y |
|
||||
| completeness guard tests | `tests/test_candidate_graph_completeness_guard.py` | FINAL | Pins the §9 hard negative matrix, with solve/refusal controls. | Y |
|
||||
| question-layer gap survey | `docs/analysis/question-layer-gap-survey.md` | FINAL | Canonical audited 44-refusal partition and backlog interpretation. | Y |
|
||||
| solver operation coverage | `docs/analysis/solver-operation-coverage.md` | FINAL | Canonical read-only audit of existing solver op coverage and representation gaps. | Y |
|
||||
| composition capability scope | `docs/analysis/composition-capability-scope.md` | FINAL | Canonical execution-lane v2 scope for ADR-0174 Phase 5b emission/representation and §9 guard precondition. | Y |
|
||||
| comprehension primitive inventory | `docs/analysis/comprehension-primitive-inventory.md` | FINAL | Canonical consolidated inventory and cross-subject leverage map from the execution lane. | Y |
|
||||
| handoff | `HANDOFF-gpt55-2026-06-03.md` | FINAL | Merge/readiness manifest for this thread. | Y |
|
||||
|
||||
## Validation
|
||||
|
||||
Completed on branch `codex/ntimes-completeness-guard`:
|
||||
|
||||
- `uv run python -m pytest tests/test_candidate_graph_completeness_guard.py -q`
|
||||
- `21 passed`
|
||||
- `uv run python -m pytest tests/test_adr_0131_G2_comparatives.py tests/test_adr_0131_G2a_comparative_verb_widening.py -q`
|
||||
- `30 passed`
|
||||
- `uv run python -m pytest tests/test_aggregate_total_question_forms.py tests/test_discrete_count_open_noun_class.py tests/test_adr_0163_d2_discrete_count_injection.py -q`
|
||||
- `59 passed`
|
||||
- Train-sample probe through `generate.math_candidate_graph.parse_and_solve`
|
||||
- `6 correct / 44 refused / 0 wrong`
|
||||
- `refused_nonzero_count = 0`
|
||||
- admitted IDs: `0003`, `0014`, `0018`, `0021`, `0024`, `0042`
|
||||
|
||||
## New Since Last Message
|
||||
|
||||
- Broadened the §9 guard from phrase-specific `times as many|more` matching to the structural captured shape: cardinal immediately followed by `times` inside the discrete-count injector.
|
||||
- Added no-reference hard negatives for `3 times the number of apples` and `3 times the apples`.
|
||||
- Added explicit safe-refusal controls for no-reference `twice`, no-reference `double`, and case 605.
|
||||
- Added canonical execution-lane `composition-capability-scope.md`.
|
||||
- Added canonical execution-lane `comprehension-primitive-inventory.md`.
|
||||
- Added this handoff manifest.
|
||||
|
||||
## Not Done / WIP
|
||||
|
||||
- No 5b emission/representation slice was implemented.
|
||||
- No <=20-case validation sub-corpus was authored.
|
||||
- No solver operation kinds or binding-graph node types were added.
|
||||
- No serving/eval/claims-ledger files were changed.
|
||||
- Superseded `0001` / `0002` / `0003` patch files from the Opus handoff directory were not committed.
|
||||
244
docs/analysis/composition-capability-scope.md
Normal file
244
docs/analysis/composition-capability-scope.md
Normal file
|
|
@ -0,0 +1,244 @@
|
|||
<!-- CANONICAL | composition-capability-scope.md | updated 2026-06-03 | execution lane | supersedes prior copies -->
|
||||
|
||||
# Composition-Capability Scope — v2 (re-anchored to shipped reality)
|
||||
|
||||
Status: Proposed analysis / build-scoping. No serving or eval edits. Source of
|
||||
truth: `docs/claims_ledger.md`, `evals/gsm8k_math/train_sample/v1/report.json`,
|
||||
ADR-0174 **read in full (644 lines, incl. Implementation Notes + amended Phase 5,
|
||||
lines 300–533)**. Every empirical claim below reproduced live against `main` @ `3e29559`.
|
||||
|
||||
> **v1 correction.** v1 aimed the plan at "land Phase 1; the unlanded value is
|
||||
> Phase 3/4." That was wrong — scoped from ADR-0174's forward sections only,
|
||||
> missing the ~225 lines documenting that **Phases 1–4 already shipped and are
|
||||
> wired into serving**. Verified on `main`:
|
||||
> `generate/comprehension/{state,lookback,contemplate,constraint_propagation}.py`
|
||||
> exist with `HYPOTHESIS_CAP`/`open_hypotheses`/`Hypothesis`/`UnknownHeld`/
|
||||
> `reevaluate`/`contemplate`, and `reevaluate`+`contemplate` are imported and
|
||||
> invoked inside `generate/math_candidate_graph.py::parse_and_solve`. Metric still
|
||||
> **6/44/0**. v2 re-anchors to the ADR's own amended Phase 5.
|
||||
|
||||
## 0. The finding that scopes everything (corrected)
|
||||
|
||||
The held-hypothesis machinery (ADR-0174 Phases 1–4) is **live in serving and the
|
||||
metric did not move.** A live `parse_and_solve` audit of all 44 refusals (§1)
|
||||
locates the wall precisely: **44/44 refuse at `branches_enumerated = 0` — upstream
|
||||
of the solver, which is never reached.** So the lift is **not** more reader
|
||||
machinery, and **not** solver operation coverage (the 8 op-kinds are never reached;
|
||||
see §1). The wall is **emission/representation**: the recognizer/parser/binding
|
||||
layer cannot construct an admissible multi-step candidate for these shapes, so
|
||||
nothing is fed to the (already-capable) solver. ADR-0174's "removing the 3–5
|
||||
narrowness layers per case" is that emission/representation work.
|
||||
|
||||
This inverts v1's thesis *and* supersedes the intermediate "operation coverage"
|
||||
thesis: the composition wall is at **emission/representation upstream of the
|
||||
solver**, not at the operation layer.
|
||||
|
||||
## 1. Verified current state on `main`
|
||||
|
||||
| Layer | State |
|
||||
|---|---|
|
||||
| Held-hypothesis reader (P1–P4: state, constraint-propagation, lookback, contemplate) | **Shipped, wired into serving** candidate-graph |
|
||||
| `HYPOTHESIS_CAP`, vault>packs>audit precedence | already set/enforced (0174 OQ#1/#3 moot) |
|
||||
| `lifecycle.py` GSM8K-scoring dispatch | **inert** (admits 0/50) — retirable in 5a |
|
||||
| `lifecycle.py` / `audit.py` reader surface | **load-bearing** for the ADR-0172 teaching corridor — **keep** |
|
||||
| Solver operation kinds | **8 already exist** (verified by discovery): `add`, `subtract`, `transfer`, `multiply`, `divide`, `apply_rate`, `compare_additive`, `compare_multiplicative`. Missing arithmetic op kinds is **not** the blocker. |
|
||||
| **Live audit of all 44 refusals (ran `parse_and_solve` on `main`)** | **44/44 refuse at `branches_enumerated = 0`** — i.e. *upstream of the solver*. No branch is built, so the 8 operation kinds are **never reached**. `rate`/`ratio` also exist as first-class `SEMANTIC_ROLES`/`QUESTION_FORMS` (`binding_graph/model.py:42–65`); only `percent`/`accumulate` lack a named op-kind — moot while nothing reaches the solver. |
|
||||
| train_sample metric | **6 / 44 / 0** (`correct_min=10` not yet passed) |
|
||||
|
||||
## 2. The actual remaining work (ADR-0174 amended Phase 5)
|
||||
|
||||
### Phase 5a — retire the inert parallel parser (structural, ~0 lift)
|
||||
Retire the GSM8K-scoring-only inert dispatch (`_try_comprehension_reader` /
|
||||
`_try_reader_for_question`) + adapter + `use_reader` plumbing. Net **−1,038 LOC**
|
||||
(code + tests). **Keep** `lifecycle.py`/`audit.py` — their reader *refusals* feed
|
||||
the ADR-0172 math-contemplation teaching corridor (`teaching/math_*`,
|
||||
`evals/flywheel_demo`, `core/cli.py`). Low-risk; single-path serving.
|
||||
Acceptance: 6/44/0 byte-identical, capability-axis lanes 100% `wrong=0`,
|
||||
pinned-SHAs pass.
|
||||
|
||||
### Phase 5b — emission / representation buildout (semantic — the real lift)
|
||||
"This is where `correct` climbs toward 25. It is not a refactor." **Verified
|
||||
diagnosis (live `parse_and_solve` audit of all 44, on `main`):** every refusal
|
||||
occurs at `branches_enumerated = 0` — *upstream of the solver*. The 8 operation
|
||||
kinds are never reached. So 5b is **not** an operation-execution problem and
|
||||
**not** chiefly a new-primitive problem. It is an **emission/representation**
|
||||
problem: the recognizer/parser/binding layer cannot construct an admissible
|
||||
multi-step candidate for these shapes, so nothing is fed to the (already-capable)
|
||||
solver.
|
||||
|
||||
The 44 split into two emission failure modes (verified, matches `report.json`):
|
||||
- **32 "recognizer matched but produced no injection"** — an anchor fired but the
|
||||
hypothesis-emitter declined to construct a candidate (cannot represent the
|
||||
multi-clause / derived / scoped structure).
|
||||
- **12 "no admissible candidate"** — the parser produced nothing admissible.
|
||||
|
||||
**Sequencing implication:** fix emission/representation first; the
|
||||
`percent`/`accumulate` primitive question is **secondary** — answerable only after
|
||||
cases actually reach the solver. This *lowers* the wrong=0 risk: emitting existing
|
||||
ops behind the same admissibility predicates is lower-risk than new solver math.
|
||||
|
||||
**5b first PR (smallest verified-tractable slice):** pick the emission failure mode
|
||||
with the highest count whose representation is bounded — and prove it reaches branch
|
||||
enumeration on ≥1 case without breaking `wrong=0`. **But the §9 completeness-gate
|
||||
precondition lands before any emission PR.**
|
||||
|
||||
## 3. Test anchors (reusable, relabeled to 5a/5b)
|
||||
1. **Serving gate (`gsm8k_math`):** 5a byte-identical 6/44/0; 5b climbs
|
||||
(`correct ≥ 10` clears the ADR-0126 exit, target → 15 → 25), `wrong = 0` at
|
||||
every step.
|
||||
2. **Regression nets (100% `wrong=0`):** G1–G5, S1; `anti_regression`; case-0050 pin.
|
||||
3. **Determinism:** `trace_hash` over `open_hypotheses`; replay-equivalence.
|
||||
4. **Serving freeze:** `verify_lane_shas.py` passes each step.
|
||||
5. **Still-needed new asset (0174 OQ#5):** the curated ≤20-case multi-step
|
||||
validation sub-corpus — **unbuilt, still required for 5b measurement.** The §9
|
||||
hard negatives are the first deposits into it. Author before 5b measurement.
|
||||
|
||||
## 4. Corrected sequencing
|
||||
0. **Precondition (§9):** harden the comparative-multiplicative completeness tripwire so a dropped `<N>×` clause refuses (as `twice` does). 5b emission on these shapes is unsafe until this lands. Driver test is FINAL and RED on `main` (see §9).
|
||||
1. **5b gating analysis — done (§8):** the **emission/representation** audit ran and the 32 no-injection cases are sub-classified by representation. Highest-count bounded gap = **R1 derived/intermediate symbol (24/44)**.
|
||||
2. **Author the ≤20-case multi-step validation sub-corpus** (test anchor for 5b).
|
||||
3. **Phase 5a** (optional, can run in parallel — structural, low-risk).
|
||||
4. **Phase 5b** — build the **emitter/representation** for R1 (derived-symbol) on the §8 near-pure exemplars (prove ≥1 case reaches `branches_enumerated > 0` and admits), each behind §3 gates **and the §9 precondition**. Not new operations — the 8 exist and are unreached.
|
||||
|
||||
(Do **not** "land Phase 1" — shipped. Do **not** treat P3/P4 as the lever — shipped, metric flat.)
|
||||
|
||||
## 5. Relation to cross-subject testing
|
||||
5b maturing the math `DomainSolver`'s emission/representation is still the precondition
|
||||
for wiring symbolic_logic as arena #2; the held-hypothesis reader + disagreement
|
||||
gate remain the shared, arena-portable primitives. `cross_domain_transfer` /
|
||||
`monotonic_learning` (both exist, with `contract.md`/`holdouts/`) become live tests
|
||||
once arena #2 exists.
|
||||
|
||||
## 6. Honest ceiling (corrected)
|
||||
The ≥15 climb is a **5b** outcome (emission/representation), **not** a P4 outcome — P4
|
||||
shipped and we are at 6. Cases needing world knowledge (0040 legs) or representations 5b
|
||||
does not add stay refused = `wrong = 0` holding, not failure.
|
||||
|
||||
## 7. Open questions for the build lane
|
||||
1. **Answered by the live audit:** the binding constraint is emission upstream of
|
||||
the solver (44/44 at `branches_enumerated=0`), not operation execution. Remaining:
|
||||
rank the §8 R-classes by the size of the bounded-representation slice per class.
|
||||
2. Of the 44, which are emission-fixable with existing ops vs (a) genuinely need a
|
||||
`percent`/`accumulate` primitive *after* emission, vs (b) world-knowledge/out-of-
|
||||
scope permanent refusals (e.g. 0040 legs-per-animal). The audit gives the upstream
|
||||
verdict; this split needs per-case reader-trace reads.
|
||||
3. 5a pre-flight: confirm the exact inert-dispatch LOC and that no teaching-corridor
|
||||
consumer breaks when the GSM8K-scoring dispatch is retired.
|
||||
|
||||
---
|
||||
|
||||
## 8. Verified per-case representation classification (live audit, all 44)
|
||||
Structural reading of each of the 44 (what representation the emitter must build for
|
||||
emission to succeed), grounded in the case text + the live `parse_and_solve` locus.
|
||||
Multi-tagged; a case usually needs several. Frequency across the 44:
|
||||
|
||||
| Representation gap | # cases (multi-tagged) |
|
||||
|---|---:|
|
||||
| R5 — multi-step rate/duration/scalar | 27 |
|
||||
| R1 — derived/intermediate symbol | 24 |
|
||||
| R6 — percent/fraction mutation (no op-kind) | 18 |
|
||||
| R4 — accumulation/residual | 10 |
|
||||
| R2 — inverse target | 6 |
|
||||
| R3 — subset/partition scope | 3 |
|
||||
| R7 — world-knowledge (permanent refusal) | 1 |
|
||||
|
||||
**Highest-leverage gap: R1 (derived/intermediate symbol)** — 24/44 need to compute an
|
||||
intermediate quantity and reuse it downstream. Its lowest-arity exemplars are nearly
|
||||
pure R1: 0027 (Twitter = half of IG+FB), 0008 (total beads → ÷ per bracelet),
|
||||
0029 (keyboard = 3× mouse → sum), 0038 (×3 → sum). Contrastive proof it is the emission
|
||||
gap (parseable aggregate question form): `"Nicole has 400 cards. Cindy has 800 cards.
|
||||
How many cards do they have together?"` **admits (1200)**; `"…Cindy has twice as many
|
||||
cards. How many cards do they have together?"` reaches `branches_enumerated=1` but
|
||||
**refuses** (completeness: scalar `2.0` unconsumed). The stated-sum reaches the solver;
|
||||
the derived form does not. R7 (0040 legs-per-animal) is a permanent refusal — world
|
||||
knowledge.
|
||||
|
||||
**First 5b slice (recommended):** derived/intermediate-symbol emission, validated on the
|
||||
four near-pure exemplars above — move them from refused to admitted, `wrong=0` preserved.
|
||||
**But see §9 first — it is a hard precondition.**
|
||||
|
||||
## 9. ⚠ Latent `wrong=0` hazard surfaced by the live audit (gate gap)
|
||||
|
||||
Contrastive probes (not in the 50-case sample) surface a reproducible **admitted-wrong**
|
||||
path. **Use the parseable aggregate question form `"...do they have together?"`** — the
|
||||
short form `"How many apples together?"` refuses upstream at question-parse
|
||||
(`branches_enumerated=0`) and does NOT exercise this `be=1` completeness gap.
|
||||
|
||||
```
|
||||
"Tom has 7 apples. Jerry has 3 times as many apples. How many apples do they have together?"
|
||||
→ admitted=True, answer=7, branches_enumerated=1 (correct = 28) # base returned, clause dropped
|
||||
"...Jerry has five times as many apples. ..." → admitted, answer=7 (correct = 42)
|
||||
"...Jerry has twice as many apples. ..." → refused, be=1 (completeness: scalar 2.0 unconsumed) # safe
|
||||
```
|
||||
|
||||
**Broadened surface (all verified RED on `main`).** The hazard is the **no-reference**
|
||||
comparative-multiplier surface across all four connectives and both cardinal forms:
|
||||
|
||||
```
|
||||
<N> times as many <unit>
|
||||
<N> times more <unit>
|
||||
<N> times the number of <unit>
|
||||
<N> times the <unit>
|
||||
```
|
||||
…for `<N>` ∈ {digit 2/3/5/…, word two/three/five/…, N≥2}. Every one admits the base.
|
||||
`twice` / `double` / `double the` (no digit/cardinal) refuse safely.
|
||||
|
||||
**Root cause (verified by reading the emitted `MathProblemGraph`).** The admitted graph
|
||||
contains a spurious initial:
|
||||
`InitialPossession(entity='Jerry', quantity=Quantity(value=3, unit='times'))`.
|
||||
|
||||
1. `quantity_values_in_text` is **symmetric** (registers 2.0 for `twice` *and* 3.0 for
|
||||
`3 times`) — **not** a quantity-extraction asymmetry (this overturns an earlier guess).
|
||||
2. For the no-ref surface, **neither** serving comparative regex fires
|
||||
(`_COMPARE_MULT_ANCHOR_RE` / `_COMPARE_MULT_NTIMES_RE` both require an "as `<REF>`"
|
||||
tail the probe lacks). `comparatives.py::_N_TIMES_RE` is the **disjoint** `derivation/`
|
||||
reader — off the serving path — so it is *not* the locus either.
|
||||
3. Instead `recognizer_match.py::_match_discrete_count_statement` (open regex
|
||||
`_extract_discrete_count_re_open`) captures the multiplier cardinal as a **count**, and
|
||||
`recognizer_anchor_inject.py::_build_initial_from_discrete_count` builds
|
||||
`CandidateInitial(value=N, unit='times', entity=<actor>)` — note the unit is the literal
|
||||
`'times'` token, **not** the counted unit.
|
||||
4. That bogus initial **consumes** the scalar N, so completeness sees
|
||||
`uncovered = {N, base} − {N, base} = ∅` → admits. The answer sums only the counted unit
|
||||
(`apples`) and returns the base. `twice`/`double` carry no cardinal the discrete-count
|
||||
regex grabs → no spurious initial → scalar unconsumed → existing guard refuses.
|
||||
|
||||
**Recommended guard shape (refusal-only, `wrong=0` preserving).** Make the discrete-count
|
||||
recognizer **decline** when its cardinal sits in a
|
||||
`<N> times {as many | more | the number of | the} <unit>` comparative-multiplier context
|
||||
— equivalently, refuse to build a discrete-count initial whose unit is the literal
|
||||
`'times'`. The no-ref form then refuses (like no-ref `twice`) until real no-ref
|
||||
comparative-multiplicative *emission* lands in 5b. The fix MUST NOT regress the controls
|
||||
below.
|
||||
|
||||
**Controls (verified on `main`).**
|
||||
- **With-ref `<N> times` already solves — must stay green.** train_sample **case 0024**:
|
||||
*"Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on
|
||||
Thursday. Brooke does three times as many jumping jacks as Sidney. How many jumping
|
||||
jacks did Brooke do?"* → **438** (committed verdict: correct). Also `dice 3×` → 80.
|
||||
- **Existing safe refusals — must stay refused.** no-ref `twice`, no-ref `double the`,
|
||||
with-ref `twice` (Ivan/Jerry dice).
|
||||
|
||||
**Driver test (FINAL, RED on `main`).** `test_completeness_guard_ntimes_noref_hazard.py`
|
||||
— **10 no-ref hazard cases** (the broadened surface × digit/word) that MUST refuse, plus
|
||||
**2 with-ref must-solve controls** (0024 → 438, dice → 80) and **3 must-still-refuse
|
||||
controls**. Verified live: **10 failed, 5 passed** on `main` (hazard RED, controls green).
|
||||
This is the first concrete 5b PR, ahead of any R1 emission work.
|
||||
|
||||
**Scope — stated precisely.** This is a **latent gate gap, not a live-metric violation.**
|
||||
The two layers are distinct: the §1 wall is at `branches_enumerated=0`
|
||||
(emission/question-parse); this hazard lives at `branches_enumerated=1` (the completeness
|
||||
gate). The real 44 all refuse at `be=0` — upstream of `be=1` — so none reach this gap
|
||||
today; that is *why* it is latent. train_sample 6/44/0 is reproduced live. The full-test
|
||||
`0/0/1319` is **not** re-verified here — it is a sealed, recorded measurement per
|
||||
`docs/claims_ledger.md` row A / `ADR-0119.7` (ciphertext
|
||||
`evals/gsm8k_math/holdouts/v1/cases.jsonl.age`), not a CI-reproducible artifact; cited,
|
||||
not re-verified (the citation and ciphertext both exist on `main`).
|
||||
|
||||
**Why it gates 5b.** Phase 5b's explicit goal is to make comparative-multiplicative cases
|
||||
*reach the graph*. If emission improves while this completeness gap remains, 5b would
|
||||
**convert latent into live** — admitting wrong answers on exactly the `<N> times` shape it
|
||||
is trying to solve. Therefore: **hardening the comparative-multiplicative completeness
|
||||
tripwire (so a dropped `<N>×` clause refuses, as `twice` does) is a hard precondition for
|
||||
any 5b emission PR.** The §9 driver test is the ready-made hard negative for the ADR-0174
|
||||
OQ#5 validation sub-corpus.
|
||||
211
docs/analysis/comprehension-primitive-inventory.md
Normal file
211
docs/analysis/comprehension-primitive-inventory.md
Normal file
|
|
@ -0,0 +1,211 @@
|
|||
<!-- CANONICAL | comprehension-primitive-inventory.md | updated 2026-06-03 | execution lane | supersedes prior copies -->
|
||||
|
||||
# Comprehension-Primitive Inventory & Cross-Subject Leverage Map
|
||||
|
||||
Status: draft / proposal-only
|
||||
Scope: read-only analysis from `main`, **verified** in the Claude lane (see appendix)
|
||||
Task: Task A from `docs/handoff/NEXT-SUBJECTS-CHATGPT-HANDOFF.md`
|
||||
|
||||
## Operating constraints observed
|
||||
|
||||
This artifact is analysis only. It proposes no serving-path edits, no eval edits, no ADR number, and no empirical claims in the inventory body. Any correctness, coverage, or `wrong=0` claim in the body is a structural reading of the code. The **Claude-lane verification appendix** at the end records what was checked against `main` with real reads of the committed report and source; it is the only section permitted to assert empirical state.
|
||||
|
||||
Read surfaces:
|
||||
|
||||
- `docs/handoff/NEXT-SUBJECTS-CHATGPT-HANDOFF.md`
|
||||
- `CLAUDE.md`
|
||||
- `generate/derivation/model.py`
|
||||
- `generate/derivation/extract.py`
|
||||
- `generate/derivation/clauses.py`
|
||||
- `generate/derivation/comparatives.py`
|
||||
- `generate/derivation/search.py`
|
||||
- `generate/derivation/multistep.py`
|
||||
- `generate/derivation/target.py`
|
||||
- `generate/derivation/compose.py`
|
||||
- `generate/derivation/accumulate.py`
|
||||
- `generate/derivation/pool.py`
|
||||
- `generate/derivation/product_bridge.py`
|
||||
- `generate/derivation/state/bind.py`
|
||||
- `generate/derivation/state/change.py`
|
||||
- `generate/math_candidate_parser.py`
|
||||
- `generate/math_candidate_graph.py`
|
||||
- `generate/recognizer_anchor_inject.py`
|
||||
- Skimmed referenced ADR surface in code/docstrings, especially ADR-0126, ADR-0131, ADR-0136, ADR-0163, ADR-0170, ADR-0175, ADR-0176, ADR-0178, ADR-0182, ADR-0184, ADR-0186, ADR-0189a, ADR-0191, ADR-0193, ADR-0194, ADR-0195.
|
||||
|
||||
## Inventory table
|
||||
|
||||
| # | Primitive | File / function(s) actually read | One-line description | Subject-general vs math-specific? |
|
||||
|---:|---|---|---|---|
|
||||
| 1 | Grounded quantity object | `generate/derivation/model.py::Quantity` | Represents a text-sourced numeric value with unit and source token provenance. | **Subject-general core, math-shaped payload.** Provenance-bearing "observed fact" objects transfer broadly; the `value/unit` fields are math-specific. |
|
||||
| 2 | Grounded derivation step | `generate/derivation/model.py::Step` | Represents one operation, its operand, and the licensing cue that must ground in text. | **Subject-general.** The pattern "claim/action must carry its own evidence cue" transfers to logic, reading comprehension, measurement, and any rule-bound subject. |
|
||||
| 3 | Deterministic left-fold derivation | `generate/derivation/model.py::GroundedDerivation.answer` | Computes a candidate result by left-folding validated steps over a start quantity. | **Mostly math-specific.** The arithmetic fold is math-specific; the generalizable primitive is ordered, evidence-carrying state transition. |
|
||||
| 4 | Primary-unit answer tracking | `generate/derivation/model.py::GroundedDerivation.answer_unit` | Carries the start quantity's unit as the answer unit under current derivation assumptions. | **Math-specific.** It is specifically dimensional arithmetic; the cross-subject analogue is "result type/class propagation." |
|
||||
| 5 | Digit quantity extraction | `generate/derivation/extract.py::extract_quantities`, `_QTY_RE`, `_quantity` | Extracts digit values followed by single unit tokens into `Quantity` records. | **Subject-general extraction pattern, math-specific symbols.** Literal-span extraction with provenance transfers; numeric parsing/unit attachment is math-specific. |
|
||||
| 6 | Word-number extraction | `generate/derivation/extract.py::_WORD_QTY_RE`, `_resolve_word_number`, `extract_quantities` | Resolves closed-set word numerals and conservative hyphen compounds into quantities. | **Broadly reusable.** Any subject with controlled lexical facts can use the same closed-vocabulary grounding discipline. |
|
||||
| 7 | Function-word unit hygiene | `generate/derivation/extract.py::_NON_UNIT_WORDS`, `_clean_unit` | Blanks function words that would otherwise be misread as units. | **Subject-general.** This is a lexical false-positive suppression primitive; math uses it for units, but other subjects need equivalent stop-token guards. |
|
||||
| 8 | List-unit inheritance | `generate/derivation/extract.py::_LIST_WITH_TRAILING_UNIT_RE`, `extract_quantities` | Assigns a trailing unit to every number in a same-list numeric sequence. | **Mixed.** The list inheritance pattern transfers to reading/measurement; the inherited object is math-specific. |
|
||||
| 9 | Sentence-final bare number extraction | `generate/derivation/extract.py::_FINAL_NUMBER_RE`, `extract_quantities` | Keeps terminal numbers available with unknown/empty unit rather than inventing a unit. | **Subject-general refusal-first grounding.** It preserves observed evidence without hallucinating missing attributes. |
|
||||
| 10 | Hyphen-bonded quantity extraction | `generate/derivation/extract.py::_HYPHEN_QTY_RE`, `extract_quantities` | Extracts tight `number-unit` surfaces such as `25-foot` without admitting open-ended multi-word units. | **Mixed.** Hyphenated modifier handling transfers; the payload is measurement/math-specific. |
|
||||
| 11 | Clause segmentation | `generate/derivation/clauses.py::segment_clauses` | Splits problem text into sentence-level clauses using terminal punctuation. | **Subject-general.** Clause segmentation is a foundational reading primitive; the implementation is intentionally orthographic and conservative. |
|
||||
| 12 | Clause-local sub-derivation | `generate/derivation/clauses.py::clause_local_results` | Derives each clause's local contribution or holds unresolved on ambiguity. | **Subject-general.** "Resolve locally before composing globally" transfers directly to reading comprehension, logic proof steps, and multi-sentence science/measurement tasks. |
|
||||
| 13 | Comparative scalar extraction | `generate/derivation/comparatives.py::extract_comparative_scalars`, `_load_comparatives`, `_N_TIMES_RE` | Maps closed comparative lexemes and `<N> times` phrases into scalar operations. | **Mixed.** Closed lexical relation extraction is subject-general; scalar multiplication is math-specific. |
|
||||
| 14 | Comparative-to-step bridge | `generate/derivation/comparatives.py::comparative_step` | Converts a comparative scalar into a derivation step whose grounding comes from the cue, not necessarily a literal numeric token. | **Subject-general.** The idea that an irreducible lexical fact licenses a typed transformation transfers strongly; the concrete operation is math-specific. |
|
||||
| 15 | Multiplicative cue hypothesis | `generate/derivation/search.py::MULTIPLICATIVE_CUES`, `_sentence_candidates` | Uses a closed cue set to propose in-clause product candidates only when a multiplicative cue is present. | **Mixed.** Cue-licensed candidate generation is general; multiplication/product semantics are math-specific. |
|
||||
| 16 | Bounded candidate generation | `generate/derivation/search.py::MAX_QUANTITIES`, `multiplicative_candidates`, `search_multiplicative`; `generate/derivation/multistep.py::MAX_QUANTITIES`, `candidate_chains` | Refuses rather than enumerating unbounded candidate spaces. | **Subject-general.** This is a core safety/performance primitive for any new subject. |
|
||||
| 17 | Target extraction from question clause | `generate/derivation/target.py::_question_clause`, `extract_target` | Extracts question quantities, aggregation cues, and units named in the question. | **Strongly subject-general.** Every subject lane needs "what is being asked?" extraction; current fields are math-specific. |
|
||||
| 18 | Prior-state question guard | `generate/derivation/target.py::asks_prior_state`, `_PRIOR_STATE_RE` | Detects questions asking for an earlier temporal state that forward derivation does not compute. | **Subject-general.** Temporal target mismatch is common across reading comprehension, science word problems, and procedural reasoning. |
|
||||
| 19 | Aggregation hint extraction | `generate/derivation/target.py::_AGG_WORDS`, `_AGG_PHRASES`, `extract_target` | Detects aggregation words/phrases such as `total`, `combined`, and `in all`. | **Mixed.** Aggregation-cue extraction transfers; summation semantics are math-specific. |
|
||||
| 20 | Question unit intersection | `generate/derivation/target.py::extract_target` | Treats asked units as body-known units that appear in the question. | **Mixed.** Target-slot/body-slot intersection transfers; unit semantics are math-specific. |
|
||||
| 21 | Shape-based multi-step chain enumeration | `generate/derivation/multistep.py::_candidate_chains`, `_chain`, `candidate_chains` | Builds a small deterministic set of product/sum chains, optionally followed by comparative tail steps. | **Mixed.** Shape-pruned candidate enumeration is general; product/sum chain templates are math-specific. |
|
||||
| 22 | Same-unit list-sum composition | `generate/derivation/compose.py::compose_sequential`, `_same_unit`, `_ADDITIVE_CUES` | Composes same-unit quantities within one clause using additive cues, with comparative tail application. | **Mixed.** Same-scope list composition transfers to reading/logic lists; same-unit arithmetic is math-specific. |
|
||||
| 23 | Clause-scoped referent guard | `generate/derivation/compose.py::compose_sequential` | Refuses when a list-sum structure spans multiple quantity-bearing clauses or has out-of-clause comparatives. | **Subject-general.** Scope containment is a central comprehension primitive and directly transfers to reading comprehension. |
|
||||
| 24 | Single-referent accumulation chaining | `generate/derivation/accumulate.py::_build_accumulation`, `compose_accumulation` | Chains gain/loss changes across clauses only when a later clause safely continues the anchor referent. | **Strongly subject-general.** This is state tracking over discourse; math uses numeric state, but the primitive is broadly useful. |
|
||||
| 25 | Foreign-distractor candidate handling | `generate/derivation/accumulate.py::_build_accumulation`, `accumulation_candidates`; `generate/derivation/verify.py::classify_derivation` | Allows isolated foreign quantities to enter as disagreement-only/exempt readings rather than commit candidates. | **Subject-general safety primitive.** Distractor evidence handling transfers to all comprehension lanes with irrelevant details. |
|
||||
| 26 | Sub-clause splitting | `generate/derivation/accumulate.py::_sub_clauses`, `_CONJUNCTION_SPLIT`, `_build_accumulation_anchor_skip` | Locally splits clauses on conjunctions for anchor/change discovery without changing the global segmenter. | **Subject-general.** Local structural refinement under a narrow caller-owned scope transfers well. |
|
||||
| 27 | Leading-subject extraction | `generate/derivation/state/bind.py::leading_subject_token` | Extracts a clause's leading word token as a loose subject signal. | **Subject-general.** It is a minimal discourse entity cue. |
|
||||
| 28 | Conservative same-referent continuation | `generate/derivation/state/bind.py::continues_anchor_referent`, `PRONOUNS` | Allows pronouns/same subjects/lowercase continuations and refuses new capitalized actor hazards. | **Subject-general.** This is directly reusable for reading comprehension and logic story-state tracking. |
|
||||
| 29 | Change polarity classification | `generate/derivation/state/change.py::classify_change_polarity`, `GAIN_VERBS`, `LOSS_VERBS` | Maps closed gain/loss cue sets to `+1`, `-1`, or refusal on ambiguity. | **Mixed.** Polarity classification is subject-general; gain/loss inventory is math-story specific. |
|
||||
| 30 | Grounded change cue selection | `generate/derivation/state/change.py::select_change_cue` | Chooses the actual cue lexeme that will be checked by the verifier. | **Subject-general.** Separating classification from evidence-cue selection is broadly valuable. |
|
||||
| 31 | Operand grounding gate | `generate/derivation/verify.py::self_verifies`, `_base_reasons` | Requires every non-comparative operand value token to ground in the problem text. | **Subject-general.** No invented evidence is a cross-domain invariant. |
|
||||
| 32 | Operation-cue grounding gate | `generate/derivation/verify.py::_base_reasons` | Requires every operation's licensing cue to appear in the text. | **Subject-general.** Every subject lane should require transformation rules to be evidence-licensed. |
|
||||
| 33 | Unit consistency gate | `generate/derivation/verify.py::_base_reasons`, `_SAME_UNIT_REQUIRED` | Requires same units for add/subtract while allowing multiply/divide composition. | **Math-specific with transferable type discipline.** The gate's type-checking role transfers; the unit rules are math-specific. |
|
||||
| 34 | Completeness gate | `generate/derivation/verify.py::_unused_quantities`, `self_verifies` | Refuses derivations that leave problem quantities unused. | **Subject-general.** "Account for all salient evidence" is central to reading, logic, measurement, and science tasks. |
|
||||
| 35 | Branch disagreement / uniqueness gate | `generate/derivation/verify.py::select_self_verified`; `generate/derivation/pool.py::resolve_pooled`; `generate/math_candidate_graph.py::parse_and_solve` | Commits only when verified candidates collapse to one distinct answer; otherwise refuses. | **Strongly subject-general.** This is one of the most transferable wrong=0-preserving primitives. |
|
||||
| 36 | Commit-eligible vs exempt classification | `generate/derivation/verify.py::classify_derivation` | Classifies readings as complete, exempt, or invalid; exempt readings can force disagreement but cannot commit alone. | **Subject-general.** "Counter-reading can block commitment without becoming an answer" is broadly useful. |
|
||||
| 37 | Repeated-unit product hazard detector | `generate/derivation/verify.py::_is_repeated_unit_product` | Marks pure products that repeat non-empty dimensions as commit-ineligible. | **Math-specific.** The general form is domain-type impossibility detection. |
|
||||
| 38 | Cross-composer pooling | `generate/derivation/pool.py::pooled_candidates`, `resolve_pooled` | Pools accumulation, multiplicative, and target-guided chain readings before applying disagreement/commit rules. | **Subject-general architecture.** Multiple independent readers should meet at a common disagreement gate. |
|
||||
| 39 | Serving promotion bridge | `generate/derivation/product_bridge.py::resolve_promotable_product`, `_has_hazard_surface`, `_has_product_target` | Promotes only complete pure-product readings whose question target and blocker checks make them safe for serving exposure. | **Mixed.** Promotion-boundary pattern is subject-general; current target/hazard surfaces are math-specific. |
|
||||
| 40 | Candidate initial-state extraction | `generate/math_candidate_parser.py::extract_initial_candidates`, `CandidateInitial` | Emits initial possession/state candidates with source-span provenance. | **Subject-general.** Initial state extraction is foundational for any story/world model; possession quantity is math-specific. |
|
||||
| 41 | Value-slot resolution | `generate/math_candidate_parser.py::_resolve_value`, `_resolve_currency`, `_is_indefinite_quantifier` | Resolves digits, money, fractions, word numbers, and hyphenated cardinals; refuses indefinite/unparseable values. | **Mixed.** Refusal-first lexical resolution transfers; supported value types are math-specific. |
|
||||
| 42 | Unit canonicalization | `generate/math_candidate_parser.py::_canonicalize_unit`, `_money_unit_normalization` | Maps surface unit tokens to canonical/plural units, including money normalization. | **Math/measurement-specific with transferable normalization boundary.** Other subjects need similar canonicalization for entities, predicates, or labels. |
|
||||
| 43 | Operation candidate extraction | `generate/math_candidate_parser.py::extract_operation_candidates`, `_op_pattern`, `_build_op_candidate` | Emits add/subtract/transfer operation candidates from canonical subject-verb-value-unit shapes. | **Mixed.** Typed event extraction transfers; arithmetic operation kinds are math-specific. |
|
||||
| 44 | Comparative operation extraction | `generate/math_candidate_parser.py::_compare_additive_candidates`, `_compare_multiplicative_candidates`, `_compare_nested_candidates`, `_resolve_reference_token` | Emits comparison candidates using closed comparison anchors and reference grounding. | **Mixed.** Comparative relation extraction transfers strongly; numeric delta/factor semantics are math-specific. |
|
||||
| 45 | Question candidate extraction | `generate/math_candidate_parser.py::extract_question_candidates`, `CandidateUnknown` | Emits unknown target candidates from closed question shapes. | **Subject-general.** Question-frame parsing is a primary cross-subject bottleneck. |
|
||||
| 46 | Aggregate question frames | `generate/math_candidate_parser.py::_Q_TOTAL_RE`, `_Q_THERE_RE`, `extract_question_candidates` | Maps total-across question surfaces to `Unknown(entity=None, unit=...)`. | **Mixed.** Aggregate target framing transfers; "unit total" is math-specific. |
|
||||
| 47 | Activity question frame | `generate/math_candidate_parser.py::_Q_DID_RE`, `extract_question_candidates` | Handles `How many <unit> did <Entity> <verb>?` activity-count questions. | **Mixed.** Activity target extraction transfers; counted activity quantity is math-specific. |
|
||||
| 48 | Conditional-prefix stripping | `generate/math_candidate_graph.py::_strip_conditional_prefix`, `_filtered_question_choices` | Retries question parsing after removing an `If X,` prefix. | **Subject-general.** Conditional-wrapper removal is broadly useful across logic and reading comprehension. |
|
||||
| 49 | Comparative-question refusal detector | `generate/math_candidate_parser.py::_pattern_b_comparative_candidates`, `_pattern_b_detects` | Recognizes "how many more" questions but emits no candidate until solver semantics exist. | **Subject-general safety primitive.** Detection-only recognizers can force clean refusal without pretending capability. |
|
||||
| 50 | Pronoun question resolution | `generate/math_candidate_parser.py::_resolve_pronoun_entity`, `_resolve_question_entity`, `_pattern_c_pronoun_verb_candidates` | Resolves gendered pronoun question entities only when exactly one whitelisted antecedent is present. | **Subject-general, implementation narrow.** The refuse-on-ambiguity pattern transfers; current name lists are GSM8K-specific. |
|
||||
| 51 | Statement context classifier | `generate/math_candidate_parser.py::has_numeric_token`, `classify_sentence` | Skips non-numeric context statements while preserving numeric-state-bearing statements as required parse/refuse inputs. | **Mixed.** Context filtering transfers; numeric-token criterion is math-specific. |
|
||||
| 52 | Capacity/rate extraction | `generate/math_candidate_parser.py::extract_capacity_candidates`, `extract_capacity_question_candidates`, `_to_seconds`; `generate/math_candidate_graph.py::parse_and_solve` | Extracts capacity per time and matching time-target questions, then computes scaled rate answers in a guarded short-circuit. | **Math/measurement-specific.** The broader primitive is matched statement/question rate-frame binding. |
|
||||
| 53 | Earnings-rate extraction | `generate/math_candidate_parser.py::extract_earnings_candidates`, `extract_earnings_question_candidates`; `generate/math_candidate_graph.py::parse_and_solve` | Extracts currency-per-time statements and matching money-over-time questions. | **Math/measurement-specific.** Transfers mainly to measurement/finance-like lanes. |
|
||||
| 54 | Conditional operation question | `generate/math_candidate_parser.py::extract_conditional_op_question_candidates`; `generate/math_candidate_graph.py::parse_and_solve` | Handles `If entity changes by N, how many ... left/now?` by matching one existing initial state and applying polarity. | **Mixed.** Conditional hypothetical target binding transfers strongly; arithmetic update is math-specific. |
|
||||
| 55 | Sentence splitting / one-question invariant | `generate/math_candidate_graph.py::_split_sentences`, `parse_and_solve` | Splits text, requires exactly one question sentence, and refuses otherwise. | **Subject-general.** Most subject lanes need explicit problem/question segmentation and clean refusal on malformed tasks. |
|
||||
| 56 | Per-sentence round-trip filtering | `generate/math_candidate_graph.py::_filtered_statement_choices`, `_filtered_question_choices`, `_initial_admissible`, `_question_admissible` | Filters emitted candidates by structural grounding before graph assembly. | **Subject-general.** Candidate emission and admissibility must remain separate in every subject. |
|
||||
| 57 | Most-grounded-slots tiebreaker | `generate/math_candidate_graph.py::_slot_count`, `_collapse_per_sentence_ties` | Collapses same-sentence candidates to the most grounded candidate when appropriate. | **Subject-general but hazardous if overused.** It transfers as a deterministic tiebreaker, but each subject must prove it cannot mask ambiguity. |
|
||||
| 58 | Graph construction with referential integrity | `generate/math_candidate_graph.py::_build_graph` | Builds a `MathProblemGraph`, rejecting branches whose question references unknown entities or violate graph invariants. | **Subject-general architecture, math-specific graph type.** Every subject needs typed graph construction with integrity checks. |
|
||||
| 59 | Cartesian branch enumeration cap | `generate/math_candidate_graph.py::MAX_TOTAL_BRANCHES`, `parse_and_solve` | Bounds branch enumeration and refuses when the space would exceed the cap. | **Subject-general.** Essential for deterministic safety and performance. |
|
||||
| 60 | Recognizer registry fallback | `generate/math_candidate_graph.py::_load_ratified_registry_or_empty`, `parse_and_solve` | Consults ratified recognizers only when parser choices are empty, and treats registry failures as empty. | **Subject-general.** Reviewed recognizer fallback with fail-closed behavior transfers directly. |
|
||||
| 61 | Anchor injection dispatch | `generate/recognizer_anchor_inject.py::inject_from_match` | Converts recognized anchors into typed solver primitives or returns empty on unsupported/unsafe categories. | **Subject-general.** This is a reusable boundary between recognizers and solver primitives. |
|
||||
| 62 | Composition registry consultation | `generate/recognizer_anchor_inject.py::_consult_composition_registry` | Admits pre-composed payloads only when the composition registry affirms their surface shape. | **Subject-general.** Reviewed structural-shape admission is reusable for logic, reading, and geometry. |
|
||||
| 63 | Discrete-count anchor injection | `generate/recognizer_anchor_inject.py::inject_discrete_count_statement`, `_build_initial_from_discrete_count`, `_build_operation_from_discrete_count_acquisition` | Builds initial-state or add-operation candidates from discrete-count recognizer anchors. | **Mixed.** Anchor-to-typed-fact injection is general; discrete count semantics are math-specific. |
|
||||
| 64 | Sealed injector lane | `generate/recognizer_anchor_inject.py::_SEALED_INJECTORS`, `inject_from_match`; `generate/math_candidate_graph.py::parse_and_solve(sealed=...)` | Keeps in-development injectors out of default serving until reviewed promotion. | **Subject-general.** This is a major reusable safety boundary for new subject lanes. |
|
||||
| 65 | Lookback pronoun resolution / ambiguity defense | `generate/math_candidate_graph.py::parse_and_solve` recognizer-injection section | Holds pronoun-requiring injected candidates until a discourse antecedent or pack-backed disambiguation is available; otherwise drops them. | **Strongly subject-general.** This is directly relevant to reading-comprehension and story-state subjects. |
|
||||
| 66 | Reader trace events | `generate/math_candidate_graph.py::CandidateGraphResult.reader_trace`, pronoun/lookback trace appends in `parse_and_solve` | Carries JSON-encoded trace events for reader phases and elimination/refusal causes. | **Subject-general.** Traceability/replay evidence is central to every future lane. |
|
||||
|
||||
## Cross-subject leverage map
|
||||
|
||||
### Strong transfer primitives
|
||||
|
||||
These are the highest-leverage primitives for new subjects because they are not inherently arithmetic:
|
||||
|
||||
1. **Evidence-carrying candidate objects** — anchors: `Quantity`, `Step`, `CandidateInitial`, `CandidateOperation`, `CandidateUnknown`. Cross-subject use: claims, propositions, logical premises, reading-comprehension facts, geometry givens.
|
||||
2. **Candidate emission separated from admissibility** — anchors: `extract_*_candidates`, `_initial_admissible`, `_question_admissible`, `roundtrip_admissible`, `self_verifies`. Cross-subject use: emit possible readings, then require grounding/type/consistency before commitment.
|
||||
3. **Refusal-first ambiguity handling** — anchors: `select_self_verified`, `resolve_pooled`, `parse_and_solve` decision rule. Cross-subject use: when multiple interpretations remain, refuse instead of choosing.
|
||||
4. **Scope/referent guards** — anchors: `segment_clauses`, `compose_sequential` clause-local guard, `continues_anchor_referent`, `_resolve_pronoun_entity`, lookback ambiguity defense. Cross-subject use: reading comprehension, narrative state tracking, logic variable binding.
|
||||
5. **Question/target extraction** — anchors: `extract_target`, `extract_question_candidates`, conditional prefix stripping, capacity/earnings/conditional question extractors. Cross-subject use: target-frame parsing is the obvious shared bottleneck across math, logic, reading, and measurement.
|
||||
6. **Completeness and distractor classification** — anchors: `_unused_quantities`, `classify_derivation`, exempt readings, context classifier. Cross-subject use: all subjects need "account for all relevant evidence" without forcing irrelevant distractors into the committed answer.
|
||||
7. **Promotion boundaries** — anchors: `resolve_promotable_product`, sealed injectors, ratified registry fallback. Cross-subject use: experimental readers can exist without becoming served behavior.
|
||||
|
||||
### Math-specific primitives with reusable analogues
|
||||
|
||||
| Math-specific primitive | Why math-specific | Reusable analogue |
|
||||
|---|---|---|
|
||||
| Unit consistency | Depends on dimensional arithmetic rules. | Type consistency / sort checking. |
|
||||
| Product/sum chain enumeration | Depends on arithmetic operator semantics. | Bounded proof/action sequence enumeration. |
|
||||
| Comparative scalar multiplication | Numeric scalar operation. | Relation-strength or predicate-transform facts from closed packs. |
|
||||
| Capacity/earnings rate short-circuits | Rate arithmetic over time/currency. | Matched statement-target frame with deterministic transformation. |
|
||||
| Repeated-unit product hazard | Dimensional impossibility. | Domain-type impossibility detector. |
|
||||
| Money/currency normalization | Numeric unit system. | Canonical symbol/entity normalization. |
|
||||
|
||||
## Observed composition wall
|
||||
|
||||
The current substrate already has many individually strong primitives. The bottleneck is not lack of primitives; it is safe composition among them:
|
||||
|
||||
- Clause-local reasoning exists, but cross-clause reasoning remains guarded and narrow.
|
||||
- Question target extraction exists, but many target frames still require closed shape support.
|
||||
- Referent continuation exists, but pronoun/coreference resolution is intentionally conservative.
|
||||
- Candidate pooling exists, but promotion to serving requires narrow target/hazard gates.
|
||||
- Completeness is strong, but it can over-force distractors unless exempt/disagreement paths are present.
|
||||
|
||||
This confirms the brief's framing: the next-subject work should exercise the same composition primitives without creating live serving risk.
|
||||
|
||||
## What transfers to other subjects
|
||||
|
||||
- **Reading comprehension should reuse the most math-relevant primitives immediately:** clause segmentation, referent guards, pronoun ambiguity refusal, target-frame parsing, completeness, and branch disagreement are already the exact pain points behind the math composition wall.
|
||||
- **Symbolic/deductive logic can reuse the candidate/admissibility/disagreement architecture:** premises become evidence-bearing candidates, inference rules become cue- or schema-licensed steps, and ambiguous proof branches refuse rather than commit.
|
||||
- **Measurement/geometry can reuse the most math-specific substrate with low conceptual impedance:** quantity extraction, unit canonicalization, unit/type consistency, target-unit matching, rate/measurement frames, and dimensional impossibility checks are already close to that domain.
|
||||
- **All future subjects should preserve the sealed/promotion boundary pattern:** draft readers and recognizers can be explored only as proposal-only or sealed lanes until the Claude lane verifies the relevant invariants.
|
||||
- **The highest cross-subject ROI is not a new corpus first; it is a small capability-axis spec that stresses target extraction, referent binding, completeness, and disagreement without weakening `wrong=0`.**
|
||||
|
||||
## Open questions for the Claude lane
|
||||
|
||||
1. Verify whether any functions above are currently serving-active vs sealed/practice-only on `main`; this read-only pass did not run lane-sha checks or tests.
|
||||
2. Confirm the exact current serving count and wrong/refusal distribution through the pinned eval lane before using this document as planning evidence.
|
||||
3. Decide whether Task B should treat `product_bridge.resolve_promotable_product` as part of the active question layer or as a promotion boundary around the derivation reader.
|
||||
4. Inspect coverage for the "most-grounded-slots-wins" tiebreaker before reusing that pattern in any new subject; it is powerful but could mask ambiguity if applied too broadly.
|
||||
5. For Task C, compare candidate subject ordering against the actual contents of `evals/symbolic_logic/` and `evals/math_capability_axes/` before drafting any subject-specific axes.
|
||||
|
||||
---
|
||||
|
||||
## Claude-lane verification (landed)
|
||||
|
||||
Verified against `main` at commit `3e29559` by reading the committed serving report and source. Method note: the inventory above was authored read-only; the checks below resolve its five open questions. The full `core test`/MLX/Rust suite was **not** re-run in this lane (Apple-Silicon/MLX substrate unavailable here); the serving metric cited is the committed, pinned report — the authoritative source of truth for the frozen serving path — not a fresh run.
|
||||
|
||||
**Definition-of-done check (Task A):** all 66 primitives resolve to real files on `main`. Every referenced module exists (`generate/derivation/{model,extract,clauses,comparatives,search,multistep,target,compose,accumulate,pool,product_bridge,verify}.py`, `generate/derivation/state/{bind,change}.py`, `generate/{math_candidate_parser,math_candidate_graph,recognizer_anchor_inject}.py`). No invented APIs found.
|
||||
|
||||
### Q1 — serving-active vs sealed
|
||||
- `_SEALED_INJECTORS = {}` is **empty** on `main`. Nothing is currently sealed. Inventory row #64 describes a real mechanism, but it is presently inert — so "sealed lane" is not what is suppressing any current behavior.
|
||||
- `discrete_count_statement` is **serving-active**: it is wired directly into the live dispatch map (`ShapeCategory.DISCRETE_COUNT_STATEMENT: inject_discrete_count_statement`). Its empty injections (see Q2) are genuine conservatism in the active injector, not sealing.
|
||||
- The frozen-serving gate (`scripts/verify_lane_shas.py`) pins the **SHA-256 of report outputs** for 8 eval lanes (reviewer_registry, miner_loop_closure, curriculum_loop_closure, domain_contract_validation, fabrication_control, demo_composition, public_demo, math_teaching_corpus). It freezes serving by making any drift in those outputs detectable; it does not pin a static list of serving source files.
|
||||
|
||||
### Q2 — exact serving distribution (CONFIRMED)
|
||||
Pinned report `evals/gsm8k_math/train_sample/v1/report.json` (ADR-0126, sample_count=50):
|
||||
|
||||
- **6 correct / 44 refused / 0 wrong.** `wrong=0` holds. `exit_criterion.correct_min=10` → `passed: false`.
|
||||
|
||||
The 44 non-correct cases decompose as:
|
||||
|
||||
| Failure mode | Count |
|
||||
|---|---:|
|
||||
| Recognizer matched but produced **no injection** | 32 |
|
||||
| **No admissible candidate** (parser emitted nothing usable) | 12 |
|
||||
|
||||
Locus of the 44: statement (recognizer) 32 · statement (parser) 7 · question (parser) 5.
|
||||
|
||||
Recognizer-fired-but-empty-injection (32) by category:
|
||||
|
||||
| Category | Count |
|
||||
|---|---:|
|
||||
| `discrete_count_statement` | 18 |
|
||||
| `descriptive_setup_no_quantity` | 4 |
|
||||
| `rate_with_currency` | 3 |
|
||||
| `multiplicative_aggregation` | 3 |
|
||||
| `currency_amount` | 3 |
|
||||
| `temporal_aggregation` | 1 |
|
||||
|
||||
**Headline:** the single largest refusal bucket is `discrete_count_statement` — **18 of 44 (41%)** — where the serving-active recognizer fires on a count-like token but the injector returns empty. **This marks *where* the composition wall surfaces; it is not a lever to widen.** As the corrected Net-read below establishes, all 18 are 2–4 capability compositions the injector *correctly* declines (emitting an initial-state there is metric-inert). The concentration is diagnostic — the most common surface form of the wall — not a backlog item, and it touches the entity/initial-state primitives (#40, #63) only as evidence that the wall sits *downstream* of extraction, in composition.
|
||||
|
||||
### Q3 — `product_bridge.resolve_promotable_product` classification (RESOLVED)
|
||||
It is part of the **active serving question layer, behaving as a promotion boundary around the derivation reader** — both, not either/or. Its module docstring places it on "the serving candidate-graph path," and it returns a "serving-safe product resolution" only after passing `_has_hazard_surface` and `_has_product_target`. Recommendation for Task B: treat it as the guarded gate by which derivation-reader products reach serving, i.e. a promotion boundary that is itself live — not a sealed/practice-only reader.
|
||||
|
||||
### Q4 — "most-grounded-slots-wins" tiebreaker coverage (CAUTION CONFIRMED, scope corrected)
|
||||
`_collapse_per_sentence_ties` / `_slot_count` are invoked at two serving sites in `parse_and_solve` (lines 958, 999). No test references those functions **by name** (no white-box test). However — correcting an earlier overstatement in this appendix — the collapse **is** behaviorally covered on the happy path: `tests/test_math_candidate_graph.py::TestAmbiguityResolution::test_gives_with_target_resolves_to_transfer` exercises the slot-count collapse ("Sam gives 3 apples to Tom" → transfer reading wins on more grounded slots) and would fail if the collapse broke. The accurate, narrower gap is therefore: happy-path collapse is covered; what is **missing** is (a) a white-box test naming the functions and (b) an **adversarial "high-slot-but-wrong vs low-slot-but-right"** case — the scenario where "more slots = better" selects the wrong reading. Recommendation: add both before reusing this pattern in any new subject.
|
||||
|
||||
### Q5 — Task C input (DEFERRED to Task C execution)
|
||||
Not resolved here; Task C explicitly requires comparing candidate subject ordering against the live contents of `evals/symbolic_logic/` and `evals/math_capability_axes/`. Flagged for the Task C pass so it is not double-counted as Task A scope.
|
||||
|
||||
### Net read for planning (corrected)
|
||||
An earlier version of this section recommended widening the serving-active `discrete_count_statement` injector as "the highest-count, lowest-risk math lever (18/44)." **That conclusion was wrong and is retracted.** Reading all 18 of those cases in full shows they are **2–4 capability compositions** (ratio chains 0020/0029/0033, multi-step rate/percent 0032/0034/0044, accumulate-against-target 0037/0039, and 0040 which needs per-entity attribute lookup before any arithmetic). The recognizer fires on the first count token ("2 horses"); the injector **correctly declines** because the surrounding problem is not a bare count. Emitting an initial-state there is **metric-inert** — the graph still cannot compose to the answer. The 18/44 concentration is the **composition wall surfacing at the most common recognizer category**, not an injector to widen. This is reinforced by **ADR-0174 (Proposed)**, which deprecates the per-category injector dispatch table as the runtime admission path (injectors become hypothesis-emitters in a held-hypothesis reader), and by the wrong=0 hazard of that surface (case-0050 canary on the same serving path).
|
||||
|
||||
Corrected steer: primitives are not the bottleneck; **safe composition is.** The honest next lever is a **composition capability** over the existing grounded primitives — multi-quantity chains (ratio, multi-step rate/percent, accumulate-against-target). The direct GSM8K-metric lever is **ADR-0174's held-hypothesis reader (Proposed)**; the adjacent proof-DAG substrate — binding-graph acyclicity, proof-graph builder, modus-ponens disagreement — is already **Accepted** (ADR-0203/0204/0205, proof_chain phase 2.1–2.3). So the work is composition through the held-hypothesis reader on an accepted proof substrate, **not** category-dispatch widening. For Task B: group all 44 and rank by **composition-arity** (1-capability gaps = tractable; 2–4-capability compositions = the wall), not by raw recognizer-category count.
|
||||
133
docs/analysis/question-layer-gap-survey.md
Normal file
133
docs/analysis/question-layer-gap-survey.md
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
<!-- FILE: docs/analysis/question-layer-gap-survey.md -->
|
||||
|
||||
# Question-Layer Gap Survey
|
||||
|
||||
Status: Proposed analysis draft. No serving behavior is changed.
|
||||
|
||||
Source of truth for the metric is `docs/claims_ledger.md` and
|
||||
`evals/gsm8k_math/train_sample/v1/report.json`: the current real
|
||||
`train_sample` result is 6 correct / 44 refused / 0 wrong. This survey assigns
|
||||
each of the 44 refused case ids to exactly one current failure group. It does
|
||||
not claim that any group would pass if widened; it names the layer that refuses
|
||||
today and estimates whether the missing capability is single-step or
|
||||
composition-bound.
|
||||
|
||||
The requested Task A file, `docs/analysis/comprehension-primitive-inventory.md`,
|
||||
was not present in this worktree. I used the embedded older handoff only as
|
||||
orientation and grounded the grouping below in the report plus the live parser /
|
||||
recognizer code.
|
||||
|
||||
## Code Map
|
||||
|
||||
Current refusal topology:
|
||||
|
||||
- `generate/math_candidate_graph.py` first extracts statement choices, then
|
||||
consults the ratified recognizer registry when a numeric statement has no
|
||||
parser candidate. If recognition succeeds but injection yields no typed
|
||||
solver primitive, it now refuses explicitly instead of dropping the statement.
|
||||
- `generate/recognizer_anchor_inject.py` is still a category dispatch surface.
|
||||
Only `discrete_count_statement` and a narrow `multiplicative_aggregation`
|
||||
entry can emit today; other categories return empty and therefore become
|
||||
explicit refusals.
|
||||
- `generate/math_candidate_parser.py` admits a closed question grammar:
|
||||
total-across `How many ... do they have ...`, existential aggregate
|
||||
`How many ... are there ...`, entity possession, activity `did`, and three
|
||||
ADR-0163.D.4 patterns. If these emit no `CandidateUnknown`, the graph refuses
|
||||
before branch enumeration.
|
||||
- ADR-0174 explicitly deprecates the per-category injector dispatch table as
|
||||
the long-term runtime admission path. Injector widening should therefore be
|
||||
treated as stopgap or hypothesis-emitter work, not the primary strategic
|
||||
direction.
|
||||
|
||||
## Assignment Table
|
||||
|
||||
Tractability scoring is case-level and arity-aware:
|
||||
|
||||
- High: one closed primitive or one local parser frame is plausibly missing.
|
||||
- Medium: one local frame/schema is clear, but downstream composition is still
|
||||
likely needed.
|
||||
- Low: the current refusal is an early stop in a 2-4 capability derivation.
|
||||
|
||||
| Group | Count | Case ids | Current failing layer | Tractability | Representative report reason excerpts | Interpretation |
|
||||
|---|---:|---|---|---|---|---|
|
||||
| DCS high-arity composition wall | 18 | 0002, 0015, 0016, 0020, 0029, 0031, 0032, 0033, 0034, 0036, 0037, 0038, 0039, 0040, 0041, 0044, 0047, 0049 | Matcher -> injector -> composition | Low | `candidate_graph: recognizer matched but produced no injection for statement: 'She splits it up into 25-foot sections.' (category=discrete_count_statement)`; `candidate_graph: recognizer matched but produced no injection for statement: 'Malcolm is trying to find the fastest walk to school and is currently comparing two routes.' (category=discrete_count_statement)` | The recognizer often fires on a count-like token, but the actual derivation needs division, rest-state, route comparison, percent/rate, chained comparisons, target residuals, or per-entity attributes. Widening the discrete-count injector alone is metric-inert and risks incomplete readings. |
|
||||
| Missing inverse/residual/comparative question frames | 5 | 0007, 0008, 0009, 0025, 0035 | Question parser / admissibility | Low | `candidate_graph: no admissible candidate for question: 'How many more boxes do they need if Francine has a total of 85 crayons?'`; `candidate_graph: no admissible candidate for question: 'How many more apples would Martha need to give away to be left with only 4 of them?'` | These are not just new surface phrasings for `Unknown(entity, unit)`. They ask for a missing operand, inverse relation, target residual, or conditional total. The question layer must bind the requested slot to a derivation, not merely extract a unit. |
|
||||
| Rate, currency-rate, and tariff statements | 4 | 0001, 0011, 0017, 0022 | Recognizer -> injector/schema | Medium | `candidate_graph: recognizer matched but produced no injection for statement: 'Tina makes $18.00 an hour.' (category=rate_with_currency)`; `candidate_graph: recognizer matched but produced no injection for statement: 'He’s charging $50.00 per day or $500.00 for 14 days.' (category=temporal_aggregation)` | The local schema gap is visible: `rate_with_currency` and `temporal_aggregation` need typed rate/tariff hypotheses. Case-level admission still needs overtime, profit, historical+today aggregation, or piecewise tariff composition. |
|
||||
| Non-quantitative relation category used as an early stop | 4 | 0012, 0023, 0027, 0046 | Matcher -> injector | Low | `candidate_graph: recognizer matched but produced no injection for statement: 'He put all of them in his aquarium but his fish ate half of them.' (category=descriptive_setup_no_quantity)`; `candidate_graph: recognizer matched but produced no injection for statement: 'Half of the students are girls, the other half are boys.' (category=descriptive_setup_no_quantity)` | The category name is accurate for the injector: it cannot emit a concrete primitive from the matched surface. The cases need fraction-of-prior, combined-total binding, partition, and percentage-of-subgroup reasoning. |
|
||||
| Multiplicative aggregate beyond the narrow emitted shapes | 3 | 0006, 0013, 0045 | Recognizer -> injector / composition registry | Medium | `candidate_graph: recognizer matched but produced no injection for statement: 'Mandy started reading books with only 8 pages when she was 6 years old.' (category=multiplicative_aggregation)`; `candidate_graph: recognizer matched but produced no injection for statement: 'Each survey has 10 questions.' (category=multiplicative_aggregation)` | There is already a narrow product injector, but these rows need time/age chains, month segmentation, doubled rates, or survey-count composition. Single product emission may help local state, but full cases still depend on multi-step composition. |
|
||||
| Financial currency amount / percent mutation statements | 3 | 0019, 0028, 0043 | Recognizer -> injector / mutation schema | Low | `candidate_graph: recognizer matched but produced no injection for statement: 'After the first appointment, John paid $100 for pet insurance that covers 80% of the subsequent visits.' (category=currency_amount)`; `candidate_graph: recognizer matched but produced no injection for statement: 'Her mother gave her an additional $4, and her father twice as much as her mother.' (category=currency_amount)` | The surface contains currency, but the required reading is coverage after first event, percent daily operating cost, revenue target, or comparative gift amount. A `CandidateInitial` currency emission would be an incomplete graph. |
|
||||
| Fractional relational statements with no parser candidate | 3 | 0004, 0005, 0010 | Statement parser / admissibility | Low | `candidate_graph: no admissible candidate for statement: 'Half of the kids are going to soccer camp, and 1/4 of the kids going to soccer camp are going to soccer camp in the morning.'`; `candidate_graph: no admissible candidate for statement: 'Marion has 1/4 more than what Yun currently has, plus 7.'` | The parser has fraction literals and comparative operations, but these surfaces are relational fractions over prior or unknown quantities. They require held equations or derivation nodes, not a flat possession/operation candidate. |
|
||||
| Duration and recurrence statement frames | 3 | 0030, 0048, 0050 | Statement parser / temporal composition | Medium | `candidate_graph: no admissible candidate for statement: 'It is a 2-hour drive each way.'`; `candidate_graph: no admissible candidate for statement: 'Mark does a gig every other day for 2 weeks.'` | These expose bounded duration-multiplier or temporal-frequency frames. Each frame is local and deterministic, but the cases still need composition across trip stages, weekly deltas to target, or per-event song duration totals. |
|
||||
| Relational conjoined-subject each initial | 1 | 0026 | Statement parser / entity binding | Medium | `candidate_graph: no admissible candidate for statement: 'Aaron and his brother Carson each saved up $40 to go to dinner.'` | The parser has an `each` extractor for two named subjects, but this sentence uses a possessive relational subject (`his brother Carson`) and a purpose tail. The local parse is probably narrower than the concept; the full case remains multi-step because bill fraction and shared scoop count follow. |
|
||||
|
||||
Total assigned: 18 + 5 + 4 + 4 + 3 + 3 + 3 + 3 + 1 = 44.
|
||||
|
||||
## Backlog Interpretation
|
||||
|
||||
The audited partition is the stable artifact. A single `count x tractability`
|
||||
sort is misleading here because count ranges from 1 to 18 while tractability is
|
||||
coarse; it would place the known composition wall at the top as if it were an
|
||||
incremental injector work order. In this survey, count is impact evidence, not
|
||||
the sort key for near-term changes.
|
||||
|
||||
### Composition-Bound Work
|
||||
|
||||
These groups should feed the ADR-0174 held-hypothesis / derivation-composer
|
||||
scope, not category-specific injector widening. They need multi-clause state,
|
||||
referent binding, ratio/fraction relations, target-slot questions, or
|
||||
event-scope composition before any answer can be safe.
|
||||
|
||||
| Group | Count | How to use the count |
|
||||
|---|---:|---|
|
||||
| DCS high-arity composition wall | 18 | Main evidence that discrete-count recognition is surfacing a composition wall, not an injector backlog. |
|
||||
| Missing inverse/residual/comparative question frames | 5 | Question-layer evidence that unknowns must bind to derivation slots, not just noun units. |
|
||||
| Rate, currency-rate, and tariff statements | 4 | Rate/tariff hypotheses are useful only if downstream overtime, profit, history, or piecewise composition can refuse partial readings. |
|
||||
| Non-quantitative relation category used as an early stop | 4 | Evidence for relation/partition composition and for avoiding hard-stop loss of later structure. |
|
||||
| Multiplicative aggregate beyond the narrow emitted shapes | 3 | Product-like anchors need day/month, age-chain, or survey-count composition before admission. |
|
||||
| Financial currency amount / percent mutation statements | 3 | Currency is not enough; these need percent coverage, operating-cost, or spend/residual mutation. |
|
||||
| Fractional relational statements with no parser candidate | 3 | Needs relational fraction/equation hypotheses, not flat possession candidates. |
|
||||
| Duration and recurrence statement frames | 3 | Duration-multiplier and recurrence frames are local signals whose answers require temporal composition. |
|
||||
|
||||
### Bounded Near-Term Fixes
|
||||
|
||||
These are smaller, lower-blast-radius probes that may be useful as executable
|
||||
follow-ups or regression tests. They should still preserve refusal-first
|
||||
admission and should not be represented as the main path to the metric.
|
||||
|
||||
| Candidate fix | Case ids | Why bounded |
|
||||
|---|---|---|
|
||||
| Relational conjoined-subject `each` binding | 0026 | One row exposes a local entity-binding gap around `Aaron and his brother Carson each ...`; useful as a narrow parser/binding probe even though the full problem remains multi-step. |
|
||||
| Descriptive/no-quantity early-stop handling | 0012, 0023, 0027, 0046 | The bounded mechanism is to avoid losing later structure when a descriptive relation cannot emit state; the cases behind it still require composition, so this is not a direct answer path. |
|
||||
| Single-slot question-frame probes | subset of 0007, 0008, 0035 | A few question surfaces can become narrow probes for residual or divisor target-slot binding. They must be held as unknown slots and refused unless the derivation is complete. |
|
||||
|
||||
## Layer Notes
|
||||
|
||||
The high-count `discrete_count_statement` bucket should not be read as "add more
|
||||
discrete-count injectors." The report is saying the matcher saw something
|
||||
count-shaped before the engine had a safe composed reading. In the live code,
|
||||
recognized-but-uninjected statements refuse explicitly because dropping them
|
||||
would permit incomplete graphs. ADR-0174 points in the same direction:
|
||||
injectors should become hypothesis emitters inside a held-hypothesis reader,
|
||||
where branch disagreement, constraint propagation, and completeness can reject
|
||||
partial readings.
|
||||
|
||||
The five question refusals are genuinely question-layer refusals in the narrow
|
||||
code sense: `extract_question_candidates` emits no `CandidateUnknown`. But all
|
||||
five ask for an inverse or residual target. A wider regex that only extracts
|
||||
the noun would not identify the unknown slot and would still be unsafe under
|
||||
`wrong = 0`.
|
||||
|
||||
## Open Questions for the Claude Lane
|
||||
|
||||
- Confirm whether `docs/analysis/comprehension-primitive-inventory.md` was meant
|
||||
to be landed in this worktree or only supplied out-of-band; this draft could
|
||||
be amended to cite its exact primitive table once present.
|
||||
- Run an instrumented read of the 44 refused cases to distinguish "injector
|
||||
returned empty because no parsed anchors existed" from "anchors existed but
|
||||
constraint propagation eliminated them."
|
||||
- For the five question refusals, collect the intended `Unknown` slot shape
|
||||
without using gold answers: missing operand, residual-to-target, inverse
|
||||
divisor, or aggregate total.
|
||||
- Decide whether the next executable lane should prototype rate/tariff
|
||||
hypotheses or question-target slots first; both are safer as held hypotheses
|
||||
than as direct category admissions.
|
||||
148
docs/analysis/solver-operation-coverage.md
Normal file
148
docs/analysis/solver-operation-coverage.md
Normal file
|
|
@ -0,0 +1,148 @@
|
|||
<!-- FILE: docs/analysis/solver-operation-coverage.md -->
|
||||
|
||||
# Solver Operation Coverage Audit
|
||||
|
||||
Status: Proposed analysis draft. No serving behavior is changed. This is a
|
||||
read-only structural audit to de-risk ADR-0174 Phase 5b / Phase-4-style
|
||||
composition work. Verdicts below are code-reading conclusions only; they must be
|
||||
verified in the Claude lane with executable solver/binding-graph cases before
|
||||
any promotion claim.
|
||||
|
||||
## Scope
|
||||
|
||||
Read surfaces:
|
||||
|
||||
- `generate/math_problem_graph.py`
|
||||
- `generate/math_solver.py`
|
||||
- `generate/binding_graph/{model,adapter,admissibility,allocation,question_target,units}.py`
|
||||
- ADR-0116, ADR-0117, ADR-0132, ADR-0133, ADR-0134, ADR-0135
|
||||
- Skimmed ADR-0174 Phase 5b and ADR-0203/0204/0205 to confirm current
|
||||
composition/proof-DAG framing.
|
||||
|
||||
Important correction to the relay: the current solver/graph vocabulary is not
|
||||
only `{add, subtract, transfer, multiply, divide}`. `MathProblemGraph` and
|
||||
`math_solver` both include the eight operation kinds:
|
||||
|
||||
```text
|
||||
add, subtract, transfer, multiply, divide,
|
||||
apply_rate, compare_additive, compare_multiplicative
|
||||
```
|
||||
|
||||
ADR-0174 Phase 5b states the same: the solver is already waiting for these
|
||||
operations; the gap is reader -> injector -> `Operation` front-end wiring plus
|
||||
composition.
|
||||
|
||||
## Existing Operation Substrate
|
||||
|
||||
| Surface | Evidence | Consequence |
|
||||
|---|---|---|
|
||||
| Closed operation vocabulary | `generate/math_problem_graph.py` defines `VALID_OPERATION_KINDS` with eight kinds. | New arithmetic verbs are not needed for ordinary multiply/divide/rate/comparison chains. |
|
||||
| Pack-bound solver dispatch | `generate/math_solver.py` maps all eight kinds to `en_arithmetic_v1` lemmas before solving. | Missing pack lemmas fail loudly; no hidden operation fallback. |
|
||||
| Stateful solver semantics | `_apply` mutates `(actor, unit)` terminal state for `add`, `subtract`, `transfer`, `multiply`, `divide`; `_apply_rate` produces numerator-unit state from denominator-unit state; comparisons derive an actor state from a reference actor. | The solver is good at forward state trajectories, but weak at keeping multiple same-unit derived intermediates alive under one actor. |
|
||||
| Unknown shape | `Unknown(entity, unit)` resolves either terminal state for one entity or total-across all entities with that unit. | Target questions that ask for a missing operand, number of iterations, or intermediate state are not represented by `Unknown` alone. |
|
||||
| Binding-graph equation/data model | `BoundEquation(operation_kind=...)`, `BoundUnknown(question_form=...)`, semantic roles include `rate`, `duration`, `difference`, `ratio`. | The graph can name richer forms than the solver's final `Unknown`, but current adapter still comes from existing `MathProblemGraph` operation chains. |
|
||||
| Unit admissibility | `check_admissibility` covers additive, multiplicative, divide, `apply_rate`, and comparison kinds. | Dimensional proofs exist for the current eight operation kinds; new node types would need explicit admissibility rules. |
|
||||
| Question-target binding | `infer_question_form` recognizes `ratio`, `difference`, `rate`, `total`, and `count` from operation kinds touching the unknown. | It can label answer form, but it does not solve inverse targets or select intermediate operation indices yet. |
|
||||
| Acyclicity | ADR-0203 adds `circular_dependency` refusal to the shared binding-graph constructor. | Any new equation/intermediate-node extension must remain a DAG, not a cyclic algebra system hidden inside the graph. |
|
||||
|
||||
## Phase-4 Target Chains
|
||||
|
||||
| Target chain | Verdict | Existing operations that can carry it | What is missing / risk |
|
||||
|---|---|---|---|
|
||||
| Multi-step rate-sum | Expressible via composition for straight-line rate applications and sums. Piecewise tariffs/conditionals need scoped selection before admission. | `apply_rate` computes `X/Y * Y -> X`; `add`/`subtract` can aggregate generated same-unit totals. Binding admissibility has an `apply_rate` rule requiring one rate dep plus one duration/count dep. | The reader must emit the base duration/count quantities, rate hypotheses, and sum operations without clobbering unrelated same-unit state. Piecewise tariffs such as "$50/day or $500/14 days" need a tariff/choice scope or explicit branch-disagreement gate; that is a binding/composition problem, not a missing arithmetic verb. |
|
||||
| Ratio chain | Expressible via composition for forward ratio chains. Inverse ratio equations need a target/equation extension. | `compare_multiplicative` supports "actor = factor * reference"; `multiply`/`divide` support scalar transformations; `infer_question_form` maps touching `compare_multiplicative` to `ratio`. | Forward chains like `A`, `B = 2A`, `C = 1/2(A+B)` can be represented if the reader emits the right reference actors and order. Inverse forms such as "ducks are 10 more than 4x chickens; ducks = 150; find total birds" require solving for an unknown reference, not just applying a forward operation. That needs equation/target solving or a new binding node shape, not a new `ratio` operation kind. |
|
||||
| Accumulate-against-target | Needs new binding-graph target/equation capability; not expressible as a fixed existing operation chain in the current solver. | `add`, `subtract`, `multiply`, `divide`, and `apply_rate` can express the arithmetic once the iteration count or missing operand is known. | The current solver consumes a fully specified graph in source order, then resolves terminal `Unknown(entity, unit)`. It cannot represent "after how many weeks", "how many cups to sell to reach profit", or "how long to make back cost" as an unknown operand/iteration count. This needs a first-class target-slot/equation node or bounded inverse solver with refusal/disagreement, plus proof that no cyclic dependency is introduced. |
|
||||
| Percent/fraction mutation | Mostly expressible via composition for forward mutations; needs intermediate-symbol/scope support for same-unit derived amounts and event subsets. | Percent/fraction values can be scalar `multiply`/`divide` factors, with `add`/`subtract` for mutation and `compare_multiplicative` for relative quantities. Binding admissibility already covers multiply/divide dimensions. | Some cases are safe forward mutations ("eat 75% of a pan", "lose half"). Others require original and derived same-unit quantities to coexist, e.g. principal plus interest, insured vs uninsured portions, operating cost as percent of startup cost. The current solver overwrites `(actor, unit)` for multiply/divide and `apply_rate`, so these need derived intermediate symbols/event scopes or separate binding nodes. A new `percent` operation kind is not structurally necessary; a new scoped intermediate/equation representation may be. |
|
||||
|
||||
## Structural Verdict
|
||||
|
||||
Phase 4/5b is mostly **not** blocked by missing arithmetic operation kinds. The
|
||||
current operation vocabulary already covers the primitive arithmetic field:
|
||||
addition/subtraction/transfer, scalar multiply/divide, rate application, and
|
||||
additive/multiplicative comparison.
|
||||
|
||||
The real scope risk is representation:
|
||||
|
||||
- Can the reader emit a chain of typed operations from scattered clauses while
|
||||
preserving all grounded quantities?
|
||||
- Can the graph retain intermediate derived quantities when they share the same
|
||||
actor/unit as their source?
|
||||
- Can a question bind to a missing operand, iteration count, or intermediate
|
||||
state instead of only terminal `Unknown(entity, unit)`?
|
||||
- Can inverse/equation targets be solved under a disagreement rule and the
|
||||
ADR-0203 acyclicity invariant?
|
||||
|
||||
That makes the likely build a **binding-target / intermediate-symbol /
|
||||
derivation-composer extension**, not a broad new solver primitive pack.
|
||||
|
||||
## Chain-Specific Notes
|
||||
|
||||
### Multi-step rate-sum
|
||||
|
||||
`apply_rate` is first-class: the solver's `_apply_rate` reads the actor's
|
||||
denominator-unit state and writes numerator-unit state. The binding-graph
|
||||
adapter synthesizes a rate symbol with composite unit `<num>_per_<denom>`, and
|
||||
admissibility checks that the denominator cancels. Therefore simple rate-sum is
|
||||
expressible as:
|
||||
|
||||
```text
|
||||
duration/count fact -> apply_rate -> produced total
|
||||
produced totals -> add/subtract -> final total
|
||||
```
|
||||
|
||||
The unsafe part is not the operation. It is branch selection and scope:
|
||||
overtime, tariffs, and "including today" require deciding which rate applies to
|
||||
which event subset.
|
||||
|
||||
### Ratio chain
|
||||
|
||||
`compare_multiplicative` already gives a forward ratio operation. It refuses
|
||||
when the reference actor has no quantity or multiple ambiguous units, which is
|
||||
the right wrong=0 boundary. Binding target can label ratio-form questions.
|
||||
|
||||
The gap is inverse ratio. Current `Operation` is directional: it mutates the
|
||||
actor from a known reference. If the reference is the unknown and the actor is
|
||||
given, the solver has no inverse-equation mode.
|
||||
|
||||
### Accumulate-against-target
|
||||
|
||||
This is the clearest "needs new graph shape" target. A terminal state solver can
|
||||
answer "what is the total after N weeks"; it cannot answer "what N reaches total
|
||||
T" unless N is already present as a quantity and a divide operation has been
|
||||
materialized by the reader. A safe implementation needs a target slot that names
|
||||
the missing operand/iteration count and an admissibility/disagreement rule for
|
||||
the inverse derivation.
|
||||
|
||||
### Percent/fraction mutation
|
||||
|
||||
Percent/fraction does not need a new arithmetic verb. It needs:
|
||||
|
||||
- scalar extraction (`75% -> 0.75`, `1/4 -> 0.25`);
|
||||
- complement derivation when the text says "covers 80%" but the cost asks for
|
||||
the uncovered part;
|
||||
- event/subset scoping ("after the first appointment", "subsequent visits");
|
||||
- intermediate symbols when original principal/cost and derived interest/cost
|
||||
must both remain available.
|
||||
|
||||
The binding graph is the natural home for these intermediates because it already
|
||||
has `BoundEquation`, dependency sets, unit proofs, question forms, and
|
||||
acyclicity checks.
|
||||
|
||||
## Open Questions for the Claude Lane
|
||||
|
||||
- Does `apply_rate` overwriting `(actor, numerator_unit)` create a concrete
|
||||
hazard in rate-sum cases where the actor already holds money? If yes, the
|
||||
build needs derived result symbols before promotion.
|
||||
- Should inverse target solving be introduced as a new `BoundUnknown` form, a
|
||||
new `BoundEquation` operation kind, or a separate proof/derivation rule over
|
||||
existing equations?
|
||||
- Can forward ratio chains be admitted through `compare_multiplicative` without
|
||||
widening the parser's reference-actor ambiguity beyond the existing refusal
|
||||
discipline?
|
||||
- What is the smallest executable probe set that distinguishes "new operation
|
||||
kind required" from "same operation, new intermediate symbol required" for
|
||||
percent/fraction mutations?
|
||||
- When proof-DAG consumers and math binding graphs share
|
||||
`SemanticSymbolicBindingGraph`, should proposition-specific operation kinds
|
||||
remain isolated from math admissibility, or should the admissibility dispatcher
|
||||
split into math/proof entrypoints before more equation kinds are added?
|
||||
|
|
@ -304,6 +304,17 @@ def _build_initial_from_discrete_count(
|
|||
if value is None:
|
||||
return None
|
||||
|
||||
# A surface like "Jerry has 3 times as many apples", "3 times more
|
||||
# apples", or "3 times the apples" is not an initial possession of
|
||||
# "3 times"; it is an incomplete comparative-multiplicative clause.
|
||||
# Letting this through as an initial consumes the scalar token and
|
||||
# defeats the ADR-0191 completeness guard. Refuse here until a real
|
||||
# compare_multiplicative operation can be emitted.
|
||||
if counted_noun.lower() == "times" and _count_token_followed_by_times(
|
||||
sentence, count_token
|
||||
):
|
||||
return None
|
||||
|
||||
# CandidateInitial requires an anchor verb token recognized in its
|
||||
# post-init whitelist (has/have/had/owns/owned/holds/held/contains/
|
||||
# contained — matched by the recognizer's narrowness rule). We pick
|
||||
|
|
@ -433,6 +444,26 @@ def _locate_token(sentence: str, target_lc: str) -> str | None:
|
|||
return None
|
||||
|
||||
|
||||
def _count_token_followed_by_times(sentence: str, count_token: str) -> bool:
|
||||
"""True when the count surface is immediately followed by ``times``.
|
||||
|
||||
The discrete-count recognizer can otherwise misread comparative
|
||||
multiplier surfaces as an initial possession of ``<N> times``. This
|
||||
check intentionally sits at the injector boundary: it only suppresses
|
||||
the malformed initial candidate and does not create any new
|
||||
admitting path.
|
||||
"""
|
||||
target = count_token.lower()
|
||||
tokens = [
|
||||
raw.strip(".,;:!?\"'()[]{}").lower()
|
||||
for raw in sentence.split()
|
||||
]
|
||||
for i, tok in enumerate(tokens[:-1]):
|
||||
if tok == target and tokens[i + 1] == "times":
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _resolve_count_value(count_token: str, count_kind: str) -> int | None:
|
||||
"""Map ``count_token`` to a numeric value.
|
||||
|
||||
|
|
|
|||
|
|
@ -89,3 +89,64 @@ def test_guard_is_refusal_only_not_answer_changing() -> None:
|
|||
# Same value, same unit-bearing graph — guard does not mutate solving.
|
||||
assert res.answer == 438.0
|
||||
assert res.selected_graph is not None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("factor", "comparative"),
|
||||
[
|
||||
("2", "as many"),
|
||||
("3", "as many"),
|
||||
("5", "as many"),
|
||||
("two", "as many"),
|
||||
("three", "as many"),
|
||||
("five", "as many"),
|
||||
("3", "more"),
|
||||
("three", "more"),
|
||||
("3", "the number of"),
|
||||
("3", "the"),
|
||||
],
|
||||
)
|
||||
def test_n_times_without_reference_refuses(
|
||||
factor: str, comparative: str
|
||||
) -> None:
|
||||
"""A dropped comparative multiplier must not admit as an initial
|
||||
possession of ``<N> times``.
|
||||
|
||||
This is the Phase-5b hard negative: if future emission work makes
|
||||
comparative-multiplicative surfaces reach the graph, the gate must
|
||||
refuse incomplete ``<N> times ...`` clauses instead of consuming
|
||||
the scalar as an ordinary count.
|
||||
"""
|
||||
res = parse_and_solve(
|
||||
"Tom has 7 apples. "
|
||||
f"Jerry has {factor} times {comparative} apples. "
|
||||
"How many apples do they have together?"
|
||||
)
|
||||
assert res.answer is None
|
||||
|
||||
|
||||
def test_n_times_as_many_with_reference_still_solves() -> None:
|
||||
"""The guard only blocks incomplete comparative clauses; a fully
|
||||
referenced compare_multiplicative graph still solves."""
|
||||
res = parse_and_solve(
|
||||
"Tom has 7 apples. Jerry has 3 times as many apples as Tom. "
|
||||
"How many apples do they have together?"
|
||||
)
|
||||
assert res.answer == 28.0
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"question",
|
||||
[
|
||||
"Tom has 7 apples. Jerry has twice as many apples. "
|
||||
"How many apples do they have together?",
|
||||
"Tom has 7 apples. Jerry has double the apples. "
|
||||
"How many apples do they have together?",
|
||||
"Ivan has 20 dice. Jerry has twice as many dice as Ivan. "
|
||||
"How many dice do they have altogether?",
|
||||
],
|
||||
)
|
||||
def test_existing_multiplier_refusals_stay_refused(question: str) -> None:
|
||||
"""Existing safe multiplier refusals must not become admissions."""
|
||||
res = parse_and_solve(question)
|
||||
assert res.answer is None
|
||||
|
|
|
|||
Loading…
Reference in a new issue