core/docs/handoff/VERB-EXPANSION-DISPATCH-PACK.md
Shay 796b446b49 docs(handoff): require multi-actor pronoun safety test in VE-A/B acceptance
Adds second mandatory hazard test to VE-A/B acceptance criteria:
multi-actor pronoun ambiguity must trigger no_antecedent_ambiguous
refusal per ADR-0174 Phase 3a defense.

Verb expansion widens the cases that reach Phase 3a lookback wiring;
without this test the multi-actor wrong=0 hazard could fire silently
in production. Surfaced by 2026-05-28 Phase 1-3a lookback review.

References: project-adr-0174-multi-actor-pronoun-hazard memory,
CLAUDE.md §Lookback Review Discipline.
2026-05-28 09:19:46 -07:00

377 lines
16 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Verb-Coverage Expansion — Dispatch Pack
**Goal:** Widen the narrowness boundary that gates `discrete_count_statement`
admission so the held-hypothesis substrate landed in ADR-0174 Phase 3a
has cases to act on. Concretely: review the 11 verbs surfacing in
`evals/gsm8k_math/train_sample/v1` empty-anchor refusals and decide,
per verb, whether to add to `_ACQUISITION_VERBS` / `_POSSESSION_VERBS`,
add to a new depletion-verb class, or refuse and route to a different
recognizer category.
**Parent ADRs:** ADR-0163 (path to GSM8K mastery), ADR-0150/0152/0155/0161
(contemplation → proposal → HITL → ratification corridor), ADR-0167
(audit-as-teaching-evidence), ADR-0174 (held-hypothesis comprehension)
**Parent briefs:**
- `docs/handoff/PHASE-3.1-FOLLOWUP-RECOGNIZER-EXPANSION.md` — analysis
identifying the verb-coverage bottleneck.
**Type:** Implementation dispatch + operator decision pack; not a
doctrine ADR. Touches a hardcoded narrowness boundary so every
acceptance is operator-reviewed and individually justified against the
`wrong = 0` invariant.
---
## Why this brief exists
ADR-0174 Phase 3a (PR #423) shipped the lookback substrate
(`reevaluate`, `PronounResolution`, held-hypothesis emission). The
substrate is correct and tested but **does not fire on any
train_sample case** because all 21 empty-anchor
`discrete_count_statement` refusals fail at the verb-whitelist layer
**before** reaching the pronoun-resolution check.
The whitelist lives in `generate/recognizer_match.py`:
```python
_POSSESSION_VERBS: Final[frozenset[str]] = frozenset({
"has", "have", "had",
})
_ACQUISITION_VERBS: Final[frozenset[str]] = frozenset({
"collected", "collects", "collect",
"received", "receives", "receive",
"bought", "buys", "buy",
"got", "gets", "get",
})
```
This is a deliberate narrowness boundary, not registry-derived data.
Widening it is a code change with operator review (not a pack
mutation through the HITL corridor). The brief below produces the
evidence the operator needs to decide each verb.
---
## Bundling rule
**Multiple PRs, one per verb class (not one per verb).** Each PR
ships:
1. The narrowness expansion in `generate/recognizer_match.py`
2. A test in `tests/test_adr_0174_phase3_lookback.py` (or sibling)
asserting the widened class admits cases the held-hypothesis
substrate now handles
3. A `wrong = 0` proof against `train_sample/v1` AND case 0050 (the
canary).
Three suggested PRs, each independently mergeable:
- **VE-A — Acquisition widening** (`feat/verb-expansion-acquisition`)
Adds reviewed acquisition verbs to `_ACQUISITION_VERBS`.
- **VE-B — Depletion class introduction** (`feat/verb-expansion-depletion`)
Adds a new `_DEPLETION_VERBS: Final[frozenset[str]]` and routes
it to `CandidateOperation(kind="subtract")`. Includes its own
`anchor_kind="depletion"` branch in
`inject_discrete_count_statement`.
- **VE-C — Refusal-typed evidence** (`feat/verb-expansion-non-arithmetic`)
For verbs that should NOT widen any whitelist (the verb does not
grammatically denote possession / acquisition / depletion), emit a
structured refusal-evidence record so the contemplation lane can
decide whether a new recognizer category is needed (e.g.
`capacity_statement`, `descriptive_action_frame`). No production
whitelist changes; just better trace.
VE-A and VE-B are independent. VE-C requires nothing in the others.
All three can be in flight in parallel.
---
## Operator profile
- **Opus** for VE-A (load-bearing wrong=0 surface — widening the
acquisition class touches every train_sample case via the recognizer
pipeline). Case 0050 hazard pin is mandatory.
- **Opus** for VE-B (adds an entirely new operation-kind path; same
wrong=0 hazard surface). Includes the depletion-class case-0050 hazard
check (a depletion-verb addition must not flip case 0050 from refused
to wrong).
- **Sonnet** for VE-C (instrumentation-only; no production behavior
change). Tight scope, no architectural risk.
---
## The 11 verbs — recommended classification
Each row is the operator's input for the dispatch. Final
classification is the operator's call after reviewing the contemplation
evidence; this column is a recommendation only.
| Verb (lemma) | Surface forms in evidence | Recommended | Why |
|---|---|---|---|
| `gain` | gained, gains, gain | **Acquisition** (VE-A) | Grammatically gains quantity to actor: "Orlando gained 5 pounds" → +5. Hazard: `gained` can be delta-of-attribute (weight, age) — the existing `_ACQUISITION_VERBS` comment explicitly excludes it for this reason. Operator must confirm. |
| `earn` | earned, earns, earn | **Acquisition** (VE-A) | Same shape as `gained`; safer because monetary/discrete-count semantics are more uniform. |
| `save` | saved, saves, save | **Acquisition** (VE-A) | Ambiguous per existing comment ("saved time" vs "saved money"). Refusal-preferring approach: require the noun class (time vs money) to gate admission. |
| `accumulate` | accumulated, accumulates, accumulate | **Acquisition** (VE-A) | Unambiguous. |
| `acquire` | acquired, acquires, acquire | **Acquisition** (VE-A) | Unambiguous. |
| `donate` | donated, donates, donate | **Depletion** (VE-B) | Grammatically removes quantity from actor: "The bookstore donated 48 boxes" → -48. Subject is the giver. |
| `give` | gave, gives, give | **Depletion** (VE-B) | Same shape as `donate`. Existing `TRANSFER_VERBS` handles `give to X` — depletion is the no-target form. |
| `lose` | lost, loses, lose | **Depletion** (VE-B) | "She lost 3 marbles" → -3. Hazard: `lost` can semantically invert direction in comparisons ("Alice lost 3 more than Bob"). Existing comparison code excludes `lost` for this reason. Depletion-class addition must preserve that exclusion in comparison contexts. |
| `spend` | spent, spends, spend | **Depletion** (VE-B) | Money-class depletion; safer than `lose` because monetary context constrains semantics. |
| `eat` | eats, ate, eat | **Depletion** (VE-B) | "The guests eat 1 pan" → -1. Constrained to food-noun classes for safety. |
| `bench presses` / `splits` / `runs` / `bakes` / `invests` / `is` / `wants` | — | **Non-arithmetic** (VE-C) | These verbs do NOT carry possession/acquisition/depletion semantics. Routing them through the discrete_count_statement recognizer is the wrong intent. They want a separate recognizer category (e.g. `rate_statement`, `capacity_statement`, `descriptive_frame`) that is out of scope for this brief. |
### Hazard pinning — non-negotiable
Every VE-A / VE-B PR MUST include **two** safety tests.
#### Hazard test #1 — case 0050 canary
```python
def test_case_0050_remains_refused_after_verb_widening(self) -> None:
"""gsm8k-train-sample-v1-0050 is the wrong=0 canary. The verb
widening must NOT flip it from refused to wrong."""
from generate.math_candidate_graph import parse_and_solve
text = (
"Mark does a gig every other day for 2 weeks. "
"He gets paid $50 per gig. He then gets a 50% raise. "
"How much money does he make per week?"
)
r = parse_and_solve(text)
assert r.answer is None, (
f"case 0050 wrong=0 hazard violated: verb widening admitted "
f"answer {r.answer!r}; must remain refused"
)
```
Reference: memory `feedback-wrong-zero-hazard-case-0050`.
#### Hazard test #2 — multi-actor pronoun safety (ADR-0174 Phase 3a)
VE-A / VE-B widens the verb whitelist, which means more
discrete_count_statement cases will reach the ADR-0174 Phase 3a
lookback wiring. When the widened verb appears in a multi-actor
problem, the Phase 3a defense (`no_antecedent_ambiguous` refusal on
multiple distinct prior subjects) MUST fire — otherwise the pronoun
could be mis-resolved and produce a wrong answer with no downstream
safety net catching it.
```python
def test_multi_actor_pronoun_with_widened_verb_refuses_safely(self) -> None:
"""ADR-0174 Phase 3a multi-actor hazard — when more than one
proper-noun subject appears in prior context, the pronoun
resolver must refuse rather than guess the most-recent.
Verb widening must preserve this defense."""
from generate.math_candidate_graph import parse_and_solve
import json
# Replace <widened_verb> with one this PR adds.
text = (
"Alice has 5 marbles. "
"Bob has 3 marbles. "
"She <widened_verb> 2 marbles. "
"How many marbles does Bob have?"
)
r = parse_and_solve(text)
# MUST refuse — wrong attribution is the hazard.
assert r.answer is None
# Trace MUST include no_antecedent_ambiguous.
lookback_events = [
json.loads(ev) for ev in r.reader_trace
if json.loads(ev).get("layer") == "lookback"
]
assert any(
ev.get("outcome") == "no_antecedent_ambiguous"
for ev in lookback_events
), f"multi-actor defense did not fire; trace={lookback_events}"
```
Reference: memory `project-adr-0174-multi-actor-pronoun-hazard`,
CLAUDE.md §Lookback Review Discipline (the discipline that surfaced
this hazard at the right time).
---
## Per-PR workflow
### VE-A — Acquisition widening
```bash
git fetch origin main && \
git worktree add /tmp/wt-ve-a origin/main && \
cd /tmp/wt-ve-a && \
git checkout -b feat/verb-expansion-acquisition
```
Touchpoints:
- `generate/recognizer_match.py:835` — add reviewed lemmas to
`_ACQUISITION_VERBS` frozenset. Each addition has an inline comment
citing the hazard review (e.g. `# Reviewed 2026-MM-DD: 'gained'
grounds as monetary/discrete-count delta; refuses on
delta-of-attribute via unit-noun class check upstream.`).
- `generate/math_roundtrip.py:60` — ensure each added lemma is also
in `ADD_VERBS` so `roundtrip_admissible.verb_registered` passes.
- New tests in
`tests/test_adr_0174_phase3_lookback.py::TestVerbExpansionAcquisition`
covering: (1) each added verb admits a synthetic
pronoun-subject sentence through the held-hypothesis path,
(2) case 0050 remains refused, (3) at least one real train_sample
case lifts from refused to correct.
Acceptance:
- ≥ 1 case lift on train_sample/v1 (verify via `uv run python -m
evals.gsm8k_math.train_sample.v1.runner`)
- wrong=0 preserved across train_sample AND case 0050
- smoke 67/67, packs 141+/141+, lanes 8/8
- All Phase 13a tests still pass (113 baseline)
### VE-B — Depletion class introduction
```bash
git fetch origin main && \
git worktree add /tmp/wt-ve-b origin/main && \
cd /tmp/wt-ve-b && \
git checkout -b feat/verb-expansion-depletion
```
Touchpoints:
- `generate/recognizer_match.py` — add
`_DEPLETION_VERBS: Final[frozenset[str]]` near
`_ACQUISITION_VERBS`. Extend `_try_extract_discrete_count_anchor`'s
verb dispatch to include `elif verb in _DEPLETION_VERBS: anchor_kind
= "depletion"`.
- `generate/recognizer_anchor_inject.py` — add
`anchor_kind == "depletion"` branch in
`inject_discrete_count_statement`; emits
`CandidateOperation(kind="subtract")` via a new
`_build_operation_from_discrete_count_depletion` helper that
mirrors the existing acquisition helper but with `kind="subtract"`.
- `generate/math_roundtrip.py` — ensure each depletion lemma is in
`SUBTRACT_VERBS`.
Acceptance:
- Synthetic depletion test passes (e.g. "Sam has 10 apples. He gave 3
apples to a friend. How many apples does Sam have?")
- ≥ 1 case lift on train_sample/v1 from the depletion class
- wrong=0 preserved across train_sample AND case 0050
- Comparison-context hazard: "Alice lost 3 more than Bob" must not
parse as depletion (already excluded from comparison verbs).
### VE-C — Non-arithmetic refusal evidence
```bash
git fetch origin main && \
git worktree add /tmp/wt-ve-c origin/main && \
cd /tmp/wt-ve-c && \
git checkout -b feat/verb-expansion-non-arithmetic
```
Touchpoints (no production behavior change):
- `generate/recognizer_match.py` — in
`_try_extract_discrete_count_anchor`, when the verb is not in any
whitelist, augment the existing `return None` with a marker on the
anchor (something like
`_DISCARDED_VERB_TRACE: dict[str, list[str]] = {}` that captures
the verb + sentence for downstream inspection). DO NOT emit
candidates; the production path remains a clean refusal.
- New CLI surface: `core teaching unrecognized-verbs --since <case-id>`
that surfaces the captured verbs as candidate evidence for a future
ADR-0163.D.x recognizer-category proposal.
Acceptance:
- All 7 non-arithmetic verbs surface in the new CLI output when run
against train_sample/v1.
- No score change; no admission behavior change.
- Refusal trace events gain a `verb_class_unknown` outcome with the
verb lemma.
---
## Truth tests
Before any VE-A/B/C PR merges, verify:
1. **wrong=0 invariant.** `uv run python scripts/verify_lane_shas.py`
passes. `train_sample/v1` report counts have `wrong: 0`.
2. **Case 0050 canary.** Explicit test that 0050 remains refused.
3. **Lift evidence.** For VE-A and VE-B, at least one train_sample
case moves from `refused` to `correct`. If zero lift, the PR is
premature — the verb widening exposed no real case, indicating the
hazard isn't justified. Refuse to ship and re-scope.
4. **Phase 3a substrate fires.** For VE-A and VE-B, the lifted case
must show a `lookback` trace event with
`outcome: "admitted"` — proving the held-hypothesis path was on
the critical path, not bypassed by the regex parser.
---
## Out of scope
- Comparison-verb additions (`compare_additive` / `compare_multiplicative`).
ADR-0123 owns that surface.
- Rate-verb additions. ADR-0122 / ADR-0163.D.2.2 own that surface.
- Transfer-verb additions (`give to X`, `send to X`). Already covered
by `TRANSFER_VERBS`; widening that is a separate brief.
- Verb-class proposals that would change the **structure** of
`_try_extract_discrete_count_anchor`'s narrowness chain (e.g.
registry-loadable verb sets). That is its own ADR; this brief
preserves the hardcoded boundary because that boundary is the
wrong=0 safety surface.
- ADR-0174 Phase 3b (compound-clause held hypotheses). Wait until
VE-A or VE-B lands so the compound-clause logic has cases to admit.
---
## Sequencing recommendation
```
VE-C (instrumentation)
VE-A (acquisition widening) ─┐
├─→ measure: does Phase 3a lookback fire?
VE-B (depletion class) ─┘ if yes: ADR-0174 Phase 3b
if no: re-scope non-firing cause
```
VE-C is independent and can land first to give us better evidence on
which Phase 3b targets are worth pursuing. VE-A and VE-B can land in
parallel; their case-0050 tests are independent.
---
## Reads required FIRST (for the operator running each VE-x)
- `generate/recognizer_match.py:812-840` — current verb-set
definitions and the design comments justifying each exclusion.
- `generate/recognizer_match.py:912-1010` — the discrete-count
anchor extractor (the narrowness chain).
- `generate/math_roundtrip.py:60-150``KIND_TO_VERBS`,
`ADD_VERBS`, `SUBTRACT_VERBS` (must add lemmas here too for
`roundtrip_admissible` parity).
- `generate/recognizer_anchor_inject.py:186-300` — injector dispatch;
depletion needs its own builder.
- `docs/handoff/PHASE-3.1-FOLLOWUP-RECOGNIZER-EXPANSION.md`
empirical analysis this brief operationalises.
- Memory: `feedback-wrong-zero-hazard-case-0050` (case 0050 hazard
is mandatory pinning for every PR).
- Memory: `thesis-decoding-not-generating` (each verb decision is
"does this teach the engine to find better?" — non-arithmetic
verbs route to VE-C precisely because widening for them would be
"storing another found thing").
---
## Decision needed before dispatch
- **Authorise VE-A acquisition widening?** Specifically: which of
`gain`, `earn`, `save`, `accumulate`, `acquire` (and their
inflections) are in scope. Each adds a `wrong=0` hazard surface.
- **Authorise VE-B depletion class introduction?** Specifically:
which of `donate`, `give`, `lose`, `spend`, `eat`. The depletion
class is a new operation-kind path; whether to introduce it AT
ALL is a structural decision.
- **Authorise VE-C non-arithmetic refusal evidence?** No production
risk; gives downstream contemplation/ADR work better evidence.
No timelines proposed; this is operator decision input plus dispatch
templates. The PRs ship when the operator says go.