310aed9ff0
* docs: consolidate governance anchors and clean up test registries * refactor(cli): decompose cli into dedicated modules * test: fix broken test baselines and formatting * docs: add domain boundary READMEs for governance anchors * test: update baseline for determination lane * test: fix capability_pass expectation * test: fix CORE_SHOWCASE_SKIP_BUDGET enforcement * chore: cleanup CLI extraction and unreachable code
179 lines
8.2 KiB
Markdown
179 lines
8.2 KiB
Markdown
# Phase 3.1 Follow-up — Verb-coverage bottleneck on train_sample/v1
|
|
|
|
**Status:** Open recommendation
|
|
**Date:** 2026-05-28
|
|
**Author:** Shay (analysis surfaced during ADR-0174 Phase 3a)
|
|
**Parent:** [ADR-0174 — Held-Hypothesis Comprehension](../adr/ADR-0174-held-hypothesis-comprehension.md)
|
|
**Related ADRs:** ADR-0163 (path to GSM8K mastery), ADR-0167 (audit-as-teaching-evidence), ADR-0150/0152/0155/0161 (HITL corridor)
|
|
|
|
---
|
|
|
|
## Context
|
|
|
|
ADR-0174 Phase 3 specified a `correct ≥ 8` lift target on
|
|
`evals/gsm8k_math/train_sample/v1` (≥ 5 of the 21 currently-empty
|
|
`discrete_count_statement` anchors admitted via lookback). Empirical
|
|
analysis during Phase 3a implementation found this target is
|
|
**not achievable through lookback alone** on this corpus. The
|
|
substrate is built correctly; the bottleneck is elsewhere.
|
|
|
|
## What Phase 3a shipped
|
|
|
|
- `generate/comprehension/lookback.py` — the `reevaluate` operator,
|
|
`PronounResolution` refinement type, `ReevaluateResult` dataclass.
|
|
- Held-anchor emission in `recognizer_match._try_extract_discrete_count_anchor`
|
|
(pronoun-subject statements carry `requires_pronoun_resolution=True`
|
|
rather than refusing).
|
|
- Lookback wiring at `math_candidate_graph.parse_and_solve`'s
|
|
recognizer-injection branch — applies `PronounResolution` against
|
|
the existing `_discourse_prior_subjects` map; emits `lookback` JSON
|
|
trace events with `outcome ∈ {admitted, eliminated, no_antecedent}`.
|
|
- 17 acceptance tests proving the wiring works on synthetic problems
|
|
(`tests/test_adr_0174_phase3_lookback.py`).
|
|
- `wrong = 0` invariant preserved; score unchanged at 3/47/0.
|
|
|
|
## Why Phase 3a did not lift the score
|
|
|
|
The 21 empty-anchor `discrete_count_statement` refusals on
|
|
train_sample/v1 break down as:
|
|
|
|
| Structural cause | Cases |
|
|
|---|---|
|
|
| Pronoun-only (no compound clause) | 2 — 0002, 0034 |
|
|
| Compound-only | 8 |
|
|
| Pronoun + compound | 5 |
|
|
| Other narrowness fail (verb/structure) | 6 |
|
|
|
|
For Phase 3a to lift any case, **three conditions** must all hold:
|
|
|
|
1. The matcher's recognizer registry recognises the statement.
|
|
2. The extractor passes every narrowness layer **before** the pronoun
|
|
check. Specifically the verb must be in `_POSSESSION_VERBS` (`has`,
|
|
`have`, `had`) or `_ACQUISITION_VERBS` (`collected`, `collects`,
|
|
`collect`, `received`, `receives`, `receive`, `bought`, `buys`,
|
|
`buy`, `got`, `gets`, `get`).
|
|
3. The candidate-graph's regex path (`_filtered_statement_choices`)
|
|
must return empty for the same statement — otherwise the regex
|
|
path commits the candidate (with the pronoun still as actor) and
|
|
the recognizer-injection branch never runs.
|
|
|
|
Verb checks against the 13 cases with compound/pronoun structure:
|
|
|
|
| Case | Statement (excerpt) | Verb | In whitelist? |
|
|
|---|---|---|---|
|
|
| 0002 | She **splits** it up... | splits | No |
|
|
| 0034 | He can **run** 40 yards... | run | No |
|
|
| 0020 | Two puppies, two kittens... **were for sale**... | were | No |
|
|
| 0021 | He **bench presses** 15 pounds... | presses | No |
|
|
| 0027 | Malcolm **has** 240 followers... | has | **Yes** |
|
|
| 0033 | Rachel **is** 12 years old... | is | No |
|
|
| 0040 | He now **has** 2 horses... | has | **Yes** |
|
|
| 0041 | Troy **bakes** 2 pans... | bakes | No |
|
|
| 0044 | John **invests** in a bank... | invests | No |
|
|
| 0045 | On Monday he **finished** 3 surveys... | finished | No |
|
|
| 0047 | John **bakes** 12 coconut macaroons... | bakes | No |
|
|
| ... | | | |
|
|
|
|
Only **two** cases (0027, 0040) cross the verb whitelist. Both also
|
|
fail at the compound-clause narrowness layer (which comes earlier
|
|
than the pronoun check), so even adding compound-clause held
|
|
hypotheses (Phase 3b) would have to fire first.
|
|
|
|
**Conclusion:** the empirical bottleneck on train_sample/v1 is
|
|
**verb-set coverage**, not lookback or held hypotheses. ADR-0174 is
|
|
the wrong tool for moving this score.
|
|
|
|
## Recommended path forward
|
|
|
|
ADR-0163 is the correct scope for verb-coverage expansion via the
|
|
HITL corridor. The path:
|
|
|
|
1. **Run `core eval math-contemplation` on the 11 failing verbs** —
|
|
`splits`, `run`, `bench presses`, `is`, `bakes`, `invests`,
|
|
`finished`, `donated`, `wants`, `gained`, `eat`. These surface as
|
|
`MathReaderRefusalEvidence` audit rows that the contemplation lane
|
|
already consumes (ADR-0167).
|
|
2. **Operator review in workbench** — categorise each verb:
|
|
- Acquisition-class (engine should treat as `add`): `received`,
|
|
`bought`, etc. — verbs that grammatically gain quantity to actor.
|
|
Candidates from list: `gained`, `won`, `earned`, `saved`,
|
|
`accumulated`, `acquired`.
|
|
- Depletion-class (engine should treat as `subtract`): `gives`,
|
|
`loses`, `spends`. Candidates: `donated`, `gave`, `eats`,
|
|
`consumed`, `lost`, `spent`.
|
|
- Non-arithmetic verbs (engine should refuse and ask): `is`,
|
|
`wants`, `bench presses`, `splits`, `run`, `bakes`, `invests`.
|
|
These do not carry possession/acquisition semantics; the right
|
|
answer is a different intent (rate / capacity / descriptive),
|
|
not a wider `add`/`subtract` whitelist.
|
|
|
|
The first two classes ratify into the registry via the existing
|
|
ADR-0150/0152 corridor (proposal → review → packed). The third
|
|
class becomes refusal-typed evidence that informs whether a
|
|
separate recognizer category is needed (e.g. a `capacity_statement`
|
|
recognizer for "He can run 40 yards in 5 seconds" rather than
|
|
forcing it into `discrete_count_statement`).
|
|
|
|
3. **After verb widening lands** — re-run Phase 3a's lookback wiring
|
|
on the corpus. The cases that were previously verb-blocked now
|
|
reach the pronoun-check layer, and the held-hypothesis path admits
|
|
them. Expected lift from this combination: roughly the 13 cases
|
|
with pronoun/compound structure that have an arithmetic-class verb
|
|
under the widened whitelist.
|
|
|
|
## What this means for ADR-0174
|
|
|
|
The held-hypothesis substrate (Phase 1 + 2 + 3a) is correct
|
|
architecture and load-bearing for Phase 4 (in-loop contemplation) and
|
|
Phase 5 (legacy-parser removal). Its **eval impact** depends on
|
|
upstream recognizer coverage maturing through the ADR-0163.x
|
|
corridor. These two efforts are complementary, not competing — the
|
|
substrate makes lookback possible, the recognizer expansion gives
|
|
lookback something to fire on.
|
|
|
|
The cleanest sequencing is:
|
|
|
|
1. **ADR-0174 Phase 3a (this PR)** — substrate landed.
|
|
2. **ADR-0163.x verb expansion** (this brief's recommendation) —
|
|
widens the corpus surface that the substrate can act on.
|
|
3. **ADR-0174 Phase 3b** — compound-clause held hypotheses. Once the
|
|
verb-coverage bottleneck is gone, compound-clause expansion
|
|
surfaces real cases. Currently it would surface zero on
|
|
train_sample for the same reason Phase 3a does: most compound
|
|
cases also fail the verb check before reaching the clause-split
|
|
narrowness layer.
|
|
4. **ADR-0174 Phase 4** — in-loop contemplation. Builds on Phase 3
|
|
substrate.
|
|
5. **ADR-0174 Phase 5** — legacy parser removal.
|
|
|
|
## Decision needed (from operator)
|
|
|
|
- **Authorise the ADR-0163.x verb-expansion contemplation pass?**
|
|
Concretely: run `core eval math-contemplation` against the 11
|
|
failing verbs above; review the proposals in workbench; ratify
|
|
acquisition/depletion entries that are unambiguous.
|
|
|
|
- **Re-scope ADR-0174 Phase 3b** to "post-recognizer-expansion
|
|
re-measurement" rather than "compound-clause held hypotheses"?
|
|
Phase 3b should land only after verb expansion exposes cases that
|
|
exercise its compound-clause logic.
|
|
|
|
No timelines are proposed; this is a sequencing recommendation. The
|
|
substrate work in Phase 3a is already merged on its own merits
|
|
(correctness and Phase 4/5 prerequisite); Phase 3b waits on
|
|
recognizer coverage.
|
|
|
|
## Cross-references
|
|
|
|
- ADR-0174 §Phase 3 acceptance — the criteria this brief documents
|
|
as unmet (with structural-cause analysis).
|
|
- `tests/test_adr_0174_phase3_lookback.py` — proves the substrate
|
|
works on synthetic problems even though no train_sample case
|
|
exercises it.
|
|
- `feedback-wrong-zero-hazard-case-0050` memory — verb expansion
|
|
must preserve the case-0050 canary; the recommended depletion-class
|
|
additions should be reviewed against this hazard before ratification.
|
|
- `thesis-decoding-not-generating` — the verb-class
|
|
contemplation/HITL path is the right "teach the engine to find
|
|
better" mechanism; widening the static whitelist directly would be
|
|
"storing another found thing."
|