Commit graph

2 commits

Author SHA1 Message Date
Shay
9fc31eeaa4
feat(brief-11/11B): reader closure audit artifact — full taxonomy + rejected naive fix (#345)
## Summary

PR 11B in the Brief 11 sequence. Closes the missing-operator inference gap
left by 11A (#343) and ships the per-case audit artifact that Brief 11 §Gate 2
identifies as "the main Brief 11 artifact."

## Why this PR does NOT touch the reader runtime

The naive closure fix for `pre_frame_filler_sentence` (drain
`statement_terminator` at pre-frame) lifts 2 cases from refused → admitted
but creates a `wrong > 0` hazard on `gsm8k-train-sample-v1-0050`:

```
Mark does a gig every other day for 2 weeks.  For each gig, he plays 3 songs.
... How many minutes did he play?
```

With the drain enabled, the reader admits `Operation(mark, add, 3, songs)`
with unknown unit `minute` and would project to a wrong answer. The stricter
variant (`pending_entity_ref is None` + no quantities) fires on 0 of the 11
candidate cases. Per Brief 11 §"Failure modes to avoid §1 — Correct-count
greed," this PR rejects both variants and routes the closure fix to a
follow-up that adds the required verb vocabulary or sentence-intent
classifier.

## Deliverables

- `generate/comprehension/audit.py` — three new missing-operator labels:
  - `pre_frame_filler_sentence` (8 cases)
  - `descriptive_frame_question` (2 cases)
  - `question_frame_slot` (1 case)
  Closes the 11-case `None`-operator gap left by 11A.
- `evals/gsm8k_math/train_sample/v1/audit_brief_11.json` — per-case audit
  artifact pinned by tests.
- `evals/gsm8k_math/train_sample/v1/audit_brief_11.md` — narrative summary
  including the rejected-fix design tension and ranked Brief 11B-step-2
  backlog.
- `tests/test_brief_11b_audit_artifact.py` — 12 tests pinning the new labels,
  the per-case artifact, the wrong=0 invariant, and the refusal taxonomy.

## Bottleneck taxonomy (after Brief 11B labelling)

| missing_operator              | count | category               |
|-------------------------------|------:|------------------------|
| quantity_extraction           | 9     | incomplete_operation   |
| lexicon_entry                 | 9     | unknown_word           |
| multi_quantity_composition    | 8     | incomplete_operation   |
| pre_frame_filler_sentence     | 8     | unexpected_category    |
| pronoun_resolution            | 3     | unresolved_pronoun     |
| fraction_percentage_literal   | 3     | unexpected_category    |
| unit_binding                  | 3     | unattached_quantity    |
| descriptive_frame_question    | 2     | unexpected_category    |
| (others, 1 each)              | 5     | various                |

## Test plan

- 12 new tests in `tests/test_brief_11b_audit_artifact.py` pass
- 23 existing 11A tests in `tests/test_brief_11_audit.py` pass
- No runtime changes; reader byte-identical to main

## Hard invariants preserved

- `wrong == 0` — no runtime change, no new admissions
- ADR-0166 — no new canonical eval lanes added; existing
  `evals/gsm8k_math/train_sample/v1/` artifact set extended
- No teaching store / pack mutation

## Follow-up

- **11B-step-2** — verb-vocabulary expansion or sentence-intent classifier
  for `pre_frame_filler_sentence` (8 cases). See audit_brief_11.md §"design
  tension" for the rejected one-line variants and why they fail wrong=0.
- **11C** — existing-lane capability snapshot (still gated on 11B-step-2 or
  another closure pass).
2026-05-27 05:35:06 -07:00
Shay
aa53fcf78d
feat(brief-11/11A): reader closure audit — per-case refusal taxonomy, graph-completeness helpers, regression tests (#343) 2026-05-27 05:14:42 -07:00