feat(brief-11/11B): reader closure audit artifact — full taxonomy + rejected naive fix (#345)

## Summary

PR 11B in the Brief 11 sequence. Closes the missing-operator inference gap
left by 11A (#343) and ships the per-case audit artifact that Brief 11 §Gate 2
identifies as "the main Brief 11 artifact."

## Why this PR does NOT touch the reader runtime

The naive closure fix for `pre_frame_filler_sentence` (drain
`statement_terminator` at pre-frame) lifts 2 cases from refused → admitted
but creates a `wrong > 0` hazard on `gsm8k-train-sample-v1-0050`:

```
Mark does a gig every other day for 2 weeks.  For each gig, he plays 3 songs.
... How many minutes did he play?
```

With the drain enabled, the reader admits `Operation(mark, add, 3, songs)`
with unknown unit `minute` and would project to a wrong answer. The stricter
variant (`pending_entity_ref is None` + no quantities) fires on 0 of the 11
candidate cases. Per Brief 11 §"Failure modes to avoid §1 — Correct-count
greed," this PR rejects both variants and routes the closure fix to a
follow-up that adds the required verb vocabulary or sentence-intent
classifier.

## Deliverables

- `generate/comprehension/audit.py` — three new missing-operator labels:
  - `pre_frame_filler_sentence` (8 cases)
  - `descriptive_frame_question` (2 cases)
  - `question_frame_slot` (1 case)
  Closes the 11-case `None`-operator gap left by 11A.
- `evals/gsm8k_math/train_sample/v1/audit_brief_11.json` — per-case audit
  artifact pinned by tests.
- `evals/gsm8k_math/train_sample/v1/audit_brief_11.md` — narrative summary
  including the rejected-fix design tension and ranked Brief 11B-step-2
  backlog.
- `tests/test_brief_11b_audit_artifact.py` — 12 tests pinning the new labels,
  the per-case artifact, the wrong=0 invariant, and the refusal taxonomy.

## Bottleneck taxonomy (after Brief 11B labelling)

| missing_operator              | count | category               |
|-------------------------------|------:|------------------------|
| quantity_extraction           | 9     | incomplete_operation   |
| lexicon_entry                 | 9     | unknown_word           |
| multi_quantity_composition    | 8     | incomplete_operation   |
| pre_frame_filler_sentence     | 8     | unexpected_category    |
| pronoun_resolution            | 3     | unresolved_pronoun     |
| fraction_percentage_literal   | 3     | unexpected_category    |
| unit_binding                  | 3     | unattached_quantity    |
| descriptive_frame_question    | 2     | unexpected_category    |
| (others, 1 each)              | 5     | various                |

## Test plan

- 12 new tests in `tests/test_brief_11b_audit_artifact.py` pass
- 23 existing 11A tests in `tests/test_brief_11_audit.py` pass
- No runtime changes; reader byte-identical to main

## Hard invariants preserved

- `wrong == 0` — no runtime change, no new admissions
- ADR-0166 — no new canonical eval lanes added; existing
  `evals/gsm8k_math/train_sample/v1/` artifact set extended
- No teaching store / pack mutation

## Follow-up

- **11B-step-2** — verb-vocabulary expansion or sentence-intent classifier
  for `pre_frame_filler_sentence` (8 cases). See audit_brief_11.md §"design
  tension" for the rejected one-line variants and why they fail wrong=0.
- **11C** — existing-lane capability snapshot (still gated on 11B-step-2 or
  another closure pass).

evals/gsm8k_math/train_sample/v1/audit_brief_11.md

@@ -0,0 +1,143 @@
+# Brief 11B — Reader Closure Audit (GSM8K train-sample v1)
+
+This document is the human-readable companion to
+`audit_brief_11.json`. It captures the per-case Phase 2 reader audit over the
+50-case sealed train sample, with the missing-operator inference extended in
+this PR (see `generate/comprehension/audit.py`).
+
+## Per-case counts
+
+| Outcome    | Count |
+|------------|------:|
+| admitted   | 0     |
+| refused    | 50    |
+| **wrong**  | **0** |
+
+`wrong == 0` is preserved by construction: this PR does **not** modify the
+reader runtime; only the audit inference layer and a new artifact are added.
+
+## Refusal taxonomy
+
+| refusal_reason         | count |
+|------------------------|------:|
+| incomplete_operation   | 18    |
+| unexpected_category    | 14    |
+| unknown_word           | 11    |
+| unattached_quantity    | 3     |
+| unresolved_pronoun     | 3     |
+| no_question_target     | 1     |
+
+## Missing-operator taxonomy (load-bearing)
+
+This is the work backlog that any subsequent reader-closure PR must address.
+Every previously-`None` operator has now been labelled.
+
+| missing_operator              | count | category          |
+|-------------------------------|------:|-------------------|
+| quantity_extraction           | 9     | incomplete_operation |
+| lexicon_entry                 | 9     | unknown_word      |
+| multi_quantity_composition    | 8     | incomplete_operation |
+| **pre_frame_filler_sentence** | **8** | **unexpected_category (new label)** |
+| pronoun_resolution            | 3     | unresolved_pronoun |
+| fraction_percentage_literal   | 3     | unexpected_category |
+| unit_binding                  | 3     | unattached_quantity |
+| **descriptive_frame_question**| **2** | **unexpected_category (new label)** |
+| compound_numeric_literal      | 1     | unknown_word      |
+| compound_time_literal         | 1     | unknown_word      |
+| multi_subject_sentence        | 1     | unexpected_category |
+| question_target_slot          | 1     | no_question_target |
+| **question_frame_slot**       | **1** | **incomplete_operation (new label)** |
+
+## New labels introduced in this PR
+
+### `pre_frame_filler_sentence` (8 cases)
+
+A sentence whose verb the reader does not recognise as a frame-opener, so the
+sentence reaches its terminator while still at pre-frame state with a subject
+entity pinned but no quantity and no transactional verb. Examples:
+
+- `gsm8k-train-sample-v1-0028`: *"Tom opens an amusement park."*
+- `gsm8k-train-sample-v1-0035`: *"... go strawberry picking."*
+- `gsm8k-train-sample-v1-0050`: *"Mark does a gig every other day for 2 weeks."*
+
+### `pre_frame_filler_sentence` — design tension (open for operator review)
+
+A naive fix (drain `statement_terminator` at pre-frame and route through
+`_end_descriptive_frame`) **lifts 2 cases from refused → admitted** but creates
+a real `wrong > 0` hazard on case `gsm8k-train-sample-v1-0050`:
+
+```
+Mark does a gig every other day for 2 weeks.  For each gig, he plays 3 songs.
+2 of the songs are 5 minutes long and the last song is twice that long.
+How many minutes did he play?
+```
+
+With pre-frame drain enabled, the reader admits a partial graph
+`Operation(mark, add, 3, songs)` with unknown unit `minute` and would project to
+a wrong answer (~3 vs expected 280). The looser variant of the fix is therefore
+**rejected** by this PR per Brief 11 §"Failure modes to avoid §1 — Correct-count
+greed."
+
+A stricter variant (gate the drain on `pending_entity_ref is None and not
+sentence_state.quantities`) fires on 0 of the 11 candidate cases — all of them
+have a subject entity pinned before the terminator — so it is purely additive
+overhead without lift.
+
+The honest closure path requires either:
+
+1. **Verb-vocabulary expansion** — promote currently-unknown statement verbs
+   (`opens`, `does`, `goes`, `plays`, `splits`, `picking`) to recognised
+   categories that open the correct statement frame; or
+2. **Sentence-level intent classification before frame dispatch** — distinguish
+   *context-setting* from *math-content* statements deterministically before
+   committing to (or refusing) a frame.
+
+Both are real capability work, not a one-line drain rule. Brief 11B-step-2
+(future PR) is the right home.
+
+### `descriptive_frame_question` (2 cases)
+
+A `?` terminator reached while the sentence has fallen into `descriptive_frame`
+without a `question_open` token — the question target slot was missed.
+Distinct refusal mechanism from the `no_question_target` finalize check.
+
+### `question_frame_slot` (1 case)
+
+A `question_frame` opened (correct intent) but did not receive a required slot
+such as `unit_class`. Separated from generic `incomplete_operation` so the
+operator can attack question-slot recovery independently of statement-frame
+slot recovery.
+
+## Highest-leverage backlog (Brief 11B-step-2 candidates)
+
+Ranked by `count × proximity-to-existing-mechanism`:
+
+1. `lexicon_entry` (9) — vocabulary gap; lowest-risk extension since adding a
+   lemma to the cognition pack cannot create wrong admissions without also
+   passing the graph completeness check.
+2. `multi_quantity_composition` (8) — requires the reader to emit two
+   `PartialInitialPossession` or `PartialOperation` entries from a single
+   compound-noun statement ("5 full boxes and 5 loose crayons"). Real frame
+   work.
+3. `pre_frame_filler_sentence` (8) — see design tension above.
+4. `quantity_extraction` (9) — usually co-occurs with `compound_*` literals or
+   indefinite quantifiers ("some", "a few"); needs a deliberate
+   indefinite-quantity policy before any admission.
+
+## Reproducibility
+
+```bash
+uv run python -c "
+import json
+from generate.comprehension.audit import audit_problem
+from generate.comprehension.state import ReaderRefusal
+with open('evals/gsm8k_math/train_sample/v1/cases.jsonl') as f:
+    for line in f:
+        c = json.loads(line)
+        r, rows = audit_problem(c['question'], case_id=c['case_id'])
+        print(c['case_id'], type(r).__name__,
+              rows[0].missing_operator if rows else '-')
+"
+```
+
+The artifact `audit_brief_11.json` is pinned by `tests/test_brief_11b_audit_artifact.py`.

generate/comprehension/audit.py

@@ -192,6 +192,28 @@ _OPERATOR_INFERENCE_RULES: list[tuple[str, re.Pattern[str], str]] = [
         re.compile(r"."),
         "graph_construction",
     ),
+    # Brief 11B: pre-frame statement_terminator — sentence ended without
+    # opening any frame. May be context filler or a verb the reader cannot
+    # admit yet. Distinct from multi_subject because no second entity.
+    (
+        "unexpected_category",
+        re.compile(r"'statement_terminator'.*pre-frame", re.IGNORECASE),
+        "pre_frame_filler_sentence",
+    ),
+    # Brief 11B: pre-frame "?" reached in a descriptive_frame — question
+    # target slot was missed before the terminator arrived.
+    (
+        "unexpected_category",
+        re.compile(r"'question_terminator'.*descriptive_frame", re.IGNORECASE),
+        "descriptive_frame_question",
+    ),
+    # Brief 11B: question_frame opened but a required slot (e.g. unit_class)
+    # never arrived. Labelled separately from generic incomplete_operation.
+    (
+        "incomplete_operation",
+        re.compile(r"question_frame missing required slot", re.IGNORECASE),
+        "question_frame_slot",
+    ),
 ]
 
 

tests/test_brief_11b_audit_artifact.py

@@ -0,0 +1,184 @@
+"""Brief 11B — pin the audit artifact and the extended missing-operator
+inference labels added in this PR.
+
+Hard invariants:
+
+* ``wrong == 0`` (this PR does not change the reader runtime).
+* The audit artifact at ``evals/gsm8k_math/train_sample/v1/audit_brief_11.json``
+  is reproducible from ``audit_problem`` over the sealed train-sample.
+* Every refused case now carries a non-None ``missing_operator`` label.
+* The three new labels introduced in this PR fire on the documented
+  detail-string patterns.
+"""
+
+from __future__ import annotations
+
+import json
+from collections import Counter
+from pathlib import Path
+
+import pytest
+
+from generate.comprehension.audit import (
+    _infer_missing_operator,
+    audit_problem,
+)
+from generate.comprehension.state import ReaderRefusal
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+CASES_PATH = REPO_ROOT / "evals/gsm8k_math/train_sample/v1/cases.jsonl"
+ARTIFACT_PATH = REPO_ROOT / "evals/gsm8k_math/train_sample/v1/audit_brief_11.json"
+
+
+@pytest.fixture(scope="module")
+def cases() -> list[dict]:
+    with CASES_PATH.open() as f:
+        return [json.loads(line) for line in f]
+
+
+@pytest.fixture(scope="module")
+def artifact() -> dict:
+    with ARTIFACT_PATH.open() as f:
+        return json.load(f)
+
+
+# ---------------------------------------------------------------------------
+# Extended inference rule tests (the three new labels).
+# ---------------------------------------------------------------------------
+
+
+def test_pre_frame_filler_sentence_label():
+    detail = (
+        "category 'statement_terminator' (word='.') at pre-frame position 7 "
+        "not handled; may be Phase-3 scope"
+    )
+    assert (
+        _infer_missing_operator("unexpected_category", detail)
+        == "pre_frame_filler_sentence"
+    )
+
+
+def test_descriptive_frame_question_label():
+    detail = "category 'question_terminator' (word='?') not drainable in descriptive_frame"
+    assert (
+        _infer_missing_operator("unexpected_category", detail)
+        == "descriptive_frame_question"
+    )
+
+
+def test_question_frame_slot_label():
+    detail = "question_frame missing required slot(s): unit_class"
+    assert (
+        _infer_missing_operator("incomplete_operation", detail)
+        == "question_frame_slot"
+    )
+
+
+def test_pre_frame_label_does_not_capture_multi_subject():
+    detail = (
+        "second entity 'Bob' at pre-frame position 3; multi-subject sentences "
+        "are Phase-2.1 scope"
+    )
+    assert (
+        _infer_missing_operator("unexpected_category", detail)
+        == "multi_subject_sentence"
+    )
+
+
+# ---------------------------------------------------------------------------
+# Artifact reproducibility.
+# ---------------------------------------------------------------------------
+
+
+def test_artifact_exists_and_has_expected_shape(artifact):
+    assert artifact["schema_version"] == 1
+    assert artifact["case_count"] == 50
+    assert artifact["summary"]["admitted"] == 0
+    assert artifact["summary"]["refused"] == 50
+    assert artifact["invariants"]["wrong_count"] == 0
+    assert artifact["invariants"]["reader_runtime_changes_in_this_pr"] is False
+    assert len(artifact["per_case"]) == 50
+
+
+def test_artifact_reproduces_from_audit(cases, artifact):
+    """Every per-case row in the artifact must round-trip from audit_problem."""
+    by_id = {row["case_id"]: row for row in artifact["per_case"]}
+    for case in cases:
+        cid = case["case_id"]
+        result, rows = audit_problem(case["question"], case_id=cid)
+        expected = by_id[cid]
+        if isinstance(result, ReaderRefusal):
+            assert expected["outcome"] == "refused"
+            assert expected["refusal_reason"] == result.reason
+            assert expected["sentence_index"] == result.sentence_index
+            assert expected["token_index"] == result.token_index
+            assert expected["token_text"] == result.token_text
+            assert expected["refusal_detail"] == result.detail
+            if rows:
+                assert expected["missing_operator"] == rows[0].missing_operator
+                assert expected["recognized_terms"] == list(rows[0].recognized_terms)
+                assert expected["skipped_frame"] == rows[0].skipped_frame
+        elif result is None:
+            assert expected["outcome"] == "no_sentences"
+        else:
+            assert expected["outcome"] == "admitted"
+
+
+def test_every_refused_case_has_labeled_operator(artifact):
+    """No more None-operator refusals — Brief 11B closes the inference gap."""
+    for row in artifact["per_case"]:
+        if row["outcome"] != "refused":
+            continue
+        assert row["missing_operator"] is not None, (
+            f"{row['case_id']}: refused with no labeled missing_operator "
+            f"(reason={row['refusal_reason']!r}, detail={row['refusal_detail']!r})"
+        )
+
+
+def test_new_labels_present_in_summary(artifact):
+    """The three new labels must be represented in the artifact."""
+    ops = artifact["summary"]["missing_operators"]
+    assert ops.get("pre_frame_filler_sentence", 0) >= 1
+    assert ops.get("descriptive_frame_question", 0) >= 1
+    assert ops.get("question_frame_slot", 0) >= 1
+
+
+def test_invariant_no_wrong_admission(artifact):
+    """The load-bearing wrong=0 invariant."""
+    assert artifact["invariants"]["wrong_count"] == 0
+    assert artifact["summary"]["admitted"] == 0
+
+
+def test_taxonomy_total_matches_refused_count(artifact):
+    refused = artifact["summary"]["refused"]
+    reason_total = sum(artifact["summary"]["refusal_reasons"].values())
+    op_total = sum(artifact["summary"]["missing_operators"].values())
+    assert reason_total == refused
+    assert op_total == refused
+
+
+# ---------------------------------------------------------------------------
+# Audit row sample integrity — ensure recognized_terms is well-formed.
+# ---------------------------------------------------------------------------
+
+
+def test_recognized_terms_only_present_on_post_pre_frame_refusals(artifact):
+    for row in artifact["per_case"]:
+        if row["outcome"] != "refused":
+            continue
+        if row["sentence_index"] == 0 and row["token_index"] == 0:
+            assert row.get("recognized_terms", []) == []
+
+
+def test_refusal_reason_distribution_is_stable(artifact):
+    """Pin the headline refusal counts so any reader runtime change must
+    explicitly update this test alongside the artifact."""
+    expected = {
+        "incomplete_operation": 18,
+        "unexpected_category": 14,
+        "unknown_word": 11,
+        "unattached_quantity": 3,
+        "unresolved_pronoun": 3,
+        "no_question_target": 1,
+    }
+    assert Counter(artifact["summary"]["refusal_reasons"]) == Counter(expected)