core/generate
Shay 9fc31eeaa4
feat(brief-11/11B): reader closure audit artifact — full taxonomy + rejected naive fix (#345)
## Summary

PR 11B in the Brief 11 sequence. Closes the missing-operator inference gap
left by 11A (#343) and ships the per-case audit artifact that Brief 11 §Gate 2
identifies as "the main Brief 11 artifact."

## Why this PR does NOT touch the reader runtime

The naive closure fix for `pre_frame_filler_sentence` (drain
`statement_terminator` at pre-frame) lifts 2 cases from refused → admitted
but creates a `wrong > 0` hazard on `gsm8k-train-sample-v1-0050`:

```
Mark does a gig every other day for 2 weeks.  For each gig, he plays 3 songs.
... How many minutes did he play?
```

With the drain enabled, the reader admits `Operation(mark, add, 3, songs)`
with unknown unit `minute` and would project to a wrong answer. The stricter
variant (`pending_entity_ref is None` + no quantities) fires on 0 of the 11
candidate cases. Per Brief 11 §"Failure modes to avoid §1 — Correct-count
greed," this PR rejects both variants and routes the closure fix to a
follow-up that adds the required verb vocabulary or sentence-intent
classifier.

## Deliverables

- `generate/comprehension/audit.py` — three new missing-operator labels:
  - `pre_frame_filler_sentence` (8 cases)
  - `descriptive_frame_question` (2 cases)
  - `question_frame_slot` (1 case)
  Closes the 11-case `None`-operator gap left by 11A.
- `evals/gsm8k_math/train_sample/v1/audit_brief_11.json` — per-case audit
  artifact pinned by tests.
- `evals/gsm8k_math/train_sample/v1/audit_brief_11.md` — narrative summary
  including the rejected-fix design tension and ranked Brief 11B-step-2
  backlog.
- `tests/test_brief_11b_audit_artifact.py` — 12 tests pinning the new labels,
  the per-case artifact, the wrong=0 invariant, and the refusal taxonomy.

## Bottleneck taxonomy (after Brief 11B labelling)

| missing_operator              | count | category               |
|-------------------------------|------:|------------------------|
| quantity_extraction           | 9     | incomplete_operation   |
| lexicon_entry                 | 9     | unknown_word           |
| multi_quantity_composition    | 8     | incomplete_operation   |
| pre_frame_filler_sentence     | 8     | unexpected_category    |
| pronoun_resolution            | 3     | unresolved_pronoun     |
| fraction_percentage_literal   | 3     | unexpected_category    |
| unit_binding                  | 3     | unattached_quantity    |
| descriptive_frame_question    | 2     | unexpected_category    |
| (others, 1 each)              | 5     | various                |

## Test plan

- 12 new tests in `tests/test_brief_11b_audit_artifact.py` pass
- 23 existing 11A tests in `tests/test_brief_11_audit.py` pass
- No runtime changes; reader byte-identical to main

## Hard invariants preserved

- `wrong == 0` — no runtime change, no new admissions
- ADR-0166 — no new canonical eval lanes added; existing
  `evals/gsm8k_math/train_sample/v1/` artifact set extended
- No teaching store / pack mutation

## Follow-up

- **11B-step-2** — verb-vocabulary expansion or sentence-intent classifier
  for `pre_frame_filler_sentence` (8 cases). See audit_brief_11.md §"design
  tension" for the rejected one-line variants and why they fail wrong=0.
- **11C** — existing-lane capability snapshot (still gated on 11B-step-2 or
  another closure pass).
2026-05-27 05:35:06 -07:00
..
binding_graph feat(binding-graph): Phase 4 question-target binding (ADR-0135) (#179) 2026-05-23 11:24:49 -07:00
comprehension feat(brief-11/11B): reader closure audit artifact — full taxonomy + rejected naive fix (#345) 2026-05-27 05:35:06 -07:00
__init__.py chore(generate): delete unreachable agenerate (#90) 2026-05-20 19:59:28 -07:00
admissibility.py feat(adr-0026): Phase 3 — ranked admissibility with margin 2026-05-17 15:03:03 -07:00
articulation.py
articulation_legality.py feat(realizer): C1.5 — articulation legality at the realizer boundary 2026-05-20 11:11:28 -07:00
attention.py
bridge_trace.py Phase 1 — bridge trace instrumentation (observation-only) 2026-05-18 18:04:57 -07:00
dialogue.py
discourse_planner.py feat(discourse): Phase 2 — reflective rendering pronominalizes focus subject 2026-05-21 10:16:12 -07:00
exhaustion.py fix(W-011/W-012): propagate recognition refusal + catch InnerLoopExhaustion (#258) 2026-05-24 20:46:46 -07:00
graph_constraint.py fix(adr-0046): make forward-graph-constraint branch mergeable 2026-05-18 05:57:46 -07:00
graph_planner.py perf(graph): PropositionGraph.topo_order — Kahn's O(N+E) instead of O(N×E) (#92) 2026-05-20 20:37:21 -07:00
grounding_accessors.py feat(grounding): structured GroundedFact accessors for discourse planner 2026-05-19 11:19:59 -07:00
intent.py fix(intent): route 'Actually X R Y' premises to CORRECTION (inference_closure) (#117) 2026-05-22 12:33:56 -07:00
intent_bridge.py Phase 2 — proposition-slot grounding for articulate_with_intent 2026-05-18 18:18:31 -07:00
intent_ratifier.py feat(recognition): ADR-0144 — EpistemicGraph carrier + pipeline integration (#227) 2026-05-24 13:39:01 -07:00
math_candidate_graph.py feat(comprehension/10): Phase 2 statement-frame reader (ADR-0164.4) (#335) 2026-05-27 05:03:56 -07:00
math_candidate_parser.py feat(ADR-0163.D.4): question grammar extension — mass nouns, comparatives, pronoun-entity resolution (#310) 2026-05-26 16:19:37 -07:00
math_parser.py fix(quarantine): drain all 60 quarantined tests — QUARANTINE=∅ (#267) 2026-05-25 11:22:12 -07:00
math_problem_graph.py feat(parser): ADR-0123 comparison-phrasing substrate (substrate-only; lift deferred) 2026-05-23 01:56:28 -07:00
math_realizer.py feat(realizer): ADR-0123 comparison-phrasing surface (closes substrate) 2026-05-23 02:03:49 -07:00
math_roundtrip.py feat(ADR-0131.G.3.1): numerics extensions — fractions + multi-currency + multi-token cardinals + word-num-adjective 2026-05-23 15:16:46 -07:00
math_solver.py feat(parser): ADR-0123 comparison-phrasing substrate (substrate-only; lift deferred) 2026-05-23 01:56:28 -07:00
math_symbolic_equivalence.py feat(ADR-0131.1.B): harden symbolic equivalence lane with generated corpus + exact algebra (#169) 2026-05-23 10:47:57 -07:00
math_symbolic_normalizer.py feat(ADR-0131.1.B): harden symbolic equivalence lane with generated corpus + exact algebra (#169) 2026-05-23 10:47:57 -07:00
math_verifier.py feat(parser): ADR-0123 comparison-phrasing substrate (substrate-only; lift deferred) 2026-05-23 01:56:28 -07:00
math_versor_arithmetic.py feat(ADR-0141): multiply as CGA dilator versor (positive non-zero) (#216) 2026-05-24 09:09:53 -07:00
morphology.py fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
ood_surface_generator.py fix(quarantine): drain all 60 quarantined tests — QUARANTINE=∅ (#267) 2026-05-25 11:22:12 -07:00
operators.py feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
perturbation_suite.py feat: add ADR-0125 perturbation suite 2026-05-22 17:12:33 -07:00
proposition.py feat(adr-0022): Forward Semantic Control — Accepted 2026-05-17 12:10:20 -07:00
realizer.py [codex] Implement energy-modulated vault surface (#269) 2026-05-25 11:33:32 -07:00
realizer_guard.py feat(coherence): ADR-0075 — realizer slot-type guard (C1) 2026-05-19 22:35:09 -07:00
recognizer_anchor_inject.py feat(ADR-0163.D.2): parsed_anchors → MathProblemGraph state — discrete_count_statement injection v1 (#315) 2026-05-26 18:32:05 -07:00
recognizer_match.py feat(ADR-0163.D.2): parsed_anchors → MathProblemGraph state — discrete_count_statement injection v1 (#315) 2026-05-26 18:32:05 -07:00
recognizer_registry.py feat(ADR-0163.D): wire ratified RecognizerSpecs into math_candidate_graph admissibility surface (#302) 2026-05-26 13:11:47 -07:00
result.py feat(adr-0023): Forward Semantic Control proof evidence — Accepted 2026-05-17 12:55:19 -07:00
rotor_admissibility.py feat(adr-0025): Phase 4 — rotor / frame admissibility at the seam 2026-05-17 15:16:32 -07:00
salience.py
semantic_templates.py feat: vault recall index, Rust versor parity, cognitive pack expansion 2026-05-15 15:34:39 -07:00
stream.py chore(generate): make stop-tokens caller-overridable via RuntimeConfig (#87) 2026-05-20 19:59:33 -07:00
surface.py feat(surface): ADR-0031 — score-decomposition surface (per-axis hedges) 2026-05-17 20:16:22 -07:00
templates.py feat(realizer): C1.5 — articulation legality at the realizer boundary 2026-05-20 11:11:28 -07:00