core/docs/handoff/COMPREHENSION-READER-AUDIT.md

Comprehension Reader Audit

Brief: POST-RAT1-PARALLEL-BRIEFS.md §"Brief C" Date: 2026-05-27 Operator: Sonnet (investigation phase) Branch: docs/comprehension-reader-audit

---

Summary finding

The comprehension reader (ADR-0164 Phase 1 + 2) is not an inert component — it is actively exercised by tests and correctly wired into parse_and_solve. However it contributes zero eval admissions today because:

1. Phase 2 (whole-problem reader) refuses on every train-sample problem due to fraction/percentage tokens, unresolved pronouns, or multi-quantity structures that are explicitly out-of-scope per Phase 2.1.
2. Phase 1 (question-sentence hybrid) admits a CandidateUnknown but the statement side never completes — the question reader's admission can only produce a result when the statement sentences also yield per_sentence_choices; for the 47 refused cases, statements already fail at the regex level, so the question reader's partial success is unreachable.

The reader is a math substrate, not a cognition track. It is not used anywhere in the cognition eval lane. It is the designed long-term replacement for the regex front-end (ADR-0164 §Decision) but is not yet at the coverage threshold where it produces observable eval lift.

---

Q1 — Call trace: where does _try_comprehension_reader actually run?

Single caller. generate/math_candidate_graph.py::parse_and_solve (line 568). No other production code calls it.

The call tree within parse_and_solve has two paths:

parse_and_solve(text, config)
│
├── config.comprehension_reader_questions == True   [flag-gated]
│   │
│   ├── ADR-0164 Phase 2 — whole-problem reader
│   │     _try_comprehension_reader(text)          [line 569]
│   │       → begin_sentence / apply_word / end_sentence / finalize
│   │       → CandidateGraphResult on success
│   │       → None on any RefusalError (falls through to regex path)
│   │
│   └── (Phase 2 fell through) continue to regex statement loop
│         → ADR-0164 Phase 1 — question-sentence hybrid
│               _try_reader_for_question(question_sentence, ...)  [line 821]
│                 → build_problem_state_from_candidates
│                 → invoke_reader_for_question
│                 → list[CandidateUnknown] | None
│               → on reader admission: use reader's CandidateUnknown
│               → on refusal: regex question parser (Pattern A/B/C)
│
└── config is None or flag False  →  regex-only path (unchanged)

The flag comprehension_reader_questions is:

• Declared in core/config.py:288 (default False)
• Set to True only in evals/gsm8k_math/train_sample/v1/runner.py:92 when --use-reader is passed.
• Not set anywhere in the live chat/runtime.py path or any cognition eval runner.

The reader is never active in production chat turns. It is activated only by an explicit CLI flag in the train-sample eval runner.

---

Q2 — Does the reader admit anything on the cognition lane?

No. comprehension_reader_questions is not set in any cognition eval runner. The reader is purely a math-domain component. Cognition evals (core eval cognition, core test --suite cognition) do not call parse_and_solve and have no concept of comprehension_reader_questions.

There is no reader-on-cognition usage to measure.

---

Q3 — Is all-or-nothing the bottleneck, or is the reader itself refusing on simple shapes?

Both, at different phases.

Phase 2 (whole-problem reader, _try_comprehension_reader)

The all-or-nothing policy is architecturally correct but the reader itself refuses early on nearly every train-sample problem. Confirmed refusal sites in lifecycle.py:

Token type	Handling	Covers cases
fraction_token / percentage_token	Explicit refusal at line 344–356: "out-of-scope (embedded-quantifier aggregate; deferred to Phase 2.1)"	0004, 0005, 0010, 0041, etc.
unknown_word	Any word absent from the lexicon (many proper nouns, verbs not yet in lexicon)	Majority of 47 refused
pronoun_resolution failure	entity_pronoun without a resolvable prior entity	0012, 0015, etc.
multi_quantity_composition	No composition frame in Phase 2 scope	0006, 0013, 0025, etc.

For the current 47 refused cases, the Phase 2 reader fails at or before the first non-trivial token and returns None, deferring to the regex path. The all-or-nothing rule is not the marginal bottleneck; lexicon coverage is. Even with per-sentence relaxation, a sentence containing an unknown verb would refuse.

Phase 1 (question-sentence hybrid, _try_reader_for_question)

Phase 1 is more targeted — it only reads the question sentence, informed by per_sentence_choices already produced by the regex parser. Its bottleneck is the upstream statement failure: if the regex parser can't build per_sentence_choices for a statement sentence, _try_reader_for_question is called with an incomplete or empty flat list, and build_problem_state_from_candidates has insufficient context to produce meaningful question-slot resolution.

The Phase 1 reader does admit case 0027 (Malcolm/followers) per test_reader_coexistence.py::test_case_0027_malcolm_admits. That's the only confirmed admission on train_sample under flag ON. The 3 currently-correct cases are unchanged between flag OFF and flag ON (coexistence test).

Conclusion: all-or-nothing is not causing the zero-lift result. The reader admits the cases it can handle; it simply cannot yet handle most train-sample problems because their lexicon coverage and structural scope (fractions, multi-quantity, complex pronouns) exceed Phase 1/2 scope.

---

Q4 — ADR-0164 Phase 1/2 promises vs current state

Promise	Status
Phase 1: question reader with regex fallthrough	✅ implemented and tested (52 tests across 3 test files)
Phase 1: wrong=0 preserved under flag ON	✅ verified by test_reader_coexistence.py::TestWrongZeroInvariant
Phase 1: admit case 0027 (Malcolm/followers)	✅ test_case_0027_malcolm_admits
Phase 2: whole-problem reader, all-or-nothing	✅ implemented, _try_comprehension_reader
Phase 2: fraction/percentage scope declared	✅ explicit refusal at lifecycle.py:344, labeled "deferred to Phase 2.1"
Phase 2: eval delta on train_sample	✗ zero new admissions under flag ON
Phase 3 (per ADR-0164 §Phasing): remove regex question parser	Not started — reader must reach sufficient question coverage first
Lexicon: seed corpus ported from regex parser	✅ math lexicon loaded at generate/comprehension/lexicon.py::load_lexicon

The only broken promise is Phase 2's intended eval delta. The ADR did not specify a minimum lift target for Phase 2 at initial ship — it specified "measure pickup rate against train_sample/v1 per round." The current pickup rate is 0 new cases on Phase 2 and 0 new cases on Phase 1 (case 0027 is already correct via regex). This is an accurate measurement, not a latent bug.

There are no ADR-0164 Phase 1/2 contract violations. The reader is operating within its declared scope. The scope is narrow.

---

Q5 — Three options and measurable tests

Option A: Operationalize (expand scope incrementally)

What: Expand Phase 2 scope to handle one new token class per iteration:

1. Common proper nouns not in the lexicon — add lexicon entries via the ratification corridor (ADR-0150/0152/0155/0161). No code change.
2. multiplicative_aggregation structures (e.g. "6 baskets × 50 strawberries") — add a distributive_modifier frame rule to Phase 2. ADR required.
3. Fraction/percentage embedded quantifiers — add Phase 2.1 handling. Separate ADR.

This is the intended path per ADR-0164 §Phasing.

Measurable test: After each lexicon expansion, run uv run python -m evals.gsm8k_math.train_sample.v1.runner --use-reader and count new Phase 2 admissions. A single lexicon batch adding the 15 most common unknown verbs should move the Phase 2 admission count from 0 to ≥ 2.

Ship as: Series of small PRs, each adding lexicon entries or a single new frame rule. Not this brief — this brief is investigation only.

Option B: Relabel (honest documentation update)

What: Add a status section to ADR-0164 acknowledging the current measurement (Phase 2: 0 new admissions on train_sample) and naming the scope gates that block lift. Rename the reader's activation flag from the generic comprehension_reader_questions to something that signals its scope: e.g., comprehension_reader_phase2_experimental.

Measurable test: No code change → no measurable test required. The honest claim after relabeling: "the reader is implemented and correct-by-construction; it produces no eval delta because its lexicon and structural scope don't yet cover any of the 47 refused cases."

Ship as: This PR (docs-only). Low risk, zero regression surface.

Option C: Retire (remove dead code)

What: Remove _try_comprehension_reader, _try_reader_for_question, and lifecycle_runtime_adapter.py; revert the comprehension_reader_questions flag; remove the 52 tests.

Why not: The reader is not dead code. It is architecturally load-bearing:

• It is the designed replacement for the regex front-end (ADR-0164 §Decision).
• It has 52 tests, 3 test files, 1,872 lines in lifecycle.py, and a working Phase 1 admission on case 0027.
• Retiring it would require reverting ADR-0164 and ADR-0165 (regex scope rule), since those two ADRs are paired: 0165 forbids cross-word regex, and 0164 provides the replacement.

Measurable test for disconfirmation: If any of the 52 reader tests currently fail or test no meaningful property (see CLAUDE.md §Schema-Defined Proof Obligations), retirement would be justified for those specific tests. Inspection shows the tests are substantive — Phase 2 tests (test_reader_phase2.py) exercise actual statement frame parsing with real admission paths.

Verdict: retire is wrong.

---

Recommendation

Ship Option B (relabel) in this PR. Add a ## Current Status section to docs/decisions/ADR-0164-incremental-comprehension-reader.md recording:

• Phase 1: implemented, tested, 0 net new admissions (case 0027 already correct via regex; wrong=0 verified)
• Phase 2: implemented, tested, 0 new admissions (fraction/multi-quantity scope gates block all 47 refused cases; explicit "Phase 2.1" label already in code)
• Next lift path: lexicon expansion via ratification corridor (no code change) followed by Phase 2.1 fraction scope (separate ADR)

Option A (operationalize) follows naturally as the next ADR wave whenever the lexicon ratification corridor is the active dispatch target. It does not require a structural change to the reader — just lexicon entries and optionally a Phase 2.1 fraction rule.

---

Falsifiability of recommendation

The claim "lexicon expansion is the highest-leverage next step" is falsifiable:

1. Add the 10 most-common unknown verbs from the 47 refused cases to the math lexicon (via apply_lexical_claim or direct ratification).
2. Run uv run python -m evals.gsm8k_math.train_sample.v1.runner --use-reader.
3. If ≥ 1 new case admits: claim confirmed; lexicon-first is the path.
4. If 0 new admissions: claim refuted; the bottleneck is structural (frame rules), not vocabulary — escalate to Phase 2.1 ADR first.

---

Files read during investigation

• generate/math_candidate_graph.py — call sites (lines 445–515, 564–571, 809–833)
• generate/comprehension/lifecycle.py — reader implementation (1,872 lines)
• generate/comprehension/lifecycle_runtime_adapter.py — Phase 1 bridge (402 lines)
• generate/comprehension/state.py — reader state types (828 lines)
• core/config.py:288 — flag declaration
• evals/gsm8k_math/train_sample/v1/runner.py — sole production flag consumer
• docs/decisions/ADR-0164-incremental-comprehension-reader.md — doctrine
• tests/test_reader_phase2.py (19 tests), test_reader_question_frame.py (20 tests), test_reader_coexistence.py (13 tests)