core/docs/analysis/question-layer-gap-survey.md

<!-- FILE: docs/analysis/question-layer-gap-survey.md -->

# Question-Layer Gap Survey

Status: Proposed analysis draft. No serving behavior is changed.

Source of truth for the metric is `docs/claims_ledger.md` and
`evals/gsm8k_math/train_sample/v1/report.json`: the current real
`train_sample` result is 6 correct / 44 refused / 0 wrong. This survey assigns
each of the 44 refused case ids to exactly one current failure group. It does
not claim that any group would pass if widened; it names the layer that refuses
today and estimates whether the missing capability is single-step or
composition-bound.

The requested Task A file, `docs/analysis/comprehension-primitive-inventory.md`,
was not present in this worktree. I used the embedded older handoff only as
orientation and grounded the grouping below in the report plus the live parser /
recognizer code.

## Code Map

Current refusal topology:

- `generate/math_candidate_graph.py` first extracts statement choices, then
  consults the ratified recognizer registry when a numeric statement has no
  parser candidate. If recognition succeeds but injection yields no typed
  solver primitive, it now refuses explicitly instead of dropping the statement.
- `generate/recognizer_anchor_inject.py` is still a category dispatch surface.
  Only `discrete_count_statement` and a narrow `multiplicative_aggregation`
  entry can emit today; other categories return empty and therefore become
  explicit refusals.
- `generate/math_candidate_parser.py` admits a closed question grammar:
  total-across `How many ... do they have ...`, existential aggregate
  `How many ... are there ...`, entity possession, activity `did`, and three
  ADR-0163.D.4 patterns. If these emit no `CandidateUnknown`, the graph refuses
  before branch enumeration.
- ADR-0174 explicitly deprecates the per-category injector dispatch table as
  the long-term runtime admission path. Injector widening should therefore be
  treated as stopgap or hypothesis-emitter work, not the primary strategic
  direction.

## Assignment Table

Tractability scoring is case-level and arity-aware:

- High: one closed primitive or one local parser frame is plausibly missing.
- Medium: one local frame/schema is clear, but downstream composition is still
  likely needed.
- Low: the current refusal is an early stop in a 2-4 capability derivation.

| Group | Count | Case ids | Current failing layer | Tractability | Representative report reason excerpts | Interpretation |
|---|---:|---|---|---|---|---|
| DCS high-arity composition wall | 18 | 0002, 0015, 0016, 0020, 0029, 0031, 0032, 0033, 0034, 0036, 0037, 0038, 0039, 0040, 0041, 0044, 0047, 0049 | Matcher -> injector -> composition | Low | `candidate_graph: recognizer matched but produced no injection for statement: 'She splits it up into 25-foot sections.' (category=discrete_count_statement)`; `candidate_graph: recognizer matched but produced no injection for statement: 'Malcolm is trying to find the fastest walk to school and is currently comparing two routes.' (category=discrete_count_statement)` | The recognizer often fires on a count-like token, but the actual derivation needs division, rest-state, route comparison, percent/rate, chained comparisons, target residuals, or per-entity attributes. Widening the discrete-count injector alone is metric-inert and risks incomplete readings. |
| Missing inverse/residual/comparative question frames | 5 | 0007, 0008, 0009, 0025, 0035 | Question parser / admissibility | Low | `candidate_graph: no admissible candidate for question: 'How many more boxes do they need if Francine has a total of 85 crayons?'`; `candidate_graph: no admissible candidate for question: 'How many more apples would Martha need to give away to be left with only 4 of them?'` | These are not just new surface phrasings for `Unknown(entity, unit)`. They ask for a missing operand, inverse relation, target residual, or conditional total. The question layer must bind the requested slot to a derivation, not merely extract a unit. |
| Rate, currency-rate, and tariff statements | 4 | 0001, 0011, 0017, 0022 | Recognizer -> injector/schema | Medium | `candidate_graph: recognizer matched but produced no injection for statement: 'Tina makes $18.00 an hour.' (category=rate_with_currency)`; `candidate_graph: recognizer matched but produced no injection for statement: 'He’s charging $50.00 per day or $500.00 for 14 days.' (category=temporal_aggregation)` | The local schema gap is visible: `rate_with_currency` and `temporal_aggregation` need typed rate/tariff hypotheses. Case-level admission still needs overtime, profit, historical+today aggregation, or piecewise tariff composition. |
| Non-quantitative relation category used as an early stop | 4 | 0012, 0023, 0027, 0046 | Matcher -> injector | Low | `candidate_graph: recognizer matched but produced no injection for statement: 'He put all of them in his aquarium but his fish ate half of them.' (category=descriptive_setup_no_quantity)`; `candidate_graph: recognizer matched but produced no injection for statement: 'Half of the students are girls, the other half are boys.' (category=descriptive_setup_no_quantity)` | The category name is accurate for the injector: it cannot emit a concrete primitive from the matched surface. The cases need fraction-of-prior, combined-total binding, partition, and percentage-of-subgroup reasoning. |
| Multiplicative aggregate beyond the narrow emitted shapes | 3 | 0006, 0013, 0045 | Recognizer -> injector / composition registry | Medium | `candidate_graph: recognizer matched but produced no injection for statement: 'Mandy started reading books with only 8 pages when she was 6 years old.' (category=multiplicative_aggregation)`; `candidate_graph: recognizer matched but produced no injection for statement: 'Each survey has 10 questions.' (category=multiplicative_aggregation)` | There is already a narrow product injector, but these rows need time/age chains, month segmentation, doubled rates, or survey-count composition. Single product emission may help local state, but full cases still depend on multi-step composition. |
| Financial currency amount / percent mutation statements | 3 | 0019, 0028, 0043 | Recognizer -> injector / mutation schema | Low | `candidate_graph: recognizer matched but produced no injection for statement: 'After the first appointment, John paid $100 for pet insurance that covers 80% of the subsequent visits.' (category=currency_amount)`; `candidate_graph: recognizer matched but produced no injection for statement: 'Her mother gave her an additional $4, and her father twice as much as her mother.' (category=currency_amount)` | The surface contains currency, but the required reading is coverage after first event, percent daily operating cost, revenue target, or comparative gift amount. A `CandidateInitial` currency emission would be an incomplete graph. |
| Fractional relational statements with no parser candidate | 3 | 0004, 0005, 0010 | Statement parser / admissibility | Low | `candidate_graph: no admissible candidate for statement: 'Half of the kids are going to soccer camp, and 1/4 of the kids going to soccer camp are going to soccer camp in the morning.'`; `candidate_graph: no admissible candidate for statement: 'Marion has 1/4 more than what Yun currently has, plus 7.'` | The parser has fraction literals and comparative operations, but these surfaces are relational fractions over prior or unknown quantities. They require held equations or derivation nodes, not a flat possession/operation candidate. |
| Duration and recurrence statement frames | 3 | 0030, 0048, 0050 | Statement parser / temporal composition | Medium | `candidate_graph: no admissible candidate for statement: 'It is a 2-hour drive each way.'`; `candidate_graph: no admissible candidate for statement: 'Mark does a gig every other day for 2 weeks.'` | These expose bounded duration-multiplier or temporal-frequency frames. Each frame is local and deterministic, but the cases still need composition across trip stages, weekly deltas to target, or per-event song duration totals. |
| Relational conjoined-subject each initial | 1 | 0026 | Statement parser / entity binding | Medium | `candidate_graph: no admissible candidate for statement: 'Aaron and his brother Carson each saved up $40 to go to dinner.'` | The parser has an `each` extractor for two named subjects, but this sentence uses a possessive relational subject (`his brother Carson`) and a purpose tail. The local parse is probably narrower than the concept; the full case remains multi-step because bill fraction and shared scoop count follow. |

Total assigned: 18 + 5 + 4 + 4 + 3 + 3 + 3 + 3 + 1 = 44.

## Backlog Interpretation

The audited partition is the stable artifact. A single `count x tractability`
sort is misleading here because count ranges from 1 to 18 while tractability is
coarse; it would place the known composition wall at the top as if it were an
incremental injector work order. In this survey, count is impact evidence, not
the sort key for near-term changes.

### Composition-Bound Work

These groups should feed the ADR-0174 held-hypothesis / derivation-composer
scope, not category-specific injector widening. They need multi-clause state,
referent binding, ratio/fraction relations, target-slot questions, or
event-scope composition before any answer can be safe.

| Group | Count | How to use the count |
|---|---:|---|
| DCS high-arity composition wall | 18 | Main evidence that discrete-count recognition is surfacing a composition wall, not an injector backlog. |
| Missing inverse/residual/comparative question frames | 5 | Question-layer evidence that unknowns must bind to derivation slots, not just noun units. |
| Rate, currency-rate, and tariff statements | 4 | Rate/tariff hypotheses are useful only if downstream overtime, profit, history, or piecewise composition can refuse partial readings. |
| Non-quantitative relation category used as an early stop | 4 | Evidence for relation/partition composition and for avoiding hard-stop loss of later structure. |
| Multiplicative aggregate beyond the narrow emitted shapes | 3 | Product-like anchors need day/month, age-chain, or survey-count composition before admission. |
| Financial currency amount / percent mutation statements | 3 | Currency is not enough; these need percent coverage, operating-cost, or spend/residual mutation. |
| Fractional relational statements with no parser candidate | 3 | Needs relational fraction/equation hypotheses, not flat possession candidates. |
| Duration and recurrence statement frames | 3 | Duration-multiplier and recurrence frames are local signals whose answers require temporal composition. |

### Bounded Near-Term Fixes

These are smaller, lower-blast-radius probes that may be useful as executable
follow-ups or regression tests. They should still preserve refusal-first
admission and should not be represented as the main path to the metric.

| Candidate fix | Case ids | Why bounded |
|---|---|---|
| Relational conjoined-subject `each` binding | 0026 | One row exposes a local entity-binding gap around `Aaron and his brother Carson each ...`; useful as a narrow parser/binding probe even though the full problem remains multi-step. |
| Descriptive/no-quantity early-stop handling | 0012, 0023, 0027, 0046 | The bounded mechanism is to avoid losing later structure when a descriptive relation cannot emit state; the cases behind it still require composition, so this is not a direct answer path. |
| Single-slot question-frame probes | subset of 0007, 0008, 0035 | A few question surfaces can become narrow probes for residual or divisor target-slot binding. They must be held as unknown slots and refused unless the derivation is complete. |

## Layer Notes

The high-count `discrete_count_statement` bucket should not be read as "add more
discrete-count injectors." The report is saying the matcher saw something
count-shaped before the engine had a safe composed reading. In the live code,
recognized-but-uninjected statements refuse explicitly because dropping them
would permit incomplete graphs. ADR-0174 points in the same direction:
injectors should become hypothesis emitters inside a held-hypothesis reader,
where branch disagreement, constraint propagation, and completeness can reject
partial readings.

The five question refusals are genuinely question-layer refusals in the narrow
code sense: `extract_question_candidates` emits no `CandidateUnknown`. But all
five ask for an inverse or residual target. A wider regex that only extracts
the noun would not identify the unknown slot and would still be unsafe under
`wrong = 0`.

## Open Questions for the Claude Lane

- Confirm whether `docs/analysis/comprehension-primitive-inventory.md` was meant
  to be landed in this worktree or only supplied out-of-band; this draft could
  be amended to cite its exact primitive table once present.
- Run an instrumented read of the 44 refused cases to distinguish "injector
  returned empty because no parsed anchors existed" from "anchors existed but
  constraint propagation eliminated them."
- For the five question refusals, collect the intended `Unknown` slot shape
  without using gold answers: missing operand, residual-to-target, inverse
  divisor, or aggregate total.
- Decide whether the next executable lane should prototype rate/tariff
  hypotheses or question-target slots first; both are safer as held hypotheses
  than as direct category admissions.