52f2bf6f4c
* docs(ADR-0136.S.0): refusal taxonomy + S.1 brief for rate/event statement corridor
Taxonomy: deterministic classification of all 50 GSM8K train-sample refused cases
into primary + secondary barriers. Key findings:
context_filler (primary): 23/50 — legitimately refuses; not parser gaps
compound_statement: 5/50 — two ops in one sentence
rate/capacity class: 4/50 — direct S.1 targets
distributive_multiply: 1/50 primary, 5/50 secondary
long-tail (diverse): 17/50
Honest S.1 ceiling: 0/50 → ≤4/50 admission. gsm8k-0014 ('Bob can shuck 10
oysters in 5 minutes') is the only case with capacity_rate as sole barrier.
Ships:
- evals/gsm8k_math/train_sample/v1/refusal_taxonomy.json (schema v1, 50 records)
- docs/briefs/parallel-2026-05-23/L17-ADR-0136-S1-rate-event-statements.md
- full briefs archive (parallel-2026-05-23)
No implementation changes. Taxonomy and brief only.
* feat(ADR-0136.S.1): rate/event statement parsing — capacity + earnings shapes, axis lane 20/20, wrong==0, gsm8k-0014 admits
Two closed statement shapes added to candidate parser and graph:
Shape A (capacity-rate): "<Actor> can <verb> N <unit> in M <time-unit>"
- 13 closed verbs (shuck/pick/pack/make/produce/type/read/write/paint/run/score/answer/complete)
- Pronoun question form (he/she/they/it) accepted
- Time-unit conversion (second/minute/hour/day)
Shape B (earnings-rate): "<Actor> <verb> $N per/an/a <time-unit>"
- 5 closed verbs (make/earn/receive/get/charge)
- Currency: $ only, 0-2 decimal places
- Per-token alternation: per/a/an/for each/every
Short-circuit paths in parse_and_solve run before the Cartesian product,
computing rate_per_sec × T_seconds directly. Actor mismatch → refusal
(not wrong). Answer ≤ 0 → fall through to refusal.
GSM8K honest delta: 0/50 → 1/50 (gsm8k-0014: answer=240.0, correct).
23 context-filler cases correctly remain refused.
Axis lane: 20/20 pass, wrong=0.
B3 bounded-grammar lane: unchanged (wrong=0).
35 new tests including B3 regression guard and GSM8K admitted_wrong=0 rail.
77 lines
3 KiB
Markdown
77 lines
3 KiB
Markdown
# ADR-0136.S.1 — Rate/Event Statement Parsing
|
||
|
||
**Status:** Accepted
|
||
**Parent:** ADR-0136 (Statement Layer Corridor)
|
||
**Date:** 2026-05-23
|
||
|
||
## Context
|
||
|
||
The GSM8K refusal taxonomy (`evals/gsm8k_math/train_sample/v1/refusal_taxonomy.json`)
|
||
reveals that 23/50 cases are blocked by context-filler sentences (correctly
|
||
refused — no parseable numeric state), while 4/50 have rate/capacity/price as
|
||
their primary barrier. The remaining cases are compound-statement,
|
||
distributive-multiply, and diverse long-tail shapes.
|
||
|
||
This ADR targets the 4 rate-class cases with two closed statement shapes.
|
||
|
||
## Taxonomy Finding
|
||
|
||
| Primary barrier | Cases | S.1 scope? |
|
||
|------------------------|-------|------------|
|
||
| `context_filler` | 23 | No — correctly refused |
|
||
| rate/capacity/price | 4 | **Yes** |
|
||
| `compound_statement` | 5 | No |
|
||
| `distributive_multiply`| 1 (+5 secondary) | No |
|
||
| diverse long-tail | 17 | No |
|
||
|
||
## Closed Verb Sets
|
||
|
||
**Capacity verbs:** shuck, pick, pack, make, produce, type, read, write,
|
||
paint, run, score, answer, complete (+ third-person -s forms).
|
||
|
||
**Earnings verbs:** make, earn, receive, get, charge (+ third-person -s forms).
|
||
|
||
No regex wildcards for verbs — every admitted verb is explicitly listed in a
|
||
frozen set. Sentences with verbs outside the closed set are refused (not
|
||
wrong).
|
||
|
||
## Short-Circuit Rationale
|
||
|
||
Both rate shapes bypass the Cartesian-product candidate graph because the
|
||
rate computation is a direct `rate × time` multiplication with unit conversion,
|
||
not a graph of initial-possessions and operations. The short-circuit runs
|
||
before `_filtered_statement_choices` so that rate-shaped sentences don't
|
||
trigger the "no admissible candidate" refusal.
|
||
|
||
Actor matching is required: capacity questions with pronouns (`he`/`she`)
|
||
accept any actor; named-actor questions require case-insensitive match.
|
||
Mismatched actors produce refusal, not wrong answers.
|
||
|
||
## Honest GSM8K Claim
|
||
|
||
- **Pre-S.1:** 0/50 admitted (all refused).
|
||
- **Post-S.1:** 1/50 admitted — `gsm8k-0014` (Bob shucks oysters) with
|
||
answer 240.0 (correct).
|
||
- **admitted_wrong = 0** (safety rail preserved).
|
||
|
||
The other 3 rate-class cases remain blocked by context-filler sentences in
|
||
their opening statements; the rate parsing behind them is irrelevant until
|
||
those sentences parse.
|
||
|
||
## Deferred
|
||
|
||
- Context-filler gated problems (23 cases — needs semantic classification
|
||
of narrative scene-setter sentences).
|
||
- Conditional branching (overtime rules, e.g. "if she works more than 8 hours").
|
||
- Percentage/interest rates (10% simple interest).
|
||
- Multi-statement earnings (duration asserted in a separate sentence from the
|
||
rate — needs general duration-statement parser).
|
||
|
||
## Evidence
|
||
|
||
- Axis lane: `evals/math_capability_axes/S1_rate_events/v1/` — 20/20 pass,
|
||
wrong=0.
|
||
- B3 bounded-grammar lane: unchanged (wrong=0).
|
||
- GSM8K candidate-graph probe: wrong=0, admitted=1/50.
|
||
- Tests: `tests/test_adr_0136_S1_rate_events.py` — ≥15 tests including B3
|
||
regression guard and GSM8K admitted_wrong=0 rail.
|