Commit graph

3 commits

Author SHA1 Message Date
Shay
52f2bf6f4c
feat(ADR-0136.S.1): rate/event statement parsing — capacity + earnings shapes, axis lane 20/20, wrong==0, gsm8k-0014 admits (#201)
* docs(ADR-0136.S.0): refusal taxonomy + S.1 brief for rate/event statement corridor

Taxonomy: deterministic classification of all 50 GSM8K train-sample refused cases
into primary + secondary barriers. Key findings:

  context_filler (primary): 23/50 — legitimately refuses; not parser gaps
  compound_statement:         5/50 — two ops in one sentence
  rate/capacity class:        4/50 — direct S.1 targets
  distributive_multiply:      1/50 primary, 5/50 secondary
  long-tail (diverse):       17/50

Honest S.1 ceiling: 0/50 → ≤4/50 admission. gsm8k-0014 ('Bob can shuck 10
oysters in 5 minutes') is the only case with capacity_rate as sole barrier.

Ships:
- evals/gsm8k_math/train_sample/v1/refusal_taxonomy.json (schema v1, 50 records)
- docs/briefs/parallel-2026-05-23/L17-ADR-0136-S1-rate-event-statements.md
- full briefs archive (parallel-2026-05-23)

No implementation changes. Taxonomy and brief only.

* feat(ADR-0136.S.1): rate/event statement parsing — capacity + earnings shapes, axis lane 20/20, wrong==0, gsm8k-0014 admits

Two closed statement shapes added to candidate parser and graph:

Shape A (capacity-rate): "<Actor> can <verb> N <unit> in M <time-unit>"
  - 13 closed verbs (shuck/pick/pack/make/produce/type/read/write/paint/run/score/answer/complete)
  - Pronoun question form (he/she/they/it) accepted
  - Time-unit conversion (second/minute/hour/day)

Shape B (earnings-rate): "<Actor> <verb> $N per/an/a <time-unit>"
  - 5 closed verbs (make/earn/receive/get/charge)
  - Currency: $ only, 0-2 decimal places
  - Per-token alternation: per/a/an/for each/every

Short-circuit paths in parse_and_solve run before the Cartesian product,
computing rate_per_sec × T_seconds directly. Actor mismatch → refusal
(not wrong). Answer ≤ 0 → fall through to refusal.

GSM8K honest delta: 0/50 → 1/50 (gsm8k-0014: answer=240.0, correct).
23 context-filler cases correctly remain refused.
Axis lane: 20/20 pass, wrong=0.
B3 bounded-grammar lane: unchanged (wrong=0).
35 new tests including B3 regression guard and GSM8K admitted_wrong=0 rail.
2026-05-23 20:36:01 -07:00
Shay
3011fce268 feat(ADR-0131.G.3): numeric literals — money + hyphenated cardinals (axis lane 20/20, wrong==0)
First capability-axis iteration after ADR-0131.G baseline. Extends the
candidate-graph parser's <value> slot to recognize:

  - Money symbol literals: $N and $N.NN (1-2 decimals); $N.NNN refused
  - Money word forms: N dollars / N cents
  - Hyphenated multi-word cardinals: twenty-five, ninety-nine, ...

All money values normalize to integer cents, unit 'cents' — pack-aligned
with en_units_v1's canonical_unit='cent' for the money dimension.
en_numerics_v1's parse_compound_cardinal handles hyphenated cardinals.

Parser changes (generate/):
  - math_candidate_parser.py: _VALUE alternation widened; _resolve_value
    refactored to return _ResolvedValue|None carrying optional unit
    override; _INITIAL_HAS_RE unit slot made optional; dollar/dollars →
    cents normalization at candidate build.
  - math_roundtrip.py: new _unit_grounds helper (money-aware); _value_grounds
    widened for the three new literal shapes; roundtrip_admissible uses
    _unit_grounds for the unit check.
  - math_candidate_graph.py: _initial_admissible and _question_admissible
    use _unit_grounds.

New axis lane (evals/math_capability_axes/G3_numerics/v1/):
  - 26 curated cases (20 positive across 4 classes + 6 refusal probes)
  - runner.py wraps _score_one_candidate_graph; byte-equal report.json
  - 20/20 positive solved correct; 6/6 refusal probes refused typed;
    solved_wrong == 0; overall_pass == True

Tests: 27/27 in 0.19s. 420 existing candidate-parser/math-parser/pack
tests still green. GSM8K probe safety rail (admitted_wrong == 0)
preserved.

Honest scope-limit (documented in ADR): admission_rate on the GSM8K
probe stays at 0/50 because (a) the probe currently consults the legacy
parser path, not the candidate-graph pipeline G.3 extends, and (b) most
money-bearing GSM8K cases fail first on verb (G.1) or multi-clause (G.4)
shape, not on the money literal. The axis lane is the load-bearing
measurement for this iteration. Reserved follow-up: a small probe-
infra ADR to switch run_coverage_probe.py to the candidate-graph
pipeline.

Out of scope, deferred to G.3.1: fractions end-to-end (resolver supports
N/M but no axis cases), multi-currency (¢ € £ ¥ ₱), space-separated
multi-word cardinals (one hundred), word-number-adjective compositions
(five full boxes).
2026-05-23 14:23:05 -07:00
Shay
feeb64818c feat(ADR-0126 P3+P4): graph assembly + decision rule + runner wiring
P3 — generate/math_candidate_graph.py:
  Branch enumeration over per-sentence candidate choices (Cartesian
  product, cap=64). Per-sentence ambiguity tiebreaker via most-grounded-
  slots-wins (transfer beats subtract when 'to Tom' grounds). Decision
  rule: 0 admissible -> refuse; 1 -> emit; >=2 same answer -> emit;
  >=2 different answers -> refuse (preserves wrong==0 on genuine
  ambiguity). End-to-end parse_and_solve(text) -> CandidateGraphResult.

  Question extractor added to math_candidate_parser.py (CandidateUnknown,
  total + entity question shapes mirroring math_parser).

  22 new tests. Permissive verbs ('bought', 'ate', 'bakes') now produce
  correct answers via the candidate-graph path; ambiguous 'gives to Tom'
  resolves to transfer reading (Tom gets the apples) deterministically.

P4 — evals/gsm8k_math/runner.py:
  New sibling function _score_one_candidate_graph(case) -> CaseOutcome.
  Identical shape to _score_one; swaps parse_problem for parse_and_solve;
  preserves verifier/realizer/expected-answer stages. Callers (e.g.
  PR #160's train_sample/v1/runner.py) substitute the new function in
  one line to evaluate the candidate-graph topology.

  9 new wiring tests. Three groups:
    - No regression: cases legacy solves, new also solves.
    - Lift: cases legacy refuses, new solves (the architectural payoff).
    - Wrong==0: out-of-grammar refuses, never wrong.

Regression: 714/714 existing math + runner tests still green.
ADR-0126 total: 74/74 tests green across P1+P2+P3+P4.
2026-05-23 06:36:13 -07:00