Commit graph

5 commits

Author SHA1 Message Date
Shay
e7a1ffb72e
feat(ADR-0136.S.2): conditional-op question — gsm8k-0042 admits, wrong==0 (#203)
Adds CandidateConditionalOpQuestion + extractor for the closed shape:
  "If <Entity> <verb> <N> <unit>, how many <unit2> does <Entity2> <aux> [<qualifier>]?"

In parse_and_solve, when the question yields exactly one such candidate
and exactly one matching InitialPossession exists by (entity, unit) across
all statement sentences, computes initial_value ± operand (verb polarity)
and emits when answer >= 0; refuses otherwise. Structurally identical to
S.1 capacity/earnings short-circuits.

GSM8K probe: 2/50 → 3/50 (+0042, answer=30.0), wrong stays 0.

- generate/math_candidate_parser.py: _COND_SUBTRACT_VERBS / _COND_ADD_VERBS
  closed sets; _COND_OP_Q_RE; extract_conditional_op_question_candidates
- generate/math_candidate_graph.py: short-circuit after earnings path
- tests/test_adr_0136_S2_conditional_op.py: 25 tests (extractor unit tests,
  end-to-end short-circuit, B3 + S.1 regression guards, post-S.2 honest
  admission count)
- docs/decisions/ADR-0136.S.2-conditional-op-question.md
2026-05-23 21:20:52 -07:00
Shay
19ac7f94b9
feat(ADR-0136.S.0): context-sentence classifier — skip no-digit sentences, gsm8k-0018 admits (#202)
- Add classify_sentence() + has_numeric_token() to math_candidate_parser.py.
  Rule: sentence with no digit and no word-number cannot introduce parseable
  numeric state — classify as "context" and skip safely (wrong==0 preserved).

- Add pre-pass in parse_and_solve() (math_candidate_graph.py): strips context
  sentences before extraction; falls through to refusal if none remain numeric.

- Extend capacity patterns for gsm8k-0018:
  - _CAPACITY_INVERTED_RE: "During M <time-unit> <Actor> can <verb> N <unit>"
  - _CAPACITY_Q2_RE: "How many <unit> [on average] is <Actor> able to <verb>,
    when the <event> lasted for T <time-unit>?"

- GSM8K: 1/50 -> 2/50 (gsm8k-0018 admits with answer 16.0); admitted_wrong==0.
- Tests: 47/47 pass (12 new for classifier, inverted patterns, 0018 end-to-end).
2026-05-23 20:51:47 -07:00
Shay
52f2bf6f4c
feat(ADR-0136.S.1): rate/event statement parsing — capacity + earnings shapes, axis lane 20/20, wrong==0, gsm8k-0014 admits (#201)
* docs(ADR-0136.S.0): refusal taxonomy + S.1 brief for rate/event statement corridor

Taxonomy: deterministic classification of all 50 GSM8K train-sample refused cases
into primary + secondary barriers. Key findings:

  context_filler (primary): 23/50 — legitimately refuses; not parser gaps
  compound_statement:         5/50 — two ops in one sentence
  rate/capacity class:        4/50 — direct S.1 targets
  distributive_multiply:      1/50 primary, 5/50 secondary
  long-tail (diverse):       17/50

Honest S.1 ceiling: 0/50 → ≤4/50 admission. gsm8k-0014 ('Bob can shuck 10
oysters in 5 minutes') is the only case with capacity_rate as sole barrier.

Ships:
- evals/gsm8k_math/train_sample/v1/refusal_taxonomy.json (schema v1, 50 records)
- docs/briefs/parallel-2026-05-23/L17-ADR-0136-S1-rate-event-statements.md
- full briefs archive (parallel-2026-05-23)

No implementation changes. Taxonomy and brief only.

* feat(ADR-0136.S.1): rate/event statement parsing — capacity + earnings shapes, axis lane 20/20, wrong==0, gsm8k-0014 admits

Two closed statement shapes added to candidate parser and graph:

Shape A (capacity-rate): "<Actor> can <verb> N <unit> in M <time-unit>"
  - 13 closed verbs (shuck/pick/pack/make/produce/type/read/write/paint/run/score/answer/complete)
  - Pronoun question form (he/she/they/it) accepted
  - Time-unit conversion (second/minute/hour/day)

Shape B (earnings-rate): "<Actor> <verb> $N per/an/a <time-unit>"
  - 5 closed verbs (make/earn/receive/get/charge)
  - Currency: $ only, 0-2 decimal places
  - Per-token alternation: per/a/an/for each/every

Short-circuit paths in parse_and_solve run before the Cartesian product,
computing rate_per_sec × T_seconds directly. Actor mismatch → refusal
(not wrong). Answer ≤ 0 → fall through to refusal.

GSM8K honest delta: 0/50 → 1/50 (gsm8k-0014: answer=240.0, correct).
23 context-filler cases correctly remain refused.
Axis lane: 20/20 pass, wrong=0.
B3 bounded-grammar lane: unchanged (wrong=0).
35 new tests including B3 regression guard and GSM8K admitted_wrong=0 rail.
2026-05-23 20:36:01 -07:00
Shay
3011fce268 feat(ADR-0131.G.3): numeric literals — money + hyphenated cardinals (axis lane 20/20, wrong==0)
First capability-axis iteration after ADR-0131.G baseline. Extends the
candidate-graph parser's <value> slot to recognize:

  - Money symbol literals: $N and $N.NN (1-2 decimals); $N.NNN refused
  - Money word forms: N dollars / N cents
  - Hyphenated multi-word cardinals: twenty-five, ninety-nine, ...

All money values normalize to integer cents, unit 'cents' — pack-aligned
with en_units_v1's canonical_unit='cent' for the money dimension.
en_numerics_v1's parse_compound_cardinal handles hyphenated cardinals.

Parser changes (generate/):
  - math_candidate_parser.py: _VALUE alternation widened; _resolve_value
    refactored to return _ResolvedValue|None carrying optional unit
    override; _INITIAL_HAS_RE unit slot made optional; dollar/dollars →
    cents normalization at candidate build.
  - math_roundtrip.py: new _unit_grounds helper (money-aware); _value_grounds
    widened for the three new literal shapes; roundtrip_admissible uses
    _unit_grounds for the unit check.
  - math_candidate_graph.py: _initial_admissible and _question_admissible
    use _unit_grounds.

New axis lane (evals/math_capability_axes/G3_numerics/v1/):
  - 26 curated cases (20 positive across 4 classes + 6 refusal probes)
  - runner.py wraps _score_one_candidate_graph; byte-equal report.json
  - 20/20 positive solved correct; 6/6 refusal probes refused typed;
    solved_wrong == 0; overall_pass == True

Tests: 27/27 in 0.19s. 420 existing candidate-parser/math-parser/pack
tests still green. GSM8K probe safety rail (admitted_wrong == 0)
preserved.

Honest scope-limit (documented in ADR): admission_rate on the GSM8K
probe stays at 0/50 because (a) the probe currently consults the legacy
parser path, not the candidate-graph pipeline G.3 extends, and (b) most
money-bearing GSM8K cases fail first on verb (G.1) or multi-clause (G.4)
shape, not on the money literal. The axis lane is the load-bearing
measurement for this iteration. Reserved follow-up: a small probe-
infra ADR to switch run_coverage_probe.py to the candidate-graph
pipeline.

Out of scope, deferred to G.3.1: fractions end-to-end (resolver supports
N/M but no axis cases), multi-currency (¢ € £ ¥ ₱), space-separated
multi-word cardinals (one hundred), word-number-adjective compositions
(five full boxes).
2026-05-23 14:23:05 -07:00
Shay
feeb64818c feat(ADR-0126 P3+P4): graph assembly + decision rule + runner wiring
P3 — generate/math_candidate_graph.py:
  Branch enumeration over per-sentence candidate choices (Cartesian
  product, cap=64). Per-sentence ambiguity tiebreaker via most-grounded-
  slots-wins (transfer beats subtract when 'to Tom' grounds). Decision
  rule: 0 admissible -> refuse; 1 -> emit; >=2 same answer -> emit;
  >=2 different answers -> refuse (preserves wrong==0 on genuine
  ambiguity). End-to-end parse_and_solve(text) -> CandidateGraphResult.

  Question extractor added to math_candidate_parser.py (CandidateUnknown,
  total + entity question shapes mirroring math_parser).

  22 new tests. Permissive verbs ('bought', 'ate', 'bakes') now produce
  correct answers via the candidate-graph path; ambiguous 'gives to Tom'
  resolves to transfer reading (Tom gets the apples) deterministically.

P4 — evals/gsm8k_math/runner.py:
  New sibling function _score_one_candidate_graph(case) -> CaseOutcome.
  Identical shape to _score_one; swaps parse_problem for parse_and_solve;
  preserves verifier/realizer/expected-answer stages. Callers (e.g.
  PR #160's train_sample/v1/runner.py) substitute the new function in
  one line to evaluate the candidate-graph topology.

  9 new wiring tests. Three groups:
    - No regression: cases legacy solves, new also solves.
    - Lift: cases legacy refuses, new solves (the architectural payoff).
    - Wrong==0: out-of-grammar refuses, never wrong.

Regression: 714/714 existing math + runner tests still green.
ADR-0126 total: 74/74 tests green across P1+P2+P3+P4.
2026-05-23 06:36:13 -07:00