ADR-0131.3 — Benchmark 3: Bounded-Grammar Word Problems

Status

Accepted

Date

2026-05-23

Context

ADR-0131 re-targeted the mathematics_logic expert promotion to three architecture-aligned benchmarks. This decision documents the third and final benchmark of the composite expert gate (ADR-0131.3): bounded-grammar word problems.

Historically, evaluating algebraic cognitive systems against datasets like GSM8K created a paraphrase-chasing trap. GSM8K rewards natural language flexibility over algebraic soundness, leading to statistical coercion or approximate nearest-neighbor fallback, which violates CORE's core invariants.

B3 establishes the alternative, architecture-aligned claim: within a closed, bounded grammar, the engine must achieve deterministic, 100% end-to-end correctness.

Decision

Establish the math_bounded_grammar lane under evals/math_bounded_grammar/v1/.

1. Bounded Grammar as Scope Statement, Not Coverage Claim

The bounded grammar is an explicit, reviewable contract. We declare that the architecture solves problems only within this grammar; any input outside it must trigger a typed refusal. Wide coverage of arbitrary natural-language phrasing is out of scope for this benchmark and is deferred to future grammar-expansion ADRs (v1.B / v2).

The templates recognized cover the full typed math pipeline established under:

Parser & Math Graph: ADR-0115
Deterministic Solver: ADR-0116
Trace Verifier: ADR-0117
Binding Graph: ADR-0132 through ADR-0135

2. Dataset Design

We curate a version-pinned set of 50 word problems split across:

solved_correct (35 cases): Grammar-conformant problems with unique correct numeric answers, exercising all 8 operation kinds (add, subtract, transfer, multiply, divide, apply_rate, compare_additive, compare_multiplicative).
solved_wrong (5 cases): Grammar-conformant problems whose expected answers are deliberately wrong, validating that the verifier/runner catches mismatch errors and that the wrong == 0 check is load-bearing.
refused (10 cases): Out-of-grammar problems testing clean, typed parser-level or solver-level refusals.

3. Exit Criteria

wrong == 0 across all three classes.
correct_rate >= 0.95.
Deterministic byte-equality of the output report.json across consecutive runs.

Consequences

The B3 benchmark provides a clean, inspectable contract of what CORE can and cannot parse.
Any future grammar expansions must be explicitly added to the bounded grammar specification (grammar.md) and verified by regression tests.
Refusal remains a first-class, defensive property of the cognitive engine.

2.6 KiB Raw Blame History