54e6bfc0d0
Implements the 4-phase documentation reorganization master plan. - Consolidation: Merged brief/, handoff/, planning/, and decisions/ into briefs/, handoffs/, plans/, and adr/ respectively (101 ADRs relocated) - Root Cleanup: Relocated HANDOFF-gpt55-*.md and key top-level docs (runtime_contracts.md, etc.) to canonical folders. Added superseded alerts. - Indices & Navigation: Created docs/README.md navigation document, docs/sessions/README.md index, docs/adr/README.md index - Note: Also includes prior commit adding ADR-0200+ corpus hygiene governance (ADR-0225, dependency map, backfilled cross-references)
42 lines
2.6 KiB
Markdown
42 lines
2.6 KiB
Markdown
# ADR-0131.3 — Benchmark 3: Bounded-Grammar Word Problems
|
|
|
|
## Status
|
|
Accepted
|
|
|
|
## Date
|
|
2026-05-23
|
|
|
|
## Context
|
|
ADR-0131 re-targeted the `mathematics_logic` expert promotion to three architecture-aligned benchmarks. This decision documents the third and final benchmark of the composite expert gate (ADR-0131.3): bounded-grammar word problems.
|
|
|
|
Historically, evaluating algebraic cognitive systems against datasets like GSM8K created a paraphrase-chasing trap. GSM8K rewards natural language flexibility over algebraic soundness, leading to statistical coercion or approximate nearest-neighbor fallback, which violates CORE's core invariants.
|
|
|
|
B3 establishes the alternative, architecture-aligned claim: **within a closed, bounded grammar, the engine must achieve deterministic, 100% end-to-end correctness.**
|
|
|
|
## Decision
|
|
Establish the `math_bounded_grammar` lane under `evals/math_bounded_grammar/v1/`.
|
|
|
|
### 1. Bounded Grammar as Scope Statement, Not Coverage Claim
|
|
The bounded grammar is an explicit, reviewable contract. We declare that the architecture solves problems *only* within this grammar; any input outside it must trigger a typed refusal. Wide coverage of arbitrary natural-language phrasing is out of scope for this benchmark and is deferred to future grammar-expansion ADRs (v1.B / v2).
|
|
|
|
The templates recognized cover the full typed math pipeline established under:
|
|
- Parser & Math Graph: ADR-0115
|
|
- Deterministic Solver: ADR-0116
|
|
- Trace Verifier: ADR-0117
|
|
- Binding Graph: ADR-0132 through ADR-0135
|
|
|
|
### 2. Dataset Design
|
|
We curate a version-pinned set of 50 word problems split across:
|
|
- `solved_correct` (35 cases): Grammar-conformant problems with unique correct numeric answers, exercising all 8 operation kinds (`add`, `subtract`, `transfer`, `multiply`, `divide`, `apply_rate`, `compare_additive`, `compare_multiplicative`).
|
|
- `solved_wrong` (5 cases): Grammar-conformant problems whose expected answers are deliberately wrong, validating that the verifier/runner catches mismatch errors and that the `wrong == 0` check is load-bearing.
|
|
- `refused` (10 cases): Out-of-grammar problems testing clean, typed parser-level or solver-level refusals.
|
|
|
|
### 3. Exit Criteria
|
|
- `wrong == 0` across all three classes.
|
|
- `correct_rate >= 0.95`.
|
|
- Deterministic byte-equality of the output `report.json` across consecutive runs.
|
|
|
|
## Consequences
|
|
- The B3 benchmark provides a clean, inspectable contract of what CORE can and cannot parse.
|
|
- Any future grammar expansions must be explicitly added to the bounded grammar specification (`grammar.md`) and verified by regression tests.
|
|
- Refusal remains a first-class, defensive property of the cognitive engine.
|