feat(ADR-0131.G.5): aggregate answer composition — combined/together cues wired, axis lane 20/20, wrong==0 (#197)

Closes the vocabulary gap: `combined` and `together` added to `_Q_TOTAL_RE`
and `_Q_ENTITY_RE` tail alternations. Both map to `entity=None` semantics;
the solver's existing sum path is unchanged.

Ships:
- Parser one-line regex extension (`generate/math_candidate_parser.py`)
- 20-case curated axis lane (`G5_aggregate/v1/`) — 5 shapes × 4 cues
- Runner + byte-equal report (20/20 pass, wrong=0)
- 25 tests covering cue vocab, 2/3-entity sums, degenerate aggregate,
  refusals, byte-equality, B3 regression guard, GSM8K safety rail
- ADR-0131.G.5

No admission movement on GSM8K probe (statement-parse bottleneck unchanged).
This commit is contained in:
Shay 2026-05-23 19:42:55 -07:00 committed by GitHub
parent 657c74102b
commit 7f67cea400
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 626 additions and 4 deletions

View file

@ -0,0 +1,89 @@
# ADR-0131.G.5 — Aggregate Answer Composition
**Status:** Accepted
**Parent:** [ADR-0131.G — GSM8K Coverage Probe](ADR-0131.G-gsm8k-coverage-probe.md)
**Date:** 2026-05-23
## Context
The aggregate-answer path — questions like "How many apples do they have
altogether?" — was functionally complete before this ADR. The parser
(`_Q_TOTAL_RE` in `generate/math_candidate_parser.py`) already emitted
`Unknown(entity=None, unit=<unit>)` for aggregate cues, and the solver
(`generate/math_solver.py`) already summed all terminal state entries
matching the questioned unit when `entity is None`.
What was missing:
1. **Vocabulary gap (now closed):** `"combined"` and `"together"` were
absent from `_Q_TOTAL_RE`'s tail alternation, causing questions using
those cues to be refused even when the solver would have produced the
correct sum.
2. **No pinned lane:** no curated axis cases proved the 2-entity,
3-entity, and degenerate aggregate paths end-to-end through
`parse_and_solve`.
## Decision
### Closed aggregate-cue vocabulary
Exactly four cues are admitted:
| Cue | Example tail |
|-----|-------------|
| `in total` | "How many apples do they have in total?" |
| `altogether` | "How many apples do they have altogether?" |
| `combined` | "How many apples do they have combined?" |
| `together` | "How many apples do they have together?" |
All four map to `entity=None` semantics — the solver sums all state
entries whose unit matches the questioned unit, across all entities.
### Solver path (pre-existing)
The `entity is None` branch in `_resolve_unknown` was not changed. It
sums `v for (_, unit), v in state.items() if unit == unknown.unit`.
This ADR extends the cue vocabulary and pins the lane, not the solver.
### Axis lane
20 curated cases at `evals/math_capability_axes/G5_aggregate/v1/cases.jsonl`:
| Shape | Count | Purpose |
|-------|-------|---------|
| 2-entity sum, no operations | 4 | one case per cue |
| 3-entity sum, no operations | 4 | one case per cue |
| 2-entity sum with add/subtract op | 4 | mixed cues |
| Single-entity degenerate | 4 | regression guard |
| Refusal: outside closed cue | 4 | wrong==0 probe |
Refusal cases use question forms outside the closed `_Q_TOTAL_RE`
pattern (e.g., "How many apples does everyone have?", "What is the
total number of coins?") to verify the parser correctly refuses
paraphrases not in the closed cue set.
### Gate
`wrong == 0` on every axis case. GSM8K `admitted_wrong == 0` preserved
(no admission movement expected — all 50 sample cases still refuse at
statement parsing; question-layer work cannot lift that).
## Deferred
- **Implicit aggregation without a cue word:** "How many apples do Sam
and Tom have?" requires coreference resolution (named-entity →
pronoun-equivalent grouping). Out of scope for the closed-cue model.
- **Rate-based aggregation:** "How many dollars did they earn in total?"
where the unit derives from a rate operation. Requires rate-verb
support in the statement parser.
- **GSM8K admission lift:** all 50 sample cases fail at statement
parsing (rate verbs, compound sentences, implicit entities).
Question-layer cue extensions cannot move that number.
## Evidence
- Axis runner: `evals/math_capability_axes/G5_aggregate/v1/runner.py`
- Report: `evals/math_capability_axes/G5_aggregate/v1/report.json`
- Tests: `tests/test_adr_0131_G5_aggregate.py`
- B3 lane unchanged.
- GSM8K `admitted_wrong == 0` preserved.

View file

@ -0,0 +1,20 @@
{"case_id": "G5-2ent-001", "category": "2entity_no_op", "cue": "altogether", "problem": "Sam has 5 apples. Tom has 3 apples. How many apples do they have altogether?", "expected_answer": 8.0}
{"case_id": "G5-2ent-002", "category": "2entity_no_op", "cue": "in total", "problem": "Alice has 7 books. Bob has 4 books. How many books do they have in total?", "expected_answer": 11.0}
{"case_id": "G5-2ent-003", "category": "2entity_no_op", "cue": "combined", "problem": "Maya has 6 coins. Leo has 9 coins. How many coins do they have combined?", "expected_answer": 15.0}
{"case_id": "G5-2ent-004", "category": "2entity_no_op", "cue": "together", "problem": "Jade has 12 stickers. Finn has 8 stickers. How many stickers do they have together?", "expected_answer": 20.0}
{"case_id": "G5-3ent-001", "category": "3entity_no_op", "cue": "altogether", "problem": "Sam has 5 apples. Tom has 3 apples. Amy has 2 apples. How many apples do they have altogether?", "expected_answer": 10.0}
{"case_id": "G5-3ent-002", "category": "3entity_no_op", "cue": "in total", "problem": "Alice has 4 books. Bob has 6 books. Carol has 2 books. How many books do they have in total?", "expected_answer": 12.0}
{"case_id": "G5-3ent-003", "category": "3entity_no_op", "cue": "combined", "problem": "Maya has 10 coins. Leo has 5 coins. Nina has 3 coins. How many coins do they have combined?", "expected_answer": 18.0}
{"case_id": "G5-3ent-004", "category": "3entity_no_op", "cue": "together", "problem": "Jade has 7 stickers. Finn has 4 stickers. Rex has 9 stickers. How many stickers do they have together?", "expected_answer": 20.0}
{"case_id": "G5-op-001", "category": "2entity_with_op", "cue": "altogether", "problem": "Sam has 5 apples. Sam buys 3 apples. Tom has 4 apples. How many apples do they have altogether?", "expected_answer": 12.0}
{"case_id": "G5-op-002", "category": "2entity_with_op", "cue": "combined", "problem": "Alice has 10 books. Alice loses 2 books. Bob has 6 books. How many books do they have combined?", "expected_answer": 14.0}
{"case_id": "G5-op-003", "category": "2entity_with_op", "cue": "in total", "problem": "Maya has 8 coins. Leo has 5 coins. Leo finds 3 coins. How many coins do they have in total?", "expected_answer": 16.0}
{"case_id": "G5-op-004", "category": "2entity_with_op", "cue": "together", "problem": "Jade has 12 stickers. Jade gives away 4 stickers. Finn has 8 stickers. How many stickers do they have together?", "expected_answer": 16.0}
{"case_id": "G5-degen-001", "category": "single_entity_total_cue", "cue": "in total", "problem": "Sam has 5 apples. How many apples do they have in total?", "expected_answer": 5.0}
{"case_id": "G5-degen-002", "category": "single_entity_total_cue", "cue": "altogether", "problem": "Alice has 7 books. Alice buys 3 books. How many books do they have altogether?", "expected_answer": 10.0}
{"case_id": "G5-degen-003", "category": "single_entity_total_cue", "cue": "combined", "problem": "Maya has 9 coins. Maya loses 2 coins. How many coins do they have combined?", "expected_answer": 7.0}
{"case_id": "G5-degen-004", "category": "single_entity_total_cue", "cue": "together", "problem": "Finn has 6 stickers. How many stickers do they have together?", "expected_answer": 6.0}
{"case_id": "G5-refuse-001", "category": "refusal_outside_closed_cue", "cue": "none", "problem": "Sam has 5 apples. Tom has 3 apples. How many apples does everyone have?", "expected_answer": null}
{"case_id": "G5-refuse-002", "category": "refusal_outside_closed_cue", "cue": "none", "problem": "Alice has 4 coins. Bob has 6 coins. What is the total number of coins?", "expected_answer": null}
{"case_id": "G5-refuse-003", "category": "refusal_outside_closed_cue", "cue": "none", "problem": "Maya has 10 books. Leo has 5 books. How many books do Sam and Leo have?", "expected_answer": null}
{"case_id": "G5-refuse-004", "category": "refusal_outside_closed_cue", "cue": "none", "problem": "Jade has 8 stickers. Finn has 4 stickers. How many stickers are there?", "expected_answer": null}

View file

@ -0,0 +1,218 @@
{
"adr": "0131.G.5",
"axis": "aggregate",
"cases_path": "evals/math_capability_axes/G5_aggregate/v1/cases.jsonl",
"metrics": {
"cases_total": 20,
"pass_rate": 1.0,
"passed": 20,
"wrong": 0,
"wrong_count_is_zero": true,
"wrong_rate": 0.0
},
"per_case": [
{
"answer": 8.0,
"case_id": "G5-2ent-001",
"category": "2entity_no_op",
"cue": "altogether",
"expected_answer": 8.0,
"outcome": "pass",
"reason": ""
},
{
"answer": 11.0,
"case_id": "G5-2ent-002",
"category": "2entity_no_op",
"cue": "in total",
"expected_answer": 11.0,
"outcome": "pass",
"reason": ""
},
{
"answer": 15.0,
"case_id": "G5-2ent-003",
"category": "2entity_no_op",
"cue": "combined",
"expected_answer": 15.0,
"outcome": "pass",
"reason": ""
},
{
"answer": 20.0,
"case_id": "G5-2ent-004",
"category": "2entity_no_op",
"cue": "together",
"expected_answer": 20.0,
"outcome": "pass",
"reason": ""
},
{
"answer": 10.0,
"case_id": "G5-3ent-001",
"category": "3entity_no_op",
"cue": "altogether",
"expected_answer": 10.0,
"outcome": "pass",
"reason": ""
},
{
"answer": 12.0,
"case_id": "G5-3ent-002",
"category": "3entity_no_op",
"cue": "in total",
"expected_answer": 12.0,
"outcome": "pass",
"reason": ""
},
{
"answer": 18.0,
"case_id": "G5-3ent-003",
"category": "3entity_no_op",
"cue": "combined",
"expected_answer": 18.0,
"outcome": "pass",
"reason": ""
},
{
"answer": 20.0,
"case_id": "G5-3ent-004",
"category": "3entity_no_op",
"cue": "together",
"expected_answer": 20.0,
"outcome": "pass",
"reason": ""
},
{
"answer": 12.0,
"case_id": "G5-op-001",
"category": "2entity_with_op",
"cue": "altogether",
"expected_answer": 12.0,
"outcome": "pass",
"reason": ""
},
{
"answer": 14.0,
"case_id": "G5-op-002",
"category": "2entity_with_op",
"cue": "combined",
"expected_answer": 14.0,
"outcome": "pass",
"reason": ""
},
{
"answer": 16.0,
"case_id": "G5-op-003",
"category": "2entity_with_op",
"cue": "in total",
"expected_answer": 16.0,
"outcome": "pass",
"reason": ""
},
{
"answer": 16.0,
"case_id": "G5-op-004",
"category": "2entity_with_op",
"cue": "together",
"expected_answer": 16.0,
"outcome": "pass",
"reason": ""
},
{
"answer": 5.0,
"case_id": "G5-degen-001",
"category": "single_entity_total_cue",
"cue": "in total",
"expected_answer": 5.0,
"outcome": "pass",
"reason": ""
},
{
"answer": 10.0,
"case_id": "G5-degen-002",
"category": "single_entity_total_cue",
"cue": "altogether",
"expected_answer": 10.0,
"outcome": "pass",
"reason": ""
},
{
"answer": 7.0,
"case_id": "G5-degen-003",
"category": "single_entity_total_cue",
"cue": "combined",
"expected_answer": 7.0,
"outcome": "pass",
"reason": ""
},
{
"answer": 6.0,
"case_id": "G5-degen-004",
"category": "single_entity_total_cue",
"cue": "together",
"expected_answer": 6.0,
"outcome": "pass",
"reason": ""
},
{
"answer": null,
"case_id": "G5-refuse-001",
"category": "refusal_outside_closed_cue",
"cue": "none",
"expected_answer": null,
"outcome": "pass",
"reason": ""
},
{
"answer": null,
"case_id": "G5-refuse-002",
"category": "refusal_outside_closed_cue",
"cue": "none",
"expected_answer": null,
"outcome": "pass",
"reason": ""
},
{
"answer": null,
"case_id": "G5-refuse-003",
"category": "refusal_outside_closed_cue",
"cue": "none",
"expected_answer": null,
"outcome": "pass",
"reason": ""
},
{
"answer": null,
"case_id": "G5-refuse-004",
"category": "refusal_outside_closed_cue",
"cue": "none",
"expected_answer": null,
"outcome": "pass",
"reason": ""
}
],
"per_category": {
"2entity_no_op": {
"pass": 4,
"wrong": 0
},
"2entity_with_op": {
"pass": 4,
"wrong": 0
},
"3entity_no_op": {
"pass": 4,
"wrong": 0
},
"refusal_outside_closed_cue": {
"pass": 4,
"wrong": 0
},
"single_entity_total_cue": {
"pass": 4,
"wrong": 0
}
},
"schema_version": 1
}

View file

@ -0,0 +1,129 @@
"""ADR-0131.G.5 — Capability axis runner for aggregate answer composition.
Exercises the ``entity=None`` sum path in :mod:`generate.math_solver` via
:func:`generate.math_candidate_graph.parse_and_solve` against curated
coverage cases that are independent of GSM8K.
Per-case classification:
| Case category | pass criterion |
|-----------------------------|-------------------------------------------|
| 2entity_no_op | answer == expected_answer (exact float) |
| 3entity_no_op | answer == expected_answer |
| 2entity_with_op | answer == expected_answer |
| single_entity_total_cue | answer == expected_answer |
| refusal_outside_closed_cue | answer is None (question not admitted) |
``wrong`` is non-zero only if a positive case returns the wrong numeric
answer or a refusal case emits a numeric answer. ``wrong == 0`` is the
load-bearing gate (ADR-0114a Obligation #4).
Determinism: case order in ``cases.jsonl`` is the report order; same
input file byte-equal report.
"""
from __future__ import annotations
import json
from pathlib import Path
from typing import Any
from generate.math_candidate_graph import parse_and_solve
_HERE = Path(__file__).resolve().parent
_CASES_PATH = _HERE / "cases.jsonl"
_REPORT_PATH = _HERE / "report.json"
def _load_cases() -> list[dict[str, Any]]:
return [
json.loads(line)
for line in _CASES_PATH.read_text(encoding="utf-8").splitlines()
if line.strip()
]
def _score_case(case: dict[str, Any]) -> dict[str, Any]:
r = parse_and_solve(case["problem"])
exp = case["expected_answer"]
category = case["category"]
if exp is not None:
if r.answer == exp:
outcome, reason = "pass", ""
elif r.answer is None:
outcome = "wrong"
reason = f"expected {exp} but got refusal: {r.refusal_reason}"
else:
outcome = "wrong"
reason = f"expected {exp} but got {r.answer}"
else:
if r.answer is None:
outcome, reason = "pass", ""
else:
outcome = "wrong"
reason = f"expected refusal but got answer {r.answer}"
return {
"case_id": case["case_id"],
"category": category,
"cue": case.get("cue", ""),
"outcome": outcome,
"reason": reason,
"answer": r.answer,
"expected_answer": exp,
}
def build_report() -> dict[str, Any]:
cases = _load_cases()
per_case = [_score_case(c) for c in cases]
total = len(per_case)
passed = sum(1 for d in per_case if d["outcome"] == "pass")
wrong = sum(1 for d in per_case if d["outcome"] == "wrong")
by_category: dict[str, dict[str, int]] = {}
for d in per_case:
slot = by_category.setdefault(d["category"], {"pass": 0, "wrong": 0})
slot[d["outcome"]] = slot.get(d["outcome"], 0) + 1
return {
"schema_version": 1,
"adr": "0131.G.5",
"axis": "aggregate",
"cases_path": "evals/math_capability_axes/G5_aggregate/v1/cases.jsonl",
"metrics": {
"cases_total": total,
"passed": passed,
"wrong": wrong,
"pass_rate": (passed / total) if total else 0.0,
"wrong_rate": (wrong / total) if total else 0.0,
"wrong_count_is_zero": wrong == 0,
},
"per_category": {
k: dict(sorted(v.items())) for k, v in sorted(by_category.items())
},
"per_case": per_case,
}
def write_report(report: dict[str, Any]) -> None:
_REPORT_PATH.write_text(
json.dumps(report, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
def main() -> int:
report = build_report()
write_report(report)
m = report["metrics"]
print(
f"ADR-0131.G.5 aggregate: passed {m['passed']}/{m['cases_total']} "
f"({m['pass_rate']:.1%}); wrong={m['wrong']} (gate: must be 0)"
)
for cat, counts in report["per_category"].items():
print(f" {cat:30s} {counts}")
return 0 if m["wrong_count_is_zero"] else 1
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -685,11 +685,15 @@ class CandidateUnknown:
Two question shapes in P3 scope: Two question shapes in P3 scope:
- ``How many <unit> does <Entity> have [left|now|in total|altogether]?`` - ``How many <unit> does <Entity> have [left|now|in total|altogether|combined|together]?``
``Unknown(entity=<Entity>, unit=<unit>)`` ``Unknown(entity=<Entity>, unit=<unit>)``
- ``How many <unit> do they have [left|now|in total|altogether]?`` - ``How many <unit> do they have [left|now|in total|altogether|combined|together]?``
``Unknown(entity=None, unit=<unit>)`` (total-across) ``Unknown(entity=None, unit=<unit>)`` (total-across)
Closed aggregate-cue vocabulary: ``in total``, ``altogether``,
``combined``, ``together``. All four map to ``entity=None`` on the
total-across form.
The round-trip filter for questions checks the unit token and (when The round-trip filter for questions checks the unit token and (when
present) the entity token both appear in the source span. present) the entity token both appear in the source span.
""" """
@ -703,13 +707,13 @@ class CandidateUnknown:
_Q_ENTITY_RE: Final[re.Pattern[str]] = re.compile( _Q_ENTITY_RE: Final[re.Pattern[str]] = re.compile(
r"^How\s+many\s+(?P<unit>\w+)\s+(?:does|do)\s+" r"^How\s+many\s+(?P<unit>\w+)\s+(?:does|do)\s+"
rf"(?P<entity>{_ENTITY})" rf"(?P<entity>{_ENTITY})"
r"\s+have(?:\s+(?:left|now|in\s+total|altogether)){0,2}\s*\??$", r"\s+have(?:\s+(?:left|now|in\s+total|altogether|combined|together)){0,2}\s*\??$",
flags=re.IGNORECASE, flags=re.IGNORECASE,
) )
_Q_TOTAL_RE: Final[re.Pattern[str]] = re.compile( _Q_TOTAL_RE: Final[re.Pattern[str]] = re.compile(
r"^How\s+many\s+(?P<unit>\w+)\s+do\s+they\s+have" r"^How\s+many\s+(?P<unit>\w+)\s+do\s+they\s+have"
r"(?:\s+(?:in\s+total|altogether|left|now)){0,2}\s*\??$", r"(?:\s+(?:in\s+total|altogether|combined|together|left|now)){0,2}\s*\??$",
flags=re.IGNORECASE, flags=re.IGNORECASE,
) )

View file

@ -0,0 +1,162 @@
"""ADR-0131.G.5 — Aggregate answer composition axis lane tests.
Pins the closed aggregate-cue vocabulary (``in total``, ``altogether``,
``combined``, ``together``) and the end-to-end ``parse_and_solve`` path
for 2-entity, 3-entity, single-entity, and refusal shapes.
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from evals.math_capability_axes.G5_aggregate.v1.runner import build_report
from generate.math_candidate_graph import parse_and_solve
from generate.math_candidate_parser import extract_question_candidates
_REPO = Path(__file__).resolve().parent.parent
_GSM8K_LEGACY_REPORT = (
_REPO / "evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json"
)
_GSM8K_CG_REPORT = _REPO / "evals/gsm8k_math/train_sample/v1/report.json"
# ── Cue vocabulary tests ─────────────────────────────────────────────
class TestCueVocabulary:
"""Verify that combined and together parse to entity=None."""
@pytest.mark.parametrize("cue", ["combined", "together", "altogether", "in total"])
def test_cue_parses_to_entity_none(self, cue: str) -> None:
q = f"How many apples do they have {cue}?"
cands = extract_question_candidates(q)
assert len(cands) >= 1, f"no candidate for cue {cue!r}"
assert cands[0].unknown.entity is None
assert cands[0].unknown.unit == "apples"
def test_closed_cue_docstring_lists_all_four(self) -> None:
import generate.math_candidate_parser as mod
src = Path(mod.__file__).read_text(encoding="utf-8")
for cue in ("in total", "altogether", "combined", "together"):
assert cue in src, f"cue {cue!r} missing from parser source"
# ── End-to-end parse_and_solve tests ─────────────────────────────────
class TestTwoEntityNoOp:
@pytest.mark.parametrize(
"problem, expected",
[
("Sam has 5 apples. Tom has 3 apples. How many apples do they have altogether?", 8.0),
("Alice has 7 books. Bob has 4 books. How many books do they have in total?", 11.0),
("Maya has 6 coins. Leo has 9 coins. How many coins do they have combined?", 15.0),
("Jade has 12 stickers. Finn has 8 stickers. How many stickers do they have together?", 20.0),
],
)
def test_two_entity_sum(self, problem: str, expected: float) -> None:
r = parse_and_solve(problem)
assert r.answer == expected
assert r.refusal_reason is None
class TestThreeEntityNoOp:
@pytest.mark.parametrize(
"problem, expected",
[
("Sam has 5 apples. Tom has 3 apples. Amy has 2 apples. How many apples do they have altogether?", 10.0),
("Alice has 4 books. Bob has 6 books. Carol has 2 books. How many books do they have in total?", 12.0),
("Maya has 10 coins. Leo has 5 coins. Nina has 3 coins. How many coins do they have combined?", 18.0),
("Jade has 7 stickers. Finn has 4 stickers. Rex has 9 stickers. How many stickers do they have together?", 20.0),
],
)
def test_three_entity_sum(self, problem: str, expected: float) -> None:
r = parse_and_solve(problem)
assert r.answer == expected
assert r.refusal_reason is None
class TestSingleEntityDegenerate:
def test_single_entity_identity(self) -> None:
r = parse_and_solve("Sam has 5 apples. How many apples do they have in total?")
assert r.answer == 5.0
def test_single_entity_with_op(self) -> None:
r = parse_and_solve("Alice has 7 books. Alice buys 3 books. How many books do they have altogether?")
assert r.answer == 10.0
class TestMismatchedUnitRefusal:
@pytest.mark.parametrize(
"problem",
[
"Sam has 5 apples. Tom has 3 apples. How many apples does everyone have?",
"Alice has 4 coins. Bob has 6 coins. What is the total number of coins?",
"Maya has 10 books. Leo has 5 books. How many books do Sam and Leo have?",
"Jade has 8 stickers. Finn has 4 stickers. How many stickers are there?",
],
)
def test_outside_closed_cue_refuses(self, problem: str) -> None:
r = parse_and_solve(problem)
assert r.answer is None, f"expected refusal but got {r.answer}"
# ── Axis lane gate ───────────────────────────────────────────────────
class TestAxisLaneGate:
def test_wrong_is_zero(self) -> None:
report = build_report()
assert report["metrics"]["wrong"] == 0
assert report["metrics"]["wrong_count_is_zero"] is True
def test_report_byte_equal_across_runs(self) -> None:
r1 = build_report()
r2 = build_report()
s1 = json.dumps(r1, indent=2, sort_keys=True)
s2 = json.dumps(r2, indent=2, sort_keys=True)
assert s1 == s2
def test_all_categories_present(self) -> None:
report = build_report()
expected_cats = {
"2entity_no_op",
"3entity_no_op",
"2entity_with_op",
"single_entity_total_cue",
"refusal_outside_closed_cue",
}
assert set(report["per_category"].keys()) == expected_cats
# ── B3 regression guard ──────────────────────────────────────────────
def test_b3_lane_still_passes() -> None:
"""B3 bounded-grammar lane must remain green after G5 changes."""
from evals.math_bounded_grammar.v1.runner import build_report as b3_build, load_cases
cases_path = _REPO / "evals" / "math_bounded_grammar" / "v1" / "cases.jsonl"
report = b3_build(load_cases(cases_path))
assert report["metrics"]["wrong"] == 0, (
f"B3 lane regression: wrong={report['metrics']['wrong']}"
)
# ── GSM8K safety rail ────────────────────────────────────────────────
def test_gsm8k_legacy_probe_safety_rail_intact() -> None:
"""ADR-0131.G invariant: legacy probe still shows admitted_wrong == 0."""
data = json.loads(_GSM8K_LEGACY_REPORT.read_text(encoding="utf-8"))
assert data["metrics"]["admitted_wrong"] == 0
def test_gsm8k_candidate_graph_probe_wrong_zero() -> None:
"""ADR-0131.G invariant: candidate-graph probe shows wrong == 0."""
data = json.loads(_GSM8K_CG_REPORT.read_text(encoding="utf-8"))
assert data["counts"]["wrong"] == 0