a76834cd3f
ADR-0131 Benchmark 1 substrate — the primary discriminator for the
mathematics_logic expert promotion under the architecture-aligned
benchmark composite proposed in ADR-0131.
WHAT LANDED:
generate/math_symbolic_normalizer.py
Deterministic univariate polynomial normalizer. Scope: single
variable, integer coefficients, +/-/*/** operators, parens, no
division, no transcendentals. Pipeline: tokenize -> recursive-
descent parse -> expand-and-collect -> canonical string. Refusal
is first-class via SymbolicError; out-of-scope inputs refuse
rather than guess (preserves wrong == 0).
generate/math_symbolic_equivalence.py
check_equivalence(a, b) -> EquivalenceVerdict
Returns EQUIVALENT / NOT_EQUIVALENT / REFUSED with canonical
strings + reason. Compares byte-equal canonical forms.
evals/math_symbolic_equivalence/v1/
cases.jsonl — 30 hand-curated cases across 18 algebraic
identity categories + 2 out-of-scope refusals.
Coverage: commutative, distributive, square +
cube of binomial, difference of squares, FOIL,
collect like terms, zero cancellation, factoring,
exponent combination, unary negation.
runner.py — CLI entry point. Loads cases, builds report,
writes JSON, exits 0/1 on gate pass/fail.
README.md — methodology, scope, dataset categorization,
exit criterion, baseline result.
tests/
test_math_symbolic_normalizer.py — 44 tests covering parser,
algebra primitives,
canonical-form invariants,
and every refusal path.
test_math_symbolic_equivalence.py — 16 tests on the public
check_equivalence API.
test_adr_0131_1_symbolic_equivalence_lane.py
— 8 tests gating the lane:
dataset integrity, exit
criterion, wrong == 0,
determinism (byte-equal
report across runs).
EMPIRICAL RESULT (the lane PASSED):
correct = 30 / 30 (100.0%)
wrong = 0 / 30 (wrong == 0 invariant satisfied)
refused = 0 / 30 (refusals all matched expected)
correct_rate = 1.00
exit_criterion: PASSED (>= 0.95 required)
CONTRAST WITH ADR-0127-0128 GSM8K TRAIN-SAMPLE RESULT (0/0/50):
This is the first benchmark on the mathematics_logic lane where
the architecture's structural strengths fully express. The result
is the empirical inverse of the GSM8K result — and that's
exactly the architecture-benchmark fit ADR-0131 was written to
re-target toward.
REGRESSION: 1033/1033 existing tests green across math + ADR-0126
+ pack ratification + runner. Zero regressions.
SCOPE DISCIPLINE (per ADR-0131.1 v1 plan):
v1 deliberately narrow (univariate, integer, polynomial). Future
ADR-0131.1.B expansions documented in README: multi-variable,
rationals, larger dataset (~500), sealed holdout per ADR-0119.7
pattern.
PARALLEL WORK (per ADR-0131 plan to run all 3 sub-phases concurrently):
- ADR-0131.2: CORE-native teaching-corpus eval (separate PR)
- ADR-0131.3: bounded-grammar word-problem set (separate PR)
These are independent of ADR-0131.1; no shared files, no
cross-PR coordination required beyond final composite gate.
96 lines
3.3 KiB
Python
96 lines
3.3 KiB
Python
"""ADR-0131.1 — tests for the symbolic equivalence check primitive."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from generate.math_symbolic_equivalence import (
|
|
Verdict,
|
|
check_equivalence,
|
|
)
|
|
|
|
|
|
class TestEquivalent:
|
|
def test_identical_expressions(self) -> None:
|
|
v = check_equivalence("x + 1", "x + 1")
|
|
assert v.verdict == Verdict.EQUIVALENT
|
|
assert v.canonical_a == v.canonical_b == "x+1"
|
|
|
|
def test_distributive(self) -> None:
|
|
v = check_equivalence("2*(x + 3)", "2*x + 6")
|
|
assert v.verdict == Verdict.EQUIVALENT
|
|
|
|
def test_square_of_binomial(self) -> None:
|
|
v = check_equivalence("(x + 1)^2", "x^2 + 2*x + 1")
|
|
assert v.verdict == Verdict.EQUIVALENT
|
|
|
|
def test_difference_of_squares(self) -> None:
|
|
v = check_equivalence("(x + 1)*(x - 1)", "x^2 - 1")
|
|
assert v.verdict == Verdict.EQUIVALENT
|
|
|
|
def test_collect_like_terms(self) -> None:
|
|
v = check_equivalence("2*x + 3*x + x", "6*x")
|
|
assert v.verdict == Verdict.EQUIVALENT
|
|
|
|
def test_zero_cancellation(self) -> None:
|
|
v = check_equivalence("x - x + 5", "5")
|
|
assert v.verdict == Verdict.EQUIVALENT
|
|
|
|
|
|
class TestNotEquivalent:
|
|
def test_different_constant(self) -> None:
|
|
v = check_equivalence("x + 1", "x + 2")
|
|
assert v.verdict == Verdict.NOT_EQUIVALENT
|
|
assert v.canonical_a == "x+1"
|
|
assert v.canonical_b == "x+2"
|
|
|
|
def test_different_degree(self) -> None:
|
|
v = check_equivalence("x^2", "x^3")
|
|
assert v.verdict == Verdict.NOT_EQUIVALENT
|
|
|
|
def test_sign_flipped(self) -> None:
|
|
v = check_equivalence("(x + 1)^2", "(x - 1)^2")
|
|
assert v.verdict == Verdict.NOT_EQUIVALENT
|
|
|
|
|
|
class TestRefused:
|
|
def test_empty_left(self) -> None:
|
|
v = check_equivalence("", "x + 1")
|
|
assert v.verdict == Verdict.REFUSED
|
|
assert "normalize(a) refused" in v.reason
|
|
|
|
def test_out_of_scope_variable_left(self) -> None:
|
|
v = check_equivalence("x + y", "x + 1")
|
|
assert v.verdict == Verdict.REFUSED
|
|
assert "single variable" in v.reason
|
|
|
|
def test_division_refused(self) -> None:
|
|
v = check_equivalence("x/2", "x")
|
|
assert v.verdict == Verdict.REFUSED
|
|
|
|
def test_a_normalizes_b_refuses(self) -> None:
|
|
# a is fine, b uses y -> refusal with canonical_a populated
|
|
v = check_equivalence("x + 1", "y + 1")
|
|
assert v.verdict == Verdict.REFUSED
|
|
assert v.canonical_a == "x+1"
|
|
assert v.canonical_b is None
|
|
assert "normalize(b) refused" in v.reason
|
|
|
|
def test_refused_verdict_is_first_class(self) -> None:
|
|
# Refusal preserves wrong == 0 — the verdict is REFUSED, never
|
|
# silently coerced to NOT_EQUIVALENT.
|
|
v = check_equivalence("garbage(", "x")
|
|
assert v.verdict == Verdict.REFUSED
|
|
|
|
|
|
class TestDeterminism:
|
|
def test_same_inputs_same_verdict(self) -> None:
|
|
# Re-running produces byte-equal verdict.
|
|
a, b = "(x + 2)*(x - 2)", "x^2 - 4"
|
|
v1 = check_equivalence(a, b)
|
|
v2 = check_equivalence(a, b)
|
|
assert v1 == v2
|
|
|
|
def test_canonical_strings_are_byte_equal_on_equivalence(self) -> None:
|
|
v = check_equivalence("(x + 1)^2", "x^2 + 2*x + 1")
|
|
assert v.canonical_a is not None
|
|
assert v.canonical_b is not None
|
|
assert v.canonical_a.encode("utf-8") == v.canonical_b.encode("utf-8")
|