a76834cd3f
ADR-0131 Benchmark 1 substrate — the primary discriminator for the
mathematics_logic expert promotion under the architecture-aligned
benchmark composite proposed in ADR-0131.
WHAT LANDED:
generate/math_symbolic_normalizer.py
Deterministic univariate polynomial normalizer. Scope: single
variable, integer coefficients, +/-/*/** operators, parens, no
division, no transcendentals. Pipeline: tokenize -> recursive-
descent parse -> expand-and-collect -> canonical string. Refusal
is first-class via SymbolicError; out-of-scope inputs refuse
rather than guess (preserves wrong == 0).
generate/math_symbolic_equivalence.py
check_equivalence(a, b) -> EquivalenceVerdict
Returns EQUIVALENT / NOT_EQUIVALENT / REFUSED with canonical
strings + reason. Compares byte-equal canonical forms.
evals/math_symbolic_equivalence/v1/
cases.jsonl — 30 hand-curated cases across 18 algebraic
identity categories + 2 out-of-scope refusals.
Coverage: commutative, distributive, square +
cube of binomial, difference of squares, FOIL,
collect like terms, zero cancellation, factoring,
exponent combination, unary negation.
runner.py — CLI entry point. Loads cases, builds report,
writes JSON, exits 0/1 on gate pass/fail.
README.md — methodology, scope, dataset categorization,
exit criterion, baseline result.
tests/
test_math_symbolic_normalizer.py — 44 tests covering parser,
algebra primitives,
canonical-form invariants,
and every refusal path.
test_math_symbolic_equivalence.py — 16 tests on the public
check_equivalence API.
test_adr_0131_1_symbolic_equivalence_lane.py
— 8 tests gating the lane:
dataset integrity, exit
criterion, wrong == 0,
determinism (byte-equal
report across runs).
EMPIRICAL RESULT (the lane PASSED):
correct = 30 / 30 (100.0%)
wrong = 0 / 30 (wrong == 0 invariant satisfied)
refused = 0 / 30 (refusals all matched expected)
correct_rate = 1.00
exit_criterion: PASSED (>= 0.95 required)
CONTRAST WITH ADR-0127-0128 GSM8K TRAIN-SAMPLE RESULT (0/0/50):
This is the first benchmark on the mathematics_logic lane where
the architecture's structural strengths fully express. The result
is the empirical inverse of the GSM8K result — and that's
exactly the architecture-benchmark fit ADR-0131 was written to
re-target toward.
REGRESSION: 1033/1033 existing tests green across math + ADR-0126
+ pack ratification + runner. Zero regressions.
SCOPE DISCIPLINE (per ADR-0131.1 v1 plan):
v1 deliberately narrow (univariate, integer, polynomial). Future
ADR-0131.1.B expansions documented in README: multi-variable,
rationals, larger dataset (~500), sealed holdout per ADR-0119.7
pattern.
PARALLEL WORK (per ADR-0131 plan to run all 3 sub-phases concurrently):
- ADR-0131.2: CORE-native teaching-corpus eval (separate PR)
- ADR-0131.3: bounded-grammar word-problem set (separate PR)
These are independent of ADR-0131.1; no shared files, no
cross-PR coordination required beyond final composite gate.
99 lines
3.2 KiB
Python
99 lines
3.2 KiB
Python
"""ADR-0131.1 — Symbolic equivalence check (Benchmark 1 primitive).
|
|
|
|
Given two algebraic expressions A and B, produces an
|
|
:class:`EquivalenceVerdict` of EQUIVALENT, NOT_EQUIVALENT, or REFUSED
|
|
(with reason). REFUSED preserves wrong == 0: the engine refuses to
|
|
guess on out-of-scope input rather than emit a wrong verdict.
|
|
|
|
Algorithm (v1, polynomial scope):
|
|
1. Normalize A via :func:`generate.math_symbolic_normalizer.normalize`.
|
|
2. Normalize B via the same function.
|
|
3. Compare canonical strings byte-for-byte.
|
|
|
|
If either normalization raises :class:`SymbolicError`, the verdict is
|
|
REFUSED with the propagating reason. This is the wrong-answer
|
|
firewall for the benchmark — anything the normalizer cannot prove
|
|
equivalent (or prove distinct) deterministically is refused.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from dataclasses import dataclass
|
|
from enum import Enum
|
|
from typing import Final
|
|
|
|
from generate.math_symbolic_normalizer import (
|
|
SymbolicError,
|
|
normalize,
|
|
)
|
|
|
|
|
|
class Verdict(str, Enum):
|
|
EQUIVALENT = "equivalent"
|
|
NOT_EQUIVALENT = "not_equivalent"
|
|
REFUSED = "refused"
|
|
|
|
|
|
@dataclass(frozen=True, slots=True)
|
|
class EquivalenceVerdict:
|
|
verdict: Verdict
|
|
canonical_a: str | None # None when verdict is REFUSED and a couldn't normalize
|
|
canonical_b: str | None
|
|
reason: str # empty on EQUIVALENT / NOT_EQUIVALENT; non-empty on REFUSED
|
|
|
|
|
|
REFUSED_VERDICTS: Final[frozenset[Verdict]] = frozenset({Verdict.REFUSED})
|
|
"""Helper set for callers that need to gate on refusal vs decision."""
|
|
|
|
|
|
def check_equivalence(
|
|
expression_a: str,
|
|
expression_b: str,
|
|
*,
|
|
variable: str = "x",
|
|
) -> EquivalenceVerdict:
|
|
"""Return whether ``expression_a`` and ``expression_b`` are
|
|
algebraically equivalent under the v1 polynomial-normalizer scope.
|
|
|
|
Refusal cases (each surfaces a typed reason):
|
|
- Either expression is empty or non-string.
|
|
- Either expression uses an out-of-scope identifier (multi-
|
|
variable, undefined name).
|
|
- Either expression contains a syntactically invalid construct.
|
|
- Either expression uses division, transcendental functions,
|
|
non-integer coefficients, negative exponents, or non-constant
|
|
exponents.
|
|
"""
|
|
try:
|
|
canon_a = normalize(expression_a, variable=variable).to_canonical_string()
|
|
except SymbolicError as exc:
|
|
return EquivalenceVerdict(
|
|
verdict=Verdict.REFUSED,
|
|
canonical_a=None,
|
|
canonical_b=None,
|
|
reason=f"normalize(a) refused: {exc}",
|
|
)
|
|
|
|
try:
|
|
canon_b = normalize(expression_b, variable=variable).to_canonical_string()
|
|
except SymbolicError as exc:
|
|
return EquivalenceVerdict(
|
|
verdict=Verdict.REFUSED,
|
|
canonical_a=canon_a,
|
|
canonical_b=None,
|
|
reason=f"normalize(b) refused: {exc}",
|
|
)
|
|
|
|
if canon_a == canon_b:
|
|
return EquivalenceVerdict(
|
|
verdict=Verdict.EQUIVALENT,
|
|
canonical_a=canon_a,
|
|
canonical_b=canon_b,
|
|
reason="",
|
|
)
|
|
return EquivalenceVerdict(
|
|
verdict=Verdict.NOT_EQUIVALENT,
|
|
canonical_a=canon_a,
|
|
canonical_b=canon_b,
|
|
reason="",
|
|
)
|