core/docs/adr/ADR-0131.1.F-frontier-baseline-comparison.md

ADR-0131.1.F — B1 Symbolic Equivalence: Frontier-Baseline Comparison

Status: Proposed Date: 2026-05-23 Author: CORE agents + reviewers Parent: ADR-0131 Depends on: ADR-0045, ADR-0114a, ADR-0119.4, ADR-0131.1

---

Context

ADR-0131 re-targeted the math-expert promotion away from GSM8K to a composite gate of three architecture-aligned benchmarks. ADR-0131.1 shipped Benchmark 1 (symbolic equivalence v1, 30/30 wrong=0); ADR-0131.1.B hardened it (185/185 wrong=0); ADR-0131.1.S sealed a 14-case holdout under pyrage X25519 to make B1's score externally credible.

ADR-0114a §Obligation #7 requires every capability lane to pair its CORE score with at least one frontier-LLM baseline. ADR-0119.4 established the methodology for the (now-deferred) gsm8k_math lane: frozen citations + a CORE-vs-frontier comparison JSON with an explicit disclaimer about scope mismatch. This ADR adapts that methodology to B1.

The challenge for B1 specifically: univariate polynomial canonical equivalence is not a standard published benchmark, so there are no direct frontier scores to cite. Two responses:

1. Adjacent-benchmark citations. Frozen scores from MATH (Hendrycks et al. 2021), MATH-500, MMLU mathematics, AIME, etc. give the published-context anchor without claiming head-to-head numbers.
2. Live head-to-head, deterministically cached. A provider-agnostic runner queries Anthropic, OpenAI, and Google on the same B1 dataset with the same deterministic prompt, parses replies into the closed CORE verdict vocabulary (equivalent / not_equivalent / refused), and caches every response so that subsequent runs replay byte-equally without re-calling the API.

Both contexts compose into a single comparison.json artifact. The ADR pins the methodology before any head-to-head numbers are recorded, so the numbers — when they land — cannot be retrofit.

---

Decision

Ship a frontier-baseline harness for B1 with three deliverables:

Action items

1. Adjacent-benchmark citations (baselines.py). Frozen ADJACENT_BENCHMARK_CITATIONS tuple containing entries from Anthropic, OpenAI, and Google on published math benchmarks. Each citation has vendor, model, benchmark, score, metric, source_url, source_date, note. URLs are validated for https?:// shape; dates for YYYY-MM-DD shape. The note field carries the scope caveat explicitly per citation.

2. Provider-agnostic runner (frontier_runner.py). Three adapters (Anthropic / OpenAI / Google), each importing its SDK lazily so the package loads cleanly without the SDKs installed. Each provider has a documented FRONTIER_<VENDOR>_KEY env var; the runner refuses with a typed FrontierRunError if the key is absent and the cache cannot cover all cases. Responses are cached one-record-per-line at evals/math_symbolic_equivalence/v1/frontier/responses/<provider>/<model>.jsonl.

3. Comparison composer (comparison.py). Joins CORE's report.json, the cached provider responses, and the frozen citations into one deterministic comparison.json. Scoring emphasizes three architecture-aligned metrics:

   • accuracy — fraction of cases matching expected. The least-load-bearing metric: frontier models will score high on canonical polynomial equivalence.
   • refusal_correctness — fraction of expected="refused" cases the provider actually refused. CORE hits 100% by lane-gate construction; frontier models typically confabulate.
   • determinism — structural assertion (CORE byte-equal across runs; frontier varies). Numeric measurement requires multiple cached runs; the schema reserves the field.

Verdict-parser discipline

The free-text-to-closed-vocab boundary lives in parse_provider_verdict. It is conservative: ambiguous or sentinel-free replies collapse to refused. A polarized verdict is never confabulated from prose. Chain-of-thought replies that mention multiple sentinel tokens use last-token-wins (provider deliberates, then concludes). This is the load-bearing seam that prevents the runner from manufacturing scores the provider didn't actually deliver.

---

Invariants

• citations_dated — every citation has source_date matching YYYY-MM-DD and source_url matching https?://.
• citations_three_vendors — Anthropic, OpenAI, and Google all represented in ADJACENT_BENCHMARK_CITATIONS.
• scope_disclaimer_present — comparison.json contains the non-empty scope_disclaimer documenting B1's scope vs the cited benchmarks.
• verdict_parser_conservative — ambiguous replies collapse to refused, never to a polarized verdict.
• responses_cache_replayable — repeated runs with the same cache produce identical comparison.json bytes.
• no_live_api_in_tests — the test suite never calls a provider API; live calls are gated behind the FRONTIER_<VENDOR>_KEY env var and the frontier_runner CLI entry point.

---

Acceptance evidence

Accepted when:

• evals/math_symbolic_equivalence/v1/frontier/baselines.py ships at least one citation per major vendor.
• evals/math_symbolic_equivalence/v1/frontier/frontier_runner.py exposes the three provider adapters with documented env keys and cache files.
• evals/math_symbolic_equivalence/v1/frontier/comparison.py generates a deterministic comparison.json carrying the schema version, scope disclaimer, CORE score, citations, and (when present) head-to-head runs.
• tests/test_adr_0131_1_F_frontier.py passes cleanly — 22 tests covering citation shape, provider-registry shape, verdict-parser conservatism, runner caching, and comparison determinism.
• The comparison JSON is committed at evals/math_symbolic_equivalence/v1/frontier/comparison.json with CORE's 185/0/0 and zero frontier runs cached — that file becomes the durable record into which actual head-to-head numbers slot deterministically the first time a FRONTIER_*_KEY is exported.

---

Consequences

• B1 (the first leg of the ADR-0131 composite gate) satisfies the Obligation-#7 frontier-pairing requirement without claiming numbers not yet measured.
• The architecture-aligned differentiator (refusal correctness, determinism) is foregrounded by the comparison schema instead of raw accuracy — preserves the post-GSM8K-arc honest framing.
• The harness is reusable. When B2 (ADR-0131.2) and B3 (ADR-0131.3) reach this maturity, their lanes get a near-identical frontier/ subdirectory; the only per-lane bits are the prompt template and the cache directory.
• Running with a real key (e.g. FRONTIER_ANTHROPIC_KEY=...) produces durable evidence — cached per-case provider responses joined to CORE's lane result — that the math-expert promotion claim can cite. The audit trail is the JSONL cache file, not a hand-curated summary.

---

Out of scope

• Running CORE against any published math benchmark (e.g. MATH-500) — reserved for the per-lane sealed-holdout pattern from ADR-0131.1.S.
• Multi-run determinism measurement for frontier models (the schema reserves the field; the harness doesn't yet score it).
• Live API spending policy — the user controls API keys; the harness refuses gracefully when keys are absent.
• B2 and B3 frontier-baseline harnesses — left for follow-up ADRs once their lanes reach v1.B maturity.