core/docs/adr/ADR-0131.1.F-frontier-baseline-comparison.md

# ADR-0131.1.F — B1 Symbolic Equivalence: Frontier-Baseline Comparison

**Status:** Proposed
**Date:** 2026-05-23
**Author:** CORE agents + reviewers
**Parent:** [ADR-0131](./ADR-0131-math-expert-rebench.md)
**Depends on:**
[ADR-0045](./ADR-0045-long-context-recall-vs-transformer-baselines.md),
[ADR-0114a](./ADR-0114a-anti-overfitting-proof-obligations.md),
[ADR-0119.4](./ADR-0119.4-frontier-baseline-comparison.md),
[ADR-0131.1](./ADR-0131-math-expert-rebench.md)

---

## Context

ADR-0131 re-targeted the math-expert promotion away from GSM8K to a
composite gate of three architecture-aligned benchmarks. ADR-0131.1
shipped Benchmark 1 (symbolic equivalence v1, 30/30 wrong=0); ADR-0131.1.B
hardened it (185/185 wrong=0); ADR-0131.1.S sealed a 14-case holdout
under pyrage X25519 to make B1's score externally credible.

ADR-0114a §Obligation #7 requires every capability lane to pair its
CORE score with at least one frontier-LLM baseline. ADR-0119.4
established the methodology for the (now-deferred) `gsm8k_math` lane:
frozen citations + a CORE-vs-frontier comparison JSON with an explicit
disclaimer about scope mismatch. This ADR adapts that methodology to
B1.

The challenge for B1 specifically: univariate polynomial canonical
equivalence is not a standard published benchmark, so there are no
direct frontier scores to cite. Two responses:

1. **Adjacent-benchmark citations.** Frozen scores from MATH (Hendrycks
   et al. 2021), MATH-500, MMLU mathematics, AIME, etc. give the
   published-context anchor without claiming head-to-head numbers.
2. **Live head-to-head, deterministically cached.** A
   provider-agnostic runner queries Anthropic, OpenAI, and Google on
   the *same* B1 dataset with the *same* deterministic prompt, parses
   replies into the closed CORE verdict vocabulary
   (`equivalent` / `not_equivalent` / `refused`), and caches every
   response so that subsequent runs replay byte-equally without
   re-calling the API.

Both contexts compose into a single `comparison.json` artifact. The
ADR pins the methodology *before* any head-to-head numbers are
recorded, so the numbers — when they land — cannot be retrofit.

---

## Decision

Ship a frontier-baseline harness for B1 with three deliverables:

### Action items

1. **Adjacent-benchmark citations (`baselines.py`).** Frozen
   `ADJACENT_BENCHMARK_CITATIONS` tuple containing entries from
   Anthropic, OpenAI, and Google on published math benchmarks. Each
   citation has `vendor`, `model`, `benchmark`, `score`, `metric`,
   `source_url`, `source_date`, `note`. URLs are validated for
   `https?://` shape; dates for `YYYY-MM-DD` shape. The note field
   carries the scope caveat explicitly per citation.

2. **Provider-agnostic runner (`frontier_runner.py`).** Three
   adapters (Anthropic / OpenAI / Google), each importing its SDK
   lazily so the package loads cleanly without the SDKs installed.
   Each provider has a documented `FRONTIER_<VENDOR>_KEY` env var; the
   runner refuses with a typed `FrontierRunError` if the key is
   absent and the cache cannot cover all cases. Responses are cached
   one-record-per-line at
   `evals/math_symbolic_equivalence/v1/frontier/responses/<provider>/<model>.jsonl`.

3. **Comparison composer (`comparison.py`).** Joins CORE's
   `report.json`, the cached provider responses, and the frozen
   citations into one deterministic `comparison.json`. Scoring
   emphasizes three architecture-aligned metrics:
   - **`accuracy`** — fraction of cases matching `expected`. The
     least-load-bearing metric: frontier models will score high on
     canonical polynomial equivalence.
   - **`refusal_correctness`** — fraction of `expected="refused"`
     cases the provider actually refused. CORE hits 100% by lane-gate
     construction; frontier models typically confabulate.
   - **`determinism`** — structural assertion (CORE byte-equal across
     runs; frontier varies). Numeric measurement requires multiple
     cached runs; the schema reserves the field.

### Verdict-parser discipline

The free-text-to-closed-vocab boundary lives in
`parse_provider_verdict`. It is **conservative**: ambiguous or
sentinel-free replies collapse to `refused`. A polarized verdict is
never confabulated from prose. Chain-of-thought replies that mention
multiple sentinel tokens use last-token-wins (provider deliberates,
then concludes). This is the load-bearing seam that prevents the
runner from manufacturing scores the provider didn't actually
deliver.

---

## Invariants

- **`citations_dated`** — every citation has `source_date` matching
  `YYYY-MM-DD` and `source_url` matching `https?://`.
- **`citations_three_vendors`** — Anthropic, OpenAI, and Google all
  represented in `ADJACENT_BENCHMARK_CITATIONS`.
- **`scope_disclaimer_present`** — `comparison.json` contains the
  non-empty `scope_disclaimer` documenting B1's scope vs the cited
  benchmarks.
- **`verdict_parser_conservative`** — ambiguous replies collapse to
  `refused`, never to a polarized verdict.
- **`responses_cache_replayable`** — repeated runs with the same
  cache produce identical `comparison.json` bytes.
- **`no_live_api_in_tests`** — the test suite never calls a provider
  API; live calls are gated behind the `FRONTIER_<VENDOR>_KEY` env
  var and the `frontier_runner` CLI entry point.

---

## Acceptance evidence

Accepted when:

- `evals/math_symbolic_equivalence/v1/frontier/baselines.py` ships at
  least one citation per major vendor.
- `evals/math_symbolic_equivalence/v1/frontier/frontier_runner.py`
  exposes the three provider adapters with documented env keys and
  cache files.
- `evals/math_symbolic_equivalence/v1/frontier/comparison.py`
  generates a deterministic `comparison.json` carrying the schema
  version, scope disclaimer, CORE score, citations, and (when
  present) head-to-head runs.
- `tests/test_adr_0131_1_F_frontier.py` passes cleanly — 22 tests
  covering citation shape, provider-registry shape, verdict-parser
  conservatism, runner caching, and comparison determinism.
- The comparison JSON is committed at
  `evals/math_symbolic_equivalence/v1/frontier/comparison.json` with
  CORE's 185/0/0 and zero frontier runs cached — that file becomes
  the durable record into which actual head-to-head numbers slot
  deterministically the first time a `FRONTIER_*_KEY` is exported.

---

## Consequences

- B1 (the first leg of the ADR-0131 composite gate) satisfies the
  Obligation-#7 frontier-pairing requirement *without* claiming
  numbers not yet measured.
- The architecture-aligned differentiator (refusal correctness,
  determinism) is foregrounded by the comparison schema instead of
  raw accuracy — preserves the post-GSM8K-arc honest framing.
- The harness is reusable. When B2 (ADR-0131.2) and B3 (ADR-0131.3)
  reach this maturity, their lanes get a near-identical
  `frontier/` subdirectory; the only per-lane bits are the prompt
  template and the cache directory.
- Running with a real key (e.g. `FRONTIER_ANTHROPIC_KEY=...`)
  produces durable evidence — cached per-case provider responses
  joined to CORE's lane result — that the math-expert promotion
  claim can cite. The audit trail is the JSONL cache file, not a
  hand-curated summary.

---

## Out of scope

- Running CORE against any published math benchmark (e.g. MATH-500)
  — reserved for the per-lane sealed-holdout pattern from
  ADR-0131.1.S.
- Multi-run determinism *measurement* for frontier models (the
  schema reserves the field; the harness doesn't yet score it).
- Live API spending policy — the user controls API keys; the harness
  refuses gracefully when keys are absent.
- B2 and B3 frontier-baseline harnesses — left for follow-up ADRs
  once their lanes reach v1.B maturity.