core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/docs/decisions/ADR-0131.1.F-frontier-baseline-comparison.md
Shay 24f6a596fe
feat(ADR-0131.1.F): frontier-baseline comparison harness for B1 (#178) 
* feat(ADR-0131.1.F): frontier-baseline comparison harness for B1

Adapts the ADR-0119.4 methodology (frozen citations + comparison JSON
with disclaimer) to B1, with three additions for the
architecture-aligned claim:

1. A provider-agnostic live head-to-head runner. Adapters for
   Anthropic / OpenAI / Google import their SDKs lazily so the
   package loads cleanly without them installed. Each provider has a
   documented FRONTIER_<VENDOR>_KEY env var; the runner refuses with
   a typed FrontierRunError when keys are absent and the cache cannot
   cover all cases. Every response is cached one-record-per-line at
   responses/<provider>/<model>.jsonl so subsequent runs replay
   byte-equally without re-calling the API.

2. A conservative free-text-to-closed-vocab verdict parser. Ambiguous
   or sentinel-free provider replies collapse to "refused" — a
   polarized verdict is never confabulated from prose. Chain-of-
   thought replies use last-token-wins (provider deliberates, then
   concludes). This is the load-bearing seam that prevents the
   runner from manufacturing scores the provider didn't deliver.

3. Architecture-aligned comparison metrics. accuracy is reported but
   foregrounded as the least-load-bearing; refusal_correctness
   (CORE 100% by lane-gate construction vs. frontier confabulation
   rate) and determinism (CORE byte-equal vs. frontier variance) are
   the differentiators.

Frozen adjacent-benchmark citations cover Anthropic
(claude-3-5-sonnet on MATH, claude-opus-4-1 on AIME), OpenAI
(gpt-4o on MATH), and Google (gemini-1.5-pro on MATH). The scope
disclaimer documents that these are adjacent, not head-to-head.
Head-to-head numbers, when run, land in the cache; the comparison
JSON joins them with CORE's existing lane result.

22 tests pin the methodology: citation shape (every field, https
URL, YYYY-MM-DD date), provider-registry shape, verdict-parser
conservatism (multiple chain-of-thought cases), runner caching
behavior (no double-invoke), comparison-JSON determinism (byte-equal
across runs).

No live API call at test time. The harness gates real runs behind
explicit env vars + CLI invocation.

Composes with ADR-0131.1 (B1 v1), ADR-0131.1.B (v1.B hardening,
#169), ADR-0131.1.S (sealed holdout, #173).

* feat(ADR-0131.1.F): live head-to-head — anthropic/claude-sonnet-4-6

First real frontier baseline on the full B1.B 185-case set
(curated + generated). Cached one-record-per-line at
responses/anthropic/claude-sonnet-4-6.jsonl. Re-runs replay from
disk; no further API calls.

Headline (after scoring fix):

  CORE                            185/185 = 100.0% accuracy
                                  3/3     = 100.0% refusal_correctness
                                  deterministic (byte-equal across runs)

  anthropic/claude-sonnet-4-6     182/185 = 98.4%  accuracy
                                  1/3     = 33.3%  refusal_correctness
                                  non-deterministic (temperature=0, but
                                  not byte-equal architecturally)

The 1.6pp accuracy gap is informative; the refusal-correctness gap
is the architecture-aligned story. Sonnet's three misses:

  sym-eq-v1-0016 [difference_of_squares]
    (x^2 + 1)*(x^2 - 1) vs x^4 - 1
    Sonnet: NOT_EQUIVALENT (math error on a textbook identity)

  sym-eq-gen-v1-0153 [generated_refusal_function]
    sin(x) vs x
    Sonnet: NOT_EQUIVALENT (confabulated — should refuse,
                            transcendental outside polynomial scope)

  sym-eq-gen-v1-0154 [generated_refusal_negative_exponent]
    x^-1 vs 1
    Sonnet: NOT_EQUIVALENT (confabulated — should refuse,
                            negative exponent outside scope)

Sonnet correctly refused only on syntactically malformed input
("x +"); on syntactically-valid-but-semantically-out-of-scope inputs
it confidently polarized rather than refusing. CORE refuses both
classes with typed reasons.

Scoring fix: comparison.py now composes curated + generated cases
(mirroring runner.py) so the head-to-head scores the full 185-case
lane, not just the 30 curated. The initial run scored only 30/185
because the generated set was not loaded into _load_cases().

22/22 frontier-methodology tests still pass.

* feat(ADR-0131.1.F): three more head-to-head runs + Ollama adapter

Three additional providers ran against the full B1.B 185-case set,
joining the prior claude-sonnet-4-6 result:

  CORE                           185/185 = 100.0% acc | 3/3 = 100%  refusal | 33 ms
  claude-sonnet-4-6              182/185 =  98.4% acc | 1/3 = 33.3% refusal | 294 s
  claude-opus-4-7                178/185 =  96.2% acc | 1/3 = 33.3% refusal | 309 s
  gpt-5                          134/185 =  72.4% acc | 1/3 = 33.3% refusal | 1153 s
  qwen3:8b (M1 local, partial)    91/91  = 100.0% acc | n/a  no refusal-class | killed

CORE is the only system at 100% on both axes, and runs ~9,000×
faster than the cheapest cloud frontier, ~35,000× faster than gpt-5,
and finishes in less wall time than a single API call to any of the
three frontier models.

Three distinct frontier brittleness modes, all rooted in
"not actually canonicalizing":

  - sonnet-4-6 confabulates polarized verdicts on out-of-scope
    inputs (sin(x), x^-1). Misses one in-scope difference-of-squares
    identity (x^2+1)*(x^2-1) vs x^4-1.
  - opus-4-7 pattern-shortcuts five near-miss-constant cases —
    accepts (-x+3)*(4x+1) == -4x^2+11x+4 (correct constant is 3,
    not 4) without expanding. Same two out-of-scope confabulations
    as sonnet.
  - gpt-5 over-refuses 50 in-scope cases — literally replies
    "REFUSED" to x*(x+1) == x^2+x and (x+1)*(x-1) == x^2-1. Same
    two out-of-scope confabulations as sonnet/opus.

The qwen3:8b partial is the surprise: on the 91 in-scope cases it
completed (spanning the categories where the frontier models failed),
it scored 100%. Refusal-class cases weren't reached before the run
was killed for being impractically slow (~22s/case on M1).

Changes in this commit:

  - frontier_runner.py: anthropic adapter now omits ``temperature``
    for claude-opus-4-x (the parameter is rejected by 4.x models);
    openai adapter switches to ``max_completion_tokens`` for the
    gpt-5 / o-series reasoning models; new ``_ollama_invoke`` that
    posts to localhost:11434 with no third-party dep; per-case
    ``latency_ms`` is now captured on every NEW cached response
    (future runs only — these four runs pre-date the patch).
  - comparison.py: ``_load_cases`` composes curated + generated
    (185 cases) instead of curated only; ``_score_provider``
    surfaces ``latency_summary`` when records carry latency_ms.
  - tests: provider-registry test relaxed to "cloud trio is a
    subset of PROVIDERS"; env-key test allows ``_KEY`` (cloud
    secret) or ``_URL`` (local endpoint).
2026-05-23 12:14:06 -07:00

7.6 KiB
Raw Permalink Blame History

ADR-0131.1.F — B1 Symbolic Equivalence: Frontier-Baseline Comparison

Status: Proposed Date: 2026-05-23 Author: CORE agents + reviewers Parent: ADR-0131 Depends on: ADR-0045, ADR-0114a, ADR-0119.4, ADR-0131.1

Context

ADR-0131 re-targeted the math-expert promotion away from GSM8K to a composite gate of three architecture-aligned benchmarks. ADR-0131.1 shipped Benchmark 1 (symbolic equivalence v1, 30/30 wrong=0); ADR-0131.1.B hardened it (185/185 wrong=0); ADR-0131.1.S sealed a 14-case holdout under pyrage X25519 to make B1's score externally credible.

ADR-0114a §Obligation #7 requires every capability lane to pair its CORE score with at least one frontier-LLM baseline. ADR-0119.4 established the methodology for the (now-deferred) gsm8k_math lane: frozen citations + a CORE-vs-frontier comparison JSON with an explicit disclaimer about scope mismatch. This ADR adapts that methodology to B1.

The challenge for B1 specifically: univariate polynomial canonical equivalence is not a standard published benchmark, so there are no direct frontier scores to cite. Two responses:

  1. Adjacent-benchmark citations. Frozen scores from MATH (Hendrycks et al. 2021), MATH-500, MMLU mathematics, AIME, etc. give the published-context anchor without claiming head-to-head numbers.
  2. Live head-to-head, deterministically cached. A provider-agnostic runner queries Anthropic, OpenAI, and Google on the same B1 dataset with the same deterministic prompt, parses replies into the closed CORE verdict vocabulary (equivalent / not_equivalent / refused), and caches every response so that subsequent runs replay byte-equally without re-calling the API.

Both contexts compose into a single comparison.json artifact. The ADR pins the methodology before any head-to-head numbers are recorded, so the numbers — when they land — cannot be retrofit.

Decision

Ship a frontier-baseline harness for B1 with three deliverables:

Action items

  1. Adjacent-benchmark citations (baselines.py). Frozen ADJACENT_BENCHMARK_CITATIONS tuple containing entries from Anthropic, OpenAI, and Google on published math benchmarks. Each citation has vendor, model, benchmark, score, metric, source_url, source_date, note. URLs are validated for https?:// shape; dates for YYYY-MM-DD shape. The note field carries the scope caveat explicitly per citation.

  2. Provider-agnostic runner (frontier_runner.py). Three adapters (Anthropic / OpenAI / Google), each importing its SDK lazily so the package loads cleanly without the SDKs installed. Each provider has a documented FRONTIER_<VENDOR>_KEY env var; the runner refuses with a typed FrontierRunError if the key is absent and the cache cannot cover all cases. Responses are cached one-record-per-line at evals/math_symbolic_equivalence/v1/frontier/responses/<provider>/<model>.jsonl.

  3. Comparison composer (comparison.py). Joins CORE's report.json, the cached provider responses, and the frozen citations into one deterministic comparison.json. Scoring emphasizes three architecture-aligned metrics:

    • accuracy — fraction of cases matching expected. The least-load-bearing metric: frontier models will score high on canonical polynomial equivalence.
    • refusal_correctness — fraction of expected="refused" cases the provider actually refused. CORE hits 100% by lane-gate construction; frontier models typically confabulate.
    • determinism — structural assertion (CORE byte-equal across runs; frontier varies). Numeric measurement requires multiple cached runs; the schema reserves the field.

Verdict-parser discipline

The free-text-to-closed-vocab boundary lives in parse_provider_verdict. It is conservative: ambiguous or sentinel-free replies collapse to refused. A polarized verdict is never confabulated from prose. Chain-of-thought replies that mention multiple sentinel tokens use last-token-wins (provider deliberates, then concludes). This is the load-bearing seam that prevents the runner from manufacturing scores the provider didn't actually deliver.

Invariants

  • citations_dated — every citation has source_date matching YYYY-MM-DD and source_url matching https?://.
  • citations_three_vendors — Anthropic, OpenAI, and Google all represented in ADJACENT_BENCHMARK_CITATIONS.
  • scope_disclaimer_present — comparison.json contains the non-empty scope_disclaimer documenting B1's scope vs the cited benchmarks.
  • verdict_parser_conservative — ambiguous replies collapse to refused, never to a polarized verdict.
  • responses_cache_replayable — repeated runs with the same cache produce identical comparison.json bytes.
  • no_live_api_in_tests — the test suite never calls a provider API; live calls are gated behind the FRONTIER_<VENDOR>_KEY env var and the frontier_runner CLI entry point.

Acceptance evidence

Accepted when:

  • evals/math_symbolic_equivalence/v1/frontier/baselines.py ships at least one citation per major vendor.
  • evals/math_symbolic_equivalence/v1/frontier/frontier_runner.py exposes the three provider adapters with documented env keys and cache files.
  • evals/math_symbolic_equivalence/v1/frontier/comparison.py generates a deterministic comparison.json carrying the schema version, scope disclaimer, CORE score, citations, and (when present) head-to-head runs.
  • tests/test_adr_0131_1_F_frontier.py passes cleanly — 22 tests covering citation shape, provider-registry shape, verdict-parser conservatism, runner caching, and comparison determinism.
  • The comparison JSON is committed at evals/math_symbolic_equivalence/v1/frontier/comparison.json with CORE's 185/0/0 and zero frontier runs cached — that file becomes the durable record into which actual head-to-head numbers slot deterministically the first time a FRONTIER_*_KEY is exported.

Consequences

  • B1 (the first leg of the ADR-0131 composite gate) satisfies the Obligation-#7 frontier-pairing requirement without claiming numbers not yet measured.
  • The architecture-aligned differentiator (refusal correctness, determinism) is foregrounded by the comparison schema instead of raw accuracy — preserves the post-GSM8K-arc honest framing.
  • The harness is reusable. When B2 (ADR-0131.2) and B3 (ADR-0131.3) reach this maturity, their lanes get a near-identical frontier/ subdirectory; the only per-lane bits are the prompt template and the cache directory.
  • Running with a real key (e.g. FRONTIER_ANTHROPIC_KEY=...) produces durable evidence — cached per-case provider responses joined to CORE's lane result — that the math-expert promotion claim can cite. The audit trail is the JSONL cache file, not a hand-curated summary.

Out of scope

  • Running CORE against any published math benchmark (e.g. MATH-500) — reserved for the per-lane sealed-holdout pattern from ADR-0131.1.S.
  • Multi-run determinism measurement for frontier models (the schema reserves the field; the harness doesn't yet score it).
  • Live API spending policy — the user controls API keys; the harness refuses gracefully when keys are absent.
  • B2 and B3 frontier-baseline harnesses — left for follow-up ADRs once their lanes reach v1.B maturity.