core/docs/decisions/ADR-0119.4-frontier-baseline-comparison.md

ADR-0119.4 — GSM8K Math: Frontier-Baseline Comparison (ADR-0114a §Obligation #7)

Status: Accepted Date: 2026-05-22 Author: CORE agents + reviewers Amends: ADR-0119 Depends on: ADR-0045, ADR-0114a, ADR-0119

---

Context

ADR-0114a (§Obligation #7) requires that every capability lane pair CORE's capability score with at least one frontier-LLM baseline from a frozen citation. This is designed to provide a capabilities comparison without live API reliance, as established in ADR-0045.

For the gsm8k_math lane, the dev and public splits (200 CORE-original cases total) were introduced under ADR-0119.2, and the lane runner under ADR-0119.3. The frontier-baseline comparison provides the comparative context by pairing published GSM8K scores from frontier vendors with CORE's public split score, alongside the explicit caveat that frontier models are evaluated on the actual GSM8K test set rather than CORE's original public split. The actual GSM8K test set is reserved exclusively for the sealed holdout split under ADR-0119.7.

---

Decision

We establish frozen citations for frontier LLM baselines and couple them with CORE's measurement of the public split.

Action Items and Deliverables

1. Frontier Citations File (frontier.json): Includes three latest frontier LLM baseline GSM8K scores (Claude 3.5 Sonnet, GPT-4, and Gemini 1.5 Pro) with verified URLs and metadata.
2. Comparison File (comparison_v1.json): Joins CORE's public split evaluation results (cases_total: 150, correct: 150, wrong: 0, refused: 0) with the copied frontier citations and the disclaimer.
3. Disclaimer Rule: Explicitly notes that frontier scores are from public reports on the actual GSM8K test set while CORE's scores are on its original split. The qualitative differentiator is CORE's zero-wrong (no-confabulation) behavior.

---

Invariants

• citations_dated: Every citation has a source and URL dated to the actual publication.
• urls_valid: Every baseline URL matches http(s) structure and resolves correctly.
• disclaimer_present: The comparison schema contains the non-empty disclaimer addressing the split mismatch and qualitative zero-wrong distinction.
• core_measurement_current: The CORE measurement aligns with the execution of the lane runner against the 150 public cases.

---

Acceptance Evidence

Accepted when:

• frontier.json is populated with Anthropic Claude 3.5 Sonnet, OpenAI GPT-4, and Google Gemini 1.5 Pro.
• comparison_v1.json is generated and carries the current CORE public split counts (150/150 correct).
• The tests in test_adr_0119_4_frontier_baseline.py pass cleanly.

---

Consequences

• The gsm8k_math lane satisfies Obligation #7 of ADR-0114a.
• Readers have clear visibility into how CORE's zero-confabulation design contrasts with the near-perfect but probabilistic scores of frontier models.

---

Out of Scope

• Running CORE against the real GSM8K test set (reserved for the encrypted holdout split under ADR-0119.7).