57 lines
5.2 KiB
Markdown
57 lines
5.2 KiB
Markdown
# ADR-0119.4 — GSM8K Math: Frontier-Baseline Comparison (ADR-0114a §Obligation #7)
|
|
|
|
**Status:** Accepted
|
|
**Date:** 2026-05-22
|
|
**Author:** CORE agents + reviewers
|
|
**Amends:** [ADR-0119](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/docs/decisions/ADR-0119-gsm8k-eval-lane-roadmap.md)
|
|
**Depends on:** [ADR-0045](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/docs/decisions/ADR-0045-long-context-recall-vs-transformer-baselines.md), [ADR-0114a](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md), [ADR-0119](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/docs/decisions/ADR-0119-gsm8k-eval-lane-roadmap.md)
|
|
|
|
---
|
|
|
|
## Context
|
|
|
|
[ADR-0114a](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md) (§Obligation #7) requires that every capability lane pair CORE's capability score with at least one frontier-LLM baseline from a frozen citation. This is designed to provide a capabilities comparison without live API reliance, as established in [ADR-0045](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/docs/decisions/ADR-0045-long-context-recall-vs-transformer-baselines.md).
|
|
|
|
For the `gsm8k_math` lane, the dev and public splits (200 CORE-original cases total) were introduced under [ADR-0119.2](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/docs/decisions/ADR-0119.2-gsm8k-eval-corpus-dev-public.md), and the lane runner under [ADR-0119.3](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/docs/decisions/ADR-0119-gsm8k-eval-lane-roadmap.md). The frontier-baseline comparison provides the comparative context by pairing published GSM8K scores from frontier vendors with CORE's public split score, alongside the explicit caveat that frontier models are evaluated on the actual GSM8K test set rather than CORE's original public split. The actual GSM8K test set is reserved exclusively for the sealed holdout split under [ADR-0119.7](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/docs/decisions/ADR-0119-gsm8k-eval-lane-roadmap.md).
|
|
|
|
---
|
|
|
|
## Decision
|
|
|
|
We establish frozen citations for frontier LLM baselines and couple them with CORE's measurement of the public split.
|
|
|
|
### Action Items and Deliverables
|
|
1. **Frontier Citations File ([frontier.json](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/evals/gsm8k_math/baselines/frontier.json)):** Includes three latest frontier LLM baseline GSM8K scores (Claude 3.5 Sonnet, GPT-4, and Gemini 1.5 Pro) with verified URLs and metadata.
|
|
2. **Comparison File ([comparison_v1.json](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/evals/gsm8k_math/baselines/comparison_v1.json)):** Joins CORE's public split evaluation results (cases_total: 150, correct: 150, wrong: 0, refused: 0) with the copied frontier citations and the disclaimer.
|
|
3. **Disclaimer Rule:** Explicitly notes that frontier scores are from public reports on the actual GSM8K test set while CORE's scores are on its original split. The qualitative differentiator is CORE's zero-wrong (no-confabulation) behavior.
|
|
|
|
---
|
|
|
|
## Invariants
|
|
|
|
- **`citations_dated`:** Every citation has a source and URL dated to the actual publication.
|
|
- **`urls_valid`:** Every baseline URL matches `http(s)` structure and resolves correctly.
|
|
- **`disclaimer_present`:** The comparison schema contains the non-empty disclaimer addressing the split mismatch and qualitative zero-wrong distinction.
|
|
- **`core_measurement_current`:** The CORE measurement aligns with the execution of the lane runner against the 150 public cases.
|
|
|
|
---
|
|
|
|
## Acceptance Evidence
|
|
|
|
Accepted when:
|
|
- [frontier.json](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/evals/gsm8k_math/baselines/frontier.json) is populated with Anthropic Claude 3.5 Sonnet, OpenAI GPT-4, and Google Gemini 1.5 Pro.
|
|
- [comparison_v1.json](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/evals/gsm8k_math/baselines/comparison_v1.json) is generated and carries the current CORE public split counts (150/150 correct).
|
|
- The tests in [test_adr_0119_4_frontier_baseline.py](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/tests/test_adr_0119_4_frontier_baseline.py) pass cleanly.
|
|
|
|
---
|
|
|
|
## Consequences
|
|
|
|
- The `gsm8k_math` lane satisfies Obligation #7 of [ADR-0114a](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md).
|
|
- Readers have clear visibility into how CORE's zero-confabulation design contrasts with the near-perfect but probabilistic scores of frontier models.
|
|
|
|
---
|
|
|
|
## Out of Scope
|
|
|
|
- Running CORE against the real GSM8K test set (reserved for the encrypted holdout split under [ADR-0119.7](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-4-frontier-baseline/docs/decisions/ADR-0119-gsm8k-eval-lane-roadmap.md)).
|