core-labs/core
core-labs/core
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity
core/docs/sessions/gsm8k-train-proxy-rebaseline-2026-06-20.md
Shay 2dc1462b40
chore(evals): ratify GSM8K train proxy rebaseline 6/44/0 → 30/20/0 (#848) 
- evals/gsm8k_math/train_sample/v1/report.json: rebaselined from
  correct=6/refused=44/wrong=0 to correct=30/refused=20/wrong=0.
  wrong_ids == []. exit_criterion.passed: false → true.
  Written deterministically by the canonical runner (write_report /
  json.dumps sort_keys=True).

- docs/sessions/gsm8k-train-proxy-rebaseline-2026-06-20.md: new audit
  note documenting why #846's report update was unauthorized, why #847
  restored the pin, and why this PR is the explicit authorization
  boundary for the committed rebaseline.

- HANDOFF-gpt55-2026-06-20.md: small factual note appended to the
  'Latest continuity pointer' section recording this rebaseline status.

Authorization chain:
  ADR-0126 (exit criterion) →
  #846 kernel changes (produce 30/20/0) →
  #847 reconciliation (restores pin, delegates to this PR) →
  this PR (explicit rebaseline authorization)

Non-goals: no runtime, serving, teaching, capability, or kernel change.
No quantity-entity work. No new eval artifacts. No teaching proposals.
2026-06-20 19:54:21 -07:00

239 lines
8.4 KiB
Markdown
Raw Blame History

# GSM8K Train Proxy Rebaseline — 2026-06-20
**PR:** `chore/evals-ratify-gsm8k-train-proxy-rebaseline`
**Title:** `chore(evals): ratify GSM8K train proxy rebaseline`
**Date:** 2026-06-20
**Author:** Sonnet 4.6 (governance/rebaseline role)
---
## 1. Background: What Happened in #846 and #847
### PR #846 — Unauthorized Rebaseline
PR #846 (`feat(kernel): implement ProblemFrame proportional-change closure`) merged at
`a145b7c3d62ab97726f87c4a711b4b4dafe9f2b6` on 2026-06-21T02:19:07Z.
The PR was a legitimate capability PR delivering proportional-change closure work in
`generate/problem_frame.py`, `generate/problem_frame_builder.py`, and
`generate/problem_frame_contracts.py`. Its kernel and test changes were valid and
authorized.
However, the merge commit also included **two unauthorized artifact-class changes**:
1. **`evals/gsm8k_math/train_sample/v1/report.json`** — rebaselined from `6/44/0` to
`30/20/0`. The live runner output was correct, but committing an updated `report.json`
inside a capability PR violates governance doctrine: the committed proxy baseline is a
sealed artifact that requires a dedicated, explicitly ratified rebaseline PR. No ADR,
session note, or explicit PR section in #846 authorized committing the new counts.
2. **`teaching/proposals/comprehension_failures/34ce9254…json`** — a new teaching proposal
file added without authorization. Committed via a `chore(kernel)` message; no review-gate
passage; contradicted the HANDOFF claim that "teaching/review/proposal paths [were]
untouched".
### PR #847 — Reconciliation
PR #847 (`reconcile: restore eval proxy and remove unauthorized teaching proposal from #846`)
merged at `b6d422834b243073fc748754ad78b1a956774065`.
#847 performed the following narrow reversions:
- Restored `evals/gsm8k_math/train_sample/v1/report.json` to the governed 6/44/0 pin.
- Deleted the unauthorized teaching proposal artifact.
- **Preserved** the valid kernel and test changes from #846.
The rationale: the live runner output (30/20/0) was already correct; only the committed
artifact was wrong. Reverting the artifact does not revert the underlying capability.
---
## 2. Why This PR Is the Explicit Authorization Boundary
The governance doctrine for `report.json` has been enforced consistently across prior
sessions:
- `gsm8k-workstream-a-gate-a1-comparative-multiplicative-lookback-2026-06-17.md`:
"No report.json rebaseline" / "report.json rebaseline only via separate ratified PR"
- `gsm8k-workstream-a-increment-3-rate-followup-ratification-2026-06-17.md`:
"No write of updated report.json in this increment"
- `gsm8k-capability-paradigm-sprint11-lookback-2026-06-17.md`: "report.json untouched"
- `gsm8k-capability-paradigm-sprint12-lookback-2026-06-17.md`: "report.json untouched"
This PR is the **explicit authorization** for the single `report.json` update. It:
- Is opened **after** #847 is merged to main (required base integrity).
- Is based on a **fresh worktree from `origin/main`** at the #847 merge commit
(`b6d422834b243073fc748754ad78b1a956774065`).
- Changes **only** `evals/gsm8k_math/train_sample/v1/report.json` and this session note.
- Does **not** touch any kernel code, tests, packs, policy, identity, recall, vault, field,
algebra, teaching proposals, or serving runtime paths.
---
## 3. Before/After Counts
### Before (governed 6/44/0 pin, restored by #847)
```json
"counts": {
"correct": 6,
"refused": 44,
"wrong": 0
},
"exit_criterion": {
"correct_min": 10,
"passed": false,
"wrong_max": 0
}
```
### After (ratified 30/20/0 rebaseline, this PR)
```json
"counts": {
"correct": 30,
"refused": 20,
"wrong": 0
},
"exit_criterion": {
"correct_min": 10,
"passed": true,
"wrong_max": 0
}
```
**`wrong_ids == []` (sacred invariant: no wrong answers committed)**
---
## 4. Live Runner Verification
The report.json was generated by running the canonical train-sample runner from the fresh
worktree at HEAD (`b6d42283`) immediately before this PR:
```bash
# Verification command (run in core-gsm8k-train-rebaseline worktree):
uv run python - <<'PY'
from evals.gsm8k_math.train_sample.v1.runner import _CASES_PATH, _load_cases, build_report
r = build_report(_load_cases(_CASES_PATH))
print("counts:", r["counts"])
print("wrong_ids:", [x["case_id"] for x in r["per_case"] if x["verdict"] == "wrong"])
print("exit_criterion:", r["exit_criterion"])
PY
# Output:
# counts: {'correct': 30, 'wrong': 0, 'refused': 20}
# wrong_ids: []
# exit_criterion: {'correct_min': 10, 'wrong_max': 0, 'passed': True}
```
The report was then written deterministically using the runner's own `write_report()`:
```bash
uv run python - <<'PY'
from evals.gsm8k_math.train_sample.v1.runner import _CASES_PATH, _load_cases, build_report, write_report, _REPORT_PATH
r = build_report(_load_cases(_CASES_PATH))
write_report(r)
print("Written to:", _REPORT_PATH)
PY
```
The runner uses `json.dumps(report, indent=2, sort_keys=True)`, so the output is
deterministic and key-order is alphabetical throughout.
---
## 5. Scope
**Changed files in this PR:**
| File | Change |
|---|---|
| `evals/gsm8k_math/train_sample/v1/report.json` | Rebaselined from 6/44/0 to 30/20/0 |
| `docs/sessions/gsm8k-train-proxy-rebaseline-2026-06-20.md` | This audit note (new) |
**No other files changed.**
---
## 6. Non-Goals
This PR explicitly does **not**:
- Change any runtime code, serving path, or derivation organ.
- Add or modify any test.
- Touch packs, policy, identity, recall, vault, field, or algebra.
- Create any teaching proposal or teaching artifact.
- Start or advance quantity-entity implementation work.
- Rebaseline any other eval artifact (holdout, composition validation, etc.).
- Add any new eval artifact.
The quantity-entity foundational slice implementation is authorized separately by
`docs/sessions/quantity-entity-foundational-slice-authorization-2026-06-20.md` and must
not begin until that document's own gating conditions are met.
---
## 7. Remaining Risk
The primary systemic risk exposed by this sequence is:
> **Report pins must not be casually updated inside implementation or capability PRs.**
The `report.json` proxy baseline is a sealed artifact. Even when the live runner output
is correct and the capability work is authorized, committing the updated report inside
a capability PR:
- Conflates two authorization domains (capability work ≠ rebaseline authorization).
- Makes the committed artifact hard to audit (which PR actually authorized it?).
- Undermines the governed rebaseline PR discipline that has protected wrong-zero invariance
across all prior sessions.
**All future PRs must leave `evals/gsm8k_math/train_sample/v1/report.json` (and all other
`report.json` proxy baselines) untouched unless the PR is explicitly and solely dedicated
to a governed rebaseline.**
---
## 8. Authorization Chain
This rebaseline is authorized by the following chain:
1. **ADR-0126** — establishes the train-sample exit criterion (`correct >= 10`, `wrong == 0`).
2. **#846 kernel changes** — the capability work that produced the 30/20/0 live runner
output (proportional-change closure in ProblemFrame). These changes are valid and merged.
3. **#847 reconciliation** — restored the committed artifact to 6/44/0, explicitly delegating
authorization to a dedicated rebaseline PR (this PR).
4. **This PR** — the explicit, governed authorization to commit the 30/20/0 counts.
The `wrong_ids == []` invariant is preserved: no wrong answer is committed, and the
`wrong_max: 0` gate remains active.
---
## 9. Fresh-Base Verification (Pre-Edit)
```bash
# In /Users/kaizenpro/Projects/core:
git fetch origin --prune
# -> b6d42283 reconcile: restore eval proxy and remove unauthorized teaching proposal from #846 (#847)
git worktree add ../core-gsm8k-train-rebaseline origin/main \
-b chore/evals-ratify-gsm8k-train-proxy-rebaseline
# In new worktree:
echo "HEAD=$(git rev-parse HEAD)"
# HEAD=b6d422834b243073fc748754ad78b1a956774065
echo "ORIGIN_MAIN=$(git rev-parse origin/main)"
# ORIGIN_MAIN=b6d422834b243073fc748754ad78b1a956774065
git status --porcelain
# (empty — clean)
git log --oneline -n 3
# b6d42283 reconcile: restore eval proxy and remove unauthorized teaching proposal from #846 (#847)
# a145b7c3 feat(kernel): implement ProblemFrame proportional-change closure (#846)
# ad0bae29 docs(kernel): authorize quantity-entity foundational implementation slice (#845)
```
HEAD == ORIGIN_MAIN == the #847 merge commit. Pre-edit state verified clean.
Reference in a new issue View git blame Copy permalink