diff --git a/docs/decisions/ADR-0119.6-depth-curve-harness.md b/docs/decisions/ADR-0119.6-depth-curve-harness.md new file mode 100644 index 00000000..bb243757 --- /dev/null +++ b/docs/decisions/ADR-0119.6-depth-curve-harness.md @@ -0,0 +1,83 @@ +# ADR-0119.6 — GSM8K Math Depth-Curve Measurement Harness + +**Status:** Accepted +**Date:** 2026-05-23 +**Author:** CORE agents + reviewers +**Depends on:** [ADR-0114a](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md), [ADR-0119](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0119-gsm8k-eval-lane-roadmap.md), [ADR-0119.2](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0119.2-gsm8k-eval-corpus-dev-public.md) + +--- + +## Context + +[ADR-0114a](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md) (Obligation #6) requires publishing a compositional-depth vs. accuracy curve for any domain promoting to `expert`. Accuracy as a function of reasoning depth must not decay sharply (a pattern-matcher decays; a reasoning system stays flat within ε). + +With the GSM8K-style eval corpus splits authored ([ADR-0119.2](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0119.2-gsm8k-eval-corpus-dev-public.md)) and the lane runner developed ([ADR-0119.3](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/evals/gsm8k_math/runner.py)), we must ship the measurement harness that implements this curve calculation from a lane report and a case set. This fulfills the measurement-side requirement of Obligation #6. + +--- + +## Decision + +We introduce a measurement harness and CLI hook to calculate and print the depth-vs-correctness curve. + +### Implementation Details + +- **Module:** [depth_curve.py](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/evals/gsm8k_math/scoring/depth_curve.py) +- **Exposed Function:** + ```python + def compute_depth_curve(cases: list[dict], lane_report: LaneReport) -> dict[str, Any] + ``` + Returns: + ```json + { + "buckets": { + "depth_1": {"total": N, "correct": M, "rate": float}, + "depth_2-3": {"total": N, "correct": M, "rate": float}, + "depth_4-5": {"total": N, "correct": M, "rate": float}, + "depth_6-8": {"total": N, "correct": M, "rate": float} + }, + "max_depth": int, + "raw_curve": [ + {"depth": 1, "total": N, "correct": M, "rate": float}, + ... + ] + } + ``` +- **Bucket key format:** `depth_1` for single integers; `depth_2-3`, `depth_4-5`, `depth_6-8` for the documented buckets matching the corpus distribution. +- **Empty bucket safety:** Rate is initialized/defaulted to `0.0` if a bucket has zero cases (preventing NaN or division-by-zero exceptions). +- **CLI hook:** Invoking `python3 -m evals.gsm8k_math.scoring.depth_curve --split dev|public` executes the lane runner on the chosen split and prints the resulting JSON. + +--- + +## Invariants + +- **Determinism:** Given the same cases and lane report, `compute_depth_curve` outputs byte-identical results. +- **Completeness:** The sum of totals in `buckets` equals the total number of cases. +- **Correctness:** The sum of correct cases in `buckets` equals the overall correct case count in the lane report. + +--- + +## Acceptance Evidence + +Accepted when: +- The module [depth_curve.py](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/evals/gsm8k_math/scoring/depth_curve.py) is present and functional. +- The tests in [test_adr_0119_6_depth_curve.py](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/tests/test_adr_0119_6_depth_curve.py) pass cleanly, validating: + - Determinism across runs. + - Bucket totals sum to total cases. + - Bucket correct counts sum to total correct. + - Empty bucket safety (no NaN/exceptions, rate defaults to `0.0`). + - Correct totals when evaluated against the public split (150 cases total, bucket distribution 15/45/45/45). + - Correct totals on combined splits matching the documented 20/60/60/60 distribution. + - Successful execution of the curve flatness ratio calculation. + +--- + +## Consequences + +- The GSM8K-math evaluation lane now has a standard, deterministic measurement harness to construct the depth curve required by [ADR-0114a](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md). +- The measurement-side of Obligation #6 is fully discharged for GSM8K-math. + +--- + +## Out of Scope + +- Setting a specific numeric threshold ε for curve flatness (reserved for the promotion contract [ADR-0120](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0120-expert-promotion-contract.md)). diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 11c21ed4..24aea5ed 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -50,6 +50,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt | [ADR-0119.1](ADR-0119.1-sealed-holdout-fabrication-control.md) | Seal `fabrication_control` Holdout (ADR-0105 Amendment) | Accepted (2026-05-23) | | [ADR-0119.2](ADR-0119.2-gsm8k-eval-corpus-dev-public.md) | GSM8K Eval Corpus Dev/Public Splits | Accepted (2026-05-22) | | [ADR-0119.4](ADR-0119.4-frontier-baseline-comparison.md) | GSM8K Math: Frontier-Baseline Comparison (ADR-0114a §Obligation #7) | Accepted (2026-05-22) | +| [ADR-0119.6](ADR-0119.6-depth-curve-harness.md) | GSM8K Math Depth-Curve Measurement Harness | Accepted (2026-05-23) | --- @@ -90,12 +91,12 @@ The ADR-0091..0114 slate is fully accepted (0091..0113) plus one proposed-roadma - `symbolic_logic` Lane-Shape Remap (ADR-0109 amendment) — ADR-0123 - `systems_software` Audit-Passed Promotion (third successful) — ADR-0124 - `all_three_pass_rate` Synonym in `inference_shape` (ADR-0109 Amendment) — ADR-0123a -<<<<<<< HEAD - Reasoning-Isolation Perturbation Suite (224 deterministic applicable semantic perturbations; discharges ADR-0114a obligation #5 for the GSM8K-style parser dev lane) — ADR-0125 - GSM8K Eval Lane Roadmap (Phase 5; decomposes into 5.1..5.8; proposed) — ADR-0119 - Seal `fabrication_control` Holdout (ADR-0105 Amendment) — ADR-0119.1 - GSM8K Eval Corpus Dev/Public Splits (200 CORE-original cases; verify.py 200/200; sealed holdout placeholder reserved for ADR-0119.7) — ADR-0119.2 - GSM8K Math: Frontier-Baseline Comparison (citations for Claude 3.5, GPT-4, Gemini 1.5; comparison_v1.json; discharges ADR-0114a §Obligation #7) — ADR-0119.4 +- GSM8K Math Depth-Curve Measurement Harness (discharges ADR-0114a Obligation #6 measurement-side) — ADR-0119.6 ADR-0080 has also landed: Contemplation Loop Phase 1 adds a read-only frontier-compare miner that emits `SPECULATIVE` findings only. diff --git a/evals/gsm8k_math/scoring/__init__.py b/evals/gsm8k_math/scoring/__init__.py new file mode 100644 index 00000000..0dec2549 --- /dev/null +++ b/evals/gsm8k_math/scoring/__init__.py @@ -0,0 +1 @@ +# Empty package marker diff --git a/evals/gsm8k_math/scoring/depth_curve.py b/evals/gsm8k_math/scoring/depth_curve.py new file mode 100644 index 00000000..edcd7def --- /dev/null +++ b/evals/gsm8k_math/scoring/depth_curve.py @@ -0,0 +1,146 @@ +"""ADR-0119.6 — GSM8K math depth-curve measurement harness. + +Buckets the correct rate of a lane run by reasoning depth +(number of operations in the ground-truth graph). +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any + +# Ensure workspace root is in sys.path when running as script +sys.path.insert(0, str(Path(__file__).resolve().parents[3])) + +from evals.gsm8k_math.runner import LaneReport, run_lane + + +def compute_depth_curve(cases: list[dict[str, Any]], lane_report: LaneReport) -> dict[str, Any]: + """Pure, deterministic function bucketizing cases by reasoning depth. + + Reads the case list and a LaneReport, buckets cases by reasoning depth, + and returns a depth curve dictionary. + """ + # Extract outcomes by case_id from the LaneReport + outcomes = {detail["case_id"]: detail["outcome"] for detail in lane_report.case_details} + + # Initialize buckets with 0.0 rates (Obligation #6 empty bucket safety) + bucket_keys = ["depth_1", "depth_2-3", "depth_4-5", "depth_6-8"] + buckets: dict[str, dict[str, Any]] = { + k: {"total": 0, "correct": 0, "rate": 0.0} for k in bucket_keys + } + + # Extract depths of all cases + case_depths = {} + max_depth = 0 + for case in cases: + case_id = case["id"] + gt_graph = case.get("ground_truth_graph", {}) + if isinstance(gt_graph, dict): + operations = gt_graph.get("operations", []) + else: + operations = getattr(gt_graph, "operations", []) + depth = len(operations) + case_depths[case_id] = depth + if depth > max_depth: + max_depth = depth + + # Initialize raw_curve mapping for depths 1 to max_depth + raw_depths = {d: {"total": 0, "correct": 0} for d in range(1, max_depth + 1)} + + for case in cases: + case_id = case["id"] + depth = case_depths[case_id] + outcome = outcomes.get(case_id, "refused") + + is_correct = 1 if outcome == "correct" else 0 + + # Resolve bucket key + if depth == 1: + b_key = "depth_1" + elif 2 <= depth <= 3: + b_key = "depth_2-3" + elif 4 <= depth <= 5: + b_key = "depth_4-5" + elif 6 <= depth <= 8: + b_key = "depth_6-8" + else: + b_key = f"depth_{depth}" + + if b_key not in buckets: + buckets[b_key] = {"total": 0, "correct": 0, "rate": 0.0} + + buckets[b_key]["total"] += 1 + buckets[b_key]["correct"] += is_correct + + if depth not in raw_depths: + raw_depths[depth] = {"total": 0, "correct": 0} + raw_depths[depth]["total"] += 1 + raw_depths[depth]["correct"] += is_correct + + # Compute rates + for k, v in buckets.items(): + if v["total"] > 0: + v["rate"] = float(v["correct"] / v["total"]) + else: + v["rate"] = 0.0 + + raw_curve = [] + for d in sorted(raw_depths.keys()): + v = raw_depths[d] + rate = float(v["correct"] / v["total"]) if v["total"] > 0 else 0.0 + raw_curve.append({ + "depth": d, + "total": v["total"], + "correct": v["correct"], + "rate": rate + }) + + return { + "buckets": buckets, + "max_depth": max_depth, + "raw_curve": raw_curve + } + + +def main() -> None: + parser = argparse.ArgumentParser(description="Run GSM8K math lane and print depth curve.") + parser.add_argument( + "--split", + choices=["dev", "public"], + required=True, + help="The corpus split to run against.", + ) + args = parser.parse_args() + + root = Path(__file__).resolve().parents[3] + if args.split == "dev": + case_path = root / "evals/gsm8k_math/dev/cases.jsonl" + else: + case_path = root / "evals/gsm8k_math/public/v1/cases.jsonl" + + if not case_path.exists(): + print(f"Error: case file not found at {case_path}", file=sys.stderr) + sys.exit(1) + + cases = [] + with open(case_path, "r", encoding="utf-8") as f: + for line in f: + if line.strip(): + cases.append(json.loads(line)) + + # Run the lane runner to get the LaneReport + lane_report = run_lane(cases) + + # Compute depth curve + curve = compute_depth_curve(cases, lane_report) + + # Print the JSON output + print(json.dumps(curve, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/tests/test_adr_0119_6_depth_curve.py b/tests/test_adr_0119_6_depth_curve.py new file mode 100644 index 00000000..c2f0606e --- /dev/null +++ b/tests/test_adr_0119_6_depth_curve.py @@ -0,0 +1,130 @@ +"""Tests for ADR-0119.6 depth-curve harness.""" + +from __future__ import annotations + +import json +from pathlib import Path +import pytest + +from evals.gsm8k_math.runner import run_lane +from evals.gsm8k_math.scoring.depth_curve import compute_depth_curve + +_REPO_ROOT = Path(__file__).resolve().parent.parent + + +def _load_jsonl(path: Path) -> list[dict]: + return [json.loads(line) for line in path.read_text().splitlines() if line.strip()] + + +_DEV_CASES = _load_jsonl(_REPO_ROOT / "evals" / "gsm8k_math" / "dev" / "cases.jsonl") +_PUBLIC_CASES = _load_jsonl(_REPO_ROOT / "evals" / "gsm8k_math" / "public" / "v1" / "cases.jsonl") + + +def test_determinism() -> None: + """Pins (a) Determinism: same inputs -> same JSON output.""" + report1 = run_lane(_DEV_CASES) + curve1 = compute_depth_curve(_DEV_CASES, report1) + + report2 = run_lane(_DEV_CASES) + curve2 = compute_depth_curve(_DEV_CASES, report2) + + # Convert to JSON string to assert exact byte-equal structures + json1 = json.dumps(curve1, sort_keys=True) + json2 = json.dumps(curve2, sort_keys=True) + assert json1 == json2 + + +def test_bucket_totals_sum_to_total_cases() -> None: + """Pins (b) Bucket totals sum to lane_report.metrics['cases_total'].""" + report = run_lane(_DEV_CASES) + curve = compute_depth_curve(_DEV_CASES, report) + + bucket_total_sum = sum(b["total"] for b in curve["buckets"].values()) + assert bucket_total_sum == report.metrics["cases_total"] + + +def test_bucket_correct_sums_match_report() -> None: + """Pins (c) Sum of (bucket.correct) == lane_report.metrics['correct'].""" + report = run_lane(_DEV_CASES) + curve = compute_depth_curve(_DEV_CASES, report) + + bucket_correct_sum = sum(b["correct"] for b in curve["buckets"].values()) + assert bucket_correct_sum == report.metrics["correct"] + + +def test_empty_bucket_rate_is_zero() -> None: + """Pins (d) Empty bucket -> rate is 0.0 (not NaN, not exception).""" + # Filter cases to only those with depth 1 + depth_1_cases = [ + c for c in _DEV_CASES + if len(c.get("ground_truth_graph", {}).get("operations", [])) == 1 + ] + assert len(depth_1_cases) > 0, "No depth-1 cases found in dev set" + + report = run_lane(depth_1_cases) + curve = compute_depth_curve(depth_1_cases, report) + + # Buckets depth_2-3, depth_4-5, depth_6-8 should be empty and have 0.0 rate + for k in ["depth_2-3", "depth_4-5", "depth_6-8"]: + assert curve["buckets"][k]["total"] == 0 + assert curve["buckets"][k]["correct"] == 0 + assert curve["buckets"][k]["rate"] == 0.0 + + +def test_public_split_totals() -> None: + """Pins (e) Run against the public split (150 cases); confirm bucket totals.""" + report = run_lane(_PUBLIC_CASES) + curve = compute_depth_curve(_PUBLIC_CASES, report) + + # Check buckets covered: 1, 2-3, 4-5, 6-8 + expected_buckets = {"depth_1", "depth_2-3", "depth_4-5", "depth_6-8"} + assert set(curve["buckets"].keys()) == expected_buckets + + # Per-bucket totals on public alone: each bucket has the public-only contribution (15/45/45/45) + assert curve["buckets"]["depth_1"]["total"] == 15 + assert curve["buckets"]["depth_2-3"]["total"] == 45 + assert curve["buckets"]["depth_4-5"]["total"] == 45 + assert curve["buckets"]["depth_6-8"]["total"] == 45 + + +def test_total_corpus_distribution() -> None: + """Pins (e) Confirm total combined corpus distribution matches 20/60/60/60.""" + all_cases = _DEV_CASES + _PUBLIC_CASES + report = run_lane(all_cases) + curve = compute_depth_curve(all_cases, report) + + assert curve["buckets"]["depth_1"]["total"] == 20 + assert curve["buckets"]["depth_2-3"]["total"] == 60 + assert curve["buckets"]["depth_4-5"]["total"] == 60 + assert curve["buckets"]["depth_6-8"]["total"] == 60 + + +def test_curve_flatness_ratio() -> None: + """Pins (f) Verify curve flatness calculation: compute ratio without enforcing threshold.""" + report = run_lane(_DEV_CASES) + curve = compute_depth_curve(_DEV_CASES, report) + + max_depth = curve["max_depth"] + if max_depth == 1: + max_bucket_key = "depth_1" + elif 2 <= max_depth <= 3: + max_bucket_key = "depth_2-3" + elif 4 <= max_depth <= 5: + max_bucket_key = "depth_4-5" + elif 6 <= max_depth <= 8: + max_bucket_key = "depth_6-8" + else: + max_bucket_key = f"depth_{max_depth}" + + max_depth_rate = curve["buckets"][max_bucket_key]["rate"] + depth_1_rate = curve["buckets"]["depth_1"]["rate"] + + # Compute ratio + if depth_1_rate > 0.0: + ratio = max_depth_rate / depth_1_rate + else: + ratio = 0.0 + + # The test confirms that the harness CAN compute and report the ratio + assert isinstance(ratio, float) + assert ratio >= 0.0