merge origin/main and resolve conflicts

docs/decisions/ADR-0119.6-depth-curve-harness.md

@@ -0,0 +1,83 @@
+# ADR-0119.6 — GSM8K Math Depth-Curve Measurement Harness
+
+**Status:** Accepted
+**Date:** 2026-05-23
+**Author:** CORE agents + reviewers
+**Depends on:** [ADR-0114a](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md), [ADR-0119](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0119-gsm8k-eval-lane-roadmap.md), [ADR-0119.2](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0119.2-gsm8k-eval-corpus-dev-public.md)
+
+---
+
+## Context
+
+[ADR-0114a](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md) (Obligation #6) requires publishing a compositional-depth vs. accuracy curve for any domain promoting to `expert`. Accuracy as a function of reasoning depth must not decay sharply (a pattern-matcher decays; a reasoning system stays flat within ε).
+
+With the GSM8K-style eval corpus splits authored ([ADR-0119.2](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0119.2-gsm8k-eval-corpus-dev-public.md)) and the lane runner developed ([ADR-0119.3](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/evals/gsm8k_math/runner.py)), we must ship the measurement harness that implements this curve calculation from a lane report and a case set. This fulfills the measurement-side requirement of Obligation #6.
+
+---
+
+## Decision
+
+We introduce a measurement harness and CLI hook to calculate and print the depth-vs-correctness curve.
+
+### Implementation Details
+
+- **Module:** [depth_curve.py](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/evals/gsm8k_math/scoring/depth_curve.py)
+- **Exposed Function:**
+  ```python
+  def compute_depth_curve(cases: list[dict], lane_report: LaneReport) -> dict[str, Any]
+  ```
+  Returns:
+  ```json
+  {
+    "buckets": {
+      "depth_1": {"total": N, "correct": M, "rate": float},
+      "depth_2-3": {"total": N, "correct": M, "rate": float},
+      "depth_4-5": {"total": N, "correct": M, "rate": float},
+      "depth_6-8": {"total": N, "correct": M, "rate": float}
+    },
+    "max_depth": int,
+    "raw_curve": [
+      {"depth": 1, "total": N, "correct": M, "rate": float},
+      ...
+    ]
+  }
+  ```
+- **Bucket key format:** `depth_1` for single integers; `depth_2-3`, `depth_4-5`, `depth_6-8` for the documented buckets matching the corpus distribution.
+- **Empty bucket safety:** Rate is initialized/defaulted to `0.0` if a bucket has zero cases (preventing NaN or division-by-zero exceptions).
+- **CLI hook:** Invoking `python3 -m evals.gsm8k_math.scoring.depth_curve --split dev|public` executes the lane runner on the chosen split and prints the resulting JSON.
+
+---
+
+## Invariants
+
+- **Determinism:** Given the same cases and lane report, `compute_depth_curve` outputs byte-identical results.
+- **Completeness:** The sum of totals in `buckets` equals the total number of cases.
+- **Correctness:** The sum of correct cases in `buckets` equals the overall correct case count in the lane report.
+
+---
+
+## Acceptance Evidence
+
+Accepted when:
+- The module [depth_curve.py](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/evals/gsm8k_math/scoring/depth_curve.py) is present and functional.
+- The tests in [test_adr_0119_6_depth_curve.py](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/tests/test_adr_0119_6_depth_curve.py) pass cleanly, validating:
+  - Determinism across runs.
+  - Bucket totals sum to total cases.
+  - Bucket correct counts sum to total correct.
+  - Empty bucket safety (no NaN/exceptions, rate defaults to `0.0`).
+  - Correct totals when evaluated against the public split (150 cases total, bucket distribution 15/45/45/45).
+  - Correct totals on combined splits matching the documented 20/60/60/60 distribution.
+  - Successful execution of the curve flatness ratio calculation.
+
+---
+
+## Consequences
+
+- The GSM8K-math evaluation lane now has a standard, deterministic measurement harness to construct the depth curve required by [ADR-0114a](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md).
+- The measurement-side of Obligation #6 is fully discharged for GSM8K-math.
+
+---
+
+## Out of Scope
+
+- Setting a specific numeric threshold ε for curve flatness (reserved for the promotion contract [ADR-0120](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0120-expert-promotion-contract.md)).

docs/decisions/README.md

@@ -50,6 +50,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt
 | [ADR-0119.1](ADR-0119.1-sealed-holdout-fabrication-control.md) | Seal `fabrication_control` Holdout (ADR-0105 Amendment) | Accepted (2026-05-23) |
 | [ADR-0119.2](ADR-0119.2-gsm8k-eval-corpus-dev-public.md) | GSM8K Eval Corpus Dev/Public Splits | Accepted (2026-05-22) |
 | [ADR-0119.4](ADR-0119.4-frontier-baseline-comparison.md) | GSM8K Math: Frontier-Baseline Comparison (ADR-0114a §Obligation #7) | Accepted (2026-05-22) |
+| [ADR-0119.6](ADR-0119.6-depth-curve-harness.md) | GSM8K Math Depth-Curve Measurement Harness | Accepted (2026-05-23) |
 
 ---
 
@@ -90,12 +91,12 @@ The ADR-0091..0114 slate is fully accepted (0091..0113) plus one proposed-roadma
 - `symbolic_logic` Lane-Shape Remap (ADR-0109 amendment) — ADR-0123
 - `systems_software` Audit-Passed Promotion (third successful) — ADR-0124
 - `all_three_pass_rate` Synonym in `inference_shape` (ADR-0109 Amendment) — ADR-0123a
-<<<<<<< HEAD
 - Reasoning-Isolation Perturbation Suite (224 deterministic applicable semantic perturbations; discharges ADR-0114a obligation #5 for the GSM8K-style parser dev lane) — ADR-0125
 - GSM8K Eval Lane Roadmap (Phase 5; decomposes into 5.1..5.8; proposed) — ADR-0119
 - Seal `fabrication_control` Holdout (ADR-0105 Amendment) — ADR-0119.1
 - GSM8K Eval Corpus Dev/Public Splits (200 CORE-original cases; verify.py 200/200; sealed holdout placeholder reserved for ADR-0119.7) — ADR-0119.2
 - GSM8K Math: Frontier-Baseline Comparison (citations for Claude 3.5, GPT-4, Gemini 1.5; comparison_v1.json; discharges ADR-0114a §Obligation #7) — ADR-0119.4
+- GSM8K Math Depth-Curve Measurement Harness (discharges ADR-0114a Obligation #6 measurement-side) — ADR-0119.6
 
 ADR-0080 has also landed: Contemplation Loop Phase 1 adds a read-only frontier-compare miner that emits `SPECULATIVE` findings only.
 

evals/gsm8k_math/scoring/__init__.py

@@ -0,0 +1 @@
+# Empty package marker

evals/gsm8k_math/scoring/depth_curve.py

@@ -0,0 +1,146 @@
+"""ADR-0119.6 — GSM8K math depth-curve measurement harness.
+
+Buckets the correct rate of a lane run by reasoning depth
+(number of operations in the ground-truth graph).
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+from typing import Any
+
+# Ensure workspace root is in sys.path when running as script
+sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
+
+from evals.gsm8k_math.runner import LaneReport, run_lane
+
+
+def compute_depth_curve(cases: list[dict[str, Any]], lane_report: LaneReport) -> dict[str, Any]:
+    """Pure, deterministic function bucketizing cases by reasoning depth.
+
+    Reads the case list and a LaneReport, buckets cases by reasoning depth,
+    and returns a depth curve dictionary.
+    """
+    # Extract outcomes by case_id from the LaneReport
+    outcomes = {detail["case_id"]: detail["outcome"] for detail in lane_report.case_details}
+
+    # Initialize buckets with 0.0 rates (Obligation #6 empty bucket safety)
+    bucket_keys = ["depth_1", "depth_2-3", "depth_4-5", "depth_6-8"]
+    buckets: dict[str, dict[str, Any]] = {
+        k: {"total": 0, "correct": 0, "rate": 0.0} for k in bucket_keys
+    }
+
+    # Extract depths of all cases
+    case_depths = {}
+    max_depth = 0
+    for case in cases:
+        case_id = case["id"]
+        gt_graph = case.get("ground_truth_graph", {})
+        if isinstance(gt_graph, dict):
+            operations = gt_graph.get("operations", [])
+        else:
+            operations = getattr(gt_graph, "operations", [])
+        depth = len(operations)
+        case_depths[case_id] = depth
+        if depth > max_depth:
+            max_depth = depth
+
+    # Initialize raw_curve mapping for depths 1 to max_depth
+    raw_depths = {d: {"total": 0, "correct": 0} for d in range(1, max_depth + 1)}
+
+    for case in cases:
+        case_id = case["id"]
+        depth = case_depths[case_id]
+        outcome = outcomes.get(case_id, "refused")
+
+        is_correct = 1 if outcome == "correct" else 0
+
+        # Resolve bucket key
+        if depth == 1:
+            b_key = "depth_1"
+        elif 2 <= depth <= 3:
+            b_key = "depth_2-3"
+        elif 4 <= depth <= 5:
+            b_key = "depth_4-5"
+        elif 6 <= depth <= 8:
+            b_key = "depth_6-8"
+        else:
+            b_key = f"depth_{depth}"
+
+        if b_key not in buckets:
+            buckets[b_key] = {"total": 0, "correct": 0, "rate": 0.0}
+
+        buckets[b_key]["total"] += 1
+        buckets[b_key]["correct"] += is_correct
+
+        if depth not in raw_depths:
+            raw_depths[depth] = {"total": 0, "correct": 0}
+        raw_depths[depth]["total"] += 1
+        raw_depths[depth]["correct"] += is_correct
+
+    # Compute rates
+    for k, v in buckets.items():
+        if v["total"] > 0:
+            v["rate"] = float(v["correct"] / v["total"])
+        else:
+            v["rate"] = 0.0
+
+    raw_curve = []
+    for d in sorted(raw_depths.keys()):
+        v = raw_depths[d]
+        rate = float(v["correct"] / v["total"]) if v["total"] > 0 else 0.0
+        raw_curve.append({
+            "depth": d,
+            "total": v["total"],
+            "correct": v["correct"],
+            "rate": rate
+        })
+
+    return {
+        "buckets": buckets,
+        "max_depth": max_depth,
+        "raw_curve": raw_curve
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Run GSM8K math lane and print depth curve.")
+    parser.add_argument(
+        "--split",
+        choices=["dev", "public"],
+        required=True,
+        help="The corpus split to run against.",
+    )
+    args = parser.parse_args()
+
+    root = Path(__file__).resolve().parents[3]
+    if args.split == "dev":
+        case_path = root / "evals/gsm8k_math/dev/cases.jsonl"
+    else:
+        case_path = root / "evals/gsm8k_math/public/v1/cases.jsonl"
+
+    if not case_path.exists():
+        print(f"Error: case file not found at {case_path}", file=sys.stderr)
+        sys.exit(1)
+
+    cases = []
+    with open(case_path, "r", encoding="utf-8") as f:
+        for line in f:
+            if line.strip():
+                cases.append(json.loads(line))
+
+    # Run the lane runner to get the LaneReport
+    lane_report = run_lane(cases)
+
+    # Compute depth curve
+    curve = compute_depth_curve(cases, lane_report)
+
+    # Print the JSON output
+    print(json.dumps(curve, indent=2))
+
+
+if __name__ == "__main__":
+    main()

tests/test_adr_0119_6_depth_curve.py

@@ -0,0 +1,130 @@
+"""Tests for ADR-0119.6 depth-curve harness."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+import pytest
+
+from evals.gsm8k_math.runner import run_lane
+from evals.gsm8k_math.scoring.depth_curve import compute_depth_curve
+
+_REPO_ROOT = Path(__file__).resolve().parent.parent
+
+
+def _load_jsonl(path: Path) -> list[dict]:
+    return [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
+
+
+_DEV_CASES = _load_jsonl(_REPO_ROOT / "evals" / "gsm8k_math" / "dev" / "cases.jsonl")
+_PUBLIC_CASES = _load_jsonl(_REPO_ROOT / "evals" / "gsm8k_math" / "public" / "v1" / "cases.jsonl")
+
+
+def test_determinism() -> None:
+    """Pins (a) Determinism: same inputs -> same JSON output."""
+    report1 = run_lane(_DEV_CASES)
+    curve1 = compute_depth_curve(_DEV_CASES, report1)
+
+    report2 = run_lane(_DEV_CASES)
+    curve2 = compute_depth_curve(_DEV_CASES, report2)
+
+    # Convert to JSON string to assert exact byte-equal structures
+    json1 = json.dumps(curve1, sort_keys=True)
+    json2 = json.dumps(curve2, sort_keys=True)
+    assert json1 == json2
+
+
+def test_bucket_totals_sum_to_total_cases() -> None:
+    """Pins (b) Bucket totals sum to lane_report.metrics['cases_total']."""
+    report = run_lane(_DEV_CASES)
+    curve = compute_depth_curve(_DEV_CASES, report)
+
+    bucket_total_sum = sum(b["total"] for b in curve["buckets"].values())
+    assert bucket_total_sum == report.metrics["cases_total"]
+
+
+def test_bucket_correct_sums_match_report() -> None:
+    """Pins (c) Sum of (bucket.correct) == lane_report.metrics['correct']."""
+    report = run_lane(_DEV_CASES)
+    curve = compute_depth_curve(_DEV_CASES, report)
+
+    bucket_correct_sum = sum(b["correct"] for b in curve["buckets"].values())
+    assert bucket_correct_sum == report.metrics["correct"]
+
+
+def test_empty_bucket_rate_is_zero() -> None:
+    """Pins (d) Empty bucket -> rate is 0.0 (not NaN, not exception)."""
+    # Filter cases to only those with depth 1
+    depth_1_cases = [
+        c for c in _DEV_CASES
+        if len(c.get("ground_truth_graph", {}).get("operations", [])) == 1
+    ]
+    assert len(depth_1_cases) > 0, "No depth-1 cases found in dev set"
+
+    report = run_lane(depth_1_cases)
+    curve = compute_depth_curve(depth_1_cases, report)
+
+    # Buckets depth_2-3, depth_4-5, depth_6-8 should be empty and have 0.0 rate
+    for k in ["depth_2-3", "depth_4-5", "depth_6-8"]:
+        assert curve["buckets"][k]["total"] == 0
+        assert curve["buckets"][k]["correct"] == 0
+        assert curve["buckets"][k]["rate"] == 0.0
+
+
+def test_public_split_totals() -> None:
+    """Pins (e) Run against the public split (150 cases); confirm bucket totals."""
+    report = run_lane(_PUBLIC_CASES)
+    curve = compute_depth_curve(_PUBLIC_CASES, report)
+
+    # Check buckets covered: 1, 2-3, 4-5, 6-8
+    expected_buckets = {"depth_1", "depth_2-3", "depth_4-5", "depth_6-8"}
+    assert set(curve["buckets"].keys()) == expected_buckets
+
+    # Per-bucket totals on public alone: each bucket has the public-only contribution (15/45/45/45)
+    assert curve["buckets"]["depth_1"]["total"] == 15
+    assert curve["buckets"]["depth_2-3"]["total"] == 45
+    assert curve["buckets"]["depth_4-5"]["total"] == 45
+    assert curve["buckets"]["depth_6-8"]["total"] == 45
+
+
+def test_total_corpus_distribution() -> None:
+    """Pins (e) Confirm total combined corpus distribution matches 20/60/60/60."""
+    all_cases = _DEV_CASES + _PUBLIC_CASES
+    report = run_lane(all_cases)
+    curve = compute_depth_curve(all_cases, report)
+
+    assert curve["buckets"]["depth_1"]["total"] == 20
+    assert curve["buckets"]["depth_2-3"]["total"] == 60
+    assert curve["buckets"]["depth_4-5"]["total"] == 60
+    assert curve["buckets"]["depth_6-8"]["total"] == 60
+
+
+def test_curve_flatness_ratio() -> None:
+    """Pins (f) Verify curve flatness calculation: compute ratio without enforcing threshold."""
+    report = run_lane(_DEV_CASES)
+    curve = compute_depth_curve(_DEV_CASES, report)
+
+    max_depth = curve["max_depth"]
+    if max_depth == 1:
+        max_bucket_key = "depth_1"
+    elif 2 <= max_depth <= 3:
+        max_bucket_key = "depth_2-3"
+    elif 4 <= max_depth <= 5:
+        max_bucket_key = "depth_4-5"
+    elif 6 <= max_depth <= 8:
+        max_bucket_key = "depth_6-8"
+    else:
+        max_bucket_key = f"depth_{max_depth}"
+
+    max_depth_rate = curve["buckets"][max_bucket_key]["rate"]
+    depth_1_rate = curve["buckets"]["depth_1"]["rate"]
+
+    # Compute ratio
+    if depth_1_rate > 0.0:
+        ratio = max_depth_rate / depth_1_rate
+    else:
+        ratio = 0.0
+
+    # The test confirms that the harness CAN compute and report the ratio
+    assert isinstance(ratio, float)
+    assert ratio >= 0.0