# ADR-0119.6 — GSM8K Math Depth-Curve Measurement Harness

**Status:** Accepted
**Date:** 2026-05-23
**Author:** CORE agents + reviewers
**Depends on:** [ADR-0114a](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md), [ADR-0119](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0119-gsm8k-eval-lane-roadmap.md), [ADR-0119.2](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0119.2-gsm8k-eval-corpus-dev-public.md)

---

## Context

[ADR-0114a](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md) (Obligation #6) requires publishing a compositional-depth vs. accuracy curve for any domain promoting to `expert`. Accuracy as a function of reasoning depth must not decay sharply (a pattern-matcher decays; a reasoning system stays flat within ε).

With the GSM8K-style eval corpus splits authored ([ADR-0119.2](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0119.2-gsm8k-eval-corpus-dev-public.md)) and the lane runner developed ([ADR-0119.3](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/evals/gsm8k_math/runner.py)), we must ship the measurement harness that implements this curve calculation from a lane report and a case set. This fulfills the measurement-side requirement of Obligation #6.

---

## Decision

We introduce a measurement harness and CLI hook to calculate and print the depth-vs-correctness curve.

### Implementation Details

- **Module:** [depth_curve.py](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/evals/gsm8k_math/scoring/depth_curve.py)
- **Exposed Function:**
  ```python
  def compute_depth_curve(cases: list[dict], lane_report: LaneReport) -> dict[str, Any]
  ```
  Returns:
  ```json
  {
    "buckets": {
      "depth_1": {"total": N, "correct": M, "rate": float},
      "depth_2-3": {"total": N, "correct": M, "rate": float},
      "depth_4-5": {"total": N, "correct": M, "rate": float},
      "depth_6-8": {"total": N, "correct": M, "rate": float}
    },
    "max_depth": int,
    "raw_curve": [
      {"depth": 1, "total": N, "correct": M, "rate": float},
      ...
    ]
  }
  ```
- **Bucket key format:** `depth_1` for single integers; `depth_2-3`, `depth_4-5`, `depth_6-8` for the documented buckets matching the corpus distribution.
- **Empty bucket safety:** Rate is initialized/defaulted to `0.0` if a bucket has zero cases (preventing NaN or division-by-zero exceptions).
- **CLI hook:** Invoking `python3 -m evals.gsm8k_math.scoring.depth_curve --split dev|public` executes the lane runner on the chosen split and prints the resulting JSON.

---

## Invariants

- **Determinism:** Given the same cases and lane report, `compute_depth_curve` outputs byte-identical results.
- **Completeness:** The sum of totals in `buckets` equals the total number of cases.
- **Correctness:** The sum of correct cases in `buckets` equals the overall correct case count in the lane report.

---

## Acceptance Evidence

Accepted when:
- The module [depth_curve.py](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/evals/gsm8k_math/scoring/depth_curve.py) is present and functional.
- The tests in [test_adr_0119_6_depth_curve.py](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/tests/test_adr_0119_6_depth_curve.py) pass cleanly, validating:
  - Determinism across runs.
  - Bucket totals sum to total cases.
  - Bucket correct counts sum to total correct.
  - Empty bucket safety (no NaN/exceptions, rate defaults to `0.0`).
  - Correct totals when evaluated against the public split (150 cases total, bucket distribution 15/45/45/45).
  - Correct totals on combined splits matching the documented 20/60/60/60 distribution.
  - Successful execution of the curve flatness ratio calculation.

---

## Consequences

- The GSM8K-math evaluation lane now has a standard, deterministic measurement harness to construct the depth curve required by [ADR-0114a](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md).
- The measurement-side of Obligation #6 is fully discharged for GSM8K-math.

---

## Out of Scope

- Setting a specific numeric threshold ε for curve flatness (reserved for the promotion contract [ADR-0120](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0120-expert-promotion-contract.md)).