ADR-0119.6 — GSM8K Math Depth-Curve Measurement Harness

Status: Accepted Date: 2026-05-23 Author: CORE agents + reviewers Depends on: ADR-0114a, ADR-0119, ADR-0119.2

Context

ADR-0114a (Obligation #6) requires publishing a compositional-depth vs. accuracy curve for any domain promoting to expert. Accuracy as a function of reasoning depth must not decay sharply (a pattern-matcher decays; a reasoning system stays flat within ε).

With the GSM8K-style eval corpus splits authored (ADR-0119.2) and the lane runner developed (ADR-0119.3), we must ship the measurement harness that implements this curve calculation from a lane report and a case set. This fulfills the measurement-side requirement of Obligation #6.

Decision

We introduce a measurement harness and CLI hook to calculate and print the depth-vs-correctness curve.

Implementation Details

Module: depth_curve.py

Exposed Function:

def compute_depth_curve(cases: list[dict], lane_report: LaneReport) -> dict[str, Any]

Returns:

{
  "buckets": {
    "depth_1": {"total": N, "correct": M, "rate": float},
    "depth_2-3": {"total": N, "correct": M, "rate": float},
    "depth_4-5": {"total": N, "correct": M, "rate": float},
    "depth_6-8": {"total": N, "correct": M, "rate": float}
  },
  "max_depth": int,
  "raw_curve": [
    {"depth": 1, "total": N, "correct": M, "rate": float},
    ...
  ]
}

Bucket key format: depth_1 for single integers; depth_2-3, depth_4-5, depth_6-8 for the documented buckets matching the corpus distribution.
Empty bucket safety: Rate is initialized/defaulted to 0.0 if a bucket has zero cases (preventing NaN or division-by-zero exceptions).
CLI hook: Invoking python3 -m evals.gsm8k_math.scoring.depth_curve --split dev|public executes the lane runner on the chosen split and prints the resulting JSON.

Invariants

Determinism: Given the same cases and lane report, compute_depth_curve outputs byte-identical results.
Completeness: The sum of totals in buckets equals the total number of cases.
Correctness: The sum of correct cases in buckets equals the overall correct case count in the lane report.

Acceptance Evidence

Accepted when:

The module depth_curve.py is present and functional.
The tests in test_adr_0119_6_depth_curve.py pass cleanly, validating:
- Determinism across runs.
- Bucket totals sum to total cases.
- Bucket correct counts sum to total correct.
- Empty bucket safety (no NaN/exceptions, rate defaults to 0.0).
- Correct totals when evaluated against the public split (150 cases total, bucket distribution 15/45/45/45).
- Correct totals on combined splits matching the documented 20/60/60/60 distribution.
- Successful execution of the curve flatness ratio calculation.

Consequences

The GSM8K-math evaluation lane now has a standard, deterministic measurement harness to construct the depth curve required by ADR-0114a.
The measurement-side of Obligation #6 is fully discharged for GSM8K-math.

Out of Scope

Setting a specific numeric threshold ε for curve flatness (reserved for the promotion contract ADR-0120).

4.7 KiB Raw Permalink Blame History