83 lines
4.7 KiB
Markdown
83 lines
4.7 KiB
Markdown
# ADR-0119.6 — GSM8K Math Depth-Curve Measurement Harness
|
|
|
|
**Status:** Accepted
|
|
**Date:** 2026-05-23
|
|
**Author:** CORE agents + reviewers
|
|
**Depends on:** [ADR-0114a](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md), [ADR-0119](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0119-gsm8k-eval-lane-roadmap.md), [ADR-0119.2](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0119.2-gsm8k-eval-corpus-dev-public.md)
|
|
|
|
---
|
|
|
|
## Context
|
|
|
|
[ADR-0114a](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md) (Obligation #6) requires publishing a compositional-depth vs. accuracy curve for any domain promoting to `expert`. Accuracy as a function of reasoning depth must not decay sharply (a pattern-matcher decays; a reasoning system stays flat within ε).
|
|
|
|
With the GSM8K-style eval corpus splits authored ([ADR-0119.2](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0119.2-gsm8k-eval-corpus-dev-public.md)) and the lane runner developed ([ADR-0119.3](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/evals/gsm8k_math/runner.py)), we must ship the measurement harness that implements this curve calculation from a lane report and a case set. This fulfills the measurement-side requirement of Obligation #6.
|
|
|
|
---
|
|
|
|
## Decision
|
|
|
|
We introduce a measurement harness and CLI hook to calculate and print the depth-vs-correctness curve.
|
|
|
|
### Implementation Details
|
|
|
|
- **Module:** [depth_curve.py](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/evals/gsm8k_math/scoring/depth_curve.py)
|
|
- **Exposed Function:**
|
|
```python
|
|
def compute_depth_curve(cases: list[dict], lane_report: LaneReport) -> dict[str, Any]
|
|
```
|
|
Returns:
|
|
```json
|
|
{
|
|
"buckets": {
|
|
"depth_1": {"total": N, "correct": M, "rate": float},
|
|
"depth_2-3": {"total": N, "correct": M, "rate": float},
|
|
"depth_4-5": {"total": N, "correct": M, "rate": float},
|
|
"depth_6-8": {"total": N, "correct": M, "rate": float}
|
|
},
|
|
"max_depth": int,
|
|
"raw_curve": [
|
|
{"depth": 1, "total": N, "correct": M, "rate": float},
|
|
...
|
|
]
|
|
}
|
|
```
|
|
- **Bucket key format:** `depth_1` for single integers; `depth_2-3`, `depth_4-5`, `depth_6-8` for the documented buckets matching the corpus distribution.
|
|
- **Empty bucket safety:** Rate is initialized/defaulted to `0.0` if a bucket has zero cases (preventing NaN or division-by-zero exceptions).
|
|
- **CLI hook:** Invoking `python3 -m evals.gsm8k_math.scoring.depth_curve --split dev|public` executes the lane runner on the chosen split and prints the resulting JSON.
|
|
|
|
---
|
|
|
|
## Invariants
|
|
|
|
- **Determinism:** Given the same cases and lane report, `compute_depth_curve` outputs byte-identical results.
|
|
- **Completeness:** The sum of totals in `buckets` equals the total number of cases.
|
|
- **Correctness:** The sum of correct cases in `buckets` equals the overall correct case count in the lane report.
|
|
|
|
---
|
|
|
|
## Acceptance Evidence
|
|
|
|
Accepted when:
|
|
- The module [depth_curve.py](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/evals/gsm8k_math/scoring/depth_curve.py) is present and functional.
|
|
- The tests in [test_adr_0119_6_depth_curve.py](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/tests/test_adr_0119_6_depth_curve.py) pass cleanly, validating:
|
|
- Determinism across runs.
|
|
- Bucket totals sum to total cases.
|
|
- Bucket correct counts sum to total correct.
|
|
- Empty bucket safety (no NaN/exceptions, rate defaults to `0.0`).
|
|
- Correct totals when evaluated against the public split (150 cases total, bucket distribution 15/45/45/45).
|
|
- Correct totals on combined splits matching the documented 20/60/60/60 distribution.
|
|
- Successful execution of the curve flatness ratio calculation.
|
|
|
|
---
|
|
|
|
## Consequences
|
|
|
|
- The GSM8K-math evaluation lane now has a standard, deterministic measurement harness to construct the depth curve required by [ADR-0114a](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md).
|
|
- The measurement-side of Obligation #6 is fully discharged for GSM8K-math.
|
|
|
|
---
|
|
|
|
## Out of Scope
|
|
|
|
- Setting a specific numeric threshold ε for curve flatness (reserved for the promotion contract [ADR-0120](file:///Users/kaizenpro/.gemini/antigravity/worktrees/core/adr-0119-depth-curve-harness/docs/decisions/ADR-0120-expert-promotion-contract.md)).
|