core/docs/decisions/ADR-0114a.6-depth-curve-auditor.md

ADR-0114a.6 — Compositional-Depth Curve Auditor (Obligation #6 wired for B3)

Status: Accepted (mechanism); coverage gap deferred to B3-owner follow-up Date: 2026-05-23 Author: CORE main agent (Opus 4.7) Depends on: ADR-0114a (10 anti-overfitting obligations), ADR-0119.6 (GSM8K depth-curve substrate — pattern source), ADR-0131.3 (B3 bounded grammar lane), ADR-0114a.10 (PR #189 — first obligation auditor; pattern this PR mirrors) Parent: ADR-0114a Sequencing: second of 5 remaining ADR-0114a obligations for mathematics_logic. After this: #2 OOD ratio, #5 perturbation (in flight, L14 / Opus#2), #8 adversarial.

---

Context

ADR-0114a Obligation #6:

depth_curve.py produces a per-bucket curve; accuracy(N) ≥ accuracy(depth_1) · (1 − ε)^(N − 1) for ε = 0.05.

A pattern-matcher's accuracy decays sharply with reasoning depth; a deterministic reasoning system stays approximately flat. The obligation pins that decay rate at most 5% per depth-step from the depth-1 anchor.

ADR-0119.6 shipped this measurement harness for GSM8K (lives at evals/gsm8k_math/scoring/depth_curve.py). This PR ports the pattern to the math composite gate's B3 lane.

Decision

core/capability/depth_curve.py — pure auditor. For each B3 case marked expected: solved_correct:

1. Re-run the candidate-graph pipeline.
2. Bucket by len(trace.steps) (authoritative depth = the number of operations the solver actually executed).
3. Aggregate per-bucket accuracy.
4. Check each populated bucket (≥1 case) against the decay bound anchored on depth_1's accuracy.

Bucket schema (closed, mirrors ADR-0119.6)

depth_1, depth_2-3, depth_4-5, depth_6-8. Depth > 8 raises DepthCurveError rather than silently extending the schema. Depth == 0 (initial-only problems) is counted as "skipped" — zero-depth has no reasoning to decay.

Representative depth convention

representative_depth(bucket) = min(bucket) — most permissive, gives the bound the best chance of holding even when only the shallow end of the bucket is populated. Tightening (e.g., max-depth-in-bucket) requires an ADR amendment.

Verdict shape — two orthogonal facts

The report distinguishes mechanism wiring from assertion meaningfulness:

Field	Meaning
obligation_6_mechanism_wired	The auditor exists and ran end-to-end.
obligation_6_assertion_holds	Every populated bucket satisfies the decay bound. (Vacuously true when only depth_1 is populated.)
coverage_sufficient	≥2 buckets populated AND ≥3 cases per populated bucket. Required for the assertion to be statistically meaningful.

CLI exits 0 iff obligation_6_assertion_holds. coverage_sufficient flows as honest disclosure in the refusal_reason when the assertion holds only vacuously.

Empirical verdict on current main

$ python3 -m core.cli capability depth-curve

lane:                          B3_bounded_grammar
cases_total:                   50
cases_solved:                  22
epsilon:                       0.05
mechanism_wired:               True
assertion_holds:               True
coverage_sufficient:           False
populated_buckets:             ['depth_1', 'depth_2-3']

  bucket       total   correct  accuracy   bound      satisfied
  depth_1      21      21       1.0000     (anchor)   True
  depth_2-3    1       1        1.0000     0.9500     True
  depth_4-5    0       0        0.0000     (anchor)   True
  depth_6-8    0       0        0.0000     (anchor)   True

refusal_reason: assertion holds but coverage insufficient — populated
buckets: ['depth_1', 'depth_2-3'] (need ≥2 buckets, ≥3 cases each)

Both populated buckets satisfy the decay bound (depth_1 anchors at 100%; depth_2-3 at 100% > 0.95 bound). The mechanism is wired correctly. But coverage is insufficient — depth_2-3 has only 1 case, depth_4-5 and depth_6-8 have none.

Honest scope-limit + named follow-up

For obligation #6 to meaningfully gate the full ADR-0120 expert promotion, B3 v1's case set needs more depth coverage:

• ≥3 cases at depth_2-3 (currently 1)
• ≥3 cases at depth_4-5 (currently 0)
• Ideally ≥3 cases at depth_6-8 (currently 0)

This is B3-owner work, not auditor work. The case authoring is governed by ADR-0131.3's grammar contract; depth-4+ problems are admissible by B3's grammar (sequence of additive/subtractive ops with consistent units) but the v1 case set leaned heavily on single-op problems for the lane-gate's correct_rate ≥ 0.95 target.

Reserved follow-up: a small B3 v1.1 PR that authors additional multi-step cases under B3's existing grammar. After it lands, coverage_sufficient flips to True and obligation #6 becomes materially gating.

What this does NOT do

• Does NOT change the solver, the parser, or any B-lane runner.
• Does NOT modify B3's case set. (The case authoring is reserved follow-up, scoped to B3's owner.)
• Does NOT promote mathematics_logic to expert.
• Does NOT wire B1 or B2 equivalents:
  • B1 (symbolic equivalence): algebra-not-arithmetic, no solver trace, no depth notion. Different metric needed.
  • B2 (teaching corpus): chain-validation surface, not problem-solving. Different metric needed.
  • Both deferred to separate sub-ADRs.

Trust boundary

• Reads only:
  • evals/math_bounded_grammar/v1/cases.jsonl
  • The pipeline's own pack reads (transitively via parse_and_solve → solve)
• Writes only: artifact path (default evals/obligation_6_depth_curve/<lane_id>.json)
• No dynamic imports, no shell passthrough, no network.
• Pure deterministic function — verified by test_report_is_deterministic and test_artifact_emission_byte_equal.

Tests

tests/test_adr_0114a_6_depth_curve.py — 24 tests:

Group	Count	What it pins
bucket schema + classification	12	closed set; depth 1..8 maps correctly; depth ≥ 9 raises
decay bound math	5	ε = 0.05 pinned; bound = anchor · 0.95^(N-1); representative_depth = min(bucket); depth_1 has no bound
coverage policy	1	thresholds pinned (≥2 buckets, ≥3 cases each)
lane evaluation	3	passes on real B3; populates depth_1; refuses on missing cases
coverage-sufficient distinction	2	B3 today is coverage-insufficient with assertion-holds; synthetic 5+5 fixture is both
determinism	2	report identical across calls; artifact byte-equal

All pass in 0.28s.

Composition with ADR-0131.4 and obligation #10

Orthogonal. The composite math gate (ADR-0131.4 / PR #188 / merged) gates the benchmark portion of the contract. Obligation #10 (PR #189) audits pack provenance. Obligation #6 (this PR) audits reasoning-depth decay. Each is its own evaluator; a future full ADR-0120 wire-up composes them into a single signed expert_claims artifact for ledger promotion.

CLAUDE.md PR-checklist

• Capability added: depth-curve auditor with honest two-axis verdict (mechanism wired + assertion holds vs. coverage sufficient) for B3.
• Invariant proving field validity: decay bound holds on every populated bucket; ε = 0.05 pinned; representative_depth convention pinned.
• CLI/eval proving the lane: python3 -m core.cli capability depth-curve + pytest tests/test_adr_0114a_6_depth_curve.py.
• Avoided hidden normalization / stochastic / approximate / unreviewed mutation: Yes. Pure deterministic auditor.
• Trust boundary: read-only inputs from documented paths; single deterministic write; no dynamic imports.