core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

feat(ADR-0131.G): GSM8K coverage probe — honest baseline + capability-first iteration discipline

Browse source 
ADR-0131 deferred GSM8K because it rewards paraphrase flexibility,
which is the deterministic engine's structural weakness. This ADR
re-engages it on architecture-aligned terms: as a *coverage probe*
of the bounded grammar + binding graph, not a promotion gate.

The framing pinned by this ADR:

  GSM8K is not a target. The model's capability is the target.
  GSM8K passing is the symptom of capability, not the goal of
  the work.

Wrong mindset (rejected by ADR's iteration discipline):
  "Find templates that admit more GSM8K cases."

Right mindset (load-bearing):
  "Extend the model's NL-to-typed-graph capability along
  principled axes (verb classes, comparative structures, numeric
  forms, multi-clause grammar). GSM8K admission rises as a
  side effect alongside every other word-problem corpus."

Baseline pinned by this commit:

  admission_rate: 0/50 = 0.0%
  admitted_wrong: 0  (gate intact, safety rail bulletproof)
  refused:        50/50 = 100.0%

Every refusal is a typed parser error citing the specific clause
that did not match a template. Zero crashes, zero confabulations
— refusal-first works perfectly at admission rate zero.

What's in this PR:

- ``docs/decisions/ADR-0131.G-gsm8k-coverage-probe.md``: the ADR.
  Cites parents (ADR-0131, -0115/-0116/-0117, -0131.3, -0132..-0135).
  Documents the capability-first iteration discipline that every
  subsequent ADR-0131.G.<n> must follow:
    1. Name a single capability axis the iteration extends
    2. Add B3-style curated coverage cases (capability proves
       itself OUTSIDE GSM8K)
    3. Re-run both B3 lane + GSM8K probe; B3 must not regress
    4. Reject any expansion that only moves GSM8K admission

- ``evals/gsm8k_math/train_sample/v1/run_coverage_probe.py``:
  pure-adapter wrapper around the existing run_lane. Emits a
  deterministic train_sample_coverage_report.json with metrics,
  per-case outcomes, and the top refused-reason families (the
  work queue for capability extension).

- ``evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json``:
  the baseline report. Diff-able artifact every future iteration
  moves.

- ``tests/test_adr_0131_G_gsm8k_coverage_probe.py``: 8 contract
  tests pinning the safety rail (admitted_wrong == 0), typed
  refusal invariant (every refused case has non-empty reason),
  closed outcome vocabulary, deterministic replay, committed-
  report matches fresh-run.

The promotion-gate composite (B1 + B2 + B3) is unaffected.
ADR-0131.4 still consumes those three. The GSM8K probe is
empirical context for honest external claims, not a gate.
This commit is contained in:
Shay 2026-05-23 13:16:09 -07:00
parent 24f6a596fe
commit 23c126ebe0
4 changed files with 1286 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

242
docs/decisions/ADR-0131.G-gsm8k-coverage-probe.md Normal file
View file

@ -0,0 +1,242 @@
# ADR-0131.G — GSM8K Coverage Probe: Honest Measurement Under the Safety Rail
**Status:** Proposed
**Date:** 2026-05-23
**Author:** CORE agents + reviewers
**Parent:** [ADR-0131](./ADR-0131-math-expert-rebench.md)
**Depends on:**
[ADR-0115](./ADR-0115-math-problem-parser-and-graph.md),
[ADR-0116](./ADR-0116-deterministic-solver.md),
[ADR-0117](./ADR-0117-solution-trace-verifier.md),
[ADR-0131.3](./ADR-0131.3-bounded-grammar.md),
[ADR-0132](./ADR-0132-binding-graph-data-model.md)..ADR-0135
---
## Context
ADR-0131 re-targeted the math-expert promotion away from GSM8K
because GSM8K rewards paraphrase flexibility, which is the
deterministic engine's structural weakness. The composite
architecture-aligned gate (B1 symbolic equivalence, B2 teaching
corpus, B3 bounded-grammar word problems) is the *true* gate.
GSM8K does not vanish from the project; it returns as a **coverage
probe** of the bounded grammar + binding-graph layer. The framing
this ADR pins:
> GSM8K is not a promotion gate. It is a coverage measurement of
> what the bounded grammar admits. Of the admitted subset, the
> engine solves deterministically with ``wrong == 0``.
What we never claim:
> "We score X% on GSM8K"
What we may claim:
> "We admit X% of GSM8K with the current bounded grammar. Within
> that admitted subset we solve 100% with zero confabulation;
> outside it we refuse cleanly with typed reasons."
Those two sentences look superficially similar. They are not. The
first conflates parser scope with reasoning ability and would
re-open the parser-shape treadmill ADR-0131 closed. The second
preserves the architecture-aligned discipline: refusal-first,
``wrong == 0`` invariant, no confabulation under any input.
---
## Decision
Introduce a **coverage probe**:
``evals/gsm8k_math/train_sample/v1/run_coverage_probe.py``. It
adapts the GSM8K train_sample (ADR-0126 P5) into the existing
``evals.gsm8k_math.runner.run_lane`` shape and writes a
deterministic ``train_sample_coverage_report.json``. The report
fields are pinned by this ADR:
| Field | Meaning |
|---|---|
| ``admitted_solved`` | Parsed + solved + verifier-passed + correct answer/unit |
| ``admitted_wrong`` | Parsed + solved BUT verifier or answer mismatch — **gate: must == 0** |
| ``refused`` | Refused at parser or solver with a typed error |
| ``admission_rate`` | ``admitted_solved / cases_total`` |
| ``safety_rail_intact`` | ``admitted_wrong == 0`` |
| ``refused_reasons_top`` | Counter of the most common refusal reasons — the *work queue* for grammar expansion |
| ``per_case`` | Full audit trail (deterministic order) |
The probe runs the existing ``run_lane`` — the *runner is the
contract*, not the probe. The probe's job is to adapt input + emit
the report.
### Iteration discipline
**Capability-first, not coverage-first.** Each subsequent
ADR-0131.G.<n> (G.1, G.2, ...) lands a *capability extension* —
not a list of GSM8K-specific templates. The distinction is the
load-bearing one in this ADR:
- ❌ Wrong mode: "Find templates that admit more GSM8K cases."
This leads to narrow patterns that score on the test but do
not generalize. Coverage rises, capability does not.
- ✅ Right mode: "Extend the model's NL-to-typed-graph capability
along a principled axis (verb class, comparative structure,
numeric form, multi-clause grammar). GSM8K admission rises as
a *symptom* of that growth — alongside any other word-problem
corpus."
Concretely, every iteration must:
1. Name a single capability axis it extends (e.g. "rate verbs
as initial-state-introducing predicates", "comparative-
multiplicative as a binding-graph operation kind").
2. Add B3-style curated coverage cases that exercise the new
capability **independently of GSM8K** — i.e. the new template
class is exercised by hand-authored sentences in
``evals/math_bounded_grammar/v1/cases.jsonl`` (or a sibling),
not only by GSM8K cases.
3. Re-run *both* the B3 lane and the GSM8K coverage probe.
The B3 lane must still pass; the GSM8K probe ``admission_rate``
moves as a side effect.
4. **Reject any expansion that only moves GSM8K admission and
not B3 / curated coverage.** That is the smell test for a
template in disguise.
Gating at each iteration:
1. ``admitted_wrong == 0`` on the GSM8K probe (the safety rail)
2. B3 lane still passes (no regression on the curated benchmark)
3. The new capability has its own curated test cases — landed
in the same PR — that the iteration's PR diff exercises
4. ``admission_rate`` strictly increased OR ``refused_reasons_top``
shows a deliberate, documented reduction in a targeted clause
family
We never grow admission by relaxing the parser into "best-effort"
guessing. Every admitted case must round-trip through the
deterministic solver + independent verifier (ADR-0117). Every
admitted case is admitted because a *general capability* now
recognizes it — not because a specific GSM8K phrasing got
hard-coded.
### Baseline (this ADR pins the starting line)
```
admission_rate: 0/50 = 0.0%
admitted_wrong: 0 (gate intact)
refused: 50/50 = 100.0%
```
Every refusal is at the parser layer with a typed error citing the
specific clause that did not match a template. **Zero crashes,
zero confabulations.** The safety rail is bulletproof at admission
rate zero.
### What "passing GSM8K" means under this ADR
The phrase is informal shorthand for: *get the model to a
capability level where GSM8K passes as a side effect.* It is
explicitly **not** "tune the model to score on GSM8K." The mindset
distinction is the load-bearing one:
- Shoot for **test-passing capability** → game the benchmark,
produce numbers that don't transfer
- Shoot for **the level at which the test passes** → grow real
NL-to-typed-graph capability; GSM8K passes as a symptom
### On the question of an architectural ceiling
There is **no theoretical capability ceiling** for unambiguous,
well-specified word problems. Determinism excludes three things:
1. Stochastic sampling — CORE will not sample-and-pray.
2. Probabilistic best-guess on ambiguous input — CORE refuses
when interpretation is underdetermined.
3. *Not* NL understanding. The binding graph + typed compiler
*is* an NL-understanding layer. Determinism does not exclude
it; it constrains how it commits.
What CORE has instead of a confabulation ceiling is a **refusal
floor** on genuinely ambiguous or underspecified inputs. On
benchmarks like GSM8K that floor is small (problems are mostly
well-specified). The contrast with frontier:
- Frontier hits ~95% on GSM8K because it confabulates the last
~5% — cases that should have refused but produced confident
wrong answers instead.
- CORE has the opposite shape on that ~5%: typed refusal with
a cited reason, not a confidently wrong number.
The practical limit on admission_rate is **engineering effort on
the parser + binding-graph layer**, not architecture. The honest
claim, at every admission level, is the same two-sentence framing
in the Context section. If a capability extension also makes B3
stronger, makes new word-problem corpora work out-of-the-box, and
survives adversarial paraphrase tests, then GSM8K admission gain
is a real signal. If the capability only moves GSM8K, the
iteration failed its own discipline.
---
## Invariants
- **``wrong_count_is_zero``** — every report iteration must show
``admitted_wrong == 0``. This is the load-bearing safety rail
inherited from B1 / B2 / B3.
- **``refusal_is_typed``** — every refused case has a non-empty
reason citing the parser or solver layer. No crash-shaped
failures, no silent failures.
- **``report_deterministic``** — two runs over the same case set
produce byte-equal ``train_sample_coverage_report.json``.
- **``adapter_pure``** — ``_adapt_case`` is a pure function from
the train_sample dict shape to the runner shape; no I/O, no
global state.
---
## Acceptance evidence
Accepted when:
- ``evals/gsm8k_math/train_sample/v1/run_coverage_probe.py`` ships
and exits 0 on the current 50-case sample.
- ``evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json``
is committed and matches the script's deterministic output.
- ``tests/test_adr_0131_G_gsm8k_coverage_probe.py`` passes — pins
the safety rail (``admitted_wrong == 0``), the typed-refusal
invariant (every refused case has a non-empty reason), and the
report-byte-equality invariant.
- This ADR is included.
---
## Consequences
- The ``train_sample`` corpus stops being eval-adjacent dead
weight. It becomes a measured progress artifact every iteration
can move.
- Each ADR-0131.G.<n> iteration's PR diff includes a real number
delta in the report's ``admission_rate`` and a real reduction in
the ``refused_reasons_top`` work queue.
- The grammar-expansion work is auditable: each new template comes
with B3-style coverage tests and a documented refusal-reason
family it closes.
- The promotion-gate composite (B1 + B2 + B3) is unaffected.
``ADR-0131.4`` (promotion wiring) consumes those three; it does
NOT consume the GSM8K coverage probe. The probe is empirical
context, not a gate.
---
## Out of scope
- Pushing admission beyond the train sample's 50 cases — sealed
holdout (ADR-0119.7) and public split (ADR-0119.2) remain
separate measurement surfaces.
- Frontier head-to-head on GSM8K — deferred until admission is
characterized enough that a CORE-vs-frontier story is
architecture-aligned rather than treadmill-shaped.
- Promotion-gate amendment — ADR-0131.4 is the place to decide
whether GSM8K admission ever enters the promotion contract.
This ADR deliberately keeps that question open.

129
evals/gsm8k_math/train_sample/v1/run_coverage_probe.py Normal file
View file

@ -0,0 +1,129 @@
"""ADR-0131.G — GSM8K coverage probe.
Wraps the existing :mod:`evals.gsm8k_math.runner` to score the
real-GSM8K train sample (case_id / question / answer_numeric format)
against the bounded grammar + binding graph pipeline. Emits a
``train_sample_coverage_report.json`` that pins:
- ``admission_rate`` — fraction parsed + solved + verifier-passed
(i.e. ``correct``)
- ``wrong`` — must always be 0 (the gate)
- ``refused_rate`` — fraction refused with a typed reason at the
parser or solver layer
- per-case outcomes (audit trail)
- top refused clauses (the work queue for grammar expansion)
The report is committed alongside this script so admission progress
across iterations (G.1, G.2, ...) is a diff-able number, not a
narrative claim.
The probe is deterministic: same case set → byte-equal report.
"""
from __future__ import annotations
import json
from collections import Counter
from pathlib import Path
from typing import Any
from evals.gsm8k_math.runner import run_lane
_HERE = Path(__file__).resolve().parent
_CASES_PATH = _HERE / "cases.jsonl"
_REPORT_PATH = _HERE / "train_sample_coverage_report.json"
def _adapt_case(raw: dict[str, Any]) -> dict[str, Any]:
"""Translate train_sample shape (case_id / question / answer_numeric)
into the lane runner's expected shape (id / problem / expected_answer
/ expected_unit). Train-sample cases carry no unit annotation; the
empty string is used so unit-mismatch verifications don't fire on
answer-only correctness.
"""
return {
"id": raw["case_id"],
"problem": raw["question"],
"expected_answer": float(raw["answer_numeric"]),
"expected_unit": "",
}
def _summarize_refusal_reasons(case_details: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""Bucket refused outcomes by the first 120 chars of the reason
and count duplicates. Returns a deterministically-ordered list of
{count, reason} dicts.
"""
counter: Counter[str] = Counter()
for d in case_details:
if d["outcome"] != "refused":
continue
reason = d["reason"].split("\n")[0][:120]
counter[reason] += 1
return [
{"count": count, "reason": reason}
for reason, count in sorted(
counter.items(),
key=lambda kv: (-kv[1], kv[0]),
)
]
def build_report() -> dict[str, Any]:
raw_cases = [
json.loads(line)
for line in _CASES_PATH.read_text(encoding="utf-8").splitlines()
if line.strip()
]
adapted = [_adapt_case(c) for c in raw_cases]
lane_report = run_lane(adapted)
metrics = dict(lane_report.metrics)
total = metrics["cases_total"]
return {
"schema_version": 1,
"adr": "0131.G",
"probe": "gsm8k_train_sample_coverage",
"sample_path": str(_CASES_PATH.relative_to(_HERE.parent.parent.parent.parent)),
"metrics": {
"cases_total": total,
"admitted_solved": metrics["correct"],
"admitted_wrong": metrics["wrong"],
"refused": metrics["refused"],
"admission_rate": metrics["correct_rate"],
"wrong_rate": metrics["wrong_rate"],
"refused_rate": metrics["refused_rate"],
"wrong_count_is_zero": metrics["wrong_count_is_zero"],
"safety_rail_intact": metrics["wrong_count_is_zero"],
},
"refused_reasons_top": _summarize_refusal_reasons(lane_report.case_details),
"per_case": lane_report.case_details,
}
def write_report(report: dict[str, Any]) -> None:
_REPORT_PATH.write_text(
json.dumps(report, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
def main() -> int:
report = build_report()
write_report(report)
m = report["metrics"]
print(
f"admission_rate: {m['admitted_solved']}/{m['cases_total']} "
f"= {m['admission_rate']:.1%}"
)
print(f"wrong: {m['admitted_wrong']} (gate: must be 0)")
print(
f"refused: {m['refused']}/{m['cases_total']} "
f"= {m['refused_rate']:.1%}"
)
print(f"safety_rail_intact: {m['safety_rail_intact']}")
return 0 if m["wrong_count_is_zero"] else 1
if __name__ == "__main__":
raise SystemExit(main())

771
evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json Normal file
View file

@ -0,0 +1,771 @@
{
"adr": "0131.G",
"metrics": {
"admission_rate": 0.0,
"admitted_solved": 0,
"admitted_wrong": 0,
"cases_total": 50,
"refused": 50,
"refused_rate": 1.0,
"safety_rail_intact": true,
"wrong_count_is_zero": true,
"wrong_rate": 0.0
},
"per_case": [
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0001",
"expected_answer": 990.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Tina makes $18.00 an hour' (in sentence: 'Tina makes $18.00 an hour.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0002",
"expected_answer": 15.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Jan buys 1000 feet of cable' (in sentence: 'Jan buys 1000 feet of cable.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0003",
"expected_answer": 864.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'The student council sells scented erasers in the morning before school starts to help raise money for school dances' (in sentence: 'The student council sells scented erasers in the morning before school starts to help raise money for school dances.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0004",
"expected_answer": 2000.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'There are some kids in camp' (in sentence: 'There are some kids in camp.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0005",
"expected_answer": 21.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature\" (in sentence: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature.\")",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0006",
"expected_answer": 480.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Mandy started reading books with only 8 pages when she was 6 years old' (in sentence: 'Mandy started reading books with only 8 pages when she was 6 years old.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0007",
"expected_answer": 2.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Francine has five full boxes of crayons and 5 loose crayons' (in sentence: 'Francine has five full boxes of crayons and 5 loose crayons, and her friend has 27 loose crayons.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0008",
"expected_answer": 9.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Marnie makes bead bracelets' (in sentence: 'Marnie makes bead bracelets.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0009",
"expected_answer": 185.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Jen has 10 more ducks than four times the number of chickens' (in sentence: 'Jen has 10 more ducks than four times the number of chickens.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0010",
"expected_answer": 9.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Yun had 20 paperclips initially, but then lost 12' (in sentence: 'Yun had 20 paperclips initially, but then lost 12.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0011",
"expected_answer": 50.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup' (in sentence: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0012",
"expected_answer": 7.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: unit 'rocks' not in en_units_v1 and not an allowed count noun",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0013",
"expected_answer": 450.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel' (in sentence: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0014",
"expected_answer": 240.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Bob can shuck 10 oysters in 5 minutes' (in sentence: 'Bob can shuck 10 oysters in 5 minutes.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0015",
"expected_answer": 38.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride' (in sentence: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride, and then bikes the remaining distance for 8 hours.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0016",
"expected_answer": 2.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs\" (in sentence: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs.\")",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0017",
"expected_answer": 800.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Jason has a carriage house that he rents out' (in sentence: 'Jason has a carriage house that he rents out.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0018",
"expected_answer": 16.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Xavier plays football with his friends' (in sentence: 'Xavier plays football with his friends.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0019",
"expected_answer": 660.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'John adopts a dog from a shelter' (in sentence: 'John adopts a dog from a shelter.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0020",
"expected_answer": 130.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Two puppies, two kittens' (in sentence: 'Two puppies, two kittens, and three parakeets were for sale at the pet shop.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0021",
"expected_answer": 450.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'John is lifting weights' (in sentence: 'John is lifting weights.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0022",
"expected_answer": 4800.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish' (in sentence: 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0023",
"expected_answer": 150.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Nicole collected 400 Pokemon cards' (in sentence: 'Nicole collected 400 Pokemon cards.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0024",
"expected_answer": 438.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday' (in sentence: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0025",
"expected_answer": 1200.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Lilibeth and her friends go strawberry picking' (in sentence: 'Lilibeth and her friends go strawberry picking.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0026",
"expected_answer": 6.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Aaron and his brother Carson each saved up $40 to go to dinner' (in sentence: 'Aaron and his brother Carson each saved up $40 to go to dinner.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0027",
"expected_answer": 3840.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: unit 'followers' not in en_units_v1 and not an allowed count noun",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0028",
"expected_answer": 200.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Tom opens an amusement park' (in sentence: 'Tom opens an amusement park.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0029",
"expected_answer": 64.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Fabian bought a brand new computer mouse and keyboard to be able to work from home' (in sentence: 'Fabian bought a brand new computer mouse and keyboard to be able to work from home.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0030",
"expected_answer": 14.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Jake decides to go to the beach for a fun day' (in sentence: 'Jake decides to go to the beach for a fun day.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0031",
"expected_answer": 92.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Jeremie wants to go to an amusement park with 3 friends at the end of summer' (in sentence: 'Jeremie wants to go to an amusement park with 3 friends at the end of summer.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0032",
"expected_answer": 34.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'John decides to take up illustration' (in sentence: 'John decides to take up illustration.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0033",
"expected_answer": 60.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Rachel is 12 years old' (in sentence: 'Rachel is 12 years old, and her grandfather is 7 times her age.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0034",
"expected_answer": 112.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Georgie is a varsity player on a football team' (in sentence: 'Georgie is a varsity player on a football team.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0035",
"expected_answer": 4.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'She decided to split them among her friends' (in sentence: 'She decided to split them among her friends.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0036",
"expected_answer": 22.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Monica way studying for an exam' (in sentence: 'Monica way studying for an exam.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0037",
"expected_answer": 3.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Michael wants to lose 10 pounds by June' (in sentence: 'Michael wants to lose 10 pounds by June.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0038",
"expected_answer": 400.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'In a building, there are a hundred ladies on the first-floor studying' (in sentence: 'In a building, there are a hundred ladies on the first-floor studying.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0039",
"expected_answer": 20.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'At the family reunion, everyone ate too much food and gained weight' (in sentence: 'At the family reunion, everyone ate too much food and gained weight.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0040",
"expected_answer": 72.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Over several years, Daniel has adopted any stray animals he sees on the side of the road' (in sentence: 'Over several years, Daniel has adopted any stray animals he sees on the side of the road.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0041",
"expected_answer": 6.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Troy bakes 2 pans of brownies, cut into 16 pieces per pan' (in sentence: 'Troy bakes 2 pans of brownies, cut into 16 pieces per pan.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0042",
"expected_answer": 30.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in each bag' (in sentence: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in each bag.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0043",
"expected_answer": 11.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Sandra wants to buy some sweets' (in sentence: 'Sandra wants to buy some sweets.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0044",
"expected_answer": 1300.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'John invests in a bank and gets 10% simple interest' (in sentence: 'John invests in a bank and gets 10% simple interest.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0045",
"expected_answer": 14.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Bart fills out surveys to earn money' (in sentence: 'Bart fills out surveys to earn money.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0046",
"expected_answer": 15.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'A school has 100 students' (in sentence: 'A school has 100 students.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0047",
"expected_answer": 45.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'John bakes 12 coconut macaroons, each weighing 5 ounces' (in sentence: 'John bakes 12 coconut macaroons, each weighing 5 ounces.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0048",
"expected_answer": 4.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Jed collects stamp cards' (in sentence: 'Jed collects stamp cards.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0049",
"expected_answer": 18.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Malcolm is trying to find the fastest walk to school and is currently comparing two routes' (in sentence: 'Malcolm is trying to find the fastest walk to school and is currently comparing two routes.')",
"trace_hash": null
},
{
"actual_answer": null,
"actual_unit": null,
"case_id": "gsm8k-train-sample-v1-0050",
"expected_answer": 280.0,
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Mark does a gig every other day for 2 weeks' (in sentence: 'Mark does a gig every other day for 2 weeks.')",
"trace_hash": null
}
],
"probe": "gsm8k_train_sample_coverage",
"refused_reasons_top": [
{
"count": 1,
"reason": "parser: could not parse statement clause: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temp"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encoun"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'A school has 100 students' (in sentence: 'A school has 100 students.')"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Aaron and his brother Carson each saved up $40 to go to dinner' (in sentence:"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup' (in s"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to h"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'At the family reunion, everyone ate too much food and gained weight' (in sent"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Bart fills out surveys to earn money' (in sentence: 'Bart fills out surveys t"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Bob can shuck 10 oysters in 5 minutes' (in sentence: 'Bob can shuck 10 oyster"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in eac"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Erica lives near a lake where most locals sell fish as their main source of i"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Fabian bought a brand new computer mouse and keyboard to be able to work from"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Francine has five full boxes of crayons and 5 loose crayons' (in sentence: 'F"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Georgie is a varsity player on a football team' (in sentence: 'Georgie is a v"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'In a building, there are a hundred ladies on the first-floor studying' (in se"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Jake decides to go to the beach for a fun day' (in sentence: 'Jake decides to"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Jan buys 1000 feet of cable' (in sentence: 'Jan buys 1000 feet of cable.')"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Jason has a carriage house that he rents out' (in sentence: 'Jason has a carr"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Jed collects stamp cards' (in sentence: 'Jed collects stamp cards.')"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Jen has 10 more ducks than four times the number of chickens' (in sentence: '"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Jeremie wants to go to an amusement park with 3 friends at the end of summer'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'John adopts a dog from a shelter' (in sentence: 'John adopts a dog from a she"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'John bakes 12 coconut macaroons, each weighing 5 ounces' (in sentence: 'John "
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'John decides to take up illustration' (in sentence: 'John decides to take up "
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'John invests in a bank and gets 10% simple interest' (in sentence: 'John inve"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'John is lifting weights' (in sentence: 'John is lifting weights.')"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Lilibeth and her friends go strawberry picking' (in sentence: 'Lilibeth and h"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Malcolm is trying to find the fastest walk to school and is currently compari"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Mandy started reading books with only 8 pages when she was 6 years old' (in s"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Mark does a gig every other day for 2 weeks' (in sentence: 'Mark does a gig e"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Marnie makes bead bracelets' (in sentence: 'Marnie makes bead bracelets.')"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Michael wants to lose 10 pounds by June' (in sentence: 'Michael wants to lose"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Monica way studying for an exam' (in sentence: 'Monica way studying for an ex"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Nicole collected 400 Pokemon cards' (in sentence: 'Nicole collected 400 Pokem"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Over several years, Daniel has adopted any stray animals he sees on the side "
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Rachel is 12 years old' (in sentence: 'Rachel is 12 years old, and her grandf"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Sandra wants to buy some sweets' (in sentence: 'Sandra wants to buy some swee"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'She decided to split them among her friends' (in sentence: 'She decided to sp"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday' (in s"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'The student council sells scented erasers in the morning before school starts"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'There are some kids in camp' (in sentence: 'There are some kids in camp.')"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Tina makes $18.00 an hour' (in sentence: 'Tina makes $18.00 an hour.')"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Tom opens an amusement park' (in sentence: 'Tom opens an amusement park.')"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, "
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Troy bakes 2 pans of brownies, cut into 16 pieces per pan' (in sentence: 'Tro"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Two puppies, two kittens' (in sentence: 'Two puppies, two kittens, and three "
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Xavier plays football with his friends' (in sentence: 'Xavier plays football "
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Yun had 20 paperclips initially, but then lost 12' (in sentence: 'Yun had 20 "
},
{
"count": 1,
"reason": "parser: unit 'followers' not in en_units_v1 and not an allowed count noun"
},
{
"count": 1,
"reason": "parser: unit 'rocks' not in en_units_v1 and not an allowed count noun"
}
],
"sample_path": "evals/gsm8k_math/train_sample/v1/cases.jsonl",
"schema_version": 1
}

144
tests/test_adr_0131_G_gsm8k_coverage_probe.py Normal file
View file

@ -0,0 +1,144 @@
"""ADR-0131.G — GSM8K coverage probe contract tests.
Pins the invariants that every iteration (G.1, G.2, ...) must
preserve. These tests fail loudly if a future grammar expansion
relaxes the safety rail or stops emitting typed refusals.
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from evals.gsm8k_math.train_sample.v1.run_coverage_probe import (
_adapt_case,
build_report,
write_report,
)
_LANE_ROOT = (
Path(__file__).resolve().parent.parent
/ "evals" / "gsm8k_math" / "train_sample" / "v1"
)
_CASES_PATH = _LANE_ROOT / "cases.jsonl"
_REPORT_PATH = _LANE_ROOT / "train_sample_coverage_report.json"
# ---------------------------------------------------------------------------
# Adapter purity
# ---------------------------------------------------------------------------
def test_adapt_case_is_pure_function() -> None:
raw = {
"case_id": "gsm8k-train-sample-v1-9999",
"question": "Sam has 5 apples.",
"answer_numeric": 5,
"extra": "ignored",
}
a = _adapt_case(raw)
b = _adapt_case(raw)
assert a == b
assert a == {
"id": "gsm8k-train-sample-v1-9999",
"problem": "Sam has 5 apples.",
"expected_answer": 5.0,
"expected_unit": "",
}
# input untouched
assert "id" not in raw and "problem" not in raw
# ---------------------------------------------------------------------------
# Safety rail — the non-negotiable invariant
# ---------------------------------------------------------------------------
def test_admitted_wrong_is_zero() -> None:
"""``wrong == 0`` is the gate. If a future grammar expansion
confabulates an answer rather than refusing, this test fails
and the expansion must be reverted."""
report = build_report()
assert report["metrics"]["admitted_wrong"] == 0, (
"safety rail violated: a confabulated answer reached the "
"verifier. Revert the most recent grammar expansion."
)
assert report["metrics"]["safety_rail_intact"] is True
def test_every_refused_case_has_typed_reason() -> None:
"""Refusal must be first-class with a non-empty typed reason.
Empty refusal reasons indicate a crash-shaped failure path
that escaped typed error handling."""
report = build_report()
for case in report["per_case"]:
if case["outcome"] == "refused":
assert isinstance(case["reason"], str)
assert case["reason"].strip(), (
f"case {case['case_id']} refused with empty reason"
)
def test_per_case_outcomes_are_in_closed_vocabulary() -> None:
report = build_report()
allowed = {"correct", "wrong", "refused"}
for case in report["per_case"]:
assert case["outcome"] in allowed, case
# ---------------------------------------------------------------------------
# Deterministic replay
# ---------------------------------------------------------------------------
def test_report_is_deterministic_across_runs() -> None:
a = build_report()
b = build_report()
assert (
json.dumps(a, sort_keys=True)
== json.dumps(b, sort_keys=True)
)
def test_committed_report_matches_current_run(tmp_path: Path) -> None:
"""The report committed to the repo must match what the script
produces from the current cases.jsonl. If this fails, either
re-run the probe and commit the new report, or revert the
cases.jsonl change."""
if not _REPORT_PATH.exists():
pytest.skip("baseline report not committed yet")
committed = json.loads(_REPORT_PATH.read_text(encoding="utf-8"))
fresh = build_report()
assert (
json.dumps(committed, sort_keys=True)
== json.dumps(fresh, sort_keys=True)
)
# ---------------------------------------------------------------------------
# Schema shape
# ---------------------------------------------------------------------------
def test_report_schema_required_fields() -> None:
report = build_report()
required_top = {
"schema_version", "adr", "probe", "sample_path",
"metrics", "refused_reasons_top", "per_case",
}
assert required_top.issubset(report.keys())
required_metrics = {
"cases_total", "admitted_solved", "admitted_wrong",
"refused", "admission_rate", "wrong_rate", "refused_rate",
"wrong_count_is_zero", "safety_rail_intact",
}
assert required_metrics.issubset(report["metrics"].keys())
def test_refused_reasons_top_is_sorted_by_count_desc() -> None:
report = build_report()
counts = [r["count"] for r in report["refused_reasons_top"]]
assert counts == sorted(counts, reverse=True)