core/evals/refusal_taxonomy/contract.md

Refusal Taxonomy Eval Contract (ADR-0163 Phase A)

Purpose

refusal-taxonomy is a read-only evaluation lane that categorises refused GSM8K statements by statement shape, not by content. The histogram it emits is the load-bearing measurement that gates Phase B of ADR-0163: the top categories by count become the input list for hand-authored exemplar corpora.

Phase A produces no recognizers and no corpus changes. Its sole job is to turn a flat refusal list into a measured distribution of shapes so the operator can choose what to teach next.

Source of truth

The v1 case set is derived from:

evals/gsm8k_math/train_sample/v1/report.json

— every record whose verdict == "refused", with the embedded statement extracted out of the reason envelope. The case set is rebuilt deterministically by scripts/build_refusal_taxonomy_cases.py; the script's output is sorted by case_id and committed to evals/refusal_taxonomy/public/v1/cases.jsonl (the framework-standard public-split location). The Phase A v1 report artifact lives at evals/refusal_taxonomy/v1/report.json.

Non-goals

This lane MUST NOT:

• mutate any corpus, pack, or language pack
• write to engine_state
• create, accept, or reject proposals
• call an LLM, embedding model, or any learned classifier
• alter the GSM8K refusal counts elsewhere in the repo

The lane only reads its own cases.jsonl (or an operator-supplied refused-case set) and emits a histogram.

Categorizer doctrine

Per ADR-0163 §Constraint #4 and CLAUDE.md, the categorizer is rules-only:

• No LLM call, no embedding, no learned model.
• Deterministic — the output is a pure function of the input string.
• No hidden normalization — lowercasing/padding is only for substring word-boundary safety; the original statement is never mutated for downstream consumers.
• First-match-wins. Priority order is fixed in shape_categories.py.
• UNCATEGORIZED is a first-class outcome. It is honest measurement, not a failure.

Adding a new category requires citing ≥ 3 refused statements as evidence in the category's docstring. This is enforced by tests/test_refusal_taxonomy_lane.py.

Shape categories (v1)

The nine baseline categories named in ADR-0163 §Phase A, plus uncategorized:

Category	Definition
nested_question_target	"If X, how many/much Y …?"
unit_partition	hyphenated unit, e.g. "25-foot sections"
rate_with_currency	$N + per-unit framing (per hour, /kg, for one cup)
comparative_with_unit	"more than", "twice as", "N times", "as many as"
fractional_rate_of_change	fraction or % paired with a change-of-state verb
indefinite_quantity	"some", "several", "a few", "many", "any"
temporal_aggregation	"each day", "every X", "in N minutes", day-of-week enumeration
conditional_quantity	bare "If X, would/will Y" without a how-target
descriptive_setup_no_quantity	no digit, no number-word, no quantifier
uncategorized	none of the above

Case set schema

Each line in cases.jsonl is a JSON object:

{"case_id": "gsm8k-train-sample-v1-NNNN",
 "statement": "<the refused statement>",
 "refusal_reason": "candidate_graph: no admissible candidate for ..."}

Lane output

run_lane(cases) returns a LaneReport with:

metrics = {
    "total": <int>,
    "by_category": {category_value: count, ...},     # every category present
    "uncategorized": <int>,
    "categorized_rate": <float in [0, 1]>,
    "case_digest": <sha256 of canonical case_details>,
}
case_details = [
    {"case_id", "statement", "shape_category", "refusal_reason"},
    ...
]

The histogram includes every category from SHAPE_CATEGORY_ORDER — counts of zero are reported explicitly, not omitted.

Replay & determinism

For a fixed cases.jsonl at a fixed commit SHA, run_lane returns the same metrics and case_details bit-for-bit. case_digest is a sha256 over the canonical-JSON serialization of case_details and acts as a single integrity hash for downstream tooling.

ADR compatibility

This lane preserves:

• ADR-0163 Phase A boundary — measurement only, no corpus mutation
• ADR-0114a wrong = 0 invariant — the lane scores shape, not correctness, so the GSM8K and capability-axis wrong counts elsewhere in the repo are unaffected
• CLAUDE.md non-negotiables — no hidden normalization, no LLM fallback, deterministic replay