feat(ADR-0163.A): refusal taxonomy lane — shape categorization of GSM8K admissibility gaps (#297 )

ADR-0163 Phase A measurement. Reads the GSM8K train-sample refusal report
(50 cases, all refused on candidate-graph admissibility) and emits a
histogram of statement shapes. Read-only: no corpus, pack, or proposal
mutation; the categorizer is rules-only with no LLM, embedding, or
learned model.

Lane: evals/refusal_taxonomy/ (auto-discovered by evals.framework)
  - shape_categories.py — ShapeCategory enum + deterministic categorizer
    (9 ADR-mandated baseline categories + UNCATEGORIZED, first-match-wins)
  - runner.py           — pure run_lane(cases) -> LaneReport
  - contract.md         — purpose, doctrine, schema, ADR compatibility
  - public/v1/cases.jsonl — 50 refused statements (sorted by case_id)
  - v1/report.json        — first run output (categorized_rate=72%)

CLI: core teaching refusal-taxonomy [--input PATH] [--json] [--save]
     Accepts a cases JSONL or a raw GSM8K eval report.json directly.

Helper: scripts/build_refusal_taxonomy_cases.py rebuilds the v1 case set
from the GSM8K train-sample report deterministically.

Tests: tests/test_refusal_taxonomy_lane.py (21 passing) cover schema
integrity, lane auto-discovery, enum exhaustiveness, categorizer
determinism + purity + no-ML-imports, histogram correctness, replay
byte-identity, committed report match, helper extraction, and a
read-only invariant snapshot over teaching/, packs/, language_packs/data/.

v1 histogram (50-case sample):
   17  descriptive_setup_no_quantity
   14  uncategorized
    4  temporal_aggregation
    3  rate_with_currency
    3  fractional_rate_of_change
    3  indefinite_quantity
    3  comparative_with_unit
    2  nested_question_target
    1  unit_partition
    0  conditional_quantity
total=50  categorized_rate=72%  uncategorized=28% (below 50% target)

Top three by count (Phase B candidates):
  1. descriptive_setup_no_quantity (17)
  2. temporal_aggregation (4)
  3. tie at 3 — operator selects from {rate_with_currency,
     fractional_rate_of_change, indefinite_quantity, comparative_with_unit}

Phase B is not started in this PR — the ADR explicitly requires the
operator to ratify the top-N selection before any exemplar corpus is
authored.

Invariants verified:
  - tests/test_adr_0131_*.py: 224 passed, 0 wrong on G1..G5 + S1
  - core test --suite smoke -q: 67 passed
  - The refusal_taxonomy/__init__.py and runner do not import openai,
    anthropic, transformers, torch, sklearn, sentence_transformers,
    requests, or httpx — verified by test_categorizer_no_llm_or_ml_imports.

Cross-references: ADR-0163 (parent), ADR-0114a (capability obligations),
ADR-0149 (recognizer pipeline substrate that Phases C–E build on).

Refs: [[thesis-decoding-not-generating]] — the rules-only categorizer
honors the doctrine: the engine learns to find better shapes; this PR
does not stuff it with another found pattern.

2026-05-26 11:27:11 -07:00

3.7 KiB

Raw Blame History

Refusal Taxonomy — ADR-0163 Phase A

The refusal-taxonomy lane categorises every refused GSM8K statement by statement shape, not by content. It is the load-bearing measurement that gates Phase B of ADR-0163: the top categories by count become the operator's input list for hand-authored exemplar corpora.

The lane is read-only. It does not mutate the active corpus, packs, language packs, or proposal log. The categorizer is rules-only — no LLM call, no embedding, no learned classifier — per ADR-0163 §Constraint #4 and CLAUDE.md.

v1 histogram (50-case GSM8K train sample)

Count	Category
17	`descriptive_setup_no_quantity`
14	`uncategorized`
4	`temporal_aggregation`
3	`rate_with_currency`
3	`fractional_rate_of_change`
3	`indefinite_quantity`
3	`comparative_with_unit`
2	`nested_question_target`
1	`unit_partition`
0	`conditional_quantity`

Total: 50. Uncategorized rate: 28%. Categorized rate: 72%.

Case digest: d030f826cb0f4088771d90c52c8be2ff75054ab27c7d47eae8dbfe1225b2eea1.

Top three by count (Phase B candidates)

descriptive_setup_no_quantity (17) — statements with no extractable measurement at all. The candidate-graph needs to admit pure-context lines as setup rather than refusing them.
temporal_aggregation (4) — repeated/aggregated time framing ("each day", "every other day for 2 weeks", "in 5 minutes", day-of-week enumeration).
Tie at 3 — rate_with_currency, fractional_rate_of_change, indefinite_quantity, comparative_with_unit. The operator selects which member of the tie (or combination) to seed in Phase B's first round; the agent does not make that call.

ADR-0163 §Phase B ratchet is "three categories per round". The 17/4 spread suggests Round 1 should anchor on descriptive_setup_no_quantity plus two of the tied trio.

Reading the uncategorized tail (14)

uncategorized is honest measurement, not failure. The 14 statements that no rule fires for share these emergent sub-shapes (not yet promoted to first-class categories — none has ≥ 3 instances individually):

bare declarative quantity — "Nicole collected 400 Pokemon cards." / "A school has 100 students." / "Malcolm has 240 followers on Instagram and 500 followers on Facebook."
distributive — "each saved up $40" / "each weighing 5 ounces"
sequential change — "had 20 paperclips initially, lost 12"
word-number enumeration — "Two puppies, two kittens, and three parakeets" / "a hundred ladies"
percentage-rate without change verb — "10% simple interest"
quantity embedded in narrative — "3 friends at the end of summer" / "lose 10 pounds by June"

The operator decides whether any of these warrant promotion to a first-class category in v2 of the taxonomy (each promotion requires ≥ 3 cited refused statements per ADR-0163 §Risks).

How to re-run

# Run via the eval framework (uses the standard public/v1/cases.jsonl)
uv run core eval refusal_taxonomy

# Run via the teaching CLI on an arbitrary refused-case set
uv run core teaching refusal-taxonomy \
    --input evals/gsm8k_math/train_sample/v1/report.json

# Regenerate the v1 cases.jsonl from the source GSM8K report
uv run python scripts/build_refusal_taxonomy_cases.py

Phase A boundary (what this lane does NOT do)

It does not add or modify recognizers.
It does not author exemplar corpora (that is Phase B).
It does not emit recognizer proposals (that is Phase C).
It does not change the GSM8K correct/refused/wrong counts. Per ADR-0114a, the wrong = 0 invariant on G1..G5 and S1 is unchanged by this work.

3.7 KiB Raw Blame History