core/docs/refusal-taxonomy.md
Shay 310aed9ff0
chore: Refactor CLI and Governance Anchors (#926)
* docs: consolidate governance anchors and clean up test registries

* refactor(cli): decompose cli into dedicated modules

* test: fix broken test baselines and formatting

* docs: add domain boundary READMEs for governance anchors

* test: update baseline for determination lane

* test: fix capability_pass expectation

* test: fix CORE_SHOWCASE_SKIP_BUDGET enforcement

* chore: cleanup CLI extraction and unreachable code
2026-07-03 12:34:56 -07:00

3.7 KiB

Refusal Taxonomy — ADR-0163 Phase A

The refusal-taxonomy lane categorises every refused GSM8K statement by statement shape, not by content. It is the load-bearing measurement that gates Phase B of ADR-0163: the top categories by count become the operator's input list for hand-authored exemplar corpora.

The lane is read-only. It does not mutate the active corpus, packs, language packs, or proposal log. The categorizer is rules-only — no LLM call, no embedding, no learned classifier — per ADR-0163 §Constraint #4 and CLAUDE.md.

v1 histogram (50-case GSM8K train sample)

Count Category
17 descriptive_setup_no_quantity
14 uncategorized
4 temporal_aggregation
3 rate_with_currency
3 fractional_rate_of_change
3 indefinite_quantity
3 comparative_with_unit
2 nested_question_target
1 unit_partition
0 conditional_quantity

Total: 50. Uncategorized rate: 28%. Categorized rate: 72%.

Case digest: d030f826cb0f4088771d90c52c8be2ff75054ab27c7d47eae8dbfe1225b2eea1.

Top three by count (Phase B candidates)

  1. descriptive_setup_no_quantity (17) — statements with no extractable measurement at all. The candidate-graph needs to admit pure-context lines as setup rather than refusing them.
  2. temporal_aggregation (4) — repeated/aggregated time framing ("each day", "every other day for 2 weeks", "in 5 minutes", day-of-week enumeration).
  3. Tie at 3rate_with_currency, fractional_rate_of_change, indefinite_quantity, comparative_with_unit. The operator selects which member of the tie (or combination) to seed in Phase B's first round; the agent does not make that call.

ADR-0163 §Phase B ratchet is "three categories per round". The 17/4 spread suggests Round 1 should anchor on descriptive_setup_no_quantity plus two of the tied trio.

Reading the uncategorized tail (14)

uncategorized is honest measurement, not failure. The 14 statements that no rule fires for share these emergent sub-shapes (not yet promoted to first-class categories — none has ≥ 3 instances individually):

  • bare declarative quantity — "Nicole collected 400 Pokemon cards." / "A school has 100 students." / "Malcolm has 240 followers on Instagram and 500 followers on Facebook."
  • distributive — "each saved up $40" / "each weighing 5 ounces"
  • sequential change — "had 20 paperclips initially, lost 12"
  • word-number enumeration — "Two puppies, two kittens, and three parakeets" / "a hundred ladies"
  • percentage-rate without change verb — "10% simple interest"
  • quantity embedded in narrative — "3 friends at the end of summer" / "lose 10 pounds by June"

The operator decides whether any of these warrant promotion to a first-class category in v2 of the taxonomy (each promotion requires ≥ 3 cited refused statements per ADR-0163 §Risks).

How to re-run

# Run via the eval framework (uses the standard public/v1/cases.jsonl)
uv run core eval refusal_taxonomy

# Run via the teaching CLI on an arbitrary refused-case set
uv run core teaching refusal-taxonomy \
    --input evals/gsm8k_math/train_sample/v1/report.json

# Regenerate the v1 cases.jsonl from the source GSM8K report
uv run python scripts/build_refusal_taxonomy_cases.py

Phase A boundary (what this lane does NOT do)

  • It does not add or modify recognizers.
  • It does not author exemplar corpora (that is Phase B).
  • It does not emit recognizer proposals (that is Phase C).
  • It does not change the GSM8K correct/refused/wrong counts. Per ADR-0114a, the wrong = 0 invariant on G1..G5 and S1 is unchanged by this work.