5b4dcb17ca
ADR-0163 Phase A measurement. Reads the GSM8K train-sample refusal report
(50 cases, all refused on candidate-graph admissibility) and emits a
histogram of statement shapes. Read-only: no corpus, pack, or proposal
mutation; the categorizer is rules-only with no LLM, embedding, or
learned model.
Lane: evals/refusal_taxonomy/ (auto-discovered by evals.framework)
- shape_categories.py — ShapeCategory enum + deterministic categorizer
(9 ADR-mandated baseline categories + UNCATEGORIZED, first-match-wins)
- runner.py — pure run_lane(cases) -> LaneReport
- contract.md — purpose, doctrine, schema, ADR compatibility
- public/v1/cases.jsonl — 50 refused statements (sorted by case_id)
- v1/report.json — first run output (categorized_rate=72%)
CLI: core teaching refusal-taxonomy [--input PATH] [--json] [--save]
Accepts a cases JSONL or a raw GSM8K eval report.json directly.
Helper: scripts/build_refusal_taxonomy_cases.py rebuilds the v1 case set
from the GSM8K train-sample report deterministically.
Tests: tests/test_refusal_taxonomy_lane.py (21 passing) cover schema
integrity, lane auto-discovery, enum exhaustiveness, categorizer
determinism + purity + no-ML-imports, histogram correctness, replay
byte-identity, committed report match, helper extraction, and a
read-only invariant snapshot over teaching/, packs/, language_packs/data/.
v1 histogram (50-case sample):
17 descriptive_setup_no_quantity
14 uncategorized
4 temporal_aggregation
3 rate_with_currency
3 fractional_rate_of_change
3 indefinite_quantity
3 comparative_with_unit
2 nested_question_target
1 unit_partition
0 conditional_quantity
total=50 categorized_rate=72% uncategorized=28% (below 50% target)
Top three by count (Phase B candidates):
1. descriptive_setup_no_quantity (17)
2. temporal_aggregation (4)
3. tie at 3 — operator selects from {rate_with_currency,
fractional_rate_of_change, indefinite_quantity, comparative_with_unit}
Phase B is not started in this PR — the ADR explicitly requires the
operator to ratify the top-N selection before any exemplar corpus is
authored.
Invariants verified:
- tests/test_adr_0131_*.py: 224 passed, 0 wrong on G1..G5 + S1
- core test --suite smoke -q: 67 passed
- The refusal_taxonomy/__init__.py and runner do not import openai,
anthropic, transformers, torch, sklearn, sentence_transformers,
requests, or httpx — verified by test_categorizer_no_llm_or_ml_imports.
Cross-references: ADR-0163 (parent), ADR-0114a (capability obligations),
ADR-0149 (recognizer pipeline substrate that Phases C–E build on).
Refs: [[thesis-decoding-not-generating]] — the rules-only categorizer
honors the doctrine: the engine learns to find better shapes; this PR
does not stuff it with another found pattern.
96 lines
2.8 KiB
Python
96 lines
2.8 KiB
Python
"""ADR-0163 Phase A — build the refusal_taxonomy v1 case set.
|
|
|
|
Reads ``evals/gsm8k_math/train_sample/v1/report.json`` and emits one JSONL
|
|
record per refused case, extracting the embedded statement out of the
|
|
``refusal_reason`` string so the lane operates on the statement itself
|
|
(not on the reason envelope).
|
|
|
|
The output is deterministic: cases are sorted by ``case_id`` and serialized
|
|
with ``sort_keys=True`` and ``ensure_ascii=False``.
|
|
|
|
Usage::
|
|
|
|
uv run python scripts/build_refusal_taxonomy_cases.py \\
|
|
--report evals/gsm8k_math/train_sample/v1/report.json \\
|
|
--out evals/refusal_taxonomy/v1/cases.jsonl
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import json
|
|
import re
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
_STATEMENT_RE = re.compile(
|
|
r"^candidate_graph:\s*no admissible candidate for\s+"
|
|
r"(?:statement|question):\s*['\"](.+)['\"]\s*$",
|
|
re.DOTALL,
|
|
)
|
|
|
|
|
|
def extract_statement(reason: str) -> str | None:
|
|
"""Pull the embedded statement out of a refusal reason.
|
|
|
|
Returns ``None`` if the reason does not match the expected envelope.
|
|
"""
|
|
|
|
match = _STATEMENT_RE.match(reason.strip())
|
|
if not match:
|
|
return None
|
|
return match.group(1).strip()
|
|
|
|
|
|
def build_cases(report_path: Path) -> list[dict[str, str]]:
|
|
payload = json.loads(report_path.read_text())
|
|
per_case = payload.get("per_case", [])
|
|
out: list[dict[str, str]] = []
|
|
for case in per_case:
|
|
if case.get("verdict") != "refused":
|
|
continue
|
|
reason = case.get("reason", "")
|
|
statement = extract_statement(reason)
|
|
if statement is None:
|
|
continue
|
|
out.append({
|
|
"case_id": case["case_id"],
|
|
"statement": statement,
|
|
"refusal_reason": reason,
|
|
})
|
|
out.sort(key=lambda r: r["case_id"])
|
|
return out
|
|
|
|
|
|
def main(argv: list[str] | None = None) -> int:
|
|
parser = argparse.ArgumentParser(description=__doc__)
|
|
parser.add_argument(
|
|
"--report",
|
|
type=Path,
|
|
default=Path("evals/gsm8k_math/train_sample/v1/report.json"),
|
|
help="path to a GSM8K eval report containing refused cases",
|
|
)
|
|
parser.add_argument(
|
|
"--out",
|
|
type=Path,
|
|
default=Path("evals/refusal_taxonomy/public/v1/cases.jsonl"),
|
|
help="output JSONL path",
|
|
)
|
|
args = parser.parse_args(argv)
|
|
|
|
if not args.report.exists():
|
|
parser.error(f"report not found: {args.report}")
|
|
|
|
cases = build_cases(args.report)
|
|
args.out.parent.mkdir(parents=True, exist_ok=True)
|
|
with args.out.open("w", encoding="utf-8") as handle:
|
|
for record in cases:
|
|
handle.write(json.dumps(record, ensure_ascii=False, sort_keys=True))
|
|
handle.write("\n")
|
|
|
|
print(f"wrote {len(cases)} cases to {args.out}", file=sys.stderr)
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|