* chore(ADR-0163.C): land three Phase C pending proposals in live log Phase C (#301) shipped the CLI but its PR dry-run wrote to a tmp log path. This commit moves the three Phase C proposals into the live teaching/proposals/proposals.jsonl so the Phase B→C audit trail is visible in the proposal log and the proposals are ready for the operator to ratify after Phase D ships. Proposals (all state=pending, kind="exemplar_corpus"): - 59223f13722f906a1cf9b65d9b01c990 — descriptive_setup_no_quantity - 46ce297f797ff16da12db5de422ca3c9 — rate_with_currency - a3b892546977c5f0f64c578d6052adbd — temporal_aggregation Produced by `core teaching propose-from-exemplars --all` against the live Phase B corpora. No ratification (ADR-0161 §5 — only the repo owner ratifies). The Phase D admissibility-replay gate confirmed replay_equivalent=true, wrong_count_delta=0 for all three. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(ADR-0163.D): wire ratified RecognizerSpecs into math_candidate_graph admissibility surface Phase D is the first PR to extend the math admission surface. The audit (#294) said the gap was admission, not operators, algebra, substrate, or packs. Phase A measured the refusal taxonomy. Phase B authored seeds. Phase C synthesized recognizers. Phase D wires those recognizers into generate/math_candidate_graph.py. Modules - generate/recognizer_registry.py — pure projection over the proposal log. Only proposals with source.kind="exemplar_corpus" AND review_state="accepted" enter the tuple. Sorted by (review_date, proposal_id). In-process cache keyed on log (mtime, sha256) — no filesystem cache (ADR-0161 §1). Malformed accepted specs raise RegistryLoadError citing the offending proposal_id; silent drops are forbidden. - generate/recognizer_match.py — per-category rules-only matchers (no LLM, no embedding, no learned classifier). Honors the Phase C synthesizer's narrowness rule: out-of-corpus currency symbols, window units, and per-unit values do NOT match. Three matchers: _match_descriptive_setup_no_quantity (zero-quantity surface), _match_temporal_aggregation (event_count_per_window with observed_window_units/quantifiers honored), _match_rate_with_currency (currency_per_unit_rate with observed currency/per-unit/amount-kind honored). - generate/math_candidate_graph.py — narrowest-edit guard at the per-statement choice loop. Before the existing "no admissible candidate for statement" refusal, consult the ratified registry. Recognized statements are dropped from per_sentence_choices (zero math state) so the Cartesian product is identical to "this statement was never there." Empty registry is a no-op — backward compatibility preserved byte-identically. Downstream consumption of parsed_anchors (turning recognized rate/temporal surfaces into solver state that produces concrete answers) is Phase E follow-up. Tests (32 new) - tests/_phase_d_fixture.py — synthetic in-memory ratified registry built from the three Phase C pending proposals' content. Per ADR-0161 §5 the agent does NOT ratify the live log; the synthetic registry round-trips the real RecognizerSpec bytes the operator will ratify after Phase D ships. - tests/test_recognizer_registry.py (9) — empty/pending/wrong-kind filtering, sort order, malformed-spec rejection, cache hit + invalidation, live-log Phase C audit check. - tests/test_recognizer_match.py (14) — per-category positive cases, narrowness (out-of-corpus surface forms rejected), no-LLM import check. - tests/test_candidate_graph_recognizer_wiring.py (7) — empty registry preserves existing refusal; synthetic registry: recognized statements no longer trigger per-statement refusal; wrong_count_delta == 0 on GSM8K train_sample; capability axes G1.. G5+S1 wrong=0 unchanged; per-category admission counts on the refused-set; unrecognized statements still refuse with the existing reason. - tests/test_phase_d_replay_evidence.py (2) — full admissibility replay gate under synthetic registry: replay_equivalent=true, wrong_count_delta=0, every capability axis wrong=0; each ratified recognizer admits >= 1 train_sample statement (wiring is consequential). Per-category fixture-based admission counts (synthetic registry vs GSM8K train_sample refused-set sentences): - descriptive_setup_no_quantity: 40 - rate_with_currency: 2 - temporal_aggregation: 7 Narrowness-invariant negative case results (matcher correctly returns None on out-of-corpus / load-bearing-math surfaces): - rate_with_currency: "She paid $5 for the book." (no per-unit) - temporal_aggregation: "On Saturday she went to the store." (single day token) - descriptive_setup_no_quantity: "There are some kids in camp." (indefinite quantifier) Candidates for Phase B round 2 (3 of 20 temporal seeds match the spec's structural commitment but not my surface regex — author_notes explicitly flagged these as schema-gap edge cases): - ta-v1-0004 "Mark does a gig every other day for 2 weeks." - ta-v1-0012 "Robin walks 4 dogs every other day around the park." - ta-v1-0019 "The pump fills the tank with 80 gallons over 6 hours." Three landed wirings DO NOT shift the GSM8K train_sample baseline counts under fixture (correct=3, wrong=0, refused=47 unchanged) — Phase D's narrow wiring is wrong=0 safe by construction; lift to "correct" requires Phase E's downstream parser-side consumption of parsed_anchors. Capability axes G1..G5+S1 wrong=0 unchanged. Cross-refs: ADR-0163 (Phase D), ADR-0057 (proposal review), ADR-0151 (auto-proposal), ADR-0161 §5 (ratification boundary), Phase A PR #297, Phase B PR #298, Phase C PR #301. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
212 lines
7.9 KiB
Python
212 lines
7.9 KiB
Python
"""ADR-0163 Phase D — recognizer_match tests.
|
|
|
|
Pins:
|
|
- per-category match: positive hits, negative misses
|
|
- narrowness: out-of-corpus surfaces return None
|
|
- parsed_anchors carry real extracted values for admissible categories
|
|
- parsed_anchors is empty for descriptive_setup_no_quantity
|
|
- determinism: same (statement, registry) -> same result
|
|
- module-import no-LLM-no-ML test (mirror Phase A/C)
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from evals.refusal_taxonomy.shape_categories import ShapeCategory
|
|
from generate.recognizer_match import RecognizerMatch, match
|
|
from generate.recognizer_registry import RatifiedRecognizer
|
|
from tests._phase_d_fixture import build_synthetic_registry
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def registry() -> tuple[RatifiedRecognizer, ...]:
|
|
return build_synthetic_registry()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Positive matches per category
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_rate_with_currency_matches_canonical_surface(
|
|
registry: tuple[RatifiedRecognizer, ...],
|
|
) -> None:
|
|
m = match("Tina makes $18.00 an hour.", registry)
|
|
assert m is not None
|
|
assert m.category is ShapeCategory.RATE_WITH_CURRENCY
|
|
assert m.outcome == "admissible"
|
|
assert m.graph_intent == "rate"
|
|
assert len(m.parsed_anchors) == 1
|
|
a = m.parsed_anchors[0]
|
|
assert a["currency_symbol"] == "$"
|
|
assert a["amount"] == "18.00"
|
|
assert a["per_unit"] == "hour"
|
|
assert a["amount_kind"] == "decimal"
|
|
|
|
|
|
def test_rate_with_currency_matches_for_one_surface(
|
|
registry: tuple[RatifiedRecognizer, ...],
|
|
) -> None:
|
|
m = match("She sells lemonade for $2 for one cup.", registry)
|
|
assert m is not None
|
|
assert m.category is ShapeCategory.RATE_WITH_CURRENCY
|
|
assert m.parsed_anchors[0]["per_unit"] == "cup"
|
|
assert m.parsed_anchors[0]["amount"] == "2"
|
|
assert m.parsed_anchors[0]["amount_kind"] == "integer"
|
|
|
|
|
|
def test_temporal_aggregation_matches_each_day(
|
|
registry: tuple[RatifiedRecognizer, ...],
|
|
) -> None:
|
|
m = match(
|
|
"Allison uploads 10 videos each day to her channel.",
|
|
registry,
|
|
)
|
|
assert m is not None
|
|
assert m.category is ShapeCategory.TEMPORAL_AGGREGATION
|
|
assert m.graph_intent == "aggregate"
|
|
assert m.parsed_anchors[0]["window_unit"] == "day"
|
|
assert m.parsed_anchors[0]["window_quantifier"] == "each"
|
|
|
|
|
|
def test_descriptive_setup_no_quantity_matches_setup(
|
|
registry: tuple[RatifiedRecognizer, ...],
|
|
) -> None:
|
|
m = match("Marnie makes bead bracelets.", registry)
|
|
assert m is not None
|
|
assert m.category is ShapeCategory.DESCRIPTIVE_SETUP_NO_QUANTITY
|
|
assert m.outcome == "inadmissible_by_design"
|
|
assert m.parsed_anchors == ()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Narrowness — out-of-corpus surfaces must NOT match
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_rate_with_currency_rejects_unseen_currency_bitcoin(
|
|
registry: tuple[RatifiedRecognizer, ...],
|
|
) -> None:
|
|
"""Bitcoin sign ₿ is outside the spec's observed currency set."""
|
|
# Verify ₿ not in observed; if it is (it isn't in current Phase B
|
|
# corpora), fall back to any non-USD/non-GBP/non-EUR/non-JPY symbol.
|
|
m = match("She earns ₿10 per hour.", registry)
|
|
assert m is None or m.category is not ShapeCategory.RATE_WITH_CURRENCY
|
|
|
|
|
|
def test_temporal_aggregation_rejects_unseen_window_unit(
|
|
registry: tuple[RatifiedRecognizer, ...],
|
|
) -> None:
|
|
"""The Phase B seeds observe a subset of window units; the matcher
|
|
refuses statements with units outside that subset."""
|
|
rate_recognizer = next(
|
|
r for r in registry if r.shape_category is ShapeCategory.TEMPORAL_AGGREGATION
|
|
)
|
|
observed = set(rate_recognizer.canonical_pattern["observed_window_units"])
|
|
all_units = {"day", "week", "month", "year", "hour", "minute", "second"}
|
|
unseen = sorted(all_units - observed)
|
|
if not unseen:
|
|
pytest.skip("Phase B corpus already covers full window vocabulary")
|
|
fake_unit = unseen[0]
|
|
# Use 5 here so it can't collide with descriptive's no-quantity rule.
|
|
m = match(f"She does 5 things each {fake_unit}.", registry)
|
|
assert m is None or m.category is not ShapeCategory.TEMPORAL_AGGREGATION
|
|
|
|
|
|
def test_descriptive_setup_rejects_statement_with_digit(
|
|
registry: tuple[RatifiedRecognizer, ...],
|
|
) -> None:
|
|
"""A statement carrying a digit cannot be admitted as
|
|
descriptive_setup_no_quantity — that category's spec pins
|
|
quantity_anchor_count=0. Some OTHER recognizer may match
|
|
(rate/temporal), but not descriptive."""
|
|
m = match("Sally has 5 apples.", registry)
|
|
if m is not None:
|
|
assert m.category is not ShapeCategory.DESCRIPTIVE_SETUP_NO_QUANTITY
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# WRONG-COUNT SAFETY — at least one negative case per category proving
|
|
# the matcher does NOT mis-admit a math-load-bearing surface that the
|
|
# Phase C synthesizer's gate would otherwise reject (ADR-0163 §The
|
|
# Load-Bearing Judgment Call).
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_rate_with_currency_does_not_match_currency_without_per_unit(
|
|
registry: tuple[RatifiedRecognizer, ...],
|
|
) -> None:
|
|
"""'She paid $5' carries currency but no per-unit framing — not a
|
|
rate, must NOT match. Mis-admitting it would lose the
|
|
distinction between amount and rate downstream."""
|
|
m = match("She paid $5 for the book.", registry)
|
|
assert m is None or m.category is not ShapeCategory.RATE_WITH_CURRENCY
|
|
|
|
|
|
def test_temporal_aggregation_does_not_match_single_day_token(
|
|
registry: tuple[RatifiedRecognizer, ...],
|
|
) -> None:
|
|
"""A single day-of-week token without enumeration must not trip
|
|
day-windowed aggregation. This was a Phase B author_note edge
|
|
case ('Saturdays present but not enumerated')."""
|
|
m = match("On Saturday she went to the store.", registry)
|
|
assert m is None or m.category is not ShapeCategory.TEMPORAL_AGGREGATION
|
|
|
|
|
|
def test_descriptive_setup_does_not_match_indefinite_quantity(
|
|
registry: tuple[RatifiedRecognizer, ...],
|
|
) -> None:
|
|
"""'There are some kids in camp' carries an indefinite quantifier
|
|
— Phase A categorizes it as indefinite_quantity, NOT
|
|
descriptive_setup_no_quantity. The matcher must respect that
|
|
distinction."""
|
|
m = match("There are some kids in camp.", registry)
|
|
assert m is None or m.category is not ShapeCategory.DESCRIPTIVE_SETUP_NO_QUANTITY
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Determinism + purity
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_match_is_deterministic(
|
|
registry: tuple[RatifiedRecognizer, ...],
|
|
) -> None:
|
|
statement = "Tina makes $18.00 an hour."
|
|
a = match(statement, registry)
|
|
b = match(statement, registry)
|
|
assert a is not None and b is not None
|
|
assert a.category is b.category
|
|
assert a.parsed_anchors == b.parsed_anchors
|
|
|
|
|
|
def test_match_returns_none_for_empty_registry() -> None:
|
|
assert match("Tina makes $18.00 an hour.", ()) is None
|
|
|
|
|
|
def test_match_returns_none_for_empty_statement(
|
|
registry: tuple[RatifiedRecognizer, ...],
|
|
) -> None:
|
|
assert match("", registry) is None
|
|
assert match(" ", registry) is None
|
|
|
|
|
|
def test_module_imports_no_llm_or_ml() -> None:
|
|
"""Phase A/C/D matchers are rules-only."""
|
|
import generate.recognizer_match as m
|
|
module_file = m.__file__
|
|
assert module_file is not None
|
|
src = Path(module_file).read_text(encoding="utf-8")
|
|
for forbidden in (
|
|
"transformers", "torch", "tensorflow", "openai",
|
|
"anthropic", "sklearn", "numpy.random",
|
|
):
|
|
assert forbidden not in src, (
|
|
f"forbidden import {forbidden!r} in recognizer_match.py"
|
|
)
|
|
|
|
|
|
_TYPE_USED = RecognizerMatch # exported public type — silence unused import
|