core/tests/test_recognizer_match.py
Shay e9b7eb0b1f
feat(ADR-0163.D): wire ratified RecognizerSpecs into math_candidate_graph admissibility surface (#302)
* chore(ADR-0163.C): land three Phase C pending proposals in live log

Phase C (#301) shipped the CLI but its PR dry-run wrote to a tmp log
path.  This commit moves the three Phase C proposals into the live
teaching/proposals/proposals.jsonl so the Phase B→C audit trail is
visible in the proposal log and the proposals are ready for the
operator to ratify after Phase D ships.

Proposals (all state=pending, kind="exemplar_corpus"):
- 59223f13722f906a1cf9b65d9b01c990 — descriptive_setup_no_quantity
- 46ce297f797ff16da12db5de422ca3c9 — rate_with_currency
- a3b892546977c5f0f64c578d6052adbd — temporal_aggregation

Produced by `core teaching propose-from-exemplars --all` against the
live Phase B corpora.  No ratification (ADR-0161 §5 — only the repo
owner ratifies).  The Phase D admissibility-replay gate confirmed
replay_equivalent=true, wrong_count_delta=0 for all three.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(ADR-0163.D): wire ratified RecognizerSpecs into math_candidate_graph admissibility surface

Phase D is the first PR to extend the math admission surface.  The
audit (#294) said the gap was admission, not operators, algebra,
substrate, or packs.  Phase A measured the refusal taxonomy.  Phase B
authored seeds.  Phase C synthesized recognizers.  Phase D wires
those recognizers into generate/math_candidate_graph.py.

Modules
- generate/recognizer_registry.py — pure projection over the proposal
  log.  Only proposals with source.kind="exemplar_corpus" AND
  review_state="accepted" enter the tuple.  Sorted by
  (review_date, proposal_id).  In-process cache keyed on log
  (mtime, sha256) — no filesystem cache (ADR-0161 §1).  Malformed
  accepted specs raise RegistryLoadError citing the offending
  proposal_id; silent drops are forbidden.
- generate/recognizer_match.py — per-category rules-only matchers
  (no LLM, no embedding, no learned classifier).  Honors the Phase C
  synthesizer's narrowness rule: out-of-corpus currency symbols,
  window units, and per-unit values do NOT match.  Three matchers:
  _match_descriptive_setup_no_quantity (zero-quantity surface),
  _match_temporal_aggregation (event_count_per_window with
  observed_window_units/quantifiers honored), _match_rate_with_currency
  (currency_per_unit_rate with observed currency/per-unit/amount-kind
  honored).
- generate/math_candidate_graph.py — narrowest-edit guard at the
  per-statement choice loop.  Before the existing
  "no admissible candidate for statement" refusal, consult the
  ratified registry.  Recognized statements are dropped from
  per_sentence_choices (zero math state) so the Cartesian product is
  identical to "this statement was never there."  Empty registry is
  a no-op — backward compatibility preserved byte-identically.
  Downstream consumption of parsed_anchors (turning recognized
  rate/temporal surfaces into solver state that produces concrete
  answers) is Phase E follow-up.

Tests (32 new)
- tests/_phase_d_fixture.py — synthetic in-memory ratified registry
  built from the three Phase C pending proposals' content.  Per
  ADR-0161 §5 the agent does NOT ratify the live log; the synthetic
  registry round-trips the real RecognizerSpec bytes the operator
  will ratify after Phase D ships.
- tests/test_recognizer_registry.py (9) — empty/pending/wrong-kind
  filtering, sort order, malformed-spec rejection, cache hit +
  invalidation, live-log Phase C audit check.
- tests/test_recognizer_match.py (14) — per-category positive cases,
  narrowness (out-of-corpus surface forms rejected), no-LLM import
  check.
- tests/test_candidate_graph_recognizer_wiring.py (7) — empty registry
  preserves existing refusal; synthetic registry: recognized
  statements no longer trigger per-statement refusal;
  wrong_count_delta == 0 on GSM8K train_sample; capability axes G1..
  G5+S1 wrong=0 unchanged; per-category admission counts on the
  refused-set; unrecognized statements still refuse with the
  existing reason.
- tests/test_phase_d_replay_evidence.py (2) — full admissibility
  replay gate under synthetic registry: replay_equivalent=true,
  wrong_count_delta=0, every capability axis wrong=0; each
  ratified recognizer admits >= 1 train_sample statement (wiring
  is consequential).

Per-category fixture-based admission counts (synthetic registry vs
GSM8K train_sample refused-set sentences):
- descriptive_setup_no_quantity: 40
- rate_with_currency:             2
- temporal_aggregation:           7

Narrowness-invariant negative case results (matcher correctly
returns None on out-of-corpus / load-bearing-math surfaces):
- rate_with_currency:           "She paid $5 for the book." (no per-unit)
- temporal_aggregation:         "On Saturday she went to the store." (single day token)
- descriptive_setup_no_quantity: "There are some kids in camp." (indefinite quantifier)

Candidates for Phase B round 2 (3 of 20 temporal seeds match the
spec's structural commitment but not my surface regex — author_notes
explicitly flagged these as schema-gap edge cases):
- ta-v1-0004 "Mark does a gig every other day for 2 weeks."
- ta-v1-0012 "Robin walks 4 dogs every other day around the park."
- ta-v1-0019 "The pump fills the tank with 80 gallons over 6 hours."

Three landed wirings DO NOT shift the GSM8K train_sample baseline
counts under fixture (correct=3, wrong=0, refused=47 unchanged) —
Phase D's narrow wiring is wrong=0 safe by construction; lift to
"correct" requires Phase E's downstream parser-side consumption of
parsed_anchors.  Capability axes G1..G5+S1 wrong=0 unchanged.

Cross-refs: ADR-0163 (Phase D), ADR-0057 (proposal review),
ADR-0151 (auto-proposal), ADR-0161 §5 (ratification boundary),
Phase A PR #297, Phase B PR #298, Phase C PR #301.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-26 13:11:47 -07:00

212 lines
7.9 KiB
Python

"""ADR-0163 Phase D — recognizer_match tests.
Pins:
- per-category match: positive hits, negative misses
- narrowness: out-of-corpus surfaces return None
- parsed_anchors carry real extracted values for admissible categories
- parsed_anchors is empty for descriptive_setup_no_quantity
- determinism: same (statement, registry) -> same result
- module-import no-LLM-no-ML test (mirror Phase A/C)
"""
from __future__ import annotations
from pathlib import Path
import pytest
from evals.refusal_taxonomy.shape_categories import ShapeCategory
from generate.recognizer_match import RecognizerMatch, match
from generate.recognizer_registry import RatifiedRecognizer
from tests._phase_d_fixture import build_synthetic_registry
@pytest.fixture(scope="module")
def registry() -> tuple[RatifiedRecognizer, ...]:
return build_synthetic_registry()
# ---------------------------------------------------------------------------
# Positive matches per category
# ---------------------------------------------------------------------------
def test_rate_with_currency_matches_canonical_surface(
registry: tuple[RatifiedRecognizer, ...],
) -> None:
m = match("Tina makes $18.00 an hour.", registry)
assert m is not None
assert m.category is ShapeCategory.RATE_WITH_CURRENCY
assert m.outcome == "admissible"
assert m.graph_intent == "rate"
assert len(m.parsed_anchors) == 1
a = m.parsed_anchors[0]
assert a["currency_symbol"] == "$"
assert a["amount"] == "18.00"
assert a["per_unit"] == "hour"
assert a["amount_kind"] == "decimal"
def test_rate_with_currency_matches_for_one_surface(
registry: tuple[RatifiedRecognizer, ...],
) -> None:
m = match("She sells lemonade for $2 for one cup.", registry)
assert m is not None
assert m.category is ShapeCategory.RATE_WITH_CURRENCY
assert m.parsed_anchors[0]["per_unit"] == "cup"
assert m.parsed_anchors[0]["amount"] == "2"
assert m.parsed_anchors[0]["amount_kind"] == "integer"
def test_temporal_aggregation_matches_each_day(
registry: tuple[RatifiedRecognizer, ...],
) -> None:
m = match(
"Allison uploads 10 videos each day to her channel.",
registry,
)
assert m is not None
assert m.category is ShapeCategory.TEMPORAL_AGGREGATION
assert m.graph_intent == "aggregate"
assert m.parsed_anchors[0]["window_unit"] == "day"
assert m.parsed_anchors[0]["window_quantifier"] == "each"
def test_descriptive_setup_no_quantity_matches_setup(
registry: tuple[RatifiedRecognizer, ...],
) -> None:
m = match("Marnie makes bead bracelets.", registry)
assert m is not None
assert m.category is ShapeCategory.DESCRIPTIVE_SETUP_NO_QUANTITY
assert m.outcome == "inadmissible_by_design"
assert m.parsed_anchors == ()
# ---------------------------------------------------------------------------
# Narrowness — out-of-corpus surfaces must NOT match
# ---------------------------------------------------------------------------
def test_rate_with_currency_rejects_unseen_currency_bitcoin(
registry: tuple[RatifiedRecognizer, ...],
) -> None:
"""Bitcoin sign ₿ is outside the spec's observed currency set."""
# Verify ₿ not in observed; if it is (it isn't in current Phase B
# corpora), fall back to any non-USD/non-GBP/non-EUR/non-JPY symbol.
m = match("She earns ₿10 per hour.", registry)
assert m is None or m.category is not ShapeCategory.RATE_WITH_CURRENCY
def test_temporal_aggregation_rejects_unseen_window_unit(
registry: tuple[RatifiedRecognizer, ...],
) -> None:
"""The Phase B seeds observe a subset of window units; the matcher
refuses statements with units outside that subset."""
rate_recognizer = next(
r for r in registry if r.shape_category is ShapeCategory.TEMPORAL_AGGREGATION
)
observed = set(rate_recognizer.canonical_pattern["observed_window_units"])
all_units = {"day", "week", "month", "year", "hour", "minute", "second"}
unseen = sorted(all_units - observed)
if not unseen:
pytest.skip("Phase B corpus already covers full window vocabulary")
fake_unit = unseen[0]
# Use 5 here so it can't collide with descriptive's no-quantity rule.
m = match(f"She does 5 things each {fake_unit}.", registry)
assert m is None or m.category is not ShapeCategory.TEMPORAL_AGGREGATION
def test_descriptive_setup_rejects_statement_with_digit(
registry: tuple[RatifiedRecognizer, ...],
) -> None:
"""A statement carrying a digit cannot be admitted as
descriptive_setup_no_quantity — that category's spec pins
quantity_anchor_count=0. Some OTHER recognizer may match
(rate/temporal), but not descriptive."""
m = match("Sally has 5 apples.", registry)
if m is not None:
assert m.category is not ShapeCategory.DESCRIPTIVE_SETUP_NO_QUANTITY
# ---------------------------------------------------------------------------
# WRONG-COUNT SAFETY — at least one negative case per category proving
# the matcher does NOT mis-admit a math-load-bearing surface that the
# Phase C synthesizer's gate would otherwise reject (ADR-0163 §The
# Load-Bearing Judgment Call).
# ---------------------------------------------------------------------------
def test_rate_with_currency_does_not_match_currency_without_per_unit(
registry: tuple[RatifiedRecognizer, ...],
) -> None:
"""'She paid $5' carries currency but no per-unit framing — not a
rate, must NOT match. Mis-admitting it would lose the
distinction between amount and rate downstream."""
m = match("She paid $5 for the book.", registry)
assert m is None or m.category is not ShapeCategory.RATE_WITH_CURRENCY
def test_temporal_aggregation_does_not_match_single_day_token(
registry: tuple[RatifiedRecognizer, ...],
) -> None:
"""A single day-of-week token without enumeration must not trip
day-windowed aggregation. This was a Phase B author_note edge
case ('Saturdays present but not enumerated')."""
m = match("On Saturday she went to the store.", registry)
assert m is None or m.category is not ShapeCategory.TEMPORAL_AGGREGATION
def test_descriptive_setup_does_not_match_indefinite_quantity(
registry: tuple[RatifiedRecognizer, ...],
) -> None:
"""'There are some kids in camp' carries an indefinite quantifier
— Phase A categorizes it as indefinite_quantity, NOT
descriptive_setup_no_quantity. The matcher must respect that
distinction."""
m = match("There are some kids in camp.", registry)
assert m is None or m.category is not ShapeCategory.DESCRIPTIVE_SETUP_NO_QUANTITY
# ---------------------------------------------------------------------------
# Determinism + purity
# ---------------------------------------------------------------------------
def test_match_is_deterministic(
registry: tuple[RatifiedRecognizer, ...],
) -> None:
statement = "Tina makes $18.00 an hour."
a = match(statement, registry)
b = match(statement, registry)
assert a is not None and b is not None
assert a.category is b.category
assert a.parsed_anchors == b.parsed_anchors
def test_match_returns_none_for_empty_registry() -> None:
assert match("Tina makes $18.00 an hour.", ()) is None
def test_match_returns_none_for_empty_statement(
registry: tuple[RatifiedRecognizer, ...],
) -> None:
assert match("", registry) is None
assert match(" ", registry) is None
def test_module_imports_no_llm_or_ml() -> None:
"""Phase A/C/D matchers are rules-only."""
import generate.recognizer_match as m
module_file = m.__file__
assert module_file is not None
src = Path(module_file).read_text(encoding="utf-8")
for forbidden in (
"transformers", "torch", "tensorflow", "openai",
"anthropic", "sklearn", "numpy.random",
):
assert forbidden not in src, (
f"forbidden import {forbidden!r} in recognizer_match.py"
)
_TYPE_USED = RecognizerMatch # exported public type — silence unused import