diff --git a/docs/decisions/ADR-0163-gsm8k-path-to-mastery.md b/docs/decisions/ADR-0163-gsm8k-path-to-mastery.md index 2e4582e4..4f20a2dd 100644 --- a/docs/decisions/ADR-0163-gsm8k-path-to-mastery.md +++ b/docs/decisions/ADR-0163-gsm8k-path-to-mastery.md @@ -207,6 +207,26 @@ Deliverables: their shape category, exemplar coverage, replay evidence, and the ratification CLI command. - Operator accepts, rejects, or withdraws. +- Engineering wiring (landed alongside the operator surface, ADR-0163.D PR): + - `generate/recognizer_registry.py` — pure projection of + accepted `exemplar_corpus` proposals from the proposal log into + a sorted-tuple of :class:`RatifiedRecognizer` records. + In-process cache keyed on the log's (mtime, sha256). + - `generate/recognizer_match.py` — per-category rules-only + matchers (no LLM, no embedding) honoring the Phase C + synthesizer's narrowness rule: out-of-corpus surface forms + return None. ``parsed_anchors`` carry extracted tokens from + the statement. + - `generate/math_candidate_graph.py` — narrowest-edit guard at + the per-statement choice loop: before the existing "no + admissible candidate for statement" refusal, consult the + ratified registry. Recognized statements are skipped from + ``per_sentence_choices`` (contribute zero math state), + preserving wrong=0 by construction. Empty registry is a + no-op. + - Downstream consumption of ``parsed_anchors`` (turning + recognized rate/temporal surfaces into solver state) is + Phase E follow-up. #### Phase E — Re-baseline GSM8K train sample diff --git a/docs/recognizer-registry.md b/docs/recognizer-registry.md new file mode 100644 index 00000000..822e5cb1 --- /dev/null +++ b/docs/recognizer-registry.md @@ -0,0 +1,89 @@ +# Ratified Recognizer Registry (ADR-0163 Phase D) + +The recognizer registry projects accepted exemplar-corpus proposals +from the append-only proposal log into a tuple the math +candidate-graph consults before refusing on an empty per-statement +choice list. It is the connective tissue between Phase C's operator +review surface and Phase D/E's admission expansion. + +## Projection rule + +A proposal enters the registry iff **all** of: + +- `source.kind == "exemplar_corpus"` (Phase C's source kind) +- `review_state == "accepted"` (operator ratification — never agent-side) +- `proposed_chain.recognizer_spec` parses as a + `teaching.recognizer_synthesis.RecognizerSpec` + +Pending, rejected, withdrawn, and non-exemplar proposals are +invisible. Malformed accepted proposals raise `RegistryLoadError` +with the offending `proposal_id` — silent drops are forbidden. + +Registry order is `(review_date, proposal_id)` ascending — stable +across runs. + +## Match contract + +`generate.recognizer_match.match(statement, registry)` returns at most +one `RecognizerMatch` per call (first-match-wins over registry order). +Each per-category matcher is rules-only — no LLM, no embedding, no +learned classifier. A module-import test pins the no-ML constraint. + +`parsed_anchors` carry numeric tokens extracted **from the statement +text**, not from the spec. For `descriptive_setup_no_quantity`, +`parsed_anchors` is the empty tuple by design — the recognizer admits +the statement as setup context, contributing no math state. + +## Narrowness invariant + +Per ADR-0163 §Phase C The Synthesis Rule property (b), the +recognizer is the **narrowest** commitment that subsumes the seeds. +The matcher inherits that narrowness verbatim: + +- A currency symbol outside the spec's `observed_currency_symbols` + does not match `rate_with_currency`. +- A window unit outside `observed_window_units` does not match + `temporal_aggregation`. +- A statement with any digit, number word, or indefinite quantifier + does not match `descriptive_setup_no_quantity`. + +Widening happens through the corridor — wider exemplar corpus → +Phase C synthesis on wider seeds → operator ratifies the wider +proposal — never by editing the matcher's permissiveness. + +## Wiring point + +`generate/math_candidate_graph.py:parse_and_solve` consults the +registry at the per-statement choice loop, **before** the existing +`no admissible candidate for statement` refusal. When the registry +recognizes the statement, the statement is dropped from +`per_sentence_choices` and the loop continues. Empty registry → the +guard is a no-op and the existing behavior is preserved +byte-identically. + +Skipping a recognized statement contributes ZERO math state to the +solver, so the Cartesian product is identical to "this statement +was never there." This preserves `wrong = 0` by construction; the +downstream solver still refuses if the remaining statements + +question cannot produce a consistent answer. + +## Ratification boundary (ADR-0161 §5) + +The agent does not ratify the live proposal log. Phase D tests +build a synthetic in-memory `RatifiedRecognizer` tuple from the +Phase C pending proposals' content (`tests/_phase_d_fixture.py`). +The matcher and candidate-graph wiring exercise the same +RecognizerSpec bytes the operator will later ratify, with zero +modification to `teaching/proposals/proposals.jsonl`. + +The operator's ratification path is the existing +`core teaching review --accept --review-date ` +— no new CLI surface lands with Phase D. + +## Phase E follow-up + +Phase D wires the registry into the admission boundary; downstream +consumption of `parsed_anchors` (turning recognized rate/temporal +surfaces into solver state that produces concrete answers) is +deferred to Phase E. The wiring is in place; Phase E adds the +math_candidate_parser handler that consumes the typed anchors. diff --git a/generate/math_candidate_graph.py b/generate/math_candidate_graph.py index f0c7166f..d59cd7dd 100644 --- a/generate/math_candidate_graph.py +++ b/generate/math_candidate_graph.py @@ -64,6 +64,23 @@ from generate.math_solver import SolveError, solve MAX_TOTAL_BRANCHES: Final[int] = 64 """Hard cap on Cartesian-product branch enumeration; exceeding refuses.""" + +def _load_ratified_registry_or_empty() -> tuple: + """Return the ratified recognizer registry, or () on any failure. + + ADR-0163 §Phase D — the candidate-graph consults this registry + before refusing on an empty per-statement choice list. Failures + (e.g. malformed log) MUST NOT regress wrong=0; in that case the + registry is treated as empty and the existing refusal path runs + unchanged. The registry projection itself is in-process cached + by ``generate.recognizer_registry``. + """ + try: + from generate.recognizer_registry import load_ratified_registry + return load_ratified_registry() + except Exception: # pragma: no cover — defensive: empty registry on any I/O error + return () + MAX_CANDIDATES_PER_SENTENCE: Final[int] = 4 """Hard cap on per-sentence candidate emission; exceeding refuses.""" @@ -425,10 +442,26 @@ def parse_and_solve(text: str) -> CandidateGraphResult: ) # Per-sentence choice spaces (after round-trip filter + tiebreaker). + # + # ADR-0163 §Phase D — ratified-recognizer admission guard. + # Before refusing on an empty choice list, consult the ratified + # RecognizerSpec registry. When the registry recognizes the + # statement, drop it from per_sentence_choices entirely instead of + # refusing: a recognized statement contributes ZERO math state so + # the Cartesian product remains identical to "this statement was + # never there," preserving wrong=0 by construction. Downstream + # consumption of parsed_anchors (turning recognized rate/temporal + # surfaces into solver state) is Phase E follow-up work. + _ratified_registry = _load_ratified_registry_or_empty() per_sentence_choices: list[list[SentenceChoice]] = [] for s in statement_sentences: choices = _filtered_statement_choices(s) if not choices: + if _ratified_registry: + from generate.recognizer_match import match as _recognizer_match + if _recognizer_match(s, _ratified_registry) is not None: + # Recognized — skip the sentence, do not refuse. + continue return CandidateGraphResult( answer=None, selected_graph=None, refusal_reason=f"no admissible candidate for statement: {s!r}", diff --git a/generate/recognizer_match.py b/generate/recognizer_match.py new file mode 100644 index 00000000..ace1153b --- /dev/null +++ b/generate/recognizer_match.py @@ -0,0 +1,396 @@ +"""ADR-0163 Phase D — per-category recognizer match. + +Pure, rules-only matching of a natural-language statement against the +ratified recognizer registry. Returns at most one +:class:`RecognizerMatch` per call (first-match-wins over the registry +order). + +Doctrine +- No LLM call, no embedding, no learned classifier. The matcher is + the same discipline as Phase A's categorizer + Phase C's + synthesizer. A module-import test (mirroring Phase A/C) enforces + this. +- Per ADR-0163 §Phase C The Synthesis Rule property (b), the + recognizer is the *narrowest* commitment that subsumes the seeds. + This module honors that narrowness verbatim: an out-of-corpus + currency symbol, window unit, or per-unit value does NOT match. + Widening happens in operator review (Phase B round 2 → Phase C + synthesis → Phase D wiring picks up the wider spec automatically), + never here. +- ``parsed_anchors`` carry the actual numeric tokens extracted from + the statement (NOT from the spec). The extraction is rules-only + and deterministic. For + ``descriptive_setup_no_quantity``, ``parsed_anchors`` is the empty + tuple by design — the recognizer admits the statement as setup + context, contributing no math state. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from typing import Any, Final, Literal, Mapping + +from evals.refusal_taxonomy.shape_categories import ShapeCategory +from generate.recognizer_registry import RatifiedRecognizer + + +# Word numerals 1..20 plus the higher cardinals and a small set of +# multipliers ("dozen"). Mirrors the Phase A categorizer's +# _NUMBER_WORDS so the matcher's "has any quantity marker" predicate +# is the same shape as Phase A's "has no quantity marker" predicate. +_NUMBER_WORDS: Final[frozenset[str]] = frozenset({ + "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", + "ten", "eleven", "twelve", "thirteen", "fourteen", "fifteen", "sixteen", + "seventeen", "eighteen", "nineteen", "twenty", "thirty", "forty", "fifty", + "sixty", "seventy", "eighty", "ninety", + "hundred", "thousand", "million", "billion", + "dozen", "dozens", +}) + +_DIGIT_RE: Final[re.Pattern[str]] = re.compile(r"\d") +_INDEFINITE_TOKENS: Final[tuple[str, ...]] = ( + " some ", " several ", " a few ", " many ", " any ", +) + + +# Currency-per-unit "amount" regex. Matches "$18.00 an hour" / +# "$2 per cup" / "$45/hour" / "$20 for one kg". The captured +# groups are (symbol, amount, _spacer, per_unit). +_CURRENCY_AMOUNT_RE: Final[re.Pattern[str]] = re.compile( + r"""(?ix) + ([\$£€¥]) # currency symbol + \s* + (\d+(?:\.\d+)?|\d+/\d+) # amount (integer, decimal, or fraction) + \s* + (?: + an?\s+([a-z]+) # "$X an hour" / "$X a day" + | per\s+([a-z]+) # "$X per hour" + | /\s*([a-z]+) # "$X/hour" + | for\s+(?:one|each|every|a)\s+([a-z]+) + # "$X for one cup" / "for each X" + ) + """, +) + +# Temporal-aggregation event_count_per_window patterns. +# +# Matches: +# "10 oysters in 5 minutes" -> count=10, window="minute", q="per" +# "10 videos each day" -> count=10, window="day", q="each" +# "20 jumping jacks on Monday" -> day-of-week single hit +# "uploads 90 minutes daily" -> count=90, window="day", q="per" +# +# Three regexes cover the high-signal canonical surfaces. Each match +# yields (count_token, window_unit, window_quantifier). +_TEMPORAL_PATTERNS: Final[tuple[tuple[re.Pattern[str], str], ...]] = ( + # " ... each|every|per " + ( + re.compile( + r"""(?ix) + \b(\d+(?:\.\d+)?)\b # count_token + [^.,;]*? # arbitrary intervening words + \b(each|every|per)\s+ + (day|week|month|year|hour|minute|second)s?\b + """ + ), + "explicit_quantifier", + ), + # " ... in " → "per " canonical + ( + re.compile( + r"""(?ix) + \b(\d+(?:\.\d+)?)\b # count_token + [^.,;]*? # arbitrary intervening words + \bin\s+\d+(?:\.\d+)?\s+ + (day|week|month|year|hour|minute|second)s?\b + """ + ), + "in_window", + ), + # " ... ly" (adverbial: daily, weekly, monthly...) + ( + re.compile( + r"""(?ix) + \b(\d+(?:\.\d+)?)\b # count_token + [^.,;]*? # arbitrary intervening words + \b(daily|weekly|monthly|yearly|hourly)\b + """ + ), + "adverbial", + ), +) + +# Day-of-week enumeration: at least two distinct day names with at +# least one numeric count. Matches "20 ... Monday, 36 ... Tuesday". +_DAY_NAMES: Final[tuple[str, ...]] = ( + "monday", "tuesday", "wednesday", "thursday", "friday", "saturday", "sunday", +) +_DAY_HIT_RE: Final[re.Pattern[str]] = re.compile( + r"""(?ix) + \b(\d+(?:\.\d+)?)\b\s* # count_token + [^.,;]*? # arbitrary intervening words + \b(monday|tuesday|wednesday|thursday|friday|saturday|sunday)\b + """ +) + + +@dataclass(frozen=True, slots=True) +class RecognizerMatch: + """One ratified-recognizer hit against a natural-language statement. + + ``parsed_anchors`` carry the numeric content extracted from + the statement. For ``descriptive_setup_no_quantity``, the tuple + is empty by design — the recognizer admits the statement as + setup context, contributing no math state. + """ + + recognizer: RatifiedRecognizer + category: ShapeCategory + outcome: Literal["admissible", "inadmissible_by_design"] + graph_intent: Literal["setup", "aggregate", "rate"] + parsed_anchors: tuple[Mapping[str, Any], ...] + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _padded_lower(statement: str) -> str: + return " " + statement.lower().replace("\n", " ") + " " + + +def _has_number_word(padded_lower: str) -> bool: + for raw_token in padded_lower.split(): + token = raw_token.strip(".,;:!?\"'()[]{}").lower() + if token in _NUMBER_WORDS: + return True + return False + + +def _has_any_quantity_marker(statement: str, padded_lower: str) -> bool: + if _DIGIT_RE.search(statement): + return True + if _has_number_word(padded_lower): + return True + for needle in _INDEFINITE_TOKENS: + if needle in padded_lower: + return True + return False + + +# --------------------------------------------------------------------------- +# Per-category matchers +# --------------------------------------------------------------------------- + + +def _match_descriptive_setup_no_quantity( + statement: str, spec: Mapping[str, Any] +) -> tuple[tuple[Mapping[str, Any], ...], Literal["setup"]] | None: + """Match a statement that carries no extractable quantity. + + Mirrors Phase A's ``_is_descriptive_setup_no_quantity`` predicate — + a statement with NO digit, NO number word, AND NO indefinite + quantifier is the canonical setup-context shape. + + Returns ``(empty parsed_anchors, "setup")`` on a hit; ``None`` + otherwise. The spec's ``quantity_anchor_count`` MUST equal 0 — + every Phase C synthesis for this category pins that, but we read + the spec rather than hard-code. + """ + if spec.get("quantity_anchor_count") != 0: + return None + padded = _padded_lower(statement) + if _has_any_quantity_marker(statement, padded): + return None + return (tuple(), "setup") + + +def _match_temporal_aggregation( + statement: str, spec: Mapping[str, Any] +) -> tuple[tuple[Mapping[str, Any], ...], Literal["aggregate"]] | None: + """Match the event_count_per_window shape against *statement*. + + Narrowness: every extracted anchor's ``window_unit`` and + ``window_quantifier`` MUST appear in the spec's observed sets. + A statement carrying an unseen window unit / quantifier returns + ``None``. + """ + if spec.get("anchor_kind") != "event_count_per_window": + return None + observed_units = set(spec.get("observed_window_units") or ()) + observed_quantifiers = set(spec.get("observed_window_quantifiers") or ()) + if not observed_units or not observed_quantifiers: + return None + + anchors: list[Mapping[str, Any]] = [] + padded = " " + statement.lower() + " " + + # Pass 1 — day-of-week enumeration. At least two distinct day + # names + a count per day yields multi-anchor day-windowed + # aggregation. + if "day" in observed_units and ("each" in observed_quantifiers or "every" in observed_quantifiers): + day_hits: list[tuple[str, str]] = [] + for m in _DAY_HIT_RE.finditer(statement): + day_hits.append((m.group(1), m.group(2).lower())) + # Require ≥ 2 distinct day names — same threshold Phase A uses. + distinct_days = {d for _, d in day_hits} + if len(distinct_days) >= 2: + quant = "each" if "each" in observed_quantifiers else "every" + for count_token, _day in day_hits: + anchors.append({ + "kind": "event_count_per_window", + "count_token": count_token, + "window_unit": "day", + "window_quantifier": quant, + }) + if anchors: + return (tuple(anchors), "aggregate") + + # Pass 2 — explicit-quantifier and adverbial framings. + for pat, kind in _TEMPORAL_PATTERNS: + for m in pat.finditer(statement): + if kind == "explicit_quantifier": + count_token, quantifier, unit = m.group(1), m.group(2).lower(), m.group(3).lower() + elif kind == "in_window": + count_token, quantifier, unit = m.group(1), "per", m.group(2).lower() + else: # adverbial + count_token = m.group(1) + adverb = m.group(2).lower() + # Map adverb → unit. + unit_map = { + "daily": "day", "weekly": "week", "monthly": "month", + "yearly": "year", "hourly": "hour", + } + unit = unit_map[adverb] + quantifier = "per" + if unit not in observed_units: + continue + if quantifier not in observed_quantifiers: + continue + anchors.append({ + "kind": "event_count_per_window", + "count_token": count_token, + "window_unit": unit, + "window_quantifier": quantifier, + }) + + if not anchors: + return None + + # Spec narrowness: anchor_count must fall within the observed range. + cmin = int(spec.get("anchor_count_min", 1)) + cmax = int(spec.get("anchor_count_max", 1)) + if not (cmin <= len(anchors) <= cmax): + return None + return (tuple(anchors), "aggregate") + + +def _match_rate_with_currency( + statement: str, spec: Mapping[str, Any] +) -> tuple[tuple[Mapping[str, Any], ...], Literal["rate"]] | None: + """Match the currency_per_unit_rate shape against *statement*. + + Narrowness: every extracted anchor's ``currency_symbol`` and + ``per_unit`` MUST be in the spec's observed sets. A statement + carrying an unseen currency or per-unit value returns ``None``. + """ + if spec.get("anchor_kind") != "currency_per_unit_rate": + return None + observed_symbols = set(spec.get("observed_currency_symbols") or ()) + observed_per_units = set(spec.get("observed_per_units") or ()) + if not observed_symbols or not observed_per_units: + return None + + anchors: list[Mapping[str, Any]] = [] + for m in _CURRENCY_AMOUNT_RE.finditer(statement): + symbol = m.group(1) + amount_token = m.group(2) + # Per-unit is whichever group captured. + per_unit = next( + (g for g in m.groups()[2:] if g), + None, + ) + if not per_unit: + continue + per_unit_lc = per_unit.lower() + if symbol not in observed_symbols: + continue + if per_unit_lc not in observed_per_units: + continue + if "/" in amount_token: + amount_kind = "word" # fractional surface; Phase B labels as 'word' + elif "." in amount_token: + amount_kind = "decimal" + else: + amount_kind = "integer" + anchors.append({ + "kind": "currency_per_unit_rate", + "currency_symbol": symbol, + "amount": amount_token, + "amount_kind": amount_kind, + "per_unit": per_unit_lc, + }) + + if not anchors: + return None + cmin = int(spec.get("anchor_count_min", 1)) + cmax = int(spec.get("anchor_count_max", 1)) + if not (cmin <= len(anchors) <= cmax): + return None + return (tuple(anchors), "rate") + + +_MATCHERS: Final[dict[ShapeCategory, Any]] = { + ShapeCategory.DESCRIPTIVE_SETUP_NO_QUANTITY: _match_descriptive_setup_no_quantity, + ShapeCategory.TEMPORAL_AGGREGATION: _match_temporal_aggregation, + ShapeCategory.RATE_WITH_CURRENCY: _match_rate_with_currency, +} + + +# --------------------------------------------------------------------------- +# Public API +# --------------------------------------------------------------------------- + + +def match( + statement: str, + registry: tuple[RatifiedRecognizer, ...], +) -> RecognizerMatch | None: + """First-match-wins over *registry*. + + Pure: same ``(statement, registry)`` → same result, byte-identical. + Order is registry order (the projection step in + :mod:`generate.recognizer_registry` sorts by ``(review_date, + proposal_id)``). + """ + if not isinstance(statement, str) or not statement.strip(): + return None + for recognizer in registry: + matcher = _MATCHERS.get(recognizer.shape_category) + if matcher is None: + continue + result = matcher(statement, recognizer.canonical_pattern) + if result is None: + continue + parsed_anchors, graph_intent = result + outcome: Literal["admissible", "inadmissible_by_design"] = ( + "inadmissible_by_design" + if recognizer.shape_category is ShapeCategory.DESCRIPTIVE_SETUP_NO_QUANTITY + else "admissible" + ) + return RecognizerMatch( + recognizer=recognizer, + category=recognizer.shape_category, + outcome=outcome, + graph_intent=graph_intent, + parsed_anchors=parsed_anchors, + ) + return None + + +__all__ = [ + "RecognizerMatch", + "match", +] diff --git a/generate/recognizer_registry.py b/generate/recognizer_registry.py new file mode 100644 index 00000000..51205861 --- /dev/null +++ b/generate/recognizer_registry.py @@ -0,0 +1,260 @@ +"""ADR-0163 Phase D — ratified RecognizerSpec registry projection. + +Pure projection over the append-only proposal log +(`teaching/proposals/proposals.jsonl`) into a tuple of +:class:`RatifiedRecognizer` records the candidate-graph admission +surface consults. + +Trust boundary +- The projection is a *read* over the proposal log. Mutation of the + active corpus or the proposal log itself is out of scope; that path + is gated by ADR-0057's ``accept_proposal``. +- Only proposals with ``review_state == "accepted"`` AND + ``source.kind == "exemplar_corpus"`` AND a parseable + ``proposed_chain.recognizer_spec`` enter the registry. Pending, + rejected, withdrawn, and non-exemplar proposals are invisible. +- Malformed accepted proposals raise :class:`RegistryLoadError` with + the offending ``proposal_id``. Silent drops are forbidden — the + operator must see them. + +Determinism +- ``load_ratified_registry(log)`` is a pure function of the log + bytes. Same log file → byte-identical tuple, sorted by + ``(review_date, proposal_id)`` ascending. +- A module-level cache keyed on the log's (mtime, sha256) keeps a hot + in-process invocation cheap. Cache lives in process; no + filesystem-level cache is introduced (ADR-0161 §1, ADR-0163 + §Phase C constraint). +""" + +from __future__ import annotations + +import hashlib +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Mapping + +from evals.refusal_taxonomy.shape_categories import ShapeCategory +from teaching.proposals import ProposalLog + + +class RegistryLoadError(ValueError): + """Raised when an accepted proposal carries a malformed recognizer spec.""" + + +@dataclass(frozen=True, slots=True) +class RatifiedRecognizer: + """One ratified recognizer projected from the proposal log. + + ``canonical_pattern`` carries the per-category bespoke shape the + Phase C synthesizer produced; consumers MUST branch on + ``shape_category`` before reading. ``review_date`` and + ``ratified_at_revision`` are recorded for audit; matching code + reads only ``shape_category`` + ``canonical_pattern``. + """ + + proposal_id: str + shape_category: ShapeCategory + canonical_pattern: Mapping[str, Any] + spec_digest: str + review_date: str + ratified_at_revision: str + + +# --------------------------------------------------------------------------- +# Cache +# --------------------------------------------------------------------------- + +# Keyed on (log_path, mtime_ns, sha256_hex). Value: the projected tuple. +# Cache is in-process and reset by clear_registry_cache() (tests). +_CACHE: dict[tuple[str, int, str], tuple[RatifiedRecognizer, ...]] = {} + + +def clear_registry_cache() -> None: + """Reset the in-process registry cache. + + Useful in tests that mutate the proposal log between calls. In + production, the cache invalidates automatically when the log's + (mtime, sha256) changes. + """ + _CACHE.clear() + + +def _log_cache_key(log_path: Path) -> tuple[str, int, str]: + if not log_path.exists(): + return (str(log_path), 0, "") + stat = log_path.stat() + digest = hashlib.sha256(log_path.read_bytes()).hexdigest() + return (str(log_path), stat.st_mtime_ns, digest) + + +# --------------------------------------------------------------------------- +# Projection +# --------------------------------------------------------------------------- + + +def _coerce_shape_category(value: Any, proposal_id: str) -> ShapeCategory: + if not isinstance(value, str): + raise RegistryLoadError( + f"proposal {proposal_id!r}: recognizer_spec.shape_category must be " + f"a string; got {type(value).__name__}" + ) + for member in ShapeCategory: + if member.value == value: + return member + raise RegistryLoadError( + f"proposal {proposal_id!r}: shape_category {value!r} is not a " + f"ShapeCategory member" + ) + + +def _extract_recognizer( + proposal: Mapping[str, Any], +) -> tuple[ShapeCategory, Mapping[str, Any], str]: + """Pull (shape_category, canonical_pattern, spec_digest) out of *proposal*. + + Raises :class:`RegistryLoadError` for any structural break. + """ + proposal_id = str(proposal.get("proposal_id") or "") + chain = proposal.get("proposed_chain") or {} + if not isinstance(chain, Mapping): + raise RegistryLoadError( + f"proposal {proposal_id!r}: proposed_chain must be a mapping" + ) + rec_spec = chain.get("recognizer_spec") + if not isinstance(rec_spec, Mapping): + raise RegistryLoadError( + f"proposal {proposal_id!r}: proposed_chain.recognizer_spec is " + "missing or non-mapping (ADR-0163 §Phase C contract)" + ) + shape_category = _coerce_shape_category( + rec_spec.get("shape_category"), proposal_id + ) + canonical_pattern = rec_spec.get("canonical_pattern") + if not isinstance(canonical_pattern, Mapping): + raise RegistryLoadError( + f"proposal {proposal_id!r}: canonical_pattern must be a mapping" + ) + spec_digest = str(chain.get("object") or "") + if not spec_digest: + raise RegistryLoadError( + f"proposal {proposal_id!r}: proposed_chain.object (spec_digest) " + "must be non-empty" + ) + return shape_category, canonical_pattern, spec_digest + + +def _accepted_review_dates( + events: list[dict[str, Any]], +) -> dict[str, tuple[str, str]]: + """Walk the log events and return {proposal_id: (review_date, note)}. + + The accept review_date is parsed out of the transition note: the + accept_proposal() helper passes the date via operator_note; ADR-0057 + encodes the same date in the corpus-append event's provenance. + Both shapes are tolerated here. + """ + out: dict[str, tuple[str, str]] = {} + for ev in events: + kind = ev.get("event") + if kind != "transition" or ev.get("to") != "accepted": + continue + pid = str(ev.get("proposal_id") or "") + note = str(ev.get("note") or "") + # Best-effort: pull a YYYY-MM-DD from the note; fall back to "". + review_date = "" + for token in note.replace(":", " ").replace(",", " ").split(): + if len(token) == 10 and token[4] == "-" and token[7] == "-": + review_date = token + break + out[pid] = (review_date, note) + # Walk corpus_append events too — their provenance.review_date is + # the authoritative source when present. + for ev in events: + if ev.get("event") != "accepted_corpus_append": + continue + pid = str(ev.get("proposal_id") or "") + prov = ev.get("provenance") or {} + if isinstance(prov, Mapping): + rd = str(prov.get("review_date") or "") + if rd and pid in out: + out[pid] = (rd, out[pid][1]) + return out + + +def load_ratified_registry( + log: ProposalLog | None = None, +) -> tuple[RatifiedRecognizer, ...]: + """Project the proposal log into a tuple of ratified recognizers. + + Only proposals whose ``review_state`` is ``"accepted"`` AND whose + ``source.kind`` is ``"exemplar_corpus"`` AND whose + ``proposed_chain.recognizer_spec`` parses as a Phase C + :class:`teaching.recognizer_synthesis.RecognizerSpec` (validated by + :func:`_extract_recognizer`) enter the tuple. + + Returned tuple is sorted by ``(review_date, proposal_id)`` + ascending — stable across runs. + + The cache is keyed on the proposal log's (mtime, sha256) so writes + to the log between calls invalidate transparently. + """ + proposal_log = log if log is not None else ProposalLog() + log_path = proposal_log.path + cache_key = _log_cache_key(log_path) + cached = _CACHE.get(cache_key) + if cached is not None: + return cached + + state = proposal_log.current_state() + events = proposal_log.events() + accept_review_dates = _accepted_review_dates(events) + + out: list[RatifiedRecognizer] = [] + for proposal_id, record in state.items(): + if record.get("state") != "accepted": + continue + source = record.get("source") or {} + if not isinstance(source, Mapping): + continue + if source.get("kind") != "exemplar_corpus": + continue + proposal_payload = record.get("proposal") or {} + if not isinstance(proposal_payload, Mapping): + raise RegistryLoadError( + f"proposal {proposal_id!r}: missing 'proposal' payload in " + "log view" + ) + try: + shape_category, canonical_pattern, spec_digest = _extract_recognizer( + proposal_payload + ) + except RegistryLoadError: + raise + review_date, _note = accept_review_dates.get(proposal_id, ("", "")) + ratified_at_revision = str( + source.get("emitted_at_revision") or "" + ) + out.append( + RatifiedRecognizer( + proposal_id=proposal_id, + shape_category=shape_category, + canonical_pattern=dict(canonical_pattern), + spec_digest=spec_digest, + review_date=review_date, + ratified_at_revision=ratified_at_revision, + ) + ) + + out.sort(key=lambda r: (r.review_date, r.proposal_id)) + result = tuple(out) + _CACHE[cache_key] = result + return result + + +__all__ = [ + "RatifiedRecognizer", + "RegistryLoadError", + "clear_registry_cache", + "load_ratified_registry", +] diff --git a/teaching/proposals/proposals.jsonl b/teaching/proposals/proposals.jsonl index 68b1db68..4ed5e313 100644 --- a/teaching/proposals/proposals.jsonl +++ b/teaching/proposals/proposals.jsonl @@ -42,3 +42,9 @@ {"chain_id":"verification_concept_requires_definition","event":"accepted_corpus_append","proposal_id":"1652a236fd525f1c5ba3dcd14f8c9b04","provenance":{"adr_id":"adr-0057","raw":"adr-0057:discovery_promoted:2026-05-18","review_date":"2026-05-18","source":"discovery_promoted"}} {"event":"transition","note":"Curriculum v2: recall reveals memory","proposal_id":"6ea18d4e5e5a04b8ec5ed6379e0b1140","to":"accepted"} {"chain_id":"cause_recall_reveals_memory","event":"accepted_corpus_append","proposal_id":"6ea18d4e5e5a04b8ec5ed6379e0b1140","provenance":{"adr_id":"adr-0057","raw":"adr-0057:discovery_promoted:2026-05-18","review_date":"2026-05-18","source":"discovery_promoted"}} +{"event":"created","proposal":{"claim_domain":"factual","evidence":[{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:gsm8k-train-sample-v1-0003","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:gsm8k-train-sample-v1-0008","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:gsm8k-train-sample-v1-0019","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:gsm8k-train-sample-v1-0021","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:gsm8k-train-sample-v1-0048","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:dsnq-v1-0006","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:dsnq-v1-0007","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:dsnq-v1-0008","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:dsnq-v1-0009","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:dsnq-v1-0010","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:dsnq-v1-0011","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:dsnq-v1-0012","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:dsnq-v1-0013","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:dsnq-v1-0014","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:dsnq-v1-0015","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:dsnq-v1-0016","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:dsnq-v1-0017","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:dsnq-v1-0018","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:dsnq-v1-0019","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:dsnq-v1-0020","source":"corpus"}],"operator_note":"","polarity":"affirms","proposal_id":"59223f13722f906a1cf9b65d9b01c990","proposed_chain":{"connective":"recognizes","intent":"admissibility","object":"47ae70324a9a8d7057c7efacfaf180a4afd07e371896af92f9d24d0a1ef04651","recognizer_spec":{"canonical_pattern":{"graph_intent":"setup","outcome":"inadmissible_by_design","quantity_anchor_count":0,"shape_category":"descriptive_setup_no_quantity","subject_is_optional":true,"unresolved_notes":["Edge case: a single day-of-week token ('Saturdays') is present but day-of-week enumeration requires >=2 distinct days, so it must not trip temporal_aggregation.","Edge case: contains a time-unit-adjacent phrase ('lunch break') but no numeric or word-form count.","Edge case: includes a time-adjacency phrase ('after dinner') but carries no quantity marker; should not trip temporal_aggregation."]},"coverage":{"anchors_empty":20,"subject_named":16,"subject_null":4},"exemplar_count":20,"exemplar_digest":"3192f9bc16ac597aa893309b08d340f748a733ec31b567769d72f21547b9fc81","shape_category":"descriptive_setup_no_quantity"},"subject":"descriptive_setup_no_quantity"},"provenance":null,"replay_evidence":null,"review_state":"pending","source":{"emitted_at_revision":"08c5e0e82feba5835b6b5e8a40dee1c590b10263","kind":"exemplar_corpus","source_id":"3192f9bc16ac597aa893309b08d340f748a733ec31b567769d72f21547b9fc81"},"source_candidate_id":"dd984fec928021e817d593de79c01cad254d380d69060f0979c9d87df021c177"}} +{"event":"replay","proposal_id":"59223f13722f906a1cf9b65d9b01c990","replay_evidence":{"baseline":{"intent_accuracy":1.0,"surface_groundedness":1.0,"term_capture_rate":1.0,"versor_closure_rate":1.0},"candidate":{"intent_accuracy":1.0,"surface_groundedness":1.0,"term_capture_rate":1.0,"versor_closure_rate":1.0},"capability_axes":{"G1_verb_classes":{"correct":20,"refused":0,"wrong":0},"G2_comparatives":{"correct":29,"refused":0,"wrong":0},"G3_numerics":{"correct":20,"refused":6,"wrong":0},"G4_multi_clause":{"correct":32,"refused":0,"wrong":0},"G5_aggregate":{"correct":20,"refused":0,"wrong":0},"S1_rate_events":{"correct":20,"refused":0,"wrong":0}},"gsm8k_train_sample":{"correct":3,"refused":47,"wrong":0},"regressed_metrics":[],"replay_equivalent":true,"wrong_count_delta":0}} +{"event":"created","proposal":{"claim_domain":"factual","evidence":[{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:gsm8k-train-sample-v1-0001","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:gsm8k-train-sample-v1-0011","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:gsm8k-train-sample-v1-0022","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0004","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0005","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0006","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0007","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0008","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0009","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0010","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0011","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0012","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0013","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0014","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0015","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0016","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0017","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0018","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0019","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:rwc-v1-0020","source":"corpus"}],"operator_note":"","polarity":"affirms","proposal_id":"46ce297f797ff16da12db5de422ca3c9","proposed_chain":{"connective":"recognizes","intent":"admissibility","object":"7741c837933c975e52d01fe4a97d18c6609cce3dceaf934ba6eccefda0c6837a","recognizer_spec":{"canonical_pattern":{"anchor_count_max":1,"anchor_count_min":1,"anchor_kind":"currency_per_unit_rate","graph_intent":"rate","observed_amount_kinds":["decimal","integer"],"observed_currency_symbols":["$","\u00a3","\u00a5","\u20ac"],"observed_per_units":["bar","cup","dozen","event","hour","kg","month","pound","session","walk","yard","year"],"outcome":"admissible","shape_category":"rate_with_currency","unresolved_notes":["'for each X' alternative framing.","'for one X' alternative framing, matching the surface of train case 0011.","Edge case: non-USD currency (euro).","Edge case: non-USD currency (pound sterling). Tests that the recognizer generalizes the currency-symbol slot.","Edge case: non-USD currency (yen) and discrete-occurrence per_unit ('session').","Non-canonical 'for one X' framing rather than 'per X'. The recognizer must accept both surface forms.","Slash-form per-unit framing ('$45/hour') rather than 'per hour'; recognizer must accept both.","per_unit='dozen' is a multiplicative unit (12-pack), not a base unit; flagged for Phase C disambiguation.","per_unit='event' is a discrete-occurrence unit, not a base measurement unit; flagged for Phase C disambiguation."]},"coverage":{"amount_kind:decimal":1,"amount_kind:integer":19,"anchors_currency_per_unit_rate":20,"currency_symbol:$":17,"currency_symbol:\u00a3":1,"currency_symbol:\u00a5":1,"currency_symbol:\u20ac":1,"per_unit:bar":1,"per_unit:cup":2,"per_unit:dozen":1,"per_unit:event":1,"per_unit:hour":6,"per_unit:kg":1,"per_unit:month":3,"per_unit:pound":1,"per_unit:session":1,"per_unit:walk":1,"per_unit:yard":1,"per_unit:year":1},"exemplar_count":20,"exemplar_digest":"8cfbb6d2dad734de4256c74bc8630a5ffcf3ae23b8df85e0d3e5816333afe678","shape_category":"rate_with_currency"},"subject":"rate_with_currency"},"provenance":null,"replay_evidence":null,"review_state":"pending","source":{"emitted_at_revision":"08c5e0e82feba5835b6b5e8a40dee1c590b10263","kind":"exemplar_corpus","source_id":"8cfbb6d2dad734de4256c74bc8630a5ffcf3ae23b8df85e0d3e5816333afe678"},"source_candidate_id":"1f2e7e29c5357b00446cf8acdfda39688962d40f053ec9bdc9632c7b444b957e"}} +{"event":"replay","proposal_id":"46ce297f797ff16da12db5de422ca3c9","replay_evidence":{"baseline":{"intent_accuracy":1.0,"surface_groundedness":1.0,"term_capture_rate":1.0,"versor_closure_rate":1.0},"candidate":{"intent_accuracy":1.0,"surface_groundedness":1.0,"term_capture_rate":1.0,"versor_closure_rate":1.0},"capability_axes":{"G1_verb_classes":{"correct":20,"refused":0,"wrong":0},"G2_comparatives":{"correct":29,"refused":0,"wrong":0},"G3_numerics":{"correct":20,"refused":6,"wrong":0},"G4_multi_clause":{"correct":32,"refused":0,"wrong":0},"G5_aggregate":{"correct":20,"refused":0,"wrong":0},"S1_rate_events":{"correct":20,"refused":0,"wrong":0}},"gsm8k_train_sample":{"correct":3,"refused":47,"wrong":0},"regressed_metrics":[],"replay_equivalent":true,"wrong_count_delta":0}} +{"event":"created","proposal":{"claim_domain":"factual","evidence":[{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:gsm8k-train-sample-v1-0013","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:gsm8k-train-sample-v1-0014","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:gsm8k-train-sample-v1-0024","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:gsm8k-train-sample-v1-0050","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0005","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0006","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0007","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0008","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0009","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0010","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0011","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0012","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0013","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0014","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0015","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0016","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0017","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0018","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0019","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:ta-v1-0020","source":"corpus"}],"operator_note":"","polarity":"affirms","proposal_id":"a3b892546977c5f0f64c578d6052adbd","proposed_chain":{"connective":"recognizes","intent":"admissibility","object":"d6e31f6bc8e77067ef6388ff95b7fa47b20c51bc0ab5223c454e533909dcbb17","recognizer_spec":{"canonical_pattern":{"anchor_count_max":4,"anchor_count_min":1,"anchor_kind":"event_count_per_window","graph_intent":"aggregate","observed_window_quantifiers":["each","every","per"],"observed_window_units":["day","hour","minute","month","week"],"outcome":"admissible","shape_category":"temporal_aggregation","unresolved_notes":["'a gig' implies count=1; 'every other day' is approximated as window_quantifier='every' with unit 'day'; the outer 'for 2 weeks' duration bound is not captured by the current schema.","'daily' is treated as window_unit='day' with quantifier='per'; the adverbial form has no explicit quantifier word.","'every other day' approximated as window_quantifier='every'; the 'other' (skip-one) cadence is not represented in the schema.","Day-of-week enumeration is encoded as four day-windowed event_count anchors; specific day-of-week labels (Mon/Tue/Wed/Thu) are not captured because the schema lacks a day-of-week field.","Edge case: 'in N minutes' window framing (no 'each/every/per' word). Mirrors train case 0014 but with a different count/unit pairing; window_quantifier is mapped to 'per' and the '30' multiplier is implicit.","Edge case: 'over N hours' temporal-window framing. As with case 0014, the schema collapses '80 per 6 hours' to window_unit='hour' with quantifier='per'.","Edge case: minimal day-of-week enumeration (exactly two distinct day names \u2014 the threshold for the rule). Distinct from case 0024 which enumerates four days.","The 'one-hour' detail describes video length, not the aggregation window; it is intentionally omitted from quantity_anchors so Phase C generalizes the window-axis cleanly.","The window is 5 minutes (composite count + unit); the schema only encodes window_unit/quantifier, so 'per' is the closest canonical quantifier and the '5' multiplier is implicit. Phase C may need a window_count extension to express '10 per 5 minutes' versus '10 per minute' precisely.","count_token here refers to the per-day quantity (2 hours of practice); the inner unit 'hours' is part of the counted thing, not the window.","count_token=90 refers to minutes-of-practice per day; the inner unit 'minutes' is part of the counted thing."]},"coverage":{"anchors_event_count_per_window":24,"window_quantifier:each":11,"window_quantifier:every":5,"window_quantifier:per":8,"window_unit:day":15,"window_unit:hour":2,"window_unit:minute":2,"window_unit:month":1,"window_unit:week":4},"exemplar_count":20,"exemplar_digest":"cca6491bac0df6e9c935d38cefc8d421de9ea6761b34e2f8bf99f9e543c95e82","shape_category":"temporal_aggregation"},"subject":"temporal_aggregation"},"provenance":null,"replay_evidence":null,"review_state":"pending","source":{"emitted_at_revision":"08c5e0e82feba5835b6b5e8a40dee1c590b10263","kind":"exemplar_corpus","source_id":"cca6491bac0df6e9c935d38cefc8d421de9ea6761b34e2f8bf99f9e543c95e82"},"source_candidate_id":"71c76cbfd6ee52075ba78fa31b7a23490762d9bac7779c00ce293a362e23d2b2"}} +{"event":"replay","proposal_id":"a3b892546977c5f0f64c578d6052adbd","replay_evidence":{"baseline":{"intent_accuracy":1.0,"surface_groundedness":1.0,"term_capture_rate":1.0,"versor_closure_rate":1.0},"candidate":{"intent_accuracy":1.0,"surface_groundedness":1.0,"term_capture_rate":1.0,"versor_closure_rate":1.0},"capability_axes":{"G1_verb_classes":{"correct":20,"refused":0,"wrong":0},"G2_comparatives":{"correct":29,"refused":0,"wrong":0},"G3_numerics":{"correct":20,"refused":6,"wrong":0},"G4_multi_clause":{"correct":32,"refused":0,"wrong":0},"G5_aggregate":{"correct":20,"refused":0,"wrong":0},"S1_rate_events":{"correct":20,"refused":0,"wrong":0}},"gsm8k_train_sample":{"correct":3,"refused":47,"wrong":0},"regressed_metrics":[],"replay_equivalent":true,"wrong_count_delta":0}} diff --git a/tests/_phase_d_fixture.py b/tests/_phase_d_fixture.py new file mode 100644 index 00000000..ef690b53 --- /dev/null +++ b/tests/_phase_d_fixture.py @@ -0,0 +1,79 @@ +"""Phase D test fixture — synthetic ratified registry from live log. + +Per ADR-0161 §5, the agent does NOT ratify the live proposal log. The +agent's tests build a SYNTHETIC RATIFIED REGISTRY in memory from the +three live PENDING Phase C proposals, populating ``review_date`` with +a fixed synthetic date. This exercises the per-category match +functions + the candidate-graph wiring against the EXACT +RecognizerSpec content the operator will later ratify, with zero +modification of the live proposal log. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Mapping + +from evals.refusal_taxonomy.shape_categories import ShapeCategory +from generate.recognizer_registry import RatifiedRecognizer +from teaching.proposals import ProposalLog + + +PHASE_C_PROPOSAL_IDS: tuple[str, ...] = ( + "59223f13722f906a1cf9b65d9b01c990", # descriptive_setup_no_quantity + "46ce297f797ff16da12db5de422ca3c9", # rate_with_currency + "a3b892546977c5f0f64c578d6052adbd", # temporal_aggregation +) + +SYNTHETIC_REVIEW_DATE: str = "2026-05-27" + + +def build_synthetic_registry( + log_path: Path | None = None, +) -> tuple[RatifiedRecognizer, ...]: + """Build the in-memory ratified registry from live pending Phase C proposals. + + Reads ``teaching/proposals/proposals.jsonl`` (or *log_path*), pulls + the three Phase C proposals by id, and converts their pending + ``proposed_chain.recognizer_spec`` payloads into + :class:`RatifiedRecognizer` records. + + Raises :class:`AssertionError` if any of the three Phase C + proposal_ids cannot be located in the log. This is intentional — + Phase D's tests assume Phase C's exemplar-corpus proposals exist; + if they don't, the operator should re-run + ``core teaching propose-from-exemplars --all`` first. + """ + log = ProposalLog(log_path) + state = log.current_state() + recognizers: list[RatifiedRecognizer] = [] + for pid in PHASE_C_PROPOSAL_IDS: + record = state.get(pid) + assert record is not None, ( + f"Phase D fixture: proposal {pid!r} not in {log.path}; " + "run `core teaching propose-from-exemplars --all` first" + ) + chain = record["proposal"]["proposed_chain"] + spec: Mapping[str, object] = chain["recognizer_spec"] # type: ignore[assignment] + shape_category = ShapeCategory(spec["shape_category"]) # type: ignore[arg-type] + recognizers.append( + RatifiedRecognizer( + proposal_id=pid, + shape_category=shape_category, + canonical_pattern=spec["canonical_pattern"], # type: ignore[arg-type] + spec_digest=str(chain["object"]), + review_date=SYNTHETIC_REVIEW_DATE, + ratified_at_revision=str( + record["proposal"]["source"]["emitted_at_revision"] + ), + ) + ) + recognizers.sort(key=lambda r: (r.review_date, r.proposal_id)) + return tuple(recognizers) + + +__all__ = [ + "PHASE_C_PROPOSAL_IDS", + "SYNTHETIC_REVIEW_DATE", + "build_synthetic_registry", +] diff --git a/tests/test_candidate_graph_recognizer_wiring.py b/tests/test_candidate_graph_recognizer_wiring.py new file mode 100644 index 00000000..d477ed3e --- /dev/null +++ b/tests/test_candidate_graph_recognizer_wiring.py @@ -0,0 +1,260 @@ +"""ADR-0163 Phase D — candidate-graph recognizer wiring tests. + +Pins: +- empty registry: candidate_graph behaves identically to main (no regression) +- non-empty (synthetic) registry: a previously-refused statement that + matches a recognizer no longer triggers the per-statement refusal +- wrong_count_delta == 0 under the synthetic registry against the + GSM8K train_sample (the load-bearing wrong=0 invariant test) +- capability axes G1..G5 + S1 report wrong=0 unchanged under the + synthetic registry (wiring does not regress the wrong=0 floor) +- per-category admission counts: how many GSM8K train_sample + statements the matcher admits per Phase B category (fixture-based, + not live re-baseline) +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +import pytest + +import generate.math_candidate_graph as cg +from evals.refusal_taxonomy.shape_categories import ShapeCategory +from generate.recognizer_match import match as _matcher +from generate.recognizer_registry import RatifiedRecognizer +from tests._phase_d_fixture import build_synthetic_registry + + +_REPO_ROOT = Path(__file__).resolve().parent.parent +_GSM8K_CASES = _REPO_ROOT / "evals" / "gsm8k_math" / "train_sample" / "v1" / "cases.jsonl" +_GSM8K_REPORT = _REPO_ROOT / "evals" / "gsm8k_math" / "train_sample" / "v1" / "report.json" + + +@pytest.fixture(scope="module") +def synthetic_registry() -> tuple[RatifiedRecognizer, ...]: + return build_synthetic_registry() + + +@pytest.fixture +def with_synthetic_registry( + monkeypatch: pytest.MonkeyPatch, + synthetic_registry: tuple[RatifiedRecognizer, ...], +) -> tuple[RatifiedRecognizer, ...]: + """Patch ``math_candidate_graph._load_ratified_registry_or_empty`` to + return the synthetic registry for the duration of the test.""" + monkeypatch.setattr( + cg, "_load_ratified_registry_or_empty", lambda: synthetic_registry, + ) + return synthetic_registry + + +# --------------------------------------------------------------------------- +# Empty registry: no behavioral change +# --------------------------------------------------------------------------- + + +def test_empty_registry_preserves_existing_refusal_reason() -> None: + """The live proposal log on main has zero accepted exemplar_corpus + proposals; the candidate-graph must refuse with the existing + reason string on a statement that has no admissible candidate.""" + # No monkeypatch — uses the real (empty) live registry projection. + result = cg.parse_and_solve( + "Tina makes $18.00 an hour. How much does Tina earn after 8 hours?" + ) + assert result.answer is None + assert result.refusal_reason is not None + # The refusal reason format is unchanged. + assert "no admissible candidate" in result.refusal_reason + + +# --------------------------------------------------------------------------- +# Non-empty synthetic registry: recognized statements skip refusal +# --------------------------------------------------------------------------- + + +def test_recognized_rate_statement_no_longer_triggers_per_statement_refusal( + with_synthetic_registry: tuple[RatifiedRecognizer, ...], +) -> None: + """With the rate_with_currency recognizer loaded, 'Tina makes + $18.00 an hour' is recognized and skipped in the per-statement + loop. The problem may still refuse downstream (the question + cannot be solved without the skipped rate's content), but the + refusal reason is no longer the per-statement + 'no admissible candidate for statement' string.""" + result = cg.parse_and_solve( + "Tina makes $18.00 an hour. How much does Tina earn after 8 hours?" + ) + if result.refusal_reason is not None: + # The recognized sentence is no longer the cause of refusal. + assert "Tina makes $18.00 an hour" not in (result.refusal_reason or "") + + +def test_recognized_descriptive_statement_no_longer_triggers_per_statement_refusal( + with_synthetic_registry: tuple[RatifiedRecognizer, ...], +) -> None: + """Descriptive_setup_no_quantity statements that survive the + numeric pre-filter (e.g., when all statements are non-numeric) + must not trigger the per-statement refusal under the wiring.""" + # Construct a problem whose statements are ALL non-numeric so + # the pre-filter does NOT strip them, forcing them to the + # per-statement loop. + result = cg.parse_and_solve( + "Marnie makes bead bracelets. John adopts a dog from a shelter. " + "How many things happened?" + ) + if result.refusal_reason is not None: + assert "Marnie makes bead bracelets" not in (result.refusal_reason or "") + assert "John adopts a dog from a shelter" not in (result.refusal_reason or "") + + +# --------------------------------------------------------------------------- +# WRONG-COUNT INVARIANT — the load-bearing safety test +# --------------------------------------------------------------------------- + + +def _run_gsm8k_train_sample_with_patch( + monkeypatch: pytest.MonkeyPatch, + registry: tuple[RatifiedRecognizer, ...], +) -> dict[str, int]: + """Re-run the gsm8k train_sample under the patched registry and + return the {correct, wrong, refused} counts.""" + monkeypatch.setattr( + cg, "_load_ratified_registry_or_empty", lambda: registry, + ) + import importlib + runner_mod = importlib.import_module( + "evals.gsm8k_math.train_sample.v1.runner" + ) + cases = runner_mod._load_cases(runner_mod._CASES_PATH) + report = runner_mod.build_report(cases) + return { + "correct": int(report["counts"]["correct"]), + "wrong": int(report["counts"]["wrong"]), + "refused": int(report["counts"]["refused"]), + } + + +def test_wrong_count_stays_zero_under_synthetic_registry( + monkeypatch: pytest.MonkeyPatch, + synthetic_registry: tuple[RatifiedRecognizer, ...], +) -> None: + """The load-bearing Phase D invariant: with the synthetic Phase C + registry loaded into the candidate-graph, the GSM8K train_sample + MUST NOT report wrong > 0. Any lift the wiring produces is + refused→correct, never refused→wrong.""" + baseline_report = json.loads(_GSM8K_REPORT.read_text(encoding="utf-8")) + baseline_counts = baseline_report["counts"] + candidate_counts = _run_gsm8k_train_sample_with_patch( + monkeypatch, synthetic_registry, + ) + assert candidate_counts["wrong"] == 0, ( + f"Phase D wiring regressed wrong=0: {candidate_counts}" + ) + wrong_delta = candidate_counts["wrong"] - int(baseline_counts.get("wrong", 0)) + assert wrong_delta == 0, f"wrong_count_delta={wrong_delta}" + + +def test_capability_axis_wrong_unchanged_under_synthetic_registry( + monkeypatch: pytest.MonkeyPatch, + synthetic_registry: tuple[RatifiedRecognizer, ...], +) -> None: + """G1..G5 + S1 must still report wrong=0 with the synthetic + registry loaded. Phase D's wiring is a per-sentence skip + guarded by a narrow recognizer; it cannot mis-admit a + well-parsed capability-axis statement.""" + monkeypatch.setattr( + cg, "_load_ratified_registry_or_empty", lambda: synthetic_registry, + ) + import importlib + lanes = [ + ("G1_verb_classes", "evals.math_capability_axes.G1_verb_classes.v1.runner"), + ("G2_comparatives", "evals.math_capability_axes.G2_comparatives.v1.runner"), + ("G3_numerics", "evals.math_capability_axes.G3_numerics.v1.runner"), + ("G4_multi_clause", "evals.math_capability_axes.G4_multi_clause.v1.runner"), + ("G5_aggregate", "evals.math_capability_axes.G5_aggregate.v1.runner"), + ("S1_rate_events", "evals.math_capability_axes.S1_rate_events.v1.runner"), + ] + for lane_id, mp in lanes: + mod = importlib.import_module(mp) + lc_args = mod._load_cases.__code__.co_argcount + br_args = mod.build_report.__code__.co_argcount + cases = mod._load_cases(mod._CASES_PATH) if lc_args == 1 else mod._load_cases() + report = mod.build_report(cases) if br_args >= 1 else mod.build_report() + # Per-axis wrong extraction (matches teaching/replay.py). + if "counts" in report: + wrong = int(report["counts"].get("wrong", 0)) + else: + metrics = report.get("metrics", {}) + wrong = int(metrics.get("wrong", metrics.get("solved_wrong", 0))) + assert wrong == 0, f"{lane_id}: wrong={wrong} under synthetic registry" + + +# --------------------------------------------------------------------------- +# Per-category admission counts (Phase D PR body evidence) +# --------------------------------------------------------------------------- + + +def test_per_category_admission_counts_on_gsm8k_train_sample( + synthetic_registry: tuple[RatifiedRecognizer, ...], +) -> None: + """Report the count of refused GSM8K train_sample sentences that + the matcher admits per Phase B category. This is the PR-body + evidence the brief requires (fixture-based, not live re-baseline). + + The assertion is bounded below by zero — we report counts, not + pin them to specific numbers, so the test stays robust to + Phase B corpus updates that narrow or widen specific axes. + """ + cases = [json.loads(l) for l in _GSM8K_CASES.read_text(encoding="utf-8").splitlines() if l.strip()] + report = json.loads(_GSM8K_REPORT.read_text(encoding="utf-8")) + refused_ids = {e["case_id"] for e in report["per_case"] if e["verdict"] == "refused"} + + counts: dict[str, int] = { + ShapeCategory.DESCRIPTIVE_SETUP_NO_QUANTITY.value: 0, + ShapeCategory.RATE_WITH_CURRENCY.value: 0, + ShapeCategory.TEMPORAL_AGGREGATION.value: 0, + } + for case in cases: + if case["case_id"] not in refused_ids: + continue + text = case["question"] + sentences = [s.strip() for s in text.replace("?", ".").split(".") if s.strip()] + for s in sentences: + m = _matcher(s, synthetic_registry) + if m is not None: + counts[m.category.value] += 1 + # Each category admits at least one statement across the 50-case + # refused-set; the PR body cites the exact counts. + assert counts[ShapeCategory.DESCRIPTIVE_SETUP_NO_QUANTITY.value] >= 1 + assert counts[ShapeCategory.RATE_WITH_CURRENCY.value] >= 1 + assert counts[ShapeCategory.TEMPORAL_AGGREGATION.value] >= 1 + # Surface the counts to stdout for the PR body. + print(f"\nPhase D admission counts (synthetic registry vs GSM8K train_sample refused-set):") + for k, v in counts.items(): + print(f" {k}: {v}") + + +# --------------------------------------------------------------------------- +# Unrecognized statement still refuses with the existing reason +# --------------------------------------------------------------------------- + + +def test_unrecognized_statement_still_refuses_with_existing_reason( + with_synthetic_registry: tuple[RatifiedRecognizer, ...], +) -> None: + """A statement that doesn't match any recognizer and that the + parser can't admit must still refuse via the existing + per-statement reason string — backward compatibility.""" + result = cg.parse_and_solve( + "Quizzical wibble fizzbuzz schnitzel 7 prog. What is the answer?" + ) + if result.refusal_reason is not None: + # Either per-statement refusal OR a later-stage refusal; both + # acceptable. The point is wrong=0 unchanged. + assert result.answer is None + + +_TYPE_USED: Any = (RatifiedRecognizer,) diff --git a/tests/test_phase_d_replay_evidence.py b/tests/test_phase_d_replay_evidence.py new file mode 100644 index 00000000..fd576ea4 --- /dev/null +++ b/tests/test_phase_d_replay_evidence.py @@ -0,0 +1,112 @@ +"""ADR-0163 Phase D — replay-evidence gate under wired registry. + +Pins: +- synthesize a registry of three accepted recognizers (the live + Phase C pending proposals' spec content) +- run the admissibility replay gate against the patched candidate-graph +- assert wrong_count_delta == 0 (the load-bearing wrong=0 invariant) +- assert each accepted recognizer's match function admits ≥ 1 + GSM8K train_sample sentence — the wiring is consequential, not + inert +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +import pytest + +import generate.math_candidate_graph as cg +import teaching.replay as replay_mod +from generate.recognizer_match import match as _matcher +from generate.recognizer_registry import RatifiedRecognizer +from tests._phase_d_fixture import build_synthetic_registry + + +_REPO_ROOT = Path(__file__).resolve().parent.parent +_GSM8K_CASES = _REPO_ROOT / "evals" / "gsm8k_math" / "train_sample" / "v1" / "cases.jsonl" + + +@pytest.fixture(autouse=True) +def _clean_replay_cache() -> Any: + replay_mod._BASELINE_CACHE.clear() + yield + replay_mod._BASELINE_CACHE.clear() + + +@pytest.fixture(scope="module") +def synthetic_registry() -> tuple[RatifiedRecognizer, ...]: + return build_synthetic_registry() + + +def test_replay_gate_wrong_count_delta_zero_under_synthetic_registry( + monkeypatch: pytest.MonkeyPatch, + synthetic_registry: tuple[RatifiedRecognizer, ...], +) -> None: + """The load-bearing Phase D invariant test. + + Patch the candidate-graph's registry loader to return the + synthetic registry, then run the full admissibility replay gate. + The wrong_count_delta must be zero — Phase D's wiring is + refused→(refused-or-correct), never refused→wrong. + """ + monkeypatch.setattr( + cg, "_load_ratified_registry_or_empty", lambda: synthetic_registry, + ) + # The replay gate runs both baseline and candidate against the + # same patched candidate-graph, so both lanes see the synthetic + # registry. This still proves wrong=0 holds under the wiring; + # the operator's live-log run will compare against the unwired + # baseline after ratification. + spec_placeholder = {"shape_category": "rate_with_currency"} # the gate ignores its content in Phase D + evidence = replay_mod.run_admissibility_replay_gate(spec_placeholder) + assert evidence.replay_equivalent is True, ( + f"replay gate rejected under synthetic registry: " + f"regressed_metrics={evidence.regressed_metrics}" + ) + assert evidence.wrong_count_delta == 0 + assert evidence.gsm8k_train_sample["wrong"] == 0 + for axis_id, counts in evidence.capability_axes.items(): + assert counts["wrong"] == 0, ( + f"{axis_id} regressed wrong=0 under synthetic registry: {counts}" + ) + + +def test_each_recognizer_admits_at_least_one_train_sample_sentence( + synthetic_registry: tuple[RatifiedRecognizer, ...], +) -> None: + """Phase D's wiring is consequential, not inert. + + For each ratified recognizer, the matcher must admit at least + one statement in the GSM8K train_sample. Proves the wiring path + will actually shift refusal causes once Phase E's parser-side + consumption lands. + """ + cases = [ + json.loads(line) + for line in _GSM8K_CASES.read_text(encoding="utf-8").splitlines() + if line.strip() + ] + statements: list[str] = [] + for case in cases: + text = case["question"] + for s in text.replace("?", ".").split("."): + s = s.strip() + if s: + statements.append(s) + + per_recognizer_hits: dict[str, int] = { + r.shape_category.value: 0 for r in synthetic_registry + } + for statement in statements: + m = _matcher(statement, synthetic_registry) + if m is not None: + per_recognizer_hits[m.category.value] += 1 + + for category, count in per_recognizer_hits.items(): + assert count >= 1, ( + f"recognizer for {category!r} admitted zero train_sample " + "sentences — wiring is inert for this category" + ) diff --git a/tests/test_recognizer_match.py b/tests/test_recognizer_match.py new file mode 100644 index 00000000..bc939ba0 --- /dev/null +++ b/tests/test_recognizer_match.py @@ -0,0 +1,212 @@ +"""ADR-0163 Phase D — recognizer_match tests. + +Pins: +- per-category match: positive hits, negative misses +- narrowness: out-of-corpus surfaces return None +- parsed_anchors carry real extracted values for admissible categories +- parsed_anchors is empty for descriptive_setup_no_quantity +- determinism: same (statement, registry) -> same result +- module-import no-LLM-no-ML test (mirror Phase A/C) +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from evals.refusal_taxonomy.shape_categories import ShapeCategory +from generate.recognizer_match import RecognizerMatch, match +from generate.recognizer_registry import RatifiedRecognizer +from tests._phase_d_fixture import build_synthetic_registry + + +@pytest.fixture(scope="module") +def registry() -> tuple[RatifiedRecognizer, ...]: + return build_synthetic_registry() + + +# --------------------------------------------------------------------------- +# Positive matches per category +# --------------------------------------------------------------------------- + + +def test_rate_with_currency_matches_canonical_surface( + registry: tuple[RatifiedRecognizer, ...], +) -> None: + m = match("Tina makes $18.00 an hour.", registry) + assert m is not None + assert m.category is ShapeCategory.RATE_WITH_CURRENCY + assert m.outcome == "admissible" + assert m.graph_intent == "rate" + assert len(m.parsed_anchors) == 1 + a = m.parsed_anchors[0] + assert a["currency_symbol"] == "$" + assert a["amount"] == "18.00" + assert a["per_unit"] == "hour" + assert a["amount_kind"] == "decimal" + + +def test_rate_with_currency_matches_for_one_surface( + registry: tuple[RatifiedRecognizer, ...], +) -> None: + m = match("She sells lemonade for $2 for one cup.", registry) + assert m is not None + assert m.category is ShapeCategory.RATE_WITH_CURRENCY + assert m.parsed_anchors[0]["per_unit"] == "cup" + assert m.parsed_anchors[0]["amount"] == "2" + assert m.parsed_anchors[0]["amount_kind"] == "integer" + + +def test_temporal_aggregation_matches_each_day( + registry: tuple[RatifiedRecognizer, ...], +) -> None: + m = match( + "Allison uploads 10 videos each day to her channel.", + registry, + ) + assert m is not None + assert m.category is ShapeCategory.TEMPORAL_AGGREGATION + assert m.graph_intent == "aggregate" + assert m.parsed_anchors[0]["window_unit"] == "day" + assert m.parsed_anchors[0]["window_quantifier"] == "each" + + +def test_descriptive_setup_no_quantity_matches_setup( + registry: tuple[RatifiedRecognizer, ...], +) -> None: + m = match("Marnie makes bead bracelets.", registry) + assert m is not None + assert m.category is ShapeCategory.DESCRIPTIVE_SETUP_NO_QUANTITY + assert m.outcome == "inadmissible_by_design" + assert m.parsed_anchors == () + + +# --------------------------------------------------------------------------- +# Narrowness — out-of-corpus surfaces must NOT match +# --------------------------------------------------------------------------- + + +def test_rate_with_currency_rejects_unseen_currency_bitcoin( + registry: tuple[RatifiedRecognizer, ...], +) -> None: + """Bitcoin sign ₿ is outside the spec's observed currency set.""" + # Verify ₿ not in observed; if it is (it isn't in current Phase B + # corpora), fall back to any non-USD/non-GBP/non-EUR/non-JPY symbol. + m = match("She earns ₿10 per hour.", registry) + assert m is None or m.category is not ShapeCategory.RATE_WITH_CURRENCY + + +def test_temporal_aggregation_rejects_unseen_window_unit( + registry: tuple[RatifiedRecognizer, ...], +) -> None: + """The Phase B seeds observe a subset of window units; the matcher + refuses statements with units outside that subset.""" + rate_recognizer = next( + r for r in registry if r.shape_category is ShapeCategory.TEMPORAL_AGGREGATION + ) + observed = set(rate_recognizer.canonical_pattern["observed_window_units"]) + all_units = {"day", "week", "month", "year", "hour", "minute", "second"} + unseen = sorted(all_units - observed) + if not unseen: + pytest.skip("Phase B corpus already covers full window vocabulary") + fake_unit = unseen[0] + # Use 5 here so it can't collide with descriptive's no-quantity rule. + m = match(f"She does 5 things each {fake_unit}.", registry) + assert m is None or m.category is not ShapeCategory.TEMPORAL_AGGREGATION + + +def test_descriptive_setup_rejects_statement_with_digit( + registry: tuple[RatifiedRecognizer, ...], +) -> None: + """A statement carrying a digit cannot be admitted as + descriptive_setup_no_quantity — that category's spec pins + quantity_anchor_count=0. Some OTHER recognizer may match + (rate/temporal), but not descriptive.""" + m = match("Sally has 5 apples.", registry) + if m is not None: + assert m.category is not ShapeCategory.DESCRIPTIVE_SETUP_NO_QUANTITY + + +# --------------------------------------------------------------------------- +# WRONG-COUNT SAFETY — at least one negative case per category proving +# the matcher does NOT mis-admit a math-load-bearing surface that the +# Phase C synthesizer's gate would otherwise reject (ADR-0163 §The +# Load-Bearing Judgment Call). +# --------------------------------------------------------------------------- + + +def test_rate_with_currency_does_not_match_currency_without_per_unit( + registry: tuple[RatifiedRecognizer, ...], +) -> None: + """'She paid $5' carries currency but no per-unit framing — not a + rate, must NOT match. Mis-admitting it would lose the + distinction between amount and rate downstream.""" + m = match("She paid $5 for the book.", registry) + assert m is None or m.category is not ShapeCategory.RATE_WITH_CURRENCY + + +def test_temporal_aggregation_does_not_match_single_day_token( + registry: tuple[RatifiedRecognizer, ...], +) -> None: + """A single day-of-week token without enumeration must not trip + day-windowed aggregation. This was a Phase B author_note edge + case ('Saturdays present but not enumerated').""" + m = match("On Saturday she went to the store.", registry) + assert m is None or m.category is not ShapeCategory.TEMPORAL_AGGREGATION + + +def test_descriptive_setup_does_not_match_indefinite_quantity( + registry: tuple[RatifiedRecognizer, ...], +) -> None: + """'There are some kids in camp' carries an indefinite quantifier + — Phase A categorizes it as indefinite_quantity, NOT + descriptive_setup_no_quantity. The matcher must respect that + distinction.""" + m = match("There are some kids in camp.", registry) + assert m is None or m.category is not ShapeCategory.DESCRIPTIVE_SETUP_NO_QUANTITY + + +# --------------------------------------------------------------------------- +# Determinism + purity +# --------------------------------------------------------------------------- + + +def test_match_is_deterministic( + registry: tuple[RatifiedRecognizer, ...], +) -> None: + statement = "Tina makes $18.00 an hour." + a = match(statement, registry) + b = match(statement, registry) + assert a is not None and b is not None + assert a.category is b.category + assert a.parsed_anchors == b.parsed_anchors + + +def test_match_returns_none_for_empty_registry() -> None: + assert match("Tina makes $18.00 an hour.", ()) is None + + +def test_match_returns_none_for_empty_statement( + registry: tuple[RatifiedRecognizer, ...], +) -> None: + assert match("", registry) is None + assert match(" ", registry) is None + + +def test_module_imports_no_llm_or_ml() -> None: + """Phase A/C/D matchers are rules-only.""" + import generate.recognizer_match as m + module_file = m.__file__ + assert module_file is not None + src = Path(module_file).read_text(encoding="utf-8") + for forbidden in ( + "transformers", "torch", "tensorflow", "openai", + "anthropic", "sklearn", "numpy.random", + ): + assert forbidden not in src, ( + f"forbidden import {forbidden!r} in recognizer_match.py" + ) + + +_TYPE_USED = RecognizerMatch # exported public type — silence unused import diff --git a/tests/test_recognizer_registry.py b/tests/test_recognizer_registry.py new file mode 100644 index 00000000..618b87d3 --- /dev/null +++ b/tests/test_recognizer_registry.py @@ -0,0 +1,289 @@ +"""ADR-0163 Phase D — recognizer_registry tests. + +Pins: +- load_ratified_registry returns empty tuple when log is empty +- filters by state=accepted + kind=exemplar_corpus +- order: sorted by (review_date, proposal_id) +- malformed spec -> RegistryLoadError with the offending proposal_id +- cache hit on identical log; cache invalidates on log mtime change +- pure: monkeypatch open() to count log reads +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +import pytest + +from generate.recognizer_registry import ( + RegistryLoadError, + clear_registry_cache, + load_ratified_registry, +) +from teaching.proposals import ProposalLog + + +@pytest.fixture(autouse=True) +def _clear_cache() -> Any: + clear_registry_cache() + yield + clear_registry_cache() + + +def _make_proposal( + *, + proposal_id: str, + shape_category: str, + review_state: str, + kind: str = "exemplar_corpus", +) -> dict[str, Any]: + """Build a proposal dict the live log shape accepts. + + Mirrors the JSONL shape Phase C's CLI writes: source.kind + + proposed_chain.recognizer_spec carry the load-bearing fields. + """ + return { + "claim_domain": "factual", + "evidence": [ + { + "epistemic_status": "coherent", + "polarity": "affirms", + "ref": "exemplar:test", + "source": "corpus", + } + ], + "operator_note": "", + "polarity": "affirms", + "proposal_id": proposal_id, + "proposed_chain": { + "subject": shape_category, + "intent": "admissibility", + "connective": "recognizes", + "object": "abc123def456", + "recognizer_spec": { + "shape_category": shape_category, + "canonical_pattern": { + "shape_category": shape_category, + "graph_intent": "setup" + if shape_category == "descriptive_setup_no_quantity" + else "aggregate", + "outcome": "inadmissible_by_design" + if shape_category == "descriptive_setup_no_quantity" + else "admissible", + "quantity_anchor_count": 0, + "unresolved_notes": [], + }, + "exemplar_count": 1, + "exemplar_digest": "deadbeef", + "coverage": {}, + }, + }, + "provenance": None, + "replay_evidence": None, + "review_state": review_state, + "source": { + "emitted_at_revision": "abc", + "kind": kind, + "source_id": "digest" if kind != "operator" else "", + }, + "source_candidate_id": f"cand-{proposal_id}", + } + + +def _write_log(path: Path, events: list[dict[str, Any]]) -> None: + """Write a synthetic proposal log JSONL at *path*.""" + with path.open("w", encoding="utf-8") as fh: + for ev in events: + fh.write(json.dumps(ev, sort_keys=True, separators=(",", ":")) + "\n") + + +def test_empty_log_returns_empty_registry(tmp_path: Path) -> None: + log_path = tmp_path / "proposals.jsonl" + log_path.write_text("", encoding="utf-8") + log = ProposalLog(log_path) + assert load_ratified_registry(log) == () + + +def test_pending_proposals_not_in_registry(tmp_path: Path) -> None: + log_path = tmp_path / "proposals.jsonl" + _write_log(log_path, [ + { + "event": "created", + "proposal": _make_proposal( + proposal_id="aaaa1111", + shape_category="descriptive_setup_no_quantity", + review_state="pending", + ), + }, + ]) + assert load_ratified_registry(ProposalLog(log_path)) == () + + +def test_non_exemplar_corpus_kind_not_in_registry(tmp_path: Path) -> None: + log_path = tmp_path / "proposals.jsonl" + _write_log(log_path, [ + { + "event": "created", + "proposal": _make_proposal( + proposal_id="bbbb2222", + shape_category="descriptive_setup_no_quantity", + review_state="pending", + kind="contemplation", + ), + }, + { + "event": "transition", + "proposal_id": "bbbb2222", + "to": "accepted", + "note": "2026-05-27", + }, + ]) + assert load_ratified_registry(ProposalLog(log_path)) == () + + +def test_accepted_exemplar_proposal_enters_registry(tmp_path: Path) -> None: + log_path = tmp_path / "proposals.jsonl" + _write_log(log_path, [ + { + "event": "created", + "proposal": _make_proposal( + proposal_id="cccc3333", + shape_category="rate_with_currency", + review_state="pending", + ), + }, + { + "event": "transition", + "proposal_id": "cccc3333", + "to": "accepted", + "note": "2026-05-27", + }, + ]) + reg = load_ratified_registry(ProposalLog(log_path)) + assert len(reg) == 1 + assert reg[0].proposal_id == "cccc3333" + assert reg[0].shape_category.value == "rate_with_currency" + assert reg[0].review_date == "2026-05-27" + + +def test_registry_sort_order_is_review_date_then_id(tmp_path: Path) -> None: + log_path = tmp_path / "proposals.jsonl" + _write_log(log_path, [ + {"event": "created", "proposal": _make_proposal( + proposal_id="zzzzzzzz", + shape_category="rate_with_currency", + review_state="pending", + )}, + {"event": "transition", "proposal_id": "zzzzzzzz", "to": "accepted", "note": "2026-05-27"}, + {"event": "created", "proposal": _make_proposal( + proposal_id="aaaaaaaa", + shape_category="rate_with_currency", + review_state="pending", + )}, + {"event": "transition", "proposal_id": "aaaaaaaa", "to": "accepted", "note": "2026-05-26"}, + ]) + reg = load_ratified_registry(ProposalLog(log_path)) + # Earlier date first. + assert [r.proposal_id for r in reg] == ["aaaaaaaa", "zzzzzzzz"] + + +def test_malformed_spec_raises_with_proposal_id(tmp_path: Path) -> None: + log_path = tmp_path / "proposals.jsonl" + broken = _make_proposal( + proposal_id="badbadbad", + shape_category="rate_with_currency", + review_state="pending", + ) + # Corrupt: shape_category is not a member of ShapeCategory. + broken["proposed_chain"]["recognizer_spec"]["shape_category"] = "not_a_category" + _write_log(log_path, [ + {"event": "created", "proposal": broken}, + {"event": "transition", "proposal_id": "badbadbad", "to": "accepted", "note": "2026-05-27"}, + ]) + with pytest.raises(RegistryLoadError, match="badbadbad"): + load_ratified_registry(ProposalLog(log_path)) + + +def test_cache_hit_avoids_re_read(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + log_path = tmp_path / "proposals.jsonl" + _write_log(log_path, [ + {"event": "created", "proposal": _make_proposal( + proposal_id="cccc3333", shape_category="rate_with_currency", review_state="pending", + )}, + {"event": "transition", "proposal_id": "cccc3333", "to": "accepted", "note": "2026-05-27"}, + ]) + log = ProposalLog(log_path) + + read_counter = {"n": 0} + real_read = Path.read_bytes + + def _tracking_read_bytes(self: Path) -> bytes: + if str(self) == str(log_path): + read_counter["n"] += 1 + return real_read(self) + + monkeypatch.setattr(Path, "read_bytes", _tracking_read_bytes) + load_ratified_registry(log) + first = read_counter["n"] + load_ratified_registry(log) + # Second call uses cache: at most one extra read (the cache-key + # mtime+sha lookup itself reads bytes), not a full re-projection. + assert read_counter["n"] - first <= 1 + + +def test_cache_invalidates_on_log_change(tmp_path: Path) -> None: + log_path = tmp_path / "proposals.jsonl" + _write_log(log_path, [ + {"event": "created", "proposal": _make_proposal( + proposal_id="cccc3333", shape_category="rate_with_currency", review_state="pending", + )}, + {"event": "transition", "proposal_id": "cccc3333", "to": "accepted", "note": "2026-05-27"}, + ]) + log = ProposalLog(log_path) + reg_a = load_ratified_registry(log) + assert len(reg_a) == 1 + + # Append another accepted proposal; cache must invalidate. + import time + time.sleep(0.01) + with log_path.open("a", encoding="utf-8") as fh: + fh.write(json.dumps({"event": "created", "proposal": _make_proposal( + proposal_id="dddd4444", + shape_category="temporal_aggregation", + review_state="pending", + )}) + "\n") + fh.write(json.dumps({ + "event": "transition", "proposal_id": "dddd4444", + "to": "accepted", "note": "2026-05-28", + }) + "\n") + reg_b = load_ratified_registry(log) + assert len(reg_b) == 2 + + +def test_live_proposal_log_has_phase_c_pending_proposals() -> None: + """Audit-level check: the live log carries the three Phase C + pending proposals. If this fails the operator has not run + ``core teaching propose-from-exemplars --all`` since Phase B, + and Phase D's downstream tests will be unable to build the + synthetic fixture (ADR-0161 §5).""" + from tests._phase_d_fixture import PHASE_C_PROPOSAL_IDS + + log = ProposalLog() + state = log.current_state() + missing = [pid for pid in PHASE_C_PROPOSAL_IDS if pid not in state] + assert not missing, ( + f"live proposal log is missing Phase C pendings {missing}; " + "run `core teaching propose-from-exemplars --all` first" + ) + # And they are ALL pending — no agent-side ratification (ADR-0161 §5). + for pid in PHASE_C_PROPOSAL_IDS: + assert state[pid]["state"] == "pending", ( + f"proposal {pid} state={state[pid]['state']!r}; " + "ADR-0161 §5 forbids agent-side ratification" + ) + # Live registry stays empty until the operator ratifies. + assert load_ratified_registry(log) == () + +