From c8037cfa0da71d590424628d21385d16cc1149e6 Mon Sep 17 00:00:00 2001 From: Shay Date: Mon, 18 May 2026 06:51:46 -0700 Subject: [PATCH] feat(adr-0049): head-noun subject extraction in intent classifier MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a deterministic, pack-agnostic post-processor in `generate/intent.py` that runs after the `_RULES` table fires: - DEFINITION / RECALL / PROCEDURE: strip trailing punctuation + leading articles; preserve multi-word noun phrases - CAUSE / VERIFICATION: additionally strip leading aux verbs; return the head noun Closed-set frozen sets (`_ARTICLES`, `_AUX_VERBS`) make the transform inspectable. No pack load, no algebra change — touches only `DialogueIntent.subject`. Cognition eval (13-case public split): surface_groundedness 46.2% → 61.5% (+15.3 pp) term_capture_rate 33.3% → 50.0% (+16.7 pp) intent_accuracy 100.0% (=) versor_closure_rate 100.0% (=) Two cases lift through the ADR-0048 pack path (definition_procedure_023, definition_relation_026 — both "What is a X?" → subject=X via article stripping). CAUSE / VERIFICATION subjects are now clean head nouns, foundational for future COMPARISON pack path / teaching-store inference. Tests: tests/test_intent_subject_extraction.py (30 tests). Lanes green: smoke (67), cognition (121), runtime (19), algebra (132), teaching (17), packs (6). --- .../ADR-0049-intent-subject-extraction.md | 226 ++++++++++++++++++ docs/decisions/README.md | 1 + generate/intent.py | 64 +++++ tests/test_intent_subject_extraction.py | 220 +++++++++++++++++ 4 files changed, 511 insertions(+) create mode 100644 docs/decisions/ADR-0049-intent-subject-extraction.md create mode 100644 tests/test_intent_subject_extraction.py diff --git a/docs/decisions/ADR-0049-intent-subject-extraction.md b/docs/decisions/ADR-0049-intent-subject-extraction.md new file mode 100644 index 00000000..4b0cc050 --- /dev/null +++ b/docs/decisions/ADR-0049-intent-subject-extraction.md @@ -0,0 +1,226 @@ +# ADR-0049 — Intent Classifier Head-Noun Subject Extraction + +**Status:** Accepted +**Date:** 2026-05-18 +**Author:** Shay + +--- + +## Context + +[ADR-0048](./ADR-0048-pack-grounded-surface.md) added a pack-grounded +surface for cold-start DEFINITION / RECALL turns where the subject lemma +is in `en_core_cognition_v1`. The eval lift was real but partial. +Investigating the misses showed they were not pack gaps — the pack +*does* carry the lemmas — they were **subject-extraction gaps** in +`generate/intent.py`: + +| Prompt | Pre-0049 `subject` | Reason for miss | +|---------------------------------|-------------------------------|----------------------------| +| `What is a procedure?` | `"a procedure"` | Article not stripped | +| `What is a relation?` | `"a relation"` | Article not stripped | +| `Why does light exist?` | `"does light exist"` | Aux verb + tail not stripped| +| `Why does knowledge require evidence?` | `"does knowledge require evidence"` | Aux verb + tail not stripped| +| `Does memory require recall?` | `"Does memory require recall?"` | Whole prompt; rule matched full string | +| `Is light a wave?` | `"Is light a wave?"` | Whole prompt; rule matched full string | + +The `_RULES` table in `generate/intent.py` was producing **subject +spans**, not **subject lemmas**. Downstream consumers +(`graph_planner.graph_from_intent`, ADR-0048 +`_maybe_pack_grounded_surface`, future teaching-store inference) need +the lemma — they cannot match a noun phrase like `"a procedure"` +against a lexicon keyed on `procedure`, and they cannot key a graph +node off `"does light exist"` cleanly. + +The cleanest fix is at the classifier boundary: produce a clean +lemma in `DialogueIntent.subject` so every consumer benefits without +each implementing its own article-stripping heuristic. + +--- + +## Decision + +Add a deterministic, pack-agnostic post-processor +`_normalize_subject(phrase, tag)` in `generate/intent.py` that runs +after the rule table fires and rewrites the subject span according +to its intent's syntactic shape. + +### Behaviour by intent + +| Intent | Transform | +|---------------------------|--------------------------------------------------------| +| `DEFINITION` / `RECALL` / `PROCEDURE` | strip trailing punctuation, strip leading articles; preserve multi-word noun phrases (e.g. `"artificial intelligence"`) | +| `CAUSE` / `VERIFICATION` | strip trailing punctuation, strip leading aux verbs (`is`, `are`, `does`, `do`, `can`, `could`, …), strip leading articles, return the **head noun** (first remaining token) | +| `CORRECTION` | strip trailing punctuation, strip leading articles | +| `UNKNOWN` | bypass (preserve raw input for debugging) | +| `COMPARISON` / `TRANSITIVE_QUERY` / `FRAME_TRANSFER` | already captured by their own named-group regexes; not routed through `_RULES` | + +### Aux-verb and article sets + +Frozen sets in `generate/intent.py`: + +```python +_ARTICLES = frozenset({"a", "an", "the"}) +_AUX_VERBS = frozenset({ + "is", "are", "am", "was", "were", "be", "been", "being", + "does", "do", "did", + "has", "have", "had", + "can", "could", "would", "should", "shall", "will", + "might", "may", "must", +}) +``` + +These are **closed** word lists. The normalizer does not depend on +the cognition pack, the language pack manifold, or any external state +— it is a pure syntactic transform. + +### Fallback + +If stripping aux verbs and articles would empty the subject (e.g. +`"What is the?"`), the normalizer returns the cleaned original phrase +rather than producing an empty subject. Downstream consumers +(`_maybe_pack_grounded_surface`) already handle empty subjects +correctly (return `None`), but preserving a non-empty subject keeps +debugging output and trace surfaces readable. + +--- + +## Why this is doctrine-aligned + +CLAUDE.md prohibits *opaque LLM fallbacks, stochastic sampling, hidden +normalisation*. This ADR: + +- **Is not opaque.** Both word sets are static frozen Python sets, + visible at module scope. Every transformation rule is explicit. +- **Is not stochastic.** Identical input produces byte-identical + `DialogueIntent` (`test_normalization_is_deterministic`). +- **Is not hidden normalisation of the algebra.** The normalizer + rewrites a *typed dataclass field*, not a versor, not a manifold, + not a field state. No hot-path math is touched. No + `versor_condition` invariant is in scope. +- **Is not coupled to a specific pack.** The aux-verb / article + lists are English syntactic categories, not pack vocabulary. The + ADR-0048 pack lookup remains the *consumer* of the cleaner lemma; + the classifier itself does not load any pack. + +The trust-boundary discipline is preserved: user-controlled text is +still escaped at all log/display sites by their respective handlers; +this ADR changes only the in-process classification output. + +--- + +## Characterisation — `core eval cognition` + +A/B run on the 13-case public cognition split, identical +`RuntimeConfig` except for the merge of this ADR: + +| Metric | Pre-ADR-0049 | Post-ADR-0049 | Δ | +|---------------------------|--------------|---------------|-------------| +| `intent_accuracy` | 100.0 % | 100.0 % | 0 | +| `surface_groundedness` | 46.2 % | **61.5 %** | **+15.3 pp**| +| `term_capture_rate` | 33.3 % | **50.0 %** | **+16.7 pp**| +| `versor_closure_rate` | 100.0 % | 100.0 % | 0 | +| `versor_condition < 1e-6` | preserved | preserved | invariant | + +The two cases that lift through the pack-grounded path +(`definition_procedure_023` and `definition_relation_026`) carry the +article-stripping flow: + +```text +"What is a procedure?" -> intent.subject == "procedure" +"What is a relation?" -> intent.subject == "relation" +``` + +Both then match the cognition pack lexicon and emit a pack-grounded +surface via ADR-0048. + +The CAUSE / VERIFICATION head-noun extraction (`"Why does light +exist?"` → `"light"`, `"Does memory require recall?"` → `"memory"`) +does not directly move the eval on this split because CAUSE and +VERIFICATION intents are scope-excluded from ADR-0048's pack path. +That work is **foundational for the next ADRs**: a future +COMPARISON / CAUSE / VERIFICATION pack path or teaching-store +inference will inherit clean lemmas without re-implementing the +extraction. + +--- + +## Consequences + +### What changes + +- `generate/intent.py` gains the `_normalize_subject` post-processor + and two closed-set frozen sets (`_ARTICLES`, `_AUX_VERBS`). +- `DialogueIntent.subject` is now a clean lemma (or noun phrase) for + every intent that routes through `_RULES`. +- ADR-0048 pack-grounded surface coverage broadens from + 4 → 6 of 13 cognition-eval cases. + +### What does not change + +- `IntentTag` enum is unchanged. +- The rule table (`_RULES`) is unchanged — the post-processor runs + after a rule fires. +- COMPARISON, TRANSITIVE_QUERY, FRAME_TRANSFER, and BELONG_QUERY + paths use their own named-group regexes and were already producing + clean subjects; they are not routed through `_normalize_subject`. +- `UnknownDomainGate` semantics are unchanged. +- `versor_condition(F) < 1e-6` invariant — no algebra is touched. + +### Scope limits + +- English only. The aux-verb / article lists are English; a future + multilingual cognition pack ADR would extend the sets or move them + into the language pack itself. +- The PROCEDURE intent's `"How can I VERB ARTICLE NOUN"` shape + (`"How can I correct an error?"`) is not handled: stripping the + verb requires either part-of-speech tagging or a closed list of + imperative verbs. Out of scope here. The case + `procedure_correct_035` has empty `expected_surface_contains` in + the eval anyway, so it does not affect surface_groundedness. +- Multi-word noun phrases for DEFINITION / RECALL (e.g. + `"artificial intelligence"`) are preserved as-is. Pack lookup + matches on the cleaned phrase; if the pack carries the multi-word + lemma, it lifts; if not, it falls through to the universal + disclosure. This is the doctrinally correct behaviour. + +--- + +## Cross-References + +- [ADR-0018](./ADR-0018-tool-use-scope.md) — defines + `DialogueIntent` and the `_RULES` table this ADR post-processes. +- [ADR-0048](./ADR-0048-pack-grounded-surface.md) — the consumer + whose pack-lookup gap this ADR closes by producing clean lemmas. +- [ADR-0046](./ADR-0046-forward-graph-constraint.md) / + [ADR-0047](./ADR-0047-wire-forward-graph-constraint.md) — the + forward-graph-constraint pipeline that consumes `intent.subject` + via `graph_planner.graph_from_intent`; cleaner subjects make + graph nodes single-lemma rather than noun-phrase, increasing the + chance the AdmissibilityRegion's CGA neighbourhood intersects the + walk's candidate pool. + +--- + +## Verification + +``` +tests/test_intent_subject_extraction.py — 30 tests, all green +tests/test_intent_proposition_graph.py — pre-existing tests still green +tests/test_pack_grounding.py — pre-existing tests still green +tests/test_semantic_realizer_integration.py — pre-existing tests still green + +Lanes (all green on this branch): + core test --suite smoke 67 passed + core test --suite cognition 121 passed + core test --suite runtime 19 passed + +core eval cognition (pre → post): + intent_accuracy 100.0% → 100.0% (=) + surface_groundedness 46.2% → 61.5% (+15.3 pp) + term_capture_rate 33.3% → 50.0% (+16.7 pp) + versor_closure_rate 100.0% → 100.0% (=) +``` + +The non-negotiable field invariant (`versor_condition(F) < 1e-6`) is +preserved: this ADR touches a typed dataclass field, no algebra. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index c310162f..1218d680 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -58,6 +58,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt | [ADR-0046](ADR-0046-forward-graph-constraint.md) | PropositionGraph as forward AdmissibilityRegion + industry demos | Accepted (2026-05-18) | | [ADR-0047](ADR-0047-wire-forward-graph-constraint.md) | Wire forward graph constraint into the chat hot path (opt-in) | Accepted (2026-05-18) | | [ADR-0048](ADR-0048-pack-grounded-surface.md) | Pack-grounded surface for cold-start DEFINITION / RECALL | Accepted (2026-05-18) | +| [ADR-0049](ADR-0049-intent-subject-extraction.md) | Intent classifier head-noun subject extraction | Accepted (2026-05-18) | --- diff --git a/generate/intent.py b/generate/intent.py index 994a2496..442c4d54 100644 --- a/generate/intent.py +++ b/generate/intent.py @@ -95,6 +95,69 @@ _RULES: tuple[tuple[re.Pattern[str], IntentTag], ...] = ( ) +# ADR-0049 — deterministic head-noun extraction from subject phrases. +# +# After a rule fires, the raw subject span often still carries auxiliary +# verbs, articles, or trailing punctuation: +# +# "What is a procedure?" -> raw subject "a procedure" +# "Why does light exist?" -> raw subject "does light exist" +# "Does memory require recall?" -> raw subject (whole prompt) +# +# Downstream consumers (graph_planner, ADR-0048 pack-grounded surface, +# future teaching-store inference) expect a clean lemma so they can +# match the ratified pack lexicon, build single-subject graphs, or +# consult the teaching store keyed by lemma. +# +# This normalizer is *pack-agnostic* — it does not load or consult any +# pack. It is a pure syntactic head-noun extractor: strip aux verbs, +# strip articles, return either the head noun (CAUSE / VERIFICATION) +# or the cleaned noun phrase (DEFINITION / RECALL / PROCEDURE). +_ARTICLES = frozenset({"a", "an", "the"}) +_AUX_VERBS = frozenset({ + "is", "are", "am", "was", "were", "be", "been", "being", + "does", "do", "did", + "has", "have", "had", + "can", "could", "would", "should", "shall", "will", "might", "may", "must", +}) + + +def _normalize_subject(phrase: str, tag: IntentTag) -> str: + """Strip aux verbs, articles, and trailing punctuation from a subject phrase. + + For CAUSE and VERIFICATION the subject phrase typically contains the + full predicate ("does light exist"), and we return the head noun. + For DEFINITION / RECALL / PROCEDURE we keep multi-word noun phrases + intact (so e.g. "artificial intelligence" is preserved), only + stripping leading articles and trailing punctuation. + + Falls back to the original phrase if normalization would empty it. + """ + if not phrase: + return phrase + cleaned = phrase.strip().rstrip("?.!").strip() + if not cleaned: + return "" + tokens = cleaned.split() + if not tokens: + return cleaned + + if tag in (IntentTag.CAUSE, IntentTag.VERIFICATION): + while tokens and tokens[0].lower() in _AUX_VERBS: + tokens = tokens[1:] + + while tokens and tokens[0].lower() in _ARTICLES: + tokens = tokens[1:] + + if not tokens: + return cleaned + + if tag in (IntentTag.CAUSE, IntentTag.VERIFICATION): + return tokens[0] + + return " ".join(tokens) + + def classify_intent(prompt: str) -> DialogueIntent: text = prompt.strip() if not text: @@ -149,6 +212,7 @@ def classify_intent(prompt: str) -> DialogueIntent: subject = text[match.end():].rstrip("?").strip() if not subject: subject = text + subject = _normalize_subject(subject, tag) return DialogueIntent(tag=tag, subject=subject) return DialogueIntent(tag=IntentTag.UNKNOWN, subject=text) diff --git a/tests/test_intent_subject_extraction.py b/tests/test_intent_subject_extraction.py new file mode 100644 index 00000000..81f43bd7 --- /dev/null +++ b/tests/test_intent_subject_extraction.py @@ -0,0 +1,220 @@ +"""ADR-0049 — intent classifier subject extraction tests. + +Contract pinned here: + + - Articles ("a", "an", "the") are stripped from the subject phrase + for every intent that runs through the rule table. + - For CAUSE and VERIFICATION intents, the subject is reduced to the + head noun: leading auxiliary verbs ("does", "is", "can", ...) are + stripped, then the first remaining token is returned. + - For DEFINITION / RECALL / PROCEDURE intents, multi-word noun + phrases are preserved (only articles + trailing punctuation are + stripped) so that proper noun phrases like "artificial + intelligence" survive. + - Trailing punctuation (``?``, ``.``, ``!``) is removed. + - Empty / all-stopword inputs fall back to the original cleaned + phrase rather than producing an empty subject. + - The normalizer is pack-agnostic: no pack loading, no pack-keyed + lookup; this is a pure syntactic transform. + +These tests are intentionally narrow and pin only the post-processor +behaviour. Downstream tests (``test_pack_grounding``) cover the +end-to-end lift from this change reaching the pack-grounded surface. +""" + +from __future__ import annotations + +import pytest + +from generate.intent import ( + DialogueIntent, + IntentTag, + classify_intent, +) + + +# --------------------------------------------------------------------------- +# DEFINITION — article stripping +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "prompt,expected_subject", + [ + ("What is a procedure?", "procedure"), + ("What is a relation?", "relation"), + ("What is an answer?", "answer"), + ("What is the truth?", "truth"), + ("What is light?", "light"), # already single-word, no change + ("What is artificial intelligence?", "artificial intelligence"), # multi-word noun phrase preserved + ], +) +def test_definition_strips_articles(prompt: str, expected_subject: str) -> None: + intent = classify_intent(prompt) + assert intent.tag is IntentTag.DEFINITION + assert intent.subject == expected_subject + + +# --------------------------------------------------------------------------- +# CAUSE — head-noun extraction past leading aux verb +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "prompt,expected_subject", + [ + ("Why does light exist?", "light"), + ("Why does knowledge require evidence?", "knowledge"), + ("Why is memory important?", "memory"), + ("Why are categories useful?", "categories"), + ("Why can a procedure fail?", "procedure"), # aux 'can' then article 'a' + ], +) +def test_cause_extracts_head_noun(prompt: str, expected_subject: str) -> None: + intent = classify_intent(prompt) + assert intent.tag is IntentTag.CAUSE + assert intent.subject == expected_subject + + +# --------------------------------------------------------------------------- +# VERIFICATION — head-noun extraction past leading aux verb +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "prompt,expected_subject", + [ + ("Does memory require recall?", "memory"), + ("Is light a wave?", "light"), + ("Can a procedure fail?", "procedure"), + ("Are categories useful?", "categories"), + ("Has truth been defined?", "truth"), + ], +) +def test_verification_extracts_head_noun(prompt: str, expected_subject: str) -> None: + intent = classify_intent(prompt) + assert intent.tag is IntentTag.VERIFICATION + assert intent.subject == expected_subject + + +# --------------------------------------------------------------------------- +# RECALL — already minimal, articles still stripped +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "prompt,expected_subject", + [ + ("Remember light", "light"), + ("Remember the truth", "truth"), + ("Remember a procedure", "procedure"), + ], +) +def test_recall_strips_articles(prompt: str, expected_subject: str) -> None: + intent = classify_intent(prompt) + assert intent.tag is IntentTag.RECALL + assert intent.subject == expected_subject + + +# --------------------------------------------------------------------------- +# Edge cases — degenerate inputs do not produce empty subjects +# --------------------------------------------------------------------------- + + +def test_definition_with_only_article_falls_back() -> None: + """``What is the?`` is malformed; the normalizer must not empty the + subject — it falls back to the cleaned original.""" + intent = classify_intent("What is the?") + assert intent.tag is IntentTag.DEFINITION + assert intent.subject != "" + + +def test_verification_with_only_aux_falls_back() -> None: + """``Is is?`` is degenerate; the normalizer must not empty the subject.""" + # The rule table will match this as VERIFICATION; head-noun extraction + # would strip all tokens, so the fallback path kicks in. + intent = classify_intent("Is is is?") + assert intent.tag is IntentTag.VERIFICATION + assert intent.subject != "" + + +def test_empty_prompt_returns_unknown_with_empty_subject() -> None: + intent = classify_intent("") + assert intent.tag is IntentTag.UNKNOWN + assert intent.subject == "" + + +def test_unknown_intent_preserves_raw_subject() -> None: + """UNKNOWN-tag prompts bypass the normalizer entirely so the raw + input survives for debugging / future-pattern detection.""" + intent = classify_intent("light logos") + assert intent.tag is IntentTag.UNKNOWN + assert intent.subject == "light logos" + + +# --------------------------------------------------------------------------- +# Trailing punctuation +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "prompt", + [ + "What is light?", + "What is light.", + "What is light!", + "What is light", + ], +) +def test_trailing_punctuation_does_not_affect_subject(prompt: str) -> None: + intent = classify_intent(prompt) + assert intent.tag is IntentTag.DEFINITION + assert intent.subject == "light" + + +# --------------------------------------------------------------------------- +# Determinism +# --------------------------------------------------------------------------- + + +def test_normalization_is_deterministic() -> None: + """Same prompt must produce byte-identical DialogueIntent on repeat + classification — no randomness, no state.""" + prompt = "Why does memory require recall?" + seen: set[DialogueIntent] = set() + for _ in range(5): + seen.add(classify_intent(prompt)) + assert len(seen) == 1 + + +# --------------------------------------------------------------------------- +# Existing intent-test contract still holds (loose ``in subject.lower()``) +# --------------------------------------------------------------------------- + + +def test_legacy_loose_contract_still_holds() -> None: + """Pre-ADR-0049 tests assert ``"field" in intent.subject.lower()`` + for ``"Why does the field diverge?"`` — ADR-0049 tightens the + subject to ``"field"``, which still satisfies the substring check.""" + intent = classify_intent("Why does the field diverge?") + assert intent.tag is IntentTag.CAUSE + assert "field" in intent.subject.lower() + assert intent.subject == "field" + + +# --------------------------------------------------------------------------- +# Pack-grounded path end-to-end — ADR-0049 unblocks ADR-0048 cases +# --------------------------------------------------------------------------- + + +def test_pack_grounded_surface_lifts_with_article_stripped() -> None: + """``What is a procedure?`` was previously routed to the universal + disclosure because the subject ``"a procedure"`` did not match the + pack lemma index. Post-ADR-0049 the article is stripped and the + pack-grounded surface engages.""" + from chat.runtime import ChatRuntime + + rt = ChatRuntime() + resp = rt.chat("What is a procedure?") + assert resp.grounding_source == "pack" + assert "procedure" in resp.surface