core/tests/test_semantic_realizer_integration.py
Shay 3952da11bc fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour
Six contained-tier gap closures from the cross-phase gap audit.
Every gaps.md file involved gains a resolution block; the historical
findings are preserved for traceability.

Identity (teaching/review.py)
- _ROLE_FRAMES gains the philosophical-axis family (axiology, ontology,
  telos, ethos, epistemology, morality, ethics, virtues, norms,
  doctrine, creed, convictions, beliefs, values, principles).
- _REDIRECT_VERBS gains the dismissal family (disregard, dismiss,
  bypass, circumvent, renounce, reject, repudiate).
- Closes multi_agent_composition holdout MAC-HOLD-001 ("disregard your
  axiology") and the matching adversarial_identity gap.
- Multi-agent holdouts: 8/8 attacks rejected, 3/3 legits accepted.

Pipeline (core/cognition/pipeline.py + docs/runtime_contracts.md)
- When the unknown-domain gate fires, ChatRuntime returns the
  "I don't have field coordinates for that yet." stub and
  vault_hits == 0.  The pipeline now honours that stub as the
  user-facing surface instead of overriding with the realizer's
  fallback articulation.  walk_surface is unchanged either way.
- New contract test
  tests/test_semantic_realizer_integration.py::test_pipeline_honours_safety_stub_when_gate_fires
  locks the contract; the existing semantic-surface test now primes
  the vault first so the gate doesn't fire on the probe.
- Closes calibration gaps.md Finding 2.

Realizer morphology (generate/morphology.py)
- G1: ~100-entry irregular-verb table replaces the previous list which
  contained only regular forms.  Includes bind→bound, run→ran,
  stand→stood, write→wrote/written, eat→ate/eaten, fly→flew/flown,
  swim→swam/swum, etc.
- CVC doubling rule for -ed and -ing (stop→stopped/stopping,
  plan→planned, run→running).
- Short-ies disambiguation (die/lie/tie keep -ie- in the base; cry/fly
  collapse to -y).  Lie is also irregular (lay/lain) — uses
  _IRREGULAR_FORMS first.
- 28-case regression test (tests/test_morphology_irregular.py).

Realizer plural agreement (generate/templates.py)
- G2: under universal/existential/many/few/most quantifiers, count-noun
  subjects pluralise (molecule → molecules) and the verb de-conjugates
  (binds → bind).  Negation toggles does-not → do-not.  Aspect toggles
  has → have, is → are.  All other constructions unchanged.
- Mass nouns (evidence, wisdom, knowledge, truth, water, …) stay
  singular under quantifiers — "all evidence supports truth" is right;
  "all evidences support" would be wrong English.
- 17-case regression test
  (tests/test_realizer_quantifier_agreement.py) covering count vs mass,
  irregular plurals (child→children, analysis→analyses), and the
  quantifier-tense / quantifier-aspect / quantifier-negation grid.

Rubric punctuation tolerance (evals/grammatical_coverage/runner.py)
- G3: _check_word_order strips trailing/leading punctuation
  (.,;:!?—–) before exact-word comparison so "river," still satisfies
  word_order=["river"].  must_contain also accepts punctuation-
  stripped token matches.
- Affects every lane that uses grammatical_coverage scoring; the OOD
  case generators no longer need to pin punctuated accept_surfaces for
  C06.

Case generator + lane regeneration
- scripts/generate_english_fluency_ood.py uses generate.templates.pluralize
  for C07/C08 must_contain + word_order so case-side constraints stay
  aligned with the (more correct) realizer.
- All Phase 5 OOD lane cases (5.1, 5.4–5.7) regenerated; results files
  re-scored.

CLI (core/cli.py)
- cmd_eval no longer crashes on lanes whose case_details use "id"
  instead of "case_id" (adversarial_identity, multi_agent_composition).
- Cognition CLI lane gains the two new morphology/quantifier
  regression test files.

Lane sweep (all 100%, no regression):
  english_fluency_ood              117/117 public + 39/39 holdouts
  elementary_mathematics_ood       117/117 + 39/39
  foundational_physics_ood         117/117 + 39/39
  foundational_biology_ood         117/117 + 39/39
  classical_literature_ood         117/117 + 39/39
  grammatical_coverage             back to 100% on its own seed cases
  hebrew_fluency / koine_greek_fluency  3/3 each

CLI lane health:
  smoke 54, runtime 19, teaching 17, packs 6, cognition 103 (was 57),
  algebra 132.
2026-05-16 21:21:06 -07:00

251 lines
10 KiB
Python

"""Tests for semantic realizer integration into the cognitive pipeline.
Verifies that the semantic realizer produces structurally better surfaces
from intent + proposition graph, and that the ChatResponse contract holds.
"""
from __future__ import annotations
import pytest
from generate.intent import IntentTag, classify_intent
from generate.graph_planner import graph_from_intent, plan_articulation
from generate.realizer import realize_semantic, realize_target, RealizedPlan
from generate.semantic_templates import humanize_predicate, render_semantic
# ---------------------------------------------------------------------------
# Unit tests: semantic_templates
# ---------------------------------------------------------------------------
class TestSemanticTemplates:
def test_humanize_known_predicate(self) -> None:
assert humanize_predicate("is_defined_as") == "is defined as"
assert humanize_predicate("contrasts_with") == "contrasts with"
assert humanize_predicate("defines") == "defines"
assert humanize_predicate("means") == "means"
assert humanize_predicate("grounds") == "grounds"
assert humanize_predicate("supports") == "supports"
assert humanize_predicate("corrects") == "corrects"
def test_humanize_unknown_predicate_uses_underscore_replacement(self) -> None:
assert humanize_predicate("some_new_predicate") == "some new predicate"
def test_render_definition(self) -> None:
surface = render_semantic(
intent=IntentTag.DEFINITION,
subject="truth",
predicate="is_defined_as",
obj="coherence",
)
assert "truth" in surface
assert "is defined as" in surface
assert "coherence" in surface
def test_render_comparison(self) -> None:
surface = render_semantic(
intent=IntentTag.COMPARISON,
subject="truth",
predicate="contrasts_with",
obj="light",
secondary="light",
)
assert "truth" in surface
assert "light" in surface
def test_render_correction(self) -> None:
surface = render_semantic(
intent=IntentTag.CORRECTION,
subject="correction",
predicate="corrects",
obj="reviewed repair",
)
assert "correction" in surface.lower()
def test_pending_obj_displays_as_ellipsis(self) -> None:
surface = render_semantic(
intent=IntentTag.DEFINITION,
subject="truth",
predicate="is_defined_as",
obj="<pending>",
)
assert "<pending>" not in surface
assert "..." in surface
# ---------------------------------------------------------------------------
# Unit tests: realize_semantic
# ---------------------------------------------------------------------------
class TestRealizeSemantic:
def test_definition_prompt_uses_semantic_realizer(self) -> None:
intent = classify_intent("What is truth?")
assert intent.tag is IntentTag.DEFINITION
graph = graph_from_intent(intent)
target = plan_articulation(graph)
plan = realize_semantic(target, graph)
assert isinstance(plan, RealizedPlan)
assert plan.surface
assert "truth" in plan.surface.lower()
assert "is defined as" in plan.surface.lower()
def test_comparison_prompt_mentions_both_terms(self) -> None:
intent = classify_intent("Compare truth and light")
assert intent.tag is IntentTag.COMPARISON
graph = graph_from_intent(intent)
target = plan_articulation(graph)
plan = realize_semantic(target, graph)
assert plan.surface
assert "truth" in plan.surface.lower()
assert "light" in plan.surface.lower()
def test_correction_prompt_uses_correction_template(self) -> None:
intent = classify_intent("No, correction means reviewed repair")
assert intent.tag is IntentTag.CORRECTION
graph = graph_from_intent(intent)
target = plan_articulation(graph)
plan = realize_semantic(target, graph)
assert plan.surface
assert "correction" in plan.surface.lower()
def test_cause_prompt(self) -> None:
intent = classify_intent("Why does light exist?")
assert intent.tag is IntentTag.CAUSE
graph = graph_from_intent(intent)
target = plan_articulation(graph)
plan = realize_semantic(target, graph)
assert plan.surface
assert "is grounded in" in plan.surface.lower()
def test_empty_target_returns_empty_plan(self) -> None:
from generate.graph_planner import ArticulationTarget
plan = realize_semantic(
ArticulationTarget(steps=(), source_intent=IntentTag.UNKNOWN),
)
assert plan.surface == ""
assert plan.fragments == ()
def test_none_target_returns_empty_plan(self) -> None:
plan = realize_semantic(None)
assert plan.surface == ""
def test_seed_relation_predicates_humanize_deterministically(self) -> None:
seed_predicates = [
"defines", "means", "grounds", "supports",
"contrasts_with", "corrects", "causes", "reveals",
"precedes", "follows", "belongs_to", "answers",
]
for pred in seed_predicates:
h = humanize_predicate(pred)
assert "_" not in h, f"{pred} humanized to {h!r} still has underscores"
assert h == humanize_predicate(pred), f"{pred} not deterministic"
# ---------------------------------------------------------------------------
# Integration: realize_semantic vs realize_target produce valid plans
# ---------------------------------------------------------------------------
class TestSemanticVsRhetoricalRealization:
@pytest.mark.parametrize("prompt,expected_intent", [
("What is truth?", IntentTag.DEFINITION),
("Compare truth and light", IntentTag.COMPARISON),
("Why does light exist?", IntentTag.CAUSE),
("No, that's wrong", IntentTag.CORRECTION),
])
def test_both_realizers_produce_nonempty_surface(
self, prompt: str, expected_intent: IntentTag,
) -> None:
intent = classify_intent(prompt)
assert intent.tag is expected_intent
graph = graph_from_intent(intent)
target = plan_articulation(graph)
rhetorical = realize_target(target, graph)
semantic = realize_semantic(target, graph)
assert rhetorical.surface, f"rhetorical plan empty for {prompt!r}"
assert semantic.surface, f"semantic plan empty for {prompt!r}"
def test_semantic_surfaces_are_deterministic(self) -> None:
prompt = "What is truth?"
results = set()
for _ in range(5):
intent = classify_intent(prompt)
graph = graph_from_intent(intent)
target = plan_articulation(graph)
plan = realize_semantic(target, graph)
results.add(plan.surface)
assert len(results) == 1, f"Non-deterministic: {results}"
# ---------------------------------------------------------------------------
# Contract: ChatResponse shape still holds through the pipeline
# ---------------------------------------------------------------------------
class TestChatResponseContractStillHolds:
def test_chat_response_has_required_fields(self) -> None:
try:
from chat.runtime import ChatRuntime, ChatResponse
except Exception:
pytest.skip("ChatRuntime not importable in this environment")
runtime = ChatRuntime()
response = runtime.chat("What is truth?")
assert isinstance(response, ChatResponse)
assert isinstance(response.surface, str)
assert response.surface
assert isinstance(response.versor_condition, float)
assert response.versor_condition < 1e-6
assert response.proposition is not None
assert response.articulation is not None
assert isinstance(response.articulation_surface, str)
assert isinstance(response.walk_surface, str)
assert isinstance(response.dialogue_role, str)
assert isinstance(response.vault_hits, int)
def test_pipeline_result_uses_semantic_surface(self) -> None:
try:
from chat.runtime import ChatRuntime
from core.cognition.pipeline import CognitiveTurnPipeline
except Exception:
pytest.skip("ChatRuntime not importable in this environment")
runtime = ChatRuntime()
pipeline = CognitiveTurnPipeline(runtime)
# Prime the vault so the unknown-domain gate does not fire on the
# probe. Without priming, ChatRuntime returns the safety stub
# ("I don't have field coordinates for that yet.") which the
# pipeline now honours (calibration gaps.md Finding 2 resolution).
# The semantic-surface contract this test gates on only applies
# when the gate does not fire; priming guarantees that.
pipeline.run("truth is defined as the coherent ground of inquiry.")
result = pipeline.run("What is truth?")
assert result.surface
assert "truth" in result.surface.lower()
assert "is defined as" in result.surface.lower()
assert result.articulation_surface == result.surface
assert result.versor_condition < 1e-6
assert result.trace_hash
def test_pipeline_honours_safety_stub_when_gate_fires(self) -> None:
"""When the unknown-domain gate fires, the pipeline's surface
is the gate's safety stub — NOT the realizer's fallback
articulation. Closes calibration gaps.md Finding 2."""
try:
from chat.runtime import ChatRuntime, _UNKNOWN_DOMAIN_SURFACE
from core.cognition.pipeline import CognitiveTurnPipeline
except Exception:
pytest.skip("ChatRuntime not importable in this environment")
runtime = ChatRuntime()
pipeline = CognitiveTurnPipeline(runtime)
# Cold runtime: the very first probe should fire the gate.
result = pipeline.run("What is truth?")
assert result.vault_hits == 0, "gate-fired turn should have zero vault hits"
assert result.surface == _UNKNOWN_DOMAIN_SURFACE
assert result.articulation_surface == _UNKNOWN_DOMAIN_SURFACE
# walk_surface is unaffected by the override decision — it carries
# the realizer's evidence regardless.
assert isinstance(result.walk_surface, str)