b07fb0413c
Quantitative companion to Phase 3 (commit 664e081). Where Phase 3
emits SPECULATIVE *findings* about plan quality, Phase 4 emits
typed *measurements* — pure-function projection of a
``DiscoursePlan`` into a ``PlanMetrics`` dataclass.
Why this matters
----------------
The discourse planner now produces multi-clause grounded
articulations (Phase 1), the renderer pronominalizes across
consecutive same-subject moves (Phase 2), and the contemplation
pre-flight emits qualitative concerns about plan shape (Phase 3).
What was missing was the *aggregable* layer: per-turn structured
numbers that downstream consumers can stream across many turns
to score quality patterns the per-turn observer cannot see.
Phase 4 lands that layer. Phase 5 (offline contemplation miner)
becomes possible because there's now structured signal to mine.
What it measures
----------------
Structure
* move_count — total moves in plan
* fact_bearing_count — moves with fact != None
Move-kind distribution
* anchor_count / support_count / relation_count
/ transition_count / closure_count
Diversity
* unique_predicates — distinct predicates across
fact-bearing moves
* unique_subjects — distinct subject lemmas
* unique_sources — distinct FactSources
Topic dynamics
* topic_shift_count — consecutive pairs where
subject changed
* pronominalization_opportunities — consecutive pairs where
subject held (= Phase 2's
anaphora trigger count)
Derived ratios
* predicate_diversity_ratio — unique_predicates /
fact_bearing_count
* subject_focus_ratio — pronominalizations /
(pronominalizations +
topic_shifts)
Every field is a deterministic pure function of the plan: same
plan in → byte-equal ``PlanMetrics.as_dict()`` out. This is the
load-bearing claim that lets Phase 5 aggregate across turns
without "is this the same metric?" ambiguity.
Doctrine alignment
------------------
Per ADR-0080 contemplation discipline:
* Read-only — metrics are pure projections of the plan; no
mutation of plan, runtime state, or memory tiers.
* No autonomous learning — metrics are observations, not
learned policy. Promotion to memory still flows through
the existing proposal-review-ratify chain.
* Deterministic replay — pinned by test_metrics_are_deterministic_
and_byte_equal_as_dict plus the runtime-level
test_metrics_byte_equal_across_runs.
Wiring
------
* New ``ChatRuntime.last_plan_metrics`` property — read-only
``PlanMetrics`` from the most recent turn where the planner
engaged (and ``discourse_contemplation`` was on); ``None``
otherwise. Reset between turns alongside ``last_plan_findings``
via the existing top-of-call reset block.
* Same opt-in flag as Phase 3 (``discourse_contemplation``).
When True, the runtime computes both findings AND metrics in
the same block; when False (default), both stay at empty/None.
Demo (config: discourse_contemplation=True)
-------------------------------------------
"What is knowledge?" → metrics: None (BRIEF fast-path)
"Tell me about memory." → moves=3 fact_bearing=3
kinds=A:1/S:1/R:1/T:0/C:0
unique_predicates=3 subjects=1
pronominalization_ops=2 shifts=0
predicate_diversity=1.000
subject_focus=1.000
"What is truth, and why does
it matter?" → moves=7 fact_bearing=6
kinds=A:2/S:2/R:2/T:1/C:0
unique_predicates=4 subjects=1
pronominalization_ops=4 shifts=1
predicate_diversity=0.667 ← Phase 3
WEAK_SURFACE
quantified
subject_focus=0.800
+ 1 finding (weak_surface)
The compound-prompt numbers are particularly informative:
``predicate_diversity=0.667`` is the algebraic expression of the
Phase 3 ``WEAK_SURFACE`` rule — the rule fires precisely because
6 fact-bearing moves used only 4 distinct predicates.
``subject_focus=0.800`` quantifies that 80% of consecutive pairs
held the same subject — high topic stickiness that Phase 2's
reflective renderer leveraged into 4 ``it`` substitutions.
Tests
-----
* ``tests/test_plan_metrics.py`` — 10 unit tests pinning each
field, derived ratios, bridge-move handling (``fact=None``
resets the focus channel), and determinism via ``as_dict()``
byte-equality.
* ``tests/test_plan_metrics_runtime.py`` — 8 end-to-end tests
proving the runtime wiring: disabled by default, populated
when enabled, BRIEF prompts yield None, no cross-turn leak,
byte-equal across runs, parametrized co-population check
alongside findings.
Verification
------------
pytest tests/test_plan_metrics*.py 18/18 pass
pytest tests/test_plan_contemplation*.py 17/17 pass (Phase 3)
pytest tests/test_discourse_planner_*.py 99/99 pass
pytest tests/test_articulation_demo.py all claims supported
pytest tests/test_narrative_example_intents.py pass
pytest tests/test_runtime_config.py pass
cognition eval OFF vs ON 45/45 surface byte-equal
45/45 trace_hash byte-equal
4/4 aggregate metrics
identical
core test --suite smoke 67/67 pass
core test --suite runtime 19/19 pass
Phase 5 (logged, not built)
---------------------------
Offline contemplation miner that consumes ``last_plan_findings``
+ ``last_plan_metrics`` streams across many turns and emits
reviewable pack-mutation candidates. Still SPECULATIVE;
review-gated; never auto-promoted to memory. Now unblocked by
the structured metric surface Phase 4 lands.
120 lines
4.4 KiB
Python
120 lines
4.4 KiB
Python
"""Phase 4 — end-to-end ``last_plan_metrics`` runtime wiring.
|
|
|
|
Mirrors ``tests/test_plan_contemplation_runtime.py`` for the
|
|
quantitative companion. Pins:
|
|
|
|
* Disabled by default — ``last_plan_metrics`` stays ``None`` even
|
|
when the planner engages.
|
|
* Enabled — metrics are populated whenever the planner engages.
|
|
* BRIEF prompts (fast-path) yield ``None`` metrics.
|
|
* Metrics do not leak across turns.
|
|
* Same prompt → byte-equal ``as_dict()`` (determinism).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from chat.runtime import ChatRuntime
|
|
from core.config import RuntimeConfig
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Disabled by default
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_metrics_none_when_contemplation_disabled() -> None:
|
|
rt = ChatRuntime(config=RuntimeConfig(discourse_contemplation=False))
|
|
rt.chat("What is truth, and why does it matter?")
|
|
assert rt.last_plan_metrics is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Enabled — multi-move plan populates structured metrics
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_compound_prompt_yields_expected_shape() -> None:
|
|
rt = ChatRuntime(config=RuntimeConfig(discourse_contemplation=True))
|
|
rt.chat("What is truth, and why does it matter?")
|
|
m = rt.last_plan_metrics
|
|
assert m is not None
|
|
# Compound prompt routes through two sub-plans plus a bridge.
|
|
assert m.move_count >= 4
|
|
assert m.fact_bearing_count >= 4
|
|
# Plan re-uses the truth subject across multiple moves; should
|
|
# therefore expose pronominalization opportunities.
|
|
assert m.pronominalization_opportunities >= 1
|
|
# Diversity ratios resolve to real numbers (no None) on a
|
|
# multi-move plan with >= 1 fact-bearing move.
|
|
assert m.predicate_diversity_ratio is not None
|
|
assert 0.0 < m.predicate_diversity_ratio <= 1.0
|
|
assert m.subject_focus_ratio is not None
|
|
assert 0.0 <= m.subject_focus_ratio <= 1.0
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# BRIEF prompts (fast-path) yield no metrics
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_brief_prompt_yields_no_metrics() -> None:
|
|
rt = ChatRuntime(config=RuntimeConfig(discourse_contemplation=True))
|
|
rt.chat("What is knowledge?")
|
|
assert rt.last_plan_metrics is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Metrics do not leak across turns
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_metrics_reset_between_turns() -> None:
|
|
rt = ChatRuntime(config=RuntimeConfig(discourse_contemplation=True))
|
|
rt.chat("What is truth, and why does it matter?")
|
|
assert rt.last_plan_metrics is not None # sanity
|
|
rt.chat("What is knowledge?") # BRIEF — should clear
|
|
assert rt.last_plan_metrics is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Determinism across two runs
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_metrics_byte_equal_across_runs() -> None:
|
|
rt1 = ChatRuntime(config=RuntimeConfig(discourse_contemplation=True))
|
|
rt2 = ChatRuntime(config=RuntimeConfig(discourse_contemplation=True))
|
|
rt1.chat("Tell me about memory.")
|
|
rt2.chat("Tell me about memory.")
|
|
m1 = rt1.last_plan_metrics
|
|
m2 = rt2.last_plan_metrics
|
|
assert m1 is not None and m2 is not None
|
|
assert m1.as_dict() == m2.as_dict()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Findings and metrics co-populate cleanly
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"prompt",
|
|
[
|
|
"Tell me about memory.",
|
|
"Explain truth.",
|
|
"What is truth, and why does it matter?",
|
|
],
|
|
)
|
|
def test_findings_and_metrics_populate_together(prompt: str) -> None:
|
|
rt = ChatRuntime(config=RuntimeConfig(discourse_contemplation=True))
|
|
rt.chat(prompt)
|
|
metrics = rt.last_plan_metrics
|
|
# Whenever metrics is populated, the planner engaged; findings
|
|
# is at least an empty tuple (never None on engaged turns).
|
|
assert metrics is not None
|
|
assert isinstance(rt.last_plan_findings, tuple)
|
|
# And the metrics' fact_bearing_count is non-zero on every
|
|
# engaged turn.
|
|
assert metrics.fact_bearing_count >= 1
|