b07fb0413c
Quantitative companion to Phase 3 (commit 664e081). Where Phase 3
emits SPECULATIVE *findings* about plan quality, Phase 4 emits
typed *measurements* — pure-function projection of a
``DiscoursePlan`` into a ``PlanMetrics`` dataclass.
Why this matters
----------------
The discourse planner now produces multi-clause grounded
articulations (Phase 1), the renderer pronominalizes across
consecutive same-subject moves (Phase 2), and the contemplation
pre-flight emits qualitative concerns about plan shape (Phase 3).
What was missing was the *aggregable* layer: per-turn structured
numbers that downstream consumers can stream across many turns
to score quality patterns the per-turn observer cannot see.
Phase 4 lands that layer. Phase 5 (offline contemplation miner)
becomes possible because there's now structured signal to mine.
What it measures
----------------
Structure
* move_count — total moves in plan
* fact_bearing_count — moves with fact != None
Move-kind distribution
* anchor_count / support_count / relation_count
/ transition_count / closure_count
Diversity
* unique_predicates — distinct predicates across
fact-bearing moves
* unique_subjects — distinct subject lemmas
* unique_sources — distinct FactSources
Topic dynamics
* topic_shift_count — consecutive pairs where
subject changed
* pronominalization_opportunities — consecutive pairs where
subject held (= Phase 2's
anaphora trigger count)
Derived ratios
* predicate_diversity_ratio — unique_predicates /
fact_bearing_count
* subject_focus_ratio — pronominalizations /
(pronominalizations +
topic_shifts)
Every field is a deterministic pure function of the plan: same
plan in → byte-equal ``PlanMetrics.as_dict()`` out. This is the
load-bearing claim that lets Phase 5 aggregate across turns
without "is this the same metric?" ambiguity.
Doctrine alignment
------------------
Per ADR-0080 contemplation discipline:
* Read-only — metrics are pure projections of the plan; no
mutation of plan, runtime state, or memory tiers.
* No autonomous learning — metrics are observations, not
learned policy. Promotion to memory still flows through
the existing proposal-review-ratify chain.
* Deterministic replay — pinned by test_metrics_are_deterministic_
and_byte_equal_as_dict plus the runtime-level
test_metrics_byte_equal_across_runs.
Wiring
------
* New ``ChatRuntime.last_plan_metrics`` property — read-only
``PlanMetrics`` from the most recent turn where the planner
engaged (and ``discourse_contemplation`` was on); ``None``
otherwise. Reset between turns alongside ``last_plan_findings``
via the existing top-of-call reset block.
* Same opt-in flag as Phase 3 (``discourse_contemplation``).
When True, the runtime computes both findings AND metrics in
the same block; when False (default), both stay at empty/None.
Demo (config: discourse_contemplation=True)
-------------------------------------------
"What is knowledge?" → metrics: None (BRIEF fast-path)
"Tell me about memory." → moves=3 fact_bearing=3
kinds=A:1/S:1/R:1/T:0/C:0
unique_predicates=3 subjects=1
pronominalization_ops=2 shifts=0
predicate_diversity=1.000
subject_focus=1.000
"What is truth, and why does
it matter?" → moves=7 fact_bearing=6
kinds=A:2/S:2/R:2/T:1/C:0
unique_predicates=4 subjects=1
pronominalization_ops=4 shifts=1
predicate_diversity=0.667 ← Phase 3
WEAK_SURFACE
quantified
subject_focus=0.800
+ 1 finding (weak_surface)
The compound-prompt numbers are particularly informative:
``predicate_diversity=0.667`` is the algebraic expression of the
Phase 3 ``WEAK_SURFACE`` rule — the rule fires precisely because
6 fact-bearing moves used only 4 distinct predicates.
``subject_focus=0.800`` quantifies that 80% of consecutive pairs
held the same subject — high topic stickiness that Phase 2's
reflective renderer leveraged into 4 ``it`` substitutions.
Tests
-----
* ``tests/test_plan_metrics.py`` — 10 unit tests pinning each
field, derived ratios, bridge-move handling (``fact=None``
resets the focus channel), and determinism via ``as_dict()``
byte-equality.
* ``tests/test_plan_metrics_runtime.py`` — 8 end-to-end tests
proving the runtime wiring: disabled by default, populated
when enabled, BRIEF prompts yield None, no cross-turn leak,
byte-equal across runs, parametrized co-population check
alongside findings.
Verification
------------
pytest tests/test_plan_metrics*.py 18/18 pass
pytest tests/test_plan_contemplation*.py 17/17 pass (Phase 3)
pytest tests/test_discourse_planner_*.py 99/99 pass
pytest tests/test_articulation_demo.py all claims supported
pytest tests/test_narrative_example_intents.py pass
pytest tests/test_runtime_config.py pass
cognition eval OFF vs ON 45/45 surface byte-equal
45/45 trace_hash byte-equal
4/4 aggregate metrics
identical
core test --suite smoke 67/67 pass
core test --suite runtime 19/19 pass
Phase 5 (logged, not built)
---------------------------
Offline contemplation miner that consumes ``last_plan_findings``
+ ``last_plan_metrics`` streams across many turns and emits
reviewable pack-mutation candidates. Still SPECULATIVE;
review-gated; never auto-promoted to memory. Now unblocked by
the structured metric surface Phase 4 lands.
351 lines
11 KiB
Python
351 lines
11 KiB
Python
"""Phase 4 — per-plan articulation telemetry metrics.
|
|
|
|
Pins ``core.contemplation.plan_metrics.compute_plan_metrics`` against:
|
|
|
|
* Trivial cases (empty plan, single anchor)
|
|
* Structural counts (move_kind distribution)
|
|
* Diversity counts (unique predicates / subjects / sources)
|
|
* Topic dynamics (pronominalization opportunities, topic shifts)
|
|
* Derived ratios (predicate_diversity_ratio, subject_focus_ratio)
|
|
* Determinism (same plan → byte-equal metrics dict)
|
|
* Bridge-move handling (fact=None resets focus channel)
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from core.contemplation.plan_metrics import compute_plan_metrics
|
|
from generate.discourse_planner import (
|
|
DiscourseMove,
|
|
DiscourseMoveKind,
|
|
DiscoursePlan,
|
|
FactSource,
|
|
GroundedFact,
|
|
)
|
|
from generate.intent import DialogueIntent, IntentTag, ResponseMode
|
|
|
|
|
|
def _fact(
|
|
subject: str,
|
|
predicate: str,
|
|
obj: str,
|
|
*,
|
|
source: FactSource = FactSource.PACK,
|
|
source_id: str = "test_pack_v1",
|
|
) -> GroundedFact:
|
|
return GroundedFact(
|
|
subject=subject,
|
|
predicate=predicate,
|
|
obj=obj,
|
|
source=source,
|
|
source_id=source_id,
|
|
)
|
|
|
|
|
|
def _intent(subject: str = "truth") -> DialogueIntent:
|
|
return DialogueIntent(tag=IntentTag.DEFINITION, subject=subject)
|
|
|
|
|
|
def _move(
|
|
kind: DiscourseMoveKind, fact: GroundedFact | None = None,
|
|
) -> DiscourseMove:
|
|
topic = fact.subject if fact is not None else ""
|
|
return DiscourseMove(
|
|
kind=kind, topic=topic, given=(), new=(),
|
|
relation_to_previous=None, fact=fact,
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Empty plan
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_empty_plan_yields_zero_metrics() -> None:
|
|
plan = DiscoursePlan(intent=_intent(), mode=ResponseMode.BRIEF, moves=())
|
|
m = compute_plan_metrics(plan)
|
|
assert m.move_count == 0
|
|
assert m.fact_bearing_count == 0
|
|
assert m.anchor_count == 0
|
|
assert m.unique_predicates == 0
|
|
assert m.unique_subjects == 0
|
|
assert m.unique_sources == 0
|
|
assert m.topic_shift_count == 0
|
|
assert m.pronominalization_opportunities == 0
|
|
assert m.predicate_diversity_ratio is None
|
|
assert m.subject_focus_ratio is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Single-anchor plan
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_single_anchor_plan_metrics() -> None:
|
|
plan = DiscoursePlan(
|
|
intent=_intent(),
|
|
mode=ResponseMode.BRIEF,
|
|
moves=(
|
|
_move(
|
|
DiscourseMoveKind.ANCHOR,
|
|
_fact("truth", "is_defined_as", "what is true"),
|
|
),
|
|
),
|
|
)
|
|
m = compute_plan_metrics(plan)
|
|
assert m.move_count == 1
|
|
assert m.fact_bearing_count == 1
|
|
assert m.anchor_count == 1
|
|
assert m.support_count == 0
|
|
assert m.unique_predicates == 1
|
|
assert m.unique_subjects == 1
|
|
assert m.unique_sources == 1
|
|
assert m.topic_shift_count == 0
|
|
assert m.pronominalization_opportunities == 0
|
|
assert m.predicate_diversity_ratio == 1.0
|
|
# No consecutive pairs to measure — ratio undefined
|
|
assert m.subject_focus_ratio is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Move-kind distribution
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_move_kind_distribution_counts() -> None:
|
|
plan = DiscoursePlan(
|
|
intent=_intent(),
|
|
mode=ResponseMode.PARAGRAPH,
|
|
moves=(
|
|
_move(
|
|
DiscourseMoveKind.ANCHOR,
|
|
_fact("truth", "is_defined_as", "what is true"),
|
|
),
|
|
_move(
|
|
DiscourseMoveKind.SUPPORT,
|
|
_fact("truth", "belongs_to", "cognition.truth"),
|
|
),
|
|
_move(
|
|
DiscourseMoveKind.RELATION,
|
|
_fact("truth", "grounds", "knowledge"),
|
|
),
|
|
_move(
|
|
DiscourseMoveKind.TRANSITION,
|
|
_fact("knowledge", "belongs_to", "cognition.knowledge"),
|
|
),
|
|
_move(DiscourseMoveKind.CLOSURE), # fact=None
|
|
),
|
|
)
|
|
m = compute_plan_metrics(plan)
|
|
assert m.move_count == 5
|
|
assert m.fact_bearing_count == 4
|
|
assert m.anchor_count == 1
|
|
assert m.support_count == 1
|
|
assert m.relation_count == 1
|
|
assert m.transition_count == 1
|
|
assert m.closure_count == 1
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Pronominalization opportunities vs. topic shifts
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_three_same_subject_moves_yield_two_pronominalization_opportunities() -> None:
|
|
plan = DiscoursePlan(
|
|
intent=_intent(),
|
|
mode=ResponseMode.PARAGRAPH,
|
|
moves=(
|
|
_move(
|
|
DiscourseMoveKind.ANCHOR,
|
|
_fact("truth", "is_defined_as", "what is true"),
|
|
),
|
|
_move(
|
|
DiscourseMoveKind.SUPPORT,
|
|
_fact("truth", "belongs_to", "cognition.truth"),
|
|
),
|
|
_move(
|
|
DiscourseMoveKind.RELATION,
|
|
_fact("truth", "grounds", "knowledge"),
|
|
),
|
|
),
|
|
)
|
|
m = compute_plan_metrics(plan)
|
|
assert m.pronominalization_opportunities == 2
|
|
assert m.topic_shift_count == 0
|
|
assert m.subject_focus_ratio == 1.0
|
|
|
|
|
|
def test_topic_shift_counted_when_subject_changes() -> None:
|
|
plan = DiscoursePlan(
|
|
intent=_intent(),
|
|
mode=ResponseMode.PARAGRAPH,
|
|
moves=(
|
|
_move(
|
|
DiscourseMoveKind.ANCHOR,
|
|
_fact("truth", "is_defined_as", "what is true"),
|
|
),
|
|
_move(
|
|
DiscourseMoveKind.TRANSITION,
|
|
_fact("knowledge", "belongs_to", "cognition.knowledge"),
|
|
),
|
|
),
|
|
)
|
|
m = compute_plan_metrics(plan)
|
|
assert m.topic_shift_count == 1
|
|
assert m.pronominalization_opportunities == 0
|
|
assert m.subject_focus_ratio == 0.0
|
|
|
|
|
|
def test_bridge_move_resets_focus_channel() -> None:
|
|
"""A fact-bearing move followed by a bridge (``fact=None``) followed
|
|
by another fact-bearing move with the SAME subject must not count
|
|
as a pronominalization opportunity — the bridge breaks the
|
|
consecutive-pair channel."""
|
|
plan = DiscoursePlan(
|
|
intent=_intent(),
|
|
mode=ResponseMode.PARAGRAPH,
|
|
moves=(
|
|
_move(
|
|
DiscourseMoveKind.ANCHOR,
|
|
_fact("truth", "is_defined_as", "what is true"),
|
|
),
|
|
_move(DiscourseMoveKind.TRANSITION), # bridge, fact=None
|
|
_move(
|
|
DiscourseMoveKind.SUPPORT,
|
|
_fact("truth", "belongs_to", "cognition.truth"),
|
|
),
|
|
),
|
|
)
|
|
m = compute_plan_metrics(plan)
|
|
# Bridge counts as a shift; no pronominalization opportunity even
|
|
# though both fact-bearing moves share subject "truth".
|
|
assert m.topic_shift_count == 1
|
|
assert m.pronominalization_opportunities == 0
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Diversity counts
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_predicate_diversity_ratio_reflects_monotony() -> None:
|
|
"""Three moves with the same predicate → diversity ratio 1/3."""
|
|
plan = DiscoursePlan(
|
|
intent=_intent(),
|
|
mode=ResponseMode.PARAGRAPH,
|
|
moves=(
|
|
_move(
|
|
DiscourseMoveKind.ANCHOR,
|
|
_fact("truth", "belongs_to", "cognition.truth"),
|
|
),
|
|
_move(
|
|
DiscourseMoveKind.SUPPORT,
|
|
_fact("truth", "belongs_to", "epistemic.ground"),
|
|
),
|
|
_move(
|
|
DiscourseMoveKind.RELATION,
|
|
_fact("truth", "belongs_to", "logos.core"),
|
|
),
|
|
),
|
|
)
|
|
m = compute_plan_metrics(plan)
|
|
assert m.unique_predicates == 1
|
|
assert m.fact_bearing_count == 3
|
|
assert m.predicate_diversity_ratio is not None
|
|
assert abs(m.predicate_diversity_ratio - (1.0 / 3.0)) < 1e-9
|
|
|
|
|
|
def test_source_diversity_counts_pack_plus_teaching() -> None:
|
|
plan = DiscoursePlan(
|
|
intent=_intent(),
|
|
mode=ResponseMode.EXPLAIN,
|
|
moves=(
|
|
_move(
|
|
DiscourseMoveKind.ANCHOR,
|
|
_fact(
|
|
"truth", "is_defined_as", "what is true",
|
|
source=FactSource.PACK,
|
|
),
|
|
),
|
|
_move(
|
|
DiscourseMoveKind.RELATION,
|
|
_fact(
|
|
"truth", "grounds", "knowledge",
|
|
source=FactSource.TEACHING,
|
|
source_id="cognition_chains_v1",
|
|
),
|
|
),
|
|
),
|
|
)
|
|
m = compute_plan_metrics(plan)
|
|
assert m.unique_sources == 2
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Determinism
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_metrics_are_deterministic_and_byte_equal_as_dict() -> None:
|
|
plan = DiscoursePlan(
|
|
intent=_intent(),
|
|
mode=ResponseMode.PARAGRAPH,
|
|
moves=(
|
|
_move(
|
|
DiscourseMoveKind.ANCHOR,
|
|
_fact("truth", "is_defined_as", "what is true"),
|
|
),
|
|
_move(
|
|
DiscourseMoveKind.SUPPORT,
|
|
_fact("truth", "belongs_to", "cognition.truth"),
|
|
),
|
|
_move(
|
|
DiscourseMoveKind.RELATION,
|
|
_fact("truth", "grounds", "knowledge"),
|
|
),
|
|
),
|
|
)
|
|
a = compute_plan_metrics(plan)
|
|
b = compute_plan_metrics(plan)
|
|
assert a == b
|
|
assert a.as_dict() == b.as_dict()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# as_dict surface
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_as_dict_includes_every_field_and_derived_ratios() -> None:
|
|
plan = DiscoursePlan(
|
|
intent=_intent(),
|
|
mode=ResponseMode.EXPLAIN,
|
|
moves=(
|
|
_move(
|
|
DiscourseMoveKind.ANCHOR,
|
|
_fact("truth", "is_defined_as", "what is true"),
|
|
),
|
|
_move(
|
|
DiscourseMoveKind.SUPPORT,
|
|
_fact("truth", "belongs_to", "cognition.truth"),
|
|
),
|
|
),
|
|
)
|
|
d = compute_plan_metrics(plan).as_dict()
|
|
for required_field in (
|
|
"move_count",
|
|
"fact_bearing_count",
|
|
"anchor_count",
|
|
"support_count",
|
|
"relation_count",
|
|
"transition_count",
|
|
"closure_count",
|
|
"unique_predicates",
|
|
"unique_subjects",
|
|
"unique_sources",
|
|
"topic_shift_count",
|
|
"pronominalization_opportunities",
|
|
"predicate_diversity_ratio",
|
|
"subject_focus_ratio",
|
|
):
|
|
assert required_field in d, f"missing field {required_field!r}"
|