Quantitative companion to Phase 3 (commit 664e081). Where Phase 3
emits SPECULATIVE *findings* about plan quality, Phase 4 emits
typed *measurements* — pure-function projection of a
``DiscoursePlan`` into a ``PlanMetrics`` dataclass.
Why this matters
----------------
The discourse planner now produces multi-clause grounded
articulations (Phase 1), the renderer pronominalizes across
consecutive same-subject moves (Phase 2), and the contemplation
pre-flight emits qualitative concerns about plan shape (Phase 3).
What was missing was the *aggregable* layer: per-turn structured
numbers that downstream consumers can stream across many turns
to score quality patterns the per-turn observer cannot see.
Phase 4 lands that layer. Phase 5 (offline contemplation miner)
becomes possible because there's now structured signal to mine.
What it measures
----------------
Structure
* move_count — total moves in plan
* fact_bearing_count — moves with fact != None
Move-kind distribution
* anchor_count / support_count / relation_count
/ transition_count / closure_count
Diversity
* unique_predicates — distinct predicates across
fact-bearing moves
* unique_subjects — distinct subject lemmas
* unique_sources — distinct FactSources
Topic dynamics
* topic_shift_count — consecutive pairs where
subject changed
* pronominalization_opportunities — consecutive pairs where
subject held (= Phase 2's
anaphora trigger count)
Derived ratios
* predicate_diversity_ratio — unique_predicates /
fact_bearing_count
* subject_focus_ratio — pronominalizations /
(pronominalizations +
topic_shifts)
Every field is a deterministic pure function of the plan: same
plan in → byte-equal ``PlanMetrics.as_dict()`` out. This is the
load-bearing claim that lets Phase 5 aggregate across turns
without "is this the same metric?" ambiguity.
Doctrine alignment
------------------
Per ADR-0080 contemplation discipline:
* Read-only — metrics are pure projections of the plan; no
mutation of plan, runtime state, or memory tiers.
* No autonomous learning — metrics are observations, not
learned policy. Promotion to memory still flows through
the existing proposal-review-ratify chain.
* Deterministic replay — pinned by test_metrics_are_deterministic_
and_byte_equal_as_dict plus the runtime-level
test_metrics_byte_equal_across_runs.
Wiring
------
* New ``ChatRuntime.last_plan_metrics`` property — read-only
``PlanMetrics`` from the most recent turn where the planner
engaged (and ``discourse_contemplation`` was on); ``None``
otherwise. Reset between turns alongside ``last_plan_findings``
via the existing top-of-call reset block.
* Same opt-in flag as Phase 3 (``discourse_contemplation``).
When True, the runtime computes both findings AND metrics in
the same block; when False (default), both stay at empty/None.
Demo (config: discourse_contemplation=True)
-------------------------------------------
"What is knowledge?" → metrics: None (BRIEF fast-path)
"Tell me about memory." → moves=3 fact_bearing=3
kinds=A:1/S:1/R:1/T:0/C:0
unique_predicates=3 subjects=1
pronominalization_ops=2 shifts=0
predicate_diversity=1.000
subject_focus=1.000
"What is truth, and why does
it matter?" → moves=7 fact_bearing=6
kinds=A:2/S:2/R:2/T:1/C:0
unique_predicates=4 subjects=1
pronominalization_ops=4 shifts=1
predicate_diversity=0.667 ← Phase 3
WEAK_SURFACE
quantified
subject_focus=0.800
+ 1 finding (weak_surface)
The compound-prompt numbers are particularly informative:
``predicate_diversity=0.667`` is the algebraic expression of the
Phase 3 ``WEAK_SURFACE`` rule — the rule fires precisely because
6 fact-bearing moves used only 4 distinct predicates.
``subject_focus=0.800`` quantifies that 80% of consecutive pairs
held the same subject — high topic stickiness that Phase 2's
reflective renderer leveraged into 4 ``it`` substitutions.
Tests
-----
* ``tests/test_plan_metrics.py`` — 10 unit tests pinning each
field, derived ratios, bridge-move handling (``fact=None``
resets the focus channel), and determinism via ``as_dict()``
byte-equality.
* ``tests/test_plan_metrics_runtime.py`` — 8 end-to-end tests
proving the runtime wiring: disabled by default, populated
when enabled, BRIEF prompts yield None, no cross-turn leak,
byte-equal across runs, parametrized co-population check
alongside findings.
Verification
------------
pytest tests/test_plan_metrics*.py 18/18 pass
pytest tests/test_plan_contemplation*.py 17/17 pass (Phase 3)
pytest tests/test_discourse_planner_*.py 99/99 pass
pytest tests/test_articulation_demo.py all claims supported
pytest tests/test_narrative_example_intents.py pass
pytest tests/test_runtime_config.py pass
cognition eval OFF vs ON 45/45 surface byte-equal
45/45 trace_hash byte-equal
4/4 aggregate metrics
identical
core test --suite smoke 67/67 pass
core test --suite runtime 19/19 pass
Phase 5 (logged, not built)
---------------------------
Offline contemplation miner that consumes ``last_plan_findings``
+ ``last_plan_metrics`` streams across many turns and emits
reviewable pack-mutation candidates. Still SPECULATIVE;
review-gated; never auto-promoted to memory. Now unblocked by
the structured metric surface Phase 4 lands.
247 lines
9 KiB
Python
247 lines
9 KiB
Python
"""Phase 4 — per-plan articulation telemetry metrics.
|
|
|
|
Pure-function projection of a ``DiscoursePlan`` into structured
|
|
quantitative measurements. Mirrors Phase 3's ``plan_preflight``
|
|
contemplation:
|
|
|
|
Phase 3 (plan_preflight) → typed SPECULATIVE *findings* (qualitative)
|
|
Phase 4 (plan_metrics) → typed *measurements* (quantitative)
|
|
|
|
Both run after the planner finishes; neither mutates anything.
|
|
Findings answer "what's wrong with this plan?". Metrics answer
|
|
"what shape does this plan have?". Together they give downstream
|
|
consumers (offline contemplation miner, operator dashboards) the
|
|
signal they need to score plan quality across many turns.
|
|
|
|
Why a separate dataclass and not just a dict
|
|
--------------------------------------------
|
|
|
|
* **Typed boundary.** ``PlanMetrics`` field types make the
|
|
serialization contract explicit; a downstream consumer can't
|
|
silently break on a renamed key.
|
|
* **Deterministic identity.** ``frozen=True`` + ``slots=True`` +
|
|
positional ``as_dict()`` keys means two metrics objects built from
|
|
byte-equal plans serialize identically. This is what lets the
|
|
offline miner aggregate over time without "is this the same
|
|
metric?" ambiguity.
|
|
* **Cheap.** Computation is O(moves); no allocation per move
|
|
beyond the dataclass itself.
|
|
|
|
Doctrine notes
|
|
--------------
|
|
|
|
Metrics are pure measurements, not opinions. They never mutate
|
|
the plan, the runtime state, or the memory tiers. Promotion to
|
|
memory still flows through the existing proposal-review-ratify
|
|
chain. Where Phase 3 emits SPECULATIVE *findings* (which downstream
|
|
review may accept), Phase 4 emits raw numbers (which downstream
|
|
analytics may aggregate).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from collections import Counter
|
|
from dataclasses import dataclass
|
|
from typing import Any
|
|
|
|
from generate.discourse_planner import (
|
|
DiscourseMoveKind,
|
|
DiscoursePlan,
|
|
)
|
|
|
|
|
|
@dataclass(frozen=True, slots=True)
|
|
class PlanMetrics:
|
|
"""Quantitative measurements of one ``DiscoursePlan``.
|
|
|
|
Every field is a pure function of the plan; same plan in →
|
|
byte-identical metrics out. Used by Phase 5 (offline miner)
|
|
to aggregate plan-quality signal across many turns and surface
|
|
deeper structural patterns that single-plan contemplation
|
|
(Phase 3) cannot see.
|
|
"""
|
|
|
|
# ------ Structure ------
|
|
|
|
move_count: int
|
|
"""Total moves in the plan, including those without facts (e.g.
|
|
bridge ``TRANSITION`` moves with ``fact=None`` and ``CLOSURE``
|
|
moves without summary facts)."""
|
|
|
|
fact_bearing_count: int
|
|
"""Moves with ``fact is not None`` — these are the moves the
|
|
renderer actually emits clauses for. ``fact_bearing_count``
|
|
< ``move_count`` indicates structural moves (bridges, closures)
|
|
the renderer elides."""
|
|
|
|
# ------ Move-kind distribution ------
|
|
|
|
anchor_count: int
|
|
support_count: int
|
|
relation_count: int
|
|
transition_count: int
|
|
closure_count: int
|
|
|
|
# ------ Diversity ------
|
|
|
|
unique_predicates: int
|
|
"""Number of distinct predicate strings across fact-bearing moves.
|
|
Low absolute counts paired with high move_count signal predicate
|
|
monotony (the WEAK_SURFACE finding from Phase 3)."""
|
|
|
|
unique_subjects: int
|
|
"""Number of distinct subject lemmas across fact-bearing moves."""
|
|
|
|
unique_sources: int
|
|
"""Number of distinct ``FactSource`` values across fact-bearing
|
|
moves. ``unique_sources == 1`` with multi-move plans signals
|
|
the COVERAGE_GAP finding from Phase 3."""
|
|
|
|
# ------ Topic dynamics ------
|
|
|
|
topic_shift_count: int
|
|
"""Number of consecutive-move pairs where the fact subject
|
|
changed. Counts transitions across the visible focus channel
|
|
that Phase 2 reflective rendering uses; ``topic_shift_count``
|
|
+ ``pronominalization_opportunities`` + 1 (for the anchor) sums
|
|
to ``fact_bearing_count`` minus zero-fact moves."""
|
|
|
|
pronominalization_opportunities: int
|
|
"""Number of consecutive-move pairs where the fact subject
|
|
repeated. Phase 2's reflective renderer takes each opportunity
|
|
to swap the subject token to ``it``."""
|
|
|
|
def as_dict(self) -> dict[str, Any]:
|
|
return {
|
|
"move_count": self.move_count,
|
|
"fact_bearing_count": self.fact_bearing_count,
|
|
"anchor_count": self.anchor_count,
|
|
"support_count": self.support_count,
|
|
"relation_count": self.relation_count,
|
|
"transition_count": self.transition_count,
|
|
"closure_count": self.closure_count,
|
|
"unique_predicates": self.unique_predicates,
|
|
"unique_subjects": self.unique_subjects,
|
|
"unique_sources": self.unique_sources,
|
|
"topic_shift_count": self.topic_shift_count,
|
|
"pronominalization_opportunities": (
|
|
self.pronominalization_opportunities
|
|
),
|
|
# Derived ratios — included in the wire format so consumers
|
|
# don't recompute them inconsistently. ``None`` when undefined
|
|
# (e.g. empty plan, single-move plan with no pairs).
|
|
"predicate_diversity_ratio": self.predicate_diversity_ratio,
|
|
"subject_focus_ratio": self.subject_focus_ratio,
|
|
}
|
|
|
|
# ---- Derived ratios ----
|
|
|
|
@property
|
|
def predicate_diversity_ratio(self) -> float | None:
|
|
"""``unique_predicates / fact_bearing_count`` — ``None`` when
|
|
no fact-bearing moves (nothing to divide by).
|
|
|
|
1.0 = every fact-bearing move uses a distinct predicate (most
|
|
diverse). Trending toward 0 = predicates repeating (Phase 3
|
|
``WEAK_SURFACE`` candidate).
|
|
"""
|
|
if self.fact_bearing_count == 0:
|
|
return None
|
|
return self.unique_predicates / self.fact_bearing_count
|
|
|
|
@property
|
|
def subject_focus_ratio(self) -> float | None:
|
|
"""Fraction of consecutive-move pairs that held subject focus
|
|
(i.e. the inverse of topic-shift rate). ``None`` when there
|
|
are no consecutive pairs (< 2 fact-bearing moves).
|
|
|
|
1.0 = perfectly stuck on one topic (every pronominalization
|
|
opportunity engaged). Trending toward 0 = topic shifts on
|
|
every move (compound or wandering plan).
|
|
"""
|
|
total_pairs = (
|
|
self.pronominalization_opportunities + self.topic_shift_count
|
|
)
|
|
if total_pairs == 0:
|
|
return None
|
|
return self.pronominalization_opportunities / total_pairs
|
|
|
|
|
|
def compute_plan_metrics(plan: DiscoursePlan) -> PlanMetrics:
|
|
"""Project a :class:`DiscoursePlan` into a :class:`PlanMetrics`.
|
|
|
|
Pure deterministic function: ``compute_plan_metrics(p) ==
|
|
compute_plan_metrics(p)`` byte-identical for any plan ``p``.
|
|
|
|
Empty plans yield a zero-valued ``PlanMetrics`` so downstream
|
|
consumers can use the same shape regardless of plan engagement.
|
|
"""
|
|
|
|
if plan.is_empty():
|
|
return PlanMetrics(
|
|
move_count=0,
|
|
fact_bearing_count=0,
|
|
anchor_count=0,
|
|
support_count=0,
|
|
relation_count=0,
|
|
transition_count=0,
|
|
closure_count=0,
|
|
unique_predicates=0,
|
|
unique_subjects=0,
|
|
unique_sources=0,
|
|
topic_shift_count=0,
|
|
pronominalization_opportunities=0,
|
|
)
|
|
|
|
kind_counts: Counter[DiscourseMoveKind] = Counter()
|
|
predicates: set[str] = set()
|
|
subjects: set[str] = set()
|
|
sources: set[Any] = set()
|
|
fact_bearing = 0
|
|
prior_subject: str | None = None
|
|
topic_shifts = 0
|
|
pronominalizations = 0
|
|
|
|
for move in plan.moves:
|
|
kind_counts[move.kind] += 1
|
|
if move.fact is None:
|
|
# Bridge / closure-without-summary moves don't carry a
|
|
# subject focus — they reset the channel. Track a topic
|
|
# shift so the focus_ratio reflects the discontinuity but
|
|
# do NOT update prior_subject (the next fact-bearing move
|
|
# establishes new focus from scratch).
|
|
if prior_subject is not None:
|
|
topic_shifts += 1
|
|
prior_subject = None
|
|
continue
|
|
fact_bearing += 1
|
|
predicates.add(move.fact.predicate)
|
|
subjects.add(move.fact.subject)
|
|
sources.add(move.fact.source)
|
|
if prior_subject is not None:
|
|
if move.fact.subject == prior_subject:
|
|
pronominalizations += 1
|
|
else:
|
|
topic_shifts += 1
|
|
prior_subject = move.fact.subject
|
|
|
|
return PlanMetrics(
|
|
move_count=len(plan.moves),
|
|
fact_bearing_count=fact_bearing,
|
|
anchor_count=kind_counts.get(DiscourseMoveKind.ANCHOR, 0),
|
|
support_count=kind_counts.get(DiscourseMoveKind.SUPPORT, 0),
|
|
relation_count=kind_counts.get(DiscourseMoveKind.RELATION, 0),
|
|
transition_count=kind_counts.get(DiscourseMoveKind.TRANSITION, 0),
|
|
closure_count=kind_counts.get(DiscourseMoveKind.CLOSURE, 0),
|
|
unique_predicates=len(predicates),
|
|
unique_subjects=len(subjects),
|
|
unique_sources=len(sources),
|
|
topic_shift_count=topic_shifts,
|
|
pronominalization_opportunities=pronominalizations,
|
|
)
|
|
|
|
|
|
__all__ = [
|
|
"PlanMetrics",
|
|
"compute_plan_metrics",
|
|
]
|