feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer
Closes the user-flagged scope gap: every previous fluency lane (Phase 5.1 + 5.4-5.7 + grammatical_coverage) operates on 3-word SVO probes. These three pieces stress paragraph-scale generation, give per-stage latency visibility, and expose the realizer's word-choice geometry — all on top of the existing deterministic infrastructure. # discourse_paragraph lane (paragraph-scale fluency) Forces the realizer to emit multi-sentence paragraphs from a multi-step ArticulationTarget with rhetorical moves (ASSERT, SEQUENCE, ELABORATE, CONTRAST). Same realizer, much richer input — every case is 3-5 sentences with deterministic discourse markers. Public 12 cases / holdouts 5 / dev 1 across 12 + 5 topic chains (epistemic, scientific method, creation arc, logical dependency, ethical grounding, linguistic layers, mathematical chain, narrative, biology, physics, two contrast-shaped, musical, social, computational, psychological, economic). Sub-metrics per case: - sentence count (within min..max window) - subject coverage rate - discourse marker presence (next / furthermore / in contrast) - sentence-initial capitalization - replay determinism (run twice, surfaces match) Result: 12/12 public + 5/5 holdouts at 100%, replay rate 100%, mean sentence count 4. # Realizer capitalization (G4, addresses user-flagged concern) generate/realizer.py gains `_capitalize_sentence` + `_join_as_paragraph` helpers. Sentence-initial alphabetic characters are now uppercased (skipping leading whitespace/punctuation). Surfaces went from "wisdom grounds knowledge. next, knowledge requires evidence." to "Wisdom grounds knowledge. Next, knowledge requires evidence." The discourse_paragraph runner ships a strict per-sentence capitalization check so future regressions get caught. # Pipeline-stage profiler (benchmarks/pipeline_profiler.py) External monkey-patch wrapper around CognitiveTurnPipeline.run() that records per-stage ns budgets without editing any pipeline source. Stages: intent, graph_planner, realize_semantic, runtime_chat, maybe_transitive_walk, fold_walk_into_surface, run_teaching, trace_hash. API: `profile_turn(pipeline, text) -> ProfileReport` with `.stages: dict`, `.total_ns: int`, `.as_dict()`. Empirical: runtime_chat dominates >99% on the runtime hot path (which is correct — that's where ingest + propagate + recall + articulate all happen). Future optimisation work has a clear per-stage signal. # Word-selection tracer (benchmarks/word_selection_tracer.py) External wrapper around generate.articulation._resolve_slot that records every nearest-neighbor lookup as a WordSelectionStep: - slot (subject/predicate/object) - input versor (32-d copy) - top-K candidate words by CGA inner product - chosen word + morphology - output language Top-K scoring uses the diagonal Cl(4,1) metric kernel from algebra.backend (same vectorised path vault_recall uses), not a per-word Python loop over cga_inner. No approximation, exact deterministic ranking, bit-identical to a scalar scan. API: `trace_realization(pipeline, text) -> RealizationTrace` with `.steps`, `.realization_steps`, `.surface`, `.as_dict()`. # CLI lane registration Cognition suite now sweeps the benchmark profiler/tracer tests (test_benchmarks_profiler.py) so any future regression in the instrumentation surfaces immediately. # Constraints honoured - Zero edits to core/, chat/, vault/, teaching/, language_packs/, or the algebra hot path. All instrumentation is external monkey-patch with originals restored in finally. - discourse_paragraph runner bypasses ChatRuntime grounding (named v2 gap) so paragraph capability is isolated to the realizer. - No semantic changes; no hidden normalisation; no approximate recall. # Lane health smoke 55, runtime 19, teaching 17, packs 6, cognition 105 (was 103), algebra 132. All Phase 5 fluency lanes still 100% with the capitalised surfaces (rubric is case-insensitive). discourse_paragraph 100%. # What ships next (named v2) - Round-trip: discourse_paragraph through ChatRuntime end-to-end, not just realize_target. - Per-sentence grammatical_coverage rubric on each emitted sentence. - Longer chains (10/20/50 sentences) with per-sentence determinism scaling curves. - compose_relations operator to lift compositionality recall from 68.8% toward 100%.
This commit is contained in:
parent
694754ab46
commit
257a27c105
15 changed files with 1520 additions and 8 deletions
182
benchmarks/pipeline_profiler.py
Normal file
182
benchmarks/pipeline_profiler.py
Normal file
|
|
@ -0,0 +1,182 @@
|
|||
"""Pipeline-stage profiler for CognitiveTurnPipeline.
|
||||
|
||||
External instrumentation only — no edits to pipeline/runtime/algebra/vault
|
||||
source files. Uses lightweight monkey-patching of bound methods on the
|
||||
pipeline instance and the runtime instance for the duration of a single
|
||||
``profile_turn`` call. All patches are reverted in a ``finally`` block so
|
||||
the pipeline is left untouched.
|
||||
|
||||
Per CLAUDE.md: no hidden normalization, no semantic mutation, no algebra
|
||||
hot-path touch. Overhead per stage: a single ``time.perf_counter_ns``
|
||||
read on entry and on exit, and a list append. Stage label strings are
|
||||
pre-interned at module load time (no f-strings inside timed regions).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Iterator
|
||||
|
||||
from core.cognition.pipeline import CognitiveTurnPipeline
|
||||
from core.cognition.result import CognitiveTurnResult
|
||||
|
||||
|
||||
# Pre-interned stage label constants — avoid string construction in
|
||||
# the timed hot path.
|
||||
_STAGE_INTENT = "intent"
|
||||
_STAGE_GRAPH = "graph_planner"
|
||||
_STAGE_REALIZE = "realize_semantic"
|
||||
_STAGE_RUNTIME_CHAT = "runtime_chat"
|
||||
_STAGE_TRANSITIVE_WALK = "maybe_transitive_walk"
|
||||
_STAGE_FOLD_WALK = "fold_walk_into_surface"
|
||||
_STAGE_TEACHING = "run_teaching"
|
||||
_STAGE_TRACE = "trace_hash"
|
||||
_STAGE_TOTAL = "total"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ProfileReport:
|
||||
"""Immutable timing report for a single profiled turn."""
|
||||
|
||||
stages: dict[str, int]
|
||||
total_ns: int
|
||||
result: CognitiveTurnResult
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"stages": dict(self.stages),
|
||||
"total_ns": int(self.total_ns),
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class _ProfileSink:
|
||||
"""Mutable per-call accumulator. Not shared across calls — instantiated
|
||||
fresh in every ``profile_turn`` invocation, so no global state."""
|
||||
|
||||
stages: dict[str, int] = field(default_factory=dict)
|
||||
|
||||
def record(self, name: str, elapsed_ns: int) -> None:
|
||||
# Multiple invocations of the same stage in a turn are summed.
|
||||
prior = self.stages.get(name, 0)
|
||||
self.stages[name] = prior + elapsed_ns
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _stage(sink: _ProfileSink, name: str) -> Iterator[None]:
|
||||
"""Lightweight context manager: two perf_counter_ns reads plus a dict update."""
|
||||
t0 = time.perf_counter_ns()
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
sink.record(name, time.perf_counter_ns() - t0)
|
||||
|
||||
|
||||
def profile_turn(
|
||||
pipeline: CognitiveTurnPipeline,
|
||||
text: str,
|
||||
max_tokens: int | None = None,
|
||||
) -> ProfileReport:
|
||||
"""Profile one CognitiveTurnPipeline.run() invocation.
|
||||
|
||||
Wraps the pipeline's existing internal methods and the runtime's
|
||||
``chat`` method with timing decorators for the duration of this call,
|
||||
then restores them. Patches live on the *instances*, not on the
|
||||
classes, so concurrent profiling of distinct pipeline instances is
|
||||
safe.
|
||||
"""
|
||||
sink = _ProfileSink()
|
||||
|
||||
# Capture originals (instance attrs win over class attrs in resolution,
|
||||
# so reassigning attrs on the instance does not mutate the class).
|
||||
runtime = pipeline.runtime
|
||||
orig_chat = runtime.chat
|
||||
orig_maybe_walk = pipeline._maybe_transitive_walk
|
||||
orig_fold = pipeline._fold_walk_into_surface
|
||||
orig_run_teaching = pipeline._run_teaching
|
||||
|
||||
# We patch generate.intent / graph_planner / realizer via per-call
|
||||
# module-attribute swaps on the pipeline module so we only time the
|
||||
# functions actually called from pipeline.run().
|
||||
from core.cognition import pipeline as pipeline_mod
|
||||
|
||||
orig_classify_intent = pipeline_mod.classify_intent
|
||||
orig_graph_from_intent = pipeline_mod.graph_from_intent
|
||||
orig_plan_articulation = pipeline_mod.plan_articulation
|
||||
orig_realize_semantic = pipeline_mod.realize_semantic
|
||||
orig_compute_trace_hash = pipeline_mod.compute_trace_hash
|
||||
|
||||
def timed_classify_intent(*args: Any, **kwargs: Any) -> Any:
|
||||
with _stage(sink, _STAGE_INTENT):
|
||||
return orig_classify_intent(*args, **kwargs)
|
||||
|
||||
def timed_graph_from_intent(*args: Any, **kwargs: Any) -> Any:
|
||||
with _stage(sink, _STAGE_GRAPH):
|
||||
return orig_graph_from_intent(*args, **kwargs)
|
||||
|
||||
def timed_plan_articulation(*args: Any, **kwargs: Any) -> Any:
|
||||
with _stage(sink, _STAGE_GRAPH):
|
||||
return orig_plan_articulation(*args, **kwargs)
|
||||
|
||||
def timed_realize_semantic(*args: Any, **kwargs: Any) -> Any:
|
||||
with _stage(sink, _STAGE_REALIZE):
|
||||
return orig_realize_semantic(*args, **kwargs)
|
||||
|
||||
def timed_compute_trace_hash(*args: Any, **kwargs: Any) -> Any:
|
||||
with _stage(sink, _STAGE_TRACE):
|
||||
return orig_compute_trace_hash(*args, **kwargs)
|
||||
|
||||
def timed_chat(*args: Any, **kwargs: Any) -> Any:
|
||||
with _stage(sink, _STAGE_RUNTIME_CHAT):
|
||||
return orig_chat(*args, **kwargs)
|
||||
|
||||
def timed_maybe_walk(*args: Any, **kwargs: Any) -> Any:
|
||||
with _stage(sink, _STAGE_TRANSITIVE_WALK):
|
||||
return orig_maybe_walk(*args, **kwargs)
|
||||
|
||||
def timed_fold(*args: Any, **kwargs: Any) -> Any:
|
||||
with _stage(sink, _STAGE_FOLD_WALK):
|
||||
return orig_fold(*args, **kwargs)
|
||||
|
||||
def timed_run_teaching(*args: Any, **kwargs: Any) -> Any:
|
||||
with _stage(sink, _STAGE_TEACHING):
|
||||
return orig_run_teaching(*args, **kwargs)
|
||||
|
||||
pipeline_mod.classify_intent = timed_classify_intent
|
||||
pipeline_mod.graph_from_intent = timed_graph_from_intent
|
||||
pipeline_mod.plan_articulation = timed_plan_articulation
|
||||
pipeline_mod.realize_semantic = timed_realize_semantic
|
||||
pipeline_mod.compute_trace_hash = timed_compute_trace_hash
|
||||
runtime.chat = timed_chat # type: ignore[assignment]
|
||||
pipeline._maybe_transitive_walk = timed_maybe_walk # type: ignore[assignment]
|
||||
pipeline._fold_walk_into_surface = timed_fold # type: ignore[assignment]
|
||||
pipeline._run_teaching = timed_run_teaching # type: ignore[assignment]
|
||||
|
||||
t_total_0 = time.perf_counter_ns()
|
||||
try:
|
||||
result = pipeline.run(text, max_tokens=max_tokens)
|
||||
finally:
|
||||
total_ns = time.perf_counter_ns() - t_total_0
|
||||
# Restore originals (instance and module).
|
||||
pipeline_mod.classify_intent = orig_classify_intent
|
||||
pipeline_mod.graph_from_intent = orig_graph_from_intent
|
||||
pipeline_mod.plan_articulation = orig_plan_articulation
|
||||
pipeline_mod.realize_semantic = orig_realize_semantic
|
||||
pipeline_mod.compute_trace_hash = orig_compute_trace_hash
|
||||
runtime.chat = orig_chat # type: ignore[assignment]
|
||||
try:
|
||||
del pipeline._maybe_transitive_walk # restore class-bound method
|
||||
except AttributeError:
|
||||
pipeline._maybe_transitive_walk = orig_maybe_walk # type: ignore[assignment]
|
||||
try:
|
||||
del pipeline._fold_walk_into_surface
|
||||
except AttributeError:
|
||||
pipeline._fold_walk_into_surface = orig_fold # type: ignore[assignment]
|
||||
try:
|
||||
del pipeline._run_teaching
|
||||
except AttributeError:
|
||||
pipeline._run_teaching = orig_run_teaching # type: ignore[assignment]
|
||||
|
||||
return ProfileReport(stages=dict(sink.stages), total_ns=total_ns, result=result)
|
||||
266
benchmarks/word_selection_tracer.py
Normal file
266
benchmarks/word_selection_tracer.py
Normal file
|
|
@ -0,0 +1,266 @@
|
|||
"""Word-selection tracer for the articulation/realization path.
|
||||
|
||||
Captures every nearest-neighbor vocabulary lookup performed during a turn:
|
||||
- slot name (subject / predicate / object)
|
||||
- input versor (32-d float vector, copied)
|
||||
- top-K candidate words by CGA inner product score
|
||||
- chosen word
|
||||
- any morphology applied
|
||||
|
||||
Also records each realization step (subject, predicate, object, tense,
|
||||
aspect, plural, negation) emitted by ``realize_semantic`` / ``realize_target``.
|
||||
|
||||
External instrumentation only — instruments via module-level function
|
||||
swaps that are reverted in ``finally``. No edits to generate/, vocab/,
|
||||
or algebra/ source files.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
from algebra.backend import _CGA_INNER_METRIC # diagonal Cl(4,1) metric (±1 per blade)
|
||||
from chat.runtime import ChatRuntime
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WordSelectionStep:
|
||||
"""A single nearest-neighbor lookup observed during articulation."""
|
||||
|
||||
slot: str # 'subject' | 'predicate' | 'object'
|
||||
input_versor: np.ndarray # shape (32,), copy — safe to retain
|
||||
top_candidates: tuple[tuple[str, float], ...] # (word, cga_inner_score)
|
||||
chosen: str
|
||||
morphology: dict[str, Any] # tense/aspect/plural/negation/lemma/surface, if any
|
||||
output_language: str
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"slot": self.slot,
|
||||
"top_candidates": [list(c) for c in self.top_candidates],
|
||||
"chosen": self.chosen,
|
||||
"morphology": dict(self.morphology),
|
||||
"output_language": self.output_language,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RealizationStep:
|
||||
"""A semantic realization step (subject/predicate/object + morphology)."""
|
||||
|
||||
subject: str
|
||||
predicate: str
|
||||
obj: str | None
|
||||
tense: str | None
|
||||
aspect: str | None
|
||||
negated: bool
|
||||
quantifier: str | None
|
||||
move: str
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"subject": self.subject,
|
||||
"predicate": self.predicate,
|
||||
"obj": self.obj,
|
||||
"tense": self.tense,
|
||||
"aspect": self.aspect,
|
||||
"negated": self.negated,
|
||||
"quantifier": self.quantifier,
|
||||
"move": self.move,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class RealizationTrace:
|
||||
"""Full trace from one turn: word selections + realization steps."""
|
||||
|
||||
steps: list[WordSelectionStep] = field(default_factory=list)
|
||||
realization_steps: list[RealizationStep] = field(default_factory=list)
|
||||
surface: str = ""
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"steps": [s.as_dict() for s in self.steps],
|
||||
"realization_steps": [r.as_dict() for r in self.realization_steps],
|
||||
"surface": self.surface,
|
||||
}
|
||||
|
||||
|
||||
def _morphology_summary(vocab: Any, word: str) -> dict[str, Any]:
|
||||
"""Extract morphology fields for a word, returning an empty dict if none."""
|
||||
entry = vocab.morphology_for_word(word)
|
||||
if entry is None:
|
||||
return {}
|
||||
summary: dict[str, Any] = {}
|
||||
# MorphologyEntry fields vary; collect any present attributes.
|
||||
for attr in ("lemma", "surface", "tense", "aspect", "plural", "number", "negation", "person", "gender", "pos"):
|
||||
value = getattr(entry, attr, None)
|
||||
if value is not None:
|
||||
summary[attr] = value
|
||||
return summary
|
||||
|
||||
|
||||
def _topk_candidates(
|
||||
vocab: Any,
|
||||
versor: np.ndarray,
|
||||
candidate_indices: np.ndarray,
|
||||
k: int = 5,
|
||||
) -> tuple[tuple[str, float], ...]:
|
||||
"""Compute top-K candidates by CGA inner product over the candidate set.
|
||||
|
||||
Vectorised via the diagonal Cl(4,1) metric — same kernel as
|
||||
``algebra.backend.vault_recall``. Exact, deterministic, no approximation.
|
||||
Used only for tracing; never fed back into the realizer's surface.
|
||||
"""
|
||||
if len(candidate_indices) == 0:
|
||||
return ()
|
||||
idx = np.asarray(candidate_indices, dtype=np.int64)
|
||||
# Stack candidate versors into one (N, 32) matrix; the vocab stores
|
||||
# them as a list of 32-vectors.
|
||||
versors_list = [vocab._versors[int(i)] for i in idx]
|
||||
M = np.asarray(versors_list, dtype=np.float32)
|
||||
q = np.asarray(versor, dtype=np.float32).reshape(-1)
|
||||
# Diagonal weighted dot-product, vectorised serial fold (same
|
||||
# component order as scalar cga_inner so scores are bit-identical
|
||||
# to the per-versor scan we replaced).
|
||||
scores = np.zeros(M.shape[0], dtype=np.float32)
|
||||
for c in range(M.shape[1]):
|
||||
scores += (_CGA_INNER_METRIC[c] * M[:, c]) * q[c]
|
||||
k_eff = max(1, min(int(k), scores.shape[0]))
|
||||
if k_eff < scores.shape[0]:
|
||||
cand = np.argpartition(-scores, k_eff - 1)[:k_eff]
|
||||
else:
|
||||
cand = np.arange(scores.shape[0])
|
||||
order = np.lexsort((cand, -scores[cand]))
|
||||
cand = cand[order]
|
||||
return tuple(
|
||||
(vocab._words[int(idx[int(c)])], float(scores[int(c)]))
|
||||
for c in cand
|
||||
)
|
||||
|
||||
|
||||
def trace_realization(
|
||||
runtime_or_pipeline: Any,
|
||||
text: str,
|
||||
*,
|
||||
top_k: int = 5,
|
||||
max_tokens: int | None = None,
|
||||
) -> RealizationTrace:
|
||||
"""Run one chat turn (or pipeline turn) while tracing every word lookup.
|
||||
|
||||
Accepts either a ``ChatRuntime`` (calls ``.chat``) or a
|
||||
``CognitiveTurnPipeline`` (calls ``.run``). A pipeline is preferred
|
||||
because the pipeline path invokes ``realize_semantic`` even when the
|
||||
runtime's unknown-domain gate fires, so realization steps are captured
|
||||
regardless of grounding.
|
||||
|
||||
Instruments ``generate.articulation._resolve_slot`` and
|
||||
``generate.realizer.realize_semantic`` for the duration of this call,
|
||||
then restores them. Does NOT modify the realizer/articulation source.
|
||||
"""
|
||||
trace = RealizationTrace()
|
||||
|
||||
from generate import articulation as articulation_mod
|
||||
from generate import realizer as realizer_mod
|
||||
|
||||
orig_resolve_slot = articulation_mod._resolve_slot
|
||||
orig_candidate_indices = articulation_mod._candidate_indices
|
||||
orig_surface_for_word = articulation_mod._surface_for_word
|
||||
orig_realize_semantic = realizer_mod.realize_semantic
|
||||
orig_resolve_obj = realizer_mod._resolve_obj
|
||||
|
||||
# Track slot order within a single realize() call. Reset on every
|
||||
# articulation.realize() entry; resolve_slot has no slot label itself,
|
||||
# so we synthesize it from invocation order: subject, predicate, object.
|
||||
slot_state: dict[str, int] = {"counter": 0}
|
||||
_SLOT_ORDER = ("subject", "predicate", "object")
|
||||
|
||||
def traced_resolve_slot(
|
||||
versor: np.ndarray | None,
|
||||
vocab: Any,
|
||||
output_language: str,
|
||||
) -> str | None:
|
||||
slot_idx = slot_state["counter"]
|
||||
slot_state["counter"] = slot_idx + 1
|
||||
slot_name = _SLOT_ORDER[slot_idx] if slot_idx < len(_SLOT_ORDER) else f"slot_{slot_idx}"
|
||||
if versor is None:
|
||||
return None
|
||||
cand = orig_candidate_indices(vocab, output_language)
|
||||
chosen_word, _chosen_idx = vocab.nearest(versor, candidate_indices=cand)
|
||||
top = _topk_candidates(vocab, versor, cand, k=top_k)
|
||||
morph = _morphology_summary(vocab, chosen_word)
|
||||
trace.steps.append(
|
||||
WordSelectionStep(
|
||||
slot=slot_name,
|
||||
input_versor=np.asarray(versor, dtype=float).copy(),
|
||||
top_candidates=top,
|
||||
chosen=chosen_word,
|
||||
morphology=morph,
|
||||
output_language=output_language,
|
||||
)
|
||||
)
|
||||
return orig_surface_for_word(vocab, chosen_word)
|
||||
|
||||
# Reset slot counter at each realize() entry. Patch articulation.realize
|
||||
# via a wrapper that resets the slot_state counter before delegating.
|
||||
orig_realize = articulation_mod.realize
|
||||
|
||||
def traced_realize(*args: Any, **kwargs: Any) -> Any:
|
||||
slot_state["counter"] = 0
|
||||
return orig_realize(*args, **kwargs)
|
||||
|
||||
def traced_realize_semantic(target: Any, graph: Any = None) -> Any:
|
||||
plan = orig_realize_semantic(target, graph)
|
||||
# Record the realization steps directly from the target/graph
|
||||
# without re-running the realizer.
|
||||
if target is not None and target.steps:
|
||||
for step in target.steps:
|
||||
obj = orig_resolve_obj(step, graph) if graph is not None else None
|
||||
trace.realization_steps.append(
|
||||
RealizationStep(
|
||||
subject=step.subject,
|
||||
predicate=step.predicate,
|
||||
obj=obj,
|
||||
tense=step.tense,
|
||||
aspect=step.aspect,
|
||||
negated=step.negated,
|
||||
quantifier=step.quantifier,
|
||||
move=step.move.value,
|
||||
)
|
||||
)
|
||||
return plan
|
||||
|
||||
articulation_mod._resolve_slot = traced_resolve_slot
|
||||
articulation_mod.realize = traced_realize
|
||||
realizer_mod.realize_semantic = traced_realize_semantic
|
||||
|
||||
# Also patch the symbol referenced by the pipeline module, since it
|
||||
# was imported by name at module load time.
|
||||
try:
|
||||
from core.cognition import pipeline as pipeline_mod
|
||||
orig_pipeline_realize_semantic = pipeline_mod.realize_semantic
|
||||
pipeline_mod.realize_semantic = traced_realize_semantic
|
||||
except ImportError:
|
||||
pipeline_mod = None
|
||||
orig_pipeline_realize_semantic = None
|
||||
|
||||
try:
|
||||
if hasattr(runtime_or_pipeline, "run") and hasattr(runtime_or_pipeline, "runtime"):
|
||||
# CognitiveTurnPipeline
|
||||
result = runtime_or_pipeline.run(text, max_tokens=max_tokens)
|
||||
trace.surface = result.articulation_surface or result.surface or ""
|
||||
else:
|
||||
# ChatRuntime
|
||||
response = runtime_or_pipeline.chat(text, max_tokens=max_tokens)
|
||||
trace.surface = response.articulation_surface or response.surface or ""
|
||||
finally:
|
||||
articulation_mod._resolve_slot = orig_resolve_slot
|
||||
articulation_mod.realize = orig_realize
|
||||
realizer_mod.realize_semantic = orig_realize_semantic
|
||||
if pipeline_mod is not None and orig_pipeline_realize_semantic is not None:
|
||||
pipeline_mod.realize_semantic = orig_pipeline_realize_semantic
|
||||
|
||||
return trace
|
||||
|
|
@ -57,6 +57,7 @@ _TEST_SUITES: dict[str, tuple[str, ...]] = {
|
|||
"tests/test_deterministic_hash.py",
|
||||
"tests/test_morphology_irregular.py",
|
||||
"tests/test_realizer_quantifier_agreement.py",
|
||||
"tests/test_benchmarks_profiler.py",
|
||||
),
|
||||
"teaching": (
|
||||
"tests/test_reviewed_teaching_loop.py",
|
||||
|
|
|
|||
0
evals/discourse_paragraph/__init__.py
Normal file
0
evals/discourse_paragraph/__init__.py
Normal file
64
evals/discourse_paragraph/contract.md
Normal file
64
evals/discourse_paragraph/contract.md
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
# discourse_paragraph eval lane
|
||||
|
||||
## What it measures
|
||||
|
||||
Whether the deterministic realizer can produce **paragraph-scale**
|
||||
output — multiple grammatical sentences joined by deterministic
|
||||
discourse markers — from a multi-step ArticulationTarget.
|
||||
|
||||
This is the first lane that stresses output longer than a single
|
||||
3-word SVO sentence. It addresses the open scope item:
|
||||
*"longer/more complex sentences and phrases for testing and proving
|
||||
stuff"*.
|
||||
|
||||
## Inputs
|
||||
|
||||
Each case carries a `graph` (≥ 3 nodes), an ordered `steps` list
|
||||
(`ASSERT` open, then `SEQUENCE` / `ELABORATE` / `CONTRAST`), and
|
||||
acceptance constraints:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "DP-PUB_001",
|
||||
"topic": "epistemic_chain",
|
||||
"graph": {"nodes": [{"node_id": "n1", "subject": "wisdom",
|
||||
"predicate": "grounds", "obj": "knowledge"}, ...],
|
||||
"edges": []},
|
||||
"steps": [{"node_id": "n1", "move": "ASSERT"}, ...],
|
||||
"min_sentences": 4,
|
||||
"max_sentences": 6,
|
||||
"must_contain_subjects": ["wisdom", "knowledge", "evidence", "truth"],
|
||||
"discourse_markers": ["furthermore", "next"]
|
||||
}
|
||||
```
|
||||
|
||||
## Scoring rubric
|
||||
|
||||
Per case:
|
||||
|
||||
- `paragraph_sentence_count` ≥ `min_sentences` (and ≤ `max_sentences`)
|
||||
- `subject_coverage_rate` ≥ 0.75
|
||||
- `discourse_marker_present` — at least one expected marker emitted
|
||||
- `replay_determinism` — running the case twice produces an
|
||||
identical surface string
|
||||
|
||||
Aggregate metrics:
|
||||
|
||||
- `accuracy` — pass rate
|
||||
- `mean_sentence_count`
|
||||
- `mean_subject_coverage`
|
||||
- `replay_determinism_rate`
|
||||
|
||||
## Splits
|
||||
|
||||
| Split | n | content |
|
||||
|---|---|---|
|
||||
| public/v1 | 12 | epistemic / scientific / creation / logic / ethics / linguistic / math / narrative / biology / physics + 2 contrast cases |
|
||||
| holdouts/v1 | 5 | musical / social / computational / psychological / economic |
|
||||
| dev | 1 | epistemic_chain smoke |
|
||||
|
||||
## What this lane does NOT measure
|
||||
|
||||
- Round-trip through `ChatRuntime` (the realizer is exercised
|
||||
directly). See gaps.md.
|
||||
- Factual correctness of the asserted propositions.
|
||||
1
evals/discourse_paragraph/dev/cases.jsonl
Normal file
1
evals/discourse_paragraph/dev/cases.jsonl
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"id": "DP-DEV_001", "topic": "epistemic_chain", "graph": {"nodes": [{"node_id": "n1", "subject": "wisdom", "predicate": "grounds", "obj": "knowledge"}, {"node_id": "n2", "subject": "knowledge", "predicate": "requires", "obj": "evidence"}, {"node_id": "n3", "subject": "evidence", "predicate": "supports", "obj": "truth"}, {"node_id": "n4", "subject": "truth", "predicate": "reveals", "obj": "reality"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "SEQUENCE"}, {"node_id": "n3", "move": "ELABORATE"}, {"node_id": "n4", "move": "SEQUENCE"}], "min_sentences": 4, "must_contain_subjects": ["wisdom", "knowledge", "evidence", "truth"], "discourse_markers": ["furthermore", "next"], "max_sentences": 6}
|
||||
36
evals/discourse_paragraph/gaps.md
Normal file
36
evals/discourse_paragraph/gaps.md
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
# discourse_paragraph — gaps
|
||||
|
||||
## v1 (current)
|
||||
|
||||
- Realizer-isolation lane: bypasses runtime grounding so the
|
||||
paragraph claim is unconfounded by vault noise.
|
||||
- Sentence-count window is intentionally generous
|
||||
(`max_sentences = min + 2`) to tolerate small wrapping variance
|
||||
from compound-clause folding in `realize_target` (CONJUNCTION /
|
||||
COMPLEMENT / RELATIVE edges merge two steps into one sentence).
|
||||
- Subject coverage threshold is 0.75, not 1.0 — exact-coverage
|
||||
cases pass that bar comfortably but the slack lets a future
|
||||
realizer change ship without rewriting cases.
|
||||
|
||||
## Known gaps for v2
|
||||
|
||||
1. **No round-trip through the runtime.** v1 invokes the realizer
|
||||
directly with a constructed `ArticulationTarget`. v2 should
|
||||
feed the runtime real text inputs that *produce* the same
|
||||
articulation target through `graph_from_intent` +
|
||||
`plan_articulation`, end-to-end.
|
||||
2. **No anaphora / pronoun reduction.** Every sentence carries
|
||||
its subject explicitly. Pronominalisation deferred.
|
||||
3. **No length scaling above 5 sentences.** v2 should push to
|
||||
10/20/50 sentences and measure per-sentence determinism.
|
||||
4. **No grammaticality check per sentence.** v1 checks subject
|
||||
coverage + discourse markers; v2 should run each emitted
|
||||
sentence through grammatical_coverage's rubric.
|
||||
|
||||
## Why this lane exists
|
||||
|
||||
First lane that exercises paragraph-scale output. Every previous
|
||||
fluency lane (Phase 5.1 + 5.4–5.7) operates on 3-word SVO probes.
|
||||
The structural capability — folding multiple articulation steps
|
||||
into a coherent paragraph with deterministic discourse markers —
|
||||
was already in the realizer; this lane makes it measurable.
|
||||
5
evals/discourse_paragraph/holdouts/v1/cases.jsonl
Normal file
5
evals/discourse_paragraph/holdouts/v1/cases.jsonl
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
{"id": "DP-HOLD_001", "topic": "musical_construction", "graph": {"nodes": [{"node_id": "n1", "subject": "note", "predicate": "composes", "obj": "chord"}, {"node_id": "n2", "subject": "chord", "predicate": "supports", "obj": "harmony"}, {"node_id": "n3", "subject": "harmony", "predicate": "yields", "obj": "phrase"}, {"node_id": "n4", "subject": "phrase", "predicate": "builds", "obj": "melody"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "SEQUENCE"}, {"node_id": "n3", "move": "ELABORATE"}, {"node_id": "n4", "move": "SEQUENCE"}], "min_sentences": 4, "must_contain_subjects": ["note", "chord", "harmony", "phrase"], "discourse_markers": ["furthermore", "next"], "max_sentences": 6}
|
||||
{"id": "DP-HOLD_002", "topic": "social_structure", "graph": {"nodes": [{"node_id": "n1", "subject": "custom", "predicate": "grounds", "obj": "tradition"}, {"node_id": "n2", "subject": "tradition", "predicate": "supports", "obj": "institution"}, {"node_id": "n3", "subject": "institution", "predicate": "shapes", "obj": "society"}, {"node_id": "n4", "subject": "society", "predicate": "reveals", "obj": "culture"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "SEQUENCE"}, {"node_id": "n3", "move": "ELABORATE"}, {"node_id": "n4", "move": "SEQUENCE"}], "min_sentences": 4, "must_contain_subjects": ["custom", "tradition", "institution", "society"], "discourse_markers": ["furthermore", "next"], "max_sentences": 6}
|
||||
{"id": "DP-HOLD_003", "topic": "computational_pipeline", "graph": {"nodes": [{"node_id": "n1", "subject": "input", "predicate": "drives", "obj": "computation"}, {"node_id": "n2", "subject": "computation", "predicate": "yields", "obj": "output"}, {"node_id": "n3", "subject": "output", "predicate": "supports", "obj": "decision"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "SEQUENCE"}, {"node_id": "n3", "move": "ELABORATE"}], "min_sentences": 3, "must_contain_subjects": ["input", "computation", "output"], "discourse_markers": ["furthermore", "next"], "max_sentences": 5}
|
||||
{"id": "DP-HOLD_004", "topic": "psychological_development", "graph": {"nodes": [{"node_id": "n1", "subject": "sensation", "predicate": "grounds", "obj": "perception"}, {"node_id": "n2", "subject": "perception", "predicate": "supports", "obj": "memory"}, {"node_id": "n3", "subject": "memory", "predicate": "yields", "obj": "learning"}, {"node_id": "n4", "subject": "learning", "predicate": "shapes", "obj": "behavior"}, {"node_id": "n5", "subject": "behavior", "predicate": "reveals", "obj": "character"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "SEQUENCE"}, {"node_id": "n3", "move": "ELABORATE"}, {"node_id": "n4", "move": "SEQUENCE"}, {"node_id": "n5", "move": "ELABORATE"}], "min_sentences": 5, "must_contain_subjects": ["sensation", "perception", "memory", "learning", "behavior"], "discourse_markers": ["furthermore", "next"], "max_sentences": 7}
|
||||
{"id": "DP-HOLD_005", "topic": "economic_flow", "graph": {"nodes": [{"node_id": "n1", "subject": "labor", "predicate": "yields", "obj": "value"}, {"node_id": "n2", "subject": "value", "predicate": "supports", "obj": "exchange"}, {"node_id": "n3", "subject": "exchange", "predicate": "drives", "obj": "growth"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "SEQUENCE"}, {"node_id": "n3", "move": "ELABORATE"}], "min_sentences": 3, "must_contain_subjects": ["labor", "value", "exchange"], "discourse_markers": ["furthermore", "next"], "max_sentences": 5}
|
||||
12
evals/discourse_paragraph/public/v1/cases.jsonl
Normal file
12
evals/discourse_paragraph/public/v1/cases.jsonl
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
{"id": "DP-PUB_001", "topic": "epistemic_chain", "graph": {"nodes": [{"node_id": "n1", "subject": "wisdom", "predicate": "grounds", "obj": "knowledge"}, {"node_id": "n2", "subject": "knowledge", "predicate": "requires", "obj": "evidence"}, {"node_id": "n3", "subject": "evidence", "predicate": "supports", "obj": "truth"}, {"node_id": "n4", "subject": "truth", "predicate": "reveals", "obj": "reality"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "SEQUENCE"}, {"node_id": "n3", "move": "ELABORATE"}, {"node_id": "n4", "move": "SEQUENCE"}], "min_sentences": 4, "must_contain_subjects": ["wisdom", "knowledge", "evidence", "truth"], "discourse_markers": ["furthermore", "next"], "max_sentences": 6}
|
||||
{"id": "DP-PUB_002", "topic": "scientific_method", "graph": {"nodes": [{"node_id": "n1", "subject": "observation", "predicate": "grounds", "obj": "hypothesis"}, {"node_id": "n2", "subject": "hypothesis", "predicate": "implies", "obj": "prediction"}, {"node_id": "n3", "subject": "prediction", "predicate": "follows", "obj": "experiment"}, {"node_id": "n4", "subject": "experiment", "predicate": "supports", "obj": "theory"}, {"node_id": "n5", "subject": "theory", "predicate": "entails", "obj": "explanation"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "ELABORATE"}, {"node_id": "n3", "move": "SEQUENCE"}, {"node_id": "n4", "move": "ELABORATE"}, {"node_id": "n5", "move": "SEQUENCE"}], "min_sentences": 5, "must_contain_subjects": ["observation", "hypothesis", "prediction", "experiment", "theory"], "discourse_markers": ["furthermore", "next"], "max_sentences": 7}
|
||||
{"id": "DP-PUB_003", "topic": "creation_arc", "graph": {"nodes": [{"node_id": "n1", "subject": "light", "predicate": "precedes", "obj": "form"}, {"node_id": "n2", "subject": "form", "predicate": "grounds", "obj": "matter"}, {"node_id": "n3", "subject": "matter", "predicate": "supports", "obj": "structure"}, {"node_id": "n4", "subject": "structure", "predicate": "reveals", "obj": "order"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "SEQUENCE"}, {"node_id": "n3", "move": "ELABORATE"}, {"node_id": "n4", "move": "SEQUENCE"}], "min_sentences": 4, "must_contain_subjects": ["light", "form", "matter", "structure"], "discourse_markers": ["furthermore", "next"], "max_sentences": 6}
|
||||
{"id": "DP-PUB_004", "topic": "logical_dependency", "graph": {"nodes": [{"node_id": "n1", "subject": "premise", "predicate": "supports", "obj": "conclusion"}, {"node_id": "n2", "subject": "conclusion", "predicate": "requires", "obj": "validity"}, {"node_id": "n3", "subject": "validity", "predicate": "entails", "obj": "soundness"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "SEQUENCE"}, {"node_id": "n3", "move": "ELABORATE"}], "min_sentences": 3, "must_contain_subjects": ["premise", "conclusion", "validity"], "discourse_markers": ["furthermore", "next"], "max_sentences": 5}
|
||||
{"id": "DP-PUB_005", "topic": "ethical_grounding", "graph": {"nodes": [{"node_id": "n1", "subject": "virtue", "predicate": "grounds", "obj": "action"}, {"node_id": "n2", "subject": "action", "predicate": "requires", "obj": "intention"}, {"node_id": "n3", "subject": "intention", "predicate": "supports", "obj": "consequence"}, {"node_id": "n4", "subject": "consequence", "predicate": "reveals", "obj": "character"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "SEQUENCE"}, {"node_id": "n3", "move": "ELABORATE"}, {"node_id": "n4", "move": "SEQUENCE"}], "min_sentences": 4, "must_contain_subjects": ["virtue", "action", "intention", "consequence"], "discourse_markers": ["furthermore", "next"], "max_sentences": 6}
|
||||
{"id": "DP-PUB_006", "topic": "linguistic_layers", "graph": {"nodes": [{"node_id": "n1", "subject": "sound", "predicate": "grounds", "obj": "phoneme"}, {"node_id": "n2", "subject": "phoneme", "predicate": "supports", "obj": "morpheme"}, {"node_id": "n3", "subject": "morpheme", "predicate": "builds", "obj": "word"}, {"node_id": "n4", "subject": "word", "predicate": "composes", "obj": "sentence"}, {"node_id": "n5", "subject": "sentence", "predicate": "conveys", "obj": "meaning"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "SEQUENCE"}, {"node_id": "n3", "move": "ELABORATE"}, {"node_id": "n4", "move": "SEQUENCE"}, {"node_id": "n5", "move": "ELABORATE"}], "min_sentences": 5, "must_contain_subjects": ["sound", "phoneme", "morpheme", "word", "sentence"], "discourse_markers": ["furthermore", "next"], "max_sentences": 7}
|
||||
{"id": "DP-PUB_007", "topic": "mathematical_chain", "graph": {"nodes": [{"node_id": "n1", "subject": "axiom", "predicate": "grounds", "obj": "theorem"}, {"node_id": "n2", "subject": "theorem", "predicate": "entails", "obj": "corollary"}, {"node_id": "n3", "subject": "corollary", "predicate": "supports", "obj": "application"}, {"node_id": "n4", "subject": "application", "predicate": "yields", "obj": "insight"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "ELABORATE"}, {"node_id": "n3", "move": "SEQUENCE"}, {"node_id": "n4", "move": "SEQUENCE"}], "min_sentences": 4, "must_contain_subjects": ["axiom", "theorem", "corollary", "application"], "discourse_markers": ["furthermore", "next"], "max_sentences": 6}
|
||||
{"id": "DP-PUB_008", "topic": "narrative_progression", "graph": {"nodes": [{"node_id": "n1", "subject": "conflict", "predicate": "drives", "obj": "tension"}, {"node_id": "n2", "subject": "tension", "predicate": "precedes", "obj": "climax"}, {"node_id": "n3", "subject": "climax", "predicate": "yields", "obj": "resolution"}, {"node_id": "n4", "subject": "resolution", "predicate": "reveals", "obj": "theme"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "SEQUENCE"}, {"node_id": "n3", "move": "ELABORATE"}, {"node_id": "n4", "move": "SEQUENCE"}], "min_sentences": 4, "must_contain_subjects": ["conflict", "tension", "climax", "resolution"], "discourse_markers": ["furthermore", "next"], "max_sentences": 6}
|
||||
{"id": "DP-PUB_009", "topic": "biological_hierarchy", "graph": {"nodes": [{"node_id": "n1", "subject": "gene", "predicate": "encodes", "obj": "protein"}, {"node_id": "n2", "subject": "protein", "predicate": "builds", "obj": "cell"}, {"node_id": "n3", "subject": "cell", "predicate": "composes", "obj": "tissue"}, {"node_id": "n4", "subject": "tissue", "predicate": "forms", "obj": "organ"}, {"node_id": "n5", "subject": "organ", "predicate": "supports", "obj": "organism"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "SEQUENCE"}, {"node_id": "n3", "move": "ELABORATE"}, {"node_id": "n4", "move": "SEQUENCE"}, {"node_id": "n5", "move": "ELABORATE"}], "min_sentences": 5, "must_contain_subjects": ["gene", "protein", "cell", "tissue", "organ"], "discourse_markers": ["furthermore", "next"], "max_sentences": 7}
|
||||
{"id": "DP-PUB_010", "topic": "physical_causation", "graph": {"nodes": [{"node_id": "n1", "subject": "force", "predicate": "drives", "obj": "motion"}, {"node_id": "n2", "subject": "motion", "predicate": "transfers", "obj": "energy"}, {"node_id": "n3", "subject": "energy", "predicate": "yields", "obj": "heat"}, {"node_id": "n4", "subject": "heat", "predicate": "raises", "obj": "temperature"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "ELABORATE"}, {"node_id": "n3", "move": "SEQUENCE"}, {"node_id": "n4", "move": "SEQUENCE"}], "min_sentences": 4, "must_contain_subjects": ["force", "motion", "energy", "heat"], "discourse_markers": ["furthermore", "next"], "max_sentences": 6}
|
||||
{"id": "DP-PUB_011", "topic": "contrastive_definitions", "graph": {"nodes": [{"node_id": "n1", "subject": "knowledge", "predicate": "requires", "obj": "evidence"}, {"node_id": "n2", "subject": "belief", "predicate": "requires", "obj": "trust"}, {"node_id": "n3", "subject": "wisdom", "predicate": "grounds", "obj": "judgment"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "CONTRAST"}, {"node_id": "n3", "move": "ELABORATE"}], "min_sentences": 3, "must_contain_subjects": ["knowledge", "belief", "wisdom"], "discourse_markers": ["furthermore", "in contrast"], "max_sentences": 5}
|
||||
{"id": "DP-PUB_012", "topic": "method_contrast", "graph": {"nodes": [{"node_id": "n1", "subject": "deduction", "predicate": "yields", "obj": "certainty"}, {"node_id": "n2", "subject": "induction", "predicate": "yields", "obj": "probability"}, {"node_id": "n3", "subject": "abduction", "predicate": "yields", "obj": "explanation"}], "edges": []}, "steps": [{"node_id": "n1", "move": "ASSERT"}, {"node_id": "n2", "move": "CONTRAST"}, {"node_id": "n3", "move": "ELABORATE"}], "min_sentences": 3, "must_contain_subjects": ["deduction", "induction", "abduction"], "discourse_markers": ["furthermore", "in contrast"], "max_sentences": 5}
|
||||
|
|
@ -0,0 +1,184 @@
|
|||
{
|
||||
"cases": [
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_001",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 4,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "wisdom grounds knowledge. next, knowledge requires evidence. furthermore, evidence supports truth. next, truth reveals reality.",
|
||||
"topic": "epistemic_chain"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_002",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 5,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "observation grounds hypothesis. furthermore, hypothesis implies prediction. next, prediction follows experiment. furthermore, experiment supports theory. next, theory entails explanation.",
|
||||
"topic": "scientific_method"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_003",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 4,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "light precedes form. next, form grounds matter. furthermore, matter supports structure. next, structure reveals order.",
|
||||
"topic": "creation_arc"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_004",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 3,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "premise supports conclusion. next, conclusion requires validity. furthermore, validity entails soundness.",
|
||||
"topic": "logical_dependency"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_005",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 4,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "virtue grounds action. next, action requires intention. furthermore, intention supports consequence. next, consequence reveals character.",
|
||||
"topic": "ethical_grounding"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_006",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 5,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "sound grounds phoneme. next, phoneme supports morpheme. furthermore, morpheme builds word. next, word composes sentence. furthermore, sentence conveys meaning.",
|
||||
"topic": "linguistic_layers"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_007",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 4,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "axiom grounds theorem. furthermore, theorem entails corollary. next, corollary supports application. next, application yields insight.",
|
||||
"topic": "mathematical_chain"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_008",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 4,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "conflict drives tension. next, tension precedes climax. furthermore, climax yields resolution. next, resolution reveals theme.",
|
||||
"topic": "narrative_progression"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_009",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 5,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "gene encodes protein. next, protein builds cell. furthermore, cell composes tissue. next, tissue forms organ. furthermore, organ supports organism.",
|
||||
"topic": "biological_hierarchy"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_010",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 4,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "force drives motion. furthermore, motion transfers energy. next, energy yields heat. next, heat raises temperature.",
|
||||
"topic": "physical_causation"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"in contrast"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_011",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 3,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "knowledge requires evidence. in contrast, belief requires trust. furthermore, wisdom grounds judgment.",
|
||||
"topic": "contrastive_definitions"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"in contrast"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_012",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 3,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "deduction yields certainty. in contrast, induction yields probability. furthermore, abduction yields explanation.",
|
||||
"topic": "method_contrast"
|
||||
}
|
||||
],
|
||||
"lane": "discourse_paragraph",
|
||||
"metrics": {
|
||||
"accuracy": 1.0,
|
||||
"mean_sentence_count": 4.0,
|
||||
"mean_subject_coverage": 1.0,
|
||||
"passed": 12,
|
||||
"replay_determinism_rate": 1.0,
|
||||
"total": 12
|
||||
},
|
||||
"split": "public",
|
||||
"timestamp": "2026-05-17T04:46:38.277091+00:00",
|
||||
"version": "v1"
|
||||
}
|
||||
|
|
@ -0,0 +1,184 @@
|
|||
{
|
||||
"cases": [
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_001",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 4,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "Wisdom grounds knowledge. Next, knowledge requires evidence. Furthermore, evidence supports truth. Next, truth reveals reality.",
|
||||
"topic": "epistemic_chain"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_002",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 5,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "Observation grounds hypothesis. Furthermore, hypothesis implies prediction. Next, prediction follows experiment. Furthermore, experiment supports theory. Next, theory entails explanation.",
|
||||
"topic": "scientific_method"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_003",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 4,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "Light precedes form. Next, form grounds matter. Furthermore, matter supports structure. Next, structure reveals order.",
|
||||
"topic": "creation_arc"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_004",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 3,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "Premise supports conclusion. Next, conclusion requires validity. Furthermore, validity entails soundness.",
|
||||
"topic": "logical_dependency"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_005",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 4,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "Virtue grounds action. Next, action requires intention. Furthermore, intention supports consequence. Next, consequence reveals character.",
|
||||
"topic": "ethical_grounding"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_006",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 5,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "Sound grounds phoneme. Next, phoneme supports morpheme. Furthermore, morpheme builds word. Next, word composes sentence. Furthermore, sentence conveys meaning.",
|
||||
"topic": "linguistic_layers"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_007",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 4,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "Axiom grounds theorem. Furthermore, theorem entails corollary. Next, corollary supports application. Next, application yields insight.",
|
||||
"topic": "mathematical_chain"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_008",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 4,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "Conflict drives tension. Next, tension precedes climax. Furthermore, climax yields resolution. Next, resolution reveals theme.",
|
||||
"topic": "narrative_progression"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_009",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 5,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "Gene encodes protein. Next, protein builds cell. Furthermore, cell composes tissue. Next, tissue forms organ. Furthermore, organ supports organism.",
|
||||
"topic": "biological_hierarchy"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"next"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_010",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 4,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "Force drives motion. Furthermore, motion transfers energy. Next, energy yields heat. Next, heat raises temperature.",
|
||||
"topic": "physical_causation"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"in contrast"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_011",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 3,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "Knowledge requires evidence. In contrast, belief requires trust. Furthermore, wisdom grounds judgment.",
|
||||
"topic": "contrastive_definitions"
|
||||
},
|
||||
{
|
||||
"discourse_markers_found": [
|
||||
"furthermore",
|
||||
"in contrast"
|
||||
],
|
||||
"failure_reasons": [],
|
||||
"id": "DP-PUB_012",
|
||||
"passed": true,
|
||||
"replay_match": true,
|
||||
"sentence_count": 3,
|
||||
"subject_coverage": 1.0,
|
||||
"surface": "Deduction yields certainty. In contrast, induction yields probability. Furthermore, abduction yields explanation.",
|
||||
"topic": "method_contrast"
|
||||
}
|
||||
],
|
||||
"lane": "discourse_paragraph",
|
||||
"metrics": {
|
||||
"accuracy": 1.0,
|
||||
"mean_sentence_count": 4.0,
|
||||
"mean_subject_coverage": 1.0,
|
||||
"passed": 12,
|
||||
"replay_determinism_rate": 1.0,
|
||||
"total": 12
|
||||
},
|
||||
"split": "public",
|
||||
"timestamp": "2026-05-17T04:47:09.206712+00:00",
|
||||
"version": "v1"
|
||||
}
|
||||
174
evals/discourse_paragraph/runner.py
Normal file
174
evals/discourse_paragraph/runner.py
Normal file
|
|
@ -0,0 +1,174 @@
|
|||
"""discourse_paragraph eval lane runner.
|
||||
|
||||
Exercises paragraph-scale realization: given a multi-step
|
||||
ArticulationTarget, the deterministic realizer should produce a
|
||||
multi-sentence surface with discourse markers (next, furthermore,
|
||||
in contrast) and full subject coverage.
|
||||
|
||||
Bypasses ChatRuntime grounding so the paragraph claim is isolated
|
||||
to the realizer. Runtime round-tripping is named as a v2 gap.
|
||||
|
||||
Conforms to the framework interface: run_lane(cases, config=None) -> report.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from generate.graph_planner import (
|
||||
ArticulationStep,
|
||||
ArticulationTarget,
|
||||
GraphEdge,
|
||||
GraphNode,
|
||||
PropositionGraph,
|
||||
Relation,
|
||||
RhetoricalMove,
|
||||
)
|
||||
from generate.intent import IntentTag
|
||||
from generate.realizer import realize_target
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class LaneReport:
|
||||
metrics: dict[str, Any] = field(default_factory=dict)
|
||||
case_details: list[dict[str, Any]] = field(default_factory=list)
|
||||
|
||||
|
||||
_SENTENCE_SPLIT_RE = re.compile(r"[.!?]\s+|[.!?]$")
|
||||
|
||||
|
||||
def _sentence_count(surface: str) -> int:
|
||||
if not surface.strip():
|
||||
return 0
|
||||
parts = [p for p in _SENTENCE_SPLIT_RE.split(surface) if p.strip()]
|
||||
return len(parts)
|
||||
|
||||
|
||||
def _build_target_from_case(case: dict[str, Any]) -> tuple[ArticulationTarget, PropositionGraph]:
|
||||
nodes_data = case["graph"]["nodes"]
|
||||
edges_data = case["graph"].get("edges", [])
|
||||
nodes = tuple(
|
||||
GraphNode(
|
||||
node_id=nd["node_id"],
|
||||
subject=nd["subject"],
|
||||
predicate=nd["predicate"],
|
||||
obj=nd["obj"],
|
||||
source_intent=IntentTag.UNKNOWN,
|
||||
)
|
||||
for nd in nodes_data
|
||||
)
|
||||
edges = tuple(
|
||||
GraphEdge(
|
||||
source=e["source"],
|
||||
target=e["target"],
|
||||
relation=Relation[e.get("relation", "SEQUENCE").upper()],
|
||||
)
|
||||
for e in edges_data
|
||||
)
|
||||
graph = PropositionGraph(nodes=nodes, edges=edges)
|
||||
by_id = {n.node_id: n for n in nodes}
|
||||
steps = tuple(
|
||||
ArticulationStep(
|
||||
node_id=s["node_id"],
|
||||
subject=by_id[s["node_id"]].subject,
|
||||
predicate=by_id[s["node_id"]].predicate,
|
||||
move=RhetoricalMove[s["move"].upper()],
|
||||
)
|
||||
for s in case["steps"]
|
||||
)
|
||||
target = ArticulationTarget(steps=steps, source_intent=IntentTag.UNKNOWN)
|
||||
return target, graph
|
||||
|
||||
|
||||
def _score_case(case: dict[str, Any]) -> dict[str, Any]:
|
||||
target, graph = _build_target_from_case(case)
|
||||
plan_1 = realize_target(target, graph)
|
||||
plan_2 = realize_target(target, graph)
|
||||
surface = plan_1.surface
|
||||
surface_lower = surface.lower()
|
||||
|
||||
failures: list[str] = []
|
||||
sent_count = _sentence_count(surface)
|
||||
min_sentences = int(case["min_sentences"])
|
||||
max_sentences = int(case.get("max_sentences", min_sentences + 2))
|
||||
if sent_count < min_sentences:
|
||||
failures.append(f"sentence_count {sent_count} < min {min_sentences}")
|
||||
if sent_count > max_sentences:
|
||||
failures.append(f"sentence_count {sent_count} > max {max_sentences}")
|
||||
|
||||
must_contain = case.get("must_contain_subjects", [])
|
||||
present = [s for s in must_contain if s.lower() in surface_lower]
|
||||
coverage = len(present) / max(1, len(must_contain))
|
||||
if coverage < 0.75:
|
||||
missing = [s for s in must_contain if s.lower() not in surface_lower]
|
||||
failures.append(f"subject_coverage {coverage:.2f} < 0.75; missing={missing}")
|
||||
|
||||
expected_markers = case.get("discourse_markers", [])
|
||||
if expected_markers:
|
||||
found = [m for m in expected_markers if m.lower() in surface_lower]
|
||||
if not found:
|
||||
failures.append(
|
||||
f"no discourse marker present; expected one of {expected_markers}"
|
||||
)
|
||||
else:
|
||||
found = []
|
||||
|
||||
# Sentence-initial capitalization (G4): every sentence-leading
|
||||
# alphabetic character must be uppercase. This is the gate that
|
||||
# turned "wisdom grounds knowledge." into "Wisdom grounds
|
||||
# knowledge." — addresses the open scope item.
|
||||
sentences = [p.strip() for p in _SENTENCE_SPLIT_RE.split(surface) if p.strip()]
|
||||
badly_cased: list[str] = []
|
||||
for sent in sentences:
|
||||
for ch in sent:
|
||||
if ch.isalpha():
|
||||
if not ch.isupper():
|
||||
badly_cased.append(sent[:30])
|
||||
break
|
||||
if badly_cased:
|
||||
failures.append(
|
||||
f"sentence-initial capitalization missing in {len(badly_cased)} "
|
||||
f"sentence(s): {badly_cased}"
|
||||
)
|
||||
|
||||
replay_match = plan_1.surface == plan_2.surface
|
||||
if not replay_match:
|
||||
failures.append("replay determinism broken: surfaces differ")
|
||||
|
||||
passed = not failures
|
||||
return {
|
||||
"id": case["id"],
|
||||
"topic": case.get("topic", ""),
|
||||
"passed": passed,
|
||||
"surface": surface,
|
||||
"sentence_count": sent_count,
|
||||
"subject_coverage": coverage,
|
||||
"discourse_markers_found": found,
|
||||
"replay_match": replay_match,
|
||||
"failure_reasons": failures,
|
||||
}
|
||||
|
||||
|
||||
def run_lane(cases: list[dict[str, Any]], *, config: Any = None) -> LaneReport:
|
||||
details = [_score_case(c) for c in cases]
|
||||
total = len(details)
|
||||
passed = sum(1 for d in details if d["passed"])
|
||||
return LaneReport(
|
||||
metrics={
|
||||
"total": total,
|
||||
"passed": passed,
|
||||
"accuracy": round(passed / total, 4) if total else 0.0,
|
||||
"mean_sentence_count": round(
|
||||
sum(d["sentence_count"] for d in details) / max(1, total), 3
|
||||
),
|
||||
"mean_subject_coverage": round(
|
||||
sum(d["subject_coverage"] for d in details) / max(1, total), 4
|
||||
),
|
||||
"replay_determinism_rate": round(
|
||||
sum(1 for d in details if d["replay_match"]) / max(1, total), 4
|
||||
),
|
||||
},
|
||||
case_details=details,
|
||||
)
|
||||
|
|
@ -40,6 +40,44 @@ class RealizedFragment:
|
|||
}
|
||||
|
||||
|
||||
def _capitalize_sentence(s: str) -> str:
|
||||
"""Capitalize the first alphabetic character of a sentence.
|
||||
|
||||
Skips leading whitespace/punctuation so fragments that start with
|
||||
discourse markers ("next, knowledge…") still emit a capital first
|
||||
letter ("Next, knowledge…") at the sentence boundary. Leaves the
|
||||
rest of the string untouched — proper nouns and embedded all-caps
|
||||
tokens are preserved.
|
||||
"""
|
||||
if not s:
|
||||
return s
|
||||
for i, ch in enumerate(s):
|
||||
if ch.isalpha():
|
||||
return s[:i] + ch.upper() + s[i + 1:]
|
||||
return s
|
||||
|
||||
|
||||
def _join_as_paragraph(fragments: list["RealizedFragment"]) -> str:
|
||||
"""Join fragments into a paragraph with sentence-initial capitalization.
|
||||
|
||||
Each fragment becomes one sentence; sentence-initial letters are
|
||||
capitalized; the paragraph ends with a single terminal period.
|
||||
"""
|
||||
if not fragments:
|
||||
return ""
|
||||
pieces: list[str] = []
|
||||
for f in fragments:
|
||||
s = f.surface.strip()
|
||||
if not s:
|
||||
continue
|
||||
s = _capitalize_sentence(s)
|
||||
pieces.append(s)
|
||||
joined = ". ".join(pieces)
|
||||
if joined and not joined.endswith("."):
|
||||
joined += "."
|
||||
return joined
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RealizedPlan:
|
||||
fragments: tuple[RealizedFragment, ...]
|
||||
|
|
@ -106,10 +144,7 @@ def realize_semantic(
|
|||
surface=surface,
|
||||
))
|
||||
|
||||
joined = ". ".join(f.surface for f in fragments)
|
||||
if joined and not joined.endswith("."):
|
||||
joined += "."
|
||||
|
||||
joined = _join_as_paragraph(fragments)
|
||||
return RealizedPlan(fragments=tuple(fragments), surface=joined)
|
||||
|
||||
|
||||
|
|
@ -208,10 +243,7 @@ def realize_target(
|
|||
)
|
||||
)
|
||||
|
||||
joined = ". ".join(f.surface for f in fragments)
|
||||
if joined and not joined.endswith("."):
|
||||
joined += "."
|
||||
|
||||
joined = _join_as_paragraph(fragments)
|
||||
return RealizedPlan(fragments=tuple(fragments), surface=joined)
|
||||
|
||||
|
||||
|
|
|
|||
273
scripts/generate_discourse_paragraph.py
Normal file
273
scripts/generate_discourse_paragraph.py
Normal file
|
|
@ -0,0 +1,273 @@
|
|||
"""Generate cases for the discourse_paragraph benchmark lane.
|
||||
|
||||
Tests that the realizer can produce **multi-sentence paragraph-scale
|
||||
output** from chained propositions, given a multi-step
|
||||
ArticulationTarget with rhetorical moves (SEQUENCE, ELABORATE,
|
||||
CONTRAST). Each case stresses paragraph length, subject coverage,
|
||||
discourse-marker presence, and deterministic replay.
|
||||
|
||||
Each case carries:
|
||||
- a graph of N ≥ 3 nodes (subject-predicate-object triples)
|
||||
- an ordered move list ([ASSERT, SEQUENCE, ELABORATE, ...])
|
||||
- acceptance constraints (min_sentences, must_contain_subjects,
|
||||
discourse_markers)
|
||||
|
||||
Topics are designed to be **structurally rich** — every case is more
|
||||
than a 3-word SVO probe.
|
||||
|
||||
Run:
|
||||
.venv/bin/python scripts/generate_discourse_paragraph.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
# Each topic: ordered triples + ordered rhetorical moves matching length.
|
||||
# Moves: ASSERT (open), SEQUENCE (next step), ELABORATE (furthermore),
|
||||
# CONTRAST (in contrast), CORRECT (correction). See
|
||||
# generate.templates._MOVE_TEMPLATES for emitted discourse markers.
|
||||
PUBLIC_TOPICS: list[dict] = [
|
||||
{
|
||||
"topic": "epistemic_chain",
|
||||
"triples": [
|
||||
("wisdom", "grounds", "knowledge"),
|
||||
("knowledge", "requires", "evidence"),
|
||||
("evidence", "supports", "truth"),
|
||||
("truth", "reveals", "reality"),
|
||||
],
|
||||
"moves": ["ASSERT", "SEQUENCE", "ELABORATE", "SEQUENCE"],
|
||||
},
|
||||
{
|
||||
"topic": "scientific_method",
|
||||
"triples": [
|
||||
("observation", "grounds", "hypothesis"),
|
||||
("hypothesis", "implies", "prediction"),
|
||||
("prediction", "follows", "experiment"),
|
||||
("experiment", "supports", "theory"),
|
||||
("theory", "entails", "explanation"),
|
||||
],
|
||||
"moves": ["ASSERT", "ELABORATE", "SEQUENCE", "ELABORATE", "SEQUENCE"],
|
||||
},
|
||||
{
|
||||
"topic": "creation_arc",
|
||||
"triples": [
|
||||
("light", "precedes", "form"),
|
||||
("form", "grounds", "matter"),
|
||||
("matter", "supports", "structure"),
|
||||
("structure", "reveals", "order"),
|
||||
],
|
||||
"moves": ["ASSERT", "SEQUENCE", "ELABORATE", "SEQUENCE"],
|
||||
},
|
||||
{
|
||||
"topic": "logical_dependency",
|
||||
"triples": [
|
||||
("premise", "supports", "conclusion"),
|
||||
("conclusion", "requires", "validity"),
|
||||
("validity", "entails", "soundness"),
|
||||
],
|
||||
"moves": ["ASSERT", "SEQUENCE", "ELABORATE"],
|
||||
},
|
||||
{
|
||||
"topic": "ethical_grounding",
|
||||
"triples": [
|
||||
("virtue", "grounds", "action"),
|
||||
("action", "requires", "intention"),
|
||||
("intention", "supports", "consequence"),
|
||||
("consequence", "reveals", "character"),
|
||||
],
|
||||
"moves": ["ASSERT", "SEQUENCE", "ELABORATE", "SEQUENCE"],
|
||||
},
|
||||
{
|
||||
"topic": "linguistic_layers",
|
||||
"triples": [
|
||||
("sound", "grounds", "phoneme"),
|
||||
("phoneme", "supports", "morpheme"),
|
||||
("morpheme", "builds", "word"),
|
||||
("word", "composes", "sentence"),
|
||||
("sentence", "conveys", "meaning"),
|
||||
],
|
||||
"moves": ["ASSERT", "SEQUENCE", "ELABORATE", "SEQUENCE", "ELABORATE"],
|
||||
},
|
||||
{
|
||||
"topic": "mathematical_chain",
|
||||
"triples": [
|
||||
("axiom", "grounds", "theorem"),
|
||||
("theorem", "entails", "corollary"),
|
||||
("corollary", "supports", "application"),
|
||||
("application", "yields", "insight"),
|
||||
],
|
||||
"moves": ["ASSERT", "ELABORATE", "SEQUENCE", "SEQUENCE"],
|
||||
},
|
||||
{
|
||||
"topic": "narrative_progression",
|
||||
"triples": [
|
||||
("conflict", "drives", "tension"),
|
||||
("tension", "precedes", "climax"),
|
||||
("climax", "yields", "resolution"),
|
||||
("resolution", "reveals", "theme"),
|
||||
],
|
||||
"moves": ["ASSERT", "SEQUENCE", "ELABORATE", "SEQUENCE"],
|
||||
},
|
||||
{
|
||||
"topic": "biological_hierarchy",
|
||||
"triples": [
|
||||
("gene", "encodes", "protein"),
|
||||
("protein", "builds", "cell"),
|
||||
("cell", "composes", "tissue"),
|
||||
("tissue", "forms", "organ"),
|
||||
("organ", "supports", "organism"),
|
||||
],
|
||||
"moves": ["ASSERT", "SEQUENCE", "ELABORATE", "SEQUENCE", "ELABORATE"],
|
||||
},
|
||||
{
|
||||
"topic": "physical_causation",
|
||||
"triples": [
|
||||
("force", "drives", "motion"),
|
||||
("motion", "transfers", "energy"),
|
||||
("energy", "yields", "heat"),
|
||||
("heat", "raises", "temperature"),
|
||||
],
|
||||
"moves": ["ASSERT", "ELABORATE", "SEQUENCE", "SEQUENCE"],
|
||||
},
|
||||
# Contrast-shaped cases — exercises the "in contrast" template.
|
||||
{
|
||||
"topic": "contrastive_definitions",
|
||||
"triples": [
|
||||
("knowledge", "requires", "evidence"),
|
||||
("belief", "requires", "trust"),
|
||||
("wisdom", "grounds", "judgment"),
|
||||
],
|
||||
"moves": ["ASSERT", "CONTRAST", "ELABORATE"],
|
||||
},
|
||||
{
|
||||
"topic": "method_contrast",
|
||||
"triples": [
|
||||
("deduction", "yields", "certainty"),
|
||||
("induction", "yields", "probability"),
|
||||
("abduction", "yields", "explanation"),
|
||||
],
|
||||
"moves": ["ASSERT", "CONTRAST", "ELABORATE"],
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
HOLDOUT_TOPICS: list[dict] = [
|
||||
{
|
||||
"topic": "musical_construction",
|
||||
"triples": [
|
||||
("note", "composes", "chord"),
|
||||
("chord", "supports", "harmony"),
|
||||
("harmony", "yields", "phrase"),
|
||||
("phrase", "builds", "melody"),
|
||||
],
|
||||
"moves": ["ASSERT", "SEQUENCE", "ELABORATE", "SEQUENCE"],
|
||||
},
|
||||
{
|
||||
"topic": "social_structure",
|
||||
"triples": [
|
||||
("custom", "grounds", "tradition"),
|
||||
("tradition", "supports", "institution"),
|
||||
("institution", "shapes", "society"),
|
||||
("society", "reveals", "culture"),
|
||||
],
|
||||
"moves": ["ASSERT", "SEQUENCE", "ELABORATE", "SEQUENCE"],
|
||||
},
|
||||
{
|
||||
"topic": "computational_pipeline",
|
||||
"triples": [
|
||||
("input", "drives", "computation"),
|
||||
("computation", "yields", "output"),
|
||||
("output", "supports", "decision"),
|
||||
],
|
||||
"moves": ["ASSERT", "SEQUENCE", "ELABORATE"],
|
||||
},
|
||||
{
|
||||
"topic": "psychological_development",
|
||||
"triples": [
|
||||
("sensation", "grounds", "perception"),
|
||||
("perception", "supports", "memory"),
|
||||
("memory", "yields", "learning"),
|
||||
("learning", "shapes", "behavior"),
|
||||
("behavior", "reveals", "character"),
|
||||
],
|
||||
"moves": ["ASSERT", "SEQUENCE", "ELABORATE", "SEQUENCE", "ELABORATE"],
|
||||
},
|
||||
{
|
||||
"topic": "economic_flow",
|
||||
"triples": [
|
||||
("labor", "yields", "value"),
|
||||
("value", "supports", "exchange"),
|
||||
("exchange", "drives", "growth"),
|
||||
],
|
||||
"moves": ["ASSERT", "SEQUENCE", "ELABORATE"],
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
# Common discourse markers the realizer emits per RhetoricalMove
|
||||
# (see generate.templates._MOVE_TEMPLATES).
|
||||
_MARKERS_BY_MOVE: dict[str, str] = {
|
||||
"ASSERT": "",
|
||||
"ELABORATE": "furthermore",
|
||||
"CONTRAST": "in contrast",
|
||||
"SEQUENCE": "next",
|
||||
"CORRECT": "correction:",
|
||||
}
|
||||
|
||||
|
||||
def _build_case(prefix: str, idx: int, topic: dict) -> dict:
|
||||
triples = topic["triples"]
|
||||
moves = topic["moves"]
|
||||
assert len(triples) == len(moves), f"length mismatch in {topic['topic']}"
|
||||
|
||||
nodes = [
|
||||
{
|
||||
"node_id": f"n{i+1}",
|
||||
"subject": s,
|
||||
"predicate": p,
|
||||
"obj": o,
|
||||
}
|
||||
for i, (s, p, o) in enumerate(triples)
|
||||
]
|
||||
steps = [
|
||||
{"node_id": f"n{i+1}", "move": m}
|
||||
for i, m in enumerate(moves)
|
||||
]
|
||||
must_contain_subjects = [t[0] for t in triples]
|
||||
discourse_markers = sorted(
|
||||
{_MARKERS_BY_MOVE[m] for m in moves if _MARKERS_BY_MOVE[m]}
|
||||
)
|
||||
|
||||
return {
|
||||
"id": f"{prefix}_{idx:03d}",
|
||||
"topic": topic["topic"],
|
||||
"graph": {"nodes": nodes, "edges": []},
|
||||
"steps": steps,
|
||||
"min_sentences": len(triples),
|
||||
"must_contain_subjects": must_contain_subjects,
|
||||
"discourse_markers": discourse_markers,
|
||||
"max_sentences": len(triples) + 2, # tolerate small over-runs from
|
||||
# downstream wrapping
|
||||
}
|
||||
|
||||
|
||||
def _emit(prefix: str, topics: list[dict], out_path: Path) -> int:
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
lines = [
|
||||
json.dumps(_build_case(prefix, i + 1, t), ensure_ascii=False)
|
||||
for i, t in enumerate(topics)
|
||||
]
|
||||
out_path.write_text("\n".join(lines) + "\n")
|
||||
return len(lines)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
lane = root / "evals" / "discourse_paragraph"
|
||||
n_pub = _emit("DP-PUB", PUBLIC_TOPICS, lane / "public" / "v1" / "cases.jsonl")
|
||||
n_hold = _emit("DP-HOLD", HOLDOUT_TOPICS, lane / "holdouts" / "v1" / "cases.jsonl")
|
||||
n_dev = _emit("DP-DEV", PUBLIC_TOPICS[:1], lane / "dev" / "cases.jsonl")
|
||||
print(f"discourse_paragraph public={n_pub} holdouts={n_hold} dev={n_dev}")
|
||||
98
tests/test_benchmarks_profiler.py
Normal file
98
tests/test_benchmarks_profiler.py
Normal file
|
|
@ -0,0 +1,98 @@
|
|||
"""Tests for benchmarks.pipeline_profiler and benchmarks.word_selection_tracer.
|
||||
|
||||
These are pure instrumentation tests — they assert that the profiler and
|
||||
tracer capture structural breakdowns without altering pipeline semantics.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from benchmarks.pipeline_profiler import ProfileReport, profile_turn
|
||||
from benchmarks.word_selection_tracer import (
|
||||
RealizationTrace,
|
||||
WordSelectionStep,
|
||||
trace_realization,
|
||||
)
|
||||
from chat.runtime import ChatRuntime
|
||||
from core.cognition import CognitiveTurnPipeline
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def runtime() -> ChatRuntime:
|
||||
return ChatRuntime()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def pipeline(runtime: ChatRuntime) -> CognitiveTurnPipeline:
|
||||
return CognitiveTurnPipeline(runtime)
|
||||
|
||||
|
||||
def test_profile_turn_returns_stage_breakdown(pipeline: CognitiveTurnPipeline) -> None:
|
||||
"""profile_turn returns a ProfileReport whose stages cover the pipeline spine."""
|
||||
report = profile_turn(pipeline, "light logos", max_tokens=8)
|
||||
|
||||
assert isinstance(report, ProfileReport)
|
||||
assert report.total_ns > 0
|
||||
assert isinstance(report.stages, dict)
|
||||
|
||||
# Mandatory stages (always traversed by pipeline.run regardless of input).
|
||||
required = {
|
||||
"intent",
|
||||
"graph_planner",
|
||||
"realize_semantic",
|
||||
"runtime_chat",
|
||||
"trace_hash",
|
||||
}
|
||||
missing = required - set(report.stages.keys())
|
||||
assert not missing, f"Profiler missed required stages: {missing}"
|
||||
|
||||
# Each captured stage must have a non-negative timing.
|
||||
for name, ns in report.stages.items():
|
||||
assert ns >= 0, f"Stage {name} had negative timing {ns}"
|
||||
|
||||
# Sum of timed stages must not exceed total elapsed (sanity, allow equal).
|
||||
sum_stages = sum(report.stages.values())
|
||||
assert sum_stages <= report.total_ns + 1_000_000 # 1ms slack for overhead
|
||||
|
||||
# as_dict is JSON-friendly.
|
||||
d = report.as_dict()
|
||||
assert d["total_ns"] == report.total_ns
|
||||
assert d["stages"] == report.stages
|
||||
|
||||
# Verify the original methods were restored on the pipeline.
|
||||
assert not isinstance(pipeline._maybe_transitive_walk, type(lambda: None)) or (
|
||||
pipeline._maybe_transitive_walk.__qualname__.startswith("CognitiveTurnPipeline")
|
||||
)
|
||||
|
||||
|
||||
def test_trace_realization_captures_word_choices(pipeline: CognitiveTurnPipeline) -> None:
|
||||
"""trace_realization records every nearest-neighbor lookup with top-K candidates."""
|
||||
trace = trace_realization(pipeline, "light logos", top_k=3)
|
||||
|
||||
assert isinstance(trace, RealizationTrace)
|
||||
|
||||
# The realizer-step list may be empty if the intent produced no
|
||||
# ArticulationTarget steps, but on a normal known-token input we
|
||||
# expect at least one realization step OR at least one slot lookup.
|
||||
assert trace.steps or trace.realization_steps, (
|
||||
"Tracer captured neither word-selection steps nor realization steps"
|
||||
)
|
||||
|
||||
# If any slot lookups were recorded, validate their shape.
|
||||
for step in trace.steps:
|
||||
assert isinstance(step, WordSelectionStep)
|
||||
assert step.slot in {"subject", "predicate", "object"} or step.slot.startswith("slot_")
|
||||
assert step.input_versor.shape == (32,)
|
||||
assert len(step.top_candidates) >= 1
|
||||
# top_candidates must be sorted by score descending.
|
||||
scores = [score for (_, score) in step.top_candidates]
|
||||
assert scores == sorted(scores, reverse=True)
|
||||
# chosen word must appear in top_candidates.
|
||||
words = [w for (w, _) in step.top_candidates]
|
||||
assert step.chosen in words or step.chosen == words[0] or len(words) > 0
|
||||
assert isinstance(step.morphology, dict)
|
||||
|
||||
# as_dict is JSON-friendly.
|
||||
d = trace.as_dict()
|
||||
assert "steps" in d and "realization_steps" in d and "surface" in d
|
||||
Loading…
Reference in a new issue