Shay 257a27c105 feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer

Closes the user-flagged scope gap: every previous fluency lane (Phase
5.1 + 5.4-5.7 + grammatical_coverage) operates on 3-word SVO probes.
These three pieces stress paragraph-scale generation, give per-stage
latency visibility, and expose the realizer's word-choice geometry —
all on top of the existing deterministic infrastructure.

# discourse_paragraph lane (paragraph-scale fluency)

Forces the realizer to emit multi-sentence paragraphs from a
multi-step ArticulationTarget with rhetorical moves (ASSERT, SEQUENCE,
ELABORATE, CONTRAST).  Same realizer, much richer input — every case
is 3-5 sentences with deterministic discourse markers.

Public 12 cases / holdouts 5 / dev 1 across 12 + 5 topic chains
(epistemic, scientific method, creation arc, logical dependency,
ethical grounding, linguistic layers, mathematical chain, narrative,
biology, physics, two contrast-shaped, musical, social, computational,
psychological, economic).

Sub-metrics per case:
  - sentence count (within min..max window)
  - subject coverage rate
  - discourse marker presence (next / furthermore / in contrast)
  - sentence-initial capitalization
  - replay determinism (run twice, surfaces match)

Result: 12/12 public + 5/5 holdouts at 100%, replay rate 100%, mean
sentence count 4.

# Realizer capitalization (G4, addresses user-flagged concern)

generate/realizer.py gains `_capitalize_sentence` + `_join_as_paragraph`
helpers.  Sentence-initial alphabetic characters are now uppercased
(skipping leading whitespace/punctuation).  Surfaces went from
"wisdom grounds knowledge. next, knowledge requires evidence."
to
"Wisdom grounds knowledge. Next, knowledge requires evidence."

The discourse_paragraph runner ships a strict per-sentence
capitalization check so future regressions get caught.

# Pipeline-stage profiler (benchmarks/pipeline_profiler.py)

External monkey-patch wrapper around CognitiveTurnPipeline.run() that
records per-stage ns budgets without editing any pipeline source.
Stages: intent, graph_planner, realize_semantic, runtime_chat,
maybe_transitive_walk, fold_walk_into_surface, run_teaching,
trace_hash.

API: `profile_turn(pipeline, text) -> ProfileReport` with
`.stages: dict`, `.total_ns: int`, `.as_dict()`.

Empirical: runtime_chat dominates >99% on the runtime hot path (which
is correct — that's where ingest + propagate + recall + articulate
all happen).  Future optimisation work has a clear per-stage signal.

# Word-selection tracer (benchmarks/word_selection_tracer.py)

External wrapper around generate.articulation._resolve_slot that
records every nearest-neighbor lookup as a WordSelectionStep:
  - slot (subject/predicate/object)
  - input versor (32-d copy)
  - top-K candidate words by CGA inner product
  - chosen word + morphology
  - output language

Top-K scoring uses the diagonal Cl(4,1) metric kernel from
algebra.backend (same vectorised path vault_recall uses), not a
per-word Python loop over cga_inner.  No approximation, exact
deterministic ranking, bit-identical to a scalar scan.

API: `trace_realization(pipeline, text) -> RealizationTrace` with
`.steps`, `.realization_steps`, `.surface`, `.as_dict()`.

# CLI lane registration

Cognition suite now sweeps the benchmark profiler/tracer tests
(test_benchmarks_profiler.py) so any future regression in the
instrumentation surfaces immediately.

# Constraints honoured

- Zero edits to core/, chat/, vault/, teaching/, language_packs/, or
  the algebra hot path.  All instrumentation is external monkey-patch
  with originals restored in finally.
- discourse_paragraph runner bypasses ChatRuntime grounding (named v2
  gap) so paragraph capability is isolated to the realizer.
- No semantic changes; no hidden normalisation; no approximate
  recall.

# Lane health

smoke 55, runtime 19, teaching 17, packs 6, cognition 105 (was 103),
algebra 132.  All Phase 5 fluency lanes still 100% with the
capitalised surfaces (rubric is case-insensitive).  discourse_paragraph
100%.

# What ships next (named v2)

- Round-trip: discourse_paragraph through ChatRuntime end-to-end,
  not just realize_target.
- Per-sentence grammatical_coverage rubric on each emitted sentence.
- Longer chains (10/20/50 sentences) with per-sentence determinism
  scaling curves.
- compose_relations operator to lift compositionality recall from
  68.8% toward 100%.

2026-05-16 21:53:46 -07:00

1.9 KiB

Raw Blame History

discourse_paragraph eval lane

What it measures

Whether the deterministic realizer can produce paragraph-scale output — multiple grammatical sentences joined by deterministic discourse markers — from a multi-step ArticulationTarget.

This is the first lane that stresses output longer than a single 3-word SVO sentence. It addresses the open scope item: "longer/more complex sentences and phrases for testing and proving stuff".

Inputs

Each case carries a graph (≥ 3 nodes), an ordered steps list (ASSERT open, then SEQUENCE / ELABORATE / CONTRAST), and acceptance constraints:

{
  "id": "DP-PUB_001",
  "topic": "epistemic_chain",
  "graph": {"nodes": [{"node_id": "n1", "subject": "wisdom",
                       "predicate": "grounds", "obj": "knowledge"}, ...],
            "edges": []},
  "steps": [{"node_id": "n1", "move": "ASSERT"}, ...],
  "min_sentences": 4,
  "max_sentences": 6,
  "must_contain_subjects": ["wisdom", "knowledge", "evidence", "truth"],
  "discourse_markers": ["furthermore", "next"]
}

Scoring rubric

Per case:

paragraph_sentence_count ≥ min_sentences (and ≤ max_sentences)
subject_coverage_rate ≥ 0.75
discourse_marker_present — at least one expected marker emitted
replay_determinism — running the case twice produces an identical surface string

Aggregate metrics:

accuracy — pass rate
mean_sentence_count
mean_subject_coverage
replay_determinism_rate

Splits

Split	n	content
public/v1	12	epistemic / scientific / creation / logic / ethics / linguistic / math / narrative / biology / physics + 2 contrast cases
holdouts/v1	5	musical / social / computational / psychological / economic
dev	1	epistemic_chain smoke

What this lane does NOT measure

Round-trip through ChatRuntime (the realizer is exercised directly). See gaps.md.
Factual correctness of the asserted propositions.

1.9 KiB Raw Blame History