Shay b5d6ad6510 feat(compositionality): compose_relations operator lifts lane 68.8% → 100%

Closes the residual `novel_pair_under_seen_relation` pattern that
neither `transitive_walk` nor `multi_relation_walk` could synthesise.

- new `compose_relations(triples, head, frame, relation)` operator —
  pure lookup, returns both `R(head, ?)` and `R(frame, ?)` tails
- new `FRAME_TRANSFER` intent + `_FRAME_TRANSFER_RE` regex tried
  before generic TRANSITIVE_QUERY so "in Y" isn't truncated; handles
  "X belong to in Y" → belongs_to normalisation
- pipeline wiring: `_maybe_compose_relations`, `_fold_compose_into_surface`,
  `_serialize_compose` (folded into operator_invocation so trace_hash
  stays bit-identical across replay)
- regression: inference_closure, multi_step_reasoning,
  cross_domain_transfer all still 100% on public + holdouts

discourse_paragraph v2:
- per-sentence grammar rubric (length, capitalization, subject
  alignment) gated on `require_per_sentence_grammar`
- scaling cases at 10 / 20 / 50 sentences — 3/3 pass, 100% per-sentence
- 3 runtime round-trip cases (`mode: runtime_roundtrip`) that prime
  vault, ask question, verify bit-identical across two fresh runtimes
- new `per_sentence_grammar_pass_rate` lane metric

Long-form replay benchmark (benchmarks/replay_vs_llm.py):
- `replay_determinism_report(prompts, runs, priming)` — CORE-only
- `compare_to_llm(prompts, llm_callable)` — BYO API client, no
  provider lock-in; reports per-prompt determinism on both sides
- ships with default cognition-pack prompts; 100% bit-identical at runs=3

Lanes green: cognition 121/121, runtime 19/19, teaching 17/17,
packs 6/6, compositionality 16/16 + 10/10, inference_closure 20/20 +
12/12, multi_step_reasoning 15/15 + 10/10, cross_domain_transfer
10/10 + 8/8, discourse_paragraph v1 12/12 + v2 6/6.

2026-05-16 22:44:06 -07:00

2.7 KiB

Raw Blame History

discourse_paragraph — gaps

v1 (current)

Realizer-isolation lane: bypasses runtime grounding so the paragraph claim is unconfounded by vault noise.
Sentence-count window is intentionally generous (max_sentences = min + 2) to tolerate small wrapping variance from compound-clause folding in realize_target (CONJUNCTION / COMPLEMENT / RELATIVE edges merge two steps into one sentence).
Subject coverage threshold is 0.75, not 1.0 — exact-coverage cases pass that bar comfortably but the slack lets a future realizer change ship without rewriting cases.

Status: v2 partially shipped

Length scaling (was gap 3 — resolved): public/v2 exercises 10 / 20 / 50-sentence cases. All three pass at 100% with bit- identical replay. First lane to push paragraph output past five sentences.
Per-sentence grammaticality (was gap 4 — resolved): runner adds _check_per_sentence_grammar gated on require_per_sentence_grammar case field. Per case: each emitted sentence must be non-empty, contain ≥ 3 whitespace tokens, start with an uppercase letter, and (when align_steps_to_sentences is set) contain the aligned step's subject. Lane reports per_sentence_grammar_pass_rate.

Remaining v3 gaps

Runtime round-trip — partial (single-sentence only). v2 adds round-trip cases (mode: "runtime_roundtrip") that prime the vault, ask a question through ChatRuntime.chat, and verify the articulation surface is well-formed, capitalized, contains an expected token, and is bit-identical across two fresh runtime instances. Three cases pass at 100%. But the runtime/planner currently produces one sentence per turn — the multi-sentence-from-runtime claim still requires a planner extension (e.g. expanding a single user question into a multi-step ArticulationTarget via graph traversal). That is the real v3 gap.
No anaphora / pronoun reduction. Every sentence carries its subject explicitly. Pronominalisation deferred.
No cross-sentence grammatical_coverage rubric. The v2 per-sentence check is structural (length, capitalization, subject alignment); it does not run each sentence through evals/grammatical_coverage's constraint rubric. Reuse should be straightforward once a sentence-to-constraint mapping is designed.

Why this lane exists

First lane that exercises paragraph-scale output. Every previous fluency lane (Phase 5.1 + 5.4–5.7) operates on 3-word SVO probes. The structural capability — folding multiple articulation steps into a coherent paragraph with deterministic discourse markers — was already in the realizer; this lane makes it measurable.

2.7 KiB Raw Blame History Unescape Escape