core/evals/discourse_paragraph
Shay b5d6ad6510 feat(compositionality): compose_relations operator lifts lane 68.8% → 100%
Closes the residual `novel_pair_under_seen_relation` pattern that
neither `transitive_walk` nor `multi_relation_walk` could synthesise.

- new `compose_relations(triples, head, frame, relation)` operator —
  pure lookup, returns both `R(head, ?)` and `R(frame, ?)` tails
- new `FRAME_TRANSFER` intent + `_FRAME_TRANSFER_RE` regex tried
  before generic TRANSITIVE_QUERY so "in Y" isn't truncated; handles
  "X belong to in Y" → belongs_to normalisation
- pipeline wiring: `_maybe_compose_relations`, `_fold_compose_into_surface`,
  `_serialize_compose` (folded into operator_invocation so trace_hash
  stays bit-identical across replay)
- regression: inference_closure, multi_step_reasoning,
  cross_domain_transfer all still 100% on public + holdouts

discourse_paragraph v2:
- per-sentence grammar rubric (length, capitalization, subject
  alignment) gated on `require_per_sentence_grammar`
- scaling cases at 10 / 20 / 50 sentences — 3/3 pass, 100% per-sentence
- 3 runtime round-trip cases (`mode: runtime_roundtrip`) that prime
  vault, ask question, verify bit-identical across two fresh runtimes
- new `per_sentence_grammar_pass_rate` lane metric

Long-form replay benchmark (benchmarks/replay_vs_llm.py):
- `replay_determinism_report(prompts, runs, priming)` — CORE-only
- `compare_to_llm(prompts, llm_callable)` — BYO API client, no
  provider lock-in; reports per-prompt determinism on both sides
- ships with default cognition-pack prompts; 100% bit-identical at runs=3

Lanes green: cognition 121/121, runtime 19/19, teaching 17/17,
packs 6/6, compositionality 16/16 + 10/10, inference_closure 20/20 +
12/12, multi_step_reasoning 15/15 + 10/10, cross_domain_transfer
10/10 + 8/8, discourse_paragraph v1 12/12 + v2 6/6.
2026-05-16 22:44:06 -07:00
..
dev feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer 2026-05-16 21:53:46 -07:00
holdouts/v1 feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer 2026-05-16 21:53:46 -07:00
public feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
results feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer 2026-05-16 21:53:46 -07:00
__init__.py feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer 2026-05-16 21:53:46 -07:00
contract.md feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
gaps.md feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
runner.py feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00