docs(evals): articulation benchmark preamble — discourse-planner spine
Records the deterministic, grounded, multi-clause articulation
benchmark that the discourse-planner work has stabilised. Mirrors
the format of teaching_loop_bench.md so the four sub-benches in
benchmarks/articulation.py have a load-bearing reference document.
Headline:
* 20 independent ChatRuntime instances × 4 prompts (EXPLAIN /
PARAGRAPH / COMPOUND / WALKTHROUGH) produce 4 unique surfaces —
byte-identical determinism on the articulation path with
RuntimeConfig(discourse_planner=True).
* Every visible token traces to a pack lemma, pack gloss, reviewed
teaching-chain entry, or fixed-template connective from the
closed five-entry _MOVE_CONNECTIVE table. No synthesis.
* discourse_planner sub-bench:
cases: 4
articulate_sentence_rate: 1.0
disclosure_sentence_rate: 0.0
multi_sentence_rate: 1.0
* Compound prompt ("What is truth, and why does it matter?") emits
6 distinct grounded sentences with cross-part fact dedup, no
anchor repetition.
* Walkthrough mode walks the teaching-chain edge graph up to 3 hops,
cycle-safe, final hop as CLOSURE; no chain ⇒ degrades to ANCHOR +
SUPPORT rather than fabricating steps.
Doc explains the partitioned predicate contract
(articulate + disclosure + unarticulate = 1.0, total and disjoint)
so future readers know why ``multi_sentence_rate`` alone is not the
headline.
Companion docs cross-linked: discourse_runtime_baseline_2026-05-19.md
(lane-level delta table), the two new isolation lanes
(compound_intent_decomposition, walkthrough_chain), and the
partitioned multi_sentence_response contract.
This commit is contained in:
Shay
parent
e985790a03
commit
90fc1b40a0
1 changed files with 208 additions and 0 deletions
208
docs/evals/articulation_bench_2026-05-19.md
Normal file
208
docs/evals/articulation_bench_2026-05-19.md
Normal file
|
|
@ -0,0 +1,208 @@
|
|||
# Articulation Benchmark — Discourse Planner Spine
|
||||
|
||||
**Date:** 2026-05-19
|
||||
**Runner:** `benchmarks/articulation.py`
|
||||
**CLI:** `core bench --suite articulation [--json]`
|
||||
**Contract tests:** `tests/test_articulation_bench.py`
|
||||
**Reference commits:** [`e985790`](../../) (lanes + holdouts + bench sub-bench),
|
||||
[`4e3ddee`](../../) (WALKTHROUGH v1),
|
||||
[`7af7892`](../../) (compound decomposition + sub-plan composition)
|
||||
|
||||
## Headline claim
|
||||
|
||||
> On the substrate currently mounted (cognition + relations + minimal +
|
||||
> domain packs, the reviewed cognition-chains corpus, and the
|
||||
> cross-pack chains corpus), the discourse-planner spine produces
|
||||
> deterministic, grounded, multi-clause articulation on every prompt
|
||||
> shape it claims to handle:
|
||||
>
|
||||
> | Prompt shape | Articulate¹ | Grounding | Sentences |
|
||||
> |---|---:|---|---:|
|
||||
> | `Explain X.` | ✓ | teaching | 3 |
|
||||
> | `Write a paragraph about X.` | ✓ | teaching | 3 |
|
||||
> | `What is X, and why does it matter?` | ✓ | teaching | 6 |
|
||||
> | `Walk me through X.` | ✓ | teaching | 2 |
|
||||
>
|
||||
> Across 20 independent runtime instances per prompt, every surface is
|
||||
> **byte-identical**: 20×4 generations produce exactly 4 unique
|
||||
> surfaces. No stochastic sampling, no LLM fallback, no approximate
|
||||
> recall.
|
||||
|
||||
¹ `articulate_sentence_rate` predicate: ≥2 substantive sentences AND
|
||||
`grounding_source ∈ {pack, teaching}`. OOV invitations and refusal
|
||||
disclosures count toward `disclosure_sentence_rate`, never articulate
|
||||
— the lane partition is total and disjoint.
|
||||
|
||||
## What's measured
|
||||
|
||||
`benchmarks/articulation.py` packages four sub-benches that exercise
|
||||
the chat-spine end-to-end with `RuntimeConfig(discourse_planner=True)`:
|
||||
|
||||
| Sub-bench | Probes | What it asserts |
|
||||
|---|---|---|
|
||||
| `breadth` | 12 prompts spanning 9 intents | The classifier routes representative prompts to the expected `IntentTag` and the runtime grounds in `{pack, teaching, oov, none}` — no unclassifiable surprises in the breadth distribution. |
|
||||
| `determinism` | 5 prompts × 20 runs each | Each prompt produces exactly **1** unique surface across 20 fresh `ChatRuntime` instances. Tests for clock reads, env reads, stochastic sampling, or shared mutable state in the warm-path planner hook. |
|
||||
| `cross_topic` | 8 turns on one runtime, `thread_anaphora=True` | Counts how many turns fired the deterministic anaphora prefix. Sanity-checks ADR-0066 thread continuity under live chat conditions. |
|
||||
| `discourse_planner` | 4 prompts, one per supported mode (EXPLAIN / PARAGRAPH / COMPOUND / WALKTHROUGH) | Reports `articulate_sentence_rate`, `disclosure_sentence_rate`, `multi_sentence_rate`. Single load-bearing capability metric per prompt shape. |
|
||||
|
||||
## Today's reference numbers
|
||||
|
||||
```text
|
||||
[breadth] 12 prompts in ~3.15s
|
||||
intents: CAUSE, COMPARISON, CORRECTION, DEFINITION,
|
||||
EXAMPLE, NARRATIVE, PROCEDURE, UNKNOWN, VERIFICATION
|
||||
grounding: none, oov, pack, teaching
|
||||
|
||||
[determinism] 5 prompts × 20 runs in ~12.85s
|
||||
byte-identical across runs: True
|
||||
unique surface counts: [1, 1, 1, 1, 1]
|
||||
|
||||
[cross_topic] 8 turns single runtime in ~7.38s
|
||||
anaphora fired on 0/8 turns
|
||||
(turns in this bench are independent topics by design;
|
||||
see test_chat_anaphora_*.py for the firing path)
|
||||
|
||||
[discourse_planner] 4 prompts in ~0.53s
|
||||
metrics: {
|
||||
cases: 4,
|
||||
articulate_sentence_rate: 1.0,
|
||||
disclosure_sentence_rate: 0.0,
|
||||
multi_sentence_rate: 1.0,
|
||||
}
|
||||
|
||||
[EXPLAIN] sentences=3 grounding=teaching articulate=True
|
||||
[PARAGRAPH] sentences=3 grounding=teaching articulate=True
|
||||
[COMPOUND] sentences=6 grounding=teaching articulate=True
|
||||
[WALKTHROUGH] sentences=2 grounding=teaching articulate=True
|
||||
```
|
||||
|
||||
## Sample surfaces
|
||||
|
||||
These are the literal surfaces emitted by the planner-on chat spine —
|
||||
every visible token below is a verbatim pack lemma, a verbatim pack
|
||||
gloss, a verbatim reviewed-teaching-chain entry, or a fixed-template
|
||||
connective from `_MOVE_CONNECTIVE` in `generate/discourse_planner.py`.
|
||||
|
||||
> **EXPLAIN — `"Explain truth."`**
|
||||
>
|
||||
> Truth is a claim or state grounded by evidence and coherent
|
||||
> judgment. Furthermore, truth belongs to cognition.truth. In turn,
|
||||
> truth grounds knowledge.
|
||||
|
||||
> **PARAGRAPH — `"Write a paragraph about truth."`**
|
||||
>
|
||||
> Truth is a claim or state grounded by evidence and coherent
|
||||
> judgment. Furthermore, truth belongs to cognition.truth. In turn,
|
||||
> truth grounds knowledge.
|
||||
|
||||
> **COMPOUND — `"What is truth, and why does it matter?"`**
|
||||
>
|
||||
> Truth is a claim or state grounded by evidence and coherent
|
||||
> judgment. Furthermore, truth belongs to cognition.truth. In turn,
|
||||
> truth grounds knowledge. Truth belongs to epistemic.ground.
|
||||
> Furthermore, truth belongs to logos.core. In turn, truth requires
|
||||
> evidence.
|
||||
|
||||
> **WALKTHROUGH — `"Walk me through recall."`**
|
||||
>
|
||||
> Recall is to retrieve a stored state from memory. Recall reveals
|
||||
> memory.
|
||||
|
||||
## Why this matters
|
||||
|
||||
### 1. Determinism on the articulation path
|
||||
|
||||
CORE's design commitment (CLAUDE.md §"Philosophical Stance") is that
|
||||
the same input under the same vault state always produces the same
|
||||
articulated output — exactly enough to support deterministic replay,
|
||||
trace hashing, and reviewed teaching.
|
||||
|
||||
The `determinism` sub-bench enforces this end-to-end with the
|
||||
planner engaged: 20 independent `ChatRuntime` instances per prompt,
|
||||
one unique surface per prompt. This is the *learning-loop
|
||||
determinism* of [ADR-0055](../decisions/) and [ADR-0057](../decisions/)
|
||||
applied to the articulation spine rather than only to retrieval and
|
||||
proposal acceptance.
|
||||
|
||||
### 2. Compound prompts compose without re-sorting
|
||||
|
||||
`"What is truth, and why does it matter?"` decomposes into
|
||||
`(DEFINITION(truth), CAUSE(truth))`. The planner concatenates the
|
||||
two sub-plans in source order with cross-part fact deduplication —
|
||||
six distinct grounded sentences with no anchor repetition. This is
|
||||
the discourse-graph traversal the design memo
|
||||
([feedback-design-fix-upstream-not-beside](../../../../.claude/projects/-Users-kaizenpro-Projects-core/memory/feedback-design-fix-upstream-not-beside.md))
|
||||
recommended: lift structure upstream rather than decorate strings
|
||||
downstream.
|
||||
|
||||
### 3. Walkthroughs walk a teaching graph, not a template
|
||||
|
||||
`WALKTHROUGH` mode walks the teaching-chain edge graph
|
||||
`(subject, *, obj) → (obj, *, *)` up to 3 hops, with the final hop
|
||||
emitted as `CLOSURE` and cycle-safety enforced by the used-fact set.
|
||||
When no chain is rooted on the anchor the planner degrades to the
|
||||
expository (ANCHOR + SUPPORT) shape rather than fabricating walk
|
||||
steps.
|
||||
|
||||
### 4. Honest negative spaces
|
||||
|
||||
`disclosure_sentence_rate = 0.0` on flag-on, but the metric exists.
|
||||
OOV teaching invitations and refusal disclosures are structurally
|
||||
multi-sentence by template — they cannot be allowed to inflate
|
||||
articulate capability. The partition
|
||||
(`articulate + disclosure + unarticulate = 1.0`) is total and
|
||||
disjoint by construction; the headline rate measures *only* what
|
||||
the spine actually planned and rendered.
|
||||
|
||||
## Provenance of every token in the surface
|
||||
|
||||
For every move in every plan that produced the surfaces above, the
|
||||
`fact` object carries:
|
||||
|
||||
* `source ∈ {PACK, TEACHING}` — never `OPERATOR`, never anything
|
||||
synthesised.
|
||||
* `source_id` — a pointer back to the exact pack lemma
|
||||
(`en_core_cognition_v1:truth#gloss`) or reviewed teaching chain
|
||||
(`cognition_chains_v1#cause_truth_reveals_knowledge`).
|
||||
* `subject` / `predicate` / `obj` — verbatim from the lexicon entry
|
||||
or chain JSONL.
|
||||
|
||||
Connectives between moves are drawn exclusively from the closed
|
||||
five-entry table `_MOVE_CONNECTIVE`:
|
||||
|
||||
```python
|
||||
ANCHOR -> ""
|
||||
SUPPORT -> "Furthermore, "
|
||||
RELATION -> "In turn, "
|
||||
TRANSITION -> "Consequently, "
|
||||
CLOSURE -> ""
|
||||
```
|
||||
|
||||
There is no other source of visible text in the rendered surface.
|
||||
The articulation is deterministic *because* it's reconstructed from
|
||||
sourced atoms; the byte-identity result above is the consequence,
|
||||
not the design intent.
|
||||
|
||||
## Reproduction
|
||||
|
||||
```bash
|
||||
# Full articulation bench (requires psutil for the footprint
|
||||
# sub-bench; the other sub-benches run without it):
|
||||
core bench --suite articulation --json
|
||||
|
||||
# Planner-on sub-bench only, without psutil dependency:
|
||||
python3 -c "from benchmarks.articulation import bench_discourse_planner; \
|
||||
probes, metrics = bench_discourse_planner(); \
|
||||
print(metrics)"
|
||||
```
|
||||
|
||||
## Companion documents
|
||||
|
||||
* [`discourse_runtime_baseline_2026-05-19.md`](./discourse_runtime_baseline_2026-05-19.md)
|
||||
— full lane-level delta table across the 14-commit landing.
|
||||
* `evals/compound_intent_decomposition/contract.md` — isolation lane
|
||||
for compound decomposition (`decomposition=1.0` on public/v1).
|
||||
* `evals/walkthrough_chain/contract.md` — isolation lane for
|
||||
walkthrough teaching-chain walks (`path_exact=1.0` on public/v1).
|
||||
* `evals/multi_sentence_response/contract.md` — partitioned predicate
|
||||
contract (`articulate / disclosure / unarticulate`).
|
||||
Loading…
Reference in a new issue