docs(adr-0055-0057): writeups + asciinema captures for the demo trilogy
Three shareable demo / benchmark writeups modeled on the existing `docs/evals/phase6_comparative_demo.md` treatment, each accompanied by an asciinema-rendered GIF for at-a-glance viewing on the repo page. - docs/evals/anti_regression_demo.md — three-gate defense; per-gate table; honesty paragraph about the synthetic regression in S2 (real ReplayEvidence shape via documented run_replay= kwarg); sample run output; falsifiable claims index. - docs/evals/learning_loop_demo.md — headline before/after; CORE-vs- pretraining comparison table; trust-boundary code snippet showing the _CORPUS_PATH swap; per-scene table; full sample run; subject- selection rationale (pack-resident ∧ no active chain ∧ deterministic intent classification). - docs/evals/teaching_loop_bench.md — what's byte-identical and why it matters per artifact; 100-run reference numbers (unique=1 across all five artifacts; mean=1.849s p50=1.838s p95=1.851s); pairing paragraph with ADR-0045 (read vs write determinism). GIF captures (rendered with asciinema 3.2.0 + agg 1.8.1, github-dark theme, JetBrains Mono): - docs/evals/assets/anti_regression.gif (120K, 944x843) - docs/evals/assets/learning_loop.gif (332K, 944x1039) - docs/evals/assets/teaching_loop_bench.gif (64K, 860x1000) Raw .cast files preserved alongside the GIFs for re-rendering at different themes / speeds / sizes without re-recording. README.md — added writeup-link column to the Inter-Session Memory three-demo table.
This commit is contained in:
Shay
parent
d24e98906e
commit
763ed16d1c
10 changed files with 552 additions and 5 deletions
10
README.md
10
README.md
|
|
@ -183,11 +183,11 @@ Supersession is the second operator-direct mutation surface: `core teaching supe
|
|||
|
||||
Three live demos / benchmarks make the chain demoable end-to-end:
|
||||
|
||||
| Demo | Headline claim | Live command |
|
||||
|---|---|---|
|
||||
| **Anti-regression** | Three independent gates each fail closed; bad proposals stop at the cheapest applicable gate. | `core demo anti-regression` |
|
||||
| **Learning loop** | Same deterministic prompt: `[none] I don't know…` before, `[teaching] thought reveals meaning…` after one accept. | `core demo learning-loop` |
|
||||
| **Determinism bench** | N identical inputs → N byte-identical proposal_id / replay metrics / chain_id. 100 runs: `unique=1` everywhere, mean ≈ 1.85s. | `core bench --suite teaching-loop --runs 100` |
|
||||
| Demo | Headline claim | Live command | Writeup |
|
||||
|---|---|---|---|
|
||||
| **Anti-regression** | Three independent gates each fail closed; bad proposals stop at the cheapest applicable gate. | `core demo anti-regression` | [`docs/evals/anti_regression_demo.md`](docs/evals/anti_regression_demo.md) |
|
||||
| **Learning loop** | Same deterministic prompt: `[none] I don't know…` before, `[teaching] thought reveals meaning…` after one accept. | `core demo learning-loop` | [`docs/evals/learning_loop_demo.md`](docs/evals/learning_loop_demo.md) |
|
||||
| **Determinism bench** | N identical inputs → N byte-identical proposal_id / replay metrics / chain_id. 100 runs: `unique=1` everywhere, mean ≈ 1.85s. | `core bench --suite teaching-loop --runs 100` | [`docs/evals/teaching_loop_bench.md`](docs/evals/teaching_loop_bench.md) |
|
||||
|
||||
Operator surfaces:
|
||||
|
||||
|
|
|
|||
175
docs/evals/anti_regression_demo.md
Normal file
175
docs/evals/anti_regression_demo.md
Normal file
|
|
@ -0,0 +1,175 @@
|
|||
# Anti-Regression Demo — Three-Gate Defense Against Learning Harm
|
||||
|
||||
**Date:** 2026-05-18
|
||||
**Runner:** `evals/anti_regression/run_demo.py`
|
||||
**CLI:** `core demo anti-regression` (`--json` for machine-readable output)
|
||||
**Contract tests:** `tests/test_anti_regression_demo.py` (5 passing)
|
||||
**Reference ADRs:** [0055](../decisions/ADR-0055-inter-session-memory-discovery-promotion.md), [0056](../decisions/ADR-0056-contemplation-loop-c1.md), [0057](../decisions/ADR-0057-teaching-chain-proposal-review.md)
|
||||
|
||||
|
||||
|
||||
## What this demo shows
|
||||
|
||||
When a system extends its own knowledge, **the gate that decides what to
|
||||
admit is the load-bearing part** — not the proposer. CORE's reviewed-
|
||||
corpus extension path has three independent gates that each must pass
|
||||
before any byte is written to the active teaching corpus. The demo
|
||||
runs each gate to verdict against a real `ProposalLog` in an isolated
|
||||
temp directory and asserts the active corpus is byte-identical pre/post.
|
||||
|
||||
| Gate | What it checks | What fails it |
|
||||
|---|---|---|
|
||||
| **S1. Eligibility predicate** (mechanical, pre-replay) | polarity ∈ {affirms, falsifies} ∧ ≥1 `source='corpus'` evidence ∧ claim_domain ≠ evaluative ∧ boundary_clean=True ∧ chain complete | Raises `ProposalError`; **no log row written**. |
|
||||
| **S2. Replay-equivalence gate** (mechanical, post-eligibility) | Cognition lane runs against active corpus AND a transient-with-append copy; any strict-decrease in `intent_accuracy / surface_groundedness / term_capture_rate / versor_closure_rate` is regression. | Auto-rejects with **named** regressed metrics in operator_note. |
|
||||
| **S3. Operator review** (manual, post-replay) | Replay-equivalence is a *precondition*, not a permission. | `--accept` not run → state stays `pending` indefinitely. |
|
||||
|
||||
## Why each gate is independent
|
||||
|
||||
A defensive lattice is only useful if each layer can refuse a bad input
|
||||
on its own — composition can't rescue an inadequate single gate.
|
||||
Here, each gate has a different *kind* of refusal:
|
||||
|
||||
- **S1 is structural** — it checks shape, not behavior. Cheapest to run.
|
||||
- **S2 is behavioral** — it actually measures what happens if the
|
||||
proposal is admitted, on the live cognition lane. The most expensive
|
||||
gate and the only one that catches regressions you can't predict
|
||||
from shape alone.
|
||||
- **S3 is intentional** — it requires an operator's explicit decision.
|
||||
No bypass; no auto-apply; no scheduled-promote-after-N-hours.
|
||||
|
||||
A proposal that fails any one of these never reaches the next.
|
||||
|
||||
## The synthetic regression in S2
|
||||
|
||||
Scene 2 needs to demonstrate the **rejection lifecycle** deterministically.
|
||||
The public cognition split's test cases happen to test for subject
|
||||
lemmas (e.g. "knowledge", "light") that always appear in the
|
||||
teaching-grounded surface as subjects regardless of an override's
|
||||
connective or object — meaning engineering a real-world regression that
|
||||
fires on today's public split takes a controlled corpus.
|
||||
|
||||
Rather than ship that complexity, S2 uses the **documented** `run_replay=`
|
||||
kwarg on `propose_from_candidate` to inject a controlled `ReplayEvidence`
|
||||
that has the same shape the real gate produces when a real regression
|
||||
is detected. The operator note, log transition, and corpus-byte-identical
|
||||
invariant are all real. In production the real gate emits this same
|
||||
shape; the demo just controls the input so the rejection narrative is
|
||||
deterministic.
|
||||
|
||||
Scenes 1 and 3 both use the real production replay function.
|
||||
|
||||
## Sample run
|
||||
|
||||
```text
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
S1. Eligibility predicate refuses ineligible candidates
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
CLAIM: An undetermined-polarity candidate never enters the proposal log.
|
||||
ProposalError raised; no log row; no replay invocation.
|
||||
|
||||
candidate.polarity : undetermined
|
||||
outcome : ProposalError raised
|
||||
error : polarity must be 'affirms' or 'falsifies'; got
|
||||
'undetermined' — undetermined candidates cannot
|
||||
propose
|
||||
proposal log rows : 0
|
||||
active corpus byte-eq : True
|
||||
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
S2. Replay-equivalence gate auto-rejects a regressing chain
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
CLAIM: An eligible candidate whose append would regress the cognition
|
||||
lane is auto-rejected with the named regressed metrics in the
|
||||
operator note. Active corpus byte-identical pre/post.
|
||||
|
||||
proposal_id : fbd12201819985cb1d3d2f97123c6f0d
|
||||
baseline metrics : {'intent_accuracy': 1.0, 'surface_groundedness':
|
||||
1.0, 'term_capture_rate': 0.9167,
|
||||
'versor_closure_rate': 1.0}
|
||||
candidate metrics : {'intent_accuracy': 1.0, 'surface_groundedness':
|
||||
0.9167, 'term_capture_rate': 0.8334,
|
||||
'versor_closure_rate': 1.0}
|
||||
regressed_metrics : ['surface_groundedness', 'term_capture_rate']
|
||||
replay_equivalent : False
|
||||
state : rejected
|
||||
operator_note : auto_rollback_regression:
|
||||
surface_groundedness,term_capture_rate
|
||||
active corpus byte-eq : True
|
||||
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
S3. Real replay gate runs cognition lane; pass → pending
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
CLAIM: An eligible candidate whose append does not regress reaches
|
||||
'pending' state. Operator --accept is still required to write
|
||||
to the active corpus; the gate is a precondition, not a permission.
|
||||
|
||||
proposal_id : 30585e8e515483c810ad05888e06b572
|
||||
baseline metrics : {'intent_accuracy': 1.0, 'surface_groundedness':
|
||||
1.0, 'term_capture_rate': 0.9167,
|
||||
'versor_closure_rate': 1.0}
|
||||
candidate metrics : {'intent_accuracy': 1.0, 'surface_groundedness':
|
||||
1.0, 'term_capture_rate': 0.9167,
|
||||
'versor_closure_rate': 1.0}
|
||||
regressed_metrics : []
|
||||
replay_equivalent : True
|
||||
state : pending
|
||||
next step : core teaching review 30585e8e515483c810ad05888e06b572
|
||||
--accept --review-date YYYY-MM-DD
|
||||
active corpus byte-eq : True
|
||||
|
||||
════════════════════════════════════════════════════════════════════════
|
||||
RESULT
|
||||
════════════════════════════════════════════════════════════════════════
|
||||
all three gates held : True
|
||||
active corpus byte-eq : True
|
||||
|
||||
Each gate is independent and fails closed. Bad proposals stop at the
|
||||
cheapest applicable gate. The active corpus is never written to
|
||||
anywhere in this demo.
|
||||
```
|
||||
|
||||
## How to reproduce
|
||||
|
||||
```bash
|
||||
core demo anti-regression # human output (preamble + scenes + result)
|
||||
core demo anti-regression --json # machine-readable DemoReport
|
||||
python -m pytest tests/test_anti_regression_demo.py -q # ~10s
|
||||
```
|
||||
|
||||
## Falsifiable claims
|
||||
|
||||
If any of these stops holding, the demo's headline no longer holds:
|
||||
|
||||
- `report.all_gates_held` is `True`.
|
||||
- `report.active_corpus_byte_identical` is `True`.
|
||||
- S1: `outcome="rejected_pre_replay"`, `proposal_id is None`, `replay_evidence is None`.
|
||||
- S2: `review_state="rejected"`, `replay_evidence.replay_equivalent is False`, and the
|
||||
operator note names every regressed metric.
|
||||
- S3: `review_state="pending"`, `replay_evidence.replay_equivalent is True`,
|
||||
`regressed_metrics == []`.
|
||||
|
||||
The contract test file pins all of these.
|
||||
|
||||
## What CORE has that other systems do not
|
||||
|
||||
Continuous pre-training, RLHF, and SFT-pipelines all *can* be regression-
|
||||
aware, but the regression check is implicit in offline evaluation, not
|
||||
gated inline at the point of admission. The proposer and the gate are
|
||||
not separated; rejection is a downstream observability concern, not a
|
||||
guaranteed-fail-closed structural property. CORE's gate is:
|
||||
|
||||
- **Mechanical**, not learned (no policy that can drift).
|
||||
- **Inline**, not offline (every admission runs the full lane).
|
||||
- **Named-metric** (any regression is reported with the specific metric
|
||||
that regressed, not a single aggregate score).
|
||||
- **Byte-identical-corpus** (the production state is never partially
|
||||
mutated mid-decision).
|
||||
|
||||
This is the architecture deployments that care about *what the system
|
||||
will say tomorrow that it would not have said yesterday* need.
|
||||
|
||||
## Related
|
||||
|
||||
- Operator command surface: see the [Inter-Session Memory section in README](../../README.md#inter-session-memory--reviewed-learning).
|
||||
- Learning-loop demo: [`learning_loop_demo.md`](learning_loop_demo.md) — the inverse demo showing the path of a *good* proposal end-to-end.
|
||||
- Determinism benchmark: [`teaching_loop_bench.md`](teaching_loop_bench.md) — N-run byte-identical-artifact proof.
|
||||
22
docs/evals/assets/anti_regression.cast
Normal file
22
docs/evals/assets/anti_regression.cast
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
{"version":3,"term":{"cols":110,"rows":42},"timestamp":1779128036,"command":"core demo anti-regression","env":{"SHELL":"/bin/zsh"}}
|
||||
[0.160, "o", "\r\n================================================================================\r\n Anti-Regression — Three-Gate Defense Against Learning Harm (ADR-0057)\r\n================================================================================\r\n\r\nReference: ADR-0055 (inter-session memory), ADR-0056 (contemplation),\r\nADR-0057 (TeachingChainProposal + replay-equivalence gate).\r\n\r\nWhen a system extends its own knowledge, the gate that decides what to\r\nadmit is the load-bearing part — not the proposer. CORE's reviewed-\r\ncorpus extension path has three independent gates that each must pass\r\nbefore any byte is written to the active teaching corpus:\r\n\r\n S1. Eligibility predicate (mechanical, pre-replay)\r\n Five mechanical checks on candidate shape — polarity in\r\n {affirms, falsifies}, ≥1 source='corpus' evidence pointer,\r\n claim_domain != evaluative (unless --allow-evaluative),\r\n boundary_clean=True, proposed_chain complete.\r\n Ineligible candidates raise ProposalError; they never e"]
|
||||
[0.000, "o", "nter\r\n the proposal log.\r\n\r\n S2. Replay-equivalence gate (mechanical, post-eligibility)\r\n The full cognition lane runs against the active corpus AND\r\n against a transient copy with the proposed chain appended.\r\n Any strict-decrease in a watched metric (intent_accuracy,\r\n surface_groundedness, term_capture_rate, versor_closure_rate)\r\n auto-rejects with the metrics named in the operator note.\r\n Active corpus file bytes byte-identical pre/post.\r\n\r\n S3. Operator review (manual, post-replay)\r\n Even a replay-equivalent proposal only reaches the 'pending'\r\n state. Explicit `core teaching review <id> --accept` is\r\n required to write to the active corpus.\r\n\r\nWhat to expect:\r\n Three scenes, each printed with its CLAIM, candidate, outcome, and\r\n the byte-identical-corpus assertion. Scenes 1 and 3 use the real\r\n replay function; scene 2 injects a controlled replay (via the\r\n documented run_replay= kwarg) to deterministically demonstrate the\r\n auto-r"]
|
||||
[0.000, "o", "ejection lifecycle on a synthetic regression.\r\n\r\nTest gate:\r\n tests/test_anti_regression_demo.py (5 tests — per-scene claim +\r\n active-corpus-byte-identical invariant).\r\n\r\nMachine-readable output:\r\n core demo anti-regression --json\r\n================================================================================\r\n\r\n"]
|
||||
[0.021, "o", "\r\n────────────────────────────────────────────────────────────────────────\r\n S1. Eligibility predicate refuses ineligible candidates\r\n────────────────────────────────────────────────────────────────────────\r\n CLAIM: An undetermined-polarity candidate never enters the proposal log. ProposalError raised; no log row; no replay invocation.\r\n\r\n"]
|
||||
[0.000, "o", " candidate.polarity : undetermined\r\n outcome : ProposalError raised\r\n error : polarity must be 'affirms' or 'falsifies'; got 'undetermined' — undetermined candidates cannot propose\r\n"]
|
||||
[0.000, "o", " proposal log rows : 0\r\n active corpus byte-eq : True\r\n\r\n"]
|
||||
[0.000, "o", "────────────────────────────────────────────────────────────────────────\r\n"]
|
||||
[0.000, "o", " S2. Replay-equivalence gate auto-rejects a regressing chain\r\n"]
|
||||
[0.000, "o", "────────────────────────────────────────────────────────────────────────\r\n"]
|
||||
[0.000, "o", " CLAIM: An eligible candidate whose append would regress the cognition lane is auto-rejected with the named regressed metrics in the operator note. Active corpus byte-identical pre/post.\r\n"]
|
||||
[0.000, "o", "\r\n"]
|
||||
[0.001, "o", " proposal_id : fbd12201819985cb1d3d2f97123c6f0d\r\n baseline metrics : {'intent_accuracy': 1.0, 'surface_groundedness': 1.0, 'term_capture_rate': 0.9167, 'versor_closure_rate': 1.0}\r\n candidate metrics : {'intent_accuracy': 1.0, 'surface_groundedness': 0.9167, 'term_capture_rate': 0.8334, 'versor_closure_rate': 1.0}\r\n regressed_metrics : ['surface_groundedness', 'term_capture_rate']\r\n replay_equivalent : False\r\n"]
|
||||
[0.000, "o", " state : rejected\r\n operator_note : auto_rollback_regression: surface_groundedness,term_capture_rate\r\n"]
|
||||
[0.000, "o", " active corpus byte-eq : True\r\n\r\n"]
|
||||
[0.000, "o", "────────────────────────────────────────────────────────────────────────\r\n"]
|
||||
[0.000, "o", " S3. Real replay gate runs cognition lane; pass → pending\r\n"]
|
||||
[0.000, "o", "────────────────────────────────────────────────────────────────────────\r\n CLAIM: An eligible candidate whose append does not regress reaches 'pending' state. Operator --accept is still required to write to the active corpus; the gate is a precondition, not a permission.\r\n\r\n"]
|
||||
[2.857, "o", " proposal_id : 30585e8e515483c810ad05888e06b572\r\n baseline metrics : {'intent_accuracy': 1.0, 'surface_groundedness': 1.0, 'term_capture_rate': 0.9167, 'versor_closure_rate': 1.0}\r\n candidate metrics : {'intent_accuracy': 1.0, 'surface_groundedness': 1.0, 'term_capture_rate': 0.9167, 'versor_closure_rate': 1.0}\r\n regressed_metrics : []\r\n replay_equivalent : True\r\n state : pending\r\n next step : core teaching review 30585e8e515483c810ad05888e06b572 --accept --review-date YYYY-MM-DD\r\n active corpus byte-eq : True\r\n"]
|
||||
[0.000, "o", "\r\n════════════════════════════════════════════════════════════════════════\r\n RESULT\r\n════════════════════════════════════════════════════════════════════════\r\n all three gates held : True\r\n active corpus byte-eq : True\r\n\r\n Each gate is independent and fails closed. Bad proposals stop at the cheapest applicable gate. The active corpus is never written to anywhere in this demo.\r\n"]
|
||||
[0.000, "o", "\r\n"]
|
||||
[0.015, "x", "0"]
|
||||
BIN
docs/evals/assets/anti_regression.gif
Normal file
BIN
docs/evals/assets/anti_regression.gif
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 118 KiB |
29
docs/evals/assets/learning_loop.cast
Normal file
29
docs/evals/assets/learning_loop.cast
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
{"version":3,"term":{"cols":110,"rows":52},"timestamp":1779128039,"command":"core demo learning-loop","env":{"SHELL":"/bin/zsh"}}
|
||||
[0.178, "o", "\r\n================================================================================\r\n Learning Loop — Cold Turn to Grounded Surface, End-to-End (ADR-0055..0057)\r\n================================================================================\r\n\r\nReference: ADR-0055 (Phase B DiscoveryCandidate emission, Phase A audit\r\n+ provenance), ADR-0056 (Phase C1 contemplation), ADR-0057 (Phase C2\r\nTeachingChainProposal + replay gate + operator review).\r\n\r\nA single deterministic prompt drives every scene:\r\n\r\n \"Why does thought exist?\"\r\n\r\nHeadline claim: CORE, asked a question it cannot ground, emits\r\nstructured evidence that a reviewed chain would have helped. An\r\noperator authors a proposal from that evidence. The replay-\r\nequivalence gate confirms no regression. The operator accepts. The\r\n**same prompt now produces a deterministic teaching-grounded surface**\r\n— replayable, with full provenance back to the operator's accept.\r\n\r\n S1. Cold turn — runtime returns the universal disclosure;\r\n "]
|
||||
[0.000, "o", " grounding_source = none.\r\n S2. Discovery emission — DiscoveryCandidate emitted to the attached\r\n sink; contemplation enriches with pack/\r\n corpus evidence. Active corpus untouched.\r\n S3. Operator proposal — complete chain authored + real replay gate\r\n run + replay_equivalent=True → pending.\r\n S4. Operator accept — accept_proposal writes ONE line to a\r\n transient corpus (copy of active + new\r\n chain). Active corpus byte-identical.\r\n S5. Replay the prompt — _CORPUS_PATH swapped to the transient;\r\n same prompt now teaching-grounded with the\r\n new chain's subject / connective / object.\r\n\r\nTrust boundary:\r\n The demo writes ONLY to a tempdir-scoped transient corpus. The\r\n active teaching corpus on disk is byte-identical pre/post — same\r\n swap pattern the replay-equivalence gate use"]
|
||||
[0.000, "o", "s. No clock-time read.\r\n\r\nWhat to expect:\r\n Per-scene printout with CLAIM, prompt/inputs, outputs, and the\r\n byte-identical-corpus assertion. Final BEFORE / AFTER block shows\r\n the deterministic surface change on the same prompt.\r\n\r\nTest gate:\r\n tests/test_learning_loop_demo.py (7 tests — loop closes, before is\r\n ungrounded, after contains new chain atoms, discovery emits ≥1,\r\n replay gate reports no regression, transient adds exactly 1 line\r\n while active is byte-identical, same prompt drives both surfaces).\r\n\r\nMachine-readable output:\r\n core demo learning-loop --json\r\n================================================================================\r\n"]
|
||||
[0.000, "o", "\r\n"]
|
||||
[1.066, "o", "\r\n────────────────────────────────────────────────────────────────────────\r\n S1. Cold turn — runtime cannot ground the prompt\r\n────────────────────────────────────────────────────────────────────────\r\n CLAIM: Active corpus has no (thought, cause) chain. The runtime falls through to the universal insufficient-grounding disclosure. Identity / safety / ethics gates still run.\r\n\r\n"]
|
||||
[0.005, "o", " prompt : Why does thought exist?\r\n surface : I don't know — insufficient grounding for that yet.\r\n grounding_source : none\r\n discovery candidates : 1 (emitted post-turn)\r\n\r\n────────────────────────────────────────────────────────────────────────\r\n S2. Discovery candidate — structured evidence, not a mutation\r\n────────────────────────────────────────────────────────────────────────\r\n CLAIM: The runtime emits a DiscoveryCandidate (ADR-0055 Phase B) documenting that a reviewed (thought, cause) chain WOULD have grounded this turn. Contemplation (ADR-0056 Phase C1) enriches with pack/corpus evidence pointers. Active corpus is byte-identical — emission writes to the sink on"]
|
||||
[0.000, "o", "ly.\r\n"]
|
||||
[0.000, "o", "\r\n"]
|
||||
[0.000, "o", " candidate_id : 17673a2f15c8da21…\r\n trigger : would_have_grounded\r\n proposed_chain : {'connective': None, 'intent': 'cause', 'object': None, 'subject': 'thought'}\r\n"]
|
||||
[0.000, "o", " polarity : undetermined\r\n"]
|
||||
[0.000, "o", " claim_domain : factual\r\n"]
|
||||
[0.000, "o", " pack_consistent : True\r\n boundary_clean : True\r\n"]
|
||||
[0.000, "o", " evidence (pack-only) : [{'epistemic_status': 'coherent', 'polarity': 'affirms', 'ref': 'thought', 'source': 'pack'}]\r\n\r\n"]
|
||||
[0.000, "o", "────────────────────────────────────────────────────────────────────────\r\n S3. Operator-authored proposal — replay-equivalence gate runs\r\n────────────────────────────────────────────────────────────────────────\r\n CLAIM: From the discovery candidate's evidence, the operator authors a complete chain: thought reveals meaning. Affirming evidence is the existing corpus chain cause_creation_reveals_meaning. The real replay gate (teaching.replay.run_replay_equivalence) runs the cognition public split twice — active corpus vs. transient-with-appended-chain — and reports no regression.\r\n"]
|
||||
[0.000, "o", "\r\n"]
|
||||
[1.849, "o", " proposal_id : 016252428267e4f339969524988c4794\r\n proposed_chain : {'subject': 'thought', 'intent': 'cause', 'connective': 'reveals', 'object': 'meaning'}\r\n evidence (corpus ref) : cause_creation_reveals_meaning\r\n replay baseline : {'intent_accuracy': 1.0, 'surface_groundedness': 1.0, 'term_capture_rate': 0.9167, 'versor_closure_rate': 1.0}\r\n replay candidate : {'intent_accuracy': 1.0, 'surface_groundedness': 1.0, 'term_capture_rate': 0.9167, 'versor_closure_rate': 1.0}\r\n regressed_metrics : []\r\n replay_equivalent : True\r\n state : pending\r\n\r\n"]
|
||||
[0.000, "o", "────────────────────────────────────────────────────────────────────────\r\n S4. Operator accept — transient corpus, active corpus untouched\r\n────────────────────────────────────────────────────────────────────────\r\n CLAIM: accept_proposal writes one JSONL line to a TRANSIENT corpus (copy of active + new chain). The active corpus file bytes are byte-identical pre/post. Provenance on the new entry: adr-0057:discovery_promoted:<review_date>.\r\n"]
|
||||
[0.000, "o", "\r\n"]
|
||||
[0.001, "o", " appended chain_id : cause_thought_reveals_meaning\r\n transient corpus path : /var/folders/kg/5xbm28qd7jl55j7lv3p001f40000gn/T/learning_loop_demo__5wclh7k/cognition_chains_v1.jsonl\r\n transient lines before : 10\r\n transient lines after : 11\r\n active corpus byte-eq : True\r\n\r\n────────────────────────────────────────────────────────────────────────\r\n"]
|
||||
[0.000, "o", " S5. Same prompt — now deterministically teaching-grounded\r\n────────────────────────────────────────────────────────────────────────\r\n"]
|
||||
[0.000, "o", " CLAIM: With the runtime's corpus path swapped to the transient corpus, the same prompt now returns a teaching-grounded surface containing the operator-accepted chain: thought reveals meaning. Identical bytes for any replay of the same prompt against this corpus state.\r\n"]
|
||||
[0.000, "o", "\r\n"]
|
||||
[0.071, "o", " prompt : Why does thought exist?\r\n surface : thought — teaching-grounded (cognition_chains_v1): cognition.thought; logos.internal. thought reveals meaning (cognition.meaning). No session evidence yet.\r\n grounding_source : teaching\r\n"]
|
||||
[0.000, "o", "\r\n════════════════════════════════════════════════════════════════════════\r\n BEFORE / AFTER (single deterministic prompt, one accept between)\r\n════════════════════════════════════════════════════════════════════════\r\n prompt : Why does thought exist?\r\n before : [none] I don't know — insufficient grounding for that yet.\r\n after : [teaching] thought — teaching-grounded (cognition_chains_v1): cognition.thought; logos.internal. thought reveals meaning (cognition.meaning). No session evidence yet.\r\n"]
|
||||
[0.000, "o", "\r\n"]
|
||||
[0.000, "o", " learning_loop_closed : True\r\n"]
|
||||
[0.000, "o", " active corpus byte-identical : True\r\n\r\n"]
|
||||
[0.014, "x", "0"]
|
||||
BIN
docs/evals/assets/learning_loop.gif
Normal file
BIN
docs/evals/assets/learning_loop.gif
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 330 KiB |
6
docs/evals/assets/teaching_loop_bench.cast
Normal file
6
docs/evals/assets/teaching_loop_bench.cast
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
{"version":3,"term":{"cols":100,"rows":50},"timestamp":1779128043,"command":"core bench --suite teaching-loop --runs 10","env":{"SHELL":"/bin/zsh"}}
|
||||
[0.154, "o", "\r\n================================================================================\r\n Teaching-Loop Determinism Benchmark (ADR-0055..0057)\r\n================================================================================\r\n\r\nReference: benchmarks/teaching_loop.py, ADR-0057 (the propose →\r\nreplay → accept pipeline). Pairs naturally with ADR-0045's 100%\r\nexact-NIAH recall numbers — same epistemic class of guarantee,\r\napplied to the *learning loop* rather than only to retrieval.\r\n\r\nFor an identical candidate, the bench runs the full reviewed-corpus\r\nextension pipeline (propose_from_candidate → real run_replay_equivalence\r\n→ accept_proposal) N times against tempdir-scoped paths, and asserts\r\nbyte-identical artifacts every iteration:\r\n\r\n - proposal_id (SHA-256 of canonical-JSON payload)\r\n - replay_baseline (cognition lane metrics on active corpus)\r\n - replay_candidate (cognition lane metrics on transient corpus)\r\n - regressed_metrics (sorted tuple)\r\n - chain_id_written\r\n\r\nAl"]
|
||||
[0.000, "o", "so reports per-iteration wall-time (mean / p50 / p95) and total.\r\n\r\nTrust boundary:\r\n Every write is confined to a tempdir created inside the bench loop.\r\n Active corpus file bytes are byte-identical pre/post regardless of\r\n N. Asserted in the bench report and re-pinned in the test.\r\n\r\n100-run reference result on today's main:\r\n unique(proposal_id) = 1 unique(chain_id) = 1\r\n unique(baseline) = 1 unique(candidate) = 1\r\n active_corpus_byte_eq = True\r\n mean = 1.85s p50 = 1.84s p95 = 1.85s\r\n\r\nTest gate:\r\n tests/test_teaching_loop_bench.py (5 tests — determinism at small N,\r\n proposal_id SHA-256 shape, canonical chain_id layout, latency stats\r\n well-formed, JSON serialisation).\r\n\r\nUsage:\r\n core bench --suite teaching-loop --runs 100\r\n core bench --suite teaching-loop --runs 10 --json\r\n================================================================================\r\n"]
|
||||
[0.000, "o", "\r\n"]
|
||||
[19.411, "o", " [PASS] teaching_loop_determinism 1.0000 byte_identity_ratio\r\n 10 runs; unique(proposal_id)=1, unique(baseline)=1, unique(candidate)=1, unique(chain_id)=1; mean=1.937s p50=1.836s p95=2.393s; active_corpus_byte_eq=True\r\n\r\nALL PASSED\r\n"]
|
||||
[0.015, "x", "0"]
|
||||
BIN
docs/evals/assets/teaching_loop_bench.gif
Normal file
BIN
docs/evals/assets/teaching_loop_bench.gif
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 61 KiB |
178
docs/evals/learning_loop_demo.md
Normal file
178
docs/evals/learning_loop_demo.md
Normal file
|
|
@ -0,0 +1,178 @@
|
|||
# Learning-Loop Demo — Cold Turn to Grounded Surface, End-to-End
|
||||
|
||||
**Date:** 2026-05-18
|
||||
**Runner:** `evals/learning_loop/run_demo.py`
|
||||
**CLI:** `core demo learning-loop` (`--json` for machine-readable output)
|
||||
**Contract tests:** `tests/test_learning_loop_demo.py` (7 passing)
|
||||
**Reference ADRs:** [0055](../decisions/ADR-0055-inter-session-memory-discovery-promotion.md), [0056](../decisions/ADR-0056-contemplation-loop-c1.md), [0057](../decisions/ADR-0057-teaching-chain-proposal-review.md)
|
||||
|
||||
|
||||
|
||||
## Headline claim
|
||||
|
||||
> A single deterministic prompt, `"Why does thought exist?"`, produces:
|
||||
>
|
||||
> - **Before** the loop runs: `[none] I don't know — insufficient grounding for that yet.`
|
||||
> - **After** one operator accept: `[teaching] thought — teaching-grounded (cognition_chains_v1): cognition.thought; logos.internal. thought reveals meaning (cognition.meaning). No session evidence yet.`
|
||||
>
|
||||
> The active corpus on disk is byte-identical pre/post. The change lives entirely in a transient corpus the demo writes to and then swaps the runtime's `_CORPUS_PATH` to — the same pattern the replay-equivalence gate uses.
|
||||
|
||||
## What CORE has that other systems do not
|
||||
|
||||
| Property | Continuous pre-training / RLHF | CORE learning loop |
|
||||
|---|---|---|
|
||||
| **Per-fact provenance** | None (gradient updates are diffuse) | `Provenance(adr_id, source, review_date, raw)` on every appended chain |
|
||||
| **Replay-equivalence guarantee** | Offline eval at checkpoint cadence | Inline gate runs the full cognition lane on every admission |
|
||||
| **Audit trail** | Training logs | `ProposalLog` events: `created` → `replay` → `transition` → `accepted_corpus_append` |
|
||||
| **Replayable across runs** | No (stochastic; weight checkpoints diverge) | SHA-256 deterministic `proposal_id`; bit-identical artifacts (see [`teaching_loop_bench.md`](teaching_loop_bench.md)) |
|
||||
| **Operator gate** | Implicit (deployment cadence) | Explicit `core teaching review <id> --accept --review-date YYYY-MM-DD` |
|
||||
| **Roll-back semantics** | Restore checkpoint | `core teaching supersede <chain_id>` (append-only at disk; active view derived) |
|
||||
|
||||
This is the architecture deployments that need to answer *"why did the
|
||||
system say this today that it would not have said yesterday?"* require.
|
||||
|
||||
## Trust boundary
|
||||
|
||||
The demo writes only to a tempdir-scoped transient corpus. The active
|
||||
teaching corpus on disk is byte-identical pre/post. The swap pattern:
|
||||
|
||||
```python
|
||||
real_path = _tg._CORPUS_PATH
|
||||
try:
|
||||
_tg._CORPUS_PATH = transient
|
||||
_tg._corpus_index.cache_clear()
|
||||
rt2 = ChatRuntime()
|
||||
response = rt2.chat("Why does thought exist?")
|
||||
finally:
|
||||
_tg._CORPUS_PATH = real_path
|
||||
_tg._corpus_index.cache_clear()
|
||||
```
|
||||
|
||||
This is the same mechanism `teaching/replay.py:_swap_corpus_path` uses
|
||||
during the replay-equivalence gate. No clock-time read anywhere in
|
||||
the loop.
|
||||
|
||||
## Five scenes
|
||||
|
||||
| Scene | What runs | Trust property |
|
||||
|---|---|---|
|
||||
| **S1. Cold turn** | Real `ChatRuntime.chat("Why does thought exist?")` | No `(thought, cause)` chain exists → universal disclosure; `grounding_source=none`. |
|
||||
| **S2. Discovery emission** | Discovery sink + contemplation enrich the candidate | Active corpus untouched; emission is sink-only. |
|
||||
| **S3. Operator proposal** | `propose_from_candidate()` runs real `run_replay_equivalence()` | Cognition lane runs twice; no regression → `state=pending`. |
|
||||
| **S4. Operator accept** | `accept_proposal()` against a **transient** corpus path | Active corpus byte-identical; transient gains exactly 1 line; provenance `adr-0057:discovery_promoted:2026-05-18`. |
|
||||
| **S5. Replay** | `_CORPUS_PATH` swapped to transient; fresh `ChatRuntime` runs the same prompt | Surface contains subject / humanised connective / object; `grounding_source=teaching`. |
|
||||
|
||||
## Sample run
|
||||
|
||||
```text
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
S1. Cold turn — runtime cannot ground the prompt
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
prompt : Why does thought exist?
|
||||
surface : I don't know — insufficient grounding for that yet.
|
||||
grounding_source : none
|
||||
discovery candidates : 1 (emitted post-turn)
|
||||
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
S2. Discovery candidate — structured evidence, not a mutation
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
candidate_id : 17673a2f15c8da21…
|
||||
trigger : would_have_grounded
|
||||
proposed_chain : {'connective': None, 'intent': 'cause',
|
||||
'object': None, 'subject': 'thought'}
|
||||
polarity : undetermined
|
||||
claim_domain : factual
|
||||
pack_consistent : True
|
||||
boundary_clean : True
|
||||
evidence (pack-only) : [{'epistemic_status': 'coherent',
|
||||
'polarity': 'affirms', 'ref': 'thought',
|
||||
'source': 'pack'}]
|
||||
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
S3. Operator-authored proposal — replay-equivalence gate runs
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
proposal_id : 016252428267e4f339969524988c4794
|
||||
proposed_chain : {'subject': 'thought', 'intent': 'cause',
|
||||
'connective': 'reveals', 'object': 'meaning'}
|
||||
evidence (corpus ref) : cause_creation_reveals_meaning
|
||||
replay baseline : {'intent_accuracy': 1.0, 'surface_groundedness':
|
||||
1.0, 'term_capture_rate': 0.9167,
|
||||
'versor_closure_rate': 1.0}
|
||||
replay candidate : {'intent_accuracy': 1.0, 'surface_groundedness':
|
||||
1.0, 'term_capture_rate': 0.9167,
|
||||
'versor_closure_rate': 1.0}
|
||||
regressed_metrics : []
|
||||
replay_equivalent : True
|
||||
state : pending
|
||||
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
S4. Operator accept — transient corpus, active corpus untouched
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
appended chain_id : cause_thought_reveals_meaning
|
||||
transient corpus path : /tmp/learning_loop_demo_xxxxxx/cognition_chains_v1.jsonl
|
||||
transient lines before : 10
|
||||
transient lines after : 11
|
||||
active corpus byte-eq : True
|
||||
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
S5. Same prompt — now deterministically teaching-grounded
|
||||
────────────────────────────────────────────────────────────────────────
|
||||
prompt : Why does thought exist?
|
||||
surface : thought — teaching-grounded (cognition_chains_v1):
|
||||
cognition.thought; logos.internal.
|
||||
thought reveals meaning (cognition.meaning).
|
||||
No session evidence yet.
|
||||
grounding_source : teaching
|
||||
|
||||
════════════════════════════════════════════════════════════════════════
|
||||
BEFORE / AFTER (single deterministic prompt, one accept between)
|
||||
════════════════════════════════════════════════════════════════════════
|
||||
prompt : Why does thought exist?
|
||||
before : [none] I don't know — insufficient grounding for that yet.
|
||||
after : [teaching] thought — teaching-grounded (cognition_chains_v1):
|
||||
cognition.thought; logos.internal.
|
||||
thought reveals meaning (cognition.meaning).
|
||||
No session evidence yet.
|
||||
|
||||
learning_loop_closed : True
|
||||
active corpus byte-identical : True
|
||||
```
|
||||
|
||||
## How to reproduce
|
||||
|
||||
```bash
|
||||
core demo learning-loop # human output (preamble + scenes + before/after)
|
||||
core demo learning-loop --json # machine-readable DemoReport
|
||||
python -m pytest tests/test_learning_loop_demo.py -q # ~15s
|
||||
```
|
||||
|
||||
## Falsifiable claims
|
||||
|
||||
If any of these stops holding, the headline claim no longer holds:
|
||||
|
||||
- `report.learning_loop_closed` is `True`.
|
||||
- `report.active_corpus_byte_identical` is `True`.
|
||||
- `report.before.grounding_source == "none"`; surface contains `"insufficient grounding"`.
|
||||
- `report.after.grounding_source == "teaching"`; surface contains `"thought"` AND `"reveal"` AND `"meaning"` AND `"teaching-grounded"`.
|
||||
- S3: `replay_evidence.replay_equivalent is True`, `regressed_metrics == []`, `state == "pending"`.
|
||||
- S4: `transient_lines_after == transient_lines_before + 1` AND `active_corpus_byte_identical is True`.
|
||||
- The same prompt drives both surfaces (`report.prompt == "Why does thought exist?"`).
|
||||
|
||||
## Why "thought" is the demo subject
|
||||
|
||||
The subject must satisfy three pre-conditions for the demo to fire deterministically:
|
||||
|
||||
1. **Pack-resident** (otherwise the discovery candidate isn't emitted) — confirmed by `'thought' in _pack_index()`.
|
||||
2. **No active `(thought, cause)` chain** (otherwise the cold turn would already be teaching-grounded) — confirmed by the active corpus snapshot.
|
||||
3. **Intent classifier picks `CAUSE` on a natural prompt** — `"Why does thought exist?"` classifies as `CAUSE / subject="thought"` deterministically.
|
||||
|
||||
The operator-authored chain (`thought reveals meaning`) cites
|
||||
`cause_creation_reveals_meaning` as affirming evidence. Both endpoint
|
||||
lemmas (`thought`, `meaning`) are pack-resident; the connective
|
||||
`reveals` is in the canonical predicate set.
|
||||
|
||||
## Related
|
||||
|
||||
- Anti-regression demo: [`anti_regression_demo.md`](anti_regression_demo.md) — the inverse demo showing each gate refusing a bad proposal.
|
||||
- Determinism benchmark: [`teaching_loop_bench.md`](teaching_loop_bench.md) — N-run byte-identical-artifact proof on this exact pipeline.
|
||||
- Operator surface: see the [Inter-Session Memory section in README](../../README.md#inter-session-memory--reviewed-learning).
|
||||
137
docs/evals/teaching_loop_bench.md
Normal file
137
docs/evals/teaching_loop_bench.md
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
# Teaching-Loop Determinism Benchmark
|
||||
|
||||
**Date:** 2026-05-18
|
||||
**Runner:** `benchmarks/teaching_loop.py`
|
||||
**CLI:** `core bench --suite teaching-loop [--runs N] [--json]`
|
||||
**Contract tests:** `tests/test_teaching_loop_bench.py` (5 passing)
|
||||
**Reference ADRs:** [0055](../decisions/ADR-0055-inter-session-memory-discovery-promotion.md), [0057](../decisions/ADR-0057-teaching-chain-proposal-review.md), [0045](../decisions/ADR-0045-long-context-recall-vs-transformer-baselines.md)
|
||||
|
||||
|
||||
|
||||
## Headline claim
|
||||
|
||||
> For an identical candidate, N runs of the full reviewed-corpus
|
||||
> extension pipeline (`propose_from_candidate` → real
|
||||
> `run_replay_equivalence` → `accept_proposal`) produce N
|
||||
> **byte-identical** artifacts at every observable point.
|
||||
>
|
||||
> The active teaching corpus on disk is byte-identical pre/post,
|
||||
> regardless of N.
|
||||
|
||||
This is the determinism guarantee for the *learning loop itself* —
|
||||
the analog of [ADR-0045's 100% exact-NIAH recall](../decisions/ADR-0045-long-context-recall-vs-transformer-baselines.md)
|
||||
result, applied to the learning path rather than only to retrieval.
|
||||
|
||||
## What's asserted byte-identical
|
||||
|
||||
| Artifact | How it's derived | Why this matters |
|
||||
|---|---|---|
|
||||
| `proposal_id` | SHA-256 prefix of canonical-JSON `(source_candidate_id, proposed_chain)` | If hashing of inputs ever drifts (locale, dict-ordering, float formatting), this changes. |
|
||||
| `replay_baseline` | Cognition lane metrics on the active corpus | If any cognition-lane component became non-deterministic, this varies across runs. |
|
||||
| `replay_candidate` | Cognition lane metrics on transient-with-append corpus | Same as above, run against a different corpus state. |
|
||||
| `regressed_metrics` | Sorted tuple of strict-decrease metric names | A 1-element drift would expose comparison non-determinism. |
|
||||
| `chain_id_written` | Canonical `<intent>_<subject>_<connective>_<object>` | Append-side identifier derivation. |
|
||||
|
||||
If determinism breaks anywhere in the pipeline — proposal hashing, the
|
||||
replay-equivalence gate, accept-side corpus-append, or `ProposalLog`
|
||||
replay — at least one of the `unique_*` counts exceeds 1 and the bench
|
||||
fails.
|
||||
|
||||
## 100-run reference result (today's main)
|
||||
|
||||
```text
|
||||
unique(proposal_id) = 1 unique(chain_id) = 1
|
||||
unique(baseline) = 1 unique(candidate) = 1
|
||||
unique(regressed_metrics) = 1
|
||||
active_corpus_byte_eq = True
|
||||
|
||||
Latency per iteration:
|
||||
mean = 1.849s p50 = 1.838s p95 = 1.851s total = ~185s
|
||||
```
|
||||
|
||||
The p95 sits within 1% of the p50 — the loop's per-iteration cost is
|
||||
dominated by the two cognition-lane runs inside the replay gate, both
|
||||
of which are themselves deterministic in time as well as output.
|
||||
|
||||
## Sample 10-run output
|
||||
|
||||
```text
|
||||
================================================================================
|
||||
Teaching-Loop Determinism Benchmark (ADR-0055..0057)
|
||||
================================================================================
|
||||
...
|
||||
[PASS] teaching_loop_determinism 1.0000 byte_identity_ratio
|
||||
10 runs; unique(proposal_id)=1, unique(baseline)=1,
|
||||
unique(candidate)=1, unique(chain_id)=1;
|
||||
mean=1.948s p50=1.846s p95=2.406s; active_corpus_byte_eq=True
|
||||
|
||||
ALL PASSED
|
||||
```
|
||||
|
||||
(p95 in any single 10-run sample is noisier than the 100-run number — a
|
||||
single warm-cache vs cold-cache iteration can move it ~30%. The 100-run
|
||||
distribution is the canonical reference.)
|
||||
|
||||
## Trust boundary
|
||||
|
||||
Every write is confined to a tempdir created inside the bench loop:
|
||||
|
||||
```python
|
||||
for _ in range(runs):
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
log_path = Path(tmpdir) / "proposals.jsonl"
|
||||
transient = Path(tmpdir) / "cognition_chains_v1.jsonl"
|
||||
shutil.copyfile(active_path, transient)
|
||||
...
|
||||
```
|
||||
|
||||
The active corpus is read at the start and at the end. Any byte
|
||||
difference would fail the bench. Re-pinned by
|
||||
`test_teaching_loop_is_deterministic_across_three_runs` in
|
||||
`tests/test_teaching_loop_bench.py`.
|
||||
|
||||
## How to reproduce
|
||||
|
||||
```bash
|
||||
core bench --suite teaching-loop --runs 100 # canonical reference run
|
||||
core bench --suite teaching-loop --runs 10 # quick smoke (~20s)
|
||||
core bench --suite teaching-loop --runs 100 --json # machine-readable
|
||||
|
||||
python -m pytest tests/test_teaching_loop_bench.py -q # ~25s
|
||||
```
|
||||
|
||||
## Falsifiable claims
|
||||
|
||||
If any of these stops holding, the headline claim no longer holds:
|
||||
|
||||
- `report.deterministic` is `True` (all five `unique_*` counts are 1).
|
||||
- `report.active_corpus_byte_identical` is `True`.
|
||||
- `report.sample_proposal_id` is 32 lowercase hex chars (SHA-256 prefix).
|
||||
- `report.sample_chain_id == "cause_thought_reveals_meaning"`.
|
||||
- `report.elapsed_p95_s >= report.elapsed_p50_s`.
|
||||
- `report.elapsed_total_s >= mean × runs × 0.9` (sanity check on wall-time accounting).
|
||||
|
||||
The contract test file pins all of these at low N for fast CI; the
|
||||
100-run reference number is informational, not gated.
|
||||
|
||||
## Why this pairs with ADR-0045
|
||||
|
||||
[ADR-0045](../decisions/ADR-0045-long-context-recall-vs-transformer-baselines.md) showed
|
||||
CORE achieves **100% exact recall** at N ∈ {100, 1k, 10k, 100k} in a
|
||||
needle-in-a-haystack scan — the *retrieval* path is bit-exact.
|
||||
|
||||
This benchmark shows the **learning path** is also bit-exact: the same
|
||||
candidate, run N times, produces the same accepted chain. Together
|
||||
they form the two halves of the deterministic-cognition claim:
|
||||
|
||||
- **Read path** (ADR-0045): exact, scale-invariant, no approximation.
|
||||
- **Write path** (this bench): exact, replayable, no non-determinism.
|
||||
|
||||
No LLM-based system has published equivalent numbers on either path,
|
||||
let alone both.
|
||||
|
||||
## Related
|
||||
|
||||
- Anti-regression demo: [`anti_regression_demo.md`](anti_regression_demo.md) — what the gate does when a regression *is* detected.
|
||||
- Learning-loop demo: [`learning_loop_demo.md`](learning_loop_demo.md) — the same pipeline as a narrative walkthrough.
|
||||
- Long-context comparison: [ADR-0045 / `long-context-comparison`](../decisions/ADR-0045-long-context-recall-vs-transformer-baselines.md) — the sibling determinism number for the *read* path.
|
||||
Loading…
Reference in a new issue