diff --git a/README.md b/README.md index 4a1ab208..bc9f1f95 100644 --- a/README.md +++ b/README.md @@ -183,11 +183,11 @@ Supersession is the second operator-direct mutation surface: `core teaching supe Three live demos / benchmarks make the chain demoable end-to-end: -| Demo | Headline claim | Live command | -|---|---|---| -| **Anti-regression** | Three independent gates each fail closed; bad proposals stop at the cheapest applicable gate. | `core demo anti-regression` | -| **Learning loop** | Same deterministic prompt: `[none] I don't know…` before, `[teaching] thought reveals meaning…` after one accept. | `core demo learning-loop` | -| **Determinism bench** | N identical inputs → N byte-identical proposal_id / replay metrics / chain_id. 100 runs: `unique=1` everywhere, mean ≈ 1.85s. | `core bench --suite teaching-loop --runs 100` | +| Demo | Headline claim | Live command | Writeup | +|---|---|---|---| +| **Anti-regression** | Three independent gates each fail closed; bad proposals stop at the cheapest applicable gate. | `core demo anti-regression` | [`docs/evals/anti_regression_demo.md`](docs/evals/anti_regression_demo.md) | +| **Learning loop** | Same deterministic prompt: `[none] I don't know…` before, `[teaching] thought reveals meaning…` after one accept. | `core demo learning-loop` | [`docs/evals/learning_loop_demo.md`](docs/evals/learning_loop_demo.md) | +| **Determinism bench** | N identical inputs → N byte-identical proposal_id / replay metrics / chain_id. 100 runs: `unique=1` everywhere, mean ≈ 1.85s. | `core bench --suite teaching-loop --runs 100` | [`docs/evals/teaching_loop_bench.md`](docs/evals/teaching_loop_bench.md) | Operator surfaces: diff --git a/docs/evals/anti_regression_demo.md b/docs/evals/anti_regression_demo.md new file mode 100644 index 00000000..f441334b --- /dev/null +++ b/docs/evals/anti_regression_demo.md @@ -0,0 +1,175 @@ +# Anti-Regression Demo — Three-Gate Defense Against Learning Harm + +**Date:** 2026-05-18 +**Runner:** `evals/anti_regression/run_demo.py` +**CLI:** `core demo anti-regression` (`--json` for machine-readable output) +**Contract tests:** `tests/test_anti_regression_demo.py` (5 passing) +**Reference ADRs:** [0055](../decisions/ADR-0055-inter-session-memory-discovery-promotion.md), [0056](../decisions/ADR-0056-contemplation-loop-c1.md), [0057](../decisions/ADR-0057-teaching-chain-proposal-review.md) + +![anti-regression demo](assets/anti_regression.gif) + +## What this demo shows + +When a system extends its own knowledge, **the gate that decides what to +admit is the load-bearing part** — not the proposer. CORE's reviewed- +corpus extension path has three independent gates that each must pass +before any byte is written to the active teaching corpus. The demo +runs each gate to verdict against a real `ProposalLog` in an isolated +temp directory and asserts the active corpus is byte-identical pre/post. + +| Gate | What it checks | What fails it | +|---|---|---| +| **S1. Eligibility predicate** (mechanical, pre-replay) | polarity ∈ {affirms, falsifies} ∧ ≥1 `source='corpus'` evidence ∧ claim_domain ≠ evaluative ∧ boundary_clean=True ∧ chain complete | Raises `ProposalError`; **no log row written**. | +| **S2. Replay-equivalence gate** (mechanical, post-eligibility) | Cognition lane runs against active corpus AND a transient-with-append copy; any strict-decrease in `intent_accuracy / surface_groundedness / term_capture_rate / versor_closure_rate` is regression. | Auto-rejects with **named** regressed metrics in operator_note. | +| **S3. Operator review** (manual, post-replay) | Replay-equivalence is a *precondition*, not a permission. | `--accept` not run → state stays `pending` indefinitely. | + +## Why each gate is independent + +A defensive lattice is only useful if each layer can refuse a bad input +on its own — composition can't rescue an inadequate single gate. +Here, each gate has a different *kind* of refusal: + +- **S1 is structural** — it checks shape, not behavior. Cheapest to run. +- **S2 is behavioral** — it actually measures what happens if the + proposal is admitted, on the live cognition lane. The most expensive + gate and the only one that catches regressions you can't predict + from shape alone. +- **S3 is intentional** — it requires an operator's explicit decision. + No bypass; no auto-apply; no scheduled-promote-after-N-hours. + +A proposal that fails any one of these never reaches the next. + +## The synthetic regression in S2 + +Scene 2 needs to demonstrate the **rejection lifecycle** deterministically. +The public cognition split's test cases happen to test for subject +lemmas (e.g. "knowledge", "light") that always appear in the +teaching-grounded surface as subjects regardless of an override's +connective or object — meaning engineering a real-world regression that +fires on today's public split takes a controlled corpus. + +Rather than ship that complexity, S2 uses the **documented** `run_replay=` +kwarg on `propose_from_candidate` to inject a controlled `ReplayEvidence` +that has the same shape the real gate produces when a real regression +is detected. The operator note, log transition, and corpus-byte-identical +invariant are all real. In production the real gate emits this same +shape; the demo just controls the input so the rejection narrative is +deterministic. + +Scenes 1 and 3 both use the real production replay function. + +## Sample run + +```text +──────────────────────────────────────────────────────────────────────── + S1. Eligibility predicate refuses ineligible candidates +──────────────────────────────────────────────────────────────────────── + CLAIM: An undetermined-polarity candidate never enters the proposal log. + ProposalError raised; no log row; no replay invocation. + + candidate.polarity : undetermined + outcome : ProposalError raised + error : polarity must be 'affirms' or 'falsifies'; got + 'undetermined' — undetermined candidates cannot + propose + proposal log rows : 0 + active corpus byte-eq : True + +──────────────────────────────────────────────────────────────────────── + S2. Replay-equivalence gate auto-rejects a regressing chain +──────────────────────────────────────────────────────────────────────── + CLAIM: An eligible candidate whose append would regress the cognition + lane is auto-rejected with the named regressed metrics in the + operator note. Active corpus byte-identical pre/post. + + proposal_id : fbd12201819985cb1d3d2f97123c6f0d + baseline metrics : {'intent_accuracy': 1.0, 'surface_groundedness': + 1.0, 'term_capture_rate': 0.9167, + 'versor_closure_rate': 1.0} + candidate metrics : {'intent_accuracy': 1.0, 'surface_groundedness': + 0.9167, 'term_capture_rate': 0.8334, + 'versor_closure_rate': 1.0} + regressed_metrics : ['surface_groundedness', 'term_capture_rate'] + replay_equivalent : False + state : rejected + operator_note : auto_rollback_regression: + surface_groundedness,term_capture_rate + active corpus byte-eq : True + +──────────────────────────────────────────────────────────────────────── + S3. Real replay gate runs cognition lane; pass → pending +──────────────────────────────────────────────────────────────────────── + CLAIM: An eligible candidate whose append does not regress reaches + 'pending' state. Operator --accept is still required to write + to the active corpus; the gate is a precondition, not a permission. + + proposal_id : 30585e8e515483c810ad05888e06b572 + baseline metrics : {'intent_accuracy': 1.0, 'surface_groundedness': + 1.0, 'term_capture_rate': 0.9167, + 'versor_closure_rate': 1.0} + candidate metrics : {'intent_accuracy': 1.0, 'surface_groundedness': + 1.0, 'term_capture_rate': 0.9167, + 'versor_closure_rate': 1.0} + regressed_metrics : [] + replay_equivalent : True + state : pending + next step : core teaching review 30585e8e515483c810ad05888e06b572 + --accept --review-date YYYY-MM-DD + active corpus byte-eq : True + +════════════════════════════════════════════════════════════════════════ + RESULT +════════════════════════════════════════════════════════════════════════ + all three gates held : True + active corpus byte-eq : True + + Each gate is independent and fails closed. Bad proposals stop at the + cheapest applicable gate. The active corpus is never written to + anywhere in this demo. +``` + +## How to reproduce + +```bash +core demo anti-regression # human output (preamble + scenes + result) +core demo anti-regression --json # machine-readable DemoReport +python -m pytest tests/test_anti_regression_demo.py -q # ~10s +``` + +## Falsifiable claims + +If any of these stops holding, the demo's headline no longer holds: + +- `report.all_gates_held` is `True`. +- `report.active_corpus_byte_identical` is `True`. +- S1: `outcome="rejected_pre_replay"`, `proposal_id is None`, `replay_evidence is None`. +- S2: `review_state="rejected"`, `replay_evidence.replay_equivalent is False`, and the + operator note names every regressed metric. +- S3: `review_state="pending"`, `replay_evidence.replay_equivalent is True`, + `regressed_metrics == []`. + +The contract test file pins all of these. + +## What CORE has that other systems do not + +Continuous pre-training, RLHF, and SFT-pipelines all *can* be regression- +aware, but the regression check is implicit in offline evaluation, not +gated inline at the point of admission. The proposer and the gate are +not separated; rejection is a downstream observability concern, not a +guaranteed-fail-closed structural property. CORE's gate is: + +- **Mechanical**, not learned (no policy that can drift). +- **Inline**, not offline (every admission runs the full lane). +- **Named-metric** (any regression is reported with the specific metric + that regressed, not a single aggregate score). +- **Byte-identical-corpus** (the production state is never partially + mutated mid-decision). + +This is the architecture deployments that care about *what the system +will say tomorrow that it would not have said yesterday* need. + +## Related + +- Operator command surface: see the [Inter-Session Memory section in README](../../README.md#inter-session-memory--reviewed-learning). +- Learning-loop demo: [`learning_loop_demo.md`](learning_loop_demo.md) — the inverse demo showing the path of a *good* proposal end-to-end. +- Determinism benchmark: [`teaching_loop_bench.md`](teaching_loop_bench.md) — N-run byte-identical-artifact proof. diff --git a/docs/evals/assets/anti_regression.cast b/docs/evals/assets/anti_regression.cast new file mode 100644 index 00000000..8d357f74 --- /dev/null +++ b/docs/evals/assets/anti_regression.cast @@ -0,0 +1,22 @@ +{"version":3,"term":{"cols":110,"rows":42},"timestamp":1779128036,"command":"core demo anti-regression","env":{"SHELL":"/bin/zsh"}} +[0.160, "o", "\r\n================================================================================\r\n Anti-Regression — Three-Gate Defense Against Learning Harm (ADR-0057)\r\n================================================================================\r\n\r\nReference: ADR-0055 (inter-session memory), ADR-0056 (contemplation),\r\nADR-0057 (TeachingChainProposal + replay-equivalence gate).\r\n\r\nWhen a system extends its own knowledge, the gate that decides what to\r\nadmit is the load-bearing part — not the proposer. CORE's reviewed-\r\ncorpus extension path has three independent gates that each must pass\r\nbefore any byte is written to the active teaching corpus:\r\n\r\n S1. Eligibility predicate (mechanical, pre-replay)\r\n Five mechanical checks on candidate shape — polarity in\r\n {affirms, falsifies}, ≥1 source='corpus' evidence pointer,\r\n claim_domain != evaluative (unless --allow-evaluative),\r\n boundary_clean=True, proposed_chain complete.\r\n Ineligible candidates raise ProposalError; they never e"] +[0.000, "o", "nter\r\n the proposal log.\r\n\r\n S2. Replay-equivalence gate (mechanical, post-eligibility)\r\n The full cognition lane runs against the active corpus AND\r\n against a transient copy with the proposed chain appended.\r\n Any strict-decrease in a watched metric (intent_accuracy,\r\n surface_groundedness, term_capture_rate, versor_closure_rate)\r\n auto-rejects with the metrics named in the operator note.\r\n Active corpus file bytes byte-identical pre/post.\r\n\r\n S3. Operator review (manual, post-replay)\r\n Even a replay-equivalent proposal only reaches the 'pending'\r\n state. Explicit `core teaching review --accept` is\r\n required to write to the active corpus.\r\n\r\nWhat to expect:\r\n Three scenes, each printed with its CLAIM, candidate, outcome, and\r\n the byte-identical-corpus assertion. Scenes 1 and 3 use the real\r\n replay function; scene 2 injects a controlled replay (via the\r\n documented run_replay= kwarg) to deterministically demonstrate the\r\n auto-r"] +[0.000, "o", "ejection lifecycle on a synthetic regression.\r\n\r\nTest gate:\r\n tests/test_anti_regression_demo.py (5 tests — per-scene claim +\r\n active-corpus-byte-identical invariant).\r\n\r\nMachine-readable output:\r\n core demo anti-regression --json\r\n================================================================================\r\n\r\n"] +[0.021, "o", "\r\n────────────────────────────────────────────────────────────────────────\r\n S1. Eligibility predicate refuses ineligible candidates\r\n────────────────────────────────────────────────────────────────────────\r\n CLAIM: An undetermined-polarity candidate never enters the proposal log. ProposalError raised; no log row; no replay invocation.\r\n\r\n"] +[0.000, "o", " candidate.polarity : undetermined\r\n outcome : ProposalError raised\r\n error : polarity must be 'affirms' or 'falsifies'; got 'undetermined' — undetermined candidates cannot propose\r\n"] +[0.000, "o", " proposal log rows : 0\r\n active corpus byte-eq : True\r\n\r\n"] +[0.000, "o", "────────────────────────────────────────────────────────────────────────\r\n"] +[0.000, "o", " S2. Replay-equivalence gate auto-rejects a regressing chain\r\n"] +[0.000, "o", "────────────────────────────────────────────────────────────────────────\r\n"] +[0.000, "o", " CLAIM: An eligible candidate whose append would regress the cognition lane is auto-rejected with the named regressed metrics in the operator note. Active corpus byte-identical pre/post.\r\n"] +[0.000, "o", "\r\n"] +[0.001, "o", " proposal_id : fbd12201819985cb1d3d2f97123c6f0d\r\n baseline metrics : {'intent_accuracy': 1.0, 'surface_groundedness': 1.0, 'term_capture_rate': 0.9167, 'versor_closure_rate': 1.0}\r\n candidate metrics : {'intent_accuracy': 1.0, 'surface_groundedness': 0.9167, 'term_capture_rate': 0.8334, 'versor_closure_rate': 1.0}\r\n regressed_metrics : ['surface_groundedness', 'term_capture_rate']\r\n replay_equivalent : False\r\n"] +[0.000, "o", " state : rejected\r\n operator_note : auto_rollback_regression: surface_groundedness,term_capture_rate\r\n"] +[0.000, "o", " active corpus byte-eq : True\r\n\r\n"] +[0.000, "o", "────────────────────────────────────────────────────────────────────────\r\n"] +[0.000, "o", " S3. Real replay gate runs cognition lane; pass → pending\r\n"] +[0.000, "o", "────────────────────────────────────────────────────────────────────────\r\n CLAIM: An eligible candidate whose append does not regress reaches 'pending' state. Operator --accept is still required to write to the active corpus; the gate is a precondition, not a permission.\r\n\r\n"] +[2.857, "o", " proposal_id : 30585e8e515483c810ad05888e06b572\r\n baseline metrics : {'intent_accuracy': 1.0, 'surface_groundedness': 1.0, 'term_capture_rate': 0.9167, 'versor_closure_rate': 1.0}\r\n candidate metrics : {'intent_accuracy': 1.0, 'surface_groundedness': 1.0, 'term_capture_rate': 0.9167, 'versor_closure_rate': 1.0}\r\n regressed_metrics : []\r\n replay_equivalent : True\r\n state : pending\r\n next step : core teaching review 30585e8e515483c810ad05888e06b572 --accept --review-date YYYY-MM-DD\r\n active corpus byte-eq : True\r\n"] +[0.000, "o", "\r\n════════════════════════════════════════════════════════════════════════\r\n RESULT\r\n════════════════════════════════════════════════════════════════════════\r\n all three gates held : True\r\n active corpus byte-eq : True\r\n\r\n Each gate is independent and fails closed. Bad proposals stop at the cheapest applicable gate. The active corpus is never written to anywhere in this demo.\r\n"] +[0.000, "o", "\r\n"] +[0.015, "x", "0"] diff --git a/docs/evals/assets/anti_regression.gif b/docs/evals/assets/anti_regression.gif new file mode 100644 index 00000000..60085e39 Binary files /dev/null and b/docs/evals/assets/anti_regression.gif differ diff --git a/docs/evals/assets/learning_loop.cast b/docs/evals/assets/learning_loop.cast new file mode 100644 index 00000000..78b60124 --- /dev/null +++ b/docs/evals/assets/learning_loop.cast @@ -0,0 +1,29 @@ +{"version":3,"term":{"cols":110,"rows":52},"timestamp":1779128039,"command":"core demo learning-loop","env":{"SHELL":"/bin/zsh"}} +[0.178, "o", "\r\n================================================================================\r\n Learning Loop — Cold Turn to Grounded Surface, End-to-End (ADR-0055..0057)\r\n================================================================================\r\n\r\nReference: ADR-0055 (Phase B DiscoveryCandidate emission, Phase A audit\r\n+ provenance), ADR-0056 (Phase C1 contemplation), ADR-0057 (Phase C2\r\nTeachingChainProposal + replay gate + operator review).\r\n\r\nA single deterministic prompt drives every scene:\r\n\r\n \"Why does thought exist?\"\r\n\r\nHeadline claim: CORE, asked a question it cannot ground, emits\r\nstructured evidence that a reviewed chain would have helped. An\r\noperator authors a proposal from that evidence. The replay-\r\nequivalence gate confirms no regression. The operator accepts. The\r\n**same prompt now produces a deterministic teaching-grounded surface**\r\n— replayable, with full provenance back to the operator's accept.\r\n\r\n S1. Cold turn — runtime returns the universal disclosure;\r\n "] +[0.000, "o", " grounding_source = none.\r\n S2. Discovery emission — DiscoveryCandidate emitted to the attached\r\n sink; contemplation enriches with pack/\r\n corpus evidence. Active corpus untouched.\r\n S3. Operator proposal — complete chain authored + real replay gate\r\n run + replay_equivalent=True → pending.\r\n S4. Operator accept — accept_proposal writes ONE line to a\r\n transient corpus (copy of active + new\r\n chain). Active corpus byte-identical.\r\n S5. Replay the prompt — _CORPUS_PATH swapped to the transient;\r\n same prompt now teaching-grounded with the\r\n new chain's subject / connective / object.\r\n\r\nTrust boundary:\r\n The demo writes ONLY to a tempdir-scoped transient corpus. The\r\n active teaching corpus on disk is byte-identical pre/post — same\r\n swap pattern the replay-equivalence gate use"] +[0.000, "o", "s. No clock-time read.\r\n\r\nWhat to expect:\r\n Per-scene printout with CLAIM, prompt/inputs, outputs, and the\r\n byte-identical-corpus assertion. Final BEFORE / AFTER block shows\r\n the deterministic surface change on the same prompt.\r\n\r\nTest gate:\r\n tests/test_learning_loop_demo.py (7 tests — loop closes, before is\r\n ungrounded, after contains new chain atoms, discovery emits ≥1,\r\n replay gate reports no regression, transient adds exactly 1 line\r\n while active is byte-identical, same prompt drives both surfaces).\r\n\r\nMachine-readable output:\r\n core demo learning-loop --json\r\n================================================================================\r\n"] +[0.000, "o", "\r\n"] +[1.066, "o", "\r\n────────────────────────────────────────────────────────────────────────\r\n S1. Cold turn — runtime cannot ground the prompt\r\n────────────────────────────────────────────────────────────────────────\r\n CLAIM: Active corpus has no (thought, cause) chain. The runtime falls through to the universal insufficient-grounding disclosure. Identity / safety / ethics gates still run.\r\n\r\n"] +[0.005, "o", " prompt : Why does thought exist?\r\n surface : I don't know — insufficient grounding for that yet.\r\n grounding_source : none\r\n discovery candidates : 1 (emitted post-turn)\r\n\r\n────────────────────────────────────────────────────────────────────────\r\n S2. Discovery candidate — structured evidence, not a mutation\r\n────────────────────────────────────────────────────────────────────────\r\n CLAIM: The runtime emits a DiscoveryCandidate (ADR-0055 Phase B) documenting that a reviewed (thought, cause) chain WOULD have grounded this turn. Contemplation (ADR-0056 Phase C1) enriches with pack/corpus evidence pointers. Active corpus is byte-identical — emission writes to the sink on"] +[0.000, "o", "ly.\r\n"] +[0.000, "o", "\r\n"] +[0.000, "o", " candidate_id : 17673a2f15c8da21…\r\n trigger : would_have_grounded\r\n proposed_chain : {'connective': None, 'intent': 'cause', 'object': None, 'subject': 'thought'}\r\n"] +[0.000, "o", " polarity : undetermined\r\n"] +[0.000, "o", " claim_domain : factual\r\n"] +[0.000, "o", " pack_consistent : True\r\n boundary_clean : True\r\n"] +[0.000, "o", " evidence (pack-only) : [{'epistemic_status': 'coherent', 'polarity': 'affirms', 'ref': 'thought', 'source': 'pack'}]\r\n\r\n"] +[0.000, "o", "────────────────────────────────────────────────────────────────────────\r\n S3. Operator-authored proposal — replay-equivalence gate runs\r\n────────────────────────────────────────────────────────────────────────\r\n CLAIM: From the discovery candidate's evidence, the operator authors a complete chain: thought reveals meaning. Affirming evidence is the existing corpus chain cause_creation_reveals_meaning. The real replay gate (teaching.replay.run_replay_equivalence) runs the cognition public split twice — active corpus vs. transient-with-appended-chain — and reports no regression.\r\n"] +[0.000, "o", "\r\n"] +[1.849, "o", " proposal_id : 016252428267e4f339969524988c4794\r\n proposed_chain : {'subject': 'thought', 'intent': 'cause', 'connective': 'reveals', 'object': 'meaning'}\r\n evidence (corpus ref) : cause_creation_reveals_meaning\r\n replay baseline : {'intent_accuracy': 1.0, 'surface_groundedness': 1.0, 'term_capture_rate': 0.9167, 'versor_closure_rate': 1.0}\r\n replay candidate : {'intent_accuracy': 1.0, 'surface_groundedness': 1.0, 'term_capture_rate': 0.9167, 'versor_closure_rate': 1.0}\r\n regressed_metrics : []\r\n replay_equivalent : True\r\n state : pending\r\n\r\n"] +[0.000, "o", "────────────────────────────────────────────────────────────────────────\r\n S4. Operator accept — transient corpus, active corpus untouched\r\n────────────────────────────────────────────────────────────────────────\r\n CLAIM: accept_proposal writes one JSONL line to a TRANSIENT corpus (copy of active + new chain). The active corpus file bytes are byte-identical pre/post. Provenance on the new entry: adr-0057:discovery_promoted:.\r\n"] +[0.000, "o", "\r\n"] +[0.001, "o", " appended chain_id : cause_thought_reveals_meaning\r\n transient corpus path : /var/folders/kg/5xbm28qd7jl55j7lv3p001f40000gn/T/learning_loop_demo__5wclh7k/cognition_chains_v1.jsonl\r\n transient lines before : 10\r\n transient lines after : 11\r\n active corpus byte-eq : True\r\n\r\n────────────────────────────────────────────────────────────────────────\r\n"] +[0.000, "o", " S5. Same prompt — now deterministically teaching-grounded\r\n────────────────────────────────────────────────────────────────────────\r\n"] +[0.000, "o", " CLAIM: With the runtime's corpus path swapped to the transient corpus, the same prompt now returns a teaching-grounded surface containing the operator-accepted chain: thought reveals meaning. Identical bytes for any replay of the same prompt against this corpus state.\r\n"] +[0.000, "o", "\r\n"] +[0.071, "o", " prompt : Why does thought exist?\r\n surface : thought — teaching-grounded (cognition_chains_v1): cognition.thought; logos.internal. thought reveals meaning (cognition.meaning). No session evidence yet.\r\n grounding_source : teaching\r\n"] +[0.000, "o", "\r\n════════════════════════════════════════════════════════════════════════\r\n BEFORE / AFTER (single deterministic prompt, one accept between)\r\n════════════════════════════════════════════════════════════════════════\r\n prompt : Why does thought exist?\r\n before : [none] I don't know — insufficient grounding for that yet.\r\n after : [teaching] thought — teaching-grounded (cognition_chains_v1): cognition.thought; logos.internal. thought reveals meaning (cognition.meaning). No session evidence yet.\r\n"] +[0.000, "o", "\r\n"] +[0.000, "o", " learning_loop_closed : True\r\n"] +[0.000, "o", " active corpus byte-identical : True\r\n\r\n"] +[0.014, "x", "0"] diff --git a/docs/evals/assets/learning_loop.gif b/docs/evals/assets/learning_loop.gif new file mode 100644 index 00000000..520e913d Binary files /dev/null and b/docs/evals/assets/learning_loop.gif differ diff --git a/docs/evals/assets/teaching_loop_bench.cast b/docs/evals/assets/teaching_loop_bench.cast new file mode 100644 index 00000000..a93abeb7 --- /dev/null +++ b/docs/evals/assets/teaching_loop_bench.cast @@ -0,0 +1,6 @@ +{"version":3,"term":{"cols":100,"rows":50},"timestamp":1779128043,"command":"core bench --suite teaching-loop --runs 10","env":{"SHELL":"/bin/zsh"}} +[0.154, "o", "\r\n================================================================================\r\n Teaching-Loop Determinism Benchmark (ADR-0055..0057)\r\n================================================================================\r\n\r\nReference: benchmarks/teaching_loop.py, ADR-0057 (the propose →\r\nreplay → accept pipeline). Pairs naturally with ADR-0045's 100%\r\nexact-NIAH recall numbers — same epistemic class of guarantee,\r\napplied to the *learning loop* rather than only to retrieval.\r\n\r\nFor an identical candidate, the bench runs the full reviewed-corpus\r\nextension pipeline (propose_from_candidate → real run_replay_equivalence\r\n→ accept_proposal) N times against tempdir-scoped paths, and asserts\r\nbyte-identical artifacts every iteration:\r\n\r\n - proposal_id (SHA-256 of canonical-JSON payload)\r\n - replay_baseline (cognition lane metrics on active corpus)\r\n - replay_candidate (cognition lane metrics on transient corpus)\r\n - regressed_metrics (sorted tuple)\r\n - chain_id_written\r\n\r\nAl"] +[0.000, "o", "so reports per-iteration wall-time (mean / p50 / p95) and total.\r\n\r\nTrust boundary:\r\n Every write is confined to a tempdir created inside the bench loop.\r\n Active corpus file bytes are byte-identical pre/post regardless of\r\n N. Asserted in the bench report and re-pinned in the test.\r\n\r\n100-run reference result on today's main:\r\n unique(proposal_id) = 1 unique(chain_id) = 1\r\n unique(baseline) = 1 unique(candidate) = 1\r\n active_corpus_byte_eq = True\r\n mean = 1.85s p50 = 1.84s p95 = 1.85s\r\n\r\nTest gate:\r\n tests/test_teaching_loop_bench.py (5 tests — determinism at small N,\r\n proposal_id SHA-256 shape, canonical chain_id layout, latency stats\r\n well-formed, JSON serialisation).\r\n\r\nUsage:\r\n core bench --suite teaching-loop --runs 100\r\n core bench --suite teaching-loop --runs 10 --json\r\n================================================================================\r\n"] +[0.000, "o", "\r\n"] +[19.411, "o", " [PASS] teaching_loop_determinism 1.0000 byte_identity_ratio\r\n 10 runs; unique(proposal_id)=1, unique(baseline)=1, unique(candidate)=1, unique(chain_id)=1; mean=1.937s p50=1.836s p95=2.393s; active_corpus_byte_eq=True\r\n\r\nALL PASSED\r\n"] +[0.015, "x", "0"] diff --git a/docs/evals/assets/teaching_loop_bench.gif b/docs/evals/assets/teaching_loop_bench.gif new file mode 100644 index 00000000..e20f55df Binary files /dev/null and b/docs/evals/assets/teaching_loop_bench.gif differ diff --git a/docs/evals/learning_loop_demo.md b/docs/evals/learning_loop_demo.md new file mode 100644 index 00000000..8857c190 --- /dev/null +++ b/docs/evals/learning_loop_demo.md @@ -0,0 +1,178 @@ +# Learning-Loop Demo — Cold Turn to Grounded Surface, End-to-End + +**Date:** 2026-05-18 +**Runner:** `evals/learning_loop/run_demo.py` +**CLI:** `core demo learning-loop` (`--json` for machine-readable output) +**Contract tests:** `tests/test_learning_loop_demo.py` (7 passing) +**Reference ADRs:** [0055](../decisions/ADR-0055-inter-session-memory-discovery-promotion.md), [0056](../decisions/ADR-0056-contemplation-loop-c1.md), [0057](../decisions/ADR-0057-teaching-chain-proposal-review.md) + +![learning-loop demo](assets/learning_loop.gif) + +## Headline claim + +> A single deterministic prompt, `"Why does thought exist?"`, produces: +> +> - **Before** the loop runs: `[none] I don't know — insufficient grounding for that yet.` +> - **After** one operator accept: `[teaching] thought — teaching-grounded (cognition_chains_v1): cognition.thought; logos.internal. thought reveals meaning (cognition.meaning). No session evidence yet.` +> +> The active corpus on disk is byte-identical pre/post. The change lives entirely in a transient corpus the demo writes to and then swaps the runtime's `_CORPUS_PATH` to — the same pattern the replay-equivalence gate uses. + +## What CORE has that other systems do not + +| Property | Continuous pre-training / RLHF | CORE learning loop | +|---|---|---| +| **Per-fact provenance** | None (gradient updates are diffuse) | `Provenance(adr_id, source, review_date, raw)` on every appended chain | +| **Replay-equivalence guarantee** | Offline eval at checkpoint cadence | Inline gate runs the full cognition lane on every admission | +| **Audit trail** | Training logs | `ProposalLog` events: `created` → `replay` → `transition` → `accepted_corpus_append` | +| **Replayable across runs** | No (stochastic; weight checkpoints diverge) | SHA-256 deterministic `proposal_id`; bit-identical artifacts (see [`teaching_loop_bench.md`](teaching_loop_bench.md)) | +| **Operator gate** | Implicit (deployment cadence) | Explicit `core teaching review --accept --review-date YYYY-MM-DD` | +| **Roll-back semantics** | Restore checkpoint | `core teaching supersede ` (append-only at disk; active view derived) | + +This is the architecture deployments that need to answer *"why did the +system say this today that it would not have said yesterday?"* require. + +## Trust boundary + +The demo writes only to a tempdir-scoped transient corpus. The active +teaching corpus on disk is byte-identical pre/post. The swap pattern: + +```python +real_path = _tg._CORPUS_PATH +try: + _tg._CORPUS_PATH = transient + _tg._corpus_index.cache_clear() + rt2 = ChatRuntime() + response = rt2.chat("Why does thought exist?") +finally: + _tg._CORPUS_PATH = real_path + _tg._corpus_index.cache_clear() +``` + +This is the same mechanism `teaching/replay.py:_swap_corpus_path` uses +during the replay-equivalence gate. No clock-time read anywhere in +the loop. + +## Five scenes + +| Scene | What runs | Trust property | +|---|---|---| +| **S1. Cold turn** | Real `ChatRuntime.chat("Why does thought exist?")` | No `(thought, cause)` chain exists → universal disclosure; `grounding_source=none`. | +| **S2. Discovery emission** | Discovery sink + contemplation enrich the candidate | Active corpus untouched; emission is sink-only. | +| **S3. Operator proposal** | `propose_from_candidate()` runs real `run_replay_equivalence()` | Cognition lane runs twice; no regression → `state=pending`. | +| **S4. Operator accept** | `accept_proposal()` against a **transient** corpus path | Active corpus byte-identical; transient gains exactly 1 line; provenance `adr-0057:discovery_promoted:2026-05-18`. | +| **S5. Replay** | `_CORPUS_PATH` swapped to transient; fresh `ChatRuntime` runs the same prompt | Surface contains subject / humanised connective / object; `grounding_source=teaching`. | + +## Sample run + +```text +──────────────────────────────────────────────────────────────────────── + S1. Cold turn — runtime cannot ground the prompt +──────────────────────────────────────────────────────────────────────── + prompt : Why does thought exist? + surface : I don't know — insufficient grounding for that yet. + grounding_source : none + discovery candidates : 1 (emitted post-turn) + +──────────────────────────────────────────────────────────────────────── + S2. Discovery candidate — structured evidence, not a mutation +──────────────────────────────────────────────────────────────────────── + candidate_id : 17673a2f15c8da21… + trigger : would_have_grounded + proposed_chain : {'connective': None, 'intent': 'cause', + 'object': None, 'subject': 'thought'} + polarity : undetermined + claim_domain : factual + pack_consistent : True + boundary_clean : True + evidence (pack-only) : [{'epistemic_status': 'coherent', + 'polarity': 'affirms', 'ref': 'thought', + 'source': 'pack'}] + +──────────────────────────────────────────────────────────────────────── + S3. Operator-authored proposal — replay-equivalence gate runs +──────────────────────────────────────────────────────────────────────── + proposal_id : 016252428267e4f339969524988c4794 + proposed_chain : {'subject': 'thought', 'intent': 'cause', + 'connective': 'reveals', 'object': 'meaning'} + evidence (corpus ref) : cause_creation_reveals_meaning + replay baseline : {'intent_accuracy': 1.0, 'surface_groundedness': + 1.0, 'term_capture_rate': 0.9167, + 'versor_closure_rate': 1.0} + replay candidate : {'intent_accuracy': 1.0, 'surface_groundedness': + 1.0, 'term_capture_rate': 0.9167, + 'versor_closure_rate': 1.0} + regressed_metrics : [] + replay_equivalent : True + state : pending + +──────────────────────────────────────────────────────────────────────── + S4. Operator accept — transient corpus, active corpus untouched +──────────────────────────────────────────────────────────────────────── + appended chain_id : cause_thought_reveals_meaning + transient corpus path : /tmp/learning_loop_demo_xxxxxx/cognition_chains_v1.jsonl + transient lines before : 10 + transient lines after : 11 + active corpus byte-eq : True + +──────────────────────────────────────────────────────────────────────── + S5. Same prompt — now deterministically teaching-grounded +──────────────────────────────────────────────────────────────────────── + prompt : Why does thought exist? + surface : thought — teaching-grounded (cognition_chains_v1): + cognition.thought; logos.internal. + thought reveals meaning (cognition.meaning). + No session evidence yet. + grounding_source : teaching + +════════════════════════════════════════════════════════════════════════ + BEFORE / AFTER (single deterministic prompt, one accept between) +════════════════════════════════════════════════════════════════════════ + prompt : Why does thought exist? + before : [none] I don't know — insufficient grounding for that yet. + after : [teaching] thought — teaching-grounded (cognition_chains_v1): + cognition.thought; logos.internal. + thought reveals meaning (cognition.meaning). + No session evidence yet. + + learning_loop_closed : True + active corpus byte-identical : True +``` + +## How to reproduce + +```bash +core demo learning-loop # human output (preamble + scenes + before/after) +core demo learning-loop --json # machine-readable DemoReport +python -m pytest tests/test_learning_loop_demo.py -q # ~15s +``` + +## Falsifiable claims + +If any of these stops holding, the headline claim no longer holds: + +- `report.learning_loop_closed` is `True`. +- `report.active_corpus_byte_identical` is `True`. +- `report.before.grounding_source == "none"`; surface contains `"insufficient grounding"`. +- `report.after.grounding_source == "teaching"`; surface contains `"thought"` AND `"reveal"` AND `"meaning"` AND `"teaching-grounded"`. +- S3: `replay_evidence.replay_equivalent is True`, `regressed_metrics == []`, `state == "pending"`. +- S4: `transient_lines_after == transient_lines_before + 1` AND `active_corpus_byte_identical is True`. +- The same prompt drives both surfaces (`report.prompt == "Why does thought exist?"`). + +## Why "thought" is the demo subject + +The subject must satisfy three pre-conditions for the demo to fire deterministically: + +1. **Pack-resident** (otherwise the discovery candidate isn't emitted) — confirmed by `'thought' in _pack_index()`. +2. **No active `(thought, cause)` chain** (otherwise the cold turn would already be teaching-grounded) — confirmed by the active corpus snapshot. +3. **Intent classifier picks `CAUSE` on a natural prompt** — `"Why does thought exist?"` classifies as `CAUSE / subject="thought"` deterministically. + +The operator-authored chain (`thought reveals meaning`) cites +`cause_creation_reveals_meaning` as affirming evidence. Both endpoint +lemmas (`thought`, `meaning`) are pack-resident; the connective +`reveals` is in the canonical predicate set. + +## Related + +- Anti-regression demo: [`anti_regression_demo.md`](anti_regression_demo.md) — the inverse demo showing each gate refusing a bad proposal. +- Determinism benchmark: [`teaching_loop_bench.md`](teaching_loop_bench.md) — N-run byte-identical-artifact proof on this exact pipeline. +- Operator surface: see the [Inter-Session Memory section in README](../../README.md#inter-session-memory--reviewed-learning). diff --git a/docs/evals/teaching_loop_bench.md b/docs/evals/teaching_loop_bench.md new file mode 100644 index 00000000..f39d584c --- /dev/null +++ b/docs/evals/teaching_loop_bench.md @@ -0,0 +1,137 @@ +# Teaching-Loop Determinism Benchmark + +**Date:** 2026-05-18 +**Runner:** `benchmarks/teaching_loop.py` +**CLI:** `core bench --suite teaching-loop [--runs N] [--json]` +**Contract tests:** `tests/test_teaching_loop_bench.py` (5 passing) +**Reference ADRs:** [0055](../decisions/ADR-0055-inter-session-memory-discovery-promotion.md), [0057](../decisions/ADR-0057-teaching-chain-proposal-review.md), [0045](../decisions/ADR-0045-long-context-recall-vs-transformer-baselines.md) + +![teaching-loop benchmark](assets/teaching_loop_bench.gif) + +## Headline claim + +> For an identical candidate, N runs of the full reviewed-corpus +> extension pipeline (`propose_from_candidate` → real +> `run_replay_equivalence` → `accept_proposal`) produce N +> **byte-identical** artifacts at every observable point. +> +> The active teaching corpus on disk is byte-identical pre/post, +> regardless of N. + +This is the determinism guarantee for the *learning loop itself* — +the analog of [ADR-0045's 100% exact-NIAH recall](../decisions/ADR-0045-long-context-recall-vs-transformer-baselines.md) +result, applied to the learning path rather than only to retrieval. + +## What's asserted byte-identical + +| Artifact | How it's derived | Why this matters | +|---|---|---| +| `proposal_id` | SHA-256 prefix of canonical-JSON `(source_candidate_id, proposed_chain)` | If hashing of inputs ever drifts (locale, dict-ordering, float formatting), this changes. | +| `replay_baseline` | Cognition lane metrics on the active corpus | If any cognition-lane component became non-deterministic, this varies across runs. | +| `replay_candidate` | Cognition lane metrics on transient-with-append corpus | Same as above, run against a different corpus state. | +| `regressed_metrics` | Sorted tuple of strict-decrease metric names | A 1-element drift would expose comparison non-determinism. | +| `chain_id_written` | Canonical `___` | Append-side identifier derivation. | + +If determinism breaks anywhere in the pipeline — proposal hashing, the +replay-equivalence gate, accept-side corpus-append, or `ProposalLog` +replay — at least one of the `unique_*` counts exceeds 1 and the bench +fails. + +## 100-run reference result (today's main) + +```text +unique(proposal_id) = 1 unique(chain_id) = 1 +unique(baseline) = 1 unique(candidate) = 1 +unique(regressed_metrics) = 1 +active_corpus_byte_eq = True + +Latency per iteration: + mean = 1.849s p50 = 1.838s p95 = 1.851s total = ~185s +``` + +The p95 sits within 1% of the p50 — the loop's per-iteration cost is +dominated by the two cognition-lane runs inside the replay gate, both +of which are themselves deterministic in time as well as output. + +## Sample 10-run output + +```text +================================================================================ + Teaching-Loop Determinism Benchmark (ADR-0055..0057) +================================================================================ +... + [PASS] teaching_loop_determinism 1.0000 byte_identity_ratio + 10 runs; unique(proposal_id)=1, unique(baseline)=1, + unique(candidate)=1, unique(chain_id)=1; + mean=1.948s p50=1.846s p95=2.406s; active_corpus_byte_eq=True + +ALL PASSED +``` + +(p95 in any single 10-run sample is noisier than the 100-run number — a +single warm-cache vs cold-cache iteration can move it ~30%. The 100-run +distribution is the canonical reference.) + +## Trust boundary + +Every write is confined to a tempdir created inside the bench loop: + +```python +for _ in range(runs): + with tempfile.TemporaryDirectory() as tmpdir: + log_path = Path(tmpdir) / "proposals.jsonl" + transient = Path(tmpdir) / "cognition_chains_v1.jsonl" + shutil.copyfile(active_path, transient) + ... +``` + +The active corpus is read at the start and at the end. Any byte +difference would fail the bench. Re-pinned by +`test_teaching_loop_is_deterministic_across_three_runs` in +`tests/test_teaching_loop_bench.py`. + +## How to reproduce + +```bash +core bench --suite teaching-loop --runs 100 # canonical reference run +core bench --suite teaching-loop --runs 10 # quick smoke (~20s) +core bench --suite teaching-loop --runs 100 --json # machine-readable + +python -m pytest tests/test_teaching_loop_bench.py -q # ~25s +``` + +## Falsifiable claims + +If any of these stops holding, the headline claim no longer holds: + +- `report.deterministic` is `True` (all five `unique_*` counts are 1). +- `report.active_corpus_byte_identical` is `True`. +- `report.sample_proposal_id` is 32 lowercase hex chars (SHA-256 prefix). +- `report.sample_chain_id == "cause_thought_reveals_meaning"`. +- `report.elapsed_p95_s >= report.elapsed_p50_s`. +- `report.elapsed_total_s >= mean × runs × 0.9` (sanity check on wall-time accounting). + +The contract test file pins all of these at low N for fast CI; the +100-run reference number is informational, not gated. + +## Why this pairs with ADR-0045 + +[ADR-0045](../decisions/ADR-0045-long-context-recall-vs-transformer-baselines.md) showed +CORE achieves **100% exact recall** at N ∈ {100, 1k, 10k, 100k} in a +needle-in-a-haystack scan — the *retrieval* path is bit-exact. + +This benchmark shows the **learning path** is also bit-exact: the same +candidate, run N times, produces the same accepted chain. Together +they form the two halves of the deterministic-cognition claim: + +- **Read path** (ADR-0045): exact, scale-invariant, no approximation. +- **Write path** (this bench): exact, replayable, no non-determinism. + +No LLM-based system has published equivalent numbers on either path, +let alone both. + +## Related + +- Anti-regression demo: [`anti_regression_demo.md`](anti_regression_demo.md) — what the gate does when a regression *is* detected. +- Learning-loop demo: [`learning_loop_demo.md`](learning_loop_demo.md) — the same pipeline as a narrative walkthrough. +- Long-context comparison: [ADR-0045 / `long-context-comparison`](../decisions/ADR-0045-long-context-recall-vs-transformer-baselines.md) — the sibling determinism number for the *read* path.