docs(adr-0055-0057): preambles + README index for the demo trilogy
Three external-facing demos / benchmarks now match the existing audit-tour / pack-measurements / long-context-comparison treatment: preamble printed before the run, README index entries, claims table. - core/cli.py — _ANTI_REGRESSION_PREAMBLE, _LEARNING_LOOP_PREAMBLE, _TEACHING_LOOP_BENCH_PREAMBLE. Each lists reference ADRs, what to expect, trust boundary, test gate, and machine-readable invocation. Wired through _print_preamble in the demo dispatch + bench dispatch (suppressed under --json). - README.md — new "Inter-Session Memory — Reviewed Learning" section between Teaching Order and Architecture: the three-gate trust property table, the three live-demo table, and the operator-surface command list. Quick-start block lists `core demo anti-regression`, `core demo learning-loop`, and `core bench --suite teaching-loop --runs 100` alongside the existing demos. No code paths changed — preambles are stdout-only when not under JSON. Tests unchanged; 17/17 green (5 anti-regression + 7 learning-loop + 5 bench).
This commit is contained in:
parent
82dac4b16f
commit
d24e98906e
2 changed files with 198 additions and 0 deletions
38
README.md
38
README.md
|
|
@ -82,6 +82,8 @@ core test --suite adr-0024 # Forward Semantic Control chain (98
|
||||||
core demo audit-tour # 4-scene pack-layer audit walkthrough (ADR-0027..0041)
|
core demo audit-tour # 4-scene pack-layer audit walkthrough (ADR-0027..0041)
|
||||||
core demo pack-measurements # ADR-0043 — pack-layer claims as per-pack measurements
|
core demo pack-measurements # ADR-0043 — pack-layer claims as per-pack measurements
|
||||||
core demo long-context-comparison # ADR-0045 — CORE NIAH recall + frozen transformer baselines
|
core demo long-context-comparison # ADR-0045 — CORE NIAH recall + frozen transformer baselines
|
||||||
|
core demo anti-regression # ADR-0057 — three-gate defense against learning harm
|
||||||
|
core demo learning-loop # ADR-0055..0057 — cold turn → discovery → propose → accept → grounded
|
||||||
core demo phase6 # 3-condition comparative table (CORE vs baseline)
|
core demo phase6 # 3-condition comparative table (CORE vs baseline)
|
||||||
core demo phase5 # stratified 5-family mechanism-isolation
|
core demo phase5 # stratified 5-family mechanism-isolation
|
||||||
core demo all # both + combined summary
|
core demo all # both + combined summary
|
||||||
|
|
@ -92,6 +94,7 @@ core eval cognition # run a discovered lane
|
||||||
core trace "your text here" # one-turn field-telemetry trace
|
core trace "your text here" # one-turn field-telemetry trace
|
||||||
core pulse "What is truth?" # one full cognitive pulse
|
core pulse "What is truth?" # one full cognitive pulse
|
||||||
core bench --suite latency # benchmark harness
|
core bench --suite latency # benchmark harness
|
||||||
|
core bench --suite teaching-loop --runs 100 # ADR-0055..0057 — replayable learning loop determinism
|
||||||
core doctor --packs --rust # environment + pack + Rust status
|
core doctor --packs --rust # environment + pack + Rust status
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -164,6 +167,41 @@ Full doctrine, decision rules, and curriculum-platform locations: [`docs/teachin
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Inter-Session Memory — Reviewed Learning
|
||||||
|
|
||||||
|
CORE extends its own teaching corpus through a four-tier path: session vault → turn-event audit → reviewed teaching corpus → ratified packs. No opaque gradient updates, no uncurated ingestion. The only path to active-corpus extension is the review-gated `TeachingChainProposal` ([ADR-0057](docs/decisions/ADR-0057-teaching-chain-proposal-review.md)), built from a contemplated `DiscoveryCandidate` ([ADR-0056](docs/decisions/ADR-0056-contemplation-loop.md)) emitted by the turn loop ([ADR-0055](docs/decisions/ADR-0055-inter-session-memory.md)).
|
||||||
|
|
||||||
|
Three independent gates every extension must pass:
|
||||||
|
|
||||||
|
| Gate | What it checks | Trust property |
|
||||||
|
|---|---|---|
|
||||||
|
| **Eligibility predicate** | polarity ∈ {affirms, falsifies} ∧ ≥1 `source='corpus'` evidence ∧ claim_domain ≠ evaluative ∧ boundary_clean ∧ chain complete | Pre-replay; raises `ProposalError`; no log entry. |
|
||||||
|
| **Replay-equivalence gate** | Full cognition lane on active vs transient-with-append; any strict-decrease in `intent_accuracy / surface_groundedness / term_capture_rate / versor_closure_rate` auto-rejects with named metrics. | Active corpus byte-identical pre/post. |
|
||||||
|
| **Operator review** | Explicit `core teaching review <id> --accept` writes one JSONL line via `append_chain_to_corpus` (the sole corpus-write surface). | No auto-apply; replay-equivalence is a precondition, not a permission. |
|
||||||
|
|
||||||
|
Supersession is the second operator-direct mutation surface: `core teaching supersede <old_chain_id>` retires an active chain by appending a replacement with `superseded_by`, with byte-identical rollback on any post-audit failure.
|
||||||
|
|
||||||
|
Three live demos / benchmarks make the chain demoable end-to-end:
|
||||||
|
|
||||||
|
| Demo | Headline claim | Live command |
|
||||||
|
|---|---|---|
|
||||||
|
| **Anti-regression** | Three independent gates each fail closed; bad proposals stop at the cheapest applicable gate. | `core demo anti-regression` |
|
||||||
|
| **Learning loop** | Same deterministic prompt: `[none] I don't know…` before, `[teaching] thought reveals meaning…` after one accept. | `core demo learning-loop` |
|
||||||
|
| **Determinism bench** | N identical inputs → N byte-identical proposal_id / replay metrics / chain_id. 100 runs: `unique=1` everywhere, mean ≈ 1.85s. | `core bench --suite teaching-loop --runs 100` |
|
||||||
|
|
||||||
|
Operator surfaces:
|
||||||
|
|
||||||
|
```
|
||||||
|
core teaching audit # surface load decisions + drop reasons
|
||||||
|
core teaching propose <candidate-jsonl-path> # build a proposal, run the replay gate
|
||||||
|
core teaching proposals --state pending # inspect the proposal log
|
||||||
|
core teaching review <proposal_id> --accept --review-date YYYY-MM-DD
|
||||||
|
core teaching supersede <old_chain_id> --subject ... --intent ... --connective ... --object ... --review-date YYYY-MM-DD
|
||||||
|
core teaching supersessions # pair retired chains with replacements (orphan-aware)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|
|
||||||
160
core/cli.py
160
core/cli.py
|
|
@ -1188,6 +1188,159 @@ Machine-readable output:
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
_ANTI_REGRESSION_PREAMBLE = """
|
||||||
|
================================================================================
|
||||||
|
Anti-Regression — Three-Gate Defense Against Learning Harm (ADR-0057)
|
||||||
|
================================================================================
|
||||||
|
|
||||||
|
Reference: ADR-0055 (inter-session memory), ADR-0056 (contemplation),
|
||||||
|
ADR-0057 (TeachingChainProposal + replay-equivalence gate).
|
||||||
|
|
||||||
|
When a system extends its own knowledge, the gate that decides what to
|
||||||
|
admit is the load-bearing part — not the proposer. CORE's reviewed-
|
||||||
|
corpus extension path has three independent gates that each must pass
|
||||||
|
before any byte is written to the active teaching corpus:
|
||||||
|
|
||||||
|
S1. Eligibility predicate (mechanical, pre-replay)
|
||||||
|
Five mechanical checks on candidate shape — polarity in
|
||||||
|
{affirms, falsifies}, ≥1 source='corpus' evidence pointer,
|
||||||
|
claim_domain != evaluative (unless --allow-evaluative),
|
||||||
|
boundary_clean=True, proposed_chain complete.
|
||||||
|
Ineligible candidates raise ProposalError; they never enter
|
||||||
|
the proposal log.
|
||||||
|
|
||||||
|
S2. Replay-equivalence gate (mechanical, post-eligibility)
|
||||||
|
The full cognition lane runs against the active corpus AND
|
||||||
|
against a transient copy with the proposed chain appended.
|
||||||
|
Any strict-decrease in a watched metric (intent_accuracy,
|
||||||
|
surface_groundedness, term_capture_rate, versor_closure_rate)
|
||||||
|
auto-rejects with the metrics named in the operator note.
|
||||||
|
Active corpus file bytes byte-identical pre/post.
|
||||||
|
|
||||||
|
S3. Operator review (manual, post-replay)
|
||||||
|
Even a replay-equivalent proposal only reaches the 'pending'
|
||||||
|
state. Explicit `core teaching review <id> --accept` is
|
||||||
|
required to write to the active corpus.
|
||||||
|
|
||||||
|
What to expect:
|
||||||
|
Three scenes, each printed with its CLAIM, candidate, outcome, and
|
||||||
|
the byte-identical-corpus assertion. Scenes 1 and 3 use the real
|
||||||
|
replay function; scene 2 injects a controlled replay (via the
|
||||||
|
documented run_replay= kwarg) to deterministically demonstrate the
|
||||||
|
auto-rejection lifecycle on a synthetic regression.
|
||||||
|
|
||||||
|
Test gate:
|
||||||
|
tests/test_anti_regression_demo.py (5 tests — per-scene claim +
|
||||||
|
active-corpus-byte-identical invariant).
|
||||||
|
|
||||||
|
Machine-readable output:
|
||||||
|
core demo anti-regression --json
|
||||||
|
================================================================================
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
_LEARNING_LOOP_PREAMBLE = """
|
||||||
|
================================================================================
|
||||||
|
Learning Loop — Cold Turn to Grounded Surface, End-to-End (ADR-0055..0057)
|
||||||
|
================================================================================
|
||||||
|
|
||||||
|
Reference: ADR-0055 (Phase B DiscoveryCandidate emission, Phase A audit
|
||||||
|
+ provenance), ADR-0056 (Phase C1 contemplation), ADR-0057 (Phase C2
|
||||||
|
TeachingChainProposal + replay gate + operator review).
|
||||||
|
|
||||||
|
A single deterministic prompt drives every scene:
|
||||||
|
|
||||||
|
"Why does thought exist?"
|
||||||
|
|
||||||
|
Headline claim: CORE, asked a question it cannot ground, emits
|
||||||
|
structured evidence that a reviewed chain would have helped. An
|
||||||
|
operator authors a proposal from that evidence. The replay-
|
||||||
|
equivalence gate confirms no regression. The operator accepts. The
|
||||||
|
**same prompt now produces a deterministic teaching-grounded surface**
|
||||||
|
— replayable, with full provenance back to the operator's accept.
|
||||||
|
|
||||||
|
S1. Cold turn — runtime returns the universal disclosure;
|
||||||
|
grounding_source = none.
|
||||||
|
S2. Discovery emission — DiscoveryCandidate emitted to the attached
|
||||||
|
sink; contemplation enriches with pack/
|
||||||
|
corpus evidence. Active corpus untouched.
|
||||||
|
S3. Operator proposal — complete chain authored + real replay gate
|
||||||
|
run + replay_equivalent=True → pending.
|
||||||
|
S4. Operator accept — accept_proposal writes ONE line to a
|
||||||
|
transient corpus (copy of active + new
|
||||||
|
chain). Active corpus byte-identical.
|
||||||
|
S5. Replay the prompt — _CORPUS_PATH swapped to the transient;
|
||||||
|
same prompt now teaching-grounded with the
|
||||||
|
new chain's subject / connective / object.
|
||||||
|
|
||||||
|
Trust boundary:
|
||||||
|
The demo writes ONLY to a tempdir-scoped transient corpus. The
|
||||||
|
active teaching corpus on disk is byte-identical pre/post — same
|
||||||
|
swap pattern the replay-equivalence gate uses. No clock-time read.
|
||||||
|
|
||||||
|
What to expect:
|
||||||
|
Per-scene printout with CLAIM, prompt/inputs, outputs, and the
|
||||||
|
byte-identical-corpus assertion. Final BEFORE / AFTER block shows
|
||||||
|
the deterministic surface change on the same prompt.
|
||||||
|
|
||||||
|
Test gate:
|
||||||
|
tests/test_learning_loop_demo.py (7 tests — loop closes, before is
|
||||||
|
ungrounded, after contains new chain atoms, discovery emits ≥1,
|
||||||
|
replay gate reports no regression, transient adds exactly 1 line
|
||||||
|
while active is byte-identical, same prompt drives both surfaces).
|
||||||
|
|
||||||
|
Machine-readable output:
|
||||||
|
core demo learning-loop --json
|
||||||
|
================================================================================
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
_TEACHING_LOOP_BENCH_PREAMBLE = """
|
||||||
|
================================================================================
|
||||||
|
Teaching-Loop Determinism Benchmark (ADR-0055..0057)
|
||||||
|
================================================================================
|
||||||
|
|
||||||
|
Reference: benchmarks/teaching_loop.py, ADR-0057 (the propose →
|
||||||
|
replay → accept pipeline). Pairs naturally with ADR-0045's 100%
|
||||||
|
exact-NIAH recall numbers — same epistemic class of guarantee,
|
||||||
|
applied to the *learning loop* rather than only to retrieval.
|
||||||
|
|
||||||
|
For an identical candidate, the bench runs the full reviewed-corpus
|
||||||
|
extension pipeline (propose_from_candidate → real run_replay_equivalence
|
||||||
|
→ accept_proposal) N times against tempdir-scoped paths, and asserts
|
||||||
|
byte-identical artifacts every iteration:
|
||||||
|
|
||||||
|
- proposal_id (SHA-256 of canonical-JSON payload)
|
||||||
|
- replay_baseline (cognition lane metrics on active corpus)
|
||||||
|
- replay_candidate (cognition lane metrics on transient corpus)
|
||||||
|
- regressed_metrics (sorted tuple)
|
||||||
|
- chain_id_written
|
||||||
|
|
||||||
|
Also reports per-iteration wall-time (mean / p50 / p95) and total.
|
||||||
|
|
||||||
|
Trust boundary:
|
||||||
|
Every write is confined to a tempdir created inside the bench loop.
|
||||||
|
Active corpus file bytes are byte-identical pre/post regardless of
|
||||||
|
N. Asserted in the bench report and re-pinned in the test.
|
||||||
|
|
||||||
|
100-run reference result on today's main:
|
||||||
|
unique(proposal_id) = 1 unique(chain_id) = 1
|
||||||
|
unique(baseline) = 1 unique(candidate) = 1
|
||||||
|
active_corpus_byte_eq = True
|
||||||
|
mean = 1.85s p50 = 1.84s p95 = 1.85s
|
||||||
|
|
||||||
|
Test gate:
|
||||||
|
tests/test_teaching_loop_bench.py (5 tests — determinism at small N,
|
||||||
|
proposal_id SHA-256 shape, canonical chain_id layout, latency stats
|
||||||
|
well-formed, JSON serialisation).
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
core bench --suite teaching-loop --runs 100
|
||||||
|
core bench --suite teaching-loop --runs 10 --json
|
||||||
|
================================================================================
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
_ALL_PREAMBLE = """
|
_ALL_PREAMBLE = """
|
||||||
================================================================================
|
================================================================================
|
||||||
Combined Demo — Full ADR-0024 Chain Evidence
|
Combined Demo — Full ADR-0024 Chain Evidence
|
||||||
|
|
@ -1400,6 +1553,8 @@ def cmd_demo(args: argparse.Namespace) -> int:
|
||||||
if target == "anti-regression":
|
if target == "anti-regression":
|
||||||
from evals.anti_regression.run_demo import run_demo
|
from evals.anti_regression.run_demo import run_demo
|
||||||
|
|
||||||
|
if not args.json:
|
||||||
|
_print_preamble(_ANTI_REGRESSION_PREAMBLE)
|
||||||
report = run_demo(emit_json=args.json)
|
report = run_demo(emit_json=args.json)
|
||||||
if args.json:
|
if args.json:
|
||||||
print(json.dumps(report, indent=2, sort_keys=True))
|
print(json.dumps(report, indent=2, sort_keys=True))
|
||||||
|
|
@ -1408,6 +1563,8 @@ def cmd_demo(args: argparse.Namespace) -> int:
|
||||||
if target == "learning-loop":
|
if target == "learning-loop":
|
||||||
from evals.learning_loop.run_demo import run_demo as run_loop_demo
|
from evals.learning_loop.run_demo import run_demo as run_loop_demo
|
||||||
|
|
||||||
|
if not args.json:
|
||||||
|
_print_preamble(_LEARNING_LOOP_PREAMBLE)
|
||||||
report = run_loop_demo(emit_json=args.json)
|
report = run_loop_demo(emit_json=args.json)
|
||||||
if args.json:
|
if args.json:
|
||||||
print(json.dumps(report, indent=2, sort_keys=True))
|
print(json.dumps(report, indent=2, sort_keys=True))
|
||||||
|
|
@ -1502,6 +1659,9 @@ def cmd_bench(args: argparse.Namespace) -> int:
|
||||||
|
|
||||||
from benchmarks.run_benchmarks import run_benchmarks
|
from benchmarks.run_benchmarks import run_benchmarks
|
||||||
|
|
||||||
|
if args.suite == "teaching-loop" and not args.json:
|
||||||
|
_print_preamble(_TEACHING_LOOP_BENCH_PREAMBLE)
|
||||||
|
|
||||||
report = run_benchmarks(
|
report = run_benchmarks(
|
||||||
suite=args.suite,
|
suite=args.suite,
|
||||||
runs=args.runs,
|
runs=args.runs,
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue