docs(adr-0055-0057): preambles + README index for the demo trilogy
Three external-facing demos / benchmarks now match the existing audit-tour / pack-measurements / long-context-comparison treatment: preamble printed before the run, README index entries, claims table. - core/cli.py — _ANTI_REGRESSION_PREAMBLE, _LEARNING_LOOP_PREAMBLE, _TEACHING_LOOP_BENCH_PREAMBLE. Each lists reference ADRs, what to expect, trust boundary, test gate, and machine-readable invocation. Wired through _print_preamble in the demo dispatch + bench dispatch (suppressed under --json). - README.md — new "Inter-Session Memory — Reviewed Learning" section between Teaching Order and Architecture: the three-gate trust property table, the three live-demo table, and the operator-surface command list. Quick-start block lists `core demo anti-regression`, `core demo learning-loop`, and `core bench --suite teaching-loop --runs 100` alongside the existing demos. No code paths changed — preambles are stdout-only when not under JSON. Tests unchanged; 17/17 green (5 anti-regression + 7 learning-loop + 5 bench).
This commit is contained in:
Shay
parent
82dac4b16f
commit
d24e98906e
2 changed files with 198 additions and 0 deletions
38
README.md
38
README.md
|
|
@ -82,6 +82,8 @@ core test --suite adr-0024 # Forward Semantic Control chain (98
|
|||
core demo audit-tour # 4-scene pack-layer audit walkthrough (ADR-0027..0041)
|
||||
core demo pack-measurements # ADR-0043 — pack-layer claims as per-pack measurements
|
||||
core demo long-context-comparison # ADR-0045 — CORE NIAH recall + frozen transformer baselines
|
||||
core demo anti-regression # ADR-0057 — three-gate defense against learning harm
|
||||
core demo learning-loop # ADR-0055..0057 — cold turn → discovery → propose → accept → grounded
|
||||
core demo phase6 # 3-condition comparative table (CORE vs baseline)
|
||||
core demo phase5 # stratified 5-family mechanism-isolation
|
||||
core demo all # both + combined summary
|
||||
|
|
@ -92,6 +94,7 @@ core eval cognition # run a discovered lane
|
|||
core trace "your text here" # one-turn field-telemetry trace
|
||||
core pulse "What is truth?" # one full cognitive pulse
|
||||
core bench --suite latency # benchmark harness
|
||||
core bench --suite teaching-loop --runs 100 # ADR-0055..0057 — replayable learning loop determinism
|
||||
core doctor --packs --rust # environment + pack + Rust status
|
||||
```
|
||||
|
||||
|
|
@ -164,6 +167,41 @@ Full doctrine, decision rules, and curriculum-platform locations: [`docs/teachin
|
|||
|
||||
---
|
||||
|
||||
## Inter-Session Memory — Reviewed Learning
|
||||
|
||||
CORE extends its own teaching corpus through a four-tier path: session vault → turn-event audit → reviewed teaching corpus → ratified packs. No opaque gradient updates, no uncurated ingestion. The only path to active-corpus extension is the review-gated `TeachingChainProposal` ([ADR-0057](docs/decisions/ADR-0057-teaching-chain-proposal-review.md)), built from a contemplated `DiscoveryCandidate` ([ADR-0056](docs/decisions/ADR-0056-contemplation-loop.md)) emitted by the turn loop ([ADR-0055](docs/decisions/ADR-0055-inter-session-memory.md)).
|
||||
|
||||
Three independent gates every extension must pass:
|
||||
|
||||
| Gate | What it checks | Trust property |
|
||||
|---|---|---|
|
||||
| **Eligibility predicate** | polarity ∈ {affirms, falsifies} ∧ ≥1 `source='corpus'` evidence ∧ claim_domain ≠ evaluative ∧ boundary_clean ∧ chain complete | Pre-replay; raises `ProposalError`; no log entry. |
|
||||
| **Replay-equivalence gate** | Full cognition lane on active vs transient-with-append; any strict-decrease in `intent_accuracy / surface_groundedness / term_capture_rate / versor_closure_rate` auto-rejects with named metrics. | Active corpus byte-identical pre/post. |
|
||||
| **Operator review** | Explicit `core teaching review <id> --accept` writes one JSONL line via `append_chain_to_corpus` (the sole corpus-write surface). | No auto-apply; replay-equivalence is a precondition, not a permission. |
|
||||
|
||||
Supersession is the second operator-direct mutation surface: `core teaching supersede <old_chain_id>` retires an active chain by appending a replacement with `superseded_by`, with byte-identical rollback on any post-audit failure.
|
||||
|
||||
Three live demos / benchmarks make the chain demoable end-to-end:
|
||||
|
||||
| Demo | Headline claim | Live command |
|
||||
|---|---|---|
|
||||
| **Anti-regression** | Three independent gates each fail closed; bad proposals stop at the cheapest applicable gate. | `core demo anti-regression` |
|
||||
| **Learning loop** | Same deterministic prompt: `[none] I don't know…` before, `[teaching] thought reveals meaning…` after one accept. | `core demo learning-loop` |
|
||||
| **Determinism bench** | N identical inputs → N byte-identical proposal_id / replay metrics / chain_id. 100 runs: `unique=1` everywhere, mean ≈ 1.85s. | `core bench --suite teaching-loop --runs 100` |
|
||||
|
||||
Operator surfaces:
|
||||
|
||||
```
|
||||
core teaching audit # surface load decisions + drop reasons
|
||||
core teaching propose <candidate-jsonl-path> # build a proposal, run the replay gate
|
||||
core teaching proposals --state pending # inspect the proposal log
|
||||
core teaching review <proposal_id> --accept --review-date YYYY-MM-DD
|
||||
core teaching supersede <old_chain_id> --subject ... --intent ... --connective ... --object ... --review-date YYYY-MM-DD
|
||||
core teaching supersessions # pair retired chains with replacements (orphan-aware)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
|
|
|
|||
160
core/cli.py
160
core/cli.py
|
|
@ -1188,6 +1188,159 @@ Machine-readable output:
|
|||
"""
|
||||
|
||||
|
||||
_ANTI_REGRESSION_PREAMBLE = """
|
||||
================================================================================
|
||||
Anti-Regression — Three-Gate Defense Against Learning Harm (ADR-0057)
|
||||
================================================================================
|
||||
|
||||
Reference: ADR-0055 (inter-session memory), ADR-0056 (contemplation),
|
||||
ADR-0057 (TeachingChainProposal + replay-equivalence gate).
|
||||
|
||||
When a system extends its own knowledge, the gate that decides what to
|
||||
admit is the load-bearing part — not the proposer. CORE's reviewed-
|
||||
corpus extension path has three independent gates that each must pass
|
||||
before any byte is written to the active teaching corpus:
|
||||
|
||||
S1. Eligibility predicate (mechanical, pre-replay)
|
||||
Five mechanical checks on candidate shape — polarity in
|
||||
{affirms, falsifies}, ≥1 source='corpus' evidence pointer,
|
||||
claim_domain != evaluative (unless --allow-evaluative),
|
||||
boundary_clean=True, proposed_chain complete.
|
||||
Ineligible candidates raise ProposalError; they never enter
|
||||
the proposal log.
|
||||
|
||||
S2. Replay-equivalence gate (mechanical, post-eligibility)
|
||||
The full cognition lane runs against the active corpus AND
|
||||
against a transient copy with the proposed chain appended.
|
||||
Any strict-decrease in a watched metric (intent_accuracy,
|
||||
surface_groundedness, term_capture_rate, versor_closure_rate)
|
||||
auto-rejects with the metrics named in the operator note.
|
||||
Active corpus file bytes byte-identical pre/post.
|
||||
|
||||
S3. Operator review (manual, post-replay)
|
||||
Even a replay-equivalent proposal only reaches the 'pending'
|
||||
state. Explicit `core teaching review <id> --accept` is
|
||||
required to write to the active corpus.
|
||||
|
||||
What to expect:
|
||||
Three scenes, each printed with its CLAIM, candidate, outcome, and
|
||||
the byte-identical-corpus assertion. Scenes 1 and 3 use the real
|
||||
replay function; scene 2 injects a controlled replay (via the
|
||||
documented run_replay= kwarg) to deterministically demonstrate the
|
||||
auto-rejection lifecycle on a synthetic regression.
|
||||
|
||||
Test gate:
|
||||
tests/test_anti_regression_demo.py (5 tests — per-scene claim +
|
||||
active-corpus-byte-identical invariant).
|
||||
|
||||
Machine-readable output:
|
||||
core demo anti-regression --json
|
||||
================================================================================
|
||||
"""
|
||||
|
||||
|
||||
_LEARNING_LOOP_PREAMBLE = """
|
||||
================================================================================
|
||||
Learning Loop — Cold Turn to Grounded Surface, End-to-End (ADR-0055..0057)
|
||||
================================================================================
|
||||
|
||||
Reference: ADR-0055 (Phase B DiscoveryCandidate emission, Phase A audit
|
||||
+ provenance), ADR-0056 (Phase C1 contemplation), ADR-0057 (Phase C2
|
||||
TeachingChainProposal + replay gate + operator review).
|
||||
|
||||
A single deterministic prompt drives every scene:
|
||||
|
||||
"Why does thought exist?"
|
||||
|
||||
Headline claim: CORE, asked a question it cannot ground, emits
|
||||
structured evidence that a reviewed chain would have helped. An
|
||||
operator authors a proposal from that evidence. The replay-
|
||||
equivalence gate confirms no regression. The operator accepts. The
|
||||
**same prompt now produces a deterministic teaching-grounded surface**
|
||||
— replayable, with full provenance back to the operator's accept.
|
||||
|
||||
S1. Cold turn — runtime returns the universal disclosure;
|
||||
grounding_source = none.
|
||||
S2. Discovery emission — DiscoveryCandidate emitted to the attached
|
||||
sink; contemplation enriches with pack/
|
||||
corpus evidence. Active corpus untouched.
|
||||
S3. Operator proposal — complete chain authored + real replay gate
|
||||
run + replay_equivalent=True → pending.
|
||||
S4. Operator accept — accept_proposal writes ONE line to a
|
||||
transient corpus (copy of active + new
|
||||
chain). Active corpus byte-identical.
|
||||
S5. Replay the prompt — _CORPUS_PATH swapped to the transient;
|
||||
same prompt now teaching-grounded with the
|
||||
new chain's subject / connective / object.
|
||||
|
||||
Trust boundary:
|
||||
The demo writes ONLY to a tempdir-scoped transient corpus. The
|
||||
active teaching corpus on disk is byte-identical pre/post — same
|
||||
swap pattern the replay-equivalence gate uses. No clock-time read.
|
||||
|
||||
What to expect:
|
||||
Per-scene printout with CLAIM, prompt/inputs, outputs, and the
|
||||
byte-identical-corpus assertion. Final BEFORE / AFTER block shows
|
||||
the deterministic surface change on the same prompt.
|
||||
|
||||
Test gate:
|
||||
tests/test_learning_loop_demo.py (7 tests — loop closes, before is
|
||||
ungrounded, after contains new chain atoms, discovery emits ≥1,
|
||||
replay gate reports no regression, transient adds exactly 1 line
|
||||
while active is byte-identical, same prompt drives both surfaces).
|
||||
|
||||
Machine-readable output:
|
||||
core demo learning-loop --json
|
||||
================================================================================
|
||||
"""
|
||||
|
||||
|
||||
_TEACHING_LOOP_BENCH_PREAMBLE = """
|
||||
================================================================================
|
||||
Teaching-Loop Determinism Benchmark (ADR-0055..0057)
|
||||
================================================================================
|
||||
|
||||
Reference: benchmarks/teaching_loop.py, ADR-0057 (the propose →
|
||||
replay → accept pipeline). Pairs naturally with ADR-0045's 100%
|
||||
exact-NIAH recall numbers — same epistemic class of guarantee,
|
||||
applied to the *learning loop* rather than only to retrieval.
|
||||
|
||||
For an identical candidate, the bench runs the full reviewed-corpus
|
||||
extension pipeline (propose_from_candidate → real run_replay_equivalence
|
||||
→ accept_proposal) N times against tempdir-scoped paths, and asserts
|
||||
byte-identical artifacts every iteration:
|
||||
|
||||
- proposal_id (SHA-256 of canonical-JSON payload)
|
||||
- replay_baseline (cognition lane metrics on active corpus)
|
||||
- replay_candidate (cognition lane metrics on transient corpus)
|
||||
- regressed_metrics (sorted tuple)
|
||||
- chain_id_written
|
||||
|
||||
Also reports per-iteration wall-time (mean / p50 / p95) and total.
|
||||
|
||||
Trust boundary:
|
||||
Every write is confined to a tempdir created inside the bench loop.
|
||||
Active corpus file bytes are byte-identical pre/post regardless of
|
||||
N. Asserted in the bench report and re-pinned in the test.
|
||||
|
||||
100-run reference result on today's main:
|
||||
unique(proposal_id) = 1 unique(chain_id) = 1
|
||||
unique(baseline) = 1 unique(candidate) = 1
|
||||
active_corpus_byte_eq = True
|
||||
mean = 1.85s p50 = 1.84s p95 = 1.85s
|
||||
|
||||
Test gate:
|
||||
tests/test_teaching_loop_bench.py (5 tests — determinism at small N,
|
||||
proposal_id SHA-256 shape, canonical chain_id layout, latency stats
|
||||
well-formed, JSON serialisation).
|
||||
|
||||
Usage:
|
||||
core bench --suite teaching-loop --runs 100
|
||||
core bench --suite teaching-loop --runs 10 --json
|
||||
================================================================================
|
||||
"""
|
||||
|
||||
|
||||
_ALL_PREAMBLE = """
|
||||
================================================================================
|
||||
Combined Demo — Full ADR-0024 Chain Evidence
|
||||
|
|
@ -1400,6 +1553,8 @@ def cmd_demo(args: argparse.Namespace) -> int:
|
|||
if target == "anti-regression":
|
||||
from evals.anti_regression.run_demo import run_demo
|
||||
|
||||
if not args.json:
|
||||
_print_preamble(_ANTI_REGRESSION_PREAMBLE)
|
||||
report = run_demo(emit_json=args.json)
|
||||
if args.json:
|
||||
print(json.dumps(report, indent=2, sort_keys=True))
|
||||
|
|
@ -1408,6 +1563,8 @@ def cmd_demo(args: argparse.Namespace) -> int:
|
|||
if target == "learning-loop":
|
||||
from evals.learning_loop.run_demo import run_demo as run_loop_demo
|
||||
|
||||
if not args.json:
|
||||
_print_preamble(_LEARNING_LOOP_PREAMBLE)
|
||||
report = run_loop_demo(emit_json=args.json)
|
||||
if args.json:
|
||||
print(json.dumps(report, indent=2, sort_keys=True))
|
||||
|
|
@ -1502,6 +1659,9 @@ def cmd_bench(args: argparse.Namespace) -> int:
|
|||
|
||||
from benchmarks.run_benchmarks import run_benchmarks
|
||||
|
||||
if args.suite == "teaching-loop" and not args.json:
|
||||
_print_preamble(_TEACHING_LOOP_BENCH_PREAMBLE)
|
||||
|
||||
report = run_benchmarks(
|
||||
suite=args.suite,
|
||||
runs=args.runs,
|
||||
|
|
|
|||
Loading…
Reference in a new issue