diff --git a/docs/decisions/ADR-0129-spaced-correction-replay-deferred.md b/docs/decisions/ADR-0129-spaced-correction-replay-deferred.md new file mode 100644 index 00000000..8bda41f9 --- /dev/null +++ b/docs/decisions/ADR-0129-spaced-correction-replay-deferred.md @@ -0,0 +1,198 @@ +# ADR-0129 — Spaced Reviewed-Correction Replay (Deferred Proposal) + +**Status:** Proposed — Deferred (backlog item; no implementation +scheduled until the GSM8K-math substrate arc through ADR-0127 / +ADR-0128 resolves Path-A vs Path-B) +**Date:** 2026-05-23 +**Author:** CORE agents + reviewers +**Depends on:** ADR-0040 (telemetry sink), ADR-0042 (audit tour), +ADR-0043 (pack measurements phase 2), ADR-0059 (correction-pass +telemetry), the entire `teaching/*` subsystem +**Supersedes:** none + +--- + +## Context + +A research review of *Beyond Traditional Pedagogy* (`/Users/kaizenpro/Downloads/...md`) +plus follow-up literature confirmation surfaced two pedagogy +findings with unusually strong empirical support and clean structural +mapping onto CORE's existing teaching loop: + +1. **Retrieval practice for retention of practiced material** — + among the most robust findings in cognitive psychology + (Roediger & Karpicke 2006 and ~two decades of replications). +2. **Spaced > massed practice** — Cepeda et al. 2006 meta-analysis; + not seriously contested in any subsequent literature. + +The combined "spaced retrieval" effect is consistently the single +highest-effect-size pedagogy intervention in well-replicated +literature. Far-transfer claims for retrieval are weaker (Pan & +Rickard 2018, Glaser & Richter 2025) — but **transfer to other +material is NOT the claim here**. The claim is *retention of +already-corrected material across long time horizons*, which is +precisely what retrieval-with-spacing addresses. + +The full research-and-review context lives at +`docs/sessions/SESSION-2026-05-23-pedagogy-research-and-teaching-loop-pivot.md`. + +### What CORE already has + +`teaching/store.py` retains reviewed corrections. `teaching/review.py` +and `teaching/correction.py` provide the reviewed-write path. When a +correction is consulted (e.g., during a turn that touches the +corrected case), CORE recalls it from the vault — exact, deterministic. + +### What CORE does NOT have + +No **deterministic schedule** that proactively re-runs CORE +against past corrections at expanding intervals to verify the +correction still produces the intended behavior under the +*current* runtime state (which has since absorbed other +corrections, pack updates, ratifications). Reviewed corrections +sit in the store until something queries them; nothing pulls +them back into circulation on a cadence. + +This is the gap "spaced retrieval" maps onto. In human pedagogy: +re-quiz the learner on previously-learned material at 1-day, +1-week, 1-month intervals to verify retention. In CORE: re-run +the deterministic pipeline against the past correction's input +on a fixed cadence and verify the output still matches the +correction's expected outcome. + +--- + +## Decision (proposed; deferred) + +Add a deterministic spaced-replay scheduler to the teaching +subsystem that, on a fixed cadence, re-runs the pipeline against +every retained reviewed-correction's input case and asserts the +output still matches the correction's expected outcome. Failures +become first-class "regression-against-prior-correction" events +emitted to the telemetry sink and surfaced in the operator +verdicts bundle. + +### Proposed shape (non-binding sketch — implementation defers) + +- **Cadence**: bounded, deterministic intervals. Initial proposal: + every reviewed correction is replayed at session counts `{5, 25, + 125, 625}` past the original correction (geometric, mirroring + spaced-repetition literature). Cadence drift is forbidden — + same input session count → same replay event. +- **Replay path**: pure read; the replay does NOT mutate any + state. It calls the standard pipeline against the correction's + recorded input, compares actual output to the correction's + expected output, emits an event. +- **Event shape**: + ```json + { + "type": "spaced_correction_replay", + "correction_id": "...", + "original_session_count": N, + "replay_session_count": M, + "interval": M - N, + "passed": , + "actual_output_digest": "", + "expected_output_digest": "", + "trace_hash_delta": "" + } + ``` +- **Failure handling**: a failed replay is NOT silently + re-corrected. It becomes an operator-visible event requiring + human review (preserves the "no unreviewed mutation" doctrine). + The original correction remains in the store; the new + divergence is logged as a separate event linked by + `correction_id`. +- **Determinism**: same `(input_sequence, pack_versions, + correction_store_state)` → byte-equal replay event sequence. +- **Cost ceiling**: per-session replay cost bounded — at session + count K, replays only fire for corrections whose + `(K - original) ∈ {5, 25, 125, 625}`. Most corrections fire + zero times per session; total replay cost is amortized. + +### Invariants + +| Invariant | Status | +|-----------|--------| +| `wrong == 0` | Preserved — replay is observational, not mutating | +| Trace determinism | Preserved — replay path is the standard deterministic pipeline | +| No unreviewed mutation | Preserved — replay failures emit events, do not auto-correct | +| Reviewed teaching only | Preserved — the scheduler operates only on already-reviewed corrections | +| `versor_condition(F) < 1e-6` | Untouched | + +### Why this is deferred, not accepted + +1. **Path-B uncertainty.** The GSM8K-math architectural arc + through ADR-0126 / 0127 / 0128 may resolve to a benchmark + re-targeting. If the math expert lane pivots, the + correction-store population characteristics change, and the + right cadence shape may differ. +2. **No measured regression.** ADR-0042's audit-tour demo + ADR-0043's + pack measurements already prove replay-equality at the snapshot + level. There is no observed instance of a past correction + silently regressing under subsequent pack updates. Spaced + replay would *detect* such regressions if they occur — but + we don't currently have evidence they do. +3. **Cost/benefit unmeasured.** The scheduler adds bounded but + nonzero per-session cost. Without an observed regression + incident, the lift is theoretical. +4. **Pedagogy analog is suggestive, not proof.** The mapping + from human-learner spaced retrieval to deterministic-engine + correction replay is structurally clean but is not itself + empirically validated *for engines*. CORE's exact-recall + property may obviate the human-learner-style decay this + addresses. + +## Exit criteria for un-deferral + +This ADR becomes a candidate for acceptance if any of: + +1. A reviewed correction is observed to silently regress against + current state (the failure mode the scheduler would have + caught). One real incident promotes from "theoretical + defense" to "documented incident response." +2. The teaching corpus grows past a threshold (~500 reviewed + corrections, current count is far below) where manual audit + is no longer feasible and proactive verification becomes + load-bearing for trust. +3. The GSM8K-math arc resolves and produces a stable correction + population whose retention characteristics can be + characterized, removing the Path-B uncertainty. + +## Alternatives considered + +### A. Build the scheduler now as defensive infrastructure. +Rejected per reason #2 above — no observed regression. + +### B. Run a single one-shot replay-all-corrections diagnostic. +Considered as a smaller alternative. May be worth a short ADR +of its own (`ADR-XXXX-correction-store-snapshot-audit`) if any +of the un-deferral exit criteria fire. Not pursued now. + +### C. Make this a runtime-mode flag the operator can enable. +Considered. The current operator surface (CLI lanes, telemetry +sink, verdicts bundle) is already busy; adding another opt-in +toggle increases surface area without a clear use case. + +## PR checklist (if/when proposed for acceptance) + +``` +What capability did this add? + → Deterministic spaced verification of past reviewed + corrections; defense against silent regression. +What invariant proves the field remains valid? + → wrong == 0 (replay is observational); trace determinism + (standard pipeline); no unreviewed mutation (failures emit + events, do not auto-correct). +Which CLI suite/eval proves the lane? + → New `core test --suite teaching-replay` lane runs replays + against a fixture correction store and asserts deterministic + event sequence; verdicts-bundle integration tested. +Did this avoid hidden normalization, stochastic fallback, +approximate recall, unreviewed mutation? + → Yes. Cadence is fixed-integer. Replay path is the standard + pipeline. Failures require human review. +If it touches user input, what trust boundary was enforced? + → No user-input surface. Replays consume correction-store + records, which are already ratified. +``` diff --git a/docs/decisions/ADR-0130-pre-articulation-calibration-deferred.md b/docs/decisions/ADR-0130-pre-articulation-calibration-deferred.md new file mode 100644 index 00000000..0f86f20f --- /dev/null +++ b/docs/decisions/ADR-0130-pre-articulation-calibration-deferred.md @@ -0,0 +1,221 @@ +# ADR-0130 — Pre-Articulation Calibration Logging (Deferred Proposal) + +**Status:** Proposed — Deferred (backlog item; no implementation +scheduled until the GSM8K-math substrate arc through ADR-0127 / +ADR-0128 resolves Path-A vs Path-B) +**Date:** 2026-05-23 +**Author:** CORE agents + reviewers +**Depends on:** ADR-0035 (turn-loop verdicts), ADR-0036 (safety +refusal), ADR-0040 (telemetry sink), ADR-0043 (pack measurements +phase 2), ADR-0059 (correction-pass telemetry) +**Supersedes:** none + +--- + +## Context + +The same research review that motivated ADR-0129 +(`docs/sessions/SESSION-2026-05-23-pedagogy-research-and-teaching-loop-pivot.md`) +surfaced a second pedagogy finding with strong empirical support +and a clean structural mapping: **metacognitive calibration via +prediction-outcome comparison**. + +In human pedagogy: learners predict their performance before a +task, compare prediction to outcome, and use the gap to recalibrate +their judgments of learning. Repeated calibration cycles shrink the +gap (Bjork, Dunlosky, Koriat, and successors). The mechanism is +that *uncalibrated confidence* is the central failure mode of +self-regulated learning — over-confidence leads to under-study; +under-confidence to over-study; both waste capacity. + +CORE has analogs at the runtime layer: +- `ADR-0035` end-of-turn safety + ethics verdicts +- `ADR-0036` typed safety refusal +- `ADR-0040` structured telemetry sink + +But CORE has **no analog at the teaching layer**. Before a +reviewed correction lands, no event captures the answer CORE +*would have produced* on the case-pre-correction; after correction, +no event captures the gap. + +This is a genuine information loss. Unlike a human learner, CORE +can do this without subjective bias: the pre-correction answer is +deterministic and exactly recordable. The gap is a real +measurement, not a self-report. + +--- + +## Decision (proposed; deferred) + +Add a deterministic "pre-correction prediction" capture step to +the teaching subsystem. When a correction is proposed (before +review), record CORE's current pipeline output on the case as a +prediction event. When the correction is accepted (review passes, +correction lands in store), emit a paired calibration event +recording the delta between prediction and the corrected outcome. + +### Proposed shape (non-binding sketch — implementation defers) + +- **Pre-correction prediction event** (emitted at correction + *proposal*): + ```json + { + "type": "pre_correction_prediction", + "correction_proposal_id": "...", + "input_digest": "", + "predicted_output_digest": "", + "predicted_trace_hash": "...", + "predicted_verdict": "correct | wrong | refused", + "current_runtime_state_digest": "" + } + ``` +- **Post-correction calibration event** (emitted at correction + *acceptance*): + ```json + { + "type": "post_correction_calibration", + "correction_id": "...", + "linked_prediction_id": "...", + "predicted_output_digest": "", + "corrected_output_digest": "", + "delta_class": + "no_change | answer_value | answer_unit | trace_only | refused_to_correct | correct_to_refused", + "pack_provenance_diff": [...] + } + ``` +- **Aggregation** (offline, periodic): a `calibration_report.json` + in the teaching subsystem's reports directory, summarizing: + - rate at which predictions matched corrections (no-change), + - distribution of delta classes, + - per-pack-version cohort comparisons (does calibration improve + after pack ratifications?). +- **No runtime gating.** The prediction is observational. It does + NOT alter what the operator can or cannot do; it does NOT veto + any correction. It's measurement, not control. + +### Invariants + +| Invariant | Status | +|-----------|--------| +| `wrong == 0` | Preserved — prediction is observational | +| Trace determinism | Preserved — prediction uses standard pipeline | +| No unreviewed mutation | Preserved — prediction does not write to correction store | +| Reviewed teaching only | Preserved — calibration emits only at proposal AND acceptance, both of which are operator-mediated | +| Telemetry redaction defaults | Preserved — input digests, not raw input | +| `versor_condition(F) < 1e-6` | Untouched | + +### What this enables that's not currently possible + +1. **Empirical answer to "is CORE actually getting better?"** Per-pack- + version calibration trends would show whether ratifications + improve pre-correction accuracy or just shift the surface. +2. **Audit story strengthens.** Today operators see that + corrections happen; they don't see how often CORE was *already* + right before the correction. The calibration gap is exactly + that signal. +3. **Misconfigured-pack detection.** A pack version that suddenly + spikes pre-correction error rate (vs the prior pack's rate) + is a flag worth surfacing automatically. +4. **Honest framing of operator workload.** If the calibration + shows pre-correction prediction matches the eventual correction + 95% of the time, the operator review can be lighter-touch on + that pack; if 5%, heavier-touch is warranted. + +### Why this is deferred, not accepted + +1. **Path-B uncertainty** (same as ADR-0129): the GSM8K-math arc + may produce a different correction-store population structure + that changes the right calibration cohorts. +2. **No measured calibration problem.** We don't currently have + evidence that pre-correction accuracy is misaligned with + post-correction. The proposal is "measure to find out" — but + the cost of building the measurement infrastructure should + match the prior of finding something. We don't have a strong + prior. +3. **Telemetry already substantial.** ADR-0040 / ADR-0042 / + ADR-0043 ship significant telemetry. Adding two new event + classes raises operator-noise floor; should only do so if the + signal proves worth it. +4. **Operator workload concern.** Even though prediction is + observational, a calibration report is a thing the operator + has to read. More artifacts means more attention budget; only + worth it if the artifacts surface decisions. + +## Exit criteria for un-deferral + +This ADR becomes a candidate for acceptance if any of: + +1. An incident occurs where a correction was applied unnecessarily + (CORE was already producing the right answer on that input) + AND the wasted review effort would have been visible to a + calibration-event sequence. +2. A pack ratification produces unexpected behavior whose + detection would have been faster via per-pack calibration + cohort comparison. +3. The teaching corpus grows to where operator review bandwidth + becomes a bottleneck and routing reviews by calibration + confidence would help triage. +4. The GSM8K-math arc resolves and ADR-0129 (spaced replay) is + un-deferred — at which point these two capabilities should + compose, since spaced replay events naturally produce + calibration evidence and they should share infrastructure. + +## Alternatives considered + +### A. Build calibration logging now, defer reporting. +Considered. Logging without reporting still costs telemetry +volume; without a report nobody reads the events; without reading +the events the log is decoration. Rejected per CLAUDE.md "no +decoration without integration." + +### B. Sample-based calibration (log a random 10% of proposals). +Considered. Determinism doctrine pushes against sampling — same +correction proposal should always produce same calibration event, +or none at all. Could be acceptable if sampling is content-keyed +(hash of input → bucket) so it's deterministic, but adds +complexity. Defer for now. + +### C. Manual calibration audit on demand. +The CLI could provide `core teaching calibration --window N` that +re-runs the last N corrections through the prediction path +*offline* and produces a one-shot calibration report. Lower +implementation cost than continuous logging; could be a useful +half-step. Worth its own short ADR if any of the exit criteria +above fire. + +## Composition with ADR-0129 + +If both ADRs are eventually un-deferred, they should share +infrastructure: + +- Spaced-replay events (ADR-0129) naturally yield calibration + evidence: each replay produces a prediction against the original + correction's expected outcome. The two event streams should + merge into a single calibration report. +- A correction whose spaced-replay events show repeated divergence + is a stronger signal than either system alone would catch. + +This composition is itself an argument for un-deferring both +together if either is un-deferred. + +## PR checklist (if/when proposed for acceptance) + +``` +What capability did this add? + → Deterministic measurement of pre-correction prediction + accuracy; empirical signal for "is CORE getting better." +What invariant proves the field remains valid? + → wrong == 0 (prediction is observational); trace determinism + (standard pipeline); no unreviewed mutation (calibration + writes events, not corrections). +Which CLI suite/eval proves the lane? + → New `core test --suite teaching-calibration` lane; fixture + correction-proposal sequence asserts deterministic event + pairs and report aggregation. +Did this avoid hidden normalization, stochastic fallback, +approximate recall, unreviewed mutation? + → Yes. Pure observational, deterministic pipeline call. +If it touches user input, what trust boundary was enforced? + → Telemetry emits input digests (SHA-256), not raw input, + consistent with ADR-0040's redact-by-default policy. +``` diff --git a/docs/sessions/SESSION-2026-05-23-pedagogy-research-and-teaching-loop-pivot.md b/docs/sessions/SESSION-2026-05-23-pedagogy-research-and-teaching-loop-pivot.md new file mode 100644 index 00000000..c707cc50 --- /dev/null +++ b/docs/sessions/SESSION-2026-05-23-pedagogy-research-and-teaching-loop-pivot.md @@ -0,0 +1,336 @@ +# SESSION 2026-05-23 — Pedagogy Research & Teaching-Loop Potential Pivot + +**Date:** 2026-05-23 +**Status:** Research note; load-bearing for ADR-0129 + ADR-0130 +**Trigger:** Operator-supplied review of *Beyond Traditional Pedagogy: +Research-Based and Emergent Techniques for Deep, Durable Learning* +(`/Users/kaizenpro/Downloads/Beyond Traditional Pedagogy ...md`, +2026-05-23) +**Branch:** `docs/pedagogy-review-and-teaching-backlog` + +--- + +## Why this session exists + +CORE's mid-2026 work has concentrated on the GSM8K-math substrate arc +(ADRs 0114a → 0119 → 0120 → 0121 → 0122 → 0123 / 0123a / 0123b → 0126 +candidate-graph topology → 0127 units pack → 0128 numerics pack). The +last three substrate ADRs each produced **zero sealed-holdout lift** +despite being correct work, leading to an architectural pivot (ADR-0126) +and a substrate-substrate (ADR-0127 / 0128) reframing. + +That sequence has been **all about the truth-articulation path** — +parse → graph → solve → verify → realize. The orthogonal axis — how +CORE *learns* from reviewed corrections — has not received the same +load-bearing attention since the ADR-0040-series teaching-substrate +work. The operator surfaced a pedagogy literature review as a sanity +check on whether the teaching loop, considered on its own merits, +has structural gaps that the GSM8K-math focus has been deferring. + +This session is the result of that check: the literature review of +the supplied document, follow-up confirmation research on contested +claims, and the resulting two backlog ADRs (0129 and 0130). + +--- + +## The reviewed document + +**Title:** *Beyond Traditional Pedagogy: Research-Based and Emergent +Techniques for Deep, Durable Learning* + +**Structure:** Executive summary + ~10 themed sections + a +synthesis table + 22 reference URLs. ~300 lines, well-cited within +the established cognitive-psychology / learning-science canon +(Bjork, Roediger & Karpicke, Kapur, Mayer, Collins / Brown / Newman, +Freeman et al., etc.). + +**Headline claims:** + +1. Active learning > passive lecture (Freeman et al. 2014 PNAS + meta-analysis as exemplar). +2. Retrieval practice (effortful recall) drives durable learning; + spacing + interleaving amplify. +3. Productive failure (Kapur) produces larger conceptual gains than + instruction-first ("3x" rhetoric in some references). +4. Embodied cognition: gesture, manipulation, handwriting matter for + acquisition. +5. Multimedia learning (Mayer): coordinated verbal + visual channels + subject to cognitive-load management. +6. Cognitive apprenticeship (Collins / Brown / Newman): modeling, + coaching, scaffolding, articulation, reflection, exploration. + +**Treatment quality:** sound at the survey level; weak on +calibration of contested findings. + +--- + +## Literature confirmation pass + +To avoid uncritical adoption, three areas with known replication +or boundary concerns were searched against 2024–2025 literature: + +### 1. Productive failure — calibration of the "3x" rhetoric + +**Anchor:** Sinha & Kapur 2021 meta-analysis (166 experimental +comparisons, ~12,000 participants), [SAGE](https://journals.sagepub.com/doi/full/10.3102/00346543211019105). + +| Claim | Reality | +|-------|---------| +| "3x conventional gains" | Headline from high-fidelity PF studies; meta-analysis average is **d = 0.36**, rising to **d = 0.58** at high design fidelity. Real but more modest. | +| "Broadly applicable" | **Largely a STEM finding.** Non-STEM evidence scarce; domain-general skill transfer not supported. | +| "Works for all learners" | Better effects for **older students** (secondary onwards); prior knowledge is a strong moderator (PMC 2023 study on prior math achievement). | + +**Verdict for CORE:** PF is the doc's most-overstated technique. +The structural analog inside CORE (let-attempt-then-review) +already exists in adversarial generation (ADR-0119.5), but with +a different mechanism — adversarial generation is a wrong-answer +*rejection* tool, not a learning-from-attempt tool. Adopting PF +shape inside CORE would mean intentionally allowing the engine +to attempt with knowingly-insufficient grounding and learning +from the gap. **This is the deliberate inverse of CORE's +`wrong==0` doctrine** and would require structural justification +beyond "the literature supports it." + +### 2. Retrieval practice — transfer limits + +**Anchor:** Pan & Rickard 2018 transfer meta-analysis; +Cognitive Research 2024 follow-up on far-transfer mechanisms, +[Cognitive Research](https://cognitiveresearchjournal.springeropen.com/articles/10.1186/s41235-024-00598-y). + +| Claim | Reality | +|-------|---------| +| "Retrieval drives transfer" | **Near transfer: yes (d = 0.4). Far transfer: weak/null (Pan & Rickard d = 0.16, n.s.).** | +| "Works for complex material" | Strongest for simple materials learned by rote; complex / educationally relevant materials show smaller, more contingent effects. | +| "Universal mechanism" | Recent work (Cognitive Research 2024): far-transfer benefits appear specifically when **rule-based learning** is the underlying mechanism + after delay. | +| "Lecture-hall ecological validity" | Glaser & Richter 2025 ([Teaching of Psychology](https://journals.sagepub.com/doi/10.1177/00986283231218943)): testing effect transfers poorly to studied-but-not-practiced content. | + +**Verdict for CORE:** Retrieval practice IS the most robust +finding *for retention of practiced material*. CORE's vault recall +already encodes the exact-recall ceiling of this technique. The +*spaced-retrieval* extension (spacing across time) is the part +not currently modeled in CORE's teaching loop — see ADR-0129. + +### 3. Embodied cognition — replication crisis + +**Anchor:** Machery 2024 chapter on the embodied-cognition +replication crisis, +[Routledge Handbook of Replication](https://www.taylorfrancis.com/chapters/edit/10.4324/9781003322511-50/replication-crisis-embodied-cognition-research-edouard-machery); +Frontiers in Education 2026 STEM-learning integrative review, +[Frontiers](https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2026.1811569/full). + +| Claim | Reality | +|-------|---------| +| Embodied learning effects | **Known replication crisis.** Foundational findings have failed independent replication. | +| Handwriting > typing | Strongest for very early literacy acquisition; broader generalizations are contested. | +| Universal benefit | "Embodiment sometimes facilitates learning and sometimes does not" — boundary conditions matter (Frontiers 2026). | + +**Verdict for CORE:** Not applicable directly (no body, no +sensorimotor system). Structural analogs (e.g., the +algebra/field/vault substrate as "grounding in a non-symbolic +representation") exist but the analogy is too weak to load-bear +design decisions. + +--- + +## What the doc missed (frameworks worth knowing) + +These should be on the radar even though they weren't in the +reviewed document: + +| Framework | Why it matters | +|-----------|----------------| +| **Worked-example effect** (Sweller, Paas, van Merriënboer) | Strong evidence for novice instruction; counter-evidence for experts (see expertise-reversal) | +| **Expertise-reversal effect** | Techniques that help novices actively hurt experts and vice versa. Directly relevant to CORE's `apprentice → audit-passed → expert` promotion contract (ADR-0120) | +| **Cognitive load theory** (Sweller) | Distinct intrinsic / extraneous / germane load distinction. Operationally useful for designing teaching corpora | +| **Deliberate practice** (Ericsson) | Specific goals + immediate feedback + repetition at the edge of capability. Better lens than "active learning" for skill domains | +| **Self-explanation effect** (Chi) | Narrow but strong evidence, particularly for science learning from worked examples | +| **Bloom's 2-sigma problem** (1984) | Unsolved benchmark: 1:1 tutoring delivers ~2 SD gains over conventional instruction. Most "evidence-based" techniques are attempts to approach this asymptote without the staffing cost | +| **Feedback science** (Hattie & Timperley 2007; Wisniewski et al. 2020) | Type / timing / specificity of feedback dominate effect sizes | +| **Pre-testing effect** (Carpenter, Richland) | Testing *before* studying primes attention. Distinct from retrieval practice | + +--- + +## Cross-walk to CORE architecture + +This is the load-bearing section: not "what does the literature +say" but "what does the literature say that maps onto a structural +move CORE could make." + +| Pedagogy concept | CORE analog | Status | +|------------------|-------------|--------| +| Retrieval practice | `teaching/correction.py` + vault recall | **Structurally aligned.** Every reviewed correction IS a retrieval+strengthen event. Exact-recall ceiling already met. | +| Spaced retrieval | (none) | **Genuine gap.** No deterministic spaced re-verification of past corrections. → ADR-0129 | +| Interleaving | Cross-pack chains (ADR-0064 / 0067) | **Aligned.** Cross-pack chains force discrimination across domains. | +| Metacognition / calibration (prediction vs outcome) | (none at teaching layer; partial at runtime via ADR-0035) | **Genuine gap.** No prediction-vs-outcome capture in teaching loop. → ADR-0130 | +| Cognitive apprenticeship | Ratified packs as articulated expert ontology | **Strong analog.** Packs ARE the encoded expert representation; ratification IS the "fade scaffolding" step. | +| Worked examples → fading | Teaching corpora → unsupervised generation | **Partial.** Corpora encode correct answers; less so the reasoning chain that produced them. Could be more first-class. | +| Productive failure | Adversarial generation (ADR-0119.5) | **Different mechanism.** Adversarial generation is rejection; PF would mean attempt-before-grounding. Inverse of `wrong==0`. Not recommended for direct port. | +| Pre-testing | (none) | Genuine gap. CORE always grounds before articulating; never the reverse. Adopting would conflict with `wrong==0`; not recommended. | +| Self-explanation | `SolutionTrace` provenance chain | **Structurally present.** Every answer has its derivation. Could be more first-class in teaching-store records. | +| Cognitive load theory | Substrate hierarchy: algebra → field → vault → realizer | **Implicit alignment.** CORE's layering matches CLT separation of intrinsic structure from extraneous load. | +| Expertise reversal | Pack-tier promotion (ADR-0120) | **Already encoded.** The `apprentice / audit-passed / expert` contract already knows that what helps an apprentice can ossify an expert. | +| Desirable difficulties | `wrong == 0` discipline | **Inverse mapping.** CORE refuses *undesirable* difficulty (confabulation under uncertainty). A teaching-side concept of *desirable* difficulty (challenging-but-not-impossible curriculum sequencing) is not yet first-class. | +| Feedback science | `teaching/review.py` | **Partially aligned.** Reviewed corrections ARE structured feedback. Timing / specificity dimensions could be more first-class. | + +--- + +## The two structural gaps worth addressing + +Distilled from the cross-walk, two design moves are both +*pedagogically supported by robust literature* AND *consistent +with CORE's existing determinism + provenance discipline*: + +### Gap 1 — Spaced reviewed-correction replay +**Mapped to:** retrieval-with-spacing literature (most robust +finding). +**ADR:** [ADR-0129](../decisions/ADR-0129-spaced-correction-replay-deferred.md) +**Status:** Deferred. +**Summary:** Periodic deterministic re-run of past reviewed +corrections to verify they still produce intended outcomes +under current state. Defense against silent regression as the +correction store and pack set evolves. + +### Gap 2 — Pre-articulation calibration logging +**Mapped to:** metacognitive calibration / prediction-outcome +comparison literature. +**ADR:** [ADR-0130](../decisions/ADR-0130-pre-articulation-calibration-deferred.md) +**Status:** Deferred. +**Summary:** When a correction is proposed, log CORE's +pre-correction prediction; on acceptance, emit the gap. +Provides empirical answer to "is CORE actually getting better" +across pack-version cohorts; supports operator triage. + +--- + +## What is NOT proposed (and why) + +| Considered | Rejected because | +|------------|------------------| +| Adopt productive-failure mechanism inside CORE | Inverse of `wrong==0`; would require structural justification beyond pedagogy literature. Adversarial generation (ADR-0119.5) covers the related "wrong-answer rejection" use case without the conceptual conflict. | +| Adopt pre-testing in articulation | Same conflict with `wrong==0`. CORE grounds before articulating by design. | +| Add embodied / sensorimotor layer | No body. The structural analogy (substrate as "grounding") is too weak to load-bear. | +| Add peer-learning multi-agent loop | Out of scope. Multi-agent coordination is a separate architectural question; not driven by this pedagogy review. | +| Adopt cognitive-load-theory load-balancing in realizer | Already implicit in the substrate hierarchy. Making it more explicit risks decoration without integration. | + +--- + +## Why both ADRs are deferred, not accepted + +Both ADR-0129 and ADR-0130 are **proposed but deferred**, following +the established ADR-0121 / ADR-0122-deferred pattern. The deferral +reasons compose: + +1. **Path-B uncertainty.** The active GSM8K-math arc + (ADR-0126 / 0127 / 0128) may resolve to a benchmark + re-targeting. If so, the correction-store population + characteristics change, and the right cadence (ADR-0129) / + cohort structure (ADR-0130) may differ. +2. **No observed incident.** Neither ADR has a triggering + incident. They're defensive infrastructure — useful if a + regression occurs (0129) or calibration drift develops (0130), + but speculative without that evidence. +3. **Cost/benefit unmeasured.** Both add telemetry volume and + operator review surface. Worth it only if the signal proves + load-bearing. +4. **Composition argument.** If either is un-deferred, the other + should be re-evaluated jointly — spaced-replay events + naturally yield calibration evidence; the two share + infrastructure. Deferring both together preserves that + composition. + +The exit criteria for un-deferral are documented in each ADR's +"Exit criteria for un-deferral" section. + +--- + +## Sequencing recommendation + +1. Land ADR-0126 (PR #161) — architecture. +2. Land ADR-0127 (Gemini in flight) — units pack. +3. Land ADR-0128 (Opus #2 in flight) — numerics pack. +4. Re-run train sample with both packs mounted → real Path-A vs + Path-B verdict. +5. If Path A: continue along the math expert promotion path. + ADR-0129 / 0130 remain deferred until an incident or + bandwidth pressure surfaces them. +6. If Path B: benchmark re-targeting becomes the work; ADR-0129 / + 0130 may become more relevant if the new benchmark's + correction-store characteristics are different enough to + warrant proactive verification. + +--- + +## Reference list (additional to the original document) + +- Sinha, T. & Kapur, M. (2021). When Problem Solving Followed by + Instruction Works: Evidence for Productive Failure. + [SAGE](https://journals.sagepub.com/doi/full/10.3102/00346543211019105) +- Pan, S. C. & Rickard, T. C. (2018). Transfer of test-enhanced + learning: meta-analytic review and synthesis. *Psychological Bulletin*. +- Glaser, J. & Richter, T. (2025). The Testing Effect in the + Lecture Hall: Does it Transfer to Content Studied but Not + Practiced? [Teaching of Psychology](https://journals.sagepub.com/doi/10.1177/00986283231218943) +- Cognitive Research: Principles and Implications (2024). Far + transfer of retrieval-practice benefits: rule-based learning + as the underlying mechanism. + [Springer](https://cognitiveresearchjournal.springeropen.com/articles/10.1186/s41235-024-00598-y) +- Machery, E. (2024). The Replication Crisis in Embodied Cognition + Research. *Routledge Handbook of Replication*. + [Taylor & Francis](https://www.taylorfrancis.com/chapters/edit/10.4324/9781003322511-50/replication-crisis-embodied-cognition-research-edouard-machery) +- Frontiers in Education (2026). Embodied cognition in STEM + learning: an integrative review. + [Frontiers](https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2026.1811569/full) +- Sinha & Kapur (2023). Prior math achievement and inventive + production predict learning from productive failure. + [PMC](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10185511/) +- Bloom, B. S. (1984). The 2 Sigma Problem. + *Educational Researcher 13(6)*. +- Hattie, J. & Timperley, H. (2007). The Power of Feedback. + *Review of Educational Research 77(1)*. +- Wisniewski, B., Zierer, K., Hattie, J. (2020). The Power of + Feedback Revisited: A Meta-Analysis. +- Ericsson, K. A., et al. (1993). The Role of Deliberate Practice + in the Acquisition of Expert Performance. + *Psychological Review 100(3)*. +- Sweller, J., van Merriënboer, J. J. G., Paas, F. G. W. C. (1998). + Cognitive Architecture and Instructional Design. + +--- + +## Open questions surfaced (not resolved this session) + +These are noted for future sessions; not items I'm advocating +for action: + +1. **Should teaching-corpus records carry "why" structure, not + just "what"?** Self-explanation literature suggests reasoning + chains in corpora may be more useful than answers alone. + `SolutionTrace` already exposes provenance; pushing this into + teaching corpora is a separate question. +2. **Is there a deliberate-practice analog at the curriculum + level?** ADR-0120's promotion contract already encodes + "stretch-but-pass" structure (correct_rate ≥ 0.60 floor). + Whether sub-curricula should also encode this is open. +3. **Could the pack-mutation-proposal pathway adopt a worked- + example pattern?** When a pack mutation is proposed, today + the operator sees the diff; could they also see a small + worked example showing the behavioral implication? + Speculative. +4. **Is Bloom's 2-sigma a meaningful target for CORE?** A + deterministic engine with exact recall has structural + properties that may exceed 1:1 tutoring on some axes + (consistency, replay) while underperforming on others + (adaptation, social affordances). Whether to claim this + target is an architectural framing question, not a + technical one. + +--- + +## End-of-session state + +- **ADRs added:** 0129 (deferred), 0130 (deferred). +- **Session note:** this file. +- **Branch:** `docs/pedagogy-review-and-teaching-backlog`. +- **PR plan:** single docs-only PR for the three files; lands + independently of the in-flight ADR-0126 / 0127 / 0128 chain. +- **No code changes.** No regression risk.