core-labs/core
core-labs/core
1
0
Fork 0
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity

docs: pedagogy research review + 2 deferred teaching-loop ADRs

Browse source 
Operator-supplied review of 'Beyond Traditional Pedagogy' triggered
a literature confirmation pass and a structural cross-walk against
CORE's teaching loop. Three artifacts:

1. ADR-0129 (DEFERRED) — Spaced reviewed-correction replay.
   Maps onto retrieval-with-spacing literature (most robust finding
   in cognitive psychology). Deterministic re-run of past
   corrections at fixed cadence to verify they still produce
   intended outcomes; failures emit operator-visible events
   (no auto-correction). Deferred pending GSM8K-math Path-A/B
   resolution + observed incident triggering un-deferral criteria.

2. ADR-0130 (DEFERRED) — Pre-articulation calibration logging.
   Maps onto metacognitive prediction-outcome literature. Logs
   CORE's pre-correction prediction; emits gap event on
   correction acceptance. Provides empirical signal for 'is CORE
   actually getting better' across pack-version cohorts. Deferred
   pending same conditions as ADR-0129; the two compose if
   un-deferred.

3. SESSION-2026-05-23 session note. Documents the review process:
   literature confirmation pass (productive failure overstated,
   retrieval transfer weaker than claimed, embodied cognition
   replication crisis), missed frameworks (worked-example effect,
   expertise reversal, CLT, deliberate practice, Bloom's 2-sigma),
   structural cross-walk to CORE architecture (12 mappings), and
   the rationale for ADRs 0129 + 0130 over alternative ports
   (productive failure rejected as inverse of wrong==0; pre-testing
   same; embodied learning N/A).

No code changes. Docs-only PR; lands independently of in-flight
ADR-0126 / 0127 / 0128 substrate chain.
This commit is contained in:
Shay 2026-05-23 07:01:55 -07:00
parent 5c6dfbe01d
commit f3c9436e66
3 changed files with 755 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

198
docs/decisions/ADR-0129-spaced-correction-replay-deferred.md Normal file
View file

@ -0,0 +1,198 @@
# ADR-0129 — Spaced Reviewed-Correction Replay (Deferred Proposal)
**Status:** Proposed — Deferred (backlog item; no implementation
scheduled until the GSM8K-math substrate arc through ADR-0127 /
ADR-0128 resolves Path-A vs Path-B)
**Date:** 2026-05-23
**Author:** CORE agents + reviewers
**Depends on:** ADR-0040 (telemetry sink), ADR-0042 (audit tour),
ADR-0043 (pack measurements phase 2), ADR-0059 (correction-pass
telemetry), the entire `teaching/*` subsystem
**Supersedes:** none
---
## Context
A research review of *Beyond Traditional Pedagogy* (`/Users/kaizenpro/Downloads/...md`)
plus follow-up literature confirmation surfaced two pedagogy
findings with unusually strong empirical support and clean structural
mapping onto CORE's existing teaching loop:
1. **Retrieval practice for retention of practiced material** —
among the most robust findings in cognitive psychology
(Roediger & Karpicke 2006 and ~two decades of replications).
2. **Spaced > massed practice** — Cepeda et al. 2006 meta-analysis;
not seriously contested in any subsequent literature.
The combined "spaced retrieval" effect is consistently the single
highest-effect-size pedagogy intervention in well-replicated
literature. Far-transfer claims for retrieval are weaker (Pan &
Rickard 2018, Glaser & Richter 2025) — but **transfer to other
material is NOT the claim here**. The claim is *retention of
already-corrected material across long time horizons*, which is
precisely what retrieval-with-spacing addresses.
The full research-and-review context lives at
`docs/sessions/SESSION-2026-05-23-pedagogy-research-and-teaching-loop-pivot.md`.
### What CORE already has
`teaching/store.py` retains reviewed corrections. `teaching/review.py`
and `teaching/correction.py` provide the reviewed-write path. When a
correction is consulted (e.g., during a turn that touches the
corrected case), CORE recalls it from the vault — exact, deterministic.
### What CORE does NOT have
No **deterministic schedule** that proactively re-runs CORE
against past corrections at expanding intervals to verify the
correction still produces the intended behavior under the
*current* runtime state (which has since absorbed other
corrections, pack updates, ratifications). Reviewed corrections
sit in the store until something queries them; nothing pulls
them back into circulation on a cadence.
This is the gap "spaced retrieval" maps onto. In human pedagogy:
re-quiz the learner on previously-learned material at 1-day,
1-week, 1-month intervals to verify retention. In CORE: re-run
the deterministic pipeline against the past correction's input
on a fixed cadence and verify the output still matches the
correction's expected outcome.
---
## Decision (proposed; deferred)
Add a deterministic spaced-replay scheduler to the teaching
subsystem that, on a fixed cadence, re-runs the pipeline against
every retained reviewed-correction's input case and asserts the
output still matches the correction's expected outcome. Failures
become first-class "regression-against-prior-correction" events
emitted to the telemetry sink and surfaced in the operator
verdicts bundle.
### Proposed shape (non-binding sketch — implementation defers)
- **Cadence**: bounded, deterministic intervals. Initial proposal:
every reviewed correction is replayed at session counts `{5, 25,
125, 625}` past the original correction (geometric, mirroring
spaced-repetition literature). Cadence drift is forbidden —
same input session count → same replay event.
- **Replay path**: pure read; the replay does NOT mutate any
state. It calls the standard pipeline against the correction's
recorded input, compares actual output to the correction's
expected output, emits an event.
- **Event shape**:
```json
{
"type": "spaced_correction_replay",
"correction_id": "...",
"original_session_count": N,
"replay_session_count": M,
"interval": M - N,
"passed": <bool>,
"actual_output_digest": "<sha256>",
"expected_output_digest": "<sha256>",
"trace_hash_delta": "<sha256-of-diff or empty>"
}
```
- **Failure handling**: a failed replay is NOT silently
re-corrected. It becomes an operator-visible event requiring
human review (preserves the "no unreviewed mutation" doctrine).
The original correction remains in the store; the new
divergence is logged as a separate event linked by
`correction_id`.
- **Determinism**: same `(input_sequence, pack_versions,
correction_store_state)` → byte-equal replay event sequence.
- **Cost ceiling**: per-session replay cost bounded — at session
count K, replays only fire for corrections whose
`(K - original) ∈ {5, 25, 125, 625}`. Most corrections fire
zero times per session; total replay cost is amortized.
### Invariants
| Invariant | Status |
|-----------|--------|
| `wrong == 0` | Preserved — replay is observational, not mutating |
| Trace determinism | Preserved — replay path is the standard deterministic pipeline |
| No unreviewed mutation | Preserved — replay failures emit events, do not auto-correct |
| Reviewed teaching only | Preserved — the scheduler operates only on already-reviewed corrections |
| `versor_condition(F) < 1e-6` | Untouched |
### Why this is deferred, not accepted
1. **Path-B uncertainty.** The GSM8K-math architectural arc
through ADR-0126 / 0127 / 0128 may resolve to a benchmark
re-targeting. If the math expert lane pivots, the
correction-store population characteristics change, and the
right cadence shape may differ.
2. **No measured regression.** ADR-0042's audit-tour demo + ADR-0043's
pack measurements already prove replay-equality at the snapshot
level. There is no observed instance of a past correction
silently regressing under subsequent pack updates. Spaced
replay would *detect* such regressions if they occur — but
we don't currently have evidence they do.
3. **Cost/benefit unmeasured.** The scheduler adds bounded but
nonzero per-session cost. Without an observed regression
incident, the lift is theoretical.
4. **Pedagogy analog is suggestive, not proof.** The mapping
from human-learner spaced retrieval to deterministic-engine
correction replay is structurally clean but is not itself
empirically validated *for engines*. CORE's exact-recall
property may obviate the human-learner-style decay this
addresses.
## Exit criteria for un-deferral
This ADR becomes a candidate for acceptance if any of:
1. A reviewed correction is observed to silently regress against
current state (the failure mode the scheduler would have
caught). One real incident promotes from "theoretical
defense" to "documented incident response."
2. The teaching corpus grows past a threshold (~500 reviewed
corrections, current count is far below) where manual audit
is no longer feasible and proactive verification becomes
load-bearing for trust.
3. The GSM8K-math arc resolves and produces a stable correction
population whose retention characteristics can be
characterized, removing the Path-B uncertainty.
## Alternatives considered
### A. Build the scheduler now as defensive infrastructure.
Rejected per reason #2 above — no observed regression.
### B. Run a single one-shot replay-all-corrections diagnostic.
Considered as a smaller alternative. May be worth a short ADR
of its own (`ADR-XXXX-correction-store-snapshot-audit`) if any
of the un-deferral exit criteria fire. Not pursued now.
### C. Make this a runtime-mode flag the operator can enable.
Considered. The current operator surface (CLI lanes, telemetry
sink, verdicts bundle) is already busy; adding another opt-in
toggle increases surface area without a clear use case.
## PR checklist (if/when proposed for acceptance)
```
What capability did this add?
→ Deterministic spaced verification of past reviewed
corrections; defense against silent regression.
What invariant proves the field remains valid?
→ wrong == 0 (replay is observational); trace determinism
(standard pipeline); no unreviewed mutation (failures emit
events, do not auto-correct).
Which CLI suite/eval proves the lane?
→ New `core test --suite teaching-replay` lane runs replays
against a fixture correction store and asserts deterministic
event sequence; verdicts-bundle integration tested.
Did this avoid hidden normalization, stochastic fallback,
approximate recall, unreviewed mutation?
→ Yes. Cadence is fixed-integer. Replay path is the standard
pipeline. Failures require human review.
If it touches user input, what trust boundary was enforced?
→ No user-input surface. Replays consume correction-store
records, which are already ratified.
```

221
docs/decisions/ADR-0130-pre-articulation-calibration-deferred.md Normal file
View file

@ -0,0 +1,221 @@
# ADR-0130 — Pre-Articulation Calibration Logging (Deferred Proposal)
**Status:** Proposed — Deferred (backlog item; no implementation
scheduled until the GSM8K-math substrate arc through ADR-0127 /
ADR-0128 resolves Path-A vs Path-B)
**Date:** 2026-05-23
**Author:** CORE agents + reviewers
**Depends on:** ADR-0035 (turn-loop verdicts), ADR-0036 (safety
refusal), ADR-0040 (telemetry sink), ADR-0043 (pack measurements
phase 2), ADR-0059 (correction-pass telemetry)
**Supersedes:** none
---
## Context
The same research review that motivated ADR-0129
(`docs/sessions/SESSION-2026-05-23-pedagogy-research-and-teaching-loop-pivot.md`)
surfaced a second pedagogy finding with strong empirical support
and a clean structural mapping: **metacognitive calibration via
prediction-outcome comparison**.
In human pedagogy: learners predict their performance before a
task, compare prediction to outcome, and use the gap to recalibrate
their judgments of learning. Repeated calibration cycles shrink the
gap (Bjork, Dunlosky, Koriat, and successors). The mechanism is
that *uncalibrated confidence* is the central failure mode of
self-regulated learning — over-confidence leads to under-study;
under-confidence to over-study; both waste capacity.
CORE has analogs at the runtime layer:
- `ADR-0035` end-of-turn safety + ethics verdicts
- `ADR-0036` typed safety refusal
- `ADR-0040` structured telemetry sink
But CORE has **no analog at the teaching layer**. Before a
reviewed correction lands, no event captures the answer CORE
*would have produced* on the case-pre-correction; after correction,
no event captures the gap.
This is a genuine information loss. Unlike a human learner, CORE
can do this without subjective bias: the pre-correction answer is
deterministic and exactly recordable. The gap is a real
measurement, not a self-report.
---
## Decision (proposed; deferred)
Add a deterministic "pre-correction prediction" capture step to
the teaching subsystem. When a correction is proposed (before
review), record CORE's current pipeline output on the case as a
prediction event. When the correction is accepted (review passes,
correction lands in store), emit a paired calibration event
recording the delta between prediction and the corrected outcome.
### Proposed shape (non-binding sketch — implementation defers)
- **Pre-correction prediction event** (emitted at correction
*proposal*):
```json
{
"type": "pre_correction_prediction",
"correction_proposal_id": "...",
"input_digest": "<sha256-of-input>",
"predicted_output_digest": "<sha256>",
"predicted_trace_hash": "...",
"predicted_verdict": "correct | wrong | refused",
"current_runtime_state_digest": "<sha256-of-pack-versions+correction-store>"
}
```
- **Post-correction calibration event** (emitted at correction
*acceptance*):
```json
{
"type": "post_correction_calibration",
"correction_id": "...",
"linked_prediction_id": "...",
"predicted_output_digest": "<sha256>",
"corrected_output_digest": "<sha256>",
"delta_class":
"no_change | answer_value | answer_unit | trace_only | refused_to_correct | correct_to_refused",
"pack_provenance_diff": [...]
}
```
- **Aggregation** (offline, periodic): a `calibration_report.json`
in the teaching subsystem's reports directory, summarizing:
- rate at which predictions matched corrections (no-change),
- distribution of delta classes,
- per-pack-version cohort comparisons (does calibration improve
after pack ratifications?).
- **No runtime gating.** The prediction is observational. It does
NOT alter what the operator can or cannot do; it does NOT veto
any correction. It's measurement, not control.
### Invariants
| Invariant | Status |
|-----------|--------|
| `wrong == 0` | Preserved — prediction is observational |
| Trace determinism | Preserved — prediction uses standard pipeline |
| No unreviewed mutation | Preserved — prediction does not write to correction store |
| Reviewed teaching only | Preserved — calibration emits only at proposal AND acceptance, both of which are operator-mediated |
| Telemetry redaction defaults | Preserved — input digests, not raw input |
| `versor_condition(F) < 1e-6` | Untouched |
### What this enables that's not currently possible
1. **Empirical answer to "is CORE actually getting better?"** Per-pack-
version calibration trends would show whether ratifications
improve pre-correction accuracy or just shift the surface.
2. **Audit story strengthens.** Today operators see that
corrections happen; they don't see how often CORE was *already*
right before the correction. The calibration gap is exactly
that signal.
3. **Misconfigured-pack detection.** A pack version that suddenly
spikes pre-correction error rate (vs the prior pack's rate)
is a flag worth surfacing automatically.
4. **Honest framing of operator workload.** If the calibration
shows pre-correction prediction matches the eventual correction
95% of the time, the operator review can be lighter-touch on
that pack; if 5%, heavier-touch is warranted.
### Why this is deferred, not accepted
1. **Path-B uncertainty** (same as ADR-0129): the GSM8K-math arc
may produce a different correction-store population structure
that changes the right calibration cohorts.
2. **No measured calibration problem.** We don't currently have
evidence that pre-correction accuracy is misaligned with
post-correction. The proposal is "measure to find out" — but
the cost of building the measurement infrastructure should
match the prior of finding something. We don't have a strong
prior.
3. **Telemetry already substantial.** ADR-0040 / ADR-0042 /
ADR-0043 ship significant telemetry. Adding two new event
classes raises operator-noise floor; should only do so if the
signal proves worth it.
4. **Operator workload concern.** Even though prediction is
observational, a calibration report is a thing the operator
has to read. More artifacts means more attention budget; only
worth it if the artifacts surface decisions.
## Exit criteria for un-deferral
This ADR becomes a candidate for acceptance if any of:
1. An incident occurs where a correction was applied unnecessarily
(CORE was already producing the right answer on that input)
AND the wasted review effort would have been visible to a
calibration-event sequence.
2. A pack ratification produces unexpected behavior whose
detection would have been faster via per-pack calibration
cohort comparison.
3. The teaching corpus grows to where operator review bandwidth
becomes a bottleneck and routing reviews by calibration
confidence would help triage.
4. The GSM8K-math arc resolves and ADR-0129 (spaced replay) is
un-deferred — at which point these two capabilities should
compose, since spaced replay events naturally produce
calibration evidence and they should share infrastructure.
## Alternatives considered
### A. Build calibration logging now, defer reporting.
Considered. Logging without reporting still costs telemetry
volume; without a report nobody reads the events; without reading
the events the log is decoration. Rejected per CLAUDE.md "no
decoration without integration."
### B. Sample-based calibration (log a random 10% of proposals).
Considered. Determinism doctrine pushes against sampling — same
correction proposal should always produce same calibration event,
or none at all. Could be acceptable if sampling is content-keyed
(hash of input → bucket) so it's deterministic, but adds
complexity. Defer for now.
### C. Manual calibration audit on demand.
The CLI could provide `core teaching calibration --window N` that
re-runs the last N corrections through the prediction path
*offline* and produces a one-shot calibration report. Lower
implementation cost than continuous logging; could be a useful
half-step. Worth its own short ADR if any of the exit criteria
above fire.
## Composition with ADR-0129
If both ADRs are eventually un-deferred, they should share
infrastructure:
- Spaced-replay events (ADR-0129) naturally yield calibration
evidence: each replay produces a prediction against the original
correction's expected outcome. The two event streams should
merge into a single calibration report.
- A correction whose spaced-replay events show repeated divergence
is a stronger signal than either system alone would catch.
This composition is itself an argument for un-deferring both
together if either is un-deferred.
## PR checklist (if/when proposed for acceptance)
```
What capability did this add?
→ Deterministic measurement of pre-correction prediction
accuracy; empirical signal for "is CORE getting better."
What invariant proves the field remains valid?
→ wrong == 0 (prediction is observational); trace determinism
(standard pipeline); no unreviewed mutation (calibration
writes events, not corrections).
Which CLI suite/eval proves the lane?
→ New `core test --suite teaching-calibration` lane; fixture
correction-proposal sequence asserts deterministic event
pairs and report aggregation.
Did this avoid hidden normalization, stochastic fallback,
approximate recall, unreviewed mutation?
→ Yes. Pure observational, deterministic pipeline call.
If it touches user input, what trust boundary was enforced?
→ Telemetry emits input digests (SHA-256), not raw input,
consistent with ADR-0040's redact-by-default policy.
```

336
docs/sessions/SESSION-2026-05-23-pedagogy-research-and-teaching-loop-pivot.md Normal file
View file

@ -0,0 +1,336 @@
# SESSION 2026-05-23 — Pedagogy Research & Teaching-Loop Potential Pivot
**Date:** 2026-05-23
**Status:** Research note; load-bearing for ADR-0129 + ADR-0130
**Trigger:** Operator-supplied review of *Beyond Traditional Pedagogy:
Research-Based and Emergent Techniques for Deep, Durable Learning*
(`/Users/kaizenpro/Downloads/Beyond Traditional Pedagogy ...md`,
2026-05-23)
**Branch:** `docs/pedagogy-review-and-teaching-backlog`
---
## Why this session exists
CORE's mid-2026 work has concentrated on the GSM8K-math substrate arc
(ADRs 0114a → 0119 → 0120 → 0121 → 0122 → 0123 / 0123a / 0123b → 0126
candidate-graph topology → 0127 units pack → 0128 numerics pack). The
last three substrate ADRs each produced **zero sealed-holdout lift**
despite being correct work, leading to an architectural pivot (ADR-0126)
and a substrate-substrate (ADR-0127 / 0128) reframing.
That sequence has been **all about the truth-articulation path** —
parse → graph → solve → verify → realize. The orthogonal axis — how
CORE *learns* from reviewed corrections — has not received the same
load-bearing attention since the ADR-0040-series teaching-substrate
work. The operator surfaced a pedagogy literature review as a sanity
check on whether the teaching loop, considered on its own merits,
has structural gaps that the GSM8K-math focus has been deferring.
This session is the result of that check: the literature review of
the supplied document, follow-up confirmation research on contested
claims, and the resulting two backlog ADRs (0129 and 0130).
---
## The reviewed document
**Title:** *Beyond Traditional Pedagogy: Research-Based and Emergent
Techniques for Deep, Durable Learning*
**Structure:** Executive summary + ~10 themed sections + a
synthesis table + 22 reference URLs. ~300 lines, well-cited within
the established cognitive-psychology / learning-science canon
(Bjork, Roediger & Karpicke, Kapur, Mayer, Collins / Brown / Newman,
Freeman et al., etc.).
**Headline claims:**
1. Active learning > passive lecture (Freeman et al. 2014 PNAS
meta-analysis as exemplar).
2. Retrieval practice (effortful recall) drives durable learning;
spacing + interleaving amplify.
3. Productive failure (Kapur) produces larger conceptual gains than
instruction-first ("3x" rhetoric in some references).
4. Embodied cognition: gesture, manipulation, handwriting matter for
acquisition.
5. Multimedia learning (Mayer): coordinated verbal + visual channels
subject to cognitive-load management.
6. Cognitive apprenticeship (Collins / Brown / Newman): modeling,
coaching, scaffolding, articulation, reflection, exploration.
**Treatment quality:** sound at the survey level; weak on
calibration of contested findings.
---
## Literature confirmation pass
To avoid uncritical adoption, three areas with known replication
or boundary concerns were searched against 2024–2025 literature:
### 1. Productive failure — calibration of the "3x" rhetoric
**Anchor:** Sinha & Kapur 2021 meta-analysis (166 experimental
comparisons, ~12,000 participants), [SAGE](https://journals.sagepub.com/doi/full/10.3102/00346543211019105).
| Claim | Reality |
|-------|---------|
| "3x conventional gains" | Headline from high-fidelity PF studies; meta-analysis average is **d = 0.36**, rising to **d = 0.58** at high design fidelity. Real but more modest. |
| "Broadly applicable" | **Largely a STEM finding.** Non-STEM evidence scarce; domain-general skill transfer not supported. |
| "Works for all learners" | Better effects for **older students** (secondary onwards); prior knowledge is a strong moderator (PMC 2023 study on prior math achievement). |
**Verdict for CORE:** PF is the doc's most-overstated technique.
The structural analog inside CORE (let-attempt-then-review)
already exists in adversarial generation (ADR-0119.5), but with
a different mechanism — adversarial generation is a wrong-answer
*rejection* tool, not a learning-from-attempt tool. Adopting PF
shape inside CORE would mean intentionally allowing the engine
to attempt with knowingly-insufficient grounding and learning
from the gap. **This is the deliberate inverse of CORE's
`wrong==0` doctrine** and would require structural justification
beyond "the literature supports it."
### 2. Retrieval practice — transfer limits
**Anchor:** Pan & Rickard 2018 transfer meta-analysis;
Cognitive Research 2024 follow-up on far-transfer mechanisms,
[Cognitive Research](https://cognitiveresearchjournal.springeropen.com/articles/10.1186/s41235-024-00598-y).
| Claim | Reality |
|-------|---------|
| "Retrieval drives transfer" | **Near transfer: yes (d = 0.4). Far transfer: weak/null (Pan & Rickard d = 0.16, n.s.).** |
| "Works for complex material" | Strongest for simple materials learned by rote; complex / educationally relevant materials show smaller, more contingent effects. |
| "Universal mechanism" | Recent work (Cognitive Research 2024): far-transfer benefits appear specifically when **rule-based learning** is the underlying mechanism + after delay. |
| "Lecture-hall ecological validity" | Glaser & Richter 2025 ([Teaching of Psychology](https://journals.sagepub.com/doi/10.1177/00986283231218943)): testing effect transfers poorly to studied-but-not-practiced content. |
**Verdict for CORE:** Retrieval practice IS the most robust
finding *for retention of practiced material*. CORE's vault recall
already encodes the exact-recall ceiling of this technique. The
*spaced-retrieval* extension (spacing across time) is the part
not currently modeled in CORE's teaching loop — see ADR-0129.
### 3. Embodied cognition — replication crisis
**Anchor:** Machery 2024 chapter on the embodied-cognition
replication crisis,
[Routledge Handbook of Replication](https://www.taylorfrancis.com/chapters/edit/10.4324/9781003322511-50/replication-crisis-embodied-cognition-research-edouard-machery);
Frontiers in Education 2026 STEM-learning integrative review,
[Frontiers](https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2026.1811569/full).
| Claim | Reality |
|-------|---------|
| Embodied learning effects | **Known replication crisis.** Foundational findings have failed independent replication. |
| Handwriting > typing | Strongest for very early literacy acquisition; broader generalizations are contested. |
| Universal benefit | "Embodiment sometimes facilitates learning and sometimes does not" — boundary conditions matter (Frontiers 2026). |
**Verdict for CORE:** Not applicable directly (no body, no
sensorimotor system). Structural analogs (e.g., the
algebra/field/vault substrate as "grounding in a non-symbolic
representation") exist but the analogy is too weak to load-bear
design decisions.
---
## What the doc missed (frameworks worth knowing)
These should be on the radar even though they weren't in the
reviewed document:
| Framework | Why it matters |
|-----------|----------------|
| **Worked-example effect** (Sweller, Paas, van Merriënboer) | Strong evidence for novice instruction; counter-evidence for experts (see expertise-reversal) |
| **Expertise-reversal effect** | Techniques that help novices actively hurt experts and vice versa. Directly relevant to CORE's `apprentice → audit-passed → expert` promotion contract (ADR-0120) |
| **Cognitive load theory** (Sweller) | Distinct intrinsic / extraneous / germane load distinction. Operationally useful for designing teaching corpora |
| **Deliberate practice** (Ericsson) | Specific goals + immediate feedback + repetition at the edge of capability. Better lens than "active learning" for skill domains |
| **Self-explanation effect** (Chi) | Narrow but strong evidence, particularly for science learning from worked examples |
| **Bloom's 2-sigma problem** (1984) | Unsolved benchmark: 1:1 tutoring delivers ~2 SD gains over conventional instruction. Most "evidence-based" techniques are attempts to approach this asymptote without the staffing cost |
| **Feedback science** (Hattie & Timperley 2007; Wisniewski et al. 2020) | Type / timing / specificity of feedback dominate effect sizes |
| **Pre-testing effect** (Carpenter, Richland) | Testing *before* studying primes attention. Distinct from retrieval practice |
---
## Cross-walk to CORE architecture
This is the load-bearing section: not "what does the literature
say" but "what does the literature say that maps onto a structural
move CORE could make."
| Pedagogy concept | CORE analog | Status |
|------------------|-------------|--------|
| Retrieval practice | `teaching/correction.py` + vault recall | **Structurally aligned.** Every reviewed correction IS a retrieval+strengthen event. Exact-recall ceiling already met. |
| Spaced retrieval | (none) | **Genuine gap.** No deterministic spaced re-verification of past corrections. → ADR-0129 |
| Interleaving | Cross-pack chains (ADR-0064 / 0067) | **Aligned.** Cross-pack chains force discrimination across domains. |
| Metacognition / calibration (prediction vs outcome) | (none at teaching layer; partial at runtime via ADR-0035) | **Genuine gap.** No prediction-vs-outcome capture in teaching loop. → ADR-0130 |
| Cognitive apprenticeship | Ratified packs as articulated expert ontology | **Strong analog.** Packs ARE the encoded expert representation; ratification IS the "fade scaffolding" step. |
| Worked examples → fading | Teaching corpora → unsupervised generation | **Partial.** Corpora encode correct answers; less so the reasoning chain that produced them. Could be more first-class. |
| Productive failure | Adversarial generation (ADR-0119.5) | **Different mechanism.** Adversarial generation is rejection; PF would mean attempt-before-grounding. Inverse of `wrong==0`. Not recommended for direct port. |
| Pre-testing | (none) | Genuine gap. CORE always grounds before articulating; never the reverse. Adopting would conflict with `wrong==0`; not recommended. |
| Self-explanation | `SolutionTrace` provenance chain | **Structurally present.** Every answer has its derivation. Could be more first-class in teaching-store records. |
| Cognitive load theory | Substrate hierarchy: algebra → field → vault → realizer | **Implicit alignment.** CORE's layering matches CLT separation of intrinsic structure from extraneous load. |
| Expertise reversal | Pack-tier promotion (ADR-0120) | **Already encoded.** The `apprentice / audit-passed / expert` contract already knows that what helps an apprentice can ossify an expert. |
| Desirable difficulties | `wrong == 0` discipline | **Inverse mapping.** CORE refuses *undesirable* difficulty (confabulation under uncertainty). A teaching-side concept of *desirable* difficulty (challenging-but-not-impossible curriculum sequencing) is not yet first-class. |
| Feedback science | `teaching/review.py` | **Partially aligned.** Reviewed corrections ARE structured feedback. Timing / specificity dimensions could be more first-class. |
---
## The two structural gaps worth addressing
Distilled from the cross-walk, two design moves are both
*pedagogically supported by robust literature* AND *consistent
with CORE's existing determinism + provenance discipline*:
### Gap 1 — Spaced reviewed-correction replay
**Mapped to:** retrieval-with-spacing literature (most robust
finding).
**ADR:** [ADR-0129](../decisions/ADR-0129-spaced-correction-replay-deferred.md)
**Status:** Deferred.
**Summary:** Periodic deterministic re-run of past reviewed
corrections to verify they still produce intended outcomes
under current state. Defense against silent regression as the
correction store and pack set evolves.
### Gap 2 — Pre-articulation calibration logging
**Mapped to:** metacognitive calibration / prediction-outcome
comparison literature.
**ADR:** [ADR-0130](../decisions/ADR-0130-pre-articulation-calibration-deferred.md)
**Status:** Deferred.
**Summary:** When a correction is proposed, log CORE's
pre-correction prediction; on acceptance, emit the gap.
Provides empirical answer to "is CORE actually getting better"
across pack-version cohorts; supports operator triage.
---
## What is NOT proposed (and why)
| Considered | Rejected because |
|------------|------------------|
| Adopt productive-failure mechanism inside CORE | Inverse of `wrong==0`; would require structural justification beyond pedagogy literature. Adversarial generation (ADR-0119.5) covers the related "wrong-answer rejection" use case without the conceptual conflict. |
| Adopt pre-testing in articulation | Same conflict with `wrong==0`. CORE grounds before articulating by design. |
| Add embodied / sensorimotor layer | No body. The structural analogy (substrate as "grounding") is too weak to load-bear. |
| Add peer-learning multi-agent loop | Out of scope. Multi-agent coordination is a separate architectural question; not driven by this pedagogy review. |
| Adopt cognitive-load-theory load-balancing in realizer | Already implicit in the substrate hierarchy. Making it more explicit risks decoration without integration. |
---
## Why both ADRs are deferred, not accepted
Both ADR-0129 and ADR-0130 are **proposed but deferred**, following
the established ADR-0121 / ADR-0122-deferred pattern. The deferral
reasons compose:
1. **Path-B uncertainty.** The active GSM8K-math arc
(ADR-0126 / 0127 / 0128) may resolve to a benchmark
re-targeting. If so, the correction-store population
characteristics change, and the right cadence (ADR-0129) /
cohort structure (ADR-0130) may differ.
2. **No observed incident.** Neither ADR has a triggering
incident. They're defensive infrastructure — useful if a
regression occurs (0129) or calibration drift develops (0130),
but speculative without that evidence.
3. **Cost/benefit unmeasured.** Both add telemetry volume and
operator review surface. Worth it only if the signal proves
load-bearing.
4. **Composition argument.** If either is un-deferred, the other
should be re-evaluated jointly — spaced-replay events
naturally yield calibration evidence; the two share
infrastructure. Deferring both together preserves that
composition.
The exit criteria for un-deferral are documented in each ADR's
"Exit criteria for un-deferral" section.
---
## Sequencing recommendation
1. Land ADR-0126 (PR #161) — architecture.
2. Land ADR-0127 (Gemini in flight) — units pack.
3. Land ADR-0128 (Opus #2 in flight) — numerics pack.
4. Re-run train sample with both packs mounted → real Path-A vs
Path-B verdict.
5. If Path A: continue along the math expert promotion path.
ADR-0129 / 0130 remain deferred until an incident or
bandwidth pressure surfaces them.
6. If Path B: benchmark re-targeting becomes the work; ADR-0129 /
0130 may become more relevant if the new benchmark's
correction-store characteristics are different enough to
warrant proactive verification.
---
## Reference list (additional to the original document)
- Sinha, T. & Kapur, M. (2021). When Problem Solving Followed by
Instruction Works: Evidence for Productive Failure.
[SAGE](https://journals.sagepub.com/doi/full/10.3102/00346543211019105)
- Pan, S. C. & Rickard, T. C. (2018). Transfer of test-enhanced
learning: meta-analytic review and synthesis. *Psychological Bulletin*.
- Glaser, J. & Richter, T. (2025). The Testing Effect in the
Lecture Hall: Does it Transfer to Content Studied but Not
Practiced? [Teaching of Psychology](https://journals.sagepub.com/doi/10.1177/00986283231218943)
- Cognitive Research: Principles and Implications (2024). Far
transfer of retrieval-practice benefits: rule-based learning
as the underlying mechanism.
[Springer](https://cognitiveresearchjournal.springeropen.com/articles/10.1186/s41235-024-00598-y)
- Machery, E. (2024). The Replication Crisis in Embodied Cognition
Research. *Routledge Handbook of Replication*.
[Taylor & Francis](https://www.taylorfrancis.com/chapters/edit/10.4324/9781003322511-50/replication-crisis-embodied-cognition-research-edouard-machery)
- Frontiers in Education (2026). Embodied cognition in STEM
learning: an integrative review.
[Frontiers](https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2026.1811569/full)
- Sinha & Kapur (2023). Prior math achievement and inventive
production predict learning from productive failure.
[PMC](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10185511/)
- Bloom, B. S. (1984). The 2 Sigma Problem.
*Educational Researcher 13(6)*.
- Hattie, J. & Timperley, H. (2007). The Power of Feedback.
*Review of Educational Research 77(1)*.
- Wisniewski, B., Zierer, K., Hattie, J. (2020). The Power of
Feedback Revisited: A Meta-Analysis.
- Ericsson, K. A., et al. (1993). The Role of Deliberate Practice
in the Acquisition of Expert Performance.
*Psychological Review 100(3)*.
- Sweller, J., van Merriënboer, J. J. G., Paas, F. G. W. C. (1998).
Cognitive Architecture and Instructional Design.
---
## Open questions surfaced (not resolved this session)
These are noted for future sessions; not items I'm advocating
for action:
1. **Should teaching-corpus records carry "why" structure, not
just "what"?** Self-explanation literature suggests reasoning
chains in corpora may be more useful than answers alone.
`SolutionTrace` already exposes provenance; pushing this into
teaching corpora is a separate question.
2. **Is there a deliberate-practice analog at the curriculum
level?** ADR-0120's promotion contract already encodes
"stretch-but-pass" structure (correct_rate ≥ 0.60 floor).
Whether sub-curricula should also encode this is open.
3. **Could the pack-mutation-proposal pathway adopt a worked-
example pattern?** When a pack mutation is proposed, today
the operator sees the diff; could they also see a small
worked example showing the behavioral implication?
Speculative.
4. **Is Bloom's 2-sigma a meaningful target for CORE?** A
deterministic engine with exact recall has structural
properties that may exceed 1:1 tutoring on some axes
(consistency, replay) while underperforming on others
(adaptation, social affordances). Whether to claim this
target is an architectural framing question, not a
technical one.
---
## End-of-session state
- **ADRs added:** 0129 (deferred), 0130 (deferred).
- **Session note:** this file.
- **Branch:** `docs/pedagogy-review-and-teaching-backlog`.
- **PR plan:** single docs-only PR for the three files; lands
independently of the in-flight ADR-0126 / 0127 / 0128 chain.
- **No code changes.** No regression risk.