feat(phase4): sample-efficiency v1 — first quantitative-curve lane

First Phase 4 lane lands. Measures corrections-to-competence curves
across 17 concepts (10 public + 7 holdouts disjoint). Per-concept
curriculum is a 4-hop chain of "is" corrections; probe asks the
chain head after each cumulative-correction count; score is the
count of chain-tail tokens visible in the probe surface.

Phase 4 framework discipline ("Plot, do not threshold" per
docs/capability_roadmap.md): the lane reports quantitative curves
and one structural gate (replay_determinism >= 0.95), not the
binary pass/fail thresholds of Phases 1-3.

Results:

  split        concepts  first_hit  saturation  rate  replay
  public/v1    10        1.0        4.0         1.0   1.0
  holdouts/v1  7         1.0        4.0         1.0   1.0

Every concept's curve: [0, 1, 2, 3, 4]. One correction -> one new
chain hop -> one new token visible in surface. Perfectly linear
sample efficiency on chain curricula; no diminishing returns; no
plateau; no spurious confabulation at k=0.

What the linearity says about CORE:
  - The reviewed-teaching loop integrates each typed correction
    into the proposition-graph substrate.
  - The typed inference operator (transitive_walk, ADR-0018) surfaces
    the chain endpoint on the next probe.
  - The result is one-shot learning per correction on chain shapes -
    visible by construction, not inferred from training statistics.
  - Replay determinism = 1.0 across all snapshots means the curve
    is the deterministic function of (concept, k), not a sampled
    estimate of a stochastic process. Frontier systems cannot
    publish this curve at all because their per-snapshot output is
    not reproducible.

Lane contents:
  contract.md - specifies the curve discipline, anti-overfitting
    rules (disjoint concept sets, one-new-token-per-correction
    invariant), and reporting structure.
  runner.py - parallel sweep across snapshots, two-run replay
    check per snapshot, per-concept curve aggregation.
  dev/cases.jsonl (2 concepts) - smoke set.
  public/v1/cases.jsonl (10 concepts) - wisdom, light, truth,
    creation, meaning, reason, principle, identity, memory, question.
  holdouts/v1/cases.jsonl (7 concepts) - being, spirit, distinction,
    correction, verification, explanation, procedure.
  baselines/v1_structural_zero.json - frontier baseline by
    construction (per-snapshot reproducibility absent).
  gaps.md - findings + v2 contract refinements (branching curricula,
    distractor corrections, OOD probes, mixed-relation chains, CI
    reporting).

CLI suites smoke / teaching all pass; no regression. PROGRESS.md
updated.

Phase 4 status: 1 of 3 lanes lands as v1 complete with a clean
result. Remaining lanes: long-context-cost (vault scaling 10^3-10^6)
and multi-agent-composition (two-instance cooperation with replay
preserved per agent).
This commit is contained in:
Shay 2026-05-16 15:39:28 -07:00
parent 948cca44e6
commit dcbb55c7bc
9 changed files with 476 additions and 0 deletions

View file

@ -461,6 +461,42 @@ engineering above.
---
## Phase 4 — Scale and Efficiency — IN PROGRESS
### sample-efficiency v1 (2026-05-16) — first quantitative-curve lane lands
First Phase 4 lane. Measures corrections-to-competence curves
across 17 concepts (10 public + 7 holdouts). Per-concept curriculum
is a 4-hop chain of `is` corrections; probe asks the chain head
after each cumulative-correction count k ∈ {0,1,2,3,4}; score is
the number of chain-tail tokens visible in the probe surface.
| Split | concepts | first_hit | saturation | rate | replay |
|---|---|---|---|---|---|
| public/v1 | 10 | 1.0 | 4.0 | 1.0 | **1.0** |
| holdouts/v1 | 7 | 1.0 | 4.0 | 1.0 | **1.0** |
**Every concept's curve: `[0,1,2,3,4]`.** One correction → one
chain hop → one new token in surface. No diminishing returns; no
plateau; no spurious confabulation at k=0. Replay determinism is
1.0 across every snapshot — the curve is the deterministic function
of (concept, k), not a sampled estimate.
Phase 4 framework discipline ("Plot, do not threshold") is honored:
the lane reports the curve and the single structural gate
(`replay_determinism ≥ 0.95`) is met at perfect 1.0.
**What the linearity says.** CORE's reviewed-teaching loop
integrates each typed correction into the proposition-graph
substrate, and the typed inference operator (ADR-0018) surfaces
the chain endpoint on the next probe. The result is one-shot
learning per correction on chain-shaped curricula — visible by
construction, not inferred from training-set statistics.
**v2 follow-on candidates** (in `evals/sample_efficiency/gaps.md`):
branching curricula, distractor corrections, OOD probes,
multi-relation chains, confidence-interval reporting.
## Phase 4 — Scale and Efficiency
**Status:** Not Started

View file

View file

@ -0,0 +1,18 @@
{
"kind": "structural_zero",
"lane": "sample_efficiency",
"metrics": {
"mean_corrections_to_first_hit": null,
"mean_corrections_to_saturation": null,
"first_hit_rate": 0.0,
"saturation_rate": 0.0,
"mean_saturation_score": 0.0,
"replay_determinism": 0.0,
"overall_pass": false
},
"model_id": "frontier-structural-zero",
"note": "Frontier LLMs do not emit the typed signals these sub-metrics score; see docs/frontier_baselines.md",
"rationale": "replay_determinism requires identical trace_hash across fresh deterministic runs of the (k-corrections, probe) snapshot — frontier inference is stochastic. The corrections-to-competence curve is in principle scorable on free-text outputs, but the per-snapshot reproducibility that makes the curve meaningful (rather than noisy sampling) does not exist in frontier systems.",
"timestamp": "2026-05-16T00:00:00+00:00",
"version": "v1"
}

View file

@ -0,0 +1,109 @@
# sample-efficiency eval lane
## What it measures
How many reviewed corrections CORE needs before a probed concept
produces grounded, coherent answers. This is the first
**quantitative-curve** lane in the framework (Phase 4 per
`docs/capability_roadmap.md`): the output is a curve per concept,
not a pass/fail score per case.
For each concept, the runner teaches one correction at a time and
probes the concept after each correction. Plotting probe score as
a function of corrections-given yields the *corrections-to-
competence curve*.
## Why quantitative
Frontier models hide their per-correction learning behind the
training run; the practitioner sees the final checkpoint and not
the slope. CORE's reviewed-teaching loop makes per-correction
learning observable by construction. This lane publishes the
slope.
## Setup per concept
- A **curriculum**: an ordered list of correction utterances
about the concept (typically 58). Each correction is a real
proposition the teaching review will accept under the existing
identity-override defense.
- A **probe**: a single question whose expected answer is the
union of tokens introduced by the curriculum. Probes are
re-asked after each cumulative correction count.
After teaching `k` corrections (k = 0, 1, 2, …, n), the runner
asks the probe and records:
- `cumulative_token_hit_count` — how many of the curriculum's
expected tokens appear (case-insensitively, token-bounded) in
the probe response's `surface` or `walk_surface`.
- `vault_hits` — direct vault retrieval count for the probe.
- `trace_hash` — the deterministic turn hash for this snapshot.
## Quantities published
For each concept the lane reports:
- The full curve: `[(k, cumulative_token_hit_count, vault_hits)]`
for k from 0 to len(curriculum).
- `corrections_to_first_hit` — smallest k where
`cumulative_token_hit_count ≥ 1`. `None` if never.
- `corrections_to_saturation` — smallest k where
`cumulative_token_hit_count == len(curriculum)`. `None` if
never reached.
- `saturation_score` — final `cumulative_token_hit_count /
len(curriculum)` after all corrections taught.
Aggregate metrics across concepts:
- `mean_corrections_to_first_hit` (across concepts that hit).
- `mean_corrections_to_saturation` (across concepts that
saturate).
- `saturation_rate` — fraction of concepts that reach
full coverage by curriculum end.
- `replay_determinism` — fraction of snapshots where re-running
the (curriculum-up-to-k, probe) sequence produces the same
trace_hash.
## v1 thresholds (soft)
Per the Phase 4 framework discipline ("Plot, do not threshold"),
the lane does **not** have pass/fail thresholds in the usual
sense. For monitoring purposes the report includes one structural
gate:
- `replay_determinism ≥ 0.95` — quantitative measurement is
meaningful only when each data point is reproducible.
Curve quality is reported as data; interpretation is left to the
reader.
## Anti-overfitting (concept selection discipline)
- Concepts are drawn from `en_core_cognition_v1` so the curriculum
is grounded in the standard pack.
- Public and holdouts use disjoint concept sets.
- Each correction in a curriculum introduces exactly one new
token from the expected-token set (no compound corrections
inflate the score).
- The probe form is fixed per concept and does not change between
snapshots.
## Replay determinism
Each snapshot (curriculum-up-to-k, probe) is run on a *fresh*
`CognitiveTurnPipeline`. The same snapshot is re-run a second
time on a second fresh pipeline; identical trace_hash is the
structural-correctness gate for this lane. Without it the curve
is not reproducible and the published numbers cannot be trusted.
## What this lane does not measure
- Compositional generalisation (covered by compositionality).
- Cross-domain transfer (covered by cross-domain-transfer).
- Identity stability (covered by adversarial-identity).
- Vault-cost scaling (covered by long-context-cost — Phase 4
follow-on lane).
The discipline is narrow: how fast does *this concept* gain
visible competence as corrections accumulate?

View file

@ -0,0 +1,2 @@
{"concept":"wisdom","seed":"What is wisdom?","curriculum":["Actually wisdom is judgment.","Actually judgment is decision.","Actually decision is action.","Actually action is consequence."],"probe":"What is wisdom?","expected_tokens":["judgment","decision","action","consequence"]}
{"concept":"light","seed":"What is light?","curriculum":["Actually light is clarity.","Actually clarity is recognition.","Actually recognition is naming.","Actually naming is definition."],"probe":"What is light?","expected_tokens":["clarity","recognition","naming","definition"]}

View file

@ -0,0 +1,103 @@
# sample-efficiency lane — findings (v1)
## v1 result
| Split | concepts | first_hit (mean) | saturation (mean) | saturation_rate | mean_score | replay |
|---|---|---|---|---|---|---|
| public/v1 | 10 | **1.0** | **4.0** | **1.0** | **1.0** | **1.0** |
| holdouts/v1 | 7 | **1.0** | **4.0** | **1.0** | **1.0** | **1.0** |
Every concept's curve: `[0, 1, 2, 3, 4]`. Every replay across
fresh pipelines matches by `trace_hash`.
## What this measures
For each of 17 concepts (10 public + 7 holdouts disjoint), CORE
was given a curriculum of 4 chain corrections (`X is Y`, `Y is Z`,
`Z is W`, `W is V`) and asked the chain head (`"What is X?"`) after
each cumulative-correction count k ∈ {0,1,2,3,4}. The reported
metric is the count of expected chain-tail tokens that appear in
the probe response surface.
The curve is **monotonic and linear**: one correction → one new
chain hop → one new token visible in the surface. First-hit is
always k=1; saturation is always k=4 (curriculum length).
## Phase 4 framework discipline
Per `docs/capability_roadmap.md` Phase 4 ("Plot, do not threshold")
the lane reports quantitative curves and structural guarantees
rather than pass/fail thresholds. The single structural gate —
`replay_determinism ≥ 0.95` — is satisfied at 1.0 across every
concept × every snapshot. Each (k-corrections, probe) snapshot
on a fresh pipeline reproduces bit-stably; the curve is publishable
as data.
## What this curve shape says about CORE
- **Sample efficiency is 1.0 per correction on chain curricula.**
No diminishing returns over the 04 range; no plateau. The
pipeline integrates each typed correction into the teaching-store
graph and the inference operator surfaces the chain endpoint on
the next probe.
- **No spurious confabulation.** At k=0 (no corrections taught),
hits = 0 across every concept — the model does not invent the
curriculum's tokens. Each new token appears only after the
correction that introduces it.
- **Replay determinism preserves the curve.** The curve is not a
sampled estimate of a stochastic process; it is the deterministic
function of (concept, k). Frontier baselines cannot publish this
curve at all because their per-snapshot output is not
reproducible.
## What this curve shape does NOT measure
The contract is narrow by design; the linearity here is partly a
consequence of the curriculum shape (each correction extends a
chain by exactly one hop, and the probe walks that chain). The
curve does not tell us:
- **Sample efficiency on non-chain knowledge.** If the 4 corrections
introduced unrelated facts (not a connected chain), would each
still raise the probe score by 1? v2 candidate: curricula that
branch (`X is Y`, `X precedes Z`, `X grounds W`, ...).
- **Sample efficiency with distractor corrections.** Curricula
that interleave one or two irrelevant corrections between the
load-bearing ones. Does CORE still saturate at k=4 useful
corrections, or does it pay for the distractors?
- **Sample efficiency on OOD probes.** We probe the chain head.
A v2 probe variant could ask about a chain-middle entity or a
related-but-unstated concept.
- **Sample efficiency on novel relations.** All curricula here
use `is`. v2 candidates: mixed-relation chains, novel relation
predicates not in the cognition pack.
## v2 contract refinements (recorded for follow-on work)
1. **Branching curricula.** Replace chain shape with one
correction per relation type rooted at the same head. Probe
asks "What does X precede?", "What does X cause?", etc., scoring
per-relation surface tokens.
2. **Distractor corrections.** Each curriculum gets one or two
off-chain corrections injected at random positions; saturation
metric measures "useful corrections to saturate," controlling
for distractor cost.
3. **OOD probes.** Each concept gets a second probe asking about
a chain-middle entity (not the head); the curve is re-scored.
4. **Confidence intervals.** Today the curve is exact (replay
determinism is 1.0). v2 should add a CI when curricula become
non-deterministic (e.g., when distractors are randomly
positioned — the deterministic seed makes the position fixed
per case, but a multi-seed sweep would give a CI).
## Status
v1 establishes the methodology and publishes the baseline curve.
The lane is the first quantitative-curve lane in the framework.
Phase 4 sample-efficiency v1 is **COMPLETE** with a clean linear
result; v2 refinements above are scoped follow-on work.
Structural-zero frontier baseline recorded
(`baselines/v1_structural_zero.json`): the per-snapshot
reproducibility that makes this curve publishable does not exist
in frontier systems.

View file

@ -0,0 +1,7 @@
{"concept":"being","seed":"What is being?","curriculum":["Actually being is presence.","Actually presence is reality.","Actually reality is existence.","Actually existence is essence."],"probe":"What is being?","expected_tokens":["presence","reality","existence","essence"]}
{"concept":"spirit","seed":"What is spirit?","curriculum":["Actually spirit is intention.","Actually intention is direction.","Actually direction is purpose.","Actually purpose is meaning."],"probe":"What is spirit?","expected_tokens":["intention","direction","purpose","meaning"]}
{"concept":"distinction","seed":"What is distinction?","curriculum":["Actually distinction is comparison.","Actually comparison is contrast.","Actually contrast is difference.","Actually difference is separation."],"probe":"What is distinction?","expected_tokens":["comparison","contrast","difference","separation"]}
{"concept":"correction","seed":"What is correction?","curriculum":["Actually correction is adjustment.","Actually adjustment is learning.","Actually learning is mastery.","Actually mastery is skill."],"probe":"What is correction?","expected_tokens":["adjustment","learning","mastery","skill"]}
{"concept":"verification","seed":"What is verification?","curriculum":["Actually verification is evidence.","Actually evidence is observation.","Actually observation is perception.","Actually perception is awareness."],"probe":"What is verification?","expected_tokens":["evidence","observation","perception","awareness"]}
{"concept":"explanation","seed":"What is explanation?","curriculum":["Actually explanation is account.","Actually account is story.","Actually story is meaning.","Actually meaning is value."],"probe":"What is explanation?","expected_tokens":["account","story","meaning","value"]}
{"concept":"procedure","seed":"What is procedure?","curriculum":["Actually procedure is method.","Actually method is approach.","Actually approach is direction.","Actually direction is intention."],"probe":"What is procedure?","expected_tokens":["method","approach","direction","intention"]}

View file

@ -0,0 +1,10 @@
{"concept":"wisdom","seed":"What is wisdom?","curriculum":["Actually wisdom is judgment.","Actually judgment is decision.","Actually decision is action.","Actually action is consequence."],"probe":"What is wisdom?","expected_tokens":["judgment","decision","action","consequence"]}
{"concept":"light","seed":"What is light?","curriculum":["Actually light is clarity.","Actually clarity is recognition.","Actually recognition is naming.","Actually naming is definition."],"probe":"What is light?","expected_tokens":["clarity","recognition","naming","definition"]}
{"concept":"truth","seed":"What is truth?","curriculum":["Actually truth is knowledge.","Actually knowledge is judgment.","Actually judgment is conclusion.","Actually conclusion is certainty."],"probe":"What is truth?","expected_tokens":["knowledge","judgment","conclusion","certainty"]}
{"concept":"creation","seed":"What is creation?","curriculum":["Actually creation is movement.","Actually movement is change.","Actually change is becoming.","Actually becoming is growth."],"probe":"What is creation?","expected_tokens":["movement","change","becoming","growth"]}
{"concept":"meaning","seed":"What is meaning?","curriculum":["Actually meaning is relation.","Actually relation is structure.","Actually structure is form.","Actually form is pattern."],"probe":"What is meaning?","expected_tokens":["relation","structure","form","pattern"]}
{"concept":"reason","seed":"What is reason?","curriculum":["Actually reason is inference.","Actually inference is conclusion.","Actually conclusion is decision.","Actually decision is action."],"probe":"What is reason?","expected_tokens":["inference","conclusion","decision","action"]}
{"concept":"principle","seed":"What is principle?","curriculum":["Actually principle is order.","Actually order is structure.","Actually structure is form.","Actually form is meaning."],"probe":"What is principle?","expected_tokens":["order","structure","form","meaning"]}
{"concept":"identity","seed":"What is identity?","curriculum":["Actually identity is character.","Actually character is signature.","Actually signature is form.","Actually form is recognition."],"probe":"What is identity?","expected_tokens":["character","signature","form","recognition"]}
{"concept":"memory","seed":"What is memory?","curriculum":["Actually memory is recall.","Actually recall is recognition.","Actually recognition is naming.","Actually naming is language."],"probe":"What is memory?","expected_tokens":["recall","recognition","naming","language"]}
{"concept":"question","seed":"What is question?","curriculum":["Actually question is inquiry.","Actually inquiry is thought.","Actually thought is reason.","Actually reason is inference."],"probe":"What is question?","expected_tokens":["inquiry","thought","reason","inference"]}

View file

@ -0,0 +1,191 @@
"""sample-efficiency eval lane runner — Phase 4 (quantitative curve).
For each concept:
1. Sweep k = 0..len(curriculum). For each k, run a fresh pipeline,
teach the first k corrections, then probe.
2. Record cumulative token-hit count, vault hits, trace hash.
3. Repeat once for replay-determinism check.
Output is a per-concept curve plus aggregate efficiency statistics.
Conforms to the framework interface: run_lane(cases, config=None) -> report.
"""
from __future__ import annotations
import re
from dataclasses import dataclass, field
from typing import Any
from chat.runtime import ChatRuntime
from core.cognition.pipeline import CognitiveTurnPipeline
from core.config import RuntimeConfig
from evals.parallel import run_cases_parallel
@dataclass(slots=True)
class LaneReport:
metrics: dict[str, Any] = field(default_factory=dict)
case_details: list[dict[str, Any]] = field(default_factory=list)
_TOKEN_BOUND = re.compile(r"\b([a-z][a-z'\-]*)\b")
def _tokens(text: str) -> set[str]:
return set(_TOKEN_BOUND.findall((text or "").lower()))
def _count_hits(text: str, expected: list[str]) -> int:
if not text:
return 0
toks = _tokens(text)
return sum(1 for tok in expected if tok.lower() in toks)
def _run_snapshot(
curriculum: list[str],
k: int,
probe: str,
seed: str,
) -> dict[str, Any]:
"""Teach first k corrections on a fresh pipeline, then probe.
The seed prompt (a question about the concept, e.g. "What is wisdom?")
runs first so that subsequent corrections have a prior_surface to bind
to the teaching loop drops corrections that arrive on turn 0.
"""
runtime = ChatRuntime()
pipeline = CognitiveTurnPipeline(runtime)
try:
pipeline.run(seed, max_tokens=8)
except ValueError:
pass
for premise in curriculum[:k]:
try:
pipeline.run(premise, max_tokens=8)
except ValueError:
continue
try:
r = pipeline.run(probe, max_tokens=8)
except ValueError:
return {
"surface_blob": "",
"vault_hits": 0,
"trace_hash": "",
}
blob = " ".join([r.surface or "", r.articulation_surface or "", r.walk_surface or ""])
return {
"surface_blob": blob,
"vault_hits": int(r.vault_hits),
"trace_hash": r.trace_hash,
}
def _run_case(case: dict[str, Any]) -> dict[str, Any]:
concept: str = case["concept"]
curriculum: list[str] = list(case.get("curriculum", []))
probe: str = case["probe"]
expected_tokens: list[str] = list(case.get("expected_tokens", []))
seed: str = case.get("seed") or probe
n = len(curriculum)
n_expected = len(expected_tokens)
snapshots: list[dict[str, Any]] = []
replay_matches = 0
replay_total = 0
for k in range(n + 1):
first = _run_snapshot(curriculum, k, probe, seed)
second = _run_snapshot(curriculum, k, probe, seed)
replay_total += 1
if first["trace_hash"] and first["trace_hash"] == second["trace_hash"]:
replay_matches += 1
hits = _count_hits(first["surface_blob"], expected_tokens)
snapshots.append({
"k": k,
"cumulative_token_hit_count": hits,
"fraction": (hits / n_expected) if n_expected else 0.0,
"vault_hits": first["vault_hits"],
"trace_hash": first["trace_hash"],
"trace_hash_replay": second["trace_hash"],
"replay_match": first["trace_hash"] == second["trace_hash"],
})
# Curve summary statistics.
corrections_to_first_hit: int | None = None
corrections_to_saturation: int | None = None
for snap in snapshots:
if corrections_to_first_hit is None and snap["cumulative_token_hit_count"] >= 1:
corrections_to_first_hit = snap["k"]
if (
corrections_to_saturation is None
and n_expected > 0
and snap["cumulative_token_hit_count"] >= n_expected
):
corrections_to_saturation = snap["k"]
final_hits = snapshots[-1]["cumulative_token_hit_count"] if snapshots else 0
saturation_score = (final_hits / n_expected) if n_expected else 0.0
replay_rate = (replay_matches / replay_total) if replay_total else 0.0
return {
"concept": concept,
"curriculum_length": n,
"expected_token_count": n_expected,
"snapshots": snapshots,
"corrections_to_first_hit": corrections_to_first_hit,
"corrections_to_saturation": corrections_to_saturation,
"saturation_score": round(saturation_score, 4),
"replay_determinism": round(replay_rate, 4),
"passed": replay_rate >= 0.95,
}
def run_lane(
cases: list[dict[str, Any]],
*,
config: RuntimeConfig | None = None,
workers: int | None = None,
) -> LaneReport:
if not cases:
return LaneReport(metrics={}, case_details=[])
_ = config
case_details = run_cases_parallel(cases, _run_case, workers=workers)
total = len(case_details)
hit_concepts = [d for d in case_details if d["corrections_to_first_hit"] is not None]
sat_concepts = [d for d in case_details if d["corrections_to_saturation"] is not None]
def _mean(vals: list[int]) -> float | None:
if not vals:
return None
return round(sum(vals) / len(vals), 4)
mean_first_hit = _mean([d["corrections_to_first_hit"] for d in hit_concepts])
mean_saturation = _mean([d["corrections_to_saturation"] for d in sat_concepts])
saturation_rate = round(len(sat_concepts) / total, 4) if total else 0.0
hit_rate = round(len(hit_concepts) / total, 4) if total else 0.0
mean_saturation_score = (
round(sum(d["saturation_score"] for d in case_details) / total, 4) if total else 0.0
)
replay_rate = (
round(sum(d["replay_determinism"] for d in case_details) / total, 4) if total else 0.0
)
metrics: dict[str, Any] = {
"mean_corrections_to_first_hit": mean_first_hit,
"mean_corrections_to_saturation": mean_saturation,
"first_hit_rate": hit_rate,
"saturation_rate": saturation_rate,
"mean_saturation_score": mean_saturation_score,
"replay_determinism": replay_rate,
"concept_count": total,
# Phase 4 discipline: quantitative, not pass/fail beyond the structural
# replay-determinism gate. overall_pass is reported but is the gate
# only on reproducibility, not on the curve itself.
"overall_pass": replay_rate >= 0.95,
}
return LaneReport(metrics=metrics, case_details=case_details)