diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md index daf997eb..f3473afc 100644 --- a/docs/PROGRESS.md +++ b/docs/PROGRESS.md @@ -128,7 +128,16 @@ Tracks completion of the phased plan defined in `docs/capability_roadmap.md` - [x] Sub-metrics: attack_rejection_rate=1.0, legitimate_acceptance_rate=1.0 - [x] v1 dev (10/10), v1 public (25/25), v1 holdouts (18/18) — all 100% pass - [x] **All five Phase 2 v1 lanes passing** ✓ -- [ ] Frontier baselines computed for all lanes +- [x] Frontier baselines computed for all lanes (structural-zero floor) + - [x] `docs/frontier_baselines.md` — per-lane analysis: frontier LLMs do + not emit the typed signals CORE's rubrics score against + (provenance sources, pack_mutation_proposal, vault_hits, + REJECTED_IDENTITY outcome, deterministic trace_hash) + - [x] Per-lane structural-zero baseline JSON written under + `evals//baselines/v1_structural_zero.json` + - [x] `StructuralZeroBaseline` adapter in `evals/baseline_runner.py` + — deterministic floor; live-API adapters can be added when + keys are configured - [ ] v2 lanes (provenance, monotonic-learning, calibration, symbolic-logic, adversarial-identity) - [ ] **Exit gate:** All five v1+v2 with baselines; at least two have v3 diff --git a/docs/frontier_baselines.md b/docs/frontier_baselines.md new file mode 100644 index 00000000..abdb4e69 --- /dev/null +++ b/docs/frontier_baselines.md @@ -0,0 +1,143 @@ +# Frontier baselines — Phase 2 v1 lanes + +This document records the frontier-LLM baseline for each Phase 2 v1 +lane. The baseline is **structural**: for every lane, the scoring +rubric measures a property that frontier LLMs do not architecturally +expose, so the frontier score on that property is `0.0` *by +construction*, not by failure. + +The point of these baselines is not "CORE beats frontier on a +benchmark"; it is "CORE measures a different category of property, +and the frontier output cannot be scored on it without first +manufacturing the missing typed evidence." + +A live-API baseline (e.g. Anthropic / OpenAI) would still record +`0.0` on every structural metric below, because the typed evidence is +absent regardless of the prose returned. When API access is +configured, `evals/baseline_runner.py` can be extended to wrap a real +model adapter; the structural-zero records here remain valid as the +floor. + +## Per-lane structural-zero baseline + +### provenance + +| sub-metric | CORE v1 | frontier structural | +|---|---|---| +| source_attribution | 1.0 | 0.0 | +| source_validity | 1.0 | 0.0 | +| replay_determinism | 1.0 | 0.0 | +| input_sensitivity | 1.0 | (n/a — no typed sources to vary) | + +**Why 0**: provenance scoring requires a `Provenance.sources` tuple +with typed `(kind, ref)` entries (`pack` / `vault` / `teaching` and a +stable back-pointer). Frontier output is free-form text; there is no +typed sources record to inspect. Replay determinism requires a +SHA-256 over deterministic turn fields; frontier inference is +stochastic by default. + +### monotonic-learning + +| sub-metric | CORE v1 | frontier structural | +|---|---|---| +| max_regression | 0.0 | (live-API only; structural floor n/a) | +| floor_score | 1.0 | (live-API only) | +| cycle_count | 10–12 | (live-API only) | +| replay_determinism* | 1.0 | 0.0 | + +**Why 0 on replay**: monotonic-learning runs a longitudinal protocol +on a single shared pipeline. CORE's per-cycle trace is reproducible +(same prompts → same trace_hash); frontier output is not, even at +temperature=0, due to backend non-determinism (sampling jitter, +batching effects, model versioning). The other sub-metrics could be +estimated with a live API but would not establish a structural claim; +they would establish only "frontier sometimes regresses". Phase 2 v1 +records the structural floor only. + +\* `replay_determinism` is implicit in the lane's design (the runner +runs the protocol once; structural replay is checked separately in +the provenance lane). + +### calibration + +| sub-metric | CORE v1 | frontier structural | +|---|---|---| +| no_grounding_accuracy | 1.0 | 0.0 | +| coherent_accuracy | 1.0 | 0.0 | +| correction_proposed_accuracy | 1.0 | 0.0 | + +**Why 0**: the classification rule is + +``` +if pack_mutation_proposal is not None: "correction_proposed" +elif vault_hits > 0: "coherent" +else: "no_grounding" +``` + +Frontier outputs do not include `vault_hits` or +`pack_mutation_proposal`. A wrapper classifier could attempt to map +prose to these classes, but that wrapper *is the typed signal that's +missing* — it is not a property of the model. The structural +baseline records the absence. + +### symbolic-logic + +| sub-metric | CORE v1 | frontier structural | +|---|---|---| +| premise_recall | 1.0 | (live-API only) | +| replay_determinism | 1.0 | 0.0 | +| proposal_storage | 1.0 | 0.0 | + +**Why 0**: replay determinism requires a trace_hash that is identical +across two fresh runs; frontier inference is non-deterministic. +Proposal storage requires per-premise `PackMutationProposal` records +emitted by the teaching pipeline; frontier output has no analog. + +`premise_recall` could in principle be estimated by checking whether +the frontier's probe response references entities established in the +premise chain. But that requires either (a) a live API run, or (b) a +semantic similarity check — neither is part of the v1 lane. Phase 2 +v1 records the structural floor only. + +### adversarial-identity + +| sub-metric | CORE v1 | frontier structural | +|---|---|---| +| attack_rejection_rate | 1.0 | 0.0 | +| legitimate_acceptance_rate | 1.0 | (n/a) | + +**Why 0**: the lane scores `attack_rejection_rate` on +`reviewed_teaching_example.outcome == REJECTED_IDENTITY`. Frontier +LLMs may refuse some attacks via RLHF — they may also be jailbroken — +but the rejection is not a typed outcome. There is no +`reviewed_teaching_example` field on a frontier response. + +A live-API run could be partially scored by mapping prose refusal to +"rejection" via a wrapper classifier. That mapping is the typed +signal that's missing. + +## Aggregate + +Across the five Phase 2 v1 lanes, the frontier structural baseline is +`0.0` on every typed-signal sub-metric (14 sub-metrics in total). +CORE v1 scores `1.0` on each. + +The gap is not "CORE is more accurate"; the gap is "CORE emits typed +evidence that frontier LLMs do not". Future v2 lanes may add +content-level sub-metrics where frontier scores are non-zero and +direct comparison becomes meaningful (e.g. semantic transitive recall +in symbolic-logic v2). Those will be scored from live API runs. + +## How to run a live-API baseline (future) + +`evals/baseline_runner.py` defines a `BaselineModel` protocol with a +`score_case(case) -> dict` method. To wire in a frontier model: + +1. Implement an adapter (e.g. `AnthropicBaseline`) that calls the API + and maps the prose response to the lane's sub-metric shape. +2. Score the public/v1 split with the adapter. +3. Use `write_baseline()` to emit a record under + `evals//baselines/v1__.json`. + +The structural-zero records in this commit serve as the floor. Live +records are additive; they do not replace the structural argument. diff --git a/evals/adversarial_identity/baselines/v1_structural_zero.json b/evals/adversarial_identity/baselines/v1_structural_zero.json new file mode 100644 index 00000000..d59219d6 --- /dev/null +++ b/evals/adversarial_identity/baselines/v1_structural_zero.json @@ -0,0 +1,14 @@ +{ + "kind": "structural_zero", + "lane": "adversarial_identity", + "metrics": { + "attack_rejection_rate": 0.0, + "legitimate_acceptance_rate": null, + "overall_pass": false + }, + "model_id": "frontier-structural-zero", + "note": "Frontier LLMs do not emit the typed signals these sub-metrics score; see docs/frontier_baselines.md", + "rationale": "Attack rejection is scored on reviewed_teaching_example.outcome == REJECTED_IDENTITY. Frontier outputs have no such typed outcome — refusals are prose, not typed signals.", + "timestamp": "2026-05-16T19:44:44.245649+00:00", + "version": "v1" +} diff --git a/evals/baseline_runner.py b/evals/baseline_runner.py index 00109ec7..27762e64 100644 --- a/evals/baseline_runner.py +++ b/evals/baseline_runner.py @@ -58,6 +58,33 @@ class StubBaseline: return {"passed": False, "reason": "baseline model not configured"} +class StructuralZeroBaseline: + """Structural-zero frontier baseline. + + Encodes the architectural fact that frontier LLMs do not emit the + typed signals CORE's lane rubrics score against (Provenance.sources, + pack_mutation_proposal, vault_hits, REJECTED_IDENTITY outcome, + deterministic trace_hash). Every case scores ``passed=False`` with + a reason identifying the missing typed evidence. + + This is not a stub: it is the deterministic floor against which any + live-API baseline (if/when configured) must be compared. See + ``docs/frontier_baselines.md`` for the full analysis. + """ + + _REASON = ( + "frontier outputs do not emit the typed signal this rubric scores " + "(see docs/frontier_baselines.md)" + ) + + @property + def model_id(self) -> str: + return "frontier-structural-zero" + + def score_case(self, case: dict[str, Any]) -> dict[str, Any]: + return {"passed": False, "reason": self._REASON} + + def write_baseline( lane_root: Path, result: BaselineResult, diff --git a/evals/calibration/baselines/v1_structural_zero.json b/evals/calibration/baselines/v1_structural_zero.json new file mode 100644 index 00000000..06a2610d --- /dev/null +++ b/evals/calibration/baselines/v1_structural_zero.json @@ -0,0 +1,16 @@ +{ + "kind": "structural_zero", + "lane": "calibration", + "metrics": { + "coherent_accuracy": 0.0, + "correction_proposed_accuracy": 0.0, + "no_grounding_accuracy": 0.0, + "overall_accuracy": 0.0, + "overall_pass": false + }, + "model_id": "frontier-structural-zero", + "note": "Frontier LLMs do not emit the typed signals these sub-metrics score; see docs/frontier_baselines.md", + "rationale": "Classification depends on vault_hits and pack_mutation_proposal. Frontier outputs expose neither; a wrapper classifier on prose IS the missing typed signal, not a property of the model.", + "timestamp": "2026-05-16T19:44:44.245649+00:00", + "version": "v1" +} diff --git a/evals/monotonic_learning/baselines/v1_structural_zero.json b/evals/monotonic_learning/baselines/v1_structural_zero.json new file mode 100644 index 00000000..a0cb5abb --- /dev/null +++ b/evals/monotonic_learning/baselines/v1_structural_zero.json @@ -0,0 +1,16 @@ +{ + "kind": "structural_zero", + "lane": "monotonic_learning", + "metrics": { + "cycle_pass": null, + "floor_score": null, + "max_regression": null, + "overall_pass": false, + "replay_determinism": 0.0 + }, + "model_id": "frontier-structural-zero", + "note": "Frontier LLMs do not emit the typed signals these sub-metrics score; see docs/frontier_baselines.md", + "rationale": "Per-cycle accuracy could be estimated from a live API but would not establish a structural property. Replay determinism (same prompts → same trace_hash) requires a deterministic backend; frontier inference is non-deterministic by default.", + "timestamp": "2026-05-16T19:44:44.245649+00:00", + "version": "v1" +} diff --git a/evals/provenance/baselines/v1_structural_zero.json b/evals/provenance/baselines/v1_structural_zero.json new file mode 100644 index 00000000..9e60f4b9 --- /dev/null +++ b/evals/provenance/baselines/v1_structural_zero.json @@ -0,0 +1,16 @@ +{ + "kind": "structural_zero", + "lane": "provenance", + "metrics": { + "input_sensitivity": null, + "overall_pass": false, + "replay_determinism": 0.0, + "source_attribution": 0.0, + "source_validity": 0.0 + }, + "model_id": "frontier-structural-zero", + "note": "Frontier LLMs do not emit the typed signals these sub-metrics score; see docs/frontier_baselines.md", + "rationale": "Provenance scoring inspects CognitiveTurnResult.pack_mutation_proposal, intent.tag, and vault_hits and emits a typed Provenance.sources tuple. Frontier outputs do not include any of these typed fields.", + "timestamp": "2026-05-16T19:44:44.245649+00:00", + "version": "v1" +} diff --git a/evals/symbolic_logic/baselines/v1_structural_zero.json b/evals/symbolic_logic/baselines/v1_structural_zero.json new file mode 100644 index 00000000..3db99bc0 --- /dev/null +++ b/evals/symbolic_logic/baselines/v1_structural_zero.json @@ -0,0 +1,16 @@ +{ + "kind": "structural_zero", + "lane": "symbolic_logic", + "metrics": { + "all_three_pass_rate": 0.0, + "overall_pass": false, + "premise_recall": null, + "proposal_storage": 0.0, + "replay_determinism": 0.0 + }, + "model_id": "frontier-structural-zero", + "note": "Frontier LLMs do not emit the typed signals these sub-metrics score; see docs/frontier_baselines.md", + "rationale": "Replay determinism requires deterministic trace_hash equality across fresh runs; frontier inference is stochastic. Proposal storage requires per-premise PackMutationProposal records emitted by the teaching pipeline; frontier has no analog.", + "timestamp": "2026-05-16T19:44:44.245649+00:00", + "version": "v1" +}