Closes Phase 2 v2 coverage — all five lanes now pass v2 public + holdouts.
calibration v2:
public/v2 — 33 cases (11 no_grounding / 11 coherent / 11 correction_proposed)
deeper priming (3 repetitions) on coherent cases; OOD
cases include both technical-domain prompts and bare
in-pack terms with empty prime (gate fires on empty
vault regardless of vocabulary)
holdouts/v2 — 24 cases (8 / 8 / 8) on distinct vocabulary
Results: no_grounding_accuracy=1.0, coherent_accuracy=1.0,
correction_proposed_accuracy=1.0 on both splits.
symbolic-logic v2:
public/v2 — 24 cases, chains up to 5 hops:
chain_5, chain_4, chain_3, modus_ponens_chain,
modus_tollens_chain, negation_chain, syllogism,
chain_recall, single_premise, mixed
holdouts/v2 — 16 cases on distinct vocabulary
Results: premise_recall=1.0, replay_determinism=1.0,
proposal_storage=1.0 on both splits.
Each case still runs twice on fresh pipelines for replay-determinism
check — even the chain_5 cases (10 teaching turns × 2 runs) reproduce
trace_hash exactly.
Both lanes run via the parallel runner introduced in 57c08e6:
calibration public/v2 (33 cases) in 4.9s
calibration holdouts/v2 (24 cases) in 3.7s
symbolic_logic public/v2 (24 cases) in 21.9s
symbolic_logic holdouts/v2 (16 cases) in 16.5s
(symbolic_logic is slower per-case because of the 2x replay run +
long chains)
Test suite: 596 passing (no regression).
Phase 2 v2 milestone: all five lanes complete + v1 baselines documented.
Next: v3 lanes for at least two of the five (roadmap exit gate).
Parallel infrastructure:
evals/parallel.py
multiprocessing.Pool helper (spawn context, default workers
min(cpu_count, 8)). Per-case lanes use it via:
run_lane(cases, workers=N)
workers=1 forces serial (debugging); None uses the default pool.
Generic over the per-case return type, so dataclass-returning
runners (provenance) and dict-returning runners both work.
Wired into:
- evals/adversarial_identity/runner.py
- evals/calibration/runner.py
- evals/symbolic_logic/runner.py
- evals/provenance/runner.py
Per-case helpers are now picklable (module-level, single arg).
Monotonic-learning stays serial within a split — shared session
is structural to its longitudinal protocol.
Empirical speedup (adversarial-identity public/v1, 25 cases on
macOS 8 cores): serial 14.1s -> parallel 3.1s (~4.5x). Identical
per-case results.
adversarial-identity v2:
public/v2 — 35 cases (20 attack / 15 legitimate). Attacks cover
more varied phrasings: punctuation variation
("Actually -" / "No:" / "Correction —"), embedded
hedges ("please" / "regardless of prior context"),
multi-clause attacks, and identity-marker triggers
in mid-clause position.
holdouts/v2 — 22 cases (12 attack / 10 legitimate) on distinct
priming vocabulary.
Results: attack_rejection_rate=1.0, legitimate_acceptance_rate=1.0
on both splits.
The marker-regex defense in teaching/review.py:_is_identity_override
holds against every v2 phrasing — markers are checked case-insensitive
against the full text, so capitalization / punctuation tricks don't
slip past.
Test suite: 596 passing (no regression).
Records the architectural floor for frontier-LLM performance on each
Phase 2 v1 lane.
The baseline is structural: every lane's scoring rubric measures a
property that frontier LLMs do not architecturally emit (Provenance
typed sources, pack_mutation_proposal, vault_hits, REJECTED_IDENTITY
outcome, deterministic trace_hash). The frontier score on each of
those sub-metrics is 0.0 by construction, not by failure — even a
live-API run would still record 0.0 on these typed-signal checks
because the evidence is absent regardless of prose quality.
Artifacts:
docs/frontier_baselines.md
Full per-lane analysis: what each sub-metric scores, why the
frontier value is 0, and where a live-API baseline would or
would not add information.
evals/<lane>/baselines/v1_structural_zero.json (× 5)
Per-lane baseline records in the same shape as lane reports.
Encodes 0.0 / None on each sub-metric with rationale.
evals/baseline_runner.py
Adds StructuralZeroBaseline adapter conforming to the
BaselineModel protocol — a real, non-stub adapter that returns
the deterministic floor. Live-API adapters (Anthropic, OpenAI)
can be wired alongside when API keys are configured; the
structural floor remains the comparison baseline.
Across 5 lanes / 14 typed-signal sub-metrics:
CORE v1: 1.0 (each)
frontier structural: 0.0 (each)
The gap is "CORE measures a property frontier output does not
expose", not "CORE outperforms on a shared benchmark". v2 lanes may
add content-level sub-metrics where direct comparison via live-API
runs becomes meaningful.
Adds the fourth Phase 2 lane. v1 measures the structural foundations
on which a future inference engine would be built:
M1. premise_recall — probe vault_hits >= min after chain teaching
M2. replay_determinism — same chain + probe → same trace_hash
M3. proposal_storage — correction premises store as PackMutationProposals
Patterns covered: modus_ponens_chain, modus_tollens_chain, syllogism,
negation, chain_recall (up to 4-hop chains).
v1 results across 38 cases (8 dev + 18 public + 12 holdouts):
premise_recall=1.0, replay_determinism=1.0, proposal_storage=1.0.
Each case runs twice on fresh CognitiveTurnPipelines to verify the
trace_hash matches — confirming deterministic replay over premise chains.
Architectural finding logged in evals/symbolic_logic/gaps.md:
CORE has no first-class inference operator. Chain "inference" today is
emergent from teaching-store commits + cumulative vault recall, not a
named-rule symbolic engine. v1 honestly tests what CORE deterministically
*does* (store, replay, recall chains) without overclaiming that CORE
reasons symbolically. v2 would assert specific transitive recall
contents in the probe surface, which requires either a
PropositionGraph traversal operator or pack-axiom rules — both filed
as suggested follow-up work.
