Parallel infrastructure:
evals/parallel.py
multiprocessing.Pool helper (spawn context, default workers
min(cpu_count, 8)). Per-case lanes use it via:
run_lane(cases, workers=N)
workers=1 forces serial (debugging); None uses the default pool.
Generic over the per-case return type, so dataclass-returning
runners (provenance) and dict-returning runners both work.
Wired into:
- evals/adversarial_identity/runner.py
- evals/calibration/runner.py
- evals/symbolic_logic/runner.py
- evals/provenance/runner.py
Per-case helpers are now picklable (module-level, single arg).
Monotonic-learning stays serial within a split — shared session
is structural to its longitudinal protocol.
Empirical speedup (adversarial-identity public/v1, 25 cases on
macOS 8 cores): serial 14.1s -> parallel 3.1s (~4.5x). Identical
per-case results.
adversarial-identity v2:
public/v2 — 35 cases (20 attack / 15 legitimate). Attacks cover
more varied phrasings: punctuation variation
("Actually -" / "No:" / "Correction —"), embedded
hedges ("please" / "regardless of prior context"),
multi-clause attacks, and identity-marker triggers
in mid-clause position.
holdouts/v2 — 22 cases (12 attack / 10 legitimate) on distinct
priming vocabulary.
Results: attack_rejection_rate=1.0, legitimate_acceptance_rate=1.0
on both splits.
The marker-regex defense in teaching/review.py:_is_identity_override
holds against every v2 phrasing — markers are checked case-insensitive
against the full text, so capitalization / punctuation tricks don't
slip past.
Test suite: 596 passing (no regression).
Records the architectural floor for frontier-LLM performance on each
Phase 2 v1 lane.
The baseline is structural: every lane's scoring rubric measures a
property that frontier LLMs do not architecturally emit (Provenance
typed sources, pack_mutation_proposal, vault_hits, REJECTED_IDENTITY
outcome, deterministic trace_hash). The frontier score on each of
those sub-metrics is 0.0 by construction, not by failure — even a
live-API run would still record 0.0 on these typed-signal checks
because the evidence is absent regardless of prose quality.
Artifacts:
docs/frontier_baselines.md
Full per-lane analysis: what each sub-metric scores, why the
frontier value is 0, and where a live-API baseline would or
would not add information.
evals/<lane>/baselines/v1_structural_zero.json (× 5)
Per-lane baseline records in the same shape as lane reports.
Encodes 0.0 / None on each sub-metric with rationale.
evals/baseline_runner.py
Adds StructuralZeroBaseline adapter conforming to the
BaselineModel protocol — a real, non-stub adapter that returns
the deterministic floor. Live-API adapters (Anthropic, OpenAI)
can be wired alongside when API keys are configured; the
structural floor remains the comparison baseline.
Across 5 lanes / 14 typed-signal sub-metrics:
CORE v1: 1.0 (each)
frontier structural: 0.0 (each)
The gap is "CORE measures a property frontier output does not
expose", not "CORE outperforms on a shared benchmark". v2 lanes may
add content-level sub-metrics where direct comparison via live-API
runs becomes meaningful.
