#58 shipped providers.py + model_registry.py for cross-provider
benchmarking but never connected them to runner.py — the adapters
sat unused. This PR wires them through with a clear lane split.
Why a new suite instead of refactoring existing ones
-----------------------------------------------------
The three existing suites (determinism / truth_lock / axis_orthogonality)
pull CORE-only telemetry: trace_hash, versor_condition, register_id,
register_variant_id, anchor_lens_id, register_canonical_surface.
None of those fields can come from OpenAI / Anthropic / Ollama.
Forcing those suites cross-provider would silently produce reports
where the cross-provider rows have empty telemetry — a worse failure
mode than not running them at all. So the routing is explicit:
CORE-only suites → --provider must be 'core'
Cross-provider suites → any provider; CORE is one adapter among many
Operator asks for the wrong combo → loud error with the right alternative.
New module: evals/frontier_compare/cross_provider.py
-----------------------------------------------------
- ProviderObservation dataclass — provider-agnostic observation shape
(prompt, surface, provider, model, elapsed_ms, error fields). No
CORE-internal telemetry expected.
- run_prompt_battery(adapter, *, cfg) → SuiteReport reusing existing
CaseResult / SuiteReport shapes so the report viewer renders both
lanes without schema branching.
- _PROMPT_BATTERY: 7 fixed cases spanning definition / cause /
verification / comparison / procedure / unknown intent shapes.
Stable case_ids so future re-runs against the same provider produce
diffable JSON.
- Per-case 'passed' is loose by design (non-empty surface, no
exception). Cross-provider quality is for human review — not for
the runner to silently score.
Updated CLI: evals/frontier_compare/__main__.py
-----------------------------------------------
- --provider {core, openai, anthropic, ollama} (default: core)
- --model <id> (validated via require_model_card)
- --env-file <path> (default: ./.env)
- Auto-persist non-CORE runs to
evals/frontier_compare/results/<provider>_<model>_<utc>.json
even when --report is omitted. API calls are rate-limited / paid;
losing the artifact is costly.
- Existing CORE-native behavior unchanged when --provider not set.
Results directory: evals/frontier_compare/results/
--------------------------------------------------
Created with .gitkeep — matches the convention used by other lanes
(evals/long_context_cost/results/, evals/koine_greek_fluency/results/,
etc.). Distinct from reports/ which .gitignore excludes for
transient debug output.
Tests: tests/test_frontier_compare_cross_provider.py (9 cases)
--------------------------------------------------------------
- prompt_battery runs with CORE adapter (no API needed)
- adapter exceptions recorded as failed observations, never propagated
- empty surfaces flagged distinctly from adapter errors
- CLI default runs CORE-native (no breaking change)
- CLI prompt_battery with --provider core routes through cross-provider path
- CLI rejects CORE-only suite + non-CORE provider with operator-helpful error
- --help surfaces both suite families
- unregistered model is rejected before any benchmark cycles burn
- ProviderObservation.succeeded handles error / empty / whitespace cases
Live evidence
-------------
$ core test --suite smoke -q
67 passed in 26.55s (no regression)
$ python -m evals.frontier_compare --provider core --suite prompt_battery --json
model=core-native mode=core suite=prompt_battery passed=True score=1.000
[definition_truth ] PASS Truth is a claim or state grounded by evidence...
[definition_knowledge ] PASS Knowledge is justified understanding grounded...
[cause_understanding ] PASS understanding — teaching-grounded (cognition_chains_v1)...
[verification_evidence ] PASS evidence — teaching-grounded (cognition_chains_v1)...
[comparison_knowledge_wisdom ] PASS knowledge contrasts with wisdom...
[procedure_recall ] PASS To recall means to retrieve a stored state from memory...
[unknown_term ] PASS I haven't learned 'xylomorphic' yet...
$ python -m evals.frontier_compare --provider openai --suite determinism
error: suite 'determinism' is CORE-only; pass --suite prompt_battery
(the cross-provider suite) when --provider='openai'.
.gitignore: adds frontier_wave1.json (stray report file repeatedly
written by ad-hoc test invocations).
9459f815b0