9459f815b0
#58 shipped providers.py + model_registry.py for cross-provider benchmarking but never connected them to runner.py — the adapters sat unused. This PR wires them through with a clear lane split. Why a new suite instead of refactoring existing ones ----------------------------------------------------- The three existing suites (determinism / truth_lock / axis_orthogonality) pull CORE-only telemetry: trace_hash, versor_condition, register_id, register_variant_id, anchor_lens_id, register_canonical_surface. None of those fields can come from OpenAI / Anthropic / Ollama. Forcing those suites cross-provider would silently produce reports where the cross-provider rows have empty telemetry — a worse failure mode than not running them at all. So the routing is explicit: CORE-only suites → --provider must be 'core' Cross-provider suites → any provider; CORE is one adapter among many Operator asks for the wrong combo → loud error with the right alternative. New module: evals/frontier_compare/cross_provider.py ----------------------------------------------------- - ProviderObservation dataclass — provider-agnostic observation shape (prompt, surface, provider, model, elapsed_ms, error fields). No CORE-internal telemetry expected. - run_prompt_battery(adapter, *, cfg) → SuiteReport reusing existing CaseResult / SuiteReport shapes so the report viewer renders both lanes without schema branching. - _PROMPT_BATTERY: 7 fixed cases spanning definition / cause / verification / comparison / procedure / unknown intent shapes. Stable case_ids so future re-runs against the same provider produce diffable JSON. - Per-case 'passed' is loose by design (non-empty surface, no exception). Cross-provider quality is for human review — not for the runner to silently score. Updated CLI: evals/frontier_compare/__main__.py ----------------------------------------------- - --provider {core, openai, anthropic, ollama} (default: core) - --model <id> (validated via require_model_card) - --env-file <path> (default: ./.env) - Auto-persist non-CORE runs to evals/frontier_compare/results/<provider>_<model>_<utc>.json even when --report is omitted. API calls are rate-limited / paid; losing the artifact is costly. - Existing CORE-native behavior unchanged when --provider not set. Results directory: evals/frontier_compare/results/ -------------------------------------------------- Created with .gitkeep — matches the convention used by other lanes (evals/long_context_cost/results/, evals/koine_greek_fluency/results/, etc.). Distinct from reports/ which .gitignore excludes for transient debug output. Tests: tests/test_frontier_compare_cross_provider.py (9 cases) -------------------------------------------------------------- - prompt_battery runs with CORE adapter (no API needed) - adapter exceptions recorded as failed observations, never propagated - empty surfaces flagged distinctly from adapter errors - CLI default runs CORE-native (no breaking change) - CLI prompt_battery with --provider core routes through cross-provider path - CLI rejects CORE-only suite + non-CORE provider with operator-helpful error - --help surfaces both suite families - unregistered model is rejected before any benchmark cycles burn - ProviderObservation.succeeded handles error / empty / whitespace cases Live evidence ------------- $ core test --suite smoke -q 67 passed in 26.55s (no regression) $ python -m evals.frontier_compare --provider core --suite prompt_battery --json model=core-native mode=core suite=prompt_battery passed=True score=1.000 [definition_truth ] PASS Truth is a claim or state grounded by evidence... [definition_knowledge ] PASS Knowledge is justified understanding grounded... [cause_understanding ] PASS understanding — teaching-grounded (cognition_chains_v1)... [verification_evidence ] PASS evidence — teaching-grounded (cognition_chains_v1)... [comparison_knowledge_wisdom ] PASS knowledge contrasts with wisdom... [procedure_recall ] PASS To recall means to retrieve a stored state from memory... [unknown_term ] PASS I haven't learned 'xylomorphic' yet... $ python -m evals.frontier_compare --provider openai --suite determinism error: suite 'determinism' is CORE-only; pass --suite prompt_battery (the cross-provider suite) when --provider='openai'. .gitignore: adds frontier_wave1.json (stray report file repeatedly written by ad-hoc test invocations).
23 lines
276 B
Text
23 lines
276 B
Text
__pycache__/
|
|
.pytest_cache/
|
|
.hypothesis/
|
|
*.pyc
|
|
.DS_Store
|
|
|
|
*.egg-info/
|
|
traces/
|
|
.formation_cache/
|
|
|
|
core-rs/target/
|
|
core-rs/Cargo.lock
|
|
|
|
uv.lock
|
|
|
|
# Environment secrets — never commit real keys
|
|
.env
|
|
.env.local
|
|
.env.*.local
|
|
|
|
# Benchmark report output
|
|
reports/
|
|
frontier_wave1.json
|