1e01f7794e
Implement the eval infrastructure defined in ADR-0016 before building new eval lanes. This establishes the discipline that governs the entire capability roadmap. - Generic eval framework (evals/framework.py): lane discovery, versioned scoring, result persistence - Cognition lane retrofitted into new convention: 45 cases split into stratified dev (13) / public v1 (13) / holdout (19) sets with contract, runner, and recorded results - Generalized `core eval <lane>` CLI: dynamic lane discovery, --list, --version, --split, --save, --json flags - Holdout runner scaffold: plaintext fallback, encryption interface ready - Baseline runner scaffold: pluggable frontier model interface - Fix: CognitiveTurnPipeline.run() crashed on turn_log[-1] when the unknown-domain gate returned a stub without appending to turn_log - ADR-0016, eval_methodology.md, PROGRESS.md, capability gates session log Phase 0 exit audit found two methodology issues: 1. Pipeline turn_log crash (fixed here) 2. Versor drift in multi-turn sessions (pre-existing, under investigation)
13 lines
3.4 KiB
JSON
13 lines
3.4 KiB
JSON
{"category": "cause", "expected_intent": "cause", "expected_surface_contains": ["correction"], "expected_terms": ["correction"], "id": "cause_correction_033", "prompt": "Why does correction help learning?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "cause", "expected_intent": "cause", "expected_surface_contains": ["creation"], "expected_terms": ["creation"], "id": "cause_creation_008", "prompt": "Why does creation matter?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "comparison", "expected_intent": "comparison", "expected_surface_contains": ["knowledge", "wisdom"], "expected_terms": ["knowledge", "wisdom"], "id": "comparison_knowledge_wisdom_029", "prompt": "Compare knowledge and wisdom", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "correction", "expected_intent": "correction", "expected_surface_contains": ["correction"], "expected_terms": [], "id": "correction_basic_014", "prompt": "No, that's wrong", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "definition", "expected_intent": "definition", "expected_surface_contains": ["distinction"], "expected_terms": ["distinction"], "id": "definition_distinction_025", "prompt": "What is distinction?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "definition", "expected_intent": "definition", "expected_surface_contains": ["evidence"], "expected_terms": ["evidence"], "id": "definition_evidence_021", "prompt": "What is evidence?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "definition", "expected_intent": "definition", "expected_surface_contains": ["identity"], "expected_terms": ["identity"], "id": "definition_identity_027", "prompt": "What is identity?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "definition", "expected_intent": "definition", "expected_surface_contains": ["inference"], "expected_terms": ["inference"], "id": "definition_inference_022", "prompt": "What is inference?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "definition", "expected_intent": "definition", "expected_surface_contains": ["judgment"], "expected_terms": ["judgment"], "id": "definition_judgment_044", "prompt": "What is judgment?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "procedure", "expected_intent": "procedure", "expected_surface_contains": [], "expected_terms": ["terms"], "id": "procedure_compare_011", "prompt": "How do I compare two terms?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "recall", "expected_intent": "recall", "expected_surface_contains": ["knowledge"], "expected_terms": ["knowledge"], "id": "recall_knowledge_039", "prompt": "Remember knowledge", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "unknown", "expected_intent": "unknown", "expected_surface_contains": [], "expected_terms": ["evidence", "reason"], "id": "unknown_evidence_042", "prompt": "evidence reason", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "verification", "expected_intent": "verification", "expected_surface_contains": ["light"], "expected_terms": ["light"], "id": "verification_light_017", "prompt": "Is light real?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|