1e01f7794e
Implement the eval infrastructure defined in ADR-0016 before building new eval lanes. This establishes the discipline that governs the entire capability roadmap. - Generic eval framework (evals/framework.py): lane discovery, versioned scoring, result persistence - Cognition lane retrofitted into new convention: 45 cases split into stratified dev (13) / public v1 (13) / holdout (19) sets with contract, runner, and recorded results - Generalized `core eval <lane>` CLI: dynamic lane discovery, --list, --version, --split, --save, --json flags - Holdout runner scaffold: plaintext fallback, encryption interface ready - Baseline runner scaffold: pluggable frontier model interface - Fix: CognitiveTurnPipeline.run() crashed on turn_log[-1] when the unknown-domain gate returned a stub without appending to turn_log - ADR-0016, eval_methodology.md, PROGRESS.md, capability gates session log Phase 0 exit audit found two methodology issues: 1. Pipeline turn_log crash (fixed here) 2. Versor drift in multi-turn sessions (pre-existing, under investigation)
13 lines
3.4 KiB
JSON
13 lines
3.4 KiB
JSON
{"category": "cause", "expected_intent": "cause", "expected_surface_contains": ["knowledge"], "expected_terms": ["knowledge"], "id": "cause_knowledge_032", "prompt": "Why does knowledge require evidence?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "cause", "expected_intent": "cause", "expected_surface_contains": ["light"], "expected_terms": ["light"], "id": "cause_light_007", "prompt": "Why does light exist?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "comparison", "expected_intent": "comparison", "expected_surface_contains": ["memory"], "expected_terms": ["memory"], "id": "comparison_memory_recall_030", "prompt": "Compare memory and recall", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "correction", "expected_intent": "correction", "expected_surface_contains": ["correction"], "expected_terms": ["correction"], "id": "correction_specific_015", "prompt": "No, correction means reviewed repair", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "definition", "expected_intent": "definition", "expected_surface_contains": ["knowledge"], "expected_terms": ["knowledge"], "id": "definition_knowledge_003", "prompt": "What is knowledge?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "definition", "expected_intent": "definition", "expected_surface_contains": ["light"], "expected_terms": ["light"], "id": "definition_light_002", "prompt": "What is light?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "definition", "expected_intent": "definition", "expected_surface_contains": ["meaning"], "expected_terms": ["meaning"], "id": "definition_meaning_004", "prompt": "What is meaning?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "definition", "expected_intent": "definition", "expected_surface_contains": ["procedure"], "expected_terms": ["procedure"], "id": "definition_procedure_023", "prompt": "What is a procedure?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "definition", "expected_intent": "definition", "expected_surface_contains": ["relation"], "expected_terms": ["relation"], "id": "definition_relation_026", "prompt": "What is a relation?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "procedure", "expected_intent": "procedure", "expected_surface_contains": [], "expected_terms": [], "id": "procedure_correct_035", "prompt": "How can I correct an error?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "recall", "expected_intent": "recall", "expected_surface_contains": ["light"], "expected_terms": ["light"], "id": "recall_light_013", "prompt": "Remember light", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "unknown", "expected_intent": "unknown", "expected_surface_contains": [], "expected_terms": ["light"], "id": "unknown_logos_019", "prompt": "light logos", "requires_deterministic_trace": true, "requires_versor_closure": true}
|
|
{"category": "verification", "expected_intent": "verification", "expected_surface_contains": ["memory"], "expected_terms": ["memory"], "id": "verification_memory_037", "prompt": "Does memory require recall?", "requires_deterministic_trace": true, "requires_versor_closure": true}
|