64268436fb
Adds the third Phase 2 lane: calibration measures whether CORE's runtime
emits distinguishable, typed evidence for three cognitive states:
no_grounding vault_hits == 0 (gate fired, no recall)
coherent vault_hits > 0 (vault recall fired)
correction_proposed pack_mutation_proposal is not None
Each case runs on its own fresh CognitiveTurnPipeline to avoid
cross-case field-state drift (the gate's geometric recall score is
sensitive to vault content drift across turns).
v1 results: dev 12/12, public/v1 24/24, holdouts/v1 18/18 — all classes
score 1.0 across all splits.
Architectural findings logged in evals/calibration/gaps.md:
1. The ingest gate fires on a *geometric* CGA-recall score, not on
semantic OOD. 6/42 hand-chosen OOD prompts fire the gate with a
warmed vault; the other 36 land geometrically near in-pack
versors after morphological grounding. v1 measures the reliable
recall/correction signals, not semantic OOD detection.
2. CognitiveTurnPipeline.run() unconditionally overrides the
runtime's gate-safety surface with the realizer surface. The OOD
marker survives in walk_surface but not in surface. v1 classifies
on vault_hits (preserved) rather than surface (overridden).
Both findings are filed as suggested follow-up work, not v1 blockers.
12 lines
1.4 KiB
JSON
12 lines
1.4 KiB
JSON
{"id":"CAL-DEV-001","expected_class":"no_grounding","prime":[],"prompt":"What is a qubit?"}
|
|
{"id":"CAL-DEV-002","expected_class":"no_grounding","prime":[],"prompt":"Explain photosynthesis."}
|
|
{"id":"CAL-DEV-003","expected_class":"no_grounding","prime":[],"prompt":"What is the mitochondria?"}
|
|
{"id":"CAL-DEV-004","expected_class":"no_grounding","prime":[],"prompt":"What is truth?"}
|
|
{"id":"CAL-DEV-005","expected_class":"coherent","prime":["What is truth?","What is truth?"],"prompt":"What is truth?"}
|
|
{"id":"CAL-DEV-006","expected_class":"coherent","prime":["What is wisdom?","What is wisdom?"],"prompt":"What is wisdom?"}
|
|
{"id":"CAL-DEV-007","expected_class":"coherent","prime":["What is knowledge?","What is knowledge?"],"prompt":"What is knowledge?"}
|
|
{"id":"CAL-DEV-008","expected_class":"coherent","prime":["What is light?","What is light?"],"prompt":"What is light?"}
|
|
{"id":"CAL-DEV-009","expected_class":"correction_proposed","prime":["What is truth?"],"prompt":"Actually that is not quite right."}
|
|
{"id":"CAL-DEV-010","expected_class":"correction_proposed","prime":["What is wisdom?"],"prompt":"No, wisdom is different from that."}
|
|
{"id":"CAL-DEV-011","expected_class":"correction_proposed","prime":["What is knowledge?"],"prompt":"Actually knowledge is not what you described."}
|
|
{"id":"CAL-DEV-012","expected_class":"correction_proposed","prime":["What is light?"],"prompt":"No, that is incorrect."}
|