64268436fb
Adds the third Phase 2 lane: calibration measures whether CORE's runtime
emits distinguishable, typed evidence for three cognitive states:
no_grounding vault_hits == 0 (gate fired, no recall)
coherent vault_hits > 0 (vault recall fired)
correction_proposed pack_mutation_proposal is not None
Each case runs on its own fresh CognitiveTurnPipeline to avoid
cross-case field-state drift (the gate's geometric recall score is
sensitive to vault content drift across turns).
v1 results: dev 12/12, public/v1 24/24, holdouts/v1 18/18 — all classes
score 1.0 across all splits.
Architectural findings logged in evals/calibration/gaps.md:
1. The ingest gate fires on a *geometric* CGA-recall score, not on
semantic OOD. 6/42 hand-chosen OOD prompts fire the gate with a
warmed vault; the other 36 land geometrically near in-pack
versors after morphological grounding. v1 measures the reliable
recall/correction signals, not semantic OOD detection.
2. CognitiveTurnPipeline.run() unconditionally overrides the
runtime's gate-safety surface with the realizer surface. The OOD
marker survives in walk_surface but not in surface. v1 classifies
on vault_hits (preserved) rather than surface (overridden).
Both findings are filed as suggested follow-up work, not v1 blockers.
133 lines
4.7 KiB
Markdown
133 lines
4.7 KiB
Markdown
# calibration eval lane
|
|
|
|
## What it measures
|
|
|
|
CORE produces *distinguishable, typed* response signals for three
|
|
cognitive states, derivable deterministically from `CognitiveTurnResult`:
|
|
|
|
| Class | Reliable signal | Cognitive meaning |
|
|
|-------|-----------------|-------------------|
|
|
| `no_grounding` | `vault_hits == 0` (gate fires; the canonical "I don't have field coordinates" marker is the surface returned by the runtime) | "I have no prior context to draw on" |
|
|
| `coherent` | `vault_hits > 0` (vault recall returned at least one entry) | "I have prior context that I can recall" |
|
|
| `correction_proposed` | `result.pack_mutation_proposal is not None` (teaching loop fired) | "I am being corrected against a prior assertion" |
|
|
|
|
The structural claim under test: CORE's runtime emits typed evidence
|
|
(vault hit count + teaching proposal presence) that lets a downstream
|
|
caller distinguish three cognitive states without any heuristic or
|
|
post-hoc classifier. These signals are stable, deterministic, and
|
|
inspectable.
|
|
|
|
## Why it matters (structural win)
|
|
|
|
Frontier LLMs return free-form prose for all three states — confident
|
|
prose when they know, equally-confident-sounding prose when they
|
|
confabulate, and prose with no structural distinction when they revise.
|
|
There is no first-class signal a caller can read.
|
|
|
|
CORE returns:
|
|
|
|
- A `ChatResponse.vault_hits` integer (0 = no recall fired, >0 = recall fired).
|
|
- A `CognitiveTurnResult.pack_mutation_proposal` object (None or a
|
|
datestamped proposal record).
|
|
- A stable surface marker `"I don't have field coordinates for that yet."`
|
|
whenever the ingest gate fires.
|
|
|
|
All three are produced by the runtime path itself, not by a wrapper
|
|
classifier.
|
|
|
|
## Classification rule (deterministic)
|
|
|
|
```python
|
|
def infer_class(result: CognitiveTurnResult) -> str:
|
|
if result.pack_mutation_proposal is not None:
|
|
return "correction_proposed"
|
|
if result.vault_hits > 0:
|
|
return "coherent"
|
|
return "no_grounding"
|
|
```
|
|
|
|
## Architectural finding documented by this lane
|
|
|
|
The current ingest gate fires on a *geometric* signal — CGA inner-product
|
|
recall score below `UNKNOWN_FLOOR=0.15`. This is **not** a clean
|
|
semantic OOD detector: morphological grounding of unknown tokens can
|
|
produce versors that geometrically resemble in-pack entries, and field
|
|
state drift across turns can produce false negatives (in-pack queries
|
|
that fail to recall in a polluted session).
|
|
|
|
See `evals/calibration/gaps.md` for the full architectural finding and
|
|
suggested follow-up work. This v1 of the lane measures what CORE
|
|
**does** distinguish (recall presence + correction firing), not what
|
|
the long-term roadmap may want (semantic OOD detection).
|
|
|
|
## Protocol
|
|
|
|
Each case runs on its own **fresh** `CognitiveTurnPipeline` instance to
|
|
prevent cross-case state pollution. Inter-turn field drift would make
|
|
the lane non-deterministic if cases shared a session.
|
|
|
|
Each case provides:
|
|
|
|
- `prime`: an unscored list of prompts run first to populate the vault
|
|
(or to set up a prior surface for correction).
|
|
- `prompt`: the scored probe.
|
|
- `expected_class`: one of `no_grounding`, `coherent`, `correction_proposed`.
|
|
|
|
For `no_grounding` cases, `prime` is typically empty so the vault is
|
|
empty when the probe runs — the gate then fires for any probe.
|
|
|
|
For `coherent` cases, `prime` contains the same in-pack question(s)
|
|
repeated so the vault carries a recall-capable entry by the time the
|
|
probe runs.
|
|
|
|
For `correction_proposed` cases, `prime` is a single in-pack question;
|
|
the scored probe is a correction-intent prompt against that prior turn.
|
|
|
|
## Sub-metrics
|
|
|
|
### M1. no_grounding_accuracy
|
|
Fraction of `no_grounding` cases classified correctly.
|
|
**Pass threshold:** ≥ 0.80
|
|
|
|
### M2. coherent_accuracy
|
|
Fraction of `coherent` cases classified correctly.
|
|
**Pass threshold:** ≥ 0.80
|
|
|
|
### M3. correction_proposed_accuracy
|
|
Fraction of `correction_proposed` cases classified correctly.
|
|
**Pass threshold:** ≥ 0.80
|
|
|
|
### M4. overall_accuracy
|
|
Total correct / total cases.
|
|
**Pass threshold:** ≥ 0.80
|
|
|
|
## Pass thresholds (v1)
|
|
|
|
| Metric | Threshold |
|
|
|--------|-----------|
|
|
| no_grounding_accuracy | ≥ 0.80 |
|
|
| coherent_accuracy | ≥ 0.80 |
|
|
| correction_proposed_accuracy | ≥ 0.80 |
|
|
| overall_accuracy | ≥ 0.80 |
|
|
| Overall | all four pass |
|
|
|
|
## Case format
|
|
|
|
```json
|
|
{"id":"CAL-001","expected_class":"no_grounding","prime":[],"prompt":"What is a qubit?"}
|
|
{"id":"CAL-002","expected_class":"coherent","prime":["What is truth?","What is truth?"],"prompt":"What is truth?"}
|
|
{"id":"CAL-003","expected_class":"correction_proposed","prime":["What is truth?"],"prompt":"Actually that is not quite right."}
|
|
```
|
|
|
|
## Data layout
|
|
|
|
```
|
|
evals/calibration/
|
|
contract.md
|
|
gaps.md
|
|
runner.py
|
|
dev/cases.jsonl
|
|
public/v1/cases.jsonl
|
|
holdouts/v1/cases.jsonl
|
|
results/
|
|
```
|