64268436fb
Adds the third Phase 2 lane: calibration measures whether CORE's runtime
emits distinguishable, typed evidence for three cognitive states:
no_grounding vault_hits == 0 (gate fired, no recall)
coherent vault_hits > 0 (vault recall fired)
correction_proposed pack_mutation_proposal is not None
Each case runs on its own fresh CognitiveTurnPipeline to avoid
cross-case field-state drift (the gate's geometric recall score is
sensitive to vault content drift across turns).
v1 results: dev 12/12, public/v1 24/24, holdouts/v1 18/18 — all classes
score 1.0 across all splits.
Architectural findings logged in evals/calibration/gaps.md:
1. The ingest gate fires on a *geometric* CGA-recall score, not on
semantic OOD. 6/42 hand-chosen OOD prompts fire the gate with a
warmed vault; the other 36 land geometrically near in-pack
versors after morphological grounding. v1 measures the reliable
recall/correction signals, not semantic OOD detection.
2. CognitiveTurnPipeline.run() unconditionally overrides the
runtime's gate-safety surface with the realizer surface. The OOD
marker survives in walk_surface but not in surface. v1 classifies
on vault_hits (preserved) rather than surface (overridden).
Both findings are filed as suggested follow-up work, not v1 blockers.
87 lines
3.7 KiB
Markdown
87 lines
3.7 KiB
Markdown
# calibration lane — architectural findings
|
|
|
|
This document records architectural gaps surfaced by the v1 calibration
|
|
lane. These are real findings worth follow-up work; they are not
|
|
blockers for the v1 lane (which measures around them honestly), and
|
|
they are not weakened thresholds masquerading as passes.
|
|
|
|
## Finding 1: The ingest gate is geometric, not semantic
|
|
|
|
`vault/decompose.py:UnknownDomainGate` fires when CGA inner-product
|
|
recall against the vault returns no entry with score ≥ `UNKNOWN_FLOOR`
|
|
(0.15). This is a *geometric* test in 32-dimensional Cl(4,1) versor
|
|
space, not a semantic test against pack vocabulary.
|
|
|
|
Empirical behavior observed during lane construction (fresh
|
|
`ChatRuntime` warmed with 7 in-pack queries):
|
|
|
|
- 6/42 hand-chosen OOD prompts (e.g. "qubit", "transistor",
|
|
"nucleotide", "polynomial", "mutex") fired the gate.
|
|
- 36/42 OOD prompts did not fire because morphological grounding
|
|
produced versors that scored above 0.15 against the warmed vault.
|
|
|
|
Additional drift effect: with the same priming, in-pack queries
|
|
*sometimes* fail to recall after several intermediate turns — vault
|
|
entries committed in earlier turns drift the recall geometry and the
|
|
fresh probe no longer reaches its anchor.
|
|
|
|
### Impact on this lane
|
|
|
|
The v1 lane intentionally avoids relying on the gate's semantic OOD
|
|
behavior. Instead, it tests three deterministic signals that CORE
|
|
*does* produce reliably:
|
|
|
|
1. `vault_hits > 0` for queries with primed recall.
|
|
2. `vault_hits == 0` for queries on an empty vault.
|
|
3. `pack_mutation_proposal is not None` for correction intents with a
|
|
primed prior turn.
|
|
|
|
These are sufficient to demonstrate the structural claim ("CORE emits
|
|
typed cognitive signals") without overclaiming semantic OOD detection.
|
|
|
|
### Suggested follow-up work
|
|
|
|
A semantic OOD layer could be added either:
|
|
|
|
- **At the gate**: extend `UnknownDomainGate.check()` to also consult
|
|
the vocabulary, e.g. fire when no content tokens of the prompt match
|
|
a pack `surface`/`lemma`/`stem`. This adds a vocabulary-aware
|
|
cross-check that doesn't replace the geometric check.
|
|
- **At the pipeline**: produce a separate `confidence` signal in
|
|
`CognitiveTurnResult` that combines geometric and vocabulary
|
|
signals. Surfaces stay unchanged; downstream callers gain a richer
|
|
typed evidence channel.
|
|
|
|
Either path should preserve replay determinism and avoid post-hoc
|
|
classifiers. A v2 calibration lane could re-enable semantic OOD tests
|
|
once that signal exists.
|
|
|
|
## Finding 2: Pipeline overrides the gate's safety surface
|
|
|
|
`core/cognition/pipeline.py` overrides `response.surface` with
|
|
`realized_plan.surface` unconditionally when the realizer produced a
|
|
result. The realizer always produces a result (it works from intent +
|
|
graph alone), so when the runtime gate fires and returns the
|
|
"I don't have field coordinates for that yet." stub, the pipeline
|
|
overrides it with realizer output.
|
|
|
|
The OOD marker survives in `result.walk_surface` (which is **not**
|
|
overridden), but the user-facing `result.surface` does not signal
|
|
no_grounding.
|
|
|
|
### Impact on this lane
|
|
|
|
The lane classifies on `vault_hits` (which is preserved by the
|
|
pipeline), not on `surface` (which is overridden). This is the right
|
|
choice for v1 measurement; it avoids touching pipeline contract until
|
|
a deliberate decision is made about whether the realizer should
|
|
respect the gate's safety surface.
|
|
|
|
### Suggested follow-up work
|
|
|
|
A small, contained fix: in `CognitiveTurnPipeline.run()`, only
|
|
override `surface`/`articulation_surface` when the underlying response
|
|
is *not* an OOD stub. This makes the user-facing surface honest about
|
|
no_grounding without affecting any other contract. The
|
|
`docs/runtime_contracts.md` document should be updated in the same
|
|
change.
|