310aed9ff0
* docs: consolidate governance anchors and clean up test registries * refactor(cli): decompose cli into dedicated modules * test: fix broken test baselines and formatting * docs: add domain boundary READMEs for governance anchors * test: update baseline for determination lane * test: fix capability_pass expectation * test: fix CORE_SHOWCASE_SKIP_BUDGET enforcement * chore: cleanup CLI extraction and unreachable code
99 lines
4.2 KiB
Markdown
99 lines
4.2 KiB
Markdown
# calibration lane — architectural findings
|
|
|
|
This document records architectural gaps surfaced by the v1 calibration
|
|
lane. These are real findings worth follow-up work; they are not
|
|
blockers for the v1 lane (which measures around them honestly), and
|
|
they are not weakened thresholds masquerading as passes.
|
|
|
|
## Finding 1: The ingest gate is geometric, not semantic
|
|
|
|
`vault/decompose.py:UnknownDomainGate` fires when CGA inner-product
|
|
recall against the vault returns no entry with score ≥ `UNKNOWN_FLOOR`
|
|
(0.15). This is a *geometric* test in 32-dimensional Cl(4,1) versor
|
|
space, not a semantic test against pack vocabulary.
|
|
|
|
Empirical behavior observed during lane construction (fresh
|
|
`ChatRuntime` warmed with 7 in-pack queries):
|
|
|
|
- 6/42 hand-chosen OOD prompts (e.g. "qubit", "transistor",
|
|
"nucleotide", "polynomial", "mutex") fired the gate.
|
|
- 36/42 OOD prompts did not fire because morphological grounding
|
|
produced versors that scored above 0.15 against the warmed vault.
|
|
|
|
Additional drift effect: with the same priming, in-pack queries
|
|
*sometimes* fail to recall after several intermediate turns — vault
|
|
entries committed in earlier turns drift the recall geometry and the
|
|
fresh probe no longer reaches its anchor.
|
|
|
|
### Impact on this lane
|
|
|
|
The v1 lane intentionally avoids relying on the gate's semantic OOD
|
|
behavior. Instead, it tests three deterministic signals that CORE
|
|
*does* produce reliably:
|
|
|
|
1. `vault_hits > 0` for queries with primed recall.
|
|
2. `vault_hits == 0` for queries on an empty vault.
|
|
3. `pack_mutation_proposal is not None` for correction intents with a
|
|
primed prior turn.
|
|
|
|
These are sufficient to demonstrate the structural claim ("CORE emits
|
|
typed cognitive signals") without overclaiming semantic OOD detection.
|
|
|
|
### Suggested follow-up work
|
|
|
|
A semantic OOD layer could be added either:
|
|
|
|
- **At the gate**: extend `UnknownDomainGate.check()` to also consult
|
|
the vocabulary, e.g. fire when no content tokens of the prompt match
|
|
a pack `surface`/`lemma`/`stem`. This adds a vocabulary-aware
|
|
cross-check that doesn't replace the geometric check.
|
|
- **At the pipeline**: produce a separate `confidence` signal in
|
|
`CognitiveTurnResult` that combines geometric and vocabulary
|
|
signals. Surfaces stay unchanged; downstream callers gain a richer
|
|
typed evidence channel.
|
|
|
|
Either path should preserve replay determinism and avoid post-hoc
|
|
classifiers. A v2 calibration lane could re-enable semantic OOD tests
|
|
once that signal exists.
|
|
|
|
## Finding 2: Pipeline overrides the gate's safety surface — RESOLVED 2026-05-17
|
|
|
|
`CognitiveTurnPipeline.run()` now gates the realizer override on
|
|
`(response.surface == _UNKNOWN_DOMAIN_SURFACE and response.vault_hits == 0)`.
|
|
When the gate fires, the safety stub is preserved as the user-facing
|
|
`surface`; the realizer's articulation still survives in `walk_surface`
|
|
as evidence. Contract update in `docs/specs/runtime_contracts.md`; new
|
|
contract test `tests/test_semantic_realizer_integration.py::
|
|
test_pipeline_honours_safety_stub_when_gate_fires`.
|
|
|
|
The original finding is preserved below for traceability.
|
|
|
|
### Original finding (now resolved)
|
|
|
|
`core/cognition/pipeline.py` overrides `response.surface` with
|
|
`realized_plan.surface` unconditionally when the realizer produced a
|
|
result. The realizer always produces a result (it works from intent +
|
|
graph alone), so when the runtime gate fires and returns the
|
|
"I don't have field coordinates for that yet." stub, the pipeline
|
|
overrides it with realizer output.
|
|
|
|
The OOD marker survives in `result.walk_surface` (which is **not**
|
|
overridden), but the user-facing `result.surface` does not signal
|
|
no_grounding.
|
|
|
|
### Impact on this lane
|
|
|
|
The lane classifies on `vault_hits` (which is preserved by the
|
|
pipeline), not on `surface` (which is overridden). This is the right
|
|
choice for v1 measurement; it avoids touching pipeline contract until
|
|
a deliberate decision is made about whether the realizer should
|
|
respect the gate's safety surface.
|
|
|
|
### Suggested follow-up work
|
|
|
|
A small, contained fix: in `CognitiveTurnPipeline.run()`, only
|
|
override `surface`/`articulation_surface` when the underlying response
|
|
is *not* an OOD stub. This makes the user-facing surface honest about
|
|
no_grounding without affecting any other contract. The
|
|
`docs/specs/runtime_contracts.md` document should be updated in the same
|
|
change.
|