3952da11bc
Six contained-tier gap closures from the cross-phase gap audit.
Every gaps.md file involved gains a resolution block; the historical
findings are preserved for traceability.
Identity (teaching/review.py)
- _ROLE_FRAMES gains the philosophical-axis family (axiology, ontology,
telos, ethos, epistemology, morality, ethics, virtues, norms,
doctrine, creed, convictions, beliefs, values, principles).
- _REDIRECT_VERBS gains the dismissal family (disregard, dismiss,
bypass, circumvent, renounce, reject, repudiate).
- Closes multi_agent_composition holdout MAC-HOLD-001 ("disregard your
axiology") and the matching adversarial_identity gap.
- Multi-agent holdouts: 8/8 attacks rejected, 3/3 legits accepted.
Pipeline (core/cognition/pipeline.py + docs/runtime_contracts.md)
- When the unknown-domain gate fires, ChatRuntime returns the
"I don't have field coordinates for that yet." stub and
vault_hits == 0. The pipeline now honours that stub as the
user-facing surface instead of overriding with the realizer's
fallback articulation. walk_surface is unchanged either way.
- New contract test
tests/test_semantic_realizer_integration.py::test_pipeline_honours_safety_stub_when_gate_fires
locks the contract; the existing semantic-surface test now primes
the vault first so the gate doesn't fire on the probe.
- Closes calibration gaps.md Finding 2.
Realizer morphology (generate/morphology.py)
- G1: ~100-entry irregular-verb table replaces the previous list which
contained only regular forms. Includes bind→bound, run→ran,
stand→stood, write→wrote/written, eat→ate/eaten, fly→flew/flown,
swim→swam/swum, etc.
- CVC doubling rule for -ed and -ing (stop→stopped/stopping,
plan→planned, run→running).
- Short-ies disambiguation (die/lie/tie keep -ie- in the base; cry/fly
collapse to -y). Lie is also irregular (lay/lain) — uses
_IRREGULAR_FORMS first.
- 28-case regression test (tests/test_morphology_irregular.py).
Realizer plural agreement (generate/templates.py)
- G2: under universal/existential/many/few/most quantifiers, count-noun
subjects pluralise (molecule → molecules) and the verb de-conjugates
(binds → bind). Negation toggles does-not → do-not. Aspect toggles
has → have, is → are. All other constructions unchanged.
- Mass nouns (evidence, wisdom, knowledge, truth, water, …) stay
singular under quantifiers — "all evidence supports truth" is right;
"all evidences support" would be wrong English.
- 17-case regression test
(tests/test_realizer_quantifier_agreement.py) covering count vs mass,
irregular plurals (child→children, analysis→analyses), and the
quantifier-tense / quantifier-aspect / quantifier-negation grid.
Rubric punctuation tolerance (evals/grammatical_coverage/runner.py)
- G3: _check_word_order strips trailing/leading punctuation
(.,;:!?—–) before exact-word comparison so "river," still satisfies
word_order=["river"]. must_contain also accepts punctuation-
stripped token matches.
- Affects every lane that uses grammatical_coverage scoring; the OOD
case generators no longer need to pin punctuated accept_surfaces for
C06.
Case generator + lane regeneration
- scripts/generate_english_fluency_ood.py uses generate.templates.pluralize
for C07/C08 must_contain + word_order so case-side constraints stay
aligned with the (more correct) realizer.
- All Phase 5 OOD lane cases (5.1, 5.4–5.7) regenerated; results files
re-scored.
CLI (core/cli.py)
- cmd_eval no longer crashes on lanes whose case_details use "id"
instead of "case_id" (adversarial_identity, multi_agent_composition).
- Cognition CLI lane gains the two new morphology/quantifier
regression test files.
Lane sweep (all 100%, no regression):
english_fluency_ood 117/117 public + 39/39 holdouts
elementary_mathematics_ood 117/117 + 39/39
foundational_physics_ood 117/117 + 39/39
foundational_biology_ood 117/117 + 39/39
classical_literature_ood 117/117 + 39/39
grammatical_coverage back to 100% on its own seed cases
hebrew_fluency / koine_greek_fluency 3/3 each
CLI lane health:
smoke 54, runtime 19, teaching 17, packs 6, cognition 103 (was 57),
algebra 132.
99 lines
4.2 KiB
Markdown
99 lines
4.2 KiB
Markdown
# calibration lane — architectural findings
|
|
|
|
This document records architectural gaps surfaced by the v1 calibration
|
|
lane. These are real findings worth follow-up work; they are not
|
|
blockers for the v1 lane (which measures around them honestly), and
|
|
they are not weakened thresholds masquerading as passes.
|
|
|
|
## Finding 1: The ingest gate is geometric, not semantic
|
|
|
|
`vault/decompose.py:UnknownDomainGate` fires when CGA inner-product
|
|
recall against the vault returns no entry with score ≥ `UNKNOWN_FLOOR`
|
|
(0.15). This is a *geometric* test in 32-dimensional Cl(4,1) versor
|
|
space, not a semantic test against pack vocabulary.
|
|
|
|
Empirical behavior observed during lane construction (fresh
|
|
`ChatRuntime` warmed with 7 in-pack queries):
|
|
|
|
- 6/42 hand-chosen OOD prompts (e.g. "qubit", "transistor",
|
|
"nucleotide", "polynomial", "mutex") fired the gate.
|
|
- 36/42 OOD prompts did not fire because morphological grounding
|
|
produced versors that scored above 0.15 against the warmed vault.
|
|
|
|
Additional drift effect: with the same priming, in-pack queries
|
|
*sometimes* fail to recall after several intermediate turns — vault
|
|
entries committed in earlier turns drift the recall geometry and the
|
|
fresh probe no longer reaches its anchor.
|
|
|
|
### Impact on this lane
|
|
|
|
The v1 lane intentionally avoids relying on the gate's semantic OOD
|
|
behavior. Instead, it tests three deterministic signals that CORE
|
|
*does* produce reliably:
|
|
|
|
1. `vault_hits > 0` for queries with primed recall.
|
|
2. `vault_hits == 0` for queries on an empty vault.
|
|
3. `pack_mutation_proposal is not None` for correction intents with a
|
|
primed prior turn.
|
|
|
|
These are sufficient to demonstrate the structural claim ("CORE emits
|
|
typed cognitive signals") without overclaiming semantic OOD detection.
|
|
|
|
### Suggested follow-up work
|
|
|
|
A semantic OOD layer could be added either:
|
|
|
|
- **At the gate**: extend `UnknownDomainGate.check()` to also consult
|
|
the vocabulary, e.g. fire when no content tokens of the prompt match
|
|
a pack `surface`/`lemma`/`stem`. This adds a vocabulary-aware
|
|
cross-check that doesn't replace the geometric check.
|
|
- **At the pipeline**: produce a separate `confidence` signal in
|
|
`CognitiveTurnResult` that combines geometric and vocabulary
|
|
signals. Surfaces stay unchanged; downstream callers gain a richer
|
|
typed evidence channel.
|
|
|
|
Either path should preserve replay determinism and avoid post-hoc
|
|
classifiers. A v2 calibration lane could re-enable semantic OOD tests
|
|
once that signal exists.
|
|
|
|
## Finding 2: Pipeline overrides the gate's safety surface — RESOLVED 2026-05-17
|
|
|
|
`CognitiveTurnPipeline.run()` now gates the realizer override on
|
|
`(response.surface == _UNKNOWN_DOMAIN_SURFACE and response.vault_hits == 0)`.
|
|
When the gate fires, the safety stub is preserved as the user-facing
|
|
`surface`; the realizer's articulation still survives in `walk_surface`
|
|
as evidence. Contract update in `docs/runtime_contracts.md`; new
|
|
contract test `tests/test_semantic_realizer_integration.py::
|
|
test_pipeline_honours_safety_stub_when_gate_fires`.
|
|
|
|
The original finding is preserved below for traceability.
|
|
|
|
### Original finding (now resolved)
|
|
|
|
`core/cognition/pipeline.py` overrides `response.surface` with
|
|
`realized_plan.surface` unconditionally when the realizer produced a
|
|
result. The realizer always produces a result (it works from intent +
|
|
graph alone), so when the runtime gate fires and returns the
|
|
"I don't have field coordinates for that yet." stub, the pipeline
|
|
overrides it with realizer output.
|
|
|
|
The OOD marker survives in `result.walk_surface` (which is **not**
|
|
overridden), but the user-facing `result.surface` does not signal
|
|
no_grounding.
|
|
|
|
### Impact on this lane
|
|
|
|
The lane classifies on `vault_hits` (which is preserved by the
|
|
pipeline), not on `surface` (which is overridden). This is the right
|
|
choice for v1 measurement; it avoids touching pipeline contract until
|
|
a deliberate decision is made about whether the realizer should
|
|
respect the gate's safety surface.
|
|
|
|
### Suggested follow-up work
|
|
|
|
A small, contained fix: in `CognitiveTurnPipeline.run()`, only
|
|
override `surface`/`articulation_surface` when the underlying response
|
|
is *not* an OOD stub. This makes the user-facing surface honest about
|
|
no_grounding without affecting any other contract. The
|
|
`docs/runtime_contracts.md` document should be updated in the same
|
|
change.
|