Six contained-tier gap closures from the cross-phase gap audit.
Every gaps.md file involved gains a resolution block; the historical
findings are preserved for traceability.
Identity (teaching/review.py)
- _ROLE_FRAMES gains the philosophical-axis family (axiology, ontology,
telos, ethos, epistemology, morality, ethics, virtues, norms,
doctrine, creed, convictions, beliefs, values, principles).
- _REDIRECT_VERBS gains the dismissal family (disregard, dismiss,
bypass, circumvent, renounce, reject, repudiate).
- Closes multi_agent_composition holdout MAC-HOLD-001 ("disregard your
axiology") and the matching adversarial_identity gap.
- Multi-agent holdouts: 8/8 attacks rejected, 3/3 legits accepted.
Pipeline (core/cognition/pipeline.py + docs/runtime_contracts.md)
- When the unknown-domain gate fires, ChatRuntime returns the
"I don't have field coordinates for that yet." stub and
vault_hits == 0. The pipeline now honours that stub as the
user-facing surface instead of overriding with the realizer's
fallback articulation. walk_surface is unchanged either way.
- New contract test
tests/test_semantic_realizer_integration.py::test_pipeline_honours_safety_stub_when_gate_fires
locks the contract; the existing semantic-surface test now primes
the vault first so the gate doesn't fire on the probe.
- Closes calibration gaps.md Finding 2.
Realizer morphology (generate/morphology.py)
- G1: ~100-entry irregular-verb table replaces the previous list which
contained only regular forms. Includes bind→bound, run→ran,
stand→stood, write→wrote/written, eat→ate/eaten, fly→flew/flown,
swim→swam/swum, etc.
- CVC doubling rule for -ed and -ing (stop→stopped/stopping,
plan→planned, run→running).
- Short-ies disambiguation (die/lie/tie keep -ie- in the base; cry/fly
collapse to -y). Lie is also irregular (lay/lain) — uses
_IRREGULAR_FORMS first.
- 28-case regression test (tests/test_morphology_irregular.py).
Realizer plural agreement (generate/templates.py)
- G2: under universal/existential/many/few/most quantifiers, count-noun
subjects pluralise (molecule → molecules) and the verb de-conjugates
(binds → bind). Negation toggles does-not → do-not. Aspect toggles
has → have, is → are. All other constructions unchanged.
- Mass nouns (evidence, wisdom, knowledge, truth, water, …) stay
singular under quantifiers — "all evidence supports truth" is right;
"all evidences support" would be wrong English.
- 17-case regression test
(tests/test_realizer_quantifier_agreement.py) covering count vs mass,
irregular plurals (child→children, analysis→analyses), and the
quantifier-tense / quantifier-aspect / quantifier-negation grid.
Rubric punctuation tolerance (evals/grammatical_coverage/runner.py)
- G3: _check_word_order strips trailing/leading punctuation
(.,;:!?—–) before exact-word comparison so "river," still satisfies
word_order=["river"]. must_contain also accepts punctuation-
stripped token matches.
- Affects every lane that uses grammatical_coverage scoring; the OOD
case generators no longer need to pin punctuated accept_surfaces for
C06.
Case generator + lane regeneration
- scripts/generate_english_fluency_ood.py uses generate.templates.pluralize
for C07/C08 must_contain + word_order so case-side constraints stay
aligned with the (more correct) realizer.
- All Phase 5 OOD lane cases (5.1, 5.4–5.7) regenerated; results files
re-scored.
CLI (core/cli.py)
- cmd_eval no longer crashes on lanes whose case_details use "id"
instead of "case_id" (adversarial_identity, multi_agent_composition).
- Cognition CLI lane gains the two new morphology/quantifier
regression test files.
Lane sweep (all 100%, no regression):
english_fluency_ood 117/117 public + 39/39 holdouts
elementary_mathematics_ood 117/117 + 39/39
foundational_physics_ood 117/117 + 39/39
foundational_biology_ood 117/117 + 39/39
classical_literature_ood 117/117 + 39/39
grammatical_coverage back to 100% on its own seed cases
hebrew_fluency / koine_greek_fluency 3/3 each
CLI lane health:
smoke 54, runtime 19, teaching 17, packs 6, cognition 103 (was 57),
algebra 132.
Closes Phase 2 v2 coverage — all five lanes now pass v2 public + holdouts.
calibration v2:
public/v2 — 33 cases (11 no_grounding / 11 coherent / 11 correction_proposed)
deeper priming (3 repetitions) on coherent cases; OOD
cases include both technical-domain prompts and bare
in-pack terms with empty prime (gate fires on empty
vault regardless of vocabulary)
holdouts/v2 — 24 cases (8 / 8 / 8) on distinct vocabulary
Results: no_grounding_accuracy=1.0, coherent_accuracy=1.0,
correction_proposed_accuracy=1.0 on both splits.
symbolic-logic v2:
public/v2 — 24 cases, chains up to 5 hops:
chain_5, chain_4, chain_3, modus_ponens_chain,
modus_tollens_chain, negation_chain, syllogism,
chain_recall, single_premise, mixed
holdouts/v2 — 16 cases on distinct vocabulary
Results: premise_recall=1.0, replay_determinism=1.0,
proposal_storage=1.0 on both splits.
Each case still runs twice on fresh pipelines for replay-determinism
check — even the chain_5 cases (10 teaching turns × 2 runs) reproduce
trace_hash exactly.
Both lanes run via the parallel runner introduced in 57c08e6:
calibration public/v2 (33 cases) in 4.9s
calibration holdouts/v2 (24 cases) in 3.7s
symbolic_logic public/v2 (24 cases) in 21.9s
symbolic_logic holdouts/v2 (16 cases) in 16.5s
(symbolic_logic is slower per-case because of the 2x replay run +
long chains)
Test suite: 596 passing (no regression).
Phase 2 v2 milestone: all five lanes complete + v1 baselines documented.
Next: v3 lanes for at least two of the five (roadmap exit gate).
Parallel infrastructure:
evals/parallel.py
multiprocessing.Pool helper (spawn context, default workers
min(cpu_count, 8)). Per-case lanes use it via:
run_lane(cases, workers=N)
workers=1 forces serial (debugging); None uses the default pool.
Generic over the per-case return type, so dataclass-returning
runners (provenance) and dict-returning runners both work.
Wired into:
- evals/adversarial_identity/runner.py
- evals/calibration/runner.py
- evals/symbolic_logic/runner.py
- evals/provenance/runner.py
Per-case helpers are now picklable (module-level, single arg).
Monotonic-learning stays serial within a split — shared session
is structural to its longitudinal protocol.
Empirical speedup (adversarial-identity public/v1, 25 cases on
macOS 8 cores): serial 14.1s -> parallel 3.1s (~4.5x). Identical
per-case results.
adversarial-identity v2:
public/v2 — 35 cases (20 attack / 15 legitimate). Attacks cover
more varied phrasings: punctuation variation
("Actually -" / "No:" / "Correction —"), embedded
hedges ("please" / "regardless of prior context"),
multi-clause attacks, and identity-marker triggers
in mid-clause position.
holdouts/v2 — 22 cases (12 attack / 10 legitimate) on distinct
priming vocabulary.
Results: attack_rejection_rate=1.0, legitimate_acceptance_rate=1.0
on both splits.
The marker-regex defense in teaching/review.py:_is_identity_override
holds against every v2 phrasing — markers are checked case-insensitive
against the full text, so capitalization / punctuation tricks don't
slip past.
Test suite: 596 passing (no regression).
Records the architectural floor for frontier-LLM performance on each
Phase 2 v1 lane.
The baseline is structural: every lane's scoring rubric measures a
property that frontier LLMs do not architecturally emit (Provenance
typed sources, pack_mutation_proposal, vault_hits, REJECTED_IDENTITY
outcome, deterministic trace_hash). The frontier score on each of
those sub-metrics is 0.0 by construction, not by failure — even a
live-API run would still record 0.0 on these typed-signal checks
because the evidence is absent regardless of prose quality.
Artifacts:
docs/frontier_baselines.md
Full per-lane analysis: what each sub-metric scores, why the
frontier value is 0, and where a live-API baseline would or
would not add information.
evals/<lane>/baselines/v1_structural_zero.json (× 5)
Per-lane baseline records in the same shape as lane reports.
Encodes 0.0 / None on each sub-metric with rationale.
evals/baseline_runner.py
Adds StructuralZeroBaseline adapter conforming to the
BaselineModel protocol — a real, non-stub adapter that returns
the deterministic floor. Live-API adapters (Anthropic, OpenAI)
can be wired alongside when API keys are configured; the
structural floor remains the comparison baseline.
Across 5 lanes / 14 typed-signal sub-metrics:
CORE v1: 1.0 (each)
frontier structural: 0.0 (each)
The gap is "CORE measures a property frontier output does not
expose", not "CORE outperforms on a shared benchmark". v2 lanes may
add content-level sub-metrics where direct comparison via live-API
runs becomes meaningful.
Adds the third Phase 2 lane: calibration measures whether CORE's runtime
emits distinguishable, typed evidence for three cognitive states:
no_grounding vault_hits == 0 (gate fired, no recall)
coherent vault_hits > 0 (vault recall fired)
correction_proposed pack_mutation_proposal is not None
Each case runs on its own fresh CognitiveTurnPipeline to avoid
cross-case field-state drift (the gate's geometric recall score is
sensitive to vault content drift across turns).
v1 results: dev 12/12, public/v1 24/24, holdouts/v1 18/18 — all classes
score 1.0 across all splits.
Architectural findings logged in evals/calibration/gaps.md:
1. The ingest gate fires on a *geometric* CGA-recall score, not on
semantic OOD. 6/42 hand-chosen OOD prompts fire the gate with a
warmed vault; the other 36 land geometrically near in-pack
versors after morphological grounding. v1 measures the reliable
recall/correction signals, not semantic OOD detection.
2. CognitiveTurnPipeline.run() unconditionally overrides the
runtime's gate-safety surface with the realizer surface. The OOD
marker survives in walk_surface but not in surface. v1 classifies
on vault_hits (preserved) rather than surface (overridden).
Both findings are filed as suggested follow-up work, not v1 blockers.
Shay