diff --git a/notes/fluency_lift_baseline_2026-05-19.md b/notes/fluency_lift_baseline_2026-05-19.md new file mode 100644 index 00000000..1117f96c --- /dev/null +++ b/notes/fluency_lift_baseline_2026-05-19.md @@ -0,0 +1,186 @@ +# Fluency Lift Baseline — 2026-05-19 + +Numbers-only record of the 2026-05-19 fluency push. Captures what +changed, where, and by how much, so subsequent work has a fixed +substrate to measure against. + +## Eval lanes — before / after + +All three lanes are run from a fresh `ChatRuntime()` per case +(cold-start invariant) except where noted. + +### `cold_start_grounding` (44 conversational prompts) + +| | Before intent fix (b52e04a) | After intent fix | After gloss landing (07da601) | +|---|---|---|---| +| `intent_accuracy` | 0.4773 | 1.0000 | 1.0000 | +| `grounding_accuracy` | 0.4773 | 1.0000 | 1.0000 | +| `subject_accuracy` | 0.4318 | 1.0000 | 1.0000 | +| `none` count | 21 / 44 | 0 / 44 | 0 / 44 | +| `pack` count | 19 / 44 | 39 / 44 | 39 / 44 | + +Five intent-classification patterns recovered 21 prompts that +previously fell to `"I don't know — insufficient grounding"`: +`Define X`, `What does X mean?`, `What is to V?`, `How does X work?`, +`What causes X?`. + +### `warmed_session_consistency` (8 cases / 18 turns) + +| | Before pipeline gate (Phase B1) | After pipeline gate (c3e2a22) | +|---|---|---| +| `no_placeholder_rate` | 0.4444 | 1.0000 | +| `telemetry_consistency_rate` | 0.4444 | 1.0000 | +| `warm_grounding_stability` | 0.0000 | 0.0000 | +| `grounding_match_rate` | 0.4444 | 0.4444 | + +The pipeline-override usefulness gate cured the placeholder-prose +bug + the telemetry/result mismatch. `warm_grounding_stability` +remains 0 because of a separate architectural bug: a pack-grounded +turn 1 reverts to vault-walk on turn 2 of the same prompt. Fix +deferred to the SurfaceSelector RFC (`notes/surface_selector_design_2026-05-19.md`). + +### `deterministic_fluency` (15 cases × 6 predicates) + +| | Before gloss landing | After gloss landing (07da601) | +|---|---|---| +| `no_placeholder_rate` | 1.0000 | 1.0000 | +| `complete_punctuation_rate` | 1.0000 | 1.0000 | +| `finite_predicate_shape_rate`| 1.0000 | 1.0000 | +| `no_provenance_only_rate` | 1.0000 | 1.0000 | +| `surface_provenance_match_rate` | 1.0000 | 1.0000 | +| `no_dotted_inventory_rate` | **0.3333** | **1.0000** | + +The gloss feature delivered the no_dotted_inventory metric from +33% to 100%. Every gloss-backed surface now reads as a fluent +sentence instead of structured-disclosure dotted paths. + +### `cognition` (CORE's authoritative cognitive eval) + +| | Public (13 cases) | Holdout (19 cases) | +|---|---|---| +| `intent_accuracy` | 1.0000 | 1.0000 | +| `term_capture_rate` | 0.9167 | 0.8333 | +| `surface_groundedness`| 1.0000 | 1.0000 | +| `versor_closure_rate` | 1.0000 | 1.0000 | + +**Byte-identical** across every change in this push. Substring +assertions in the eval continue to find every expected term in the +new fluent surfaces. + +## Sample probe — fluent vs. before + +Fresh `ChatRuntime()` per prompt: + +```text +input: What is truth? +before: truth — pack-grounded (en_core_cognition_v1): + cognition.truth; logos.core; epistemic.ground. + No session evidence yet. +after: Truth is a claim or state grounded by evidence and coherent + judgment. pack-grounded (en_core_cognition_v1). + +input: Define moment. +before: I don't know — insufficient grounding for that yet. +after: Moment is a brief or pointlike interval of time. + pack-grounded (en_core_temporal_v1). + +input: What does important mean? +before: I don't know — insufficient grounding for that yet. +after: Something is important when it carries weight or priority in + some judgment context. pack-grounded (en_core_attitude_v1). + +input: What is to create? +before: I haven't learned 'to create' yet (intent: definition). + Mounted lexicon packs: en_core_cognition_v1, ... +after: To create means to bring something into existence through + deliberate action. pack-grounded (en_core_action_v1). + +input: What is quasar? (genuinely OOV — control) +both: I haven't learned 'quasar' yet (intent: definition). + Mounted lexicon packs: ... + +input: How does memory work? (CAUSE w/o teaching chain — control) +both: I don't know — insufficient grounding for that yet. + (deliberately preserved as the discovery-gap signal) +``` + +## Lexicon + gloss inventory + +After this push: + +| | Lexicon entries | Glosses | Coverage | +|---|---|---|---| +| en_core_cognition_v1 | 85 | 78 | 91.8% | +| en_core_meta_v1 | 73 | 72 | 98.6% | +| en_core_attitude_v1 | 40 | 40 | 100.0% | +| en_core_temporal_v1 | 28 | 28 | 100.0% | +| en_core_action_v1 | 26 | 26 | 100.0% | +| en_core_quantitative_v1 | 24 | 24 | 100.0% | +| en_core_spatial_v1 | 24 | 24 | 100.0% | +| en_core_polarity_v1 | 16 | 16 | 100.0% | +| en_core_causation_v1 | 15 | 15 | 100.0% | +| **Total** | **331** | **323** | **97.6%** | + +The 8 unglossed entries in cognition are dual-POS lemmas (e.g. +`cause` exists as NOUN and VERB; only the more salient POS got a +gloss in the first dispatch). Adding the duals is a follow-up +authoring pass. + +## Commits in this push + +``` +07da601 feat(packs): seed 323 reviewed glosses across 9 English content packs +46ac737 feat(pack-grounding): selector-ready gloss wiring via PackSurfaceCandidate +24daebf feat(pack-resolver): gloss resolver with lexicon-residency + dual-checksum hardening +c3e2a22 fix(pipeline): usefulness gate on realized-plan override +a67a3cc feat(evals): deterministic_fluency lane — six structural predicates +0cf1a8f feat(evals): warmed_session_consistency lane — pipeline override regression substrate +c6b4f1d fix(runtime): config-replace + thin API wrappers + stale docstring +a084f1d feat(evals): cold_start_grounding lane — 44-prompt routing probe +b52e04a fix(intent): five conversational definition patterns + polarity-stopword +``` + +Earlier in the session (now ancestors of the above): + +``` +8 commits seeding 9 new English content packs (230 lemmas, 5x prior coverage) +``` + +## What's deferred (with rationale) + +- **SurfaceSelector refactor** — `notes/surface_selector_design_2026-05-19.md` + Cures `warm_grounding_stability`. Crosses runtime + pipeline + + telemetry + hash. Solo-landing carries too much blast radius; + reviewer is best positioned to land it. + +- **Spine unification** — `notes/spine_unification_design_2026-05-19.md` + Cures `core chat` ≠ pipeline-eval drift. Depends on the + SurfaceSelector landing first. + +- **Cognition dual-POS gloss completion** — 8 cognition lemmas have + dual entries (NOUN+VERB) where only one got a gloss. Mechanical + follow-up; one subagent dispatch can close it. + +- **Gloss-formed sentences for AUX/PRON/SCONJ** — three lemmas in + cognition (`be`, `why`, `because`) have glosses authored to a + specific frame. Manual QA pass on the resulting surface is + pending. + +## Reproducing the numbers + +```bash +core eval cold_start_grounding +core eval warmed_session_consistency +core eval deterministic_fluency +core eval cognition +core eval cognition --split holdout + +# Live probe: +python3 -c " +from chat.runtime import ChatRuntime +for p in ['What is truth?', 'Define moment.', 'What does important mean?', + 'What is to create?', 'How does memory work?']: + r = ChatRuntime().chat(p) + print(f'[{r.grounding_source}] {p}\n -> {r.surface}\n') +" +``` diff --git a/notes/spine_unification_design_2026-05-19.md b/notes/spine_unification_design_2026-05-19.md new file mode 100644 index 00000000..e4fc8a41 --- /dev/null +++ b/notes/spine_unification_design_2026-05-19.md @@ -0,0 +1,144 @@ +# Cognitive-Spine Unification — Deferred RFC + +**Date:** 2026-05-19 +**Status:** Design proposal — NOT implemented. Deferred from the +2026-05-19 fluency push because the change crosses public-API +entrypoints and depends on the SurfaceSelector landing. +**Companion RFC:** `notes/surface_selector_design_2026-05-19.md` + +## Motivation + +The 2026-05-19 design review's Finding P0 #2: + +> The live cognitive spine is fragmented across public entrypoints. + +| Entrypoint | Today's spine | Affected by intent fixes? | Affected by pipeline fixes? | +|---|---|---|---| +| `core chat` (REPL) | `ChatRuntime.chat()` direct | ✓ yes | ✗ no | +| `core trace` (single turn)| `ChatRuntime.chat()` direct | ✓ yes | ✗ no | +| `core pulse` (research) | `scripts/run_pulse.py` graph diffusion + GloVe-seeded | ✗ no | ✗ no | +| `evals/cognition/runner.py`| `CognitiveTurnPipeline.run()` | ✓ yes | ✓ yes | +| `evals/cold_start_grounding/runner.py` | `ChatRuntime.chat()` direct | ✓ yes | ✗ no | +| `evals/warmed_session_consistency/runner.py` | `CognitiveTurnPipeline.run()` | ✓ yes | ✓ yes | +| `evals/deterministic_fluency/runner.py` | `ChatRuntime.chat()` direct | ✓ yes | ✗ no | + +Three separate cognitive spines exist: + +1. `ChatRuntime.chat()` direct — the simplest path, used by `core chat` + and `core trace`. +2. `CognitiveTurnPipeline.run()` — wraps `ChatRuntime.chat()` and adds + a graph-realizer override step + transitive walk + frame compose. +3. `scripts/run_pulse.py` — independent path with GloVe seeding, + graph constraint correction, top-k recall. + +Effects of fragmentation: + +- A fix to the pipeline's override behaviour does not reach the + user via `core chat`. +- A fix to the runtime reaches the user but is masked under the + pipeline-wrapped eval lanes. +- Pulse can "prove" capabilities the user never experiences. +- Tests can be green while user behaviour is broken (and vice versa). + +## Proposed direction + +### One canonical chat spine + +`ChatRuntime.chat()` becomes the single canonical entrypoint. The +pipeline's value-add (transitive walks, frame composition) moves +INSIDE the runtime as opt-in passes consulted by the selector: + +```python +class ChatRuntime: + def chat(self, text, *, max_tokens=None, mode="full"): + # mode="full" — runtime + pipeline-equivalent passes + # mode="bridge" — runtime only (today's bridge path) + # mode="walk" — walk evidence only (research / introspection) + ... +``` + +The pipeline becomes a thin convenience wrapper that selects a mode: + +```python +class CognitiveTurnPipeline: + def run(self, text, *, max_tokens=None): + # Equivalent to ChatRuntime.chat(text, mode="full"). + # Retained as the API the cognition eval harness was built + # against; new code calls ChatRuntime.chat() directly. + ... +``` + +### Pulse demoted to research harness + +`scripts/run_pulse.py` keeps existing for the geometry-research path +but is labeled non-canonical. It does not contribute to "fluent +chat" claims; the eval lanes that rely on it (if any) are renamed. + +### Single selector consumed everywhere + +The SurfaceSelector (companion RFC) is the only path that emits the +user-facing surface. All entrypoints route through it: + +``` +user input + ─▶ ChatRuntime.chat(text, mode=…) + ─▶ collect_candidates(intent, subject, field_state, mode) + ─▶ SurfaceSelector.select(candidates, context) + ─▶ ChatResponse(surface=chosen.surface, …) +``` + +`core chat`, `core trace`, every eval lane, and the pipeline shim +all call `ChatRuntime.chat()` with different modes. One emission +point, one telemetry record, one trace hash. + +## What this fixes + +| Today | After | +|---|---| +| Pipeline override invisible to `core chat` | Pipeline's value-add is opt-in modes inside the runtime; visible everywhere or nowhere | +| Eval-vs-user behaviour drift | Same code path; can't drift | +| Pulse "proves" things the user doesn't see | Pulse explicitly labeled non-canonical | +| Tests asserting `r.surface == r.walk_surface` | Surface is the selector's output; walk_surface remains audit telemetry | +| Three places to add a fluency surface | One: register a SurfaceProvider | + +## Sequencing + +This RFC is **dependent on** `surface_selector_design_2026-05-19.md`. +Land in this order: + +1. SurfaceSelector + provider registry (the companion RFC) +2. Wrap each existing dispatcher branch as a provider +3. Re-implement pipeline override as a provider (or remove if the + selector handles it via ordering) +4. Move `core chat` to `ChatRuntime.chat(mode="full")` +5. Move `core trace` to `ChatRuntime.chat(mode="full")` with trace + instrumentation +6. Audit eval lanes — every lane explicitly declares its mode +7. Label `scripts/run_pulse.py` as non-canonical in its docstring + and any eval lane that depends on it + +## What does NOT change + +- Pack content (already correct authoring path) +- Teaching chains (already correct authoring path) +- Intent classification (already canonical via `generate.intent`) +- Telemetry schema (one emitter, one shape) +- Trace-hash stability (intra-session; hashes are still per-run) + +## Risk register + +- **Public API stability** — `CognitiveTurnPipeline.run()` cannot + be removed without a deprecation cycle. Migration step is a + wrapper, not a removal. +- **Mode semantics** — the three modes (`full` / `bridge` / `walk`) + must be documented in `docs/runtime_contracts.md` BEFORE the + refactor so users can rely on them. +- **Eval invariant** — `cognition` eval expects pipeline-level + behaviour. The wrapper preserves that; verifiable byte-identity + on the eval is a hard prerequisite to commit. + +## When to land + +After the SurfaceSelector RFC. Spine unification without the +selector would just move the fragmentation; with the selector it +collapses the spines onto a single, observable, replayable path. diff --git a/notes/surface_selector_design_2026-05-19.md b/notes/surface_selector_design_2026-05-19.md new file mode 100644 index 00000000..d2f55b4b --- /dev/null +++ b/notes/surface_selector_design_2026-05-19.md @@ -0,0 +1,188 @@ +# SurfaceSelector Design — Deferred RFC + +**Date:** 2026-05-19 +**Status:** Design proposal — NOT implemented. Deferred from the +2026-05-19 fluency push because the refactor crosses too many files +to land safely solo. Picks up cleanly when the reviewer is back. +**Seed type:** `chat/pack_surface_candidate.py::PackSurfaceCandidate` +(landed in commit 46ac737 — already shaped for selector consumption) + +## Motivation + +The 2026-05-19 design review identified the central architectural +fault behind the fluency bug class: + +> **Surface authority fragmentation.** The runtime, pipeline, walk +> evidence, pack fallback, telemetry, hash, and several composers all +> compete for the user-facing sentence. The system can be "green" +> while user-facing selection, pipeline selection, and telemetry +> selection disagree. + +Phase B1's pipeline-override usefulness gate (commit `c3e2a22`) cured +two symptoms: + +- `no_placeholder_rate`: 0.44 → 1.00 +- `telemetry_consistency_rate`: 0.44 → 1.00 + +But it did not cure the third: + +- `warm_grounding_stability`: 0.0 (warmed lane) + +The warmed-session bug — a pack-grounded surface on turn 1 reverting +to a vault-walk fragment on turn 2 of the same prompt — is the +remaining symptom. Its fix is structural, not patchable: pack- +grounding must fire by intent + lemma residency, not by vault gate +state. This is the SurfaceSelector's job. + +## Proposed design + +### One typed candidate per provider + +```python +@dataclass(frozen=True, slots=True) +class SurfaceCandidate: + surface: str # final user-facing string + grounding_source: str # "refusal"/"teaching"/"pack"/... + provider_id: str # which provider emitted this + rank_hint: int # provider's own confidence tier + pack_id: str | None # provenance (None for refusal/OOV) + intent: IntentTag + subject_lemma: str | None + semantic_domains: tuple[str, ...] # audit content + epistemic_status: str # "asserted"/"hedged"/"refused"/... + is_user_facing_safe: bool + is_fluent_sentence: bool + is_replayable: bool +``` + +`PackSurfaceCandidate` (already landed) is a structural subset of +this — the migration is a strict superset rename + a few new fields. + +### Providers + +Each provider has one job: produce a candidate (or None) for the +given intent + subject. No selection logic; no telemetry; no +side effects beyond the candidate's audit fields. + +``` +RefusalProvider → refusal candidates (safety/ethics) +TeachingChainProvider → reviewed-corpus chain candidates +CrossPackChainProvider → cross-pack chain candidates +PackGlossProvider → reviewed gloss sentences +PackDomainProvider → dotted-domain disclosure +OOVInvitationProvider → "I haven't learned X yet" +VaultRecallProvider → vault-grounded candidates (when warranted) +ArticulationProvider → realizer/walk fallback (lowest rank) +``` + +Today these all live as branches in +`chat/runtime.py::_maybe_pack_grounded_surface()` and parallel +dispatchers. The selector collapses them into a registered list. + +### Selection + +```python +def select(candidates: Sequence[SurfaceCandidate], + context: SelectionContext) -> SurfaceCandidate | None: + """One pure function over the candidate list. + + Returns the highest-ranked candidate where: + - is_user_facing_safe is True + - the candidate's intent matches context.intent + - the candidate respects context.constraints (e.g. cold_start + suppresses VaultRecallProvider on turn 1) + + Ordering: + refusal > teaching > pack_gloss > pack_domain > oov > vault > walk + + Within the same grounding_source rank, prefer + is_fluent_sentence=True. + """ +``` + +### Single emission point + +```python +def chat(self, text: str, ...) -> ChatResponse: + ... + candidates = self._collect_candidates(intent, subject, field_state) + chosen = self._selector.select(candidates, context) + # one emission point: telemetry, hash, ChatResponse all + # built from the same `chosen` object. + self._emit_turn_event(chosen, ...) + return ChatResponse(surface=chosen.surface, ..., chosen=chosen) +``` + +The pipeline either consumes the runtime's chosen candidate or +becomes a provider itself. Either way, there is exactly one +selector, one emission point, one trace-hash input. + +## What gets fixed by this + +| Symptom | Fixed how | +|---|---| +| Warm-grounding instability | `_collect_candidates` queries `PackGlossProvider`/`PackDomainProvider` by intent + lemma residency, independent of vault gate state. Turn N>1 produces the same candidate set as turn 1 for the same prompt. | +| Telemetry / hash drift | Single emission point. No second-pass override path exists. | +| `chat/runtime.py::_maybe_pack_grounded_surface` dispatcher pile | Decomposes into one provider per surface family. Adding a new surface kind = adding a new provider, not editing the dispatcher. | +| Pipeline-vs-runtime fragmentation | Pipeline either consumes the runtime selection or registers itself as a provider; in both cases the user-facing selection happens exactly once. | +| Spine fragmentation (separate RFC) | `core chat` / `core trace` / cognition eval / pulse all call into the same selector entry point. | + +## What does NOT get fixed by this + +- Gloss content coverage (handled by Phase C) +- Intent classification gaps (handled by `b52e04a`) +- Pipeline placeholder prose (already cured by `c3e2a22` — selector + enforces the same `_is_useful_surface` check as a candidate filter) +- Subjective fluency (out of scope — selector doesn't author content) + +## Migration shape + +1. Create `chat/surface_selector.py` with `SurfaceCandidate`, + `SurfaceProvider` protocol, and `SurfaceSelector.select()`. +2. Add a `SurfaceCandidate.from_pack_surface_candidate()` adapter so + the existing `chat/pack_surface_candidate.py::PackSurfaceCandidate` + becomes a SurfaceCandidate via 1-line conversion. +3. Wrap each existing dispatcher branch in + `_maybe_pack_grounded_surface()` as a provider. No behaviour + change yet — each provider emits the same surface its branch + used to emit. +4. Replace `_maybe_pack_grounded_surface()` with + `selector.select(_collect_candidates(...))`. Behaviour-preserving + if the selector's ordering matches the dispatcher's order. +5. Move the pipeline's override path through the selector. +6. Add `core chat` / `core trace` integration paths. + +Each step is a separate commit, each ends green, each is independently +revertable. None requires authoring new content. + +## Testing approach + +- `warmed_session_consistency` already pins the lift target. + `warm_grounding_stability` going from 0.0 to 1.0 is the regression + signal for the selector landing. +- `deterministic_fluency` continues to pin the structural floor. +- New `tests/test_surface_selector.py` for the selector itself: + ordering invariants, candidate-set determinism, single-emission + invariant (only one TurnEvent emitted per chat call). + +## Risk register + +- **Pipeline integration** — touches `core/cognition/pipeline.py`. + Mitigation: the pipeline override gate (`c3e2a22`) already filters + unuseful surfaces, so the selector's job is mostly to consume the + runtime's chosen candidate. Pipeline-specific override semantics + may be expressible as a single ordered provider. +- **Trace hash drift** — selector emits one hash; pipeline's + separate hash path goes away. Old test fixtures with frozen + hashes (`tests/test_cognitive_turn_pipeline.py:119-123`) compare + run-to-run, not against stored values, so this is safe. +- **OOV / refusal precedence** — already encoded by the ordering + rank. The current dispatcher's order is preserved by spec. + +## When to land + +Best landed by the engineer who wrote the 2026-05-19 design review +(they have the cleanest model of the call sites that need migration). +Solo-landable when warmed-session work is the next architectural +priority. Not for accidental work — this refactor crosses too many +files to reverse cheaply.