core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

docs(notes): SurfaceSelector + spine-unification RFCs + lift baseline

Browse source 
Three companion docs to the 2026-05-19 fluency push.  Captures the
deferred architectural work and the measured lift so the next
engineering pass has fixed substrate to build on.

notes/surface_selector_design_2026-05-19.md
  Deferred RFC for the typed-candidate-lattice + single-selector
  refactor the 2026-05-19 design review prescribed.  Names the
  remaining symptom this fixes (warm_grounding_stability=0 on the
  warmed lane) and the migration shape: PackSurfaceCandidate
  already shipped in commit 46ac737 is a structural subset of the
  proposed SurfaceCandidate type.  Six-step landing plan; each
  step ends green and is independently revertable.

notes/spine_unification_design_2026-05-19.md
  Companion RFC for the cognitive-spine unification.  Enumerates
  the three spines today (ChatRuntime.chat, CognitiveTurnPipeline,
  scripts/run_pulse) + 5 eval-lane runners that split between
  them.  Proposes one canonical entrypoint with opt-in mode
  parameter.  Depends on the SurfaceSelector landing first.

notes/fluency_lift_baseline_2026-05-19.md
  Numbers-only baseline.  Per-lane before/after metrics across
  cold_start_grounding, warmed_session_consistency,
  deterministic_fluency, and cognition (public + holdout).
  Sample probe showing fluent vs. structured-disclosure output
  for 6 prompts.  Lexicon + gloss coverage by pack (323/331 =
  97.6% English-pack coverage).  Reproducer command at the bottom
  so anyone can re-measure in one paste.

Both RFCs explicitly document what's IN scope (so the next pass
isn't ambiguous) and what's OUT of scope (so it isn't accidentally
absorbed).  Both flag the appropriate landing surface (reviewer's
track, not solo) and the dependency order.

No code change in this commit.  Pure documentation.
This commit is contained in:
Shay 2026-05-19 07:38:34 -07:00
parent 07da601641
commit 269372a3a8
3 changed files with 518 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

186
notes/fluency_lift_baseline_2026-05-19.md Normal file
View file

@ -0,0 +1,186 @@
# Fluency Lift Baseline — 2026-05-19
Numbers-only record of the 2026-05-19 fluency push. Captures what
changed, where, and by how much, so subsequent work has a fixed
substrate to measure against.
## Eval lanes — before / after
All three lanes are run from a fresh `ChatRuntime()` per case
(cold-start invariant) except where noted.
### `cold_start_grounding` (44 conversational prompts)
| | Before intent fix (b52e04a) | After intent fix | After gloss landing (07da601) |
|---|---|---|---|
| `intent_accuracy` | 0.4773 | 1.0000 | 1.0000 |
| `grounding_accuracy` | 0.4773 | 1.0000 | 1.0000 |
| `subject_accuracy` | 0.4318 | 1.0000 | 1.0000 |
| `none` count | 21 / 44 | 0 / 44 | 0 / 44 |
| `pack` count | 19 / 44 | 39 / 44 | 39 / 44 |
Five intent-classification patterns recovered 21 prompts that
previously fell to `"I don't know — insufficient grounding"`:
`Define X`, `What does X mean?`, `What is to V?`, `How does X work?`,
`What causes X?`.
### `warmed_session_consistency` (8 cases / 18 turns)
| | Before pipeline gate (Phase B1) | After pipeline gate (c3e2a22) |
|---|---|---|
| `no_placeholder_rate` | 0.4444 | 1.0000 |
| `telemetry_consistency_rate` | 0.4444 | 1.0000 |
| `warm_grounding_stability` | 0.0000 | 0.0000 |
| `grounding_match_rate` | 0.4444 | 0.4444 |
The pipeline-override usefulness gate cured the placeholder-prose
bug + the telemetry/result mismatch. `warm_grounding_stability`
remains 0 because of a separate architectural bug: a pack-grounded
turn 1 reverts to vault-walk on turn 2 of the same prompt. Fix
deferred to the SurfaceSelector RFC (`notes/surface_selector_design_2026-05-19.md`).
### `deterministic_fluency` (15 cases × 6 predicates)
| | Before gloss landing | After gloss landing (07da601) |
|---|---|---|
| `no_placeholder_rate` | 1.0000 | 1.0000 |
| `complete_punctuation_rate` | 1.0000 | 1.0000 |
| `finite_predicate_shape_rate`| 1.0000 | 1.0000 |
| `no_provenance_only_rate` | 1.0000 | 1.0000 |
| `surface_provenance_match_rate` | 1.0000 | 1.0000 |
| `no_dotted_inventory_rate` | **0.3333** | **1.0000** |
The gloss feature delivered the no_dotted_inventory metric from
33% to 100%. Every gloss-backed surface now reads as a fluent
sentence instead of structured-disclosure dotted paths.
### `cognition` (CORE's authoritative cognitive eval)
| | Public (13 cases) | Holdout (19 cases) |
|---|---|---|
| `intent_accuracy` | 1.0000 | 1.0000 |
| `term_capture_rate` | 0.9167 | 0.8333 |
| `surface_groundedness`| 1.0000 | 1.0000 |
| `versor_closure_rate` | 1.0000 | 1.0000 |
**Byte-identical** across every change in this push. Substring
assertions in the eval continue to find every expected term in the
new fluent surfaces.
## Sample probe — fluent vs. before
Fresh `ChatRuntime()` per prompt:
```text
input: What is truth?
before: truth — pack-grounded (en_core_cognition_v1):
cognition.truth; logos.core; epistemic.ground.
No session evidence yet.
after: Truth is a claim or state grounded by evidence and coherent
judgment. pack-grounded (en_core_cognition_v1).
input: Define moment.
before: I don't know — insufficient grounding for that yet.
after: Moment is a brief or pointlike interval of time.
pack-grounded (en_core_temporal_v1).
input: What does important mean?
before: I don't know — insufficient grounding for that yet.
after: Something is important when it carries weight or priority in
some judgment context. pack-grounded (en_core_attitude_v1).
input: What is to create?
before: I haven't learned 'to create' yet (intent: definition).
Mounted lexicon packs: en_core_cognition_v1, ...
after: To create means to bring something into existence through
deliberate action. pack-grounded (en_core_action_v1).
input: What is quasar? (genuinely OOV — control)
both: I haven't learned 'quasar' yet (intent: definition).
Mounted lexicon packs: ...
input: How does memory work? (CAUSE w/o teaching chain — control)
both: I don't know — insufficient grounding for that yet.
(deliberately preserved as the discovery-gap signal)
```
## Lexicon + gloss inventory
After this push:
| | Lexicon entries | Glosses | Coverage |
|---|---|---|---|
| en_core_cognition_v1 | 85 | 78 | 91.8% |
| en_core_meta_v1 | 73 | 72 | 98.6% |
| en_core_attitude_v1 | 40 | 40 | 100.0% |
| en_core_temporal_v1 | 28 | 28 | 100.0% |
| en_core_action_v1 | 26 | 26 | 100.0% |
| en_core_quantitative_v1 | 24 | 24 | 100.0% |
| en_core_spatial_v1 | 24 | 24 | 100.0% |
| en_core_polarity_v1 | 16 | 16 | 100.0% |
| en_core_causation_v1 | 15 | 15 | 100.0% |
| **Total** | **331** | **323** | **97.6%** |
The 8 unglossed entries in cognition are dual-POS lemmas (e.g.
`cause` exists as NOUN and VERB; only the more salient POS got a
gloss in the first dispatch). Adding the duals is a follow-up
authoring pass.
## Commits in this push
```
07da601 feat(packs): seed 323 reviewed glosses across 9 English content packs
46ac737 feat(pack-grounding): selector-ready gloss wiring via PackSurfaceCandidate
24daebf feat(pack-resolver): gloss resolver with lexicon-residency + dual-checksum hardening
c3e2a22 fix(pipeline): usefulness gate on realized-plan override
a67a3cc feat(evals): deterministic_fluency lane — six structural predicates
0cf1a8f feat(evals): warmed_session_consistency lane — pipeline override regression substrate
c6b4f1d fix(runtime): config-replace + thin API wrappers + stale docstring
a084f1d feat(evals): cold_start_grounding lane — 44-prompt routing probe
b52e04a fix(intent): five conversational definition patterns + polarity-stopword
```
Earlier in the session (now ancestors of the above):
```
8 commits seeding 9 new English content packs (230 lemmas, 5x prior coverage)
```
## What's deferred (with rationale)
- **SurfaceSelector refactor** — `notes/surface_selector_design_2026-05-19.md`
Cures `warm_grounding_stability`. Crosses runtime + pipeline +
telemetry + hash. Solo-landing carries too much blast radius;
reviewer is best positioned to land it.
- **Spine unification** — `notes/spine_unification_design_2026-05-19.md`
Cures `core chat` ≠ pipeline-eval drift. Depends on the
SurfaceSelector landing first.
- **Cognition dual-POS gloss completion** — 8 cognition lemmas have
dual entries (NOUN+VERB) where only one got a gloss. Mechanical
follow-up; one subagent dispatch can close it.
- **Gloss-formed sentences for AUX/PRON/SCONJ** — three lemmas in
cognition (`be`, `why`, `because`) have glosses authored to a
specific frame. Manual QA pass on the resulting surface is
pending.
## Reproducing the numbers
```bash
core eval cold_start_grounding
core eval warmed_session_consistency
core eval deterministic_fluency
core eval cognition
core eval cognition --split holdout
# Live probe:
python3 -c "
from chat.runtime import ChatRuntime
for p in ['What is truth?', 'Define moment.', 'What does important mean?',
'What is to create?', 'How does memory work?']:
r = ChatRuntime().chat(p)
print(f'[{r.grounding_source}] {p}\n -> {r.surface}\n')
"
```

144
notes/spine_unification_design_2026-05-19.md Normal file
View file

@ -0,0 +1,144 @@
# Cognitive-Spine Unification — Deferred RFC
**Date:** 2026-05-19
**Status:** Design proposal — NOT implemented. Deferred from the
2026-05-19 fluency push because the change crosses public-API
entrypoints and depends on the SurfaceSelector landing.
**Companion RFC:** `notes/surface_selector_design_2026-05-19.md`
## Motivation
The 2026-05-19 design review's Finding P0 #2:
> The live cognitive spine is fragmented across public entrypoints.
| Entrypoint | Today's spine | Affected by intent fixes? | Affected by pipeline fixes? |
|---|---|---|---|
| `core chat` (REPL) | `ChatRuntime.chat()` direct | ✓ yes | ✗ no |
| `core trace` (single turn)| `ChatRuntime.chat()` direct | ✓ yes | ✗ no |
| `core pulse` (research) | `scripts/run_pulse.py` graph diffusion + GloVe-seeded | ✗ no | ✗ no |
| `evals/cognition/runner.py`| `CognitiveTurnPipeline.run()` | ✓ yes | ✓ yes |
| `evals/cold_start_grounding/runner.py` | `ChatRuntime.chat()` direct | ✓ yes | ✗ no |
| `evals/warmed_session_consistency/runner.py` | `CognitiveTurnPipeline.run()` | ✓ yes | ✓ yes |
| `evals/deterministic_fluency/runner.py` | `ChatRuntime.chat()` direct | ✓ yes | ✗ no |
Three separate cognitive spines exist:
1. `ChatRuntime.chat()` direct — the simplest path, used by `core chat`
and `core trace`.
2. `CognitiveTurnPipeline.run()` — wraps `ChatRuntime.chat()` and adds
a graph-realizer override step + transitive walk + frame compose.
3. `scripts/run_pulse.py` — independent path with GloVe seeding,
graph constraint correction, top-k recall.
Effects of fragmentation:
- A fix to the pipeline's override behaviour does not reach the
user via `core chat`.
- A fix to the runtime reaches the user but is masked under the
pipeline-wrapped eval lanes.
- Pulse can "prove" capabilities the user never experiences.
- Tests can be green while user behaviour is broken (and vice versa).
## Proposed direction
### One canonical chat spine
`ChatRuntime.chat()` becomes the single canonical entrypoint. The
pipeline's value-add (transitive walks, frame composition) moves
INSIDE the runtime as opt-in passes consulted by the selector:
```python
class ChatRuntime:
def chat(self, text, *, max_tokens=None, mode="full"):
# mode="full" — runtime + pipeline-equivalent passes
# mode="bridge" — runtime only (today's bridge path)
# mode="walk" — walk evidence only (research / introspection)
...
```
The pipeline becomes a thin convenience wrapper that selects a mode:
```python
class CognitiveTurnPipeline:
def run(self, text, *, max_tokens=None):
# Equivalent to ChatRuntime.chat(text, mode="full").
# Retained as the API the cognition eval harness was built
# against; new code calls ChatRuntime.chat() directly.
...
```
### Pulse demoted to research harness
`scripts/run_pulse.py` keeps existing for the geometry-research path
but is labeled non-canonical. It does not contribute to "fluent
chat" claims; the eval lanes that rely on it (if any) are renamed.
### Single selector consumed everywhere
The SurfaceSelector (companion RFC) is the only path that emits the
user-facing surface. All entrypoints route through it:
```
user input
─▶ ChatRuntime.chat(text, mode=…)
─▶ collect_candidates(intent, subject, field_state, mode)
─▶ SurfaceSelector.select(candidates, context)
─▶ ChatResponse(surface=chosen.surface, …)
```
`core chat`, `core trace`, every eval lane, and the pipeline shim
all call `ChatRuntime.chat()` with different modes. One emission
point, one telemetry record, one trace hash.
## What this fixes
| Today | After |
|---|---|
| Pipeline override invisible to `core chat` | Pipeline's value-add is opt-in modes inside the runtime; visible everywhere or nowhere |
| Eval-vs-user behaviour drift | Same code path; can't drift |
| Pulse "proves" things the user doesn't see | Pulse explicitly labeled non-canonical |
| Tests asserting `r.surface == r.walk_surface` | Surface is the selector's output; walk_surface remains audit telemetry |
| Three places to add a fluency surface | One: register a SurfaceProvider |
## Sequencing
This RFC is **dependent on** `surface_selector_design_2026-05-19.md`.
Land in this order:
1. SurfaceSelector + provider registry (the companion RFC)
2. Wrap each existing dispatcher branch as a provider
3. Re-implement pipeline override as a provider (or remove if the
selector handles it via ordering)
4. Move `core chat` to `ChatRuntime.chat(mode="full")`
5. Move `core trace` to `ChatRuntime.chat(mode="full")` with trace
instrumentation
6. Audit eval lanes — every lane explicitly declares its mode
7. Label `scripts/run_pulse.py` as non-canonical in its docstring
and any eval lane that depends on it
## What does NOT change
- Pack content (already correct authoring path)
- Teaching chains (already correct authoring path)
- Intent classification (already canonical via `generate.intent`)
- Telemetry schema (one emitter, one shape)
- Trace-hash stability (intra-session; hashes are still per-run)
## Risk register
- **Public API stability** — `CognitiveTurnPipeline.run()` cannot
be removed without a deprecation cycle. Migration step is a
wrapper, not a removal.
- **Mode semantics** — the three modes (`full` / `bridge` / `walk`)
must be documented in `docs/runtime_contracts.md` BEFORE the
refactor so users can rely on them.
- **Eval invariant** — `cognition` eval expects pipeline-level
behaviour. The wrapper preserves that; verifiable byte-identity
on the eval is a hard prerequisite to commit.
## When to land
After the SurfaceSelector RFC. Spine unification without the
selector would just move the fragmentation; with the selector it
collapses the spines onto a single, observable, replayable path.

188
notes/surface_selector_design_2026-05-19.md Normal file
View file

@ -0,0 +1,188 @@
# SurfaceSelector Design — Deferred RFC
**Date:** 2026-05-19
**Status:** Design proposal — NOT implemented. Deferred from the
2026-05-19 fluency push because the refactor crosses too many files
to land safely solo. Picks up cleanly when the reviewer is back.
**Seed type:** `chat/pack_surface_candidate.py::PackSurfaceCandidate`
(landed in commit 46ac737 — already shaped for selector consumption)
## Motivation
The 2026-05-19 design review identified the central architectural
fault behind the fluency bug class:
> **Surface authority fragmentation.** The runtime, pipeline, walk
> evidence, pack fallback, telemetry, hash, and several composers all
> compete for the user-facing sentence. The system can be "green"
> while user-facing selection, pipeline selection, and telemetry
> selection disagree.
Phase B1's pipeline-override usefulness gate (commit `c3e2a22`) cured
two symptoms:
- `no_placeholder_rate`: 0.44 → 1.00
- `telemetry_consistency_rate`: 0.44 → 1.00
But it did not cure the third:
- `warm_grounding_stability`: 0.0 (warmed lane)
The warmed-session bug — a pack-grounded surface on turn 1 reverting
to a vault-walk fragment on turn 2 of the same prompt — is the
remaining symptom. Its fix is structural, not patchable: pack-
grounding must fire by intent + lemma residency, not by vault gate
state. This is the SurfaceSelector's job.
## Proposed design
### One typed candidate per provider
```python
@dataclass(frozen=True, slots=True)
class SurfaceCandidate:
surface: str # final user-facing string
grounding_source: str # "refusal"/"teaching"/"pack"/...
provider_id: str # which provider emitted this
rank_hint: int # provider's own confidence tier
pack_id: str | None # provenance (None for refusal/OOV)
intent: IntentTag
subject_lemma: str | None
semantic_domains: tuple[str, ...] # audit content
epistemic_status: str # "asserted"/"hedged"/"refused"/...
is_user_facing_safe: bool
is_fluent_sentence: bool
is_replayable: bool
```
`PackSurfaceCandidate` (already landed) is a structural subset of
this — the migration is a strict superset rename + a few new fields.
### Providers
Each provider has one job: produce a candidate (or None) for the
given intent + subject. No selection logic; no telemetry; no
side effects beyond the candidate's audit fields.
```
RefusalProvider → refusal candidates (safety/ethics)
TeachingChainProvider → reviewed-corpus chain candidates
CrossPackChainProvider → cross-pack chain candidates
PackGlossProvider → reviewed gloss sentences
PackDomainProvider → dotted-domain disclosure
OOVInvitationProvider → "I haven't learned X yet"
VaultRecallProvider → vault-grounded candidates (when warranted)
ArticulationProvider → realizer/walk fallback (lowest rank)
```
Today these all live as branches in
`chat/runtime.py::_maybe_pack_grounded_surface()` and parallel
dispatchers. The selector collapses them into a registered list.
### Selection
```python
def select(candidates: Sequence[SurfaceCandidate],
context: SelectionContext) -> SurfaceCandidate | None:
"""One pure function over the candidate list.
Returns the highest-ranked candidate where:
- is_user_facing_safe is True
- the candidate's intent matches context.intent
- the candidate respects context.constraints (e.g. cold_start
suppresses VaultRecallProvider on turn 1)
Ordering:
refusal > teaching > pack_gloss > pack_domain > oov > vault > walk
Within the same grounding_source rank, prefer
is_fluent_sentence=True.
"""
```
### Single emission point
```python
def chat(self, text: str, ...) -> ChatResponse:
...
candidates = self._collect_candidates(intent, subject, field_state)
chosen = self._selector.select(candidates, context)
# one emission point: telemetry, hash, ChatResponse all
# built from the same `chosen` object.
self._emit_turn_event(chosen, ...)
return ChatResponse(surface=chosen.surface, ..., chosen=chosen)
```
The pipeline either consumes the runtime's chosen candidate or
becomes a provider itself. Either way, there is exactly one
selector, one emission point, one trace-hash input.
## What gets fixed by this
| Symptom | Fixed how |
|---|---|
| Warm-grounding instability | `_collect_candidates` queries `PackGlossProvider`/`PackDomainProvider` by intent + lemma residency, independent of vault gate state. Turn N>1 produces the same candidate set as turn 1 for the same prompt. |
| Telemetry / hash drift | Single emission point. No second-pass override path exists. |
| `chat/runtime.py::_maybe_pack_grounded_surface` dispatcher pile | Decomposes into one provider per surface family. Adding a new surface kind = adding a new provider, not editing the dispatcher. |
| Pipeline-vs-runtime fragmentation | Pipeline either consumes the runtime selection or registers itself as a provider; in both cases the user-facing selection happens exactly once. |
| Spine fragmentation (separate RFC) | `core chat` / `core trace` / cognition eval / pulse all call into the same selector entry point. |
## What does NOT get fixed by this
- Gloss content coverage (handled by Phase C)
- Intent classification gaps (handled by `b52e04a`)
- Pipeline placeholder prose (already cured by `c3e2a22` — selector
enforces the same `_is_useful_surface` check as a candidate filter)
- Subjective fluency (out of scope — selector doesn't author content)
## Migration shape
1. Create `chat/surface_selector.py` with `SurfaceCandidate`,
`SurfaceProvider` protocol, and `SurfaceSelector.select()`.
2. Add a `SurfaceCandidate.from_pack_surface_candidate()` adapter so
the existing `chat/pack_surface_candidate.py::PackSurfaceCandidate`
becomes a SurfaceCandidate via 1-line conversion.
3. Wrap each existing dispatcher branch in
`_maybe_pack_grounded_surface()` as a provider. No behaviour
change yet — each provider emits the same surface its branch
used to emit.
4. Replace `_maybe_pack_grounded_surface()` with
`selector.select(_collect_candidates(...))`. Behaviour-preserving
if the selector's ordering matches the dispatcher's order.
5. Move the pipeline's override path through the selector.
6. Add `core chat` / `core trace` integration paths.
Each step is a separate commit, each ends green, each is independently
revertable. None requires authoring new content.
## Testing approach
- `warmed_session_consistency` already pins the lift target.
`warm_grounding_stability` going from 0.0 to 1.0 is the regression
signal for the selector landing.
- `deterministic_fluency` continues to pin the structural floor.
- New `tests/test_surface_selector.py` for the selector itself:
ordering invariants, candidate-set determinism, single-emission
invariant (only one TurnEvent emitted per chat call).
## Risk register
- **Pipeline integration** — touches `core/cognition/pipeline.py`.
Mitigation: the pipeline override gate (`c3e2a22`) already filters
unuseful surfaces, so the selector's job is mostly to consume the
runtime's chosen candidate. Pipeline-specific override semantics
may be expressible as a single ordered provider.
- **Trace hash drift** — selector emits one hash; pipeline's
separate hash path goes away. Old test fixtures with frozen
hashes (`tests/test_cognitive_turn_pipeline.py:119-123`) compare
run-to-run, not against stored values, so this is safe.
- **OOV / refusal precedence** — already encoded by the ordering
rank. The current dispatcher's order is preserved by spec.
## When to land
Best landed by the engineer who wrote the 2026-05-19 design review
(they have the cleanest model of the call sites that need migration).
Solo-landable when warmed-session work is the next architectural
priority. Not for accidental work — this refactor crosses too many
files to reverse cheaply.