Commit graph

15 commits

Author SHA1 Message Date
Shay
79f1678923 feat: ADR-0086 + ADR-0087 + 100-register catalog — cognition lane closure
Three load-bearing pieces:

1. ADR-0086 — UNKNOWN-intent pack-resident token surface
   New deterministic composer `pack_grounded_unknown_surface` in
   chat/pack_grounding.py.  When intent classification returns UNKNOWN
   but the prompt contains pack-resident lemmas (via cross-pack
   resolver), surface those lemmas with their semantic_domains
   instead of falling to the bare _UNKNOWN_DOMAIN_SURFACE.  Wired
   into chat/runtime.py::_maybe_pack_grounded_surface as the
   last typed-intent branch before the OOV fallback.  Null-lift
   invariant pinned: fully-OOV prompts still emit the universal
   disclosure byte-identically.  Closes four cognition-eval term
   misses: unknown_logos_019 (public), unknown_evidence_042 (dev),
   unknown_spirit_041 + unknown_word_018 (holdout).  Side effect:
   evals/results/phase2_pack_measurements.json refusal_rate drops
   from 0.25 → 0.125 across all three identity packs (no longer
   refusing on these prompts).

2. ADR-0087 — PROCEDURE selector + trailing-clause subject echo
   Two coupled changes in chat/pack_grounding.py:
   (a) Numeric-determiner downrank in _extract_procedure_topic_lemma:
       tokens whose primary semantic_domain starts with
       "quantitative.numeric." are demoted; non-numeric resident
       candidates always win.  So "compare two terms" anchors on
       `compare` not `two`.
   (b) Trailing clause echoes the full normalized subject_text
       rather than just the selected lemma, so OOV head nouns like
       "terms" reach the surface even when only the procedure verb
       is pack-resident.  Closes procedure_compare_011.

3. 100-register catalog
   New packs/register/_catalog.json — canonical machine-readable
   spec for all 100 registers (7 currently-ratified + 93 drafted)
   organized into 9 voice groups (depth/tone/stance/posture/domain/
   cultural/affective/functional/composite).  Each entry is a
   complete production input — realizer_overrides, marker palettes
   (openings/transitions/closings), depth_preference, description,
   author_notes.  All realizer_overrides use only legal keys per
   scripts/ratify_register_packs.py::_KNOWN_OVERRIDE_KEYS.
   Companion packs/register/CATALOG.md documents the production
   loop: materialize → widen REGISTER_IDS → ratify → smoke.

Cognition-eval lifts (all three splits):
  public:  term_capture 91.7% → 100.0%  (+8.3pp)
  holdout: term_capture 83.3% → 100.0%  (+16.7pp)
  dev:     term_capture 78.6% → 100.0%  (+21.4pp)
  surface_groundedness: 100% preserved on all splits
  intent_accuracy / versor_closure: 100% preserved on all splits

Tests:
  tests/test_pack_grounded_unknown.py     — 14 tests (composer
    direct + runtime engagement + null-lift invariant)
  tests/test_adr_0087_procedure_selector.py — 12 tests (selector
    numeric downrank + trailing-clause echo + regression guard)
  Existing test suites unaffected — cognition lane 120 passed / 1
  skipped both before and after.  Full lane net −3 failures vs
  pristine main (39 → 36 — none introduced).
2026-05-21 00:08:12 -07:00
Shay
4b9404a88e
feat(adr-0085): gloss-aware CAUSE composer — explanation frame from glosses (#70)
The original "Why does light exist?" complaint that motivated ADR-0084
was specifically about CAUSE-intent surfaces. ADR-0084 (substrate) +
PR #65 (content) already moved DEFINITION/RECALL to gloss-grounded
surfaces ("Light is visible medium that reveal truth."). But CAUSE
still dispatched through the chain-walk path:

  Before: light — teaching-grounded (cognition_chains_v1):
            cognition.illumination; logos.core.
            light reveals truth (cognition.truth).
            No session evidence yet.

  After:  Light exists as visible medium that reveal truth.
          pack-grounded (en_core_cognition_v1).

The chain-walk is structurally correct but the wrong SHAPE for a why-
question — it's a graph traversal, not an explanation. ADR-0085 fixes
the shape using the same gloss material that DEFINITION/RECALL already
consume, with no new content authoring.

Additive composer
  chat/pack_grounding.py:gloss_aware_cause_surface()
  - Resolves gloss via lexicon-residency-checked resolve_gloss().
  - Frames POS-aware:
      NOUN -> "{Lemma} exists as {gloss}."
      VERB -> "To {lemma} is to {gloss}."
      ADJ  -> "To be {lemma} is to {gloss}."
      *    -> falls back to _frame_gloss (predicate-identity).
  - Threads anchor lens via the existing helper (ADR-0073c parity).
  - Returns None when no gloss exists — runtime falls through to the
    existing chain-walk path. Additive: no CAUSE case loses its surface.

Runtime dispatch
  chat/runtime.py — IntentTag.CAUSE tries gloss path FIRST under the
  flag; falls through to teaching_grounded_surface* on None.
  Unconditional fallback — never silent.

Opt-in flag
  core/config.py — RuntimeConfig.gloss_aware_cause: bool = False
  Default off preserves pre-ADR-0085 chain-walk surfaces byte-
  identically (null-drop invariant, CI-pinned).

Prompt-diversity classifier update
  evals/prompt_diversity/runner.py — _CAUSE_MARKERS widened with the
  explanation-frame markers ("exists as", "is to", "to be", "is for",
  "purpose of") plus bare-form predicates ("reveal" alongside
  "reveals"). Neither composer path is penalised on shape_fit just on
  inflection grounds.

v1/public lift (flag OFF vs ON, 26 cases)
  intent_accuracy        : 65.4% -> 65.4%   ( — )
  versor_closure_rate    : 100.0% -> 100.0% ( — )
  response_shape_fit     : 57.7% -> 57.7%   ( — , both frames recognized)
  audit_in_surface_rate  : 42.3% -> 42.3%   ( — , envelope ADR's job)
  gloss_quote_rate       : 11.5% -> 23.1%   (+11.5pp, structural lift)

Tests (15)
  - 5 pure composer (NOUN/VERB frame, unknown/empty None, no chain-
    walk artifacts in surface)
  - 5 runtime dispatch (flag-off chain-walk, flag-on gloss, parametrized
    across glossed subjects, VERIFICATION unchanged under flag, no-
    gloss fallback engages)
  - 5 cognition lane invariance (aggregate metrics byte-identical
    under both flag states; surfaces deliberately shift on the 2 CAUSE
    cases with glossed subjects — the structural-change-vs-metric-
    invariance both-sides invariant)

Lanes
  smoke 67/0, cognition 120/0/1 skipped, packs 6/0, teaching 17/0,
  runtime 19/0. core eval cognition byte-identical 100/91.7/100/100
  under both flag states.

Scope limits (per ADR §Scope limits)
  - CAUSE only; VERIFICATION still chain-walks (different shape).
  - English pilot only; Greek/Hebrew packs not opted into definitional
    layer yet (ADR-0084 scope limit).
  - Single-lemma subjects; compound/anaphoric fall through.
  - Opt-in until cognition holdout confirms the lift transfers off-
    fixture. Future PR flips default on.

Out of scope
  - Surface-vs-envelope cleanup ("pack-grounded (...)" still leaks).
  - Predicate licensing (ADR-0086).
  - Content style pass (bare lemma forms in glosses — separate brief).
2026-05-20 15:55:08 -07:00
Shay
6387872051 feat(packs): en_collapse_anchors_v1 — activate chesed/shalom/tzedek lenses on EN input
ADR-0073c shipped he_chesed_v1, he_shalom_v1, he_tzedek_v1 with lossy
EN-collapse alignment edges (he-021 → en-collapse-love @ 0.63, etc.)
but the synthetic en-collapse-* targets didn't exist in any mounted
lexicon.  Result: the three lenses ratified but stayed dormant — the
runtime OOV gate fired on "What is love?" / "What is peace?" /
"What is justice?" before the lens engagement path got a chance.

This commit adds a minimal pack whose lexicon carries exactly those
three synthetic anchors:

  en-collapse-love     lemma="love"     domain=collapse_anchor.love
  en-collapse-peace    lemma="peace"    domain=collapse_anchor.peace
  en-collapse-justice  lemma="justice"  domain=collapse_anchor.justice

Mounted last in DEFAULT_RESOLVABLE_PACK_IDS — cognition / relations
packs win first-match on any future collision.  No real content pack
currently carries these lemmas (grep-confirmed) so the mount adds no
collision risk.

The pack-grounded surface for "What is love?" advertises its nature
honestly via the pack id (en_collapse_anchors_v1) and the domain
string (collapse_anchor.love) — the surface is intentionally minimal;
the substantive content arrives via the lens annotation
[lens(he_chesed_v1):covenant-love] / [lens(he_shalom_v1):wholeness-peace] /
[lens(he_tzedek_v1):right-order].

chat/pack_grounding.py:_en_lemma_to_entry_id() now reads both
en_core_cognition_v1 and en_collapse_anchors_v1, with cognition
winning on lemma collision.

New test file tests/test_en_collapse_anchors_v1_pack.py pins:
  - each anchor lemma resolves to its synthetic entry_id
  - collapse pack mounted last (precedence guarantee)
  - each of the three lenses engages on its target English prompt
  - baseline surface (no lens) still advertises anchor nature

Validation:
  - Cognition eval byte-identical (100/100/91.7/100)
  - 160 lens/pack/resolver tests pass + 8 new
  - anchor-lens-tour green
  - register-tour green
2026-05-20 10:58:07 -07:00
Shay
d7499c80b3
feat(intent): normalize confirmation-tag propositions (#45) 2026-05-19 22:55:28 -07:00
Shay
b35bec6465 feat(anchor_lens): ADR-0073c — L1.3 first lenses + composer wiring
L1.3 of the anchor-lens inside-out rollout — first substantive
surface lift on the substantive axis.  Two ratified non-trivial
lenses engage on cognition-pack lemmas via the alignment graph,
appending [lens(<id>):<mode>] annotations to the existing
pack-grounded surface.

Two ratified lenses

  grc_logos_v1 (Greek substrate)
    primary_substrate         : "grc"
    semantic_domain_preferences: ["logos.episteme.systematic_knowledge"]
    cognitive_mode_label       : "systematic"
    Engages on en "knowledge" via grc-core-cog-021 (ἐπιστήμη) →
    en-core-cog-007 alignment edge.

  he_logos_v1 (Hebrew substrate)
    primary_substrate         : "he"
    semantic_domain_preferences: ["logos.aletheia.verity"]
    cognitive_mode_label       : "covenant-verity"
    Engages on en "truth" via he-core-cog-002 (אמת) →
    en-core-cog-002 alignment edge.

  Both ratified under method anchor_lens_lifts_proposition.

Engagement rule (single)

  1. Resolve en_lemma → entry_id (cognition pack).
  2. For each substrate pack matching lens.primary_substrate, load
     alignment.jsonl; find edges where target_id == entry_id.
  3. For each such substrate lemma, if any atom in its
     semantic_domains ∈ lens.semantic_domain_preferences → engage.
  4. No match → None (no annotation; byte-identical surface).

The pivot is shared semantic_domain atoms surfaced via the
alignment graph — exactly the language-neutral commitment from
ADR-0073.  Engagement never touches non-English surface text;
entry_ids and atom strings only.

Surface lift

  no-lens : "Knowledge is X. pack-grounded (en_core_cognition_v1)."
  lens-on : "Knowledge is X. pack-grounded (en_core_cognition_v1) [lens(grc_logos_v1):systematic]."

  Annotation between existing provenance and trailing period.
  Both metadata fields are ASCII-bounded ≤64 chars at the loader
  level, so the annotation can never carry non-ASCII.

Scope deliberately narrow

  L1.3 wiring restricted to pack_grounded_surface /
  build_pack_surface_candidate (DEFINITION/RECALL only).  Other
  composers (COMPARISON / CORRECTION / PROCEDURE / NARRATIVE /
  EXAMPLE / CAUSE / VERIFICATION) accept the anchor_lens kwarg via
  forward-compat default UNANCHORED but do not yet consume it.
  L1.3b or later broadens to those intent shapes.

Ratify gate widening

  Non-null lenses must:
    - have primary_substrate ∈ {grc, he, en}
    - have a non-empty cognitive_mode_label
    - every preferred atom must exist in at least one lemma of the
      named substrate (trust boundary: operators cannot ship a lens
      pointing at atoms not on disk).
  Method: anchor_lens_lifts_proposition.  Null lenses still ratify
  under byte_identity_null_lift (L1.2 method).

Seam allow-list widening

  Truth-path modules (cognition / trace / pipeline / intent /
  propagation / vault / algebra) still refused.  Composer-side
  imports from chat/pack_grounding.py now permitted — the same way
  ADR-0069's R2 widened the register seam.

New invariants pinned (3)

  tests/test_anchor_lens_engagement_unit.py (14 tests) — resolver
  returns mode label only on intended substrate × en lemma pair;
  case-insensitive; engagement None under null lens; synthetic
  lens with unmatched atom returns None; annotation is pure ASCII.

  tests/test_anchor_lens_lifts_proposition.py (17 tests) — grc
  engages on knowledge only, he engages on truth only,
  cross-lens isolation, three-way distinctness, replay determinism
  per (lens × prompt), register-tour seam holds within each lens
  scope (orthogonality CI-pinned, parametrized over 4 lens
  choices).

  tests/test_anchor_lens_no_glyph_leak.py (5 tests) — hard
  block-scoped gate: Greek (U+0370..03FF, U+1F00..1FFF), Hebrew
  (U+0590..05FF), Syriac, Arabic.  Stylistic punctuation
  (em-dash etc.) explicitly allowed; em-dash predates L1.3 by a
  wide margin and is not a substrate-leak risk.  Tested per-lens
  across every cognition case + direct lens-metadata ASCII check.

Lane evidence

  74 anchor-lens tests pass (37 from L1.2 + 37 new).
  python -m core.cli eval cognition → public 100/100/91.7/100
  byte-identical (lens=None / default_unanchored_v1).
  core demo register-tour --json → all_claims_supported: True
  (R5 seam still holds; L1.3 doesn't perturb presentation axis).
  Full lane: 2706 passed / 4 skipped / 1 pre-existing failure
  (+37 over L1.2's 2669; the one failure remains
  test_all_preamble_explains_combined_run, unrelated).

Files

  packs/anchor_lens/grc_logos_v1.json                        NEW
  packs/anchor_lens/grc_logos_v1.mastery_report.json         NEW
  packs/anchor_lens/he_logos_v1.json                         NEW
  packs/anchor_lens/he_logos_v1.mastery_report.json          NEW

  scripts/ratify_anchor_lens_packs.py                        EDIT
    LENS_IDS adds grc_logos_v1 / he_logos_v1; gate widened.

  chat/pack_grounding.py                                     EDIT
    _resolve_anchor_lens_mode, _maybe_append_anchor_lens_annotation,
    _substrate_lexicon_by_entry_id, _en_lemma_to_entry_id.
    build_pack_surface_candidate + pack_grounded_surface gain
    anchor_lens kwarg (default UNANCHORED).

  chat/runtime.py                                            EDIT
    Thread self.anchor_lens into pack_grounded_surface() call.

  tests/test_anchor_lens_pack_seam.py                        EDIT
    Doc-comment updated for L1.3 allow-list.

  tests/test_anchor_lens_*                                   NEW (3 files)

  docs/decisions/ADR-0073c-anchor-lens-composer-wiring.md    NEW
2026-05-19 20:06:02 -07:00
Shay
6207b5fd0e feat(register): R1–R4 register pack subsystem — deterministic surface variation
Introduces the presentation axis as a fourth pack class (sibling to identity /
safety / ethics), orthogonal to the truth path. Same input + same packs +
same register ⇒ bit-for-bit reproducible surface; varying any of the three ⇒
genuinely different output. No stochastic sampling.

ADR-0068 (R1): RegisterPack frozen dataclass, loader, ratify script, seam test.
  - default_neutral_v1 ratified as null register.

ADR-0069 (R2): realizer register parameter threaded through 9 composer entry
  points; RuntimeConfig.register_pack_id; three byte-identity invariants
  (A: None ≡ pre-R2 unregistered; B: None ≡ default_neutral_v1; C: trace_hash
  invariant under register). Amended to default-with-lint after 167-call-site
  scout: composers default to UNREGISTERED, AST lint enforces explicit
  register= at runtime call sites.

ADR-0070 (R3): terse_v1 register, first non-neutral pack. realizer_overrides
  schema with known-keys allow-list (disclosure_domain_count ∈ {1,2,3}).
  build_pack_surface_candidate reads override with fail-soft clamp. New
  invariant register_invariant_grounding asserts grounding_source +
  trace_hash byte-identical across {None, neutral, terse}.

ADR-0071 (R4): seeded surface variation via convivial_v1.
  chat/register_variation.py applies SHA-256-seeded marker selection from
  bounded discourse-marker buckets. ChatResponse.pre_decoration_surface routes
  truth-path surface to core/cognition/pipeline.py so trace_hash stays
  invariant under register (the load-bearing architectural fix — initially
  invariant C failed under convivial because decoration was leaking into
  trace_hash via response.surface). Empty-string marker entries now
  legitimate ("no marker this turn" is a valid seed pick). realizer_overrides
  schema widened with per_intent nested block (validated against IntentTag
  whitelist; wired but not exercised by convivial). Two new invariants:
  seeded_variation_replay_equivalence (fresh runtimes → byte-identical) and
  seeded_variation_turn_distinct (same prompt across turns → ≥2 distinct
  surfaces).

ADR-0072 (R5, draft): telemetry + operator surface — TurnEvent gains
  register_id and register_variant_id, core chat --register flag, core demo
  register-tour. Status: Proposed; not yet implemented.

Three ratified register packs ship: default_neutral_v1 (null), terse_v1
(disclosure_domain_count=1), convivial_v1 (3 openings × 3 closings).

Verification:
  - 84 register tests pass + 1 documented skip
  - Curated lanes green: smoke 67, cognition 120+1s, teaching 17, packs 6,
    runtime 19, algebra 132
  - Cognition eval byte-identical to pre-register baseline:
    public 100/100/91.7/100, holdout 100/100/83.3/100
  - Full lane: 2608 passed, 4 skipped, 1 failed (pre-existing
    test_cli_demo.py "Combined Demo" → "Run Every Demo" rename, unrelated)

Truth-path isolation: chat/register_variation.py is realizer-side; the seam
test (tests/test_register_pack_seam.py) refuses imports of packs.register
from intent classification, propagation, vault recall, trace hashing, and
algebra.
2026-05-19 16:52:36 -07:00
Shay
07da601641 feat(packs): seed 323 reviewed glosses across 9 English content packs
Phase C of the gloss feature.  Lands the natural-language gloss
content that the resolver (Phase B2) and the runtime composer
(Phase B3) were prepared for.  This is the user-visible payoff:
cold-start DEFINITION / RECALL prompts on pack-resident lemmas now
emit fluent grounded sentences instead of dotted-domain disclosure.

Authoring: five parallel subagents in ONE message block (a single
parallel dispatch, ~20s wall-clock vs ~95s sequential).  Each
subagent received its pack's complete lemma + POS list and a strict
JSON-shape exemplar.  Total returned: 326 raw gloss entries.

Assembly (this commit): the raw entries were partitioned by
lexicon-residency lookup (the resolve_gloss invariant enforced at
storage time), deduplicated within pack, sorted by lemma, written
to ``language_packs/data/<pack>/glosses.jsonl``, and each pack's
manifest received a new ``glosses_checksum`` field.  323 glosses
landed clean; 0 rejected.

Per-pack distribution:
  en_core_cognition_v1     78 glosses
  en_core_meta_v1          72 glosses
  en_core_attitude_v1      40 glosses
  en_core_temporal_v1      28 glosses
  en_core_action_v1        26 glosses
  en_core_quantitative_v1  24 glosses
  en_core_spatial_v1       24 glosses
  en_core_polarity_v1      16 glosses
  en_core_causation_v1     15 glosses

Live-probe lift (fresh ChatRuntime per prompt):

  BEFORE:
    truth — pack-grounded (en_core_cognition_v1):
      cognition.truth; logos.core; epistemic.ground.
      No session evidence yet.

  AFTER:
    Truth is a claim or state grounded by evidence and coherent
    judgment.  pack-grounded (en_core_cognition_v1).

Same provenance.  Same audit-trail content (the dotted domains are
still in lexicon.jsonl, the resolver can still read them, the
candidate object carries them verbatim).  But the user-facing
surface is a sentence the user can actually read.

Eval-lane lift:

  deterministic_fluency       BEFORE      AFTER
    no_dotted_inventory_rate  0.3333  →   1.0000
    no_provenance_only_rate   1.0000  →   1.0000  (held)
    no_placeholder_rate       1.0000  →   1.0000  (held)
    complete_punctuation_rate 1.0000  →   1.0000  (held)
    finite_predicate_shape    1.0000  →   1.0000  (held)
    surface_provenance_match  1.0000  →   1.0000  (held)
  cold_start_grounding         all metrics held at 1.0
  warmed_session_consistency   no_placeholder + telemetry_match held at 1.0
                              (warm_grounding_stability still 0 — separate fix)
  cognition eval public        100 / 100 / 91.7 / 100   (BYTE-IDENTICAL)
  cognition eval holdout       100 / 100 / 83.3 / 100   (BYTE-IDENTICAL)

  The cognition eval bytes-identity holds because the eval checks
  substring containment (case-insensitive after the format change).
  Every lemma still appears in its fluent surface.

Hardening this commit enforces:

  Lexicon-residency at storage time
    tests/test_pack_glosses_content.py::test_every_gloss_lemma_is_lexicon_resident
    walks every glosses.jsonl and asserts every lemma is present in
    the same pack's lexicon.jsonl.  Drift in glosses (an unratified
    lemma sneaking in) fails the lane immediately.

  Dual-checksum discipline
    tests/test_pack_glosses_content.py::test_every_glossed_pack_has_matching_checksum
    re-hashes glosses.jsonl bytes-on-disk and compares against the
    manifest's glosses_checksum.  Any tampering fails.

  Immutable-lexicon invariant
    tests/test_pack_glosses_content.py::test_lexicon_checksum_unchanged_by_gloss_landing
    re-hashes lexicon.jsonl and compares against the manifest's
    (original) checksum.  Proves that adding glosses did NOT perturb
    the lexicon seal.

  High-freq lemma resolution
    32 of the most-common conversational lemmas (truth, doubt,
    fact, idea, self, true, important, now, place, make, effect,
    always, ...) all resolve to a fluent surface end-to-end.

Test-suite drift this commit absorbed:

  - tests/test_pack_grounding.py — three substring assertions
    updated to be case-insensitive (gloss-backed surfaces capitalize
    lemmas at sentence start, dotted-disclosure surfaces don't).
    "No session evidence yet" assertion replaced with the
    common-substring "pack-grounded" marker that BOTH forms emit.
  - tests/test_pack_resolver_glosses.py — the back-compat test
    pivots from en_core_cognition_v1 (now glossed) to en_minimal_v1
    (deliberately unglossed).  A new test pins the glossed case.

Files added:
  language_packs/data/<pack>/glosses.jsonl  (9 files, 323 entries)
  tests/test_pack_glosses_content.py        (9 contract tests)

Files modified:
  language_packs/data/<pack>/manifest.json  (9 files, glosses_checksum field)
  chat/pack_grounding.py                    (lowercase "pack-grounded" tag)
  tests/test_pack_grounding.py              (3 substring assertions relaxed)
  tests/test_pack_resolver_glosses.py       (back-compat test pivoted)

Verification:
  127/127 affected tests green.
  9/9 new gloss-content tests green.
  All three eval lanes report the lift documented above.
  Cognition eval byte-identical.
2026-05-19 07:34:33 -07:00
Shay
46ac737767 feat(pack-grounding): selector-ready gloss wiring via PackSurfaceCandidate
Wires the gloss resolver (Phase B2) through pack_grounded_surface
WITHOUT hard-coding around the future SurfaceSelector.  Per the
2026-05-19 design review:

  > don't let glosses hard-code around the selector.  If they ship
  > first, keep the integration deliberately narrow:
  > pack_grounded_surface() may use glosses temporarily, but the
  > data model should already look like future SurfaceCandidate
  > input.  That avoids a second migration.

The integration uses a typed intermediate dataclass that matches the
selector's expected candidate shape:

  chat/pack_surface_candidate.py
    @dataclass(frozen=True, slots=True)
    class PackSurfaceCandidate:
        surface: str               # rendered final string
        grounding_source: str      # "pack" today
        pack_id: str               # provenance
        gloss: str | None          # reviewed natural-language form
        semantic_domains: tuple    # audit-trail content
        lemma: str
        pos: str
        is_user_facing_safe: bool  # honesty flag for selector
        is_fluent_sentence: bool   # gloss-backed vs. dotted-disclosure

When the SurfaceSelector lands:
  - This type becomes one variant in the selector's typed candidate
    union (alongside RefusalCandidate, TeachingCandidate, OOVCandidate).
  - pack_grounded_surface() becomes a provider that emits the
    candidate; the selector picks across providers' candidates by
    ranked authority + is_fluent_sentence preference.
  - No data migration — only the rendering step relocates.

Surface composition (chat/pack_grounding.py):

  build_pack_surface_candidate(lemma) -> PackSurfaceCandidate
    1. resolve_lemma(lemma) — required (None when OOV).
    2. resolve_gloss(lemma) — when present AND same pack as lexicon,
       compose POS-framed fluent sentence:
         "Truth is a claim or state grounded by evidence and
          coherent judgment. Pack-grounded (en_core_cognition_v1)."
       sets is_fluent_sentence=True.
    3. Else fallback to original ADR-0048 dotted-disclosure form:
         "truth — pack-grounded (en_core_cognition_v1):
          cognition.truth; logos.core; epistemic.ground.
          No session evidence yet."
       sets is_fluent_sentence=False.

  pack_grounded_surface(lemma) -> str | None
    Renders the candidate's surface field.  Returns None for OOV.
    Both fluent and disclosure surfaces carry the
    "pack-grounded ({pack_id})" provenance marker so existing
    substring-permissive tests continue to pass through the
    transition.

POS-framed sentence templates (_frame_gloss):
    NOUN  -> "{Lemma} is {gloss}."
    VERB  -> "To {lemma} means {gloss}."
    ADJ   -> "Something is {lemma} when it {gloss}."
    ADV   -> "{Lemma} indicates {gloss}."
    ADP   -> "{Lemma} is a relation of {gloss}."
    SCONJ -> "{Lemma} introduces {gloss}."
    PRON  -> "{Lemma} asks for {gloss}."
    AUX   -> "{Lemma} expresses {gloss}."
    INTJ  -> "{Lemma} is uttered to {gloss}."
    DET   -> "{Lemma} specifies {gloss}."
    NUM   -> "{Lemma} is the cardinal value {gloss}."

Phase C glosses (sitting in /tmp from the 5-subagent parallel
dispatch) are authored to fit these frames exactly.

NO GLOSSES SHIP IN THIS COMMIT.  This is the wiring; the content
arrives in the Phase C commit.  Today every pack still emits the
original dotted-disclosure form because no glosses.jsonl exists yet
on any pack.

Verification:
  97/97 affected tests green (pack grounding, resolver, glosses,
  procedure surface, correction topic, meta pack).
  Cognition eval byte-identical on both splits.
  Live probe with no glosses: surface format identical to pre-fix.
2026-05-19 07:26:46 -07:00
Shay
b52e04a72f fix(intent): five conversational definition patterns + polarity-stopword
The 2026-05-19 cumulative live probe surfaced a stark gap: ~52% of
realistic conversational definition prompts ("Define X", "What does
X mean?", "What is to V?", "How does X work?", "What causes X?")
returned ``grounding_source="none"`` *even though every subject
lemma was pack-resident* across the 9 mounted English packs.

Root cause: the bottleneck was intent classification + subject
extraction, not lexicon coverage.  Five patterns either had no rule
or routed to an intent the runtime dispatcher couldn't handle.  The
fluency assessment at
``/Users/kaizenpro/.codex/worktrees/6533/core/notes/fluency_assessment_2026-05-19.md``
named these as Root Cause #1 ("public chat path does not use the
cognitive spine") and Root Cause #3 ("proposition graphs are too
thin").  This commit closes the surface-level half of that gap;
the deeper answer-plan layer (gloss propositions, P3 in the
assessment) is the next step.

Patterns fixed in ``generate/intent.py``:

  1. ``Define X``        — added ``^define\s+`` rule mapping to
                           DEFINITION (placed after ``^what is/are``
                           so multi-word DEFINITION patterns still
                           prefer the question form).
  2. ``What does X mean?`` — was matching TRANSITIVE_QUERY with
                            relation=``mean``.  Now re-routes to
                            DEFINITION inside ``classify_intent`` so
                            ``pack_grounded_surface`` fires on X.
                            Other transitive relations (precede,
                            ground, etc.) remain TRANSITIVE_QUERY.
  3. ``What is to V?``   — added infinitive-marker strip to
                           ``_normalize_subject`` for DEFINITION /
                           RECALL.  ``to`` is gated on intent tag so
                           it never strips a transfer preposition
                           from CAUSE / VERIFICATION.
  4. ``How does X work?`` — added ``_HOW_DOES_X_RE`` (third-person
                            mechanistic-cause).  Distinct from the
                            first-person PROCEDURE rule ("How do I
                            X?").  Verbs: work / function / operate /
                            happen / exist / behave / act / emerge.
  5. ``What causes X?``   — added causative-verb rule (causes /
                            triggers / enables / prevents / drives /
                            produces / induces / yields) routing to
                            CAUSE with X as subject.

Deliberate NON-fix: I considered adding a ``pack_grounded_surface``
fallback in the CAUSE / VERIFICATION dispatcher when no teaching
chain matches the subject.  Reverted on review — that masks the
"would_have_grounded" discovery-candidate signal the teaching
pipeline uses to identify teaching-content gaps (see
``tests/test_discovery_candidates``).  CAUSE on a pack-resident
lemma without a teaching chain stays ``grounding_source=='none'``
so the discovery layer can log the gap honestly.

``chat/pack_grounding.py``:
  Extended ``_CORRECTION_TOPIC_STOPWORDS`` to include polarity
  markers (no / yes / maybe / perhaps / hardly / indeed / surely /
  definitely).  Without this the CORRECTION composer would
  short-circuit on ``no`` from "No, my parent disagrees" and miss
  the topical lemma ``parent``.

Cumulative probe lift (44 realistic conversational prompts):
  BEFORE: pack=16  none=23  oov=4  teaching=1  (52% NONE)
  AFTER:  pack=37  none=2   oov=4  teaching=1   ( 5% NONE)

  The remaining 2 NONE responses are CAUSE-shaped prompts with no
  teaching chain — deliberately preserved as the discovery-gap
  signal described above.

Tests: tests/test_intent_classification_extensions.py — 23 new
tests covering each pattern + the lift invariant.

Verification:
  Cognition eval byte-identical on both splits (100/100/91.7/100
  public, 100/100/83.3/100 holdout).
  All 111 intent-affected tests green:
    test_intent_classification_extensions.py (23)
    test_intent_proposition_graph.py / test_intent_ratifier.py /
    test_intent_subject_extraction.py / test_narrative_example_intents.py
    test_procedure_surface.py
    test_correction_topic_lemma.py
    test_cross_pack_grounding.py (including the polarity-stopword fix)
    test_discovery_candidates.py
    test_contemplation_wiring.py
    test_en_core_polarity_v1_pack.py
2026-05-19 06:12:05 -07:00
Shay
9f83b27a7c feat(adr-0063): cross-pack surface resolver — kinship lemmas ground on live path
ADR-0063 closes the ADR-0048/0050/0053/0061 hardcoded-cognition-pack
asymmetry. New chat/pack_resolver.py provides resolve_lemma(lemma,
pack_ids) → (resolving_pack_id, semantic_domains) across an ordered
tuple of mounted lexicon packs (first-match-wins, lru_cache per-pack).

Surface composers in chat/pack_grounding.py now consult the resolver
instead of a hardcoded en_core_cognition_v1. en_core_relations_v1
joins RuntimeConfig.input_packs defaults; kinship lemmas now ground
on the live path:

  > What is a parent?
  parent — pack-grounded (en_core_relations_v1):
  kinship.ascendant.direct; kinship.parent; biology.progenitor.
  No session evidence yet.

Cross-pack comparison (knowledge × parent) renders composite tag
(en_core_cognition_v1 × en_core_relations_v1). Cognition lane
remains byte-identical: cognition is resolved first and the surface
format for cognition lemmas is unchanged.

Cognition eval (byte-identical to pre-ADR baseline):
  public  → intent 100% / surface 100% / term 91.7% / closure 100%
  holdout → intent 100% / surface 100% / term 83.3% / closure 100%

Curated lanes green: smoke 67 / cognition 121 / teaching 17 /
packs 6 / runtime 19 / algebra 132.

New tests: test_pack_resolver.py (28) + test_cross_pack_grounding.py
(17). test_en_core_relations_v1_pack.py: default-input-packs guard
inverted. test_pack_grounding.py: two stale ADR-0048 tests rewritten
(premises invalidated by ADR-0052/0061; now use fully-out-of-pack
prompts).

chat/teaching_grounding.py UNCHANGED — cognition_chains_v1 corpus
stays cognition-only. Cross-pack teaching corpora are the natural
ADR-0064.
2026-05-18 15:00:58 -07:00
Shay
bf7f7895fe feat(adr-0061): PROCEDURE intent routes to pack-grounded surface
Pre-ADR-0061 every "How do I X?" question fell through to the
universal disclosure even when X was a pack-resident lemma.  The
teaching corpus carries CAUSE/VERIFICATION chains only — procedural
knowledge is fundamentally different in kind from propositional
claims and deserves its own ratification path (deliberately out of
scope; a future parallel `procedure_chains_v1.jsonl` schema is
discussed in the ADR's non-goals).

ADR-0061 adds the honest cold-start fallback: ground the topic in
pack semantic_domains and note explicitly that ratified step-by-step
guidance does not exist yet.

Surface format:
  "procedure-grounded ({pack_id}): {lemma} ({d1}; {d2}).
   Step-by-step guidance for {lemma} is not yet ratified
   in this session."

Selector — **last** pack-resident lemma in the verb-phrase subject:
  "define a concept" → concept    (object beats verb)
  "verify a claim"   → verify     (verb wins when object is OOV)
  "correct an error" → correct
  "learn this"       → learn
  "do stuff"         → None       (falls through to universal disclosure)

Stopwords: only `be` and `have` (dialogue fillers).  Procedure verbs
are deliberately NOT stopworded so the verb-as-fallback rule fires
when the object is OOV — keeps surface coverage.

Trust-boundary invariants:
  - Every visible non-template token is lemma / pack-domain / template.
  - Deterministic: same subject_text → same bytes.
  - Returns None for fully-unknown utterances → universal disclosure
    fires.  Never fabricates surface from nothing (ADR-0053 contract).
  - "not yet ratified" trust-label preserved.

Cognition lane lift:
  public  : intent 100% / surface 100% / term 91.7% / versor 100%      (unchanged)
  holdout : intent 100% / surface 94.7%→100.0% / term 79.2%→83.3% / versor 100%

Two cases fixed:
  - procedure_define_010 ("How do I define a concept?") — surface +
    term `concept` now captured.
  - procedure_verify_034 ("How do I verify a claim?") — surface only
    (case has no expected_terms; the verb fallback grounds it).

Combined effect: holdout `surface_groundedness` closes to 100%; 4 of
5 architectural holdout misses now resolved (this ADR + ADR-0060 +
the supersede from epistemology v1).  Remaining 2 are UNKNOWN-intent
cases (unknown_spirit_041, unknown_word_018) — out of scope; deserve
their own ADR with distinct selector semantics.

- chat/pack_grounding.py — `_extract_procedure_topic_lemma` helper +
  `pack_grounded_procedure_surface` composer.
- chat/runtime.py — import + dispatch branch for `IntentTag.PROCEDURE`.
- tests/test_procedure_surface.py — 15 tests pin: extraction
  (last-wins / verb-by-elimination / be+have skipped / None on empty /
  strips punctuation / case-insensitive); surface (contains lemma /
  contains domains / pack_id / "not yet ratified" label / None for
  no-pack-lemma / deterministic); end-to-end through ChatRuntime.

Lanes (regression): smoke 67 / cognition 121 / teaching 17 /
procedure 15 — all green.

The non-negotiable field invariant (versor_condition < 1e-6) is
unaffected: this ADR changes surface composition only.
2026-05-18 14:22:19 -07:00
Shay
c9e858c266 feat(adr-0060): correction acknowledgement carries corrected-topic lemma
ADR-0053's cold-start CORRECTION surface was topic-blind: a user who
said "Actually, truth requires evidence" got a response referencing
`correction` but never `truth`.  The holdout case correction_truth_040
expected `term=['truth']` and missed — one of the architectural gaps
surfaced by the epistemology v1 curriculum unit.

ADR-0060 closes that gap by weaving the first pack-resident topical
lemma from the utterance into a fixed-template extension:

  correction received — pack-grounded ({pack_id}):
  {correction_domains}. Noted topic: {lemma} ({lemma_domains}).
  No prior turn in this session to correct yet.

Selection rule (deterministic, left-to-right token order):
  - skip stopwords: `correction`, `correct`, `be`, `have`
  - pick first pack-resident lemma
  - if none found → ADR-0053 topic-less template byte-identically

Trust-boundary invariants preserved:
  - Every visible non-template token is still lemma / pack-domain / template
  - Deterministic: same text → same bytes
  - Backward compatible: existing 15 ADR-0053 tests pass byte-identically
  - "No prior turn in this session to correct yet." trust label kept

Cognition lane lift:
  public  : intent 100% / surface 100% / term 91.7% / versor 100%   (unchanged)
  holdout : intent 100% / surface 94.7% / term 75.0%→79.2% / versor 100%

The +4.2pp matches the single-case fix exactly (correction_truth_040).
Remaining 3 holdout misses (procedure_define_010, unknown_spirit_041,
unknown_word_018) are out of scope for this ADR.

- chat/pack_grounding.py — `_extract_correction_topic_lemma` helper +
  optional `text` parameter on `pack_grounded_correction_surface`.
- chat/runtime.py — single-line call-site change to pass `text` through.
- tests/test_correction_topic_lemma.py — 14 new tests pin:
  extraction (first lemma / skips correction / skips fillers / None on
  empty / strips punctuation / case-insensitive); surface (contains
  corrected lemma / contains topic domains / degrades to ADR-0053
  byte-identically / preserves trust label / deterministic / correct
  pack_id); end-to-end (correction_truth_040 emits 'truth' / no-pack-
  lemma still grounds).

Why text-level extraction, not intent.subject:
  `intent.subject` after ADR-0049 head-noun extraction returns
  ", truth requires evidence" for the test prompt — the CORRECTION
  intent's subject-extractor preserves the post-marker tail.  Parsing
  the raw text at the surface layer is cleaner; isolates the fix;
  doesn't perturb upstream classification logic.

Lanes (regression): smoke 67 / cognition 121 / teaching 17 /
correction tests 29 (15 ADR-0053 backward-compat + 14 ADR-0060 new) —
all green.

The non-negotiable field invariant (versor_condition < 1e-6) is
unaffected: this ADR changes surface composition only.
2026-05-18 14:14:27 -07:00
Shay
e975faf8a8 feat(adr-0053): cognition lane closure — corpus expansion + CORRECTION acknowledgement
Closes both cognition splits at 100% surface_groundedness.  Three
parts:

1. Teaching corpus expansion (no code).  cognition_chains_v1.jsonl
   grows 3→10 chains.  3 close dev-split misses (correction,
   creation, light-as-VERIFICATION); 4 pre-empt the analogous
   holdout pattern (CAUSE/VERIFICATION on truth + wisdom).  Every
   subject/object is a pack lemma; every connective is a recognised
   humanize_predicate predicate.

2. CORRECTION acknowledgement branch.  New
   `pack_grounded_correction_surface()` in chat/pack_grounding.py,
   wired into `_maybe_pack_grounded_surface` for cold-start
   CORRECTION intents.  Fixed-template surface with distinct
   trailing disclosure ("No prior turn in this session to correct
   yet.") — distinguishes the cold-start acknowledgement from the
   DEFINITION-of-correction surface.  The post-correction reviewed-
   teaching path in teaching/correction.py is unchanged.

3. Diagnostic memory.  Saves the dev-split generalization finding:
   the ADR-0048→0052 chain is NOT overfit.  Public/dev gap was
   teaching-corpus content coverage, not architecture.

Eval deltas (both splits run, post-ADR-0053):
                       public   dev
  intent_accuracy        100%   100%   (=)
  surface_groundedness   100%   100%   SATURATED
  term_capture_rate    91.7%  78.6%
  versor_closure_rate    100%   100%   (=)

Public surface_groundedness: 92.3% → 100%   (+7.7 pp)
Dev    surface_groundedness: 69.2% → 100%   (+30.8 pp)

Tests: tests/test_pack_grounded_correction.py (15 new tests).
Lanes green: smoke (67), cognition (121), runtime (19),
teaching (17), packs (6).

Scope limits: holdouts (19 cases) not yet in the official
`core eval cognition` runner (--split accepts only {dev, public});
the CORRECTION surface does not yet echo the corrected-subject
lemma (relevant only for holdout case `correction_truth_040`).
2026-05-18 07:43:39 -07:00
Shay
ecd580479a feat(adr-0050): pack-grounded COMPARISON surface
Sibling to ADR-0048's DEFINITION/RECALL pack-grounded surface for
the COMPARISON intent.  `pack_grounded_comparison_surface(a, b)` in
`chat/pack_grounding.py` composes a deterministic side-by-side
surface from both lemmas' pack `semantic_domains`, joined by the
fixed connective "contrasts with":

  "{a} (d_a1; d_a2) contrasts with {b} (d_b1; d_b2) — pack-grounded
   ({pack_id}). No session evidence yet."

`chat/runtime.py:_maybe_pack_grounded_surface` gains a COMPARISON
branch that runs before the DEFINITION/RECALL check.  Engages only
when both `intent.subject` and `intent.secondary_subject` are pack
lemmas and differ (identical-lemma comparison defers to disclosure).
Order-sensitive by design — matches the graph-layer's directional
CONTRAST edge.

Cognition eval (13-case public split):
  surface_groundedness  61.5% → 69.2%  (+7.7 pp)
  term_capture_rate     50.0% → 58.3%  (+8.3 pp)
  intent_accuracy            100.0%        (=)
  versor_closure_rate        100.0%        (=)

Case lifted: comparison_memory_recall_030 ("Compare memory and
recall").  Remaining unlift cases (CAUSE×2, VERIFICATION×1,
CORRECTION×1) need teaching-store chains or operator-driven
inference — pack lookup cannot supply causal explanations,
verifications, or corrections without fabrication.

Tests: tests/test_pack_grounded_comparison.py (15 tests).
Lanes green: smoke (67), cognition (121), runtime (19), algebra
(132), teaching (17), packs (6).
2026-05-18 06:59:53 -07:00
Shay
c28e107dc7 feat(adr-0048): pack-grounded surface for cold-start DEFINITION/RECALL
Closes the surface-grounding gap isolated by ADR-0047's
characterisation.  Adds the ratified cognition pack as a second
grounding source alongside the session vault.

== chat/pack_grounding.py (new) ==

Loads en_core_cognition_v1's lexicon once (cached; immutable pack)
and exposes:

  pack_grounded_surface(lemma) -> str | None

Returns a deterministic, fully pack-sourced surface:

  "{lemma} — pack-grounded ({pack_id}): {d1}; {d2}; {d3}. No session evidence yet."

Every visible atom is the lemma or a verbatim semantic_domains
string from the pack.  No rewording, no synthesis, no LLM.

== chat/runtime.py ==

_stub_response gains optional pack_grounded_surface= parameter.
_maybe_pack_grounded_surface routes to the pack only when all four
hold: gate_source=="empty_vault", output_language=="en",
intent.tag in {DEFINITION, RECALL}, and intent.subject is a pack
lemma.  Safety/ethics refusal still takes priority above this branch.

ChatResponse and TurnEvent gain grounding_source ∈ {vault,pack,none}.
Main walk path tags responses "vault".

== core/cognition/pipeline.py ==

gate_fired detection moved from string equality on the universal
disclosure to provenance:

  gate_fired = response.vault_hits == 0 and response.grounding_source != "vault"

Same intent (suppress realizer template on gate-fired turns),
broader stub-path surface set.

== Characterisation (core eval cognition, 13-case public split) ==

  Metric                  Pre        Post     Δ
  intent_accuracy        100.0%     100.0%    0
  surface_groundedness    15.4%      46.2%   +30.8 pp
  term_capture_rate        0.0%      33.3%   +33.3 pp
  versor_closure_rate    100.0%     100.0%    0

Lift is non-uniform by design: only single-lemma DEFINITION/RECALL
on pack-known English subjects engage.  CAUSE/COMPARISON/VERIFICATION
and multi-word OOV subjects still return the universal disclosure —
fabricating those would violate the no-LLM-fallback doctrine.

== Tests ==

  tests/test_pack_grounding.py                          18 passed
  tests/test_semantic_realizer_integration.py (updated) 1 stub-path test
    pinned to the broader contract: surface is either universal
    disclosure or pack-grounded; never the realizer template.

== Lanes ==

  smoke 67  cognition 121  runtime 19  algebra 132
  teaching 17  packs 6

versor_condition(F) < 1e-6 invariant unaffected (no algebra changes).
2026-05-18 06:36:10 -07:00