audit(L5): cognition pipeline — PARTIAL (#244)
This commit is contained in:
Shay
GitHub
parent
72cede071f
commit
050b2f9222
2 changed files with 126 additions and 107 deletions
|
|
@ -13,7 +13,7 @@
|
|||
| L2 — Vault | ✅ Audited | **PARTIAL** | None — flagged learning as wiring debt | 2026-05-24 |
|
||||
| L3 — Language packs | ✅ Audited | **PARTIAL** | None — flagged readback as wiring debt | 2026-05-24 |
|
||||
| L4 — Recognition | ✅ Audited | **PARTIAL** | None — flagged connector/storage/integration as wiring debt | 2026-05-24 |
|
||||
| L5 — Cognition pipeline | ⏳ Pending | — | — | — |
|
||||
| L5 — Cognition pipeline | ✅ Audited | **PARTIAL** | `generate/render.py` deleted; `explain.py` / `provenance.py` flagged as wiring debt | 2026-05-24 |
|
||||
| L6 — Chat runtime + surface composition | ⏳ Pending | — | — | — |
|
||||
| L7 — Teaching loop | ⏳ Pending | — | — | — |
|
||||
| L8 — Inter-session memory + contemplation | ⏳ Pending | — | — | — |
|
||||
|
|
@ -884,3 +884,128 @@ awaiting integration/storage ADRs.
|
|||
reactivates one across sessions.
|
||||
|
||||
---
|
||||
|
||||
## L5 — Cognition pipeline
|
||||
|
||||
**Audit date:** 2026-05-24
|
||||
**Auditor:** primary agent (Gemini)
|
||||
**Verdict:** **PARTIAL**
|
||||
|
||||
### Scope-hypothesis correction (per audit step 0)
|
||||
|
||||
The scope's layering table cited `core/cognition/` and `generate/` concerns: intent classification, ratification, articulation target, deterministic realizer, proposition graph planner, cold-start grounding sources. Reality: the concerns are split across two packages: `core/cognition/` (orchestration, result, trace, surface resolution) and `generate/` (intent classification, intent ratification, realizer, templates, graph planner, admissibility, attention, salience, and stream walk). The scope ranges omitted key admissibility and classification ADRs. `generate/render.py` was found to be completely dormant (0 imports, 0 references, 0 tests) and was deleted during cleanup. `core/cognition/explain.py` and `core/cognition/provenance.py` were found to have 0 live production callers, and are flagged as dormant/partially live wiring debt.
|
||||
|
||||
### ADRs in scope for L5
|
||||
|
||||
| ADR | Title | Status | Belongs at L5? |
|
||||
|---|---|---|---|
|
||||
| ADR-0022 | Forward Semantic Control | Accepted | Yes — `classify_intent` and `ratify_intent` at the pipeline step |
|
||||
| ADR-0023 | Forward Semantic Control Proof | Accepted | Yes — verifies ratification execution and rates |
|
||||
| ADR-0024 | Inner-Loop Per-Rotor Admissibility | Accepted | Yes — defines admissibility regions, `check_transition`, `InnerLoopExhaustion` |
|
||||
| ADR-0025 | Rotor / Frame Admissibility | Accepted | Yes — rotor effect on current field against frame cone |
|
||||
| ADR-0026 | Ranked Admissibility with Margin | Accepted | Yes — margin-mode checks |
|
||||
| ADR-0046 | PropositionGraph as Forward Admissibility Constraint | Accepted | Yes — proposition graph planning constraints |
|
||||
| ADR-0047 | Wire the Forward Graph Constraint into the Chat Hot Path | Accepted | Yes — wires forward constraint |
|
||||
| ADR-0048 | Pack-Grounded Surface for Cold-Start DEFINITION / RECALL | Accepted | Yes |
|
||||
| ADR-0049 | Intent Classifier Head-Noun Subject Extraction | Accepted | Yes |
|
||||
| ADR-0050 | Pack-Grounded Surface for Cold-Start COMPARISON | Accepted | Yes |
|
||||
| ADR-0051 | Trust-Boundary Hardening Pass | Accepted | Yes |
|
||||
| ADR-0052 | Teaching-Grounded Surface for Cold-Start CAUSE / VERIFICATION | Accepted | Yes |
|
||||
| ADR-0053 | Cognition Lane Closure | Accepted | Yes |
|
||||
| ADR-0058 | `forward_graph_constraint`: Engaged but Inert | Accepted | Yes |
|
||||
| ADR-0061 | PROCEDURE Intent Routes to Pack-Grounded Surface | Accepted | Yes |
|
||||
| ADR-0069 | Realizer register parameter | Accepted | Yes |
|
||||
| ADR-0075 | Realizer slot-type guard | Accepted | Yes |
|
||||
| ADR-0142 | Epistemic State Taxonomy | Accepted | Boundary-only — L5/L9 boundary |
|
||||
| ADR-0143 | Recognition Spike | Accepted | Boundary-only — L4 output is L5 input |
|
||||
| ADR-0144 | PropositionGraph: Epistemic Carrier & Gate | Accepted | Yes — carrier integration with L4/L9 |
|
||||
|
||||
### Modules in scope for L5
|
||||
|
||||
| Module | Lines | Live-import sites (outside own package, outside `tests/`) | Test-import sites | Status |
|
||||
|---|---|---|---|---|
|
||||
| `core/cognition/pipeline.py` | 727 | 3 (`chat/runtime.py`, `core/cli.py`, `evals/run_cognition_eval.py`) | 12 | Live |
|
||||
| `core/cognition/result.py` | 139 | 3 | 11 | Live |
|
||||
| `core/cognition/trace.py` | 153 | 3 | 11 | Live |
|
||||
| `core/cognition/surface_resolution.py` | 108 | 1 (`pipeline.py`) | 3 | Live |
|
||||
| `core/cognition/explain.py` | 124 | 0 (only re-exported in `core/cognition/__init__.py`) | 1 | Dormant (no live production callers) |
|
||||
| `core/cognition/provenance.py` | 101 | 0 (used by `evals/provenance/runner.py`) | 1 | Partially live (evals/verification only) |
|
||||
| `generate/intent.py` | 662 | 8 | 15 | Live |
|
||||
| `generate/intent_bridge.py` | 350 | 2 | 2 | Live |
|
||||
| `generate/intent_ratifier.py` | 241 | 1 | 4 | Live |
|
||||
| `generate/realizer.py` | 272 | 3 | 7 | Live |
|
||||
| `generate/templates.py` | 232 | 3 | 3 | Live |
|
||||
| `generate/graph_planner.py` | 300 | 4 | 7 | Live |
|
||||
| `generate/admissibility.py` | 668 | 2 | 11 | Live |
|
||||
| `generate/graph_constraint.py` | 159 | 2 | 2 | Live |
|
||||
| `generate/stream.py` | 648 | 2 | 12 | Live |
|
||||
| `generate/proposition.py` | 417 | 1 | 2 | Live |
|
||||
| `generate/salience.py` | 62 | 2 | 2 | Live |
|
||||
| `generate/attention.py` | 43 | 1 | 1 | Live |
|
||||
| `generate/bridge_trace.py` | 292 | 1 | 1 | Live |
|
||||
| `generate/render.py` | — | — | — | **DELETED — 0 imports, 0 references, 0 tests** |
|
||||
|
||||
### Caller-trace evidence
|
||||
|
||||
The live chat/cognition trace is:
|
||||
`ChatRuntime.chat()` → `CognitiveTurnPipeline.run()` → `_ratify_intent()` → `graph_from_intent()` → `plan_articulation()` → `realize_semantic()` → delegated walk → `generate()` → `resolve_surface()` → `compute_trace_hash()`.
|
||||
|
||||
### Exercising suite lane
|
||||
|
||||
- `core test --suite cognition` — **120 passed, 1 skipped**.
|
||||
- `core test --suite smoke` — **67 passed**.
|
||||
- `core eval cognition` — **13/13 passed** (100% intent_accuracy, surface_groundedness, versor_closure_rate).
|
||||
- `scripts/verify_lane_shas.py` — **7/7 lanes match pinned SHAs**.
|
||||
|
||||
### Cross-layer contract check
|
||||
|
||||
**Pass 1 — mechanical (consumer-exists per exposed symbol):**
|
||||
|
||||
| Exposed symbol | Consumer evidence |
|
||||
|---|---|
|
||||
| `CognitiveTurnPipeline` | `chat/runtime.py:27`, `core/cli.py:446`, `evals/run_cognition_eval.py` |
|
||||
| `CognitiveTurnResult` | `chat/runtime.py:27`, `core/cognition/pipeline.py`, tests |
|
||||
| `resolve_surface` | `core/cognition/pipeline.py:23` |
|
||||
| `compute_trace_hash` | `core/cognition/pipeline.py:24` |
|
||||
| `classify_compound_intent` | `core/cognition/pipeline.py:25` |
|
||||
| `ratify_intent` | `core/cognition/pipeline.py:30` |
|
||||
| `graph_from_intent` | `core/cognition/pipeline.py:34` |
|
||||
| `realize_semantic` | `core/cognition/pipeline.py:41` |
|
||||
| `generate` | `chat/runtime.py:88` |
|
||||
| `build_graph_constraint` | `chat/runtime.py:88` |
|
||||
|
||||
**Pass 2 — semantic (four invariants checked):**
|
||||
|
||||
1. **Surface contract.** Verified `resolve_surface` in `core/cognition/surface_resolution.py` preserves the user-facing `surface` vs raw walk telemetry `walk_surface` split. User-facing `surface` is the semantic realizer output folded with deterministic walk/compose suffixes; raw manifold-walk telemetry goes strictly to `walk_surface`.
|
||||
2. **Normalization-site discipline.** `generate/stream.py` is a FORBIDDEN normalization site except for the documented final-state closure: `_close_final_state(state)` calls `unitize_versor(state.F)` when sealing the final state at the generation boundary. No other normalization has crept into `generate/`. `generate/admissibility.py` explicitly documents that its composed blades are *not* unitized to honor the CLAUDE.md discipline.
|
||||
3. **`refusal_reason` materialization.** Wiring exists: `ChatResponse` has the field, `CognitiveTurnPipeline` reads it, and it is folded into `trace_hash` when non-empty. **However:** `InnerLoopExhaustion` exceptions raised during generation are NOT caught in `ChatRuntime.chat()`, so `refusal_reason` is never populated with the machine-readable `RefusalReason` taxonomy during live runs (it propagates as an unhandled exception).
|
||||
4. **RatificationOutcome PASSTHROUGH split.** Confirmed sub-values (`PASSTHROUGH_NO_FIELD`, `PASSTHROUGH_NO_VOCAB`, `PASSTHROUGH_NO_VERSOR`) collapse to `"passthrough"` before trace hashing in `core/cognition/pipeline.py`. Byte-identity of existing trace hashes preserved exactly.
|
||||
|
||||
### Semantic mismatches flagged for human review
|
||||
|
||||
- **`InnerLoopExhaustion` propagation gap.** Inner-loop refusal exceptions are never caught in main `ChatRuntime.chat()` execution, meaning refusal reasons for admissibility exhaustions are never materialized to `ChatResponse.refusal_reason` during live execution. They cause unhandled exceptions unless caught externally.
|
||||
- **`explain.py` and `provenance.py` dormancy.** `core/cognition/explain.py` has 0 live production callers outside its test file. `core/cognition/provenance.py` is only used in tests and `evals/provenance/runner.py`. Review whether these should be fully wired to the live REPL or are reserved for offline audit tools.
|
||||
|
||||
### Closure criteria scorecard
|
||||
|
||||
| Criterion | Status | Evidence |
|
||||
|---|---|---|
|
||||
| 1. Design artifact | ✅ | ADR-0022..0026, ADR-0046..0053, ADR-0142..0144 |
|
||||
| 2. Code artifact | ✅ | `core/cognition/`, `generate/` (after deleting unused `render.py`) |
|
||||
| 3. Live caller | ⚠️ PARTIAL | Orchestration pipeline, ratification, realizer, stream walk live; `explain.py` dormant; inner-loop refusal exceptions unwired |
|
||||
| 4. Exercised by suite lane | ✅ | `cognition` + `smoke` pass; `eval cognition` 100% |
|
||||
| 5. Cross-layer consistency | ⚠️ PARTIAL | Hashing + surface contracts consistent; `refusal_reason` propagation from generator unwired |
|
||||
|
||||
**Verdict:** **PARTIAL** (dormant `explain.py`, partial `provenance.py`, unwired `InnerLoopExhaustion` exception materialization).
|
||||
|
||||
### Cleanup performed
|
||||
|
||||
- **`generate/render.py` deleted** — completely unused module (0 production imports, 0 test imports, 0 references). Tests + SHAs confirmed intact post-deletion. First cleanup-as-found of the audit per [[feedback-cleanup-as-you-find]].
|
||||
|
||||
### Findings / notes for downstream layers
|
||||
|
||||
- **L6 (Chat runtime) auditor:** Wire the `InnerLoopExhaustion` exceptions caught in the turn loop to populate the `ChatResponse.refusal_reason` field so machine-readable refusal codes are folded into the trace hash and result objects.
|
||||
- **L7 (Teaching loop) / L8 (Memory) auditor:** `core/cognition/explain.py` is dormant. If teaching correction review requires a natural-language explanation of what was decoded on a turn, wire `explain()` to the REPL or CLI proposal commands.
|
||||
- **L9 (Epistemic state) auditor:** ADR-0142 implementation debt #3 (wiring refusal reason) is the primary blocker for full epistemic refusal tracking. Fix this boundary in L6/L9.
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -1,106 +0,0 @@
|
|||
"""
|
||||
generate/render.py — Default TextRenderer (ADR-0011)
|
||||
|
||||
The renderer is the last thing that happens before output leaves the system.
|
||||
Nothing after the renderer touches the field.
|
||||
|
||||
Contract (Renderer protocol):
|
||||
Input: Iterable[str] — the ordered surface-string token stream from stream.py
|
||||
Output: str — the realized surface text
|
||||
Stateless: — holds no field state, modifies nothing
|
||||
Deterministic: — identical token sequences produce identical output
|
||||
|
||||
Modality-specific renderers (markdown, Hebrew RTL, Koine Greek polytonic,
|
||||
audio phoneme stream) implement the same Renderer protocol and are provided
|
||||
by the caller. The engine never selects a renderer.
|
||||
|
||||
Design note:
|
||||
stream.py yields word strings — the .surface form resolved by vocab.nearest().
|
||||
The renderer's only job is joining them correctly for the target modality.
|
||||
Linguistic depth (Hebrew morphology, Greek diacritics) is already encoded
|
||||
in the surface strings upstream in vocab/. The renderer transcribes; it
|
||||
does not interpret.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Iterable, Protocol, runtime_checkable
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class Renderer(Protocol):
|
||||
"""
|
||||
Protocol for all surface renderers.
|
||||
|
||||
The engine yields tokens; the caller provides a Renderer and calls render().
|
||||
The engine has no concept of deployment context.
|
||||
"""
|
||||
|
||||
def render(self, tokens: Iterable[str]) -> str | bytes:
|
||||
"""Realize a token stream as surface output."""
|
||||
...
|
||||
|
||||
|
||||
class TextRenderer:
|
||||
"""
|
||||
Default renderer: joins tokens into a UTF-8 string.
|
||||
|
||||
Separator logic:
|
||||
- Tokens that begin with standard punctuation (.,;:!?'\')
|
||||
are attached to the preceding token without a space.
|
||||
- All other tokens are separated by a single space.
|
||||
- Leading and trailing whitespace is stripped.
|
||||
|
||||
This handles English, Koine Greek polytonic, and Hebrew surface strings
|
||||
correctly provided the VocabEntry.surface fields are well-formed upstream.
|
||||
Hebrew RTL directionality is a property of the Unicode codepoints themselves
|
||||
and is respected automatically by any compliant display layer.
|
||||
"""
|
||||
|
||||
# Punctuation tokens that attach to the left (no preceding space)
|
||||
_ATTACH_LEFT: frozenset[str] = frozenset(".,;:!?'\")-]}’”;·")
|
||||
|
||||
def render(self, tokens: Iterable[str]) -> str:
|
||||
parts: list[str] = []
|
||||
for token in tokens:
|
||||
if not token:
|
||||
continue
|
||||
if parts and token[0] not in self._ATTACH_LEFT:
|
||||
parts.append(" ")
|
||||
parts.append(token)
|
||||
return "".join(parts).strip()
|
||||
|
||||
|
||||
class MarkdownRenderer:
|
||||
"""
|
||||
Renderer for markdown surface output.
|
||||
|
||||
Identical join logic to TextRenderer. Extends with optional
|
||||
code-fence and heading prefix support if the token stream carries
|
||||
structural markers (tokens that begin with '#' or '```').
|
||||
|
||||
This is a thin wrapper — it does not parse or validate markdown.
|
||||
It writes what the field produced.
|
||||
"""
|
||||
|
||||
_text = TextRenderer()
|
||||
|
||||
def render(self, tokens: Iterable[str]) -> str:
|
||||
return self._text.render(tokens)
|
||||
|
||||
|
||||
# Module-level default — usable directly without instantiation
|
||||
default_renderer = TextRenderer()
|
||||
|
||||
|
||||
def render(tokens: Iterable[str], renderer: Renderer | None = None) -> str:
|
||||
"""
|
||||
Convenience function: render a token stream using the provided renderer.
|
||||
Falls back to the default TextRenderer if none is given.
|
||||
|
||||
Usage:
|
||||
from generate.render import render
|
||||
text = render(generate(state, vocab, persona))
|
||||
"""
|
||||
r = renderer if renderer is not None else default_renderer
|
||||
return r.render(tokens)
|
||||
Loading…
Reference in a new issue