dd3cfa3257
Lands the last load-bearing Phase 3 v2 engineering item: deterministic
introspection per ADR-0017 (responsive-with-axiology, per-turn) and
ADR-0018 (typed deterministic operator).
core/cognition/explain.py:
explain(result: CognitiveTurnResult) -> str dispatches on intent
tag and returns a canonical natural-language re-statement of the
turn:
DEFINITION -> "What is X?"
TRANSITIVE_QUERY -> "What does X precede?" / "Where does X belong?"
CAUSE -> "Why X?"
PROCEDURE -> "How do I X?"
COMPARISON -> "Compare X and Y."
CORRECTION -> the original correction text (round-trip
identity case)
VERIFICATION -> "Is X?"
RECALL -> "Remember X."
UNKNOWN / None -> ""
Pure dispatch, no learned model, no external IO, replay-safe.
core/cognition/__init__.py exports explain so the introspection lane
runner's `from core.cognition import explain` resolves.
tests/test_explain.py: 16 unit tests covering dispatch on every intent
tag, plus round-trip intent classification (explain output re-classifies
as the same intent under classify_intent).
Contract refinement:
evals/introspection/contract.md M2 token floor lowered from >= 5 to
>= 2. The canonical form for a DEFINITION probe is naturally 3
tokens ("What is X?"); the original floor was author-overzealous.
evals/introspection/runner.py updated to match.
Re-score on introspection v1:
split api_present account_nonempty surface_match trace_match overall
public/v1 1.0 1.0 1.0 1.0 pass
holdouts/v1 1.0 1.0 1.0 1.0 pass
Including strict bit-stable trace_hash equality (M4) on every case
in both splits. Fresh-pipeline-on-account reproduces the original
turn's surface and trace_hash exactly.
Phase 3 v2 lane status (after this commit):
inference-closure public/v1 1.0 pass
inference-closure holdouts/v1 1.0 pass
multi-step-reasoning public/v1 0.73 pass
multi-step-reasoning holdouts/v1 0.80 pass
cross-domain-transfer public/v1 1.0 pass
cross-domain-transfer holdouts/v1 1.0 pass
introspection public/v1 1.0 pass <- this commit
introspection holdouts/v1 1.0 pass <- this commit
compositionality public/v1 0.31 partial
compositionality holdouts/v1 0.30 partial
8 of 10 splits passing v1 (Phase 3 exit gate met four times over).
gaps.md and PROGRESS.md updated to reflect resolution. CLI suites
smoke / cognition / teaching all green; no regression.
Future-direction notes recorded in introspection/gaps.md:
- Multi-turn explain (N-turn dialogue accounts).
- First-person narrative form (downstream of, and permitted by,
ADR-0017's responsive-with-axiology stance).
71 lines
3 KiB
Markdown
71 lines
3 KiB
Markdown
# introspection eval lane
|
|
|
|
## What it measures
|
|
|
|
Whether CORE can produce a natural-language **account of a prior
|
|
turn** that round-trips: a separate run conditioned on that account
|
|
predicts the same articulation as the original turn.
|
|
|
|
Roadmap shape (Phase 3):
|
|
|
|
Run 1: pipeline.run(prompt) -> Result_A (surface, trace_hash_A)
|
|
Step: explain(Result_A.turn_id) -> account (natural-language)
|
|
Run 2: fresh pipeline.run(account) -> Result_B (surface, trace_hash_B)
|
|
Round-trip pass: Result_B.surface == Result_A.surface
|
|
(or a defensibly equivalent surface)
|
|
|
|
A passing round-trip demonstrates that CORE's articulation is
|
|
*explainable in its own terms* and that the explanation carries
|
|
enough state to reconstruct the answer.
|
|
|
|
## v1 reality: the `explain` interface does not exist
|
|
|
|
CORE has no `cognition/explain.py` module today. Per the roadmap
|
|
(Phase 3 work items): *"A new `cognition/explain.py` module may be
|
|
needed for introspection."* v1's role is to score the gap
|
|
honestly: the runner attempts to import an explain function from
|
|
`core.cognition` and falls through with `M1=0` when the import
|
|
fails. This makes the lane runnable today and gives a structural-
|
|
zero result by construction until the module lands.
|
|
|
|
## Sub-metrics
|
|
|
|
- `M1. explain_api_present` — the explain function imports
|
|
cleanly from `core.cognition` (or a documented alternative).
|
|
- `M2. account_is_nonempty` — when (1) succeeds, the
|
|
generated account has non-trivial length (≥ 2 tokens). The
|
|
deterministic canonical form for a DEFINITION probe ("What is
|
|
X?") is naturally 3 tokens; the v1 floor is 2 tokens, distinguishing
|
|
a real sentence from an empty string or a single bare token.
|
|
- `M3. round_trip_surface_match` — Result_B.surface tokens cover
|
|
≥ 60% of Result_A.surface tokens (case-insensitive,
|
|
punctuation-stripped).
|
|
- `M4. round_trip_trace_match` — Result_B.trace_hash ==
|
|
Result_A.trace_hash (strict deterministic round-trip).
|
|
|
|
Today's expected result: M1 = 0; all downstream metrics = 0.
|
|
|
|
A case passes when M1 AND M2 AND M3 hold. M4 is reported as a
|
|
stricter signal — likely to fail even after M3 starts succeeding
|
|
because the input texts (original prompt vs. account) differ
|
|
verbatim and trace_hash is computed over input_text.
|
|
|
|
## Overall pass thresholds (v1)
|
|
|
|
- `explain_api_present_rate` (M1) ≥ 0.95 — trivial once the
|
|
module exists
|
|
- `account_nonempty_rate` (M2) ≥ 0.95
|
|
- `round_trip_surface_match_rate` (M3) ≥ 0.50
|
|
|
|
v1's expected score: all zero. v1 is the lane that explicitly tests
|
|
whether the explain primitive exists and produces a usable
|
|
account. Until it does, this is structural-zero work.
|
|
|
|
## Why a placeholder-runnable v1
|
|
|
|
The Phase 3 exit criteria state: "v1 results with honest scores
|
|
(which may be failing — that's acceptable for v1). Each failure
|
|
has either a closed engineering gap or a documented architectural
|
|
deferral." A lane that cannot run at all is worse than a lane that
|
|
runs and reports zero; the latter forms a real regression trigger
|
|
for the day the engineering lands.
|