819c8b81ac
Spreads the four remaining Phase 3 lanes to map the full reasoning-
depth surface alongside inference-closure (already landed at e509e0d).
Each lane is a v1 honest probe per the roadmap; engineering work
follows once the full surface is visible.
Results across all five Phase 3 lanes:
lane split primary signal foundation
inference-closure public/v1 0.0 1.0 / 1.0
inference-closure holdouts/v1 0.0 1.0 / 1.0
compositionality public/v1 0.0625 (1/16) 1.0 / 1.0
compositionality holdouts/v1 0.0 1.0 / 1.0
multi-step-reasoning public/v1 0.0 1.0 / 1.0
multi-step-reasoning holdouts/v1 0.0 1.0 / 1.0
introspection public/v1 0.0 (no api) n/a
introspection holdouts/v1 0.0 n/a
cross-domain-transfer public/v1 0.0 1.0 / 1.0
cross-domain-transfer holdouts/v1 0.0 1.0 / 1.0
Foundation guarantees (storage + replay) intact across every lane
that has them. The reasoning-depth signal is uniformly zero. The
five lanes triangulate four architectural gaps:
Gap 1. generate/graph_planner.py has no transitive composition.
Gap 2. field/propagate.py has no derivable-but-not-asserted recall.
Gap 3. core/cognition/explain.py module does not exist.
Gap 4. no structural-pattern recogniser (cross-subdomain transfer).
Gaps 1, 2, 4 cluster on the same code surface and may close together
as a single bounded PR. Gap 3 is independent module-creation work.
Lane scaffolding mirrors inference-closure (contract.md, runner.py,
dev + public/v1 + holdouts/v1 cases.jsonl, baselines/v1_structural_zero.json,
gaps.md). All runners are parallel-safe and use the standard
run_lane(cases, *, config, workers) interface.
Per-lane gaps.md records the engineering shape for v2 plus future
directions worth not forgetting:
- compositionality/gaps.md: metaphor is compositionality with
selective property transfer; building it is correctly downstream
of closing this lane.
- cross-domain-transfer/gaps.md: metaphor + narrative as
cross-domain operators; narrative requires the Agency open-scope
decision to pin first.
- introspection/gaps.md: explain API is also the substrate for
first-person narrative self-account.
Recommended v2 sequence in docs/PROGRESS.md:
1. Pin Agency + Tool-use open-scope decisions (deadline: before
Phase 3 engineering).
2. Engineer Gaps 1 + 2 as one bounded PR.
3. Engineer Gap 3 independently.
4. Re-author cross-domain-transfer v2 with matched-control
contract refinement.
Phase 3 v1 exit: 0/5 lanes passing, which is the expected v1 floor.
CLI suites smoke / cognition / teaching pass; no regression on
Phase 2.
68 lines
2.8 KiB
Markdown
68 lines
2.8 KiB
Markdown
# introspection eval lane
|
|
|
|
## What it measures
|
|
|
|
Whether CORE can produce a natural-language **account of a prior
|
|
turn** that round-trips: a separate run conditioned on that account
|
|
predicts the same articulation as the original turn.
|
|
|
|
Roadmap shape (Phase 3):
|
|
|
|
Run 1: pipeline.run(prompt) -> Result_A (surface, trace_hash_A)
|
|
Step: explain(Result_A.turn_id) -> account (natural-language)
|
|
Run 2: fresh pipeline.run(account) -> Result_B (surface, trace_hash_B)
|
|
Round-trip pass: Result_B.surface == Result_A.surface
|
|
(or a defensibly equivalent surface)
|
|
|
|
A passing round-trip demonstrates that CORE's articulation is
|
|
*explainable in its own terms* and that the explanation carries
|
|
enough state to reconstruct the answer.
|
|
|
|
## v1 reality: the `explain` interface does not exist
|
|
|
|
CORE has no `cognition/explain.py` module today. Per the roadmap
|
|
(Phase 3 work items): *"A new `cognition/explain.py` module may be
|
|
needed for introspection."* v1's role is to score the gap
|
|
honestly: the runner attempts to import an explain function from
|
|
`core.cognition` and falls through with `M1=0` when the import
|
|
fails. This makes the lane runnable today and gives a structural-
|
|
zero result by construction until the module lands.
|
|
|
|
## Sub-metrics
|
|
|
|
- `M1. explain_api_present` — the explain function imports
|
|
cleanly from `core.cognition` (or a documented alternative).
|
|
- `M2. account_is_nonempty` — when (1) succeeds, the
|
|
generated account has non-trivial length (≥ 5 tokens).
|
|
- `M3. round_trip_surface_match` — Result_B.surface tokens cover
|
|
≥ 60% of Result_A.surface tokens (case-insensitive,
|
|
punctuation-stripped).
|
|
- `M4. round_trip_trace_match` — Result_B.trace_hash ==
|
|
Result_A.trace_hash (strict deterministic round-trip).
|
|
|
|
Today's expected result: M1 = 0; all downstream metrics = 0.
|
|
|
|
A case passes when M1 AND M2 AND M3 hold. M4 is reported as a
|
|
stricter signal — likely to fail even after M3 starts succeeding
|
|
because the input texts (original prompt vs. account) differ
|
|
verbatim and trace_hash is computed over input_text.
|
|
|
|
## Overall pass thresholds (v1)
|
|
|
|
- `explain_api_present_rate` (M1) ≥ 0.95 — trivial once the
|
|
module exists
|
|
- `account_nonempty_rate` (M2) ≥ 0.95
|
|
- `round_trip_surface_match_rate` (M3) ≥ 0.50
|
|
|
|
v1's expected score: all zero. v1 is the lane that explicitly tests
|
|
whether the explain primitive exists and produces a usable
|
|
account. Until it does, this is structural-zero work.
|
|
|
|
## Why a placeholder-runnable v1
|
|
|
|
The Phase 3 exit criteria state: "v1 results with honest scores
|
|
(which may be failing — that's acceptable for v1). Each failure
|
|
has either a closed engineering gap or a documented architectural
|
|
deferral." A lane that cannot run at all is worse than a lane that
|
|
runs and reports zero; the latter forms a real regression trigger
|
|
for the day the engineering lands.
|