819c8b81ac
Spreads the four remaining Phase 3 lanes to map the full reasoning-
depth surface alongside inference-closure (already landed at e509e0d).
Each lane is a v1 honest probe per the roadmap; engineering work
follows once the full surface is visible.
Results across all five Phase 3 lanes:
lane split primary signal foundation
inference-closure public/v1 0.0 1.0 / 1.0
inference-closure holdouts/v1 0.0 1.0 / 1.0
compositionality public/v1 0.0625 (1/16) 1.0 / 1.0
compositionality holdouts/v1 0.0 1.0 / 1.0
multi-step-reasoning public/v1 0.0 1.0 / 1.0
multi-step-reasoning holdouts/v1 0.0 1.0 / 1.0
introspection public/v1 0.0 (no api) n/a
introspection holdouts/v1 0.0 n/a
cross-domain-transfer public/v1 0.0 1.0 / 1.0
cross-domain-transfer holdouts/v1 0.0 1.0 / 1.0
Foundation guarantees (storage + replay) intact across every lane
that has them. The reasoning-depth signal is uniformly zero. The
five lanes triangulate four architectural gaps:
Gap 1. generate/graph_planner.py has no transitive composition.
Gap 2. field/propagate.py has no derivable-but-not-asserted recall.
Gap 3. core/cognition/explain.py module does not exist.
Gap 4. no structural-pattern recogniser (cross-subdomain transfer).
Gaps 1, 2, 4 cluster on the same code surface and may close together
as a single bounded PR. Gap 3 is independent module-creation work.
Lane scaffolding mirrors inference-closure (contract.md, runner.py,
dev + public/v1 + holdouts/v1 cases.jsonl, baselines/v1_structural_zero.json,
gaps.md). All runners are parallel-safe and use the standard
run_lane(cases, *, config, workers) interface.
Per-lane gaps.md records the engineering shape for v2 plus future
directions worth not forgetting:
- compositionality/gaps.md: metaphor is compositionality with
selective property transfer; building it is correctly downstream
of closing this lane.
- cross-domain-transfer/gaps.md: metaphor + narrative as
cross-domain operators; narrative requires the Agency open-scope
decision to pin first.
- introspection/gaps.md: explain API is also the substrate for
first-person narrative self-account.
Recommended v2 sequence in docs/PROGRESS.md:
1. Pin Agency + Tool-use open-scope decisions (deadline: before
Phase 3 engineering).
2. Engineer Gaps 1 + 2 as one bounded PR.
3. Engineer Gap 3 independently.
4. Re-author cross-domain-transfer v2 with matched-control
contract refinement.
Phase 3 v1 exit: 0/5 lanes passing, which is the expected v1 floor.
CLI suites smoke / cognition / teaching pass; no regression on
Phase 2.
58 lines
2.2 KiB
Markdown
58 lines
2.2 KiB
Markdown
# introspection lane — architectural findings (v1)
|
|
|
|
## v1 result
|
|
|
|
| Split | n | explain_api_present | account_nonempty | surface_match | trace_match |
|
|
|---|---|---|---|---|---|
|
|
| public/v1 | 12 | **0.0** | 0.0 | 0.0 | 0.0 |
|
|
| holdouts/v1 | 8 | **0.0** | 0.0 | 0.0 | 0.0 |
|
|
|
|
Structural zero by construction: there is no `explain` callable to
|
|
import from `core.cognition`.
|
|
|
|
## Why this is the right v1
|
|
|
|
A lane that can't run at all is worse than a lane that runs and
|
|
reports a typed zero. The introspection lane runs today, attempts
|
|
the import, catches the failure deterministically, and emits four
|
|
sub-metrics — all zero, all explained. The day someone lands a
|
|
`core/cognition/explain.py` module, this lane immediately starts
|
|
producing real numbers without any test infrastructure change.
|
|
|
|
## Required engineering for v2
|
|
|
|
The roadmap (`docs/capability_roadmap.md` Phase 3 work items) is
|
|
explicit:
|
|
|
|
> A new `cognition/explain.py` module may be needed for
|
|
> introspection.
|
|
|
|
Concretely, an `explain(result: CognitiveTurnResult) -> str`
|
|
function that:
|
|
|
|
1. **Reads structured state from the result** — intent tag,
|
|
proposition graph, articulation target, vault hits, identity
|
|
score.
|
|
2. **Composes a deterministic natural-language account** that
|
|
re-states the trajectory in source language. Probably leans on
|
|
the same `realize_semantic` machinery currently used for
|
|
articulation but inverted: surface → structured trace → surface'.
|
|
3. **Round-trip property**: feeding the account back through the
|
|
pipeline produces an articulation whose token coverage of the
|
|
original surface is high. Strict trace-hash equivalence is the
|
|
ideal but not the v1 bar — surface token overlap ≥ 0.60 is the
|
|
v1 contract.
|
|
|
|
## Future direction (recorded here so it's not forgotten)
|
|
|
|
A working introspection API is also the substrate for **narrative
|
|
self-explanation**: the same machinery that produces "I answered X
|
|
because I retrieved Y under intent Z" is what produces an agent's
|
|
own first-person account of a turn. Per the open scope decision in
|
|
`docs/PROGRESS.md` (Agency: responsive vs. goal-directed), this
|
|
choice should pin before introspection v2 is engineered.
|
|
|
|
## Status
|
|
|
|
v1 is structural-zero scaffolding. Permanent regression evidence
|
|
of the missing module.
|