Shay 819c8b81ac feat(phase3): compositionality, multi-step-reasoning, introspection, cross-domain-transfer v1

Spreads the four remaining Phase 3 lanes to map the full reasoning-
depth surface alongside inference-closure (already landed at e509e0d).
Each lane is a v1 honest probe per the roadmap; engineering work
follows once the full surface is visible.

Results across all five Phase 3 lanes:

  lane                      split        primary signal  foundation
  inference-closure         public/v1    0.0             1.0 / 1.0
  inference-closure         holdouts/v1  0.0             1.0 / 1.0
  compositionality          public/v1   0.0625 (1/16)   1.0 / 1.0
  compositionality          holdouts/v1  0.0             1.0 / 1.0
  multi-step-reasoning      public/v1    0.0             1.0 / 1.0
  multi-step-reasoning      holdouts/v1  0.0             1.0 / 1.0
  introspection             public/v1    0.0 (no api)    n/a
  introspection             holdouts/v1  0.0             n/a
  cross-domain-transfer     public/v1    0.0             1.0 / 1.0
  cross-domain-transfer     holdouts/v1  0.0             1.0 / 1.0

Foundation guarantees (storage + replay) intact across every lane
that has them. The reasoning-depth signal is uniformly zero. The
five lanes triangulate four architectural gaps:

  Gap 1. generate/graph_planner.py has no transitive composition.
  Gap 2. field/propagate.py has no derivable-but-not-asserted recall.
  Gap 3. core/cognition/explain.py module does not exist.
  Gap 4. no structural-pattern recogniser (cross-subdomain transfer).

Gaps 1, 2, 4 cluster on the same code surface and may close together
as a single bounded PR. Gap 3 is independent module-creation work.

Lane scaffolding mirrors inference-closure (contract.md, runner.py,
dev + public/v1 + holdouts/v1 cases.jsonl, baselines/v1_structural_zero.json,
gaps.md). All runners are parallel-safe and use the standard
run_lane(cases, *, config, workers) interface.

Per-lane gaps.md records the engineering shape for v2 plus future
directions worth not forgetting:
  - compositionality/gaps.md: metaphor is compositionality with
    selective property transfer; building it is correctly downstream
    of closing this lane.
  - cross-domain-transfer/gaps.md: metaphor + narrative as
    cross-domain operators; narrative requires the Agency open-scope
    decision to pin first.
  - introspection/gaps.md: explain API is also the substrate for
    first-person narrative self-account.

Recommended v2 sequence in docs/PROGRESS.md:
  1. Pin Agency + Tool-use open-scope decisions (deadline: before
     Phase 3 engineering).
  2. Engineer Gaps 1 + 2 as one bounded PR.
  3. Engineer Gap 3 independently.
  4. Re-author cross-domain-transfer v2 with matched-control
     contract refinement.

Phase 3 v1 exit: 0/5 lanes passing, which is the expected v1 floor.
CLI suites smoke / cognition / teaching pass; no regression on
Phase 2.

2026-05-16 14:48:36 -07:00

2.8 KiB

Raw Blame History

introspection eval lane

What it measures

Whether CORE can produce a natural-language account of a prior turn that round-trips: a separate run conditioned on that account predicts the same articulation as the original turn.

Roadmap shape (Phase 3):

Run 1: pipeline.run(prompt) -> Result_A (surface, trace_hash_A) Step: explain(Result_A.turn_id) -> account (natural-language) Run 2: fresh pipeline.run(account) -> Result_B (surface, trace_hash_B) Round-trip pass: Result_B.surface == Result_A.surface (or a defensibly equivalent surface)

A passing round-trip demonstrates that CORE's articulation is explainable in its own terms and that the explanation carries enough state to reconstruct the answer.

v1 reality: the `explain` interface does not exist

CORE has no cognition/explain.py module today. Per the roadmap (Phase 3 work items): "A new cognition/explain.py module may be needed for introspection." v1's role is to score the gap honestly: the runner attempts to import an explain function from core.cognition and falls through with M1=0 when the import fails. This makes the lane runnable today and gives a structural- zero result by construction until the module lands.

Sub-metrics

M1. explain_api_present — the explain function imports cleanly from core.cognition (or a documented alternative).
M2. account_is_nonempty — when (1) succeeds, the generated account has non-trivial length (≥ 5 tokens).
M3. round_trip_surface_match — Result_B.surface tokens cover ≥ 60% of Result_A.surface tokens (case-insensitive, punctuation-stripped).
M4. round_trip_trace_match — Result_B.trace_hash == Result_A.trace_hash (strict deterministic round-trip).

Today's expected result: M1 = 0; all downstream metrics = 0.

A case passes when M1 AND M2 AND M3 hold. M4 is reported as a stricter signal — likely to fail even after M3 starts succeeding because the input texts (original prompt vs. account) differ verbatim and trace_hash is computed over input_text.

Overall pass thresholds (v1)

explain_api_present_rate (M1) ≥ 0.95 — trivial once the module exists
account_nonempty_rate (M2) ≥ 0.95
round_trip_surface_match_rate (M3) ≥ 0.50

v1's expected score: all zero. v1 is the lane that explicitly tests whether the explain primitive exists and produces a usable account. Until it does, this is structural-zero work.

Why a placeholder-runnable v1

The Phase 3 exit criteria state: "v1 results with honest scores (which may be failing — that's acceptable for v1). Each failure has either a closed engineering gap or a documented architectural deferral." A lane that cannot run at all is worse than a lane that runs and reports zero; the latter forms a real regression trigger for the day the engineering lands.

2.8 KiB Raw Blame History