Shay 819c8b81ac feat(phase3): compositionality, multi-step-reasoning, introspection, cross-domain-transfer v1

Spreads the four remaining Phase 3 lanes to map the full reasoning-
depth surface alongside inference-closure (already landed at e509e0d).
Each lane is a v1 honest probe per the roadmap; engineering work
follows once the full surface is visible.

Results across all five Phase 3 lanes:

  lane                      split        primary signal  foundation
  inference-closure         public/v1    0.0             1.0 / 1.0
  inference-closure         holdouts/v1  0.0             1.0 / 1.0
  compositionality          public/v1   0.0625 (1/16)   1.0 / 1.0
  compositionality          holdouts/v1  0.0             1.0 / 1.0
  multi-step-reasoning      public/v1    0.0             1.0 / 1.0
  multi-step-reasoning      holdouts/v1  0.0             1.0 / 1.0
  introspection             public/v1    0.0 (no api)    n/a
  introspection             holdouts/v1  0.0             n/a
  cross-domain-transfer     public/v1    0.0             1.0 / 1.0
  cross-domain-transfer     holdouts/v1  0.0             1.0 / 1.0

Foundation guarantees (storage + replay) intact across every lane
that has them. The reasoning-depth signal is uniformly zero. The
five lanes triangulate four architectural gaps:

  Gap 1. generate/graph_planner.py has no transitive composition.
  Gap 2. field/propagate.py has no derivable-but-not-asserted recall.
  Gap 3. core/cognition/explain.py module does not exist.
  Gap 4. no structural-pattern recogniser (cross-subdomain transfer).

Gaps 1, 2, 4 cluster on the same code surface and may close together
as a single bounded PR. Gap 3 is independent module-creation work.

Lane scaffolding mirrors inference-closure (contract.md, runner.py,
dev + public/v1 + holdouts/v1 cases.jsonl, baselines/v1_structural_zero.json,
gaps.md). All runners are parallel-safe and use the standard
run_lane(cases, *, config, workers) interface.

Per-lane gaps.md records the engineering shape for v2 plus future
directions worth not forgetting:
  - compositionality/gaps.md: metaphor is compositionality with
    selective property transfer; building it is correctly downstream
    of closing this lane.
  - cross-domain-transfer/gaps.md: metaphor + narrative as
    cross-domain operators; narrative requires the Agency open-scope
    decision to pin first.
  - introspection/gaps.md: explain API is also the substrate for
    first-person narrative self-account.

Recommended v2 sequence in docs/PROGRESS.md:
  1. Pin Agency + Tool-use open-scope decisions (deadline: before
     Phase 3 engineering).
  2. Engineer Gaps 1 + 2 as one bounded PR.
  3. Engineer Gap 3 independently.
  4. Re-author cross-domain-transfer v2 with matched-control
     contract refinement.

Phase 3 v1 exit: 0/5 lanes passing, which is the expected v1 floor.
CLI suites smoke / cognition / teaching pass; no regression on
Phase 2.

2026-05-16 14:48:36 -07:00

3.1 KiB

Raw Blame History

compositionality lane — architectural findings (v1)

v1 result

Split	n	compositional_recall_rate	premises_stored	replay	no_leakage
public/v1	16	0.0625 (1/16)	1.0	1.0	0.4375
holdouts/v1	10	0.0	1.0	1.0	0.4

The single public hit is consistent with a realizer-template token coincidence rather than real composition (no second hit on holdouts; no pattern in the hit; not reproducible across patterns).

Foundation intact

Every teaching turn fires a PackMutationProposal (premises_stored_rate = 1.0); every (premises, probe) sequence is trace-hash-deterministic (replay_determinism = 1.0). The Phase 2 storage + replay guarantees survive at this depth.

What v1 reveals

No composition operator. Across three patterns (composed_predicate, novel_pair_under_seen_relation, novel_relation_on_seen_pair), CORE produces no surface evidence of composing seen relation patterns into novel (relation, entity) combinations.
Same root cause as inference-closure. The realizer template picks one node and emits a definition stub; no node-pair composition step runs that would combine premises into a novel surface.

Authoring finding — leakage rate

no_leakage_rate is 0.4375 / 0.4 — i.e. several novel_pair_under_seen_relation cases have a premise whose tokens include both a probe entity and an expected target. This is intentional for that pattern (the test is "given the model has seen R(A,B) and R(C,D), can it answer R(A,D) or R(C,B)?" — both answers were taught as premise endpoints, just not together). The strict author-time leakage check fires by design here. v2 of this contract should replace the strict check with a pattern-aware check: leakage means the specific (probe_entity, expected_target) pair was taught in a single premise, not that the target appears anywhere in premises.

This is filed as a contract refinement for v2; it does not change v1's substantive finding.

Architectural gap (same family as inference-closure)

Composition requires the proposition-graph planner to walk multiple nodes and synthesize a derived articulation. plan_articulation() in generate/graph_planner.py is single-node. Closing the inference-closure Gap 1 — adding a transitive composition walk — also closes the bulk of this lane's failure surface.

Future direction (recorded here so it's not forgotten)

Metaphor and simile are structurally compositionality with selective property transfer: "the heart is a pump" is the same graph-traversal shape as the compositionality probes above, with a filter that says which relations transfer across the analogy. Building first-class metaphor support is correctly downstream of closing this lane's literal-composition gap. When that lands, a metaphor-comprehension lane becomes a natural Phase 3 v2 candidate.

Status

v1 stands as honest-failure baseline. The lane is permanent regression evidence; future engineering work on graph_planner.py that closes inference-closure Gap 1 should be re-scored here.

3.1 KiB Raw Blame History