Shay 819c8b81ac feat(phase3): compositionality, multi-step-reasoning, introspection, cross-domain-transfer v1

Spreads the four remaining Phase 3 lanes to map the full reasoning-
depth surface alongside inference-closure (already landed at e509e0d).
Each lane is a v1 honest probe per the roadmap; engineering work
follows once the full surface is visible.

Results across all five Phase 3 lanes:

  lane                      split        primary signal  foundation
  inference-closure         public/v1    0.0             1.0 / 1.0
  inference-closure         holdouts/v1  0.0             1.0 / 1.0
  compositionality          public/v1   0.0625 (1/16)   1.0 / 1.0
  compositionality          holdouts/v1  0.0             1.0 / 1.0
  multi-step-reasoning      public/v1    0.0             1.0 / 1.0
  multi-step-reasoning      holdouts/v1  0.0             1.0 / 1.0
  introspection             public/v1    0.0 (no api)    n/a
  introspection             holdouts/v1  0.0             n/a
  cross-domain-transfer     public/v1    0.0             1.0 / 1.0
  cross-domain-transfer     holdouts/v1  0.0             1.0 / 1.0

Foundation guarantees (storage + replay) intact across every lane
that has them. The reasoning-depth signal is uniformly zero. The
five lanes triangulate four architectural gaps:

  Gap 1. generate/graph_planner.py has no transitive composition.
  Gap 2. field/propagate.py has no derivable-but-not-asserted recall.
  Gap 3. core/cognition/explain.py module does not exist.
  Gap 4. no structural-pattern recogniser (cross-subdomain transfer).

Gaps 1, 2, 4 cluster on the same code surface and may close together
as a single bounded PR. Gap 3 is independent module-creation work.

Lane scaffolding mirrors inference-closure (contract.md, runner.py,
dev + public/v1 + holdouts/v1 cases.jsonl, baselines/v1_structural_zero.json,
gaps.md). All runners are parallel-safe and use the standard
run_lane(cases, *, config, workers) interface.

Per-lane gaps.md records the engineering shape for v2 plus future
directions worth not forgetting:
  - compositionality/gaps.md: metaphor is compositionality with
    selective property transfer; building it is correctly downstream
    of closing this lane.
  - cross-domain-transfer/gaps.md: metaphor + narrative as
    cross-domain operators; narrative requires the Agency open-scope
    decision to pin first.
  - introspection/gaps.md: explain API is also the substrate for
    first-person narrative self-account.

Recommended v2 sequence in docs/PROGRESS.md:
  1. Pin Agency + Tool-use open-scope decisions (deadline: before
     Phase 3 engineering).
  2. Engineer Gaps 1 + 2 as one bounded PR.
  3. Engineer Gap 3 independently.
  4. Re-author cross-domain-transfer v2 with matched-control
     contract refinement.

Phase 3 v1 exit: 0/5 lanes passing, which is the expected v1 floor.
CLI suites smoke / cognition / teaching pass; no regression on
Phase 2.

2026-05-16 14:48:36 -07:00

3.4 KiB

Raw Blame History

compositionality eval lane

What it measures

Whether CORE generalises across construction families: relation patterns and entity sets seen at teaching time should compose into novel (relation, entity) combinations at probe time, even though the specific combination was never taught directly.

This is the lane the roadmap flags as most vulnerable to overfitting (docs/capability_roadmap.md Phase 3, anti-overfitting note). The split below honours that warning:

Training (teaching turns) Test (probe)

R1(A, B), R1(C, D) R1(A, D) — seen entities, novel pair R2(A, B), R2(C, D) R2(C, B) — same R3(E, F), R3(G, H) R3 applied to seen entities only ... (NEVER teach (A, D) under R1)

The probe asks for the entailment under a relation the model has seen with both endpoints — but never with this specific pair.

Why it matters

Frontier LLMs compose well because their training set already contains nearly every short combination of common entities and relations. CORE's claim is stronger and harder: that the algebraic structure of the proposition graph itself supports composition, without requiring the specific combination to have been seen. This lane tests that claim.

Patterns covered (v1)

Pattern	Construction-family rule
`novel_pair_under_seen_relation`	`R(A,B)` and `R(C,D)` taught; probe `R(A,D)`. Pass = response references `D` (the seen RHS under R applied to seen LHS A).
`novel_relation_on_seen_pair`	`R(A,B)` and `R'(C,D)` taught with `A`, `B`, `C`, `D` independently grounded; probe `R'(A,B)`. Pass = response references the chain-derived target under `R'`.
`composed_predicate`	`is(A,B)` and `precedes(B,C)` taught; probe asks `What does A precede?` Pass = response references `C`.

Each pattern relies only on the existing en_core_cognition_v1 relation vocabulary (is, causes, precedes, follows, grounds, belongs_to, means, reveals, contrasts_with).

Sub-metrics

M1. compositional_token_hit — the expected composed-entity token appears in surface or walk_surface (case-insensitive, token-bounded).
M2. premises_stored — all teaching turns produce proposals.
M3. replay_determinism — two fresh runs match by trace_hash.
M4. no_taught_pair_leakage — the construction-family split is enforced at authoring time (verified by the lane runner: every probe is checked against the premise list to ensure the probe's exact (R, A, target) triple does NOT appear verbatim).

A case passes when M1 AND M2 AND M3 hold. M4 is a structural authoring check (true by construction); the runner reports it for audit.

Overall pass thresholds (v1)

compositional_recall_rate (M1) ≥ 0.50
premises_stored_rate ≥ 0.95
replay_determinism ≥ 0.95

This lane is built knowing the same graph_planner and field/propagate gaps that the inference-closure lane surfaced will likely cause v1 to fail uniformly. v1's value is to score the gap per pattern so the future v2 engineering can target the right one.

Anti-overfitting

Public split uses one entity set; holdouts uses a disjoint set.
No probe's (R, A, target) triple is ever a verbatim premise.
Patterns differ structurally between splits to avoid template memorisation.

3.4 KiB Raw Blame History