819c8b81ac
Spreads the four remaining Phase 3 lanes to map the full reasoning-
depth surface alongside inference-closure (already landed at e509e0d).
Each lane is a v1 honest probe per the roadmap; engineering work
follows once the full surface is visible.
Results across all five Phase 3 lanes:
lane split primary signal foundation
inference-closure public/v1 0.0 1.0 / 1.0
inference-closure holdouts/v1 0.0 1.0 / 1.0
compositionality public/v1 0.0625 (1/16) 1.0 / 1.0
compositionality holdouts/v1 0.0 1.0 / 1.0
multi-step-reasoning public/v1 0.0 1.0 / 1.0
multi-step-reasoning holdouts/v1 0.0 1.0 / 1.0
introspection public/v1 0.0 (no api) n/a
introspection holdouts/v1 0.0 n/a
cross-domain-transfer public/v1 0.0 1.0 / 1.0
cross-domain-transfer holdouts/v1 0.0 1.0 / 1.0
Foundation guarantees (storage + replay) intact across every lane
that has them. The reasoning-depth signal is uniformly zero. The
five lanes triangulate four architectural gaps:
Gap 1. generate/graph_planner.py has no transitive composition.
Gap 2. field/propagate.py has no derivable-but-not-asserted recall.
Gap 3. core/cognition/explain.py module does not exist.
Gap 4. no structural-pattern recogniser (cross-subdomain transfer).
Gaps 1, 2, 4 cluster on the same code surface and may close together
as a single bounded PR. Gap 3 is independent module-creation work.
Lane scaffolding mirrors inference-closure (contract.md, runner.py,
dev + public/v1 + holdouts/v1 cases.jsonl, baselines/v1_structural_zero.json,
gaps.md). All runners are parallel-safe and use the standard
run_lane(cases, *, config, workers) interface.
Per-lane gaps.md records the engineering shape for v2 plus future
directions worth not forgetting:
- compositionality/gaps.md: metaphor is compositionality with
selective property transfer; building it is correctly downstream
of closing this lane.
- cross-domain-transfer/gaps.md: metaphor + narrative as
cross-domain operators; narrative requires the Agency open-scope
decision to pin first.
- introspection/gaps.md: explain API is also the substrate for
first-person narrative self-account.
Recommended v2 sequence in docs/PROGRESS.md:
1. Pin Agency + Tool-use open-scope decisions (deadline: before
Phase 3 engineering).
2. Engineer Gaps 1 + 2 as one bounded PR.
3. Engineer Gap 3 independently.
4. Re-author cross-domain-transfer v2 with matched-control
contract refinement.
Phase 3 v1 exit: 0/5 lanes passing, which is the expected v1 floor.
CLI suites smoke / cognition / teaching pass; no regression on
Phase 2.
72 lines
3.1 KiB
Markdown
72 lines
3.1 KiB
Markdown
# compositionality lane — architectural findings (v1)
|
|
|
|
## v1 result
|
|
|
|
| Split | n | compositional_recall_rate | premises_stored | replay | no_leakage |
|
|
|---|---|---|---|---|---|
|
|
| public/v1 | 16 | **0.0625** (1/16) | 1.0 | 1.0 | 0.4375 |
|
|
| holdouts/v1 | 10 | **0.0** | 1.0 | 1.0 | 0.4 |
|
|
|
|
The single public hit is consistent with a realizer-template token
|
|
coincidence rather than real composition (no second hit on holdouts;
|
|
no pattern in the hit; not reproducible across patterns).
|
|
|
|
## Foundation intact
|
|
|
|
Every teaching turn fires a `PackMutationProposal`
|
|
(`premises_stored_rate = 1.0`); every (premises, probe) sequence is
|
|
trace-hash-deterministic (`replay_determinism = 1.0`). The
|
|
Phase 2 storage + replay guarantees survive at this depth.
|
|
|
|
## What v1 reveals
|
|
|
|
- **No composition operator.** Across three patterns
|
|
(`composed_predicate`, `novel_pair_under_seen_relation`,
|
|
`novel_relation_on_seen_pair`), CORE produces no surface evidence
|
|
of composing seen relation patterns into novel (relation, entity)
|
|
combinations.
|
|
- **Same root cause as inference-closure.** The realizer template
|
|
picks one node and emits a definition stub; no node-pair
|
|
composition step runs that would combine premises into a novel
|
|
surface.
|
|
|
|
## Authoring finding — leakage rate
|
|
|
|
`no_leakage_rate` is 0.4375 / 0.4 — i.e. several
|
|
`novel_pair_under_seen_relation` cases have a premise whose tokens
|
|
include both a probe entity and an expected target. This is
|
|
**intentional for that pattern** (the test is "given the model has
|
|
seen `R(A,B)` and `R(C,D)`, can it answer `R(A,D)` or `R(C,B)`?" —
|
|
both answers were taught as premise endpoints, just not together).
|
|
The strict author-time leakage check fires by design here. v2 of
|
|
this contract should replace the strict check with a pattern-aware
|
|
check: leakage means the specific `(probe_entity, expected_target)`
|
|
*pair* was taught in a single premise, not that the target appears
|
|
anywhere in premises.
|
|
|
|
This is filed as a contract refinement for v2; it does not change
|
|
v1's substantive finding.
|
|
|
|
## Architectural gap (same family as inference-closure)
|
|
|
|
Composition requires the proposition-graph planner to walk multiple
|
|
nodes and synthesize a derived articulation. `plan_articulation()`
|
|
in `generate/graph_planner.py` is single-node. Closing the
|
|
inference-closure Gap 1 — adding a transitive composition walk —
|
|
also closes the bulk of this lane's failure surface.
|
|
|
|
## Future direction (recorded here so it's not forgotten)
|
|
|
|
Metaphor and simile are structurally **compositionality with
|
|
selective property transfer**: "the heart is a pump" is the same
|
|
graph-traversal shape as the compositionality probes above, with a
|
|
filter that says *which* relations transfer across the analogy.
|
|
Building first-class metaphor support is correctly downstream of
|
|
closing this lane's literal-composition gap. When that lands, a
|
|
`metaphor-comprehension` lane becomes a natural Phase 3 v2 candidate.
|
|
|
|
## Status
|
|
|
|
v1 stands as honest-failure baseline. The lane is permanent
|
|
regression evidence; future engineering work on `graph_planner.py`
|
|
that closes inference-closure Gap 1 should be re-scored here.
|