819c8b81ac
Spreads the four remaining Phase 3 lanes to map the full reasoning-
depth surface alongside inference-closure (already landed at e509e0d).
Each lane is a v1 honest probe per the roadmap; engineering work
follows once the full surface is visible.
Results across all five Phase 3 lanes:
lane split primary signal foundation
inference-closure public/v1 0.0 1.0 / 1.0
inference-closure holdouts/v1 0.0 1.0 / 1.0
compositionality public/v1 0.0625 (1/16) 1.0 / 1.0
compositionality holdouts/v1 0.0 1.0 / 1.0
multi-step-reasoning public/v1 0.0 1.0 / 1.0
multi-step-reasoning holdouts/v1 0.0 1.0 / 1.0
introspection public/v1 0.0 (no api) n/a
introspection holdouts/v1 0.0 n/a
cross-domain-transfer public/v1 0.0 1.0 / 1.0
cross-domain-transfer holdouts/v1 0.0 1.0 / 1.0
Foundation guarantees (storage + replay) intact across every lane
that has them. The reasoning-depth signal is uniformly zero. The
five lanes triangulate four architectural gaps:
Gap 1. generate/graph_planner.py has no transitive composition.
Gap 2. field/propagate.py has no derivable-but-not-asserted recall.
Gap 3. core/cognition/explain.py module does not exist.
Gap 4. no structural-pattern recogniser (cross-subdomain transfer).
Gaps 1, 2, 4 cluster on the same code surface and may close together
as a single bounded PR. Gap 3 is independent module-creation work.
Lane scaffolding mirrors inference-closure (contract.md, runner.py,
dev + public/v1 + holdouts/v1 cases.jsonl, baselines/v1_structural_zero.json,
gaps.md). All runners are parallel-safe and use the standard
run_lane(cases, *, config, workers) interface.
Per-lane gaps.md records the engineering shape for v2 plus future
directions worth not forgetting:
- compositionality/gaps.md: metaphor is compositionality with
selective property transfer; building it is correctly downstream
of closing this lane.
- cross-domain-transfer/gaps.md: metaphor + narrative as
cross-domain operators; narrative requires the Agency open-scope
decision to pin first.
- introspection/gaps.md: explain API is also the substrate for
first-person narrative self-account.
Recommended v2 sequence in docs/PROGRESS.md:
1. Pin Agency + Tool-use open-scope decisions (deadline: before
Phase 3 engineering).
2. Engineer Gaps 1 + 2 as one bounded PR.
3. Engineer Gap 3 independently.
4. Re-author cross-domain-transfer v2 with matched-control
contract refinement.
Phase 3 v1 exit: 0/5 lanes passing, which is the expected v1 floor.
CLI suites smoke / cognition / teaching pass; no regression on
Phase 2.
81 lines
3.4 KiB
Markdown
81 lines
3.4 KiB
Markdown
# compositionality eval lane
|
|
|
|
## What it measures
|
|
|
|
Whether CORE generalises **across construction families**: relation
|
|
patterns and entity sets seen at teaching time should compose into
|
|
novel (relation, entity) combinations at probe time, even though the
|
|
specific combination was never taught directly.
|
|
|
|
This is the lane the roadmap flags as most vulnerable to overfitting
|
|
(`docs/capability_roadmap.md` Phase 3, anti-overfitting note). The
|
|
split below honours that warning:
|
|
|
|
Training (teaching turns) Test (probe)
|
|
-------------------------- ----------------------------
|
|
R1(A, B), R1(C, D) R1(A, D) — seen entities, novel pair
|
|
R2(A, B), R2(C, D) R2(C, B) — same
|
|
R3(E, F), R3(G, H) R3 applied to seen entities only
|
|
...
|
|
(NEVER teach (A, D) under R1)
|
|
|
|
The probe asks for the entailment under a relation the model has
|
|
seen with *both endpoints* — but never with this specific pair.
|
|
|
|
## Why it matters
|
|
|
|
Frontier LLMs compose well because their training set already
|
|
contains nearly every short combination of common entities and
|
|
relations. CORE's claim is stronger and harder: that the algebraic
|
|
structure of the proposition graph *itself* supports composition,
|
|
without requiring the specific combination to have been seen. This
|
|
lane tests that claim.
|
|
|
|
## Patterns covered (v1)
|
|
|
|
| Pattern | Construction-family rule |
|
|
|---|---|
|
|
| `novel_pair_under_seen_relation` | `R(A,B)` and `R(C,D)` taught; probe `R(A,D)`. Pass = response references `D` (the seen RHS under R applied to seen LHS A). |
|
|
| `novel_relation_on_seen_pair` | `R(A,B)` and `R'(C,D)` taught with `A`, `B`, `C`, `D` independently grounded; probe `R'(A,B)`. Pass = response references the chain-derived target under `R'`. |
|
|
| `composed_predicate` | `is(A,B)` and `precedes(B,C)` taught; probe asks `What does A precede?` Pass = response references `C`. |
|
|
|
|
Each pattern relies only on the existing
|
|
`en_core_cognition_v1` relation vocabulary (`is`, `causes`,
|
|
`precedes`, `follows`, `grounds`, `belongs_to`, `means`, `reveals`,
|
|
`contrasts_with`).
|
|
|
|
## Sub-metrics
|
|
|
|
- `M1. compositional_token_hit` — the expected composed-entity
|
|
token appears in `surface` or `walk_surface` (case-insensitive,
|
|
token-bounded).
|
|
- `M2. premises_stored` — all teaching turns produce
|
|
proposals.
|
|
- `M3. replay_determinism` — two fresh runs match by
|
|
`trace_hash`.
|
|
- `M4. no_taught_pair_leakage` — the construction-family split is
|
|
enforced at authoring time (verified by the lane runner: every
|
|
probe is checked against the premise list to ensure the probe's
|
|
exact `(R, A, target)` triple does NOT appear verbatim).
|
|
|
|
A case passes when M1 AND M2 AND M3 hold. M4 is a structural
|
|
authoring check (true by construction); the runner reports it for
|
|
audit.
|
|
|
|
## Overall pass thresholds (v1)
|
|
|
|
- `compositional_recall_rate` (M1) ≥ 0.50
|
|
- `premises_stored_rate` ≥ 0.95
|
|
- `replay_determinism` ≥ 0.95
|
|
|
|
This lane is built knowing the same `graph_planner` and
|
|
`field/propagate` gaps that the inference-closure lane surfaced will
|
|
likely cause v1 to fail uniformly. v1's value is to score the gap
|
|
*per pattern* so the future v2 engineering can target the right one.
|
|
|
|
## Anti-overfitting
|
|
|
|
- Public split uses one entity set; holdouts uses a disjoint set.
|
|
- No probe's `(R, A, target)` triple is ever a verbatim premise.
|
|
- Patterns differ structurally between splits to avoid template
|
|
memorisation.
|