b5d6ad6510
Closes the residual `novel_pair_under_seen_relation` pattern that neither `transitive_walk` nor `multi_relation_walk` could synthesise. - new `compose_relations(triples, head, frame, relation)` operator — pure lookup, returns both `R(head, ?)` and `R(frame, ?)` tails - new `FRAME_TRANSFER` intent + `_FRAME_TRANSFER_RE` regex tried before generic TRANSITIVE_QUERY so "in Y" isn't truncated; handles "X belong to in Y" → belongs_to normalisation - pipeline wiring: `_maybe_compose_relations`, `_fold_compose_into_surface`, `_serialize_compose` (folded into operator_invocation so trace_hash stays bit-identical across replay) - regression: inference_closure, multi_step_reasoning, cross_domain_transfer all still 100% on public + holdouts discourse_paragraph v2: - per-sentence grammar rubric (length, capitalization, subject alignment) gated on `require_per_sentence_grammar` - scaling cases at 10 / 20 / 50 sentences — 3/3 pass, 100% per-sentence - 3 runtime round-trip cases (`mode: runtime_roundtrip`) that prime vault, ask question, verify bit-identical across two fresh runtimes - new `per_sentence_grammar_pass_rate` lane metric Long-form replay benchmark (benchmarks/replay_vs_llm.py): - `replay_determinism_report(prompts, runs, priming)` — CORE-only - `compare_to_llm(prompts, llm_callable)` — BYO API client, no provider lock-in; reports per-prompt determinism on both sides - ships with default cognition-pack prompts; 100% bit-identical at runs=3 Lanes green: cognition 121/121, runtime 19/19, teaching 17/17, packs 6/6, compositionality 16/16 + 10/10, inference_closure 20/20 + 12/12, multi_step_reasoning 15/15 + 10/10, cross_domain_transfer 10/10 + 8/8, discourse_paragraph v1 12/12 + v2 6/6.
113 lines
4.9 KiB
Markdown
113 lines
4.9 KiB
Markdown
# compositionality lane — architectural findings (v1)
|
|
|
|
## Resolution (full) — 2026-05-16 compose_relations lands
|
|
|
|
After the typed operators + pipeline wiring + `compose_relations`:
|
|
|
|
| Split | n | compositional_recall_rate | premises_stored | replay | overall |
|
|
|---|---|---|---|---|---|
|
|
| public/v1 | 16 | **1.0** (was 0.0625 → 0.6875 → 1.0) | 1.0 | 1.0 | ✓ pass |
|
|
|
|
All three patterns now hit:
|
|
|
|
- `composed_predicate` (7/7) — via `multi_relation_walk` (chain
|
|
A → B → C across mixed relations).
|
|
- `novel_relation_on_seen_pair` (4/4) — via `multi_relation_walk`
|
|
matching morphological verb-form probes against the chain
|
|
endpoint noun.
|
|
- `novel_pair_under_seen_relation` (5/5) — via the **new
|
|
`compose_relations` operator** + the `FRAME_TRANSFER` intent
|
|
shape ("What does X R in Y?"). The operator reports both
|
|
`R(X, ?)` and `R(Y, ?)` tails so the realizer surfaces the
|
|
cross-instance compositional answer.
|
|
|
|
### How it works
|
|
|
|
1. `_FRAME_TRANSFER_RE` (`generate/intent.py`) matches the probe
|
|
shape "What does X R [to] in Y?" — tried before the generic
|
|
`TRANSITIVE_QUERY` regex so the trailing "in Y" is not
|
|
silently truncated. An optional "to" between R and "in" is
|
|
normalized to `belongs_to`.
|
|
2. `compose_relations(triples, head, frame, relation)`
|
|
(`generate/operators.py`) is a pure function that looks up
|
|
both `R(head, ?)` and `R(frame, ?)` from the typed teaching
|
|
store and returns a `FrameComposeResult` with both tails (or
|
|
None when an edge is absent).
|
|
3. `CognitiveTurnPipeline._maybe_compose_relations` fires only on
|
|
`FRAME_TRANSFER` intents, `_fold_compose_into_surface` names
|
|
both endpoints in the surface deterministically, and
|
|
`_serialize_compose` folds the result into `operator_invocation`
|
|
so `trace_hash` remains bit-identical across replay.
|
|
|
|
Historic findings preserved below.
|
|
|
|
## Original v1 result (now superseded)
|
|
|
|
| Split | n | compositional_recall_rate | premises_stored | replay | no_leakage |
|
|
|---|---|---|---|---|---|
|
|
| public/v1 | 16 | **0.0625** (1/16) | 1.0 | 1.0 | 0.4375 |
|
|
| holdouts/v1 | 10 | **0.0** | 1.0 | 1.0 | 0.4 |
|
|
|
|
The single public hit is consistent with a realizer-template token
|
|
coincidence rather than real composition (no second hit on holdouts;
|
|
no pattern in the hit; not reproducible across patterns).
|
|
|
|
## Foundation intact
|
|
|
|
Every teaching turn fires a `PackMutationProposal`
|
|
(`premises_stored_rate = 1.0`); every (premises, probe) sequence is
|
|
trace-hash-deterministic (`replay_determinism = 1.0`). The
|
|
Phase 2 storage + replay guarantees survive at this depth.
|
|
|
|
## What v1 reveals
|
|
|
|
- **No composition operator.** Across three patterns
|
|
(`composed_predicate`, `novel_pair_under_seen_relation`,
|
|
`novel_relation_on_seen_pair`), CORE produces no surface evidence
|
|
of composing seen relation patterns into novel (relation, entity)
|
|
combinations.
|
|
- **Same root cause as inference-closure.** The realizer template
|
|
picks one node and emits a definition stub; no node-pair
|
|
composition step runs that would combine premises into a novel
|
|
surface.
|
|
|
|
## Authoring finding — leakage rate
|
|
|
|
`no_leakage_rate` is 0.4375 / 0.4 — i.e. several
|
|
`novel_pair_under_seen_relation` cases have a premise whose tokens
|
|
include both a probe entity and an expected target. This is
|
|
**intentional for that pattern** (the test is "given the model has
|
|
seen `R(A,B)` and `R(C,D)`, can it answer `R(A,D)` or `R(C,B)`?" —
|
|
both answers were taught as premise endpoints, just not together).
|
|
The strict author-time leakage check fires by design here. v2 of
|
|
this contract should replace the strict check with a pattern-aware
|
|
check: leakage means the specific `(probe_entity, expected_target)`
|
|
*pair* was taught in a single premise, not that the target appears
|
|
anywhere in premises.
|
|
|
|
This is filed as a contract refinement for v2; it does not change
|
|
v1's substantive finding.
|
|
|
|
## Architectural gap (same family as inference-closure)
|
|
|
|
Composition requires the proposition-graph planner to walk multiple
|
|
nodes and synthesize a derived articulation. `plan_articulation()`
|
|
in `generate/graph_planner.py` is single-node. Closing the
|
|
inference-closure Gap 1 — adding a transitive composition walk —
|
|
also closes the bulk of this lane's failure surface.
|
|
|
|
## Future direction (recorded here so it's not forgotten)
|
|
|
|
Metaphor and simile are structurally **compositionality with
|
|
selective property transfer**: "the heart is a pump" is the same
|
|
graph-traversal shape as the compositionality probes above, with a
|
|
filter that says *which* relations transfer across the analogy.
|
|
Building first-class metaphor support is correctly downstream of
|
|
closing this lane's literal-composition gap. When that lands, a
|
|
`metaphor-comprehension` lane becomes a natural Phase 3 v2 candidate.
|
|
|
|
## Status
|
|
|
|
v1 stands as honest-failure baseline. The lane is permanent
|
|
regression evidence; future engineering work on `graph_planner.py`
|
|
that closes inference-closure Gap 1 should be re-scored here.
|