Shay e509e0d6d6 feat(phase3): inference-closure lane v1 — foundation OK, no operator

First Phase 3 lane. Scores whether CORE can derive entailments that
were not directly asserted, given a chain of premises taught through
the correction loop. Five transitive relation patterns drawn from
en_core_cognition_v1:

  transitive_is        A is B; B is C            -> What is A?
  transitive_precedes  A precedes B; B precedes C -> What does A precede?
  transitive_grounds   A grounds B; B grounds C   -> What does A ground?
  transitive_causes    A causes B; B causes C     -> What does A cause?
  transitive_belongs_to A belongs_to B; B belongs_to C -> Where does A belong?

Pass = expected entailment token appears in probe response surface
or walk surface (M1 or M2) AND every premise stored (M3) AND
trace_hash deterministic across two fresh runs (M4).

Results:

  split        n   derived  stored  replay  overall_pass
  public/v1    20  0.0      1.0     1.0     False
  holdouts/v1  12  0.0      1.0     1.0     False

This is the expected honest failure per docs/capability_roadmap.md
Phase 3. Foundation guarantees from Phase 2 (storage + replay) hold
at this depth; the inference-closure step itself does not yet exist
in CORE. The lane scores exactly the gap.

Concrete trace recorded in gaps.md: for premises 'wisdom is light',
'light is truth', probe 'What is wisdom?' returns the template
'wisdom is defined as ...' — vault retrieves 9 entries including
both premises, but the realizer emits a definition stub instead of
a derivation.

Architectural gaps filed (evals/inference_closure/gaps.md):

  Gap 1. generate/graph_planner.py has no transitive composition —
         plan_articulation picks a single node; there is no chained
         relation walk that produces a derived node from premises.
  Gap 2. field/propagate.py has no derivable-but-not-asserted recall
         path — vault retrieval is direct CGA inner product; no
         path-recall operator over relation-typed edges.

Both gaps are v2 engineering candidates and may share an
implementation surface. The lane is permanent regression evidence
of what specifically is missing.

Includes:
  - contract.md: pass criteria, anti-overfitting note, sub-metric
    definitions, calibration approach.
  - runner.py: parallel, fresh-pipeline-per-case, M1-M4 scoring,
    two-run replay-determinism check.
  - dev/cases.jsonl (5), public/v1 (20), holdouts/v1 (12) — disjoint
    entity sets, all five patterns covered.
  - baselines/v1_structural_zero.json: frontier LLMs do not emit
    the typed signals by construction.
  - gaps.md: full architectural finding, engineering shapes for v2.

CLI suites smoke / cognition / teaching pass; no regression on
Phase 2 work.

2026-05-16 14:33:08 -07:00

4.6 KiB

Raw Blame History

inference-closure lane — architectural findings (v1)

v1 result

Split	n	derived_recall_rate	premises_stored_rate	replay_determinism	overall_pass
public/v1	20	0.0	1.0	1.0	False
holdouts/v1	12	0.0	1.0	1.0	False

This is the expected v1 outcome documented in docs/capability_roadmap.md Phase 3: lanes may fail v1 honestly, and each failure becomes either a closed engineering gap or a documented architectural deferral.

What v1 confirms

Foundation intact. Every premise emits a PackMutationProposal (M3 = 1.0); every (premises, probe) sequence is replay-deterministic by trace_hash (M4 = 1.0). The work that landed in Phase 2 (symbolic-logic v1+v2) — storage and replay — holds at this depth.
No inference operator. Across all 32 cases (20 public + 12 holdouts) covering five relation families (is, precedes, grounds, causes, belongs_to), the probe response surface never references the derived entailment token. Both surface and walk_surface are template-driven definitions/disclaimers, not derivations.

Concrete probe trace

For INF-DEV-001 — premises wisdom is light, light is truth, probe What is wisdom? — the runtime produces:

PREMISE 'What is wisdom?'          surface='wisdom is defined as ...'  vault=0
PREMISE 'Actually wisdom is light' surface='Light write.'              vault=9
PREMISE 'What is light?'           surface='light is defined as ...'   vault=5
PREMISE 'Actually light is truth.' surface='Truth thought — λαμβάνω…'  vault=9
PROBE   'What is wisdom?'
  surface              = 'wisdom is defined as ...'
  articulation_surface = 'wisdom is defined as ...'
  walk_surface         = 'Wisdom does not with.'
  vault_hits           = 9

The probe retrieves 9 vault entries (so the premises are in recall reach) but the realizer template emits a generic definition stub. The transitive entailment (truth) appears in neither surface nor walk_surface.

Architectural gaps (where the inference closure step is missing)

The roadmap pre-identified two suspects. v1 evidence narrows it to both:

Gap 1 — `generate/graph_planner.py` has no transitive composition

When the premise wisdom is light is taught, a proposition-graph node is created. When light is truth is taught, a second node is created. No edge composition step runs that would emit a derived node wisdom is truth. The articulation target planner picks a single node to articulate; it has no mechanism for chained traversal.

Engineering shape (out of scope for v1, in scope for v2):

Extend graph_from_intent() to detect a probe whose form is "what does A R?" / "what is A?" and walk outgoing R-edges of A.
Extend plan_articulation() to optionally compose a multi-node surface when the walk produces a single deterministic chain.

Gap 2 — `field/propagate.py` has no derivable-but-not-asserted recall path

Vault retrieval scores cga_inner(query, stored_versor) and returns the top-K direct matches. There is no path-based recall that says "return X if there is a relation-chain from the query entity to X." The 9 vault hits for the probe include the premise versors but not a derivation versor (none exists).

Engineering shape (also v2 candidate, may overlap with Gap 1):

A path-recall operator over the relation-typed edges of vault entries. Preserves exact-CGA semantics: the chain composition is deterministic, not approximate.

Pass criterion review

The pass thresholds in contract.md (derived_recall_rate >= 0.50, premises_stored_rate >= 0.95, replay_determinism >= 0.95) are unchanged. v1's failure is uniform on the derived-recall metric — exactly what the contract was built to detect.

What stands today

The lane exists as a permanent regression and progress signal.
Foundation guarantees (storage, replay) are independently scored and remain at 1.0 — closing the inference gap will not cost them.
Structural-zero frontier baseline recorded (baselines/v1_structural_zero.json): frontier LLMs do not emit the typed signals these sub-metrics score by construction.

Phase 3 exit posture

This lane satisfies the roadmap's v1 expectation: "v1 results with honest scores (which may be failing — that's acceptable for v1)." Phase 3 exit requires at least two lanes passing v1 by phase exit; inference-closure's gap-1 / gap-2 engineering work, if undertaken, flips this lane from failing v1 to passing v2 and contributes to that count. Until then it stands as load-bearing evidence of the specific engineering work needed.

4.6 KiB Raw Blame History