e509e0d6d6
First Phase 3 lane. Scores whether CORE can derive entailments that
were not directly asserted, given a chain of premises taught through
the correction loop. Five transitive relation patterns drawn from
en_core_cognition_v1:
transitive_is A is B; B is C -> What is A?
transitive_precedes A precedes B; B precedes C -> What does A precede?
transitive_grounds A grounds B; B grounds C -> What does A ground?
transitive_causes A causes B; B causes C -> What does A cause?
transitive_belongs_to A belongs_to B; B belongs_to C -> Where does A belong?
Pass = expected entailment token appears in probe response surface
or walk surface (M1 or M2) AND every premise stored (M3) AND
trace_hash deterministic across two fresh runs (M4).
Results:
split n derived stored replay overall_pass
public/v1 20 0.0 1.0 1.0 False
holdouts/v1 12 0.0 1.0 1.0 False
This is the expected honest failure per docs/capability_roadmap.md
Phase 3. Foundation guarantees from Phase 2 (storage + replay) hold
at this depth; the inference-closure step itself does not yet exist
in CORE. The lane scores exactly the gap.
Concrete trace recorded in gaps.md: for premises 'wisdom is light',
'light is truth', probe 'What is wisdom?' returns the template
'wisdom is defined as ...' — vault retrieves 9 entries including
both premises, but the realizer emits a definition stub instead of
a derivation.
Architectural gaps filed (evals/inference_closure/gaps.md):
Gap 1. generate/graph_planner.py has no transitive composition —
plan_articulation picks a single node; there is no chained
relation walk that produces a derived node from premises.
Gap 2. field/propagate.py has no derivable-but-not-asserted recall
path — vault retrieval is direct CGA inner product; no
path-recall operator over relation-typed edges.
Both gaps are v2 engineering candidates and may share an
implementation surface. The lane is permanent regression evidence
of what specifically is missing.
Includes:
- contract.md: pass criteria, anti-overfitting note, sub-metric
definitions, calibration approach.
- runner.py: parallel, fresh-pipeline-per-case, M1-M4 scoring,
two-run replay-determinism check.
- dev/cases.jsonl (5), public/v1 (20), holdouts/v1 (12) — disjoint
entity sets, all five patterns covered.
- baselines/v1_structural_zero.json: frontier LLMs do not emit
the typed signals by construction.
- gaps.md: full architectural finding, engineering shapes for v2.
CLI suites smoke / cognition / teaching pass; no regression on
Phase 2 work.
5 lines
1.3 KiB
JSON
5 lines
1.3 KiB
JSON
{"id":"INF-DEV-001","pattern":"transitive_is","premises":["What is wisdom?","Actually wisdom is light.","What is light?","Actually light is truth."],"probe":"What is wisdom?","expected_entailment_tokens":["truth"],"expected_proposals":2}
|
|
{"id":"INF-DEV-002","pattern":"transitive_precedes","premises":["What is creation?","Actually creation precedes order.","What is order?","Actually order precedes meaning."],"probe":"What does creation precede?","expected_entailment_tokens":["meaning"],"expected_proposals":2}
|
|
{"id":"INF-DEV-003","pattern":"transitive_grounds","premises":["What is truth?","Actually truth grounds knowledge.","What is knowledge?","Actually knowledge grounds judgment."],"probe":"What does truth ground?","expected_entailment_tokens":["judgment"],"expected_proposals":2}
|
|
{"id":"INF-DEV-004","pattern":"transitive_causes","premises":["What is light?","Actually light causes clarity.","What is clarity?","Actually clarity causes recognition."],"probe":"What does light cause?","expected_entailment_tokens":["recognition"],"expected_proposals":2}
|
|
{"id":"INF-DEV-005","pattern":"transitive_belongs_to","premises":["What is question?","Actually question belongs_to inquiry.","What is inquiry?","Actually inquiry belongs_to thought."],"probe":"Where does question belong?","expected_entailment_tokens":["thought"],"expected_proposals":2}
|