Shay 0053648efd feat(evals): symbolic-logic lane v1 — premise-chain foundations

Adds the fourth Phase 2 lane. v1 measures the structural foundations
on which a future inference engine would be built:

  M1. premise_recall    — probe vault_hits >= min after chain teaching
  M2. replay_determinism — same chain + probe → same trace_hash
  M3. proposal_storage  — correction premises store as PackMutationProposals

Patterns covered: modus_ponens_chain, modus_tollens_chain, syllogism,
negation, chain_recall (up to 4-hop chains).

v1 results across 38 cases (8 dev + 18 public + 12 holdouts):
  premise_recall=1.0, replay_determinism=1.0, proposal_storage=1.0.

Each case runs twice on fresh CognitiveTurnPipelines to verify the
trace_hash matches — confirming deterministic replay over premise chains.

Architectural finding logged in evals/symbolic_logic/gaps.md:

  CORE has no first-class inference operator. Chain "inference" today is
  emergent from teaching-store commits + cumulative vault recall, not a
  named-rule symbolic engine. v1 honestly tests what CORE deterministically
  *does* (store, replay, recall chains) without overclaiming that CORE
  reasons symbolically. v2 would assert specific transitive recall
  contents in the probe surface, which requires either a
  PropositionGraph traversal operator or pack-axiom rules — both filed
  as suggested follow-up work.

2026-05-16 12:34:55 -07:00

3.4 KiB

Raw Blame History

symbolic-logic eval lane

What it measures

CORE's foundation for proposition-based inference: when premises are taught via the correction loop, the resulting field state and vault content carry the chain in a deterministic, premise-sensitive, recallable form.

v1 measures three structural foundations on which a future inference engine would be built; it does not test that CORE applies named inference rules (modus ponens, modus tollens, syllogism) directly. See gaps.md for the architectural finding and roadmap toward v2.

Why it matters (structural win)

Frontier LLMs produce inference-like text but provide no first-class evidence that the chain was actually stored, replayed deterministically, or that the produced trace depends on the specific premises given. Their inference is a single forward pass over a stochastic policy.

CORE returns:

CognitiveTurnResult.vault_hits — how many premises the probe recalled.
CognitiveTurnResult.trace_hash — a SHA-256 over deterministic pipeline state.
CognitiveTurnResult.pack_mutation_proposal — datestamped record for each correction-intent turn.

These let a downstream caller verify three properties of any premise-chain inference:

Recall: the probe can see the chain (vault_hits > 0).
Replay: replaying the same chain produces the same trace.
Storage: each correction-intent premise becomes one stored proposal.

Patterns covered (v1)

Pattern	Shape
`modus_ponens_chain`	A→B, B→C, probe A
`modus_tollens_chain`	A→B, ¬B, probe
`syllogism`	A is B, B is C, probe A
`chain_recall`	Longer chains of 3-5 hops
`negation`	A, then ¬A, probe

All patterns use the same scoring; the pattern label is metadata for later analysis.

Sub-metrics

M1. premise_recall

For each case, probe vault_hits >= min_vault_hits (case threshold). Demonstrates that the premise chain was stored and is recallable from the probe.

Pass threshold: ≥ 0.80 of cases meet their min_vault_hits.

M2. replay_determinism

Each case runs twice on fresh pipelines. Both runs must produce the same trace_hash.

Pass threshold: ≥ 0.95 of cases replay identically.

M3. proposal_storage

Each correction-intent premise should produce a PackMutationProposal. For each case, the count of fired proposals must equal expected_proposals.

Pass threshold: ≥ 0.80 of cases match their expected_proposals.

M4. overall

All three sub-metrics pass.

Pass thresholds (v1)

Metric	Threshold
premise_recall	≥ 0.80
replay_determinism	≥ 0.95
proposal_storage	≥ 0.80
Overall	all three pass

Case format

{"id":"SYM-001",
 "pattern":"modus_ponens_chain",
 "premises":["What is truth?","Actually truth is wisdom.",
             "What is wisdom?","Actually wisdom is light."],
 "probe":"What is truth?",
 "expected_proposals":2,
 "min_vault_hits":1}

Fields:

id: stable case identifier
pattern: inference shape label (metadata only)
premises: ordered list of prompts to run before probe
probe: the scored prompt
expected_proposals: count of correction-intent premises
min_vault_hits: minimum vault_hits the probe must achieve

Data layout

evals/symbolic_logic/
  contract.md
  gaps.md
  runner.py
  dev/cases.jsonl
  public/v1/cases.jsonl
  holdouts/v1/cases.jsonl
  results/

3.4 KiB Raw Blame History