core/evals/symbolic_logic/contract.md

symbolic-logic eval lane

What it measures

CORE's foundation for proposition-based inference: when premises are taught via the correction loop, the resulting field state and vault content carry the chain in a deterministic, premise-sensitive, recallable form.

v1 measures three structural foundations on which a future inference engine would be built; it does not test that CORE applies named inference rules (modus ponens, modus tollens, syllogism) directly. See gaps.md for the architectural finding and roadmap toward v2.

Why it matters (structural win)

Frontier LLMs produce inference-like text but provide no first-class evidence that the chain was actually stored, replayed deterministically, or that the produced trace depends on the specific premises given. Their inference is a single forward pass over a stochastic policy.

CORE returns:

• CognitiveTurnResult.vault_hits — how many premises the probe recalled.
• CognitiveTurnResult.trace_hash — a SHA-256 over deterministic pipeline state.
• CognitiveTurnResult.pack_mutation_proposal — datestamped record for each correction-intent turn.

These let a downstream caller verify three properties of any premise-chain inference:

1. Recall: the probe can see the chain (vault_hits > 0).
2. Replay: replaying the same chain produces the same trace.
3. Storage: each correction-intent premise becomes one stored proposal.

Patterns covered (v1)

Pattern	Shape
modus_ponens_chain	A→B, B→C, probe A
modus_tollens_chain	A→B, ¬B, probe
syllogism	A is B, B is C, probe A
chain_recall	Longer chains of 3-5 hops
negation	A, then ¬A, probe

All patterns use the same scoring; the pattern label is metadata for later analysis.

Sub-metrics

M1. premise_recall

For each case, probe vault_hits >= min_vault_hits (case threshold). Demonstrates that the premise chain was stored and is recallable from the probe.

Pass threshold: ≥ 0.80 of cases meet their min_vault_hits.

M2. replay_determinism

Each case runs twice on fresh pipelines. Both runs must produce the same trace_hash.

Pass threshold: ≥ 0.95 of cases replay identically.

M3. proposal_storage

Each correction-intent premise should produce a PackMutationProposal. For each case, the count of fired proposals must equal expected_proposals.

Pass threshold: ≥ 0.80 of cases match their expected_proposals.

M4. overall

All three sub-metrics pass.

Pass thresholds (v1)

Metric	Threshold
premise_recall	≥ 0.80
replay_determinism	≥ 0.95
proposal_storage	≥ 0.80
Overall	all three pass

Case format

{"id":"SYM-001",
 "pattern":"modus_ponens_chain",
 "premises":["What is truth?","Actually truth is wisdom.",
             "What is wisdom?","Actually wisdom is light."],
 "probe":"What is truth?",
 "expected_proposals":2,
 "min_vault_hits":1}

Fields:

• id: stable case identifier
• pattern: inference shape label (metadata only)
• premises: ordered list of prompts to run before probe
• probe: the scored prompt
• expected_proposals: count of correction-intent premises
• min_vault_hits: minimum vault_hits the probe must achieve

Data layout

evals/symbolic_logic/
  contract.md
  gaps.md
  runner.py
  dev/cases.jsonl
  public/v1/cases.jsonl
  holdouts/v1/cases.jsonl
  results/