89032f7abf
contradiction_detection: 0.50 → 1.00 contradiction_flag_rate, 1.00 → 0.00 false_flag_rate. Lane graduates overall. TeachingStore.add now runs a coherence checker on every new proposal. Two detection paths, both require subject token overlap: Typed path — both new and prior parse to triples with the same relation. Tails must differ in negation/opposition polarity AND share ≥1 content token. Catches (truth, is, coherence) ↔ (truth, is, not coherence). Text fallback — at least one side failed to parse a triple (e.g. relation predicate "depends" not in the cognition pack lexicon yet). Raw correction texts must differ in polarity AND share ≥2 non-discourse content tokens. ≥2 threshold prevents single-shared-subject false positives on unrelated corrections. Catches "meaning depends on use" vs "meaning is independent of use". On detection, BOTH proposals (new and conflicting prior) transition to EpistemicStatus.CONTESTED. ADR-0021: CONTESTED is not admissible as evidence until a coherence judgment ratifies one direction or falsifies the other. Runner side: v1 versor-spike heuristic retired. The new CONTESTED signal is the only one that drives `flagged`. versor_delta retained in the record for telemetry. CLAIMS.md Tier 4.5 contradiction rows CLOSED — completes the truth-seeking schema arc. All red Tier 4.5 rows from the audit are now green. docs/truth_seeking_schema.md §"Contradiction detection is not implemented" closed. Verified: smoke (67), teaching (17), cognition (121), runtime (19), architectural invariants (40) — all green.
3 KiB
contradiction-detection eval lane
What it measures
When CORE is taught two corrections that directly contradict each
other (e.g. "truth is coherence" then "truth is not coherence"),
the second event should be flagged as conflicting with the first —
either via a coherence judgment that transitions one or both to
EpistemicStatus.CONTESTED, or via a versor-condition spike that
makes the conflict structurally visible.
Why it matters (structural win, eventually)
Frontier LLMs absorb contradictory training data silently. A fine-tuning corpus that says "X is Y" in one shard and "X is not Y" in another produces a model whose answer depends on sampling, not on which claim is correct. There is no mechanism to surface the disagreement to a reviewer.
CORE's stored-relation substrate and exact CGA recall make
contradiction a detectable event in principle: two triples
(truth, IS, coherence) and (truth, IS_NOT, coherence) in the
same teaching store are inconsistent on inspection, and a coherence
judgment can transition the conflicting pair to CONTESTED before
they pollute downstream inference.
Current state — graduated 2026-05-17
Lane now passes overall: contradiction_flag_rate=1.00,
false_flag_rate=0.00. TeachingStore.add runs a coherence checker
on every new proposal:
- Typed path — when the new proposal and a prior parse to triples with the same relation, compare tails for negation/opposition differential AND shared content tokens.
- Text fallback — when one side fails to parse a triple (relation predicate not in the cognition pack yet), compare raw correction texts for polarity differential AND ≥2 shared non-discourse content tokens. The ≥2 threshold prevents single-shared-subject false positives.
On detection, BOTH the new proposal and the conflicting prior
transition to EpistemicStatus.CONTESTED — neither is admissible as
evidence until a coherence judgment ratifies one direction or
falsifies the other.
The v1 versor-spike heuristic was retired in the same commit; the runner now reads the CONTESTED transition directly.
Pass criteria
| Metric | Definition | v1 threshold | Current |
|---|---|---|---|
contradiction_flag_rate |
Fraction of paired-contradiction cases where the second event surfaces a CONTESTED transition | 0.90 | 1.00 |
false_flag_rate |
Fraction of paired-consistent cases that are incorrectly flagged as contradictory | 0.00 | 0.00 |
overall_pass |
flag_rate ≥ 0.90 AND false_flag_rate == 0 | true | true |
Cases
Each case is a pair of corrections delivered sequentially against
the same prior. The lane runner applies them in order and inspects
the second event's pack_mutation_proposal.epistemic_status and the
versor-condition delta between the two events.
- Paired contradiction — corrections that negate each other.
- Paired consistent — control pairs that elaborate without contradicting, to keep the flagger honest if/when one is built.
Runner
runner.py in this directory.