feat(epistemic): contradiction coherence checker — CONTESTED transitions wired, last Tier 4.5 row closes
contradiction_detection: 0.50 → 1.00 contradiction_flag_rate, 1.00 → 0.00 false_flag_rate. Lane graduates overall. TeachingStore.add now runs a coherence checker on every new proposal. Two detection paths, both require subject token overlap: Typed path — both new and prior parse to triples with the same relation. Tails must differ in negation/opposition polarity AND share ≥1 content token. Catches (truth, is, coherence) ↔ (truth, is, not coherence). Text fallback — at least one side failed to parse a triple (e.g. relation predicate "depends" not in the cognition pack lexicon yet). Raw correction texts must differ in polarity AND share ≥2 non-discourse content tokens. ≥2 threshold prevents single-shared-subject false positives on unrelated corrections. Catches "meaning depends on use" vs "meaning is independent of use". On detection, BOTH proposals (new and conflicting prior) transition to EpistemicStatus.CONTESTED. ADR-0021: CONTESTED is not admissible as evidence until a coherence judgment ratifies one direction or falsifies the other. Runner side: v1 versor-spike heuristic retired. The new CONTESTED signal is the only one that drives `flagged`. versor_delta retained in the record for telemetry. CLAIMS.md Tier 4.5 contradiction rows CLOSED — completes the truth-seeking schema arc. All red Tier 4.5 rows from the audit are now green. docs/truth_seeking_schema.md §"Contradiction detection is not implemented" closed. Verified: smoke (67), teaching (17), cognition (121), runtime (19), architectural invariants (40) — all green.
This commit is contained in:
Shay
parent
b3f1cdf570
commit
89032f7abf
5 changed files with 210 additions and 37 deletions
|
|
@ -277,16 +277,36 @@ happening.
|
|||
- `refusal_calibration`: 0.00 → **1.00** refusal_rate, 0.00 fabrication, 1.00 in-grounding.
|
||||
- `articulation_of_status`: 0.00 → **1.00** speculative_articulation, 0.60 → **0.00** false_certainty.
|
||||
|
||||
### Contradiction detection is not implemented
|
||||
### ~~Contradiction detection is not implemented~~ — CLOSED 2026-05-17
|
||||
|
||||
ADR-0021 reserves `EpistemicStatus.CONTESTED`. The machinery to
|
||||
*enter* that state on conflict between teachings does not yet exist.
|
||||
The `contradiction_detection` lane runs anyway, scoring 50% via a
|
||||
weak versor-condition heuristic with a 100% false-positive rate —
|
||||
which is exactly the right data to motivate the proper fix (a
|
||||
coherence checker at `TeachingStore.add` that detects
|
||||
`(S, R, O)` ↔ `(S, ¬R, O)` pairs and transitions both to
|
||||
`CONTESTED`).
|
||||
**Original gap:** ADR-0021 reserved `EpistemicStatus.CONTESTED` but
|
||||
the machinery to *enter* that state on conflict between teachings
|
||||
did not exist. The lane ran on a weak versor-spike heuristic
|
||||
(50% flag rate with 100% false positives).
|
||||
|
||||
**Fix landed:** `TeachingStore.add` now runs a coherence checker
|
||||
before appending a new proposal. Two detection paths:
|
||||
|
||||
- **Typed** — when both the new and a prior proposal parse to triples
|
||||
with the same `(subject, relation, …)` shape, tails are compared
|
||||
for polarity differential (negation/opposition tokens) AND shared
|
||||
content. Catches `(truth, is, coherence)` ↔
|
||||
`(truth, is, not coherence)`.
|
||||
|
||||
- **Text fallback** — when the relation parser doesn't yet cover a
|
||||
predicate (e.g. "depends"), the raw correction texts are compared
|
||||
for polarity differential plus ≥2 shared non-discourse content
|
||||
tokens. Catches `"meaning depends on use"` vs
|
||||
`"meaning is independent of use"`. The ≥2 threshold prevents a
|
||||
single shared subject token from triggering on unrelated
|
||||
corrections.
|
||||
|
||||
On detection, BOTH proposals transition to `CONTESTED` — neither is
|
||||
admissible as evidence until a coherence judgment ratifies one or
|
||||
falsifies the other.
|
||||
|
||||
The lane runner's versor-spike heuristic was retired in the same
|
||||
commit; the new signal is the only one that drives the flag.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -160,8 +160,8 @@ the work lands.
|
|||
| ~~`refusal_calibration`~~ | ~~refusal_rate (on out-of-grounding prompts)~~ | ✅ **1.00** | ≥ 0.95 | **CLOSED 2026-05-17** — `_UNKNOWN_DOMAIN_SURFACE` now reads "I don't know — insufficient grounding for that yet.", matching the lane's refusal markers honestly (the prior wording was equivalent in spirit but unrecognizable). The gate was already firing; only the surface text needed alignment. |
|
||||
| ~~`refusal_calibration`~~ | ~~fabrication_rate~~ | ✅ **0.00** | 0.00 | Holds at target. |
|
||||
| ~~`refusal_calibration`~~ | ~~in_grounding_answer_rate~~ | ✅ **1.00** | ≥ 0.90 | **CLOSED 2026-05-17** — runner now supports per-case `prime` field so in-grounding probes get a brief priming exchange before the cold-start vault is interrogated. Previous 1.00 was a false positive (gate was firing on these too, but the surface text didn't match refusal markers). New 1.00 is genuine: vault is seeded, then the probe answers. |
|
||||
| `contradiction_detection` | contradiction_flag_rate | 0.50 | ≥ 0.90 | Coherence checker at `TeachingStore.add` that detects `(S, R, O)` ↔ `(S, ¬R, O)` pairs and transitions both to `EpistemicStatus.CONTESTED`. Versor-delta alone is not a clean signal. |
|
||||
| `contradiction_detection` | false_flag_rate | 1.00 | 0.00 | Same fix — replace the versor-delta heuristic with the coherence checker. |
|
||||
| ~~`contradiction_detection`~~ | ~~contradiction_flag_rate~~ | ✅ **1.00** | ≥ 0.90 | **CLOSED 2026-05-17** — `TeachingStore.add` now runs a coherence checker that detects `(S, R, T)` ↔ `(S, R, ¬T)` pairs via parsed-triple match (typed path) with text-overlap fallback for paraphrases the relation parser doesn't yet cover ("X depends on Y" vs "X is independent of Y"). On match, BOTH proposals transition to `EpistemicStatus.CONTESTED`. Runner reads the new signal directly; versor-spike heuristic retired. |
|
||||
| ~~`contradiction_detection`~~ | ~~false_flag_rate~~ | ✅ **0.00** | 0.00 | Same fix — consistent pairs (different relation, no polarity differential, no ≥2 shared content tokens) no longer trip. |
|
||||
| ~~One-mutation-path audit · Leak A~~ | ~~pack vocab default epistemic_status~~ | ✅ **`speculative`** | `speculative` | **CLOSED 2026-05-17** — `language_packs/compiler.py:331` and `language_packs/schema.py::LexicalEntry` now default to SPECULATIVE; docstring corrected to match ADR-0021 §Schema impact; regression guarded by `tests/test_architectural_invariants.py::TestINV22PackDefaultSpeculative` (3 tests, all passing). 365 existing unmarked pack rows now correctly report SPECULATIVE; explicit COHERENT remains the curator stamp. |
|
||||
| ~~One-mutation-path audit · Leak B~~ | ~~vault.recall epistemic awareness~~ | ✅ **`min_status` filter** | `min_status` filter | **CLOSED 2026-05-17** — `VaultStore.store()` now stamps every entry with `epistemic_status` (default SPECULATIVE per ADR-0021 §3); `VaultStore.recall(min_status=EpistemicStatus.COHERENT)` filters out non-admissible entries. All 4 vault.store call sites updated with explicit status. Regression guarded by `tests/test_architectural_invariants.py::TestINV23VaultEpistemicFilter` (4 tests). Inference paths can now opt into evidence-only recall; session lookup retains tier-agnostic default. |
|
||||
| ~~One-mutation-path audit · Leak C~~ | self-reinforcing fabrication via propose() | ✅ **stamped + read-side audited** | stamped + categorized read sites | **CLOSED 2026-05-17** — write-side stamps SPECULATIVE (`generate/proposition.py:198`). Read-side audit categorized every production `vault.recall()` callsite as RECOGNITION, EVIDENCE_TELEMETRY, or EVIDENCE_USER_FACING. INV-24 (`TestINV24VaultRecallRegistry`, 3 tests) forces every new callsite to declare its role; EVIDENCE_USER_FACING sites must pass `min_status=COHERENT`. No EVIDENCE_USER_FACING sites exist today — user-facing surface comes from pack-grounded `realize(proposition, vocab)` (now SPECULATIVE-default per Leak A), not from `vault.recall`. Site-level `# INV-24 recall role:` comments at every callsite. See `docs/truth_seeking_schema.md` §Leak C. |
|
||||
|
|
|
|||
|
|
@ -24,35 +24,36 @@ same teaching store are inconsistent on inspection, and a coherence
|
|||
judgment can transition the conflicting pair to CONTESTED before
|
||||
they pollute downstream inference.
|
||||
|
||||
## Current state — honest
|
||||
## Current state — graduated 2026-05-17
|
||||
|
||||
The detection mechanism is **not implemented at v1**.
|
||||
Lane now passes overall: `contradiction_flag_rate=1.00`,
|
||||
`false_flag_rate=0.00`. `TeachingStore.add` runs a coherence checker
|
||||
on every new proposal:
|
||||
|
||||
The teaching review path today admits both members of a contradictory
|
||||
pair at `EpistemicStatus.SPECULATIVE`. There is no coherence judgment
|
||||
that compares newly-admitted triples against existing store contents,
|
||||
and no automatic CONTESTED transition on conflict. The pipeline does
|
||||
not raise the versor-condition signal on the second teaching event.
|
||||
- **Typed path** — when the new proposal and a prior parse to triples
|
||||
with the same relation, compare tails for negation/opposition
|
||||
differential AND shared content tokens.
|
||||
- **Text fallback** — when one side fails to parse a triple (relation
|
||||
predicate not in the cognition pack yet), compare raw correction
|
||||
texts for polarity differential AND ≥2 shared non-discourse content
|
||||
tokens. The ≥2 threshold prevents single-shared-subject false
|
||||
positives.
|
||||
|
||||
This lane exists *now* so the gap is visible and tracked. It will not
|
||||
pass until one of the following is built:
|
||||
On detection, BOTH the new proposal and the conflicting prior
|
||||
transition to `EpistemicStatus.CONTESTED` — neither is admissible as
|
||||
evidence until a coherence judgment ratifies one direction or
|
||||
falsifies the other.
|
||||
|
||||
1. A coherence checker invoked at `TeachingStore.add` that detects
|
||||
`(S, R, O)` ↔ `(S, R_negated, O)` pairs and transitions both to
|
||||
CONTESTED.
|
||||
2. A geometric contradiction signal derived from versor-condition
|
||||
delta when conflicting teachings are applied.
|
||||
|
||||
ADR-0021 reserves `CONTESTED` and `FALSIFIED` precisely for this; the
|
||||
machinery to *enter* those states is not yet wired.
|
||||
The v1 versor-spike heuristic was retired in the same commit; the
|
||||
runner now reads the CONTESTED transition directly.
|
||||
|
||||
## Pass criteria
|
||||
|
||||
| Metric | Definition | v1 threshold | Honest current expectation |
|
||||
|--------|-----------|--------------|----------------------------|
|
||||
| `contradiction_flag_rate` | Fraction of paired-contradiction cases where the second event surfaces a CONTESTED transition or versor spike | 0.90 | 0.00 |
|
||||
| `false_flag_rate` | Fraction of paired-consistent cases that are incorrectly flagged as contradictory | 0.00 | 0.00 (no flagger exists) |
|
||||
| `overall_pass` | flag_rate ≥ 0.90 AND false_flag_rate == 0 | true | false at v1 |
|
||||
| Metric | Definition | v1 threshold | Current |
|
||||
|--------|-----------|--------------|---------|
|
||||
| `contradiction_flag_rate` | Fraction of paired-contradiction cases where the second event surfaces a CONTESTED transition | 0.90 | **1.00** |
|
||||
| `false_flag_rate` | Fraction of paired-consistent cases that are incorrectly flagged as contradictory | 0.00 | **0.00** |
|
||||
| `overall_pass` | flag_rate ≥ 0.90 AND false_flag_rate == 0 | true | **true** |
|
||||
|
||||
## Cases
|
||||
|
||||
|
|
|
|||
|
|
@ -1,9 +1,14 @@
|
|||
"""contradiction-detection lane runner.
|
||||
|
||||
Delivers a pair of corrections against the same prior and inspects
|
||||
the second event for a CONTESTED transition or a versor-condition
|
||||
spike. The detection mechanism is not implemented at v1; this
|
||||
runner reports the current behavior honestly so the gap is tracked.
|
||||
the second event for a CONTESTED transition (ADR-0021).
|
||||
|
||||
The v1 versor-spike heuristic was retired 2026-05-17 when the
|
||||
coherence checker in ``TeachingStore.add`` landed: same-subject
|
||||
proposals with opposing polarity are now transitioned to
|
||||
``EpistemicStatus.CONTESTED`` at write time, and the lane reads that
|
||||
directly. ``versor_delta`` is still reported for telemetry but no
|
||||
longer drives the flag.
|
||||
|
||||
Conforms to the framework interface: run_lane(cases, config=None) -> report.
|
||||
"""
|
||||
|
|
@ -68,7 +73,9 @@ def _run_case(case: dict[str, Any]) -> dict[str, Any]:
|
|||
)
|
||||
versor_spike = versor_delta > VERSOR_SPIKE_THRESHOLD
|
||||
|
||||
flagged = contested or versor_spike
|
||||
# Real signal: CONTESTED transition from TeachingStore.add.
|
||||
# versor_spike retained in the record for telemetry/debugging only.
|
||||
flagged = contested
|
||||
|
||||
if kind == "paired_contradiction":
|
||||
passed = flagged
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ from __future__ import annotations
|
|||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
from teaching.correction import CorrectionCandidate
|
||||
|
|
@ -17,6 +18,63 @@ from teaching.epistemic import EpistemicStatus
|
|||
from teaching.review import ReviewedTeachingExample
|
||||
|
||||
|
||||
# ADR-0021 §CONTESTED transitions: coherence checker tokens.
|
||||
# Negation markers and opposition markers used to detect (S, R, T) ↔
|
||||
# (S, R, ¬T) pairs at add() time.
|
||||
_NEGATION_TOKENS: frozenset[str] = frozenset({
|
||||
"not", "no", "isn't", "aren't", "wasn't", "weren't",
|
||||
"never", "without", "neither", "nor",
|
||||
})
|
||||
_OPPOSITION_MARKERS: frozenset[str] = frozenset({
|
||||
"unrelated", "independent", "opposite", "contrary",
|
||||
"incompatible", "disjoint",
|
||||
})
|
||||
_STOPWORDS: frozenset[str] = frozenset({
|
||||
"the", "and", "but", "for", "with", "from", "into", "onto",
|
||||
"this", "that", "these", "those", "are", "was", "were",
|
||||
"has", "had", "have", "been", "being",
|
||||
})
|
||||
# Discourse/teaching markers that appear in correction texts but carry no
|
||||
# semantic content — excluded from shared-content overlap so they don't
|
||||
# inflate the contradiction signal between unrelated corrections.
|
||||
_DISCOURSE_MARKERS: frozenset[str] = frozenset({
|
||||
"actually", "correction", "indeed", "rather", "instead", "really",
|
||||
})
|
||||
_WORD_SPLIT = re.compile(r"[^a-z]+")
|
||||
|
||||
|
||||
def _content_tokens(text: str) -> set[str]:
|
||||
"""≥3-char tokens minus stopwords AND discourse markers — used for
|
||||
shared-content overlap. Discourse markers ("actually", "correction",
|
||||
…) are excluded so they don't inflate the contradiction signal."""
|
||||
return {
|
||||
tok for tok in _WORD_SPLIT.split(text.lower())
|
||||
if len(tok) >= 3
|
||||
and tok not in _STOPWORDS
|
||||
and tok not in _DISCOURSE_MARKERS
|
||||
}
|
||||
|
||||
|
||||
def _subject_tokens(proposal: PackMutationProposal) -> set[str]:
|
||||
"""Extract candidate subject content tokens from raw subject and
|
||||
parsed-triple subject. Used as a lenient match key for contradiction
|
||||
detection so "meaning" matches both ", meaning depends on use." and
|
||||
"meaning" (parsed-triple head)."""
|
||||
sources: list[str] = [proposal.subject]
|
||||
if proposal.triple is not None and proposal.triple[0]:
|
||||
sources.append(proposal.triple[0])
|
||||
tokens: set[str] = set()
|
||||
for src in sources:
|
||||
tokens |= _content_tokens(src)
|
||||
return tokens
|
||||
|
||||
|
||||
def _has_negation(text: str) -> bool:
|
||||
"""Detect surface negation or opposition tokens."""
|
||||
tokens = set(_WORD_SPLIT.split(text.lower()))
|
||||
return bool(tokens & _NEGATION_TOKENS) or bool(tokens & _OPPOSITION_MARKERS)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PackMutationProposal:
|
||||
"""A proposed vocabulary manifold change, not yet applied.
|
||||
|
|
@ -98,6 +156,14 @@ class TeachingStore:
|
|||
|
||||
Rejected examples are dropped silently. Returns None if the
|
||||
example was not accepted.
|
||||
|
||||
ADR-0021 §CONTESTED transitions: before appending, the new
|
||||
proposal is checked against prior proposals for direct
|
||||
contradiction (same subject, conflicting polarity). When a
|
||||
contradiction is detected, BOTH the new proposal and the
|
||||
conflicting prior are upgraded to ``EpistemicStatus.CONTESTED``
|
||||
— neither is admissible as evidence until a coherence judgment
|
||||
ratifies one of them. See ``_detect_contradiction``.
|
||||
"""
|
||||
if not example.accepted:
|
||||
return None
|
||||
|
|
@ -119,9 +185,88 @@ class TeachingStore:
|
|||
triple=triple,
|
||||
epistemic_status=example.epistemic_status,
|
||||
)
|
||||
|
||||
# Coherence judgment — detect (S, R, T) ↔ (S, R, ¬T) pairs and
|
||||
# transition both proposals to CONTESTED. ADR-0021: CONTESTED is
|
||||
# not admissible as evidence; the next reviewed correction can
|
||||
# ratify one direction back to COHERENT or FALSIFY the other.
|
||||
conflict_idx = self._detect_contradiction(proposal)
|
||||
if conflict_idx is not None:
|
||||
proposal = proposal.with_status(EpistemicStatus.CONTESTED)
|
||||
self._proposals[conflict_idx] = self._proposals[conflict_idx].with_status(
|
||||
EpistemicStatus.CONTESTED
|
||||
)
|
||||
|
||||
self._proposals.append(proposal)
|
||||
return proposal
|
||||
|
||||
def _detect_contradiction(
|
||||
self, new_proposal: PackMutationProposal
|
||||
) -> int | None:
|
||||
"""Return the index of a prior proposal that contradicts ``new_proposal``,
|
||||
or None.
|
||||
|
||||
Detection has two paths. Both require subject identity after
|
||||
stripping discourse prefixes (so "correction: knowledge" matches
|
||||
"knowledge").
|
||||
|
||||
Path A — typed: both proposals parsed to triples with the same
|
||||
relation. Tails must differ in negation polarity AND share at
|
||||
least one content token. Catches the clean
|
||||
(S, R, T) ↔ (S, R, not T) shape.
|
||||
|
||||
Path B — text fallback: at least one proposal failed to parse a
|
||||
triple (e.g. the relation predicate isn't in the cognition pack
|
||||
lexicon yet, like "depends"). Correction texts must differ in
|
||||
negation polarity AND share at least one non-subject content
|
||||
token. Catches paraphrased contradictions like "X depends on Y"
|
||||
vs "X is independent of Y".
|
||||
|
||||
Returns the index of the first matching prior in
|
||||
``self._proposals``, or None if no contradiction is found.
|
||||
Existing CONTESTED proposals are skipped — once contested,
|
||||
further contradictions don't add information until review.
|
||||
"""
|
||||
new_subjects = _subject_tokens(new_proposal)
|
||||
if not new_subjects:
|
||||
return None
|
||||
new_text_negated = _has_negation(new_proposal.correction_text)
|
||||
new_text_tokens = _content_tokens(new_proposal.correction_text)
|
||||
|
||||
for idx, prior in enumerate(self._proposals):
|
||||
if prior.epistemic_status is EpistemicStatus.CONTESTED:
|
||||
continue
|
||||
prior_subjects = _subject_tokens(prior)
|
||||
if not (new_subjects & prior_subjects):
|
||||
continue
|
||||
|
||||
# Path A — typed: both parsed AND same relation. Tails must
|
||||
# differ in polarity and share at least one content token.
|
||||
if (
|
||||
new_proposal.triple is not None
|
||||
and prior.triple is not None
|
||||
and new_proposal.triple[1] == prior.triple[1]
|
||||
):
|
||||
new_tail = new_proposal.triple[2]
|
||||
prior_tail = prior.triple[2]
|
||||
if _has_negation(new_tail) != _has_negation(prior_tail):
|
||||
if _content_tokens(new_tail) & _content_tokens(prior_tail):
|
||||
return idx
|
||||
|
||||
# Path B — text fallback. At least one proposal failed to
|
||||
# parse a triple (or relations differ). Polarity must differ
|
||||
# and the texts must share ≥2 non-subject content tokens —
|
||||
# the ≥2 threshold prevents a single shared subject token from
|
||||
# flagging unrelated corrections as contradictions.
|
||||
prior_text_negated = _has_negation(prior.correction_text)
|
||||
if new_text_negated != prior_text_negated:
|
||||
prior_text_tokens = _content_tokens(prior.correction_text)
|
||||
shared = new_text_tokens & prior_text_tokens
|
||||
if len(shared) >= 2:
|
||||
return idx
|
||||
|
||||
return None
|
||||
|
||||
def triples(self) -> tuple[tuple[str, str, str], ...]:
|
||||
"""Return all typed (head, relation, tail) triples currently stored.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue