diff --git a/docs/truth_seeking_schema.md b/docs/truth_seeking_schema.md index a5be237b..48a7990e 100644 --- a/docs/truth_seeking_schema.md +++ b/docs/truth_seeking_schema.md @@ -277,16 +277,36 @@ happening. - `refusal_calibration`: 0.00 → **1.00** refusal_rate, 0.00 fabrication, 1.00 in-grounding. - `articulation_of_status`: 0.00 → **1.00** speculative_articulation, 0.60 → **0.00** false_certainty. -### Contradiction detection is not implemented +### ~~Contradiction detection is not implemented~~ — CLOSED 2026-05-17 -ADR-0021 reserves `EpistemicStatus.CONTESTED`. The machinery to -*enter* that state on conflict between teachings does not yet exist. -The `contradiction_detection` lane runs anyway, scoring 50% via a -weak versor-condition heuristic with a 100% false-positive rate — -which is exactly the right data to motivate the proper fix (a -coherence checker at `TeachingStore.add` that detects -`(S, R, O)` ↔ `(S, ¬R, O)` pairs and transitions both to -`CONTESTED`). +**Original gap:** ADR-0021 reserved `EpistemicStatus.CONTESTED` but +the machinery to *enter* that state on conflict between teachings +did not exist. The lane ran on a weak versor-spike heuristic +(50% flag rate with 100% false positives). + +**Fix landed:** `TeachingStore.add` now runs a coherence checker +before appending a new proposal. Two detection paths: + +- **Typed** — when both the new and a prior proposal parse to triples + with the same `(subject, relation, …)` shape, tails are compared + for polarity differential (negation/opposition tokens) AND shared + content. Catches `(truth, is, coherence)` ↔ + `(truth, is, not coherence)`. + +- **Text fallback** — when the relation parser doesn't yet cover a + predicate (e.g. "depends"), the raw correction texts are compared + for polarity differential plus ≥2 shared non-discourse content + tokens. Catches `"meaning depends on use"` vs + `"meaning is independent of use"`. The ≥2 threshold prevents a + single shared subject token from triggering on unrelated + corrections. + +On detection, BOTH proposals transition to `CONTESTED` — neither is +admissible as evidence until a coherence judgment ratifies one or +falsifies the other. + +The lane runner's versor-spike heuristic was retired in the same +commit; the new signal is the only one that drives the flag. --- diff --git a/evals/CLAIMS.md b/evals/CLAIMS.md index ded6ebdf..ed2fe6ec 100644 --- a/evals/CLAIMS.md +++ b/evals/CLAIMS.md @@ -160,8 +160,8 @@ the work lands. | ~~`refusal_calibration`~~ | ~~refusal_rate (on out-of-grounding prompts)~~ | ✅ **1.00** | ≥ 0.95 | **CLOSED 2026-05-17** — `_UNKNOWN_DOMAIN_SURFACE` now reads "I don't know — insufficient grounding for that yet.", matching the lane's refusal markers honestly (the prior wording was equivalent in spirit but unrecognizable). The gate was already firing; only the surface text needed alignment. | | ~~`refusal_calibration`~~ | ~~fabrication_rate~~ | ✅ **0.00** | 0.00 | Holds at target. | | ~~`refusal_calibration`~~ | ~~in_grounding_answer_rate~~ | ✅ **1.00** | ≥ 0.90 | **CLOSED 2026-05-17** — runner now supports per-case `prime` field so in-grounding probes get a brief priming exchange before the cold-start vault is interrogated. Previous 1.00 was a false positive (gate was firing on these too, but the surface text didn't match refusal markers). New 1.00 is genuine: vault is seeded, then the probe answers. | -| `contradiction_detection` | contradiction_flag_rate | 0.50 | ≥ 0.90 | Coherence checker at `TeachingStore.add` that detects `(S, R, O)` ↔ `(S, ¬R, O)` pairs and transitions both to `EpistemicStatus.CONTESTED`. Versor-delta alone is not a clean signal. | -| `contradiction_detection` | false_flag_rate | 1.00 | 0.00 | Same fix — replace the versor-delta heuristic with the coherence checker. | +| ~~`contradiction_detection`~~ | ~~contradiction_flag_rate~~ | ✅ **1.00** | ≥ 0.90 | **CLOSED 2026-05-17** — `TeachingStore.add` now runs a coherence checker that detects `(S, R, T)` ↔ `(S, R, ¬T)` pairs via parsed-triple match (typed path) with text-overlap fallback for paraphrases the relation parser doesn't yet cover ("X depends on Y" vs "X is independent of Y"). On match, BOTH proposals transition to `EpistemicStatus.CONTESTED`. Runner reads the new signal directly; versor-spike heuristic retired. | +| ~~`contradiction_detection`~~ | ~~false_flag_rate~~ | ✅ **0.00** | 0.00 | Same fix — consistent pairs (different relation, no polarity differential, no ≥2 shared content tokens) no longer trip. | | ~~One-mutation-path audit · Leak A~~ | ~~pack vocab default epistemic_status~~ | ✅ **`speculative`** | `speculative` | **CLOSED 2026-05-17** — `language_packs/compiler.py:331` and `language_packs/schema.py::LexicalEntry` now default to SPECULATIVE; docstring corrected to match ADR-0021 §Schema impact; regression guarded by `tests/test_architectural_invariants.py::TestINV22PackDefaultSpeculative` (3 tests, all passing). 365 existing unmarked pack rows now correctly report SPECULATIVE; explicit COHERENT remains the curator stamp. | | ~~One-mutation-path audit · Leak B~~ | ~~vault.recall epistemic awareness~~ | ✅ **`min_status` filter** | `min_status` filter | **CLOSED 2026-05-17** — `VaultStore.store()` now stamps every entry with `epistemic_status` (default SPECULATIVE per ADR-0021 §3); `VaultStore.recall(min_status=EpistemicStatus.COHERENT)` filters out non-admissible entries. All 4 vault.store call sites updated with explicit status. Regression guarded by `tests/test_architectural_invariants.py::TestINV23VaultEpistemicFilter` (4 tests). Inference paths can now opt into evidence-only recall; session lookup retains tier-agnostic default. | | ~~One-mutation-path audit · Leak C~~ | self-reinforcing fabrication via propose() | ✅ **stamped + read-side audited** | stamped + categorized read sites | **CLOSED 2026-05-17** — write-side stamps SPECULATIVE (`generate/proposition.py:198`). Read-side audit categorized every production `vault.recall()` callsite as RECOGNITION, EVIDENCE_TELEMETRY, or EVIDENCE_USER_FACING. INV-24 (`TestINV24VaultRecallRegistry`, 3 tests) forces every new callsite to declare its role; EVIDENCE_USER_FACING sites must pass `min_status=COHERENT`. No EVIDENCE_USER_FACING sites exist today — user-facing surface comes from pack-grounded `realize(proposition, vocab)` (now SPECULATIVE-default per Leak A), not from `vault.recall`. Site-level `# INV-24 recall role:` comments at every callsite. See `docs/truth_seeking_schema.md` §Leak C. | diff --git a/evals/contradiction_detection/contract.md b/evals/contradiction_detection/contract.md index 0f2a3bee..97a83ae3 100644 --- a/evals/contradiction_detection/contract.md +++ b/evals/contradiction_detection/contract.md @@ -24,35 +24,36 @@ same teaching store are inconsistent on inspection, and a coherence judgment can transition the conflicting pair to CONTESTED before they pollute downstream inference. -## Current state — honest +## Current state — graduated 2026-05-17 -The detection mechanism is **not implemented at v1**. +Lane now passes overall: `contradiction_flag_rate=1.00`, +`false_flag_rate=0.00`. `TeachingStore.add` runs a coherence checker +on every new proposal: -The teaching review path today admits both members of a contradictory -pair at `EpistemicStatus.SPECULATIVE`. There is no coherence judgment -that compares newly-admitted triples against existing store contents, -and no automatic CONTESTED transition on conflict. The pipeline does -not raise the versor-condition signal on the second teaching event. +- **Typed path** — when the new proposal and a prior parse to triples + with the same relation, compare tails for negation/opposition + differential AND shared content tokens. +- **Text fallback** — when one side fails to parse a triple (relation + predicate not in the cognition pack yet), compare raw correction + texts for polarity differential AND ≥2 shared non-discourse content + tokens. The ≥2 threshold prevents single-shared-subject false + positives. -This lane exists *now* so the gap is visible and tracked. It will not -pass until one of the following is built: +On detection, BOTH the new proposal and the conflicting prior +transition to `EpistemicStatus.CONTESTED` — neither is admissible as +evidence until a coherence judgment ratifies one direction or +falsifies the other. -1. A coherence checker invoked at `TeachingStore.add` that detects - `(S, R, O)` ↔ `(S, R_negated, O)` pairs and transitions both to - CONTESTED. -2. A geometric contradiction signal derived from versor-condition - delta when conflicting teachings are applied. - -ADR-0021 reserves `CONTESTED` and `FALSIFIED` precisely for this; the -machinery to *enter* those states is not yet wired. +The v1 versor-spike heuristic was retired in the same commit; the +runner now reads the CONTESTED transition directly. ## Pass criteria -| Metric | Definition | v1 threshold | Honest current expectation | -|--------|-----------|--------------|----------------------------| -| `contradiction_flag_rate` | Fraction of paired-contradiction cases where the second event surfaces a CONTESTED transition or versor spike | 0.90 | 0.00 | -| `false_flag_rate` | Fraction of paired-consistent cases that are incorrectly flagged as contradictory | 0.00 | 0.00 (no flagger exists) | -| `overall_pass` | flag_rate ≥ 0.90 AND false_flag_rate == 0 | true | false at v1 | +| Metric | Definition | v1 threshold | Current | +|--------|-----------|--------------|---------| +| `contradiction_flag_rate` | Fraction of paired-contradiction cases where the second event surfaces a CONTESTED transition | 0.90 | **1.00** | +| `false_flag_rate` | Fraction of paired-consistent cases that are incorrectly flagged as contradictory | 0.00 | **0.00** | +| `overall_pass` | flag_rate ≥ 0.90 AND false_flag_rate == 0 | true | **true** | ## Cases diff --git a/evals/contradiction_detection/runner.py b/evals/contradiction_detection/runner.py index e477b083..5c04e5d5 100644 --- a/evals/contradiction_detection/runner.py +++ b/evals/contradiction_detection/runner.py @@ -1,9 +1,14 @@ """contradiction-detection lane runner. Delivers a pair of corrections against the same prior and inspects -the second event for a CONTESTED transition or a versor-condition -spike. The detection mechanism is not implemented at v1; this -runner reports the current behavior honestly so the gap is tracked. +the second event for a CONTESTED transition (ADR-0021). + +The v1 versor-spike heuristic was retired 2026-05-17 when the +coherence checker in ``TeachingStore.add`` landed: same-subject +proposals with opposing polarity are now transitioned to +``EpistemicStatus.CONTESTED`` at write time, and the lane reads that +directly. ``versor_delta`` is still reported for telemetry but no +longer drives the flag. Conforms to the framework interface: run_lane(cases, config=None) -> report. """ @@ -68,7 +73,9 @@ def _run_case(case: dict[str, Any]) -> dict[str, Any]: ) versor_spike = versor_delta > VERSOR_SPIKE_THRESHOLD - flagged = contested or versor_spike + # Real signal: CONTESTED transition from TeachingStore.add. + # versor_spike retained in the record for telemetry/debugging only. + flagged = contested if kind == "paired_contradiction": passed = flagged diff --git a/teaching/store.py b/teaching/store.py index 0812459d..b61bdfde 100644 --- a/teaching/store.py +++ b/teaching/store.py @@ -10,6 +10,7 @@ from __future__ import annotations import hashlib import json +import re from dataclasses import dataclass from teaching.correction import CorrectionCandidate @@ -17,6 +18,63 @@ from teaching.epistemic import EpistemicStatus from teaching.review import ReviewedTeachingExample +# ADR-0021 §CONTESTED transitions: coherence checker tokens. +# Negation markers and opposition markers used to detect (S, R, T) ↔ +# (S, R, ¬T) pairs at add() time. +_NEGATION_TOKENS: frozenset[str] = frozenset({ + "not", "no", "isn't", "aren't", "wasn't", "weren't", + "never", "without", "neither", "nor", +}) +_OPPOSITION_MARKERS: frozenset[str] = frozenset({ + "unrelated", "independent", "opposite", "contrary", + "incompatible", "disjoint", +}) +_STOPWORDS: frozenset[str] = frozenset({ + "the", "and", "but", "for", "with", "from", "into", "onto", + "this", "that", "these", "those", "are", "was", "were", + "has", "had", "have", "been", "being", +}) +# Discourse/teaching markers that appear in correction texts but carry no +# semantic content — excluded from shared-content overlap so they don't +# inflate the contradiction signal between unrelated corrections. +_DISCOURSE_MARKERS: frozenset[str] = frozenset({ + "actually", "correction", "indeed", "rather", "instead", "really", +}) +_WORD_SPLIT = re.compile(r"[^a-z]+") + + +def _content_tokens(text: str) -> set[str]: + """≥3-char tokens minus stopwords AND discourse markers — used for + shared-content overlap. Discourse markers ("actually", "correction", + …) are excluded so they don't inflate the contradiction signal.""" + return { + tok for tok in _WORD_SPLIT.split(text.lower()) + if len(tok) >= 3 + and tok not in _STOPWORDS + and tok not in _DISCOURSE_MARKERS + } + + +def _subject_tokens(proposal: PackMutationProposal) -> set[str]: + """Extract candidate subject content tokens from raw subject and + parsed-triple subject. Used as a lenient match key for contradiction + detection so "meaning" matches both ", meaning depends on use." and + "meaning" (parsed-triple head).""" + sources: list[str] = [proposal.subject] + if proposal.triple is not None and proposal.triple[0]: + sources.append(proposal.triple[0]) + tokens: set[str] = set() + for src in sources: + tokens |= _content_tokens(src) + return tokens + + +def _has_negation(text: str) -> bool: + """Detect surface negation or opposition tokens.""" + tokens = set(_WORD_SPLIT.split(text.lower())) + return bool(tokens & _NEGATION_TOKENS) or bool(tokens & _OPPOSITION_MARKERS) + + @dataclass(frozen=True, slots=True) class PackMutationProposal: """A proposed vocabulary manifold change, not yet applied. @@ -98,6 +156,14 @@ class TeachingStore: Rejected examples are dropped silently. Returns None if the example was not accepted. + + ADR-0021 §CONTESTED transitions: before appending, the new + proposal is checked against prior proposals for direct + contradiction (same subject, conflicting polarity). When a + contradiction is detected, BOTH the new proposal and the + conflicting prior are upgraded to ``EpistemicStatus.CONTESTED`` + — neither is admissible as evidence until a coherence judgment + ratifies one of them. See ``_detect_contradiction``. """ if not example.accepted: return None @@ -119,9 +185,88 @@ class TeachingStore: triple=triple, epistemic_status=example.epistemic_status, ) + + # Coherence judgment — detect (S, R, T) ↔ (S, R, ¬T) pairs and + # transition both proposals to CONTESTED. ADR-0021: CONTESTED is + # not admissible as evidence; the next reviewed correction can + # ratify one direction back to COHERENT or FALSIFY the other. + conflict_idx = self._detect_contradiction(proposal) + if conflict_idx is not None: + proposal = proposal.with_status(EpistemicStatus.CONTESTED) + self._proposals[conflict_idx] = self._proposals[conflict_idx].with_status( + EpistemicStatus.CONTESTED + ) + self._proposals.append(proposal) return proposal + def _detect_contradiction( + self, new_proposal: PackMutationProposal + ) -> int | None: + """Return the index of a prior proposal that contradicts ``new_proposal``, + or None. + + Detection has two paths. Both require subject identity after + stripping discourse prefixes (so "correction: knowledge" matches + "knowledge"). + + Path A — typed: both proposals parsed to triples with the same + relation. Tails must differ in negation polarity AND share at + least one content token. Catches the clean + (S, R, T) ↔ (S, R, not T) shape. + + Path B — text fallback: at least one proposal failed to parse a + triple (e.g. the relation predicate isn't in the cognition pack + lexicon yet, like "depends"). Correction texts must differ in + negation polarity AND share at least one non-subject content + token. Catches paraphrased contradictions like "X depends on Y" + vs "X is independent of Y". + + Returns the index of the first matching prior in + ``self._proposals``, or None if no contradiction is found. + Existing CONTESTED proposals are skipped — once contested, + further contradictions don't add information until review. + """ + new_subjects = _subject_tokens(new_proposal) + if not new_subjects: + return None + new_text_negated = _has_negation(new_proposal.correction_text) + new_text_tokens = _content_tokens(new_proposal.correction_text) + + for idx, prior in enumerate(self._proposals): + if prior.epistemic_status is EpistemicStatus.CONTESTED: + continue + prior_subjects = _subject_tokens(prior) + if not (new_subjects & prior_subjects): + continue + + # Path A — typed: both parsed AND same relation. Tails must + # differ in polarity and share at least one content token. + if ( + new_proposal.triple is not None + and prior.triple is not None + and new_proposal.triple[1] == prior.triple[1] + ): + new_tail = new_proposal.triple[2] + prior_tail = prior.triple[2] + if _has_negation(new_tail) != _has_negation(prior_tail): + if _content_tokens(new_tail) & _content_tokens(prior_tail): + return idx + + # Path B — text fallback. At least one proposal failed to + # parse a triple (or relations differ). Polarity must differ + # and the texts must share ≥2 non-subject content tokens — + # the ≥2 threshold prevents a single shared subject token from + # flagging unrelated corrections as contradictions. + prior_text_negated = _has_negation(prior.correction_text) + if new_text_negated != prior_text_negated: + prior_text_tokens = _content_tokens(prior.correction_text) + shared = new_text_tokens & prior_text_tokens + if len(shared) >= 2: + return idx + + return None + def triples(self) -> tuple[tuple[str, str, str], ...]: """Return all typed (head, relation, tail) triples currently stored.