refactor(contemplation): converge to shared discovery-sink plumbing (#58)
Connects ADR-0080's read-only contemplation loop to the existing teaching-pipeline plumbing without forcing a type collapse. The SPECULATIVE-only invariant from #55 is preserved verbatim; what changes is *where the findings flow*. What was wrong with the prior shape ----------------------------------- PR #55 shipped a parallel core/contemplation/ package whose findings were written as one JSON blob per CLI invocation, with no consumer. The SPECULATIVE-only invariant protected a write path that didn't exist. My closed PR #56 (second miner) would have entrenched the duplication. What this PR changes -------------------- 1. Schema (core/contemplation/schema.py) - Adds a BOUNDARY note documenting why EvidencePointer (teaching) and ContemplationEvidenceRef (core) intentionally stay separate: EvidencePointer.source is constrained to {corpus, pack, vault_coherent} — pointers into reviewed in-process memory the runtime trusts. ContemplationEvidenceRef points to external report files that have NOT been reviewed. Converging them would either widen the runtime-grounding enum (losing the "reviewed memory only" guarantee) or force benchmark reports to masquerade as vault_coherent. Both are worse than keeping them separate. - Adds format_contemplation_finding_jsonl(finding) — the canonical JSONL formatter mirroring teaching.discovery.format_candidate_jsonl. 2. Runner (core/contemplation/runner.py) - Both runners gain an optional sink: DiscoveryCandidateSink | None parameter. When supplied, each finding is emitted as one canonical JSONL line via the SHARED protocol — same protocol that backs DiscoveryBufferSink and DiscoveryMonthlyFileSink. - Sink path is additive: the ContemplationRun blob is byte-identical whether or not a sink is supplied (pinned by test). - No sink supplied → existing in-memory behavior preserved exactly. 3. CLI (core/contemplation/__main__.py) - Adds --lane {frontier_compare, contradiction_detection} flag. Default unchanged. - Adds --sink-root <path> flag. When set, instantiates a DiscoveryMonthlyFileSink and findings land at <root>/<YYYY>/<YYYY-MM>.jsonl — the SAME layout discovery candidates use, so operators can grep one stream. 4. Miner (core/contemplation/miners/contradiction_detection.py) - Restored from closed PR #56 under the unified pipeline. - Failure-mode split preserved (missed_contradiction / false_contradiction_flag) with asymmetric repair actions. What this PR does NOT do ------------------------ - Does NOT unify ContemplationFinding with DiscoveryCandidate. DiscoveryCandidate.trigger is Literal[would_have_grounded, successful_comparison, hedge_acknowledged, oov_resolved_via_decomp] — all turn-loop flavored. None describe "I parsed a benchmark report." Forcing a 5th trigger that no turn-loop extractor produces would pollute the turn-loop type for the schema's sake. - Does NOT extend teaching/gaps.py. Gap aggregates DiscoveryCandidate cells by (subject, intent) — domain nouns. ContemplationFinding subjects are namespaced ("contradiction_detection/CON-PUB-002"). Different operator views. A sibling aggregator can come later when an operator actually asks for it. Why this is the right unification point --------------------------------------- The honest convergence is at the *sink* (so all SPECULATIVE evidence lives in one rooted append-only stream), not the *aggregator* (which appropriately produces typed views per evidence family). The boundary doctrine from #55 is preserved; it now connects to existing plumbing instead of writing JSON to disk with no consumer. Tests (tests/test_contemplation_pipeline_convergence.py, 10 cases) ------------------------------------------------------------------ - DiscoveryBufferSink satisfies DiscoveryCandidateSink (shared protocol) - frontier runner emits findings to shared sink - contradiction runner emits findings to shared sink - sink is optional — no-op when absent - emission is canonical JSONL (sorted keys, no newline, deterministic) - DiscoveryMonthlyFileSink persists findings at <root>/<YYYY>/<YYYY-MM>.jsonl - sink emission does not alter the ContemplationRun blob (additive) - contradiction miner predicate split + repair-action asymmetry - config_hash differs between lanes (replay can distinguish) - BOUNDARY doc is present in schema.py (regression guard) - ContemplationEvidenceRef field invariants - format_contemplation_finding_jsonl is deterministic + canonical All 18 tests pass (5 original ADR-0080 + 13 new convergence). Live evidence ------------- $ uv run python -m core.contemplation \ evals/contradiction_detection/results/v1_public_*.json \ --lane contradiction_detection \ --sink-root /tmp/sink_demo /tmp/sink_demo/2026/2026-05.jsonl ← same layout as discovery candidates predicate=missed_contradiction subject=contradiction_detection/CON-PUB-002 predicate=missed_contradiction subject=contradiction_detection/CON-PUB-004 predicate=false_contradiction_flag subject=contradiction_detection/CON-PUB-005 predicate=false_contradiction_flag subject=contradiction_detection/CON-PUB-006
This commit is contained in:
parent
c2c1cb94e9
commit
1573064349
6 changed files with 624 additions and 10 deletions
|
|
@ -6,9 +6,17 @@ import sys
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from core.contemplation.runner import (
|
from core.contemplation.runner import (
|
||||||
|
contemplate_contradiction_reports,
|
||||||
contemplate_frontier_reports,
|
contemplate_frontier_reports,
|
||||||
write_contemplation_run,
|
write_contemplation_run,
|
||||||
)
|
)
|
||||||
|
from teaching.discovery_sink import DiscoveryMonthlyFileSink
|
||||||
|
|
||||||
|
|
||||||
|
_LANE_RUNNERS = {
|
||||||
|
"frontier_compare": contemplate_frontier_reports,
|
||||||
|
"contradiction_detection": contemplate_contradiction_reports,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def build_parser() -> argparse.ArgumentParser:
|
def build_parser() -> argparse.ArgumentParser:
|
||||||
|
|
@ -20,7 +28,13 @@ def build_parser() -> argparse.ArgumentParser:
|
||||||
"reports",
|
"reports",
|
||||||
nargs="+",
|
nargs="+",
|
||||||
type=Path,
|
type=Path,
|
||||||
help="frontier_compare JSON report path(s) to contemplate.",
|
help="Report JSON path(s) to contemplate. All paths must share --lane.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--lane",
|
||||||
|
choices=sorted(_LANE_RUNNERS.keys()),
|
||||||
|
default="frontier_compare",
|
||||||
|
help="Evidence lane the reports belong to (default: frontier_compare).",
|
||||||
)
|
)
|
||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"--pack-id",
|
"--pack-id",
|
||||||
|
|
@ -38,18 +52,38 @@ def build_parser() -> argparse.ArgumentParser:
|
||||||
"--report",
|
"--report",
|
||||||
type=Path,
|
type=Path,
|
||||||
default=None,
|
default=None,
|
||||||
help="Optional output path for the contemplation run JSON.",
|
help="Optional output path for the contemplation run JSON blob.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--sink-root",
|
||||||
|
type=Path,
|
||||||
|
default=None,
|
||||||
|
help=(
|
||||||
|
"Optional append-only JSONL sink root. When set each finding is "
|
||||||
|
"emitted as one canonical JSONL line via "
|
||||||
|
"teaching.discovery_sink.DiscoveryMonthlyFileSink at "
|
||||||
|
"<root>/<YYYY>/<YYYY-MM>.jsonl alongside discovery candidates."
|
||||||
|
),
|
||||||
)
|
)
|
||||||
return parser
|
return parser
|
||||||
|
|
||||||
|
|
||||||
def main(argv: list[str] | None = None) -> int:
|
def main(argv: list[str] | None = None) -> int:
|
||||||
args = build_parser().parse_args(argv)
|
args = build_parser().parse_args(argv)
|
||||||
run = contemplate_frontier_reports(
|
runner = _LANE_RUNNERS[args.lane]
|
||||||
args.reports,
|
sink = (
|
||||||
pack_ids=tuple(args.pack_id or ()),
|
DiscoveryMonthlyFileSink(args.sink_root) if args.sink_root is not None else None
|
||||||
notes=tuple(args.note or ()),
|
|
||||||
)
|
)
|
||||||
|
try:
|
||||||
|
run = runner(
|
||||||
|
args.reports,
|
||||||
|
pack_ids=tuple(args.pack_id or ()),
|
||||||
|
notes=tuple(args.note or ()),
|
||||||
|
sink=sink,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
if sink is not None:
|
||||||
|
sink.close()
|
||||||
if args.report is not None:
|
if args.report is not None:
|
||||||
write_contemplation_run(run, args.report)
|
write_contemplation_run(run, args.report)
|
||||||
print(json.dumps(run.as_dict(), ensure_ascii=False, indent=2, sort_keys=True))
|
print(json.dumps(run.as_dict(), ensure_ascii=False, indent=2, sort_keys=True))
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,9 @@
|
||||||
"""Read-only contemplation miners."""
|
"""Read-only contemplation miners."""
|
||||||
|
|
||||||
|
from .contradiction_detection import mine_contradiction_detection_report
|
||||||
from .frontier_compare import mine_frontier_compare_report
|
from .frontier_compare import mine_frontier_compare_report
|
||||||
|
|
||||||
__all__ = ["mine_frontier_compare_report"]
|
__all__ = [
|
||||||
|
"mine_contradiction_detection_report",
|
||||||
|
"mine_frontier_compare_report",
|
||||||
|
]
|
||||||
|
|
|
||||||
111
core/contemplation/miners/contradiction_detection.py
Normal file
111
core/contemplation/miners/contradiction_detection.py
Normal file
|
|
@ -0,0 +1,111 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from core.contemplation.schema import (
|
||||||
|
ContemplationEvidenceRef,
|
||||||
|
ContemplationFinding,
|
||||||
|
FindingKind,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
_MISSED = "missed_contradiction"
|
||||||
|
_FALSE_FLAG = "false_contradiction_flag"
|
||||||
|
|
||||||
|
|
||||||
|
def _case_iter(report: dict[str, Any]) -> tuple[dict[str, Any], ...]:
|
||||||
|
return tuple(c for c in report.get("cases", ()) if isinstance(c, dict))
|
||||||
|
|
||||||
|
|
||||||
|
def _failure_mode(case: dict[str, Any]) -> str | None:
|
||||||
|
"""Classify a failed contradiction-detection case.
|
||||||
|
|
||||||
|
Returns the predicate for the emitted finding, or ``None`` if the
|
||||||
|
case is not a failure worth flagging.
|
||||||
|
"""
|
||||||
|
if bool(case.get("passed", True)):
|
||||||
|
return None
|
||||||
|
kind = str(case.get("kind", ""))
|
||||||
|
if kind == "paired_contradiction":
|
||||||
|
return _MISSED
|
||||||
|
if kind == "paired_consistent":
|
||||||
|
return _FALSE_FLAG
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _evidence_summary(case: dict[str, Any]) -> str:
|
||||||
|
"""Compact, deterministic one-line evidence summary.
|
||||||
|
|
||||||
|
Captures the lane's quantitative signals so a reviewer can decide
|
||||||
|
without re-running the lane. Keys are sorted to keep the summary
|
||||||
|
stable across runs.
|
||||||
|
"""
|
||||||
|
keys = ("kind", "flagged", "contested", "versor_delta", "versor_spike")
|
||||||
|
parts = [f"{k}={case[k]}" for k in keys if k in case]
|
||||||
|
return ";".join(parts) if parts else "case failed"
|
||||||
|
|
||||||
|
|
||||||
|
def mine_contradiction_detection_report(
|
||||||
|
report_path: str | Path,
|
||||||
|
*,
|
||||||
|
substrate_hash: str,
|
||||||
|
) -> tuple[ContemplationFinding, ...]:
|
||||||
|
"""Convert failed contradiction-detection cases into speculative findings.
|
||||||
|
|
||||||
|
Two failure modes are surfaced as distinct predicates:
|
||||||
|
|
||||||
|
- ``missed_contradiction`` — a ``paired_contradiction`` case
|
||||||
|
the lane failed to flag (a real contradiction slipped through).
|
||||||
|
- ``false_contradiction_flag`` — a ``paired_consistent`` case
|
||||||
|
the lane wrongly flagged (the detector fired on consistent text).
|
||||||
|
|
||||||
|
Both warrant operator attention but call for different repairs, so
|
||||||
|
the predicate split is load-bearing — not cosmetic.
|
||||||
|
|
||||||
|
Read-only: parses ``report_path`` and returns immutable findings.
|
||||||
|
Never writes packs, teaching corpora, or runtime state.
|
||||||
|
"""
|
||||||
|
path = Path(report_path)
|
||||||
|
report = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
findings: list[ContemplationFinding] = []
|
||||||
|
source_id = str(path)
|
||||||
|
lane = str(report.get("lane", "contradiction_detection"))
|
||||||
|
|
||||||
|
for case in _case_iter(report):
|
||||||
|
predicate = _failure_mode(case)
|
||||||
|
if predicate is None:
|
||||||
|
continue
|
||||||
|
case_id = str(case.get("id", "unknown_case"))
|
||||||
|
evidence = ContemplationEvidenceRef(
|
||||||
|
source_type="contradiction_detection_report",
|
||||||
|
source_id=source_id,
|
||||||
|
pointer=f"lane={lane};case={case_id}",
|
||||||
|
summary=_evidence_summary(case),
|
||||||
|
)
|
||||||
|
proposed_action = (
|
||||||
|
"Inspect the paired-contradiction probe that slipped through; "
|
||||||
|
"either tighten the detector's versor-delta threshold or add a "
|
||||||
|
"focused regression for this case."
|
||||||
|
if predicate == _MISSED
|
||||||
|
else "Inspect the paired-consistent probe the detector wrongly "
|
||||||
|
"flagged; either loosen the threshold or add a counter-example "
|
||||||
|
"regression that pins the consistent reading."
|
||||||
|
)
|
||||||
|
findings.append(
|
||||||
|
ContemplationFinding(
|
||||||
|
kind=FindingKind.CONTRADICTION,
|
||||||
|
subject=f"{lane}/{case_id}",
|
||||||
|
predicate=predicate,
|
||||||
|
object=None,
|
||||||
|
evidence_refs=(evidence,),
|
||||||
|
proposed_action=proposed_action,
|
||||||
|
substrate_hash=substrate_hash,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
return tuple(findings)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["mine_contradiction_detection_report"]
|
||||||
|
|
@ -5,9 +5,17 @@ import json
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Iterable
|
from typing import Iterable
|
||||||
|
|
||||||
|
from core.contemplation.miners.contradiction_detection import (
|
||||||
|
mine_contradiction_detection_report,
|
||||||
|
)
|
||||||
from core.contemplation.miners.frontier_compare import mine_frontier_compare_report
|
from core.contemplation.miners.frontier_compare import mine_frontier_compare_report
|
||||||
from core.contemplation.schema import ContemplationFinding, ContemplationRun
|
from core.contemplation.schema import (
|
||||||
|
ContemplationFinding,
|
||||||
|
ContemplationRun,
|
||||||
|
format_contemplation_finding_jsonl,
|
||||||
|
)
|
||||||
from core.contemplation.snapshot import ContemplationSubstrate
|
from core.contemplation.snapshot import ContemplationSubstrate
|
||||||
|
from teaching.discovery_sink import DiscoveryCandidateSink
|
||||||
|
|
||||||
|
|
||||||
def _config_hash(payload: dict[str, object]) -> str:
|
def _config_hash(payload: dict[str, object]) -> str:
|
||||||
|
|
@ -20,17 +28,50 @@ def _config_hash(payload: dict[str, object]) -> str:
|
||||||
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
|
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _emit_findings(
|
||||||
|
findings: Iterable[ContemplationFinding],
|
||||||
|
sink: DiscoveryCandidateSink | None,
|
||||||
|
) -> None:
|
||||||
|
"""Stream each finding through the shared sink protocol when set.
|
||||||
|
|
||||||
|
No-op when *sink* is ``None`` — preserves the existing "build a
|
||||||
|
``ContemplationRun`` blob" path for callers that want a single
|
||||||
|
in-memory result.
|
||||||
|
|
||||||
|
When a sink is supplied, each finding is emitted as one canonical
|
||||||
|
JSONL line via :func:`format_contemplation_finding_jsonl`. This
|
||||||
|
is the unification point with the discovery candidate stream
|
||||||
|
(ADR-0055 Phase B sinks): both flow through the same
|
||||||
|
``DiscoveryCandidateSink`` protocol, both land in append-only
|
||||||
|
monthly JSONL files when paired with
|
||||||
|
:class:`teaching.discovery_sink.DiscoveryMonthlyFileSink`.
|
||||||
|
|
||||||
|
Sink errors are NOT swallowed — see ADR-0055 fail-fast contract.
|
||||||
|
"""
|
||||||
|
if sink is None:
|
||||||
|
return
|
||||||
|
for finding in findings:
|
||||||
|
sink.emit(format_contemplation_finding_jsonl(finding))
|
||||||
|
|
||||||
|
|
||||||
def contemplate_frontier_reports(
|
def contemplate_frontier_reports(
|
||||||
report_paths: Iterable[str | Path],
|
report_paths: Iterable[str | Path],
|
||||||
*,
|
*,
|
||||||
pack_ids: Iterable[str] = (),
|
pack_ids: Iterable[str] = (),
|
||||||
notes: Iterable[str] = (),
|
notes: Iterable[str] = (),
|
||||||
|
sink: DiscoveryCandidateSink | None = None,
|
||||||
) -> ContemplationRun:
|
) -> ContemplationRun:
|
||||||
"""Run ADR-0080 Phase 1 over explicit frontier-compare reports.
|
"""Run ADR-0080 Phase 1 over explicit frontier-compare reports.
|
||||||
|
|
||||||
The runner is read-only. It does not discover files implicitly, does not
|
The runner is read-only. It does not discover files implicitly, does not
|
||||||
mutate packs, does not write teaching examples, and does not promote any
|
mutate packs, does not write teaching examples, and does not promote any
|
||||||
finding beyond SPECULATIVE.
|
finding beyond SPECULATIVE.
|
||||||
|
|
||||||
|
When *sink* is supplied each finding is also emitted as one
|
||||||
|
canonical JSONL line via the shared
|
||||||
|
:class:`teaching.discovery_sink.DiscoveryCandidateSink` protocol,
|
||||||
|
so contemplation findings flow into the same append-only stream
|
||||||
|
discovery candidates use.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
paths = tuple(Path(p) for p in report_paths)
|
paths = tuple(Path(p) for p in report_paths)
|
||||||
|
|
@ -47,6 +88,7 @@ def contemplate_frontier_reports(
|
||||||
substrate_hash=substrate.substrate_hash,
|
substrate_hash=substrate.substrate_hash,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
_emit_findings(findings, sink)
|
||||||
config_hash = _config_hash(
|
config_hash = _config_hash(
|
||||||
{
|
{
|
||||||
"runner": "contemplate_frontier_reports",
|
"runner": "contemplate_frontier_reports",
|
||||||
|
|
@ -62,6 +104,51 @@ def contemplate_frontier_reports(
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def contemplate_contradiction_reports(
|
||||||
|
report_paths: Iterable[str | Path],
|
||||||
|
*,
|
||||||
|
pack_ids: Iterable[str] = (),
|
||||||
|
notes: Iterable[str] = (),
|
||||||
|
sink: DiscoveryCandidateSink | None = None,
|
||||||
|
) -> ContemplationRun:
|
||||||
|
"""Run ADR-0080 Phase 1 over explicit contradiction-detection reports.
|
||||||
|
|
||||||
|
Mirrors :func:`contemplate_frontier_reports` for the
|
||||||
|
``evals/contradiction_detection`` lane. Same read-only guarantees,
|
||||||
|
same SPECULATIVE-only finding contract, separate runner so the
|
||||||
|
config hash records which lane was contemplated.
|
||||||
|
"""
|
||||||
|
|
||||||
|
paths = tuple(Path(p) for p in report_paths)
|
||||||
|
substrate = ContemplationSubstrate.from_report_paths(
|
||||||
|
paths,
|
||||||
|
pack_ids=tuple(pack_ids),
|
||||||
|
notes=tuple(notes),
|
||||||
|
)
|
||||||
|
findings: list[ContemplationFinding] = []
|
||||||
|
for path in paths:
|
||||||
|
findings.extend(
|
||||||
|
mine_contradiction_detection_report(
|
||||||
|
path,
|
||||||
|
substrate_hash=substrate.substrate_hash,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
_emit_findings(findings, sink)
|
||||||
|
config_hash = _config_hash(
|
||||||
|
{
|
||||||
|
"runner": "contemplate_contradiction_reports",
|
||||||
|
"report_paths": [str(p) for p in paths],
|
||||||
|
"pack_ids": tuple(sorted(set(pack_ids))),
|
||||||
|
"notes": tuple(notes),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return ContemplationRun(
|
||||||
|
substrate_hash=substrate.substrate_hash,
|
||||||
|
config_hash=config_hash,
|
||||||
|
findings=tuple(findings),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def write_contemplation_run(run: ContemplationRun, path: str | Path) -> None:
|
def write_contemplation_run(run: ContemplationRun, path: str | Path) -> None:
|
||||||
target = Path(path)
|
target = Path(path)
|
||||||
target.parent.mkdir(parents=True, exist_ok=True)
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
@ -71,4 +158,8 @@ def write_contemplation_run(run: ContemplationRun, path: str | Path) -> None:
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["contemplate_frontier_reports", "write_contemplation_run"]
|
__all__ = [
|
||||||
|
"contemplate_contradiction_reports",
|
||||||
|
"contemplate_frontier_reports",
|
||||||
|
"write_contemplation_run",
|
||||||
|
]
|
||||||
|
|
|
||||||
|
|
@ -40,9 +40,40 @@ def _sha256_16(payload: dict[str, Any]) -> str:
|
||||||
return hashlib.sha256(_canonical_json(payload).encode("utf-8")).hexdigest()[:16]
|
return hashlib.sha256(_canonical_json(payload).encode("utf-8")).hexdigest()[:16]
|
||||||
|
|
||||||
|
|
||||||
|
# BOUNDARY (vs teaching/discovery.py:EvidencePointer)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# ``EvidencePointer`` (teaching/discovery.py) and ``ContemplationEvidenceRef``
|
||||||
|
# below intentionally remain separate types. They have *different
|
||||||
|
# semantics*, not just different shapes:
|
||||||
|
#
|
||||||
|
# - ``EvidencePointer.source`` is constrained to
|
||||||
|
# ``{corpus, pack, vault_coherent}`` — pointers into reviewed
|
||||||
|
# in-process memory the runtime trusts as grounding.
|
||||||
|
# - ``ContemplationEvidenceRef.source_type`` is free-form because
|
||||||
|
# it points at external artifacts (benchmark report files,
|
||||||
|
# evaluator output, lab measurements) that have not been
|
||||||
|
# reviewed by the teaching pipeline.
|
||||||
|
#
|
||||||
|
# Converging them would either widen the runtime-grounding source
|
||||||
|
# enum (a real loss — losing the "this came from reviewed memory"
|
||||||
|
# guarantee) or force benchmark reports to masquerade as
|
||||||
|
# ``vault_coherent``. Both are worse than keeping the two types
|
||||||
|
# separate and documented.
|
||||||
|
#
|
||||||
|
# What IS shared:
|
||||||
|
# - ``EpistemicStatus`` (one source of truth at teaching/epistemic.py).
|
||||||
|
# - Sink plumbing (DiscoveryCandidateSink protocol) — see
|
||||||
|
# ``core/contemplation/sink.py``.
|
||||||
|
# - Append-only monthly JSONL file convention.
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True, slots=True)
|
@dataclass(frozen=True, slots=True)
|
||||||
class ContemplationEvidenceRef:
|
class ContemplationEvidenceRef:
|
||||||
"""Pointer to evidence supporting a contemplation finding."""
|
"""Pointer to evidence supporting a contemplation finding.
|
||||||
|
|
||||||
|
Distinct from :class:`teaching.discovery.EvidencePointer` — see
|
||||||
|
the BOUNDARY note above this class for why the two stay separate.
|
||||||
|
"""
|
||||||
|
|
||||||
source_type: str
|
source_type: str
|
||||||
source_id: str
|
source_id: str
|
||||||
|
|
@ -158,9 +189,21 @@ class ContemplationRun:
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def format_contemplation_finding_jsonl(finding: ContemplationFinding) -> str:
|
||||||
|
"""Return a deterministic JSONL line for a contemplation finding.
|
||||||
|
|
||||||
|
Mirrors :func:`teaching.discovery.format_candidate_jsonl` in style
|
||||||
|
(canonical JSON, sorted keys, no trailing newline) so both flow
|
||||||
|
through the same ``DiscoveryCandidateSink`` plumbing without
|
||||||
|
schema-aware special casing.
|
||||||
|
"""
|
||||||
|
return _canonical_json(finding.as_dict())
|
||||||
|
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"ContemplationEvidenceRef",
|
"ContemplationEvidenceRef",
|
||||||
"ContemplationFinding",
|
"ContemplationFinding",
|
||||||
"ContemplationRun",
|
"ContemplationRun",
|
||||||
"FindingKind",
|
"FindingKind",
|
||||||
|
"format_contemplation_finding_jsonl",
|
||||||
]
|
]
|
||||||
|
|
|
||||||
331
tests/test_contemplation_pipeline_convergence.py
Normal file
331
tests/test_contemplation_pipeline_convergence.py
Normal file
|
|
@ -0,0 +1,331 @@
|
||||||
|
"""Tests for the contemplation→teaching pipeline convergence refactor.
|
||||||
|
|
||||||
|
Pins three properties:
|
||||||
|
|
||||||
|
1. ContemplationFinding emission flows through the SHARED sink
|
||||||
|
protocol from teaching/discovery_sink.py — not a parallel
|
||||||
|
bespoke sink.
|
||||||
|
2. The contradiction_detection miner survives and works under the
|
||||||
|
unified sink pattern (preserves the work from the closed #56
|
||||||
|
while routing it through the corrected boundary).
|
||||||
|
3. The boundary doc in core/contemplation/schema.py is intact —
|
||||||
|
EvidencePointer (teaching) and ContemplationEvidenceRef (core)
|
||||||
|
stay separate by design, not by oversight.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from core.contemplation.miners.contradiction_detection import (
|
||||||
|
mine_contradiction_detection_report,
|
||||||
|
)
|
||||||
|
from core.contemplation.runner import (
|
||||||
|
contemplate_contradiction_reports,
|
||||||
|
contemplate_frontier_reports,
|
||||||
|
)
|
||||||
|
from core.contemplation.schema import (
|
||||||
|
ContemplationEvidenceRef,
|
||||||
|
ContemplationFinding,
|
||||||
|
FindingKind,
|
||||||
|
format_contemplation_finding_jsonl,
|
||||||
|
)
|
||||||
|
from core.contemplation.snapshot import ContemplationSubstrate
|
||||||
|
from teaching.discovery_sink import (
|
||||||
|
DiscoveryBufferSink,
|
||||||
|
DiscoveryCandidateSink,
|
||||||
|
DiscoveryMonthlyFileSink,
|
||||||
|
)
|
||||||
|
from teaching.epistemic import EpistemicStatus
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Fixtures: real-shape sample reports for both lanes
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _frontier_report(path: Path) -> None:
|
||||||
|
payload = {
|
||||||
|
"benchmark_family": "frontier_compare_wave1",
|
||||||
|
"suites": [
|
||||||
|
{
|
||||||
|
"suite": "truth_lock",
|
||||||
|
"cases": [
|
||||||
|
{
|
||||||
|
"suite": "truth_lock",
|
||||||
|
"case_id": "known_truth",
|
||||||
|
"prompt": "What is truth?",
|
||||||
|
"passed": True,
|
||||||
|
"failures": [],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"suite": "truth_lock",
|
||||||
|
"case_id": "unknown_relation",
|
||||||
|
"prompt": "Why does xylomorphic matter?",
|
||||||
|
"passed": False,
|
||||||
|
"failures": ["unexpected_grounding_source:vault"],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
path.write_text(json.dumps(payload, sort_keys=True), encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def _contradiction_report(path: Path) -> None:
|
||||||
|
payload = {
|
||||||
|
"lane": "contradiction_detection",
|
||||||
|
"split": "public",
|
||||||
|
"version": "v1",
|
||||||
|
"cases": [
|
||||||
|
{
|
||||||
|
"id": "CON-PUB-001",
|
||||||
|
"kind": "paired_contradiction",
|
||||||
|
"passed": True,
|
||||||
|
"flagged": True,
|
||||||
|
"contested": False,
|
||||||
|
"versor_delta": 3.28677e-07,
|
||||||
|
"versor_spike": True,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "CON-PUB-002",
|
||||||
|
"kind": "paired_contradiction",
|
||||||
|
"passed": False,
|
||||||
|
"flagged": False,
|
||||||
|
"contested": False,
|
||||||
|
"versor_delta": 0.0,
|
||||||
|
"versor_spike": False,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "CON-PUB-005",
|
||||||
|
"kind": "paired_consistent",
|
||||||
|
"passed": False,
|
||||||
|
"flagged": True,
|
||||||
|
"contested": False,
|
||||||
|
"versor_delta": 3.24412e-07,
|
||||||
|
"versor_spike": True,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
path.write_text(json.dumps(payload, sort_keys=True), encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1. Shared sink protocol — the actual integration claim
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_sink_is_typed_as_discovery_candidate_sink() -> None:
|
||||||
|
"""DiscoveryBufferSink must satisfy DiscoveryCandidateSink — that is
|
||||||
|
the shared protocol both pipelines now feed.
|
||||||
|
|
||||||
|
A static structural check, but if someone widens the protocol in
|
||||||
|
teaching/discovery_sink.py without adapting the contemplation
|
||||||
|
emitter, this test surfaces the regression at the type level.
|
||||||
|
"""
|
||||||
|
sink: DiscoveryCandidateSink = DiscoveryBufferSink()
|
||||||
|
assert hasattr(sink, "emit")
|
||||||
|
sink.emit("{}")
|
||||||
|
assert sink.lines == ["{}"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_frontier_runner_emits_findings_to_shared_sink(tmp_path: Path) -> None:
|
||||||
|
report = tmp_path / "frontier.json"
|
||||||
|
_frontier_report(report)
|
||||||
|
sink = DiscoveryBufferSink()
|
||||||
|
|
||||||
|
run = contemplate_frontier_reports((report,), sink=sink)
|
||||||
|
|
||||||
|
# One emission per finding, no extra noise (passing cases don't emit).
|
||||||
|
assert len(sink.lines) == 1
|
||||||
|
assert len(run.findings) == 1
|
||||||
|
# The emitted line round-trips to the same payload as the run blob.
|
||||||
|
emitted = json.loads(sink.lines[0])
|
||||||
|
assert emitted == run.findings[0].as_dict()
|
||||||
|
|
||||||
|
|
||||||
|
def test_contradiction_runner_emits_findings_to_shared_sink(tmp_path: Path) -> None:
|
||||||
|
report = tmp_path / "contradiction.json"
|
||||||
|
_contradiction_report(report)
|
||||||
|
sink = DiscoveryBufferSink()
|
||||||
|
|
||||||
|
run = contemplate_contradiction_reports((report,), sink=sink)
|
||||||
|
|
||||||
|
# Two failed cases (1 missed, 1 false-flag); passing case suppressed.
|
||||||
|
assert len(sink.lines) == 2
|
||||||
|
assert len(run.findings) == 2
|
||||||
|
predicates = {json.loads(line)["predicate"] for line in sink.lines}
|
||||||
|
assert predicates == {"missed_contradiction", "false_contradiction_flag"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_sink_is_optional_no_op_when_absent(tmp_path: Path) -> None:
|
||||||
|
"""Existing in-memory ContemplationRun behavior is preserved when
|
||||||
|
no sink is supplied — guarantees the refactor is non-breaking."""
|
||||||
|
report = tmp_path / "contradiction.json"
|
||||||
|
_contradiction_report(report)
|
||||||
|
|
||||||
|
run = contemplate_contradiction_reports((report,)) # no sink kwarg
|
||||||
|
|
||||||
|
assert len(run.findings) == 2
|
||||||
|
assert all(
|
||||||
|
f.epistemic_status is EpistemicStatus.SPECULATIVE for f in run.findings
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_emission_is_jsonl_canonical(tmp_path: Path) -> None:
|
||||||
|
"""The JSONL line shape must match the discovery_sink convention:
|
||||||
|
one canonical-JSON line per finding, sorted keys, no newline,
|
||||||
|
deterministic across runs."""
|
||||||
|
report = tmp_path / "contradiction.json"
|
||||||
|
_contradiction_report(report)
|
||||||
|
sink_a = DiscoveryBufferSink()
|
||||||
|
sink_b = DiscoveryBufferSink()
|
||||||
|
|
||||||
|
contemplate_contradiction_reports((report,), sink=sink_a)
|
||||||
|
contemplate_contradiction_reports((report,), sink=sink_b)
|
||||||
|
|
||||||
|
assert sink_a.lines == sink_b.lines
|
||||||
|
for line in sink_a.lines:
|
||||||
|
assert "\n" not in line
|
||||||
|
# Canonical JSON: re-encoding with same options reproduces the line.
|
||||||
|
parsed = json.loads(line)
|
||||||
|
roundtrip = json.dumps(
|
||||||
|
parsed, ensure_ascii=False, sort_keys=True, separators=(",", ":")
|
||||||
|
)
|
||||||
|
assert line == roundtrip
|
||||||
|
|
||||||
|
|
||||||
|
def test_monthly_file_sink_persists_contemplation_findings(
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> None:
|
||||||
|
"""End-to-end: contemplation findings land in the SAME monthly JSONL
|
||||||
|
layout discovery candidates use — <root>/<YYYY>/<YYYY-MM>.jsonl."""
|
||||||
|
report = tmp_path / "contradiction.json"
|
||||||
|
_contradiction_report(report)
|
||||||
|
sink_root = tmp_path / "discovery_log"
|
||||||
|
|
||||||
|
with DiscoveryMonthlyFileSink(sink_root) as sink:
|
||||||
|
run = contemplate_contradiction_reports((report,), sink=sink)
|
||||||
|
assert len(run.findings) == 2
|
||||||
|
|
||||||
|
# Find the monthly file the sink created.
|
||||||
|
written = sorted(sink_root.rglob("*.jsonl"))
|
||||||
|
assert len(written) == 1, f"expected one monthly file, got {written}"
|
||||||
|
lines = [
|
||||||
|
line
|
||||||
|
for line in written[0].read_text(encoding="utf-8").splitlines()
|
||||||
|
if line.strip()
|
||||||
|
]
|
||||||
|
assert len(lines) == 2
|
||||||
|
# Layout matches discovery candidate convention <YYYY>/<YYYY-MM>.jsonl.
|
||||||
|
rel = written[0].relative_to(sink_root)
|
||||||
|
assert len(rel.parts) == 2 # YYYY / YYYY-MM.jsonl
|
||||||
|
assert rel.parts[1].endswith(".jsonl")
|
||||||
|
|
||||||
|
|
||||||
|
def test_sink_emission_does_not_alter_run_blob(tmp_path: Path) -> None:
|
||||||
|
"""Emitting to a sink and building the ContemplationRun blob must
|
||||||
|
produce IDENTICAL findings — the sink path is additional output,
|
||||||
|
not a parallel transformation."""
|
||||||
|
report = tmp_path / "frontier.json"
|
||||||
|
_frontier_report(report)
|
||||||
|
|
||||||
|
run_no_sink = contemplate_frontier_reports((report,))
|
||||||
|
sink = DiscoveryBufferSink()
|
||||||
|
run_with_sink = contemplate_frontier_reports((report,), sink=sink)
|
||||||
|
|
||||||
|
assert run_no_sink.run_id == run_with_sink.run_id
|
||||||
|
assert run_no_sink.as_dict() == run_with_sink.as_dict()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 2. Contradiction miner — preserved from closed #56, now under the
|
||||||
|
# unified sink contract
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_contradiction_miner_predicate_split(tmp_path: Path) -> None:
|
||||||
|
report = tmp_path / "contradiction.json"
|
||||||
|
_contradiction_report(report)
|
||||||
|
substrate = ContemplationSubstrate.from_report_paths((report,))
|
||||||
|
|
||||||
|
findings = mine_contradiction_detection_report(
|
||||||
|
report, substrate_hash=substrate.substrate_hash
|
||||||
|
)
|
||||||
|
by_predicate = {f.predicate: f for f in findings}
|
||||||
|
assert set(by_predicate.keys()) == {
|
||||||
|
"missed_contradiction",
|
||||||
|
"false_contradiction_flag",
|
||||||
|
}
|
||||||
|
# Repair-action asymmetry: tightening (missed) vs loosening (false flag).
|
||||||
|
assert "tighten" in by_predicate["missed_contradiction"].proposed_action
|
||||||
|
assert "loosen" in by_predicate["false_contradiction_flag"].proposed_action
|
||||||
|
# All findings emit FindingKind.CONTRADICTION (not BENCHMARK_CASE) —
|
||||||
|
# the lane is semantically targeted.
|
||||||
|
for finding in findings:
|
||||||
|
assert finding.kind is FindingKind.CONTRADICTION
|
||||||
|
|
||||||
|
|
||||||
|
def test_contradiction_config_hash_differs_from_frontier(tmp_path: Path) -> None:
|
||||||
|
"""Lane choice must remain load-bearing in config_hash so replay
|
||||||
|
can distinguish which lane was contemplated. Regression guard
|
||||||
|
against a future refactor that accidentally unifies the runners."""
|
||||||
|
report = tmp_path / "shared.json"
|
||||||
|
_contradiction_report(report)
|
||||||
|
a = contemplate_contradiction_reports((report,))
|
||||||
|
b = contemplate_frontier_reports((report,))
|
||||||
|
assert a.config_hash != b.config_hash
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 3. Boundary doc — the deliberate non-merger of evidence types
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_boundary_doc_present_in_schema_module() -> None:
|
||||||
|
"""The BOUNDARY note in core/contemplation/schema.py is the
|
||||||
|
durable record of why EvidencePointer and ContemplationEvidenceRef
|
||||||
|
stay separate. Pinning its presence prevents a future contributor
|
||||||
|
from silently deleting the boundary rationale."""
|
||||||
|
schema_text = Path("core/contemplation/schema.py").read_text(encoding="utf-8")
|
||||||
|
assert "BOUNDARY" in schema_text
|
||||||
|
assert "EvidencePointer" in schema_text
|
||||||
|
assert "vault_coherent" in schema_text
|
||||||
|
|
||||||
|
|
||||||
|
def test_contemplation_evidence_ref_rejects_empty_fields() -> None:
|
||||||
|
"""The existing schema invariants must survive the refactor."""
|
||||||
|
with pytest.raises(ValueError, match="source_type"):
|
||||||
|
ContemplationEvidenceRef(source_type="", source_id="x", pointer="y")
|
||||||
|
with pytest.raises(ValueError, match="source_id"):
|
||||||
|
ContemplationEvidenceRef(source_type="x", source_id="", pointer="y")
|
||||||
|
with pytest.raises(ValueError, match="pointer"):
|
||||||
|
ContemplationEvidenceRef(source_type="x", source_id="y", pointer="")
|
||||||
|
|
||||||
|
|
||||||
|
def test_format_contemplation_finding_jsonl_is_deterministic() -> None:
|
||||||
|
finding = ContemplationFinding(
|
||||||
|
kind=FindingKind.CONTRADICTION,
|
||||||
|
subject="lane/case_001",
|
||||||
|
predicate="missed_contradiction",
|
||||||
|
object=None,
|
||||||
|
evidence_refs=(
|
||||||
|
ContemplationEvidenceRef(
|
||||||
|
source_type="t", source_id="i", pointer="p", summary="s"
|
||||||
|
),
|
||||||
|
),
|
||||||
|
proposed_action="review",
|
||||||
|
substrate_hash="abc123",
|
||||||
|
)
|
||||||
|
a = format_contemplation_finding_jsonl(finding)
|
||||||
|
b = format_contemplation_finding_jsonl(finding)
|
||||||
|
assert a == b
|
||||||
|
# Round-trips through json.loads/dumps with same options.
|
||||||
|
parsed = json.loads(a)
|
||||||
|
assert (
|
||||||
|
json.dumps(parsed, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
|
||||||
|
== a
|
||||||
|
)
|
||||||
Loading…
Reference in a new issue