283680f110
ADR-0044 — Medical / clinical ethics pack (worked-example domain pack).
Ships packs/ethics/medical_clinical_ethics_v1.json with six commitments
partitioned across all three remediation tiers:
- refuse: no_dosing_recommendation, no_emergency_triage_authority
- hedge: defer_diagnosis_to_clinician, surface_evidence_grade
- audit: disclose_no_clinician_relationship, respect_patient_autonomy
Ratified end-to-end through scripts/ratify_ethics_pack.py (PACK_IDS
extended). Production-mode load via load_ethics_pack succeeds.
ChatRuntime composition includes universal safety floor + every medical
commitment. tests/test_medical_clinical_ethics_pack.py (8 tests) gates
file existence, sealed report, disjoint refusal/hedge lists, and
pack-swap visibility (default pack does NOT carry medical commitments).
ADR-0045 — Long-context recall: CORE vs transformer baselines.
Adds evals/long_context_cost/comparison_runner.py with a deterministic
needle-in-a-haystack measurement at N ∈ {100, 1_000, 10_000, 100_000}.
CORE recall = 100% at every tested N by exact cga_inner scan.
Paired with frozen citations of published transformer NIAH numbers in
evals/long_context_cost/baselines/transformer_long_context.json:
Claude 2.1 (200k, 50%), GPT-4 Turbo 128k (~71%), Gemini 1.5 Pro (99.7%),
NVIDIA RULER (varies). Each citation carries source + url.
The two components measure different inputs (synthetic versors vs NL
needles) and are not directly comparable benchmark-for-benchmark. The
comparison is at the architectural level — exact-scan recall vs
attention-based probabilistic recall. Scope and limits documented in
the ADR. tests/test_long_context_comparison.py (5 tests) gates schema,
CORE recall == 100%, and baseline citation presence.
CLI integration: two new demo targets with study-grade preambles.
- core demo pack-measurements (ADR-0043 — wired)
- core demo long-context-comparison (ADR-0045)
README + docs/PROGRESS.md cheatsheets updated. docs/decisions/README.md
index extended with ADR-0044 + ADR-0045; pack-layer chain title now
"ADR-0027 through ADR-0045".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
64 lines
2.4 KiB
Python
64 lines
2.4 KiB
Python
"""ADR-0045 — long-context recall comparison vs transformer baselines.
|
|
|
|
CORE's vault recall is exact by construction: there is no approximate
|
|
index, no embedding compression, no attention bottleneck. The
|
|
comparison runner publishes:
|
|
|
|
1. A needle-in-a-haystack measurement at multiple N showing CORE
|
|
returns the planted needle at top-1.
|
|
2. Frozen citations of published transformer long-context recall
|
|
numbers (Claude 2.1, GPT-4 Turbo 128k, Gemini 1.5 Pro, NVIDIA
|
|
RULER) as the comparator.
|
|
|
|
If a future change to the vault breaks exact recall, this suite fails.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
from evals.long_context_cost.comparison_runner import (
|
|
DEFAULT_N_VALUES,
|
|
run_comparison,
|
|
)
|
|
|
|
|
|
class TestComparisonRunner:
|
|
def test_report_schema_is_stable(self) -> None:
|
|
report = run_comparison(n_values=(100, 1_000))
|
|
assert report["schema_version"] == 1
|
|
assert "core_measurements" in report
|
|
assert "transformer_baselines" in report
|
|
core = report["core_measurements"]
|
|
assert set(core.keys()) >= {
|
|
"n_values",
|
|
"recall_pct",
|
|
"exact_by_construction",
|
|
"per_n",
|
|
}
|
|
|
|
def test_core_exact_recall_holds(self) -> None:
|
|
"""The load-bearing claim: top-1 needle recall is correct at every N."""
|
|
report = run_comparison(n_values=(100, 1_000, 10_000))
|
|
assert report["claim_supported"] is True
|
|
assert report["core_measurements"]["recall_pct"] == 100.0
|
|
for entry in report["core_measurements"]["per_n"]:
|
|
assert entry["top1_correct"] is True, entry
|
|
|
|
def test_transformer_baselines_are_frozen_citations(self) -> None:
|
|
report = run_comparison(n_values=(100,))
|
|
baselines = report["transformer_baselines"]["baselines"]
|
|
assert len(baselines) >= 3
|
|
for b in baselines:
|
|
assert "source" in b
|
|
assert "url" in b
|
|
assert "context_window_tokens" in b
|
|
|
|
def test_default_n_values_include_at_least_100k(self) -> None:
|
|
"""Ensure the comparison exercises a large-N regime by default."""
|
|
assert max(DEFAULT_N_VALUES) >= 100_000
|
|
|
|
|
|
class TestCoreGuaranteeAdvertised:
|
|
def test_baseline_file_records_core_guarantee(self) -> None:
|
|
report = run_comparison(n_values=(100,))
|
|
guarantee = report["transformer_baselines"]["core_guarantee"]
|
|
assert guarantee["recall_pct"] == 100.0
|
|
assert guarantee["recall_kind"] == "exact_cga_inner_scan"
|