64c5bc4619
Audit of the one-mutation-path invariant (ADR-0021 §3) found three leaks
where pack authority or session-state writes could substitute for coherence
judgment. All three landed fixes or partial closures in this push.
Leaks closed:
- Leak A: pack vocab defaulted to COHERENT — flipped to SPECULATIVE in
language_packs/{compiler,schema}.py; docstring corrected to align with
ADR-0021 (it was rationalizing the leak).
- Leak B: vault.recall was epistemic-blind — VaultStore.store() now stamps
every entry with EpistemicStatus (default SPECULATIVE); recall(min_status=)
filters to admissible-as-evidence tier. All 4 vault-write sites updated.
- Leak C (write-side): generate/proposition.py:198 stored articulated
propositions unmarked — now stamps SPECULATIVE, breaking the
fabrication-feedback loop in principle. Read-side audit of 5 call sites
is the residual.
New architectural invariants (tests/test_architectural_invariants.py):
- INV-21: one-mutation-path allowlist (caught Leak C on first run)
- INV-22: pack lexicon default is SPECULATIVE (Leak A guard)
- INV-23: vault recall epistemic-aware (Leak B guard)
New eval lanes:
- teaching_injection_resistance — ships GREEN at 1.00/1.00/0 (the
structural anti-injection claim is real and measurable)
- refusal_calibration — honest gap: 0% refusal, 0% fabrication
- contradiction_detection — honest gap: 50% flag via versor-delta heuristic,
100% false-positive; motivates the proper coherence-checker
- articulation_of_status — honest gap: 0% speculative articulation, 60%
false certainty; output-side leak surface
New benchmarks:
- benchmarks/footprint.py — total deployed runtime is 7.06 MiB
(109,358x smaller than Llama 3.1 405B, runs offline, no GPU)
- benchmarks/learning_curve.py — monotonic + replay-deterministic curve
per lane
Documentation:
- docs/truth_seeking_schema.md — foundational architectural commitment,
five rules, mapped to human failure modes, leaks published openly
- evals/CLAIMS.md — five-tier public claims doc; Tier 4.5 publishes
known gaps with named fixes; verification contract at top
- README.md — new pillar between algebraic substrate and language pillar
Includes in-flight formation pipeline scaffolding (formation/, tests/formation/,
docs/formation_pipeline_plan.md) and minor CLI/contracts/gitignore edits
that were already in the working tree at session start.
Verification: 798 passed, 2 skipped, 1 deselected (pre-existing pack-count
test drift unrelated to schema changes).
140 lines
4.5 KiB
Python
140 lines
4.5 KiB
Python
"""Tests for ``formation.mastery`` — self-sealed ``MasteryReport``."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from formation.course import GateMeasurement, MasteryReport
|
|
from formation.mastery import emit_report, report_to_dict, verify_report
|
|
|
|
|
|
def _gate(name: str, passed: bool = True) -> GateMeasurement:
|
|
return GateMeasurement(name=name, passed=passed, measurement="1.0", threshold="1.0")
|
|
|
|
|
|
class TestEmit:
|
|
def test_emits_sealed_report(self) -> None:
|
|
rpt = emit_report(
|
|
course_id="x.v1",
|
|
source_bundle_sha="a" * 64,
|
|
validated_set_sha="b" * 64,
|
|
course_sha256="c" * 64,
|
|
plan_sha256="d" * 64,
|
|
gates=(_gate("g1"),),
|
|
trace_hashes=("h1", "h2"),
|
|
issued_at="2026-05-16T00:00:00Z",
|
|
)
|
|
assert rpt.report_sha256 != ""
|
|
assert verify_report(rpt) is True
|
|
assert rpt.ratified is True
|
|
|
|
def test_failure_reasons_set_ratified_false(self) -> None:
|
|
rpt = emit_report(
|
|
course_id="x.v1",
|
|
source_bundle_sha="a" * 64,
|
|
validated_set_sha="b" * 64,
|
|
course_sha256="c" * 64,
|
|
plan_sha256="d" * 64,
|
|
gates=(_gate("g1"),),
|
|
trace_hashes=(),
|
|
failure_reasons=("nope",),
|
|
issued_at="2026-05-16T00:00:00Z",
|
|
)
|
|
assert rpt.ratified is False
|
|
assert verify_report(rpt) is True
|
|
|
|
def test_failing_gate_sets_ratified_false(self) -> None:
|
|
rpt = emit_report(
|
|
course_id="x.v1",
|
|
source_bundle_sha="a" * 64,
|
|
validated_set_sha="b" * 64,
|
|
course_sha256="c" * 64,
|
|
plan_sha256="d" * 64,
|
|
gates=(_gate("g1", passed=False),),
|
|
trace_hashes=(),
|
|
issued_at="2026-05-16T00:00:00Z",
|
|
)
|
|
assert rpt.ratified is False
|
|
|
|
|
|
class TestSeal:
|
|
def test_tamper_breaks_verify(self) -> None:
|
|
rpt = emit_report(
|
|
course_id="x.v1",
|
|
source_bundle_sha="a" * 64,
|
|
validated_set_sha="b" * 64,
|
|
course_sha256="c" * 64,
|
|
plan_sha256="d" * 64,
|
|
gates=(_gate("g1"),),
|
|
trace_hashes=("h1",),
|
|
issued_at="2026-05-16T00:00:00Z",
|
|
)
|
|
tampered = MasteryReport(
|
|
course_id="x.v2", # changed
|
|
source_bundle_sha=rpt.source_bundle_sha,
|
|
validated_set_sha=rpt.validated_set_sha,
|
|
course_sha256=rpt.course_sha256,
|
|
plan_sha256=rpt.plan_sha256,
|
|
gates=rpt.gates,
|
|
trace_hashes=rpt.trace_hashes,
|
|
ratified=rpt.ratified,
|
|
report_sha256=rpt.report_sha256,
|
|
issued_at=rpt.issued_at,
|
|
failure_reasons=rpt.failure_reasons,
|
|
)
|
|
assert verify_report(tampered) is False
|
|
|
|
def test_blank_seal_fails_verify(self) -> None:
|
|
bare = MasteryReport(
|
|
course_id="x.v1",
|
|
source_bundle_sha="a" * 64,
|
|
validated_set_sha="b" * 64,
|
|
course_sha256="c" * 64,
|
|
plan_sha256="d" * 64,
|
|
gates=(),
|
|
trace_hashes=(),
|
|
ratified=True,
|
|
)
|
|
assert verify_report(bare) is False
|
|
|
|
def test_seal_is_deterministic(self) -> None:
|
|
# Two emissions with identical inputs (including issued_at) → same SHA.
|
|
kw = dict(
|
|
course_id="x.v1",
|
|
source_bundle_sha="a" * 64,
|
|
validated_set_sha="b" * 64,
|
|
course_sha256="c" * 64,
|
|
plan_sha256="d" * 64,
|
|
gates=(_gate("g1"),),
|
|
trace_hashes=("h1",),
|
|
issued_at="2026-05-16T00:00:00Z",
|
|
)
|
|
a = emit_report(**kw)
|
|
b = emit_report(**kw)
|
|
assert a.report_sha256 == b.report_sha256
|
|
|
|
|
|
class TestProjection:
|
|
def test_report_to_dict_has_no_floats(self) -> None:
|
|
rpt = emit_report(
|
|
course_id="x.v1",
|
|
source_bundle_sha="a" * 64,
|
|
validated_set_sha="b" * 64,
|
|
course_sha256="c" * 64,
|
|
plan_sha256="d" * 64,
|
|
gates=(_gate("g1"),),
|
|
trace_hashes=("h1",),
|
|
issued_at="2026-05-16T00:00:00Z",
|
|
)
|
|
d = report_to_dict(rpt)
|
|
# Walk the dict and assert no floats.
|
|
def walk(x):
|
|
if isinstance(x, float):
|
|
raise AssertionError("float in report dict")
|
|
if isinstance(x, dict):
|
|
for v in x.values():
|
|
walk(v)
|
|
elif isinstance(x, list):
|
|
for v in x:
|
|
walk(v)
|
|
walk(d)
|