feat: integrate hardened CLOSE yardstick into determinism & teaching regression surfaces
- docs/testing-lanes.md + Makefile: recommended determinism regression invocation (uv run python -m evals.close_derived_climb + contract pytest) as part of standard verification story / rerun flows. - evals/anti_regression/run_demo.py + tests/test_anti_regression_demo.py: hermetic embedding of the yardstick (lived IdleTickResult flag, semantic determine rule=direct, content_replay_checksum) into the core anti-regression / teaching demo flow (core demo anti-regression now carries + reports the Claim-B signals; active corpus remains untouched). - Supporting docs: contract.md (uv + refs), anti_regression_demo.md (complementary note), runtime_contracts.md (determination surface cross-ref). - New ratification artifact (pre-impl) + all success criteria met. See ratification for chosen approach + why only correct path. All existing tests/invariants green; hermeticity preserved; no core/engine changes.
This commit is contained in:
Shay
parent
7a20356ab6
commit
f03979f025
8 changed files with 205 additions and 2 deletions
6
Makefile
6
Makefile
|
|
@ -8,6 +8,12 @@
|
|||
# until the suite is xdist-hermetic (shared repo engine_state/ + report.json +
|
||||
# teaching/proposals writers race under parallel workers). See
|
||||
# docs/testing-lanes.md "Follow-up: xdist".
|
||||
#
|
||||
# CLOSE flywheel Claim-B determinism (post-#791 hardening): after any
|
||||
# CLOSE/idle_tick/realize_derived/consolidate/vault/determine change, also run
|
||||
# uv run python -m evals.close_derived_climb
|
||||
# uv run python -m pytest tests/test_derived_close_proposals.py tests/test_architectural_invariants.py -q
|
||||
# See docs/testing-lanes.md "Recommended determinism / teaching regression invocation".
|
||||
|
||||
.PHONY: test-fast test-slow test-full
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,92 @@
|
|||
# Ratification: Integrate Hardened CLOSE Yardstick into Determinism & Teaching Regression Surfaces
|
||||
|
||||
**Date:** 2026-06-16
|
||||
**Author:** Grok (following user brief)
|
||||
**Branch:** feat/integrate-hardened-close-yardstick-determinism-teaching (freshly created)
|
||||
**Base:** Clean fetched main @7a20356a (post #791 merge; working tree explicitly restored to clean before branch)
|
||||
**Brief reference:** "Title: Integrate Hardened CLOSE Yardstick into Determinism & Teaching Regression Surfaces" (High priority; strict "ratify first" workflow required)
|
||||
|
||||
**Context (verbatim from brief + prior state):**
|
||||
PR #791 merged the hardening of `evals/close_derived_climb` to full Claim B level. It now exercises/measures via *lived* paths:
|
||||
- `ChatRuntime.idle_tick()` + `IdleTickResult.derived_close_proposals_emitted` (real flag gating, not simulation)
|
||||
- Explicit `determine()` calls asserting `Determined(True, rule='direct')` on positive probes post-materialization ("semantic_positives_determined_direct")
|
||||
- `content_replay_checksum` over canonical closure sets (structure_key + Derivation + premise_structure_keys) + proposal bodies (in addition to aggregate replay_checksum)
|
||||
- All original Claim A preserved (1→5→8 monotone/strict growth on is-a + relational-transitive, wrong_total=0, negatives/excluded preds refused, determinism, no serving/ratification mutation, SPECULATIVE/proposal_only only, INV-21/29/30/31 etc.)
|
||||
|
||||
The yardstick (`uv run python -m evals.close_derived_climb` or direct `from evals.close_derived_climb import run`) is currently manual-only. Its value as recurring protection for the CLOSE flywheel (autonomous derived-fact growth + gated proposal emission) is not realized in standard regression flows.
|
||||
|
||||
Immediate post-merge verification (on fresh main) already confirmed the Claim B behaviors live on main.
|
||||
|
||||
## Ratification Decision
|
||||
|
||||
**Before any implementation code was written, the chosen approach is:**
|
||||
|
||||
Execute a *minimal, hermetic, documentation-primary + demo-embedding integration* with exactly these targeted changes (and nothing broader):
|
||||
|
||||
1. **Determinism regression surfaces (docs + test config/scripts):**
|
||||
- Add a prominent "Recommended determinism / teaching regression invocation (post-Claim-B hardening of CLOSE yardstick)" section in `docs/testing-lanes.md` (the single auditable source of truth for lanes, determinism reruns, make targets, hermeticity rules, and "standard verification story").
|
||||
- Document the exact command `uv run python -m evals.close_derived_climb` (plus the contract pytest pair `tests/test_derived_close_proposals.py tests/test_architectural_invariants.py -q`) as a recurring step for determinism reruns, to be run alongside/after `core test --suite ...` or `make test-*` lanes.
|
||||
- Make a tiny, comment-only update in `Makefile` under the test-fast/test-slow/test-full targets (and the header) referencing the CLOSE Claim-B command so operators following the make lanes are pointed at the full verification story. No behavior change to targets.
|
||||
- Result: the hardened yardstick (with its content checksum + semantic + lived IdleTickResult) becomes part of the *recommended* determinism regression flows without any modification to `_TEST_SUITES` in core/cli.py, without new suite entries, without marker changes, and without CLI additions.
|
||||
|
||||
2. **Teaching / anti-regression demo flows (hermetic embedding of the yardstick):**
|
||||
- Edit `evals/anti_regression/run_demo.py` (the implementation behind `core demo anti-regression`, the explicit "teaching/anti-regression demonstration flow" referenced in README, docs/evals/, and cli epilog examples).
|
||||
- Import `run` from `evals.close_derived_climb` (already exposed via __init__).
|
||||
- Execute `climb_report = run()` inside the demo (the climb is fully hermetic: per-run fresh `ChatRuntime(no_load_state=True)`, internal temp only for the flag-proposal sink patch, zero writes to production teaching corpus / engine_state / shared proposal sinks).
|
||||
- Surface the critical Claim-B evidence additively in `DemoReport` / `run_demo` return / `as_dict()` under a new key `"close_derived_climb"` (or nested), including at minimum: `wrong_total`, `proposals_only_with_flag`, `content_replay_checksum`, `semantic_positives_determined_direct` aggregates, and growth numbers.
|
||||
- In the human RESULT section (when not --json), print a one-line summary of the CLOSE flywheel check (e.g. "CLOSE derived climb yardstick (Claim B): wrong_total=0, proposals gated by lived idle_tick+IdleTickResult, content_replay_checksum stable, semantic direct determines passed").
|
||||
- This *adds the yardstick (its execution + its key metrics)* directly into the anti-regression demo flow.
|
||||
- Minor supporting update in `tests/test_anti_regression_demo.py`: add 1-2 assertions on the new key (e.g. `assert "close_derived_climb" in report; assert report["close_derived_climb"]["aggregate"]["wrong_total"] == 0`) so the integration is pinned by the contract tests. Existing 5 tests' assertions remain untouched and will continue to pass.
|
||||
- Why here? The anti-regression demo is the load-bearing "anti-regression demos" surface for teaching-loop protection. CLOSE is the complementary autonomous-growth protection (derived close facts + gated speculative proposals). Embedding it here makes the demo a joint surface without conflating the concerns or touching `teaching/` review/ratification code.
|
||||
|
||||
3. **Update supporting documentation (with cross-references):**
|
||||
- `docs/testing-lanes.md` (as above; also ensure the new section references hermeticity rules, the prior Claim-B ratification, this ratification, contract.md, and the anti_regression_demo.md).
|
||||
- `evals/close_derived_climb/contract.md` (update the "Run:" line to the project-standard `uv run python -m ...`; add "Integrated into determinism lanes (testing-lanes.md) and anti-regression demo (see anti_regression_demo.md + this ratification)" + links).
|
||||
- `docs/evals/anti_regression_demo.md` (add a short "Complementary CLOSE flywheel protection" subsection noting that `core demo anti-regression` now also runs the hardened yardstick and surfaces its invariants; link to contract.md and the two ratifications).
|
||||
- (If natural: a one-sentence cross-ref in `docs/runtime_contracts.md` under the determination surface section noting that semantic answerability for realized derived facts is now exercised by the CLOSE yardstick's `determine(..., rule='direct')` asserts.)
|
||||
- This ratification artifact itself (placed in docs/analysis/ per project pattern for such decisions).
|
||||
- No other dependent references required updates (confirmed via tree-wide grep before ratification; only self + historical analysis ratif + source + git history mentioned the new metrics).
|
||||
|
||||
4. **Hermeticity + safety preservation (non-negotiable, verified at each step):**
|
||||
- Every change respects the hermeticity requirements documented in `docs/testing-lanes.md` (isolation of writers, temp dirs for proposals/logs, no races, byte-identity where claimed).
|
||||
- The CLOSE runner already satisfies this (explicit TemporaryDirectory + DEFAULT_SINK patch only for the one flag test; no persistent side effects).
|
||||
- The anti demo already wraps its ProposalLog in `with tempfile.TemporaryDirectory()`.
|
||||
- Embedding the call does not introduce new shared state or non-determinism.
|
||||
- **Zero** changes to: core engine, `chat/runtime.py`, `generate/*` (except the already-shipped derived_close), `teaching/`, `vault/`, proposal review/ratification, FrameVerdict, `RuntimeConfig` defaults, invariants, serving paths, or any mutation rules.
|
||||
- All existing behavior, `wrong_total=0` guarantee, content-level determinism, and replay checksums preserved.
|
||||
|
||||
**Why this is the *only* correct path (and must not be broadened):**
|
||||
|
||||
- It is the smallest set of changes that satisfies the Objective ("Integrate ... so it runs as part of the standard verification story") and every Success Criterion while obeying every Constraint and the "Required Workflow (Strict)" (ratify artifact *before writing any implementation code*; branch from clean main; stay in Scope's four areas; end with PR containing the mandated summary elements).
|
||||
- It directly addresses the brief's "What’s Next (The Right Path)" recommendation from the immediate post-merge actions: promote into determinism regression runs + teaching/anti-regression demos + make the improved checksum/semantic checks "part of the default verification story".
|
||||
- It makes the protection *recurring and automated* exactly where the project already runs verification (documented lanes + `core demo anti-regression` which is exercised in tests + by humans following the cli examples).
|
||||
- Any broader or different approach would violate constraints:
|
||||
- Adding a new entry to `_TEST_SUITES` + updating cli.py + test_cli_test_suites.py would be a CLI addition (explicitly "Out of Scope") and broad config refactor.
|
||||
- New make targets or changes to lane mechanics would be infrastructure refactoring (out of scope).
|
||||
- Touching teaching/queue/review or core cognition paths would risk invariants and is out of scope.
|
||||
- Duplicating yardstick logic or creating a new harness would waste the already-hardened Claim-B implementation and add non-determinism risk.
|
||||
- Ratifying this exact path *first* (this artifact) guarantees subsequent edits cannot drift; the PR will be auditable against this document.
|
||||
- Highest leverage / lowest surface area: the yardstick code itself is left untouched (its hardening stands on its own); value is realized purely by wiring it into the *existing* surfaces via docs (primary, always-run recommendation) + one hermetic call site in the demo (actual execution during anti-regression verification).
|
||||
|
||||
**Success criteria this ratification guarantees will be true post-implementation (and will be re-verified before PR):**
|
||||
- Hardened yardstick invocable as part of standard determinism regression flows (via updated lanes doc + Makefile comments + core test / make usage patterns).
|
||||
- Referenced and executed inside teaching/anti-regression demonstration paths (`core demo anti-regression` now carries the CLOSE evidence).
|
||||
- All existing tests + architectural invariants + `core test --suite teaching` etc. remain green.
|
||||
- Hermeticity preserved (no new writers, no corpus/engine_state pollution, xdist-safe).
|
||||
- Documentation (testing-lanes, contract, anti_regression_demo.md, this ratif) accurately reflects usage and cross-refs.
|
||||
- The yardstick now provides recurring, automated, Claim-B-strength protection for the CLOSE flywheel inside the project's normal verification story.
|
||||
|
||||
**References (must be linked in PR description):**
|
||||
- This ratification: `docs/analysis/integrate-hardened-close-yardstick-determinism-teaching-regression-ratification-2026-06-16.md`
|
||||
- Prior Claim-B hardening ratification: `docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md`
|
||||
- Yardstick contract + runner: `evals/close_derived_climb/contract.md` + `runner.py`
|
||||
- Anti-regression demo surface: `evals/anti_regression/run_demo.py` + `tests/test_anti_regression_demo.py` + `docs/evals/anti_regression_demo.md`
|
||||
- Lanes / hermeticity: `docs/testing-lanes.md` + `Makefile`
|
||||
- Runtime contracts (determination surface): `docs/runtime_contracts.md`
|
||||
- Post-merge verification commands (to be re-run): `uv run python -m evals.close_derived_climb` + `uv run python -m pytest tests/test_derived_close_proposals.py tests/test_architectural_invariants.py -q`
|
||||
|
||||
**Ratification Status:** COMPLETE. This artifact was written on the branch *before any search_replace, edit, or other implementation code touched source files, tests, scripts, or docs (other than creation of this file itself)*. All exploration was read-only (git, list_dir, read_file, grep).
|
||||
|
||||
Implementation may now proceed, strictly limited to the four areas above, followed by thorough verification (re-running the post-merge commands + targeted core test / demo runs + full relevant pytest) and PR creation with the exact mandated description contents.
|
||||
|
||||
(End of ratification. The single correct path is now locked.)
|
||||
|
|
@ -39,6 +39,18 @@ Here, each gate has a different *kind* of refusal:
|
|||
|
||||
A proposal that fails any one of these never reaches the next.
|
||||
|
||||
## Complementary CLOSE flywheel protection (Claim B)
|
||||
|
||||
This demo is part of the anti-regression / teaching demonstration surfaces. As of the post-#791 integration, `core demo anti-regression` (and `tests/test_anti_regression_demo.py`) also executes the hardened `evals/close_derived_climb` yardstick. This adds recurring execution of the lived-runtime CLOSE autonomous-growth path (real `idle_tick()` + `IdleTickResult.derived_close_proposals_emitted` gating, explicit `determine(..., rule='direct')` semantic asserts on materialized derived facts, and `content_replay_checksum` over canonical closures + proposal bodies) with its own invariants (wrong_total=0, proposal-only/SPECULATIVE, hermetic).
|
||||
|
||||
See:
|
||||
- `evals/close_derived_climb/contract.md`
|
||||
- `docs/testing-lanes.md` (Recommended determinism / teaching regression invocation)
|
||||
- `docs/analysis/integrate-hardened-close-yardstick-determinism-teaching-regression-ratification-2026-06-16.md`
|
||||
- `docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md`
|
||||
|
||||
The three reviewed-teaching gates (S1–S3) and the CLOSE derived-fact growth gates are complementary anti-regression surfaces; both must hold.
|
||||
|
||||
## The synthetic regression in S2
|
||||
|
||||
Scene 2 needs to demonstrate the **rejection lifecycle** deterministically.
|
||||
|
|
|
|||
|
|
@ -97,7 +97,11 @@ Contract:
|
|||
|
||||
When `accrue_realized_knowledge` is enabled and a question turn is **Determined**
|
||||
over realized knowledge (`generate.determine.determine` returns a `Determined`),
|
||||
the user-facing `surface` is the rendered determination
|
||||
the user-facing `surface` is the rendered determination. The hardened
|
||||
CLOSE derived-climb yardstick (`evals/close_derived_climb`) now exercises
|
||||
this path explicitly via `determine()` asserts with `rule='direct'` on
|
||||
materialized derived facts (see "semantic_positives_determined_direct" in
|
||||
its contract and the integration ratification in docs/analysis/).
|
||||
(`generate.determine.render_determination`) — the engine answers *from what it
|
||||
accrued in the conversation*. The basis is rendered **honestly**: SPECULATIVE
|
||||
grounds (today's only case) read as "as I was told", never "verified"; D0 only
|
||||
|
|
|
|||
|
|
@ -85,3 +85,35 @@ it is deferred, not done here.
|
|||
long tail of 1–15s `ChatRuntime` constructions, not outliers. A
|
||||
shared/session-scoped warm-runtime fixture for read-only tests would cut this
|
||||
further.
|
||||
|
||||
## Recommended determinism / teaching regression invocation (post-Claim-B hardening of CLOSE yardstick)
|
||||
|
||||
After any change touching CLOSE flywheel, idle_tick, realize_derived, consolidate_determinations, vault recall of realized facts, determine(), or the derived close proposal bridge, run the hardened yardstick as part of your determinism regression and anti-regression verification:
|
||||
|
||||
```bash
|
||||
uv run python -m evals.close_derived_climb
|
||||
uv run python -m pytest tests/test_derived_close_proposals.py tests/test_architectural_invariants.py -q
|
||||
```
|
||||
|
||||
(Also available via `core demo anti-regression` which now embeds the yardstick — see below.)
|
||||
|
||||
This is the canonical "standard verification story" invocation for the CLOSE autonomous growth surface. It is the direct follow-up to the Claim-B hardening (#791) and makes the improved measurement recurring rather than isolated.
|
||||
|
||||
**What the hardened yardstick now exercises and measures (Claim B):**
|
||||
- Real `ChatRuntime.idle_tick()` + `IdleTickResult.derived_close_proposals_emitted` (proposal flag gating via the lived runtime path, not a simulation).
|
||||
- Explicit `determine()` calls on the post-fixed-point positive probes, asserting `Determined(True, rule='direct')` ("semantic_positives_determined_direct").
|
||||
- `content_replay_checksum` covering canonical closure sets (with structure_key, Derivation, and premise_structure_keys) and full proposal bodies for exact-trajectory fidelity.
|
||||
- Retained Claim A guarantees: strict/monotone growth (1/5/8), wrong_total == 0, negatives and excluded predicates refused, full determinism, hermetic (no serving, no ratification, SPECULATIVE/proposal_only only, all INV-30/31 etc. preserved).
|
||||
|
||||
See:
|
||||
- `evals/close_derived_climb/contract.md` (metrics, scenarios, "no side effects")
|
||||
- `docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md` (the hardening ratification)
|
||||
- `docs/analysis/integrate-hardened-close-yardstick-determinism-teaching-regression-ratification-2026-06-16.md` (this integration ratification + "why only correct path")
|
||||
- `docs/runtime_contracts.md` (determination surface contract exercised by the semantic asserts)
|
||||
- `docs/evals/anti_regression_demo.md` (the anti-regression demo now runs the yardstick)
|
||||
- `tests/test_anti_regression_demo.py` (contract test that pins the embedding)
|
||||
- `Makefile` (comments under test lanes point here)
|
||||
|
||||
The yardstick itself remains hermetic per the rules in this document (fresh runtimes, internal temps only for proposal sink during flag test). It introduces no new writers to engine_state/, teaching/proposals/, or evals reports.
|
||||
|
||||
This integration (documentation promotion into the lanes + hermetic execution inside the anti-regression demo) is the highest-leverage way to ensure the project actually benefits from the hardened Claim-B measurement surface on every relevant regression run.
|
||||
|
|
|
|||
|
|
@ -52,6 +52,13 @@ from teaching.proposals import (
|
|||
propose_from_candidate,
|
||||
)
|
||||
|
||||
# Hermetic integration of the hardened CLOSE derived-climb yardstick (Claim B).
|
||||
# Executed as part of the anti-regression demo flow so that `core demo anti-regression`
|
||||
# (and its contract tests) provide recurring protection for the CLOSE flywheel
|
||||
# (lived idle_tick + IdleTickResult flag, semantic determine(rule='direct'),
|
||||
# content_replay_checksum). The climb runner is fully self-isolating.
|
||||
from evals.close_derived_climb import run as run_close_derived_climb
|
||||
|
||||
|
||||
_VERBOSE = True
|
||||
|
||||
|
|
@ -227,6 +234,11 @@ class DemoReport:
|
|||
scenes: tuple[SceneResult, ...]
|
||||
all_gates_held: bool
|
||||
active_corpus_byte_identical: bool
|
||||
# Additive: the hardened CLOSE derived climb yardstick (Claim B) is run
|
||||
# inside this anti-regression demo as a complementary hermetic protection
|
||||
# surface for autonomous derived-fact growth + gated proposal emission.
|
||||
# See ratification + testing-lanes.md. Never mutates shared state.
|
||||
close_derived_climb: dict[str, Any] | None = None
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
# ``all_claims_supported`` is the canonical cross-demo success
|
||||
|
|
@ -237,6 +249,7 @@ class DemoReport:
|
|||
"scenes": [s.as_dict() for s in self.scenes],
|
||||
"all_gates_held": self.all_gates_held,
|
||||
"active_corpus_byte_identical": self.active_corpus_byte_identical,
|
||||
"close_derived_climb": self.close_derived_climb,
|
||||
"all_claims_supported": (
|
||||
self.all_gates_held and self.active_corpus_byte_identical
|
||||
),
|
||||
|
|
@ -410,10 +423,20 @@ def run_demo(*, emit_json: bool = False) -> dict[str, Any]:
|
|||
and s2.outcome == "auto_rejected_on_regression"
|
||||
and s3.outcome == "pending_awaiting_operator"
|
||||
)
|
||||
|
||||
# Execute the hardened CLOSE derived-climb yardstick (Claim B) here.
|
||||
# This integrates the yardstick into the anti-regression / teaching demo
|
||||
# flows. The call is hermetic (climb uses only fresh runtimes + internal
|
||||
# temps for proposal sink isolation during its flag test; zero production
|
||||
# writes). Key Claim-B signals (lived IdleTickResult, semantic determine
|
||||
# rule=direct, content_replay_checksum) are captured for the report.
|
||||
close_derived_climb = run_close_derived_climb()
|
||||
|
||||
report = DemoReport(
|
||||
scenes=scenes,
|
||||
all_gates_held=all_gates_held,
|
||||
active_corpus_byte_identical=(active_bytes_before == active_bytes_after),
|
||||
close_derived_climb=close_derived_climb,
|
||||
)
|
||||
|
||||
if _VERBOSE:
|
||||
|
|
@ -423,6 +446,17 @@ def run_demo(*, emit_json: bool = False) -> dict[str, Any]:
|
|||
_say("═" * 72)
|
||||
_say(f" all three gates held : {report.all_gates_held}")
|
||||
_say(f" active corpus byte-eq : {report.active_corpus_byte_identical}")
|
||||
# CLOSE flywheel (Claim B) — now part of this anti-regression demo.
|
||||
# Exercises real idle_tick + IdleTickResult.derived_close_proposals_emitted,
|
||||
# semantic determine() with rule='direct', and content-level replay checksum.
|
||||
climb = report.close_derived_climb or {}
|
||||
agg = climb.get("aggregate", {})
|
||||
propf = climb.get("proposal_flag", {})
|
||||
_say(
|
||||
f" CLOSE derived climb (Claim B): wrong_total={agg.get('wrong_total')}, "
|
||||
f"proposals_only_with_flag={propf.get('only_with_flag')}, "
|
||||
f"content_replay_checksum={(climb.get('content_replay_checksum') or '')[:12]}..."
|
||||
)
|
||||
_say()
|
||||
_say(
|
||||
" Each gate is independent and fails closed. Bad proposals "
|
||||
|
|
|
|||
|
|
@ -25,5 +25,7 @@ This lane measures the monotone growth in directly-answerable set enabled by:
|
|||
- Uses real idle_tick path with flags for consolidation and proposal emission.
|
||||
- All realization remains SPECULATIVE; proposals are proposal_only + requires_review.
|
||||
|
||||
Run: python -m evals.close_derived_climb
|
||||
Run: uv run python -m evals.close_derived_climb
|
||||
Replays the exact trajectories (aggregates + full content) for audit. Now qualifies as full lived-runtime Claim B yardstick per post-merge hardening audit.
|
||||
|
||||
Integrated into the project's standard determinism regression and teaching/anti-regression surfaces (see docs/testing-lanes.md "Recommended determinism / teaching regression invocation", docs/evals/anti_regression_demo.md, and the integration ratification docs/analysis/integrate-hardened-close-yardstick-determinism-teaching-regression-ratification-2026-06-16.md). The prior Claim-B hardening ratification is at docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md. Cross-references runtime determination surface in docs/runtime_contracts.md.
|
||||
|
|
|
|||
|
|
@ -61,3 +61,24 @@ def test_active_corpus_never_touched_across_full_demo() -> None:
|
|||
assert report["active_corpus_byte_identical"] is True
|
||||
for scene in report["scenes"]:
|
||||
assert scene["corpus_byte_identical"] is True
|
||||
|
||||
|
||||
def test_close_derived_climb_yardstick_runs_as_part_of_anti_regression_demo() -> None:
|
||||
"""Integration pin: the hardened CLOSE yardstick (Claim B) is now executed
|
||||
inside the anti-regression demo flow. Asserts key invariants from the
|
||||
lived flag + semantic determine + content checksum surface without
|
||||
affecting the three reviewed-teaching gate claims."""
|
||||
report = run_demo(emit_json=True)
|
||||
assert "close_derived_climb" in report
|
||||
climb = report["close_derived_climb"] or {}
|
||||
assert "aggregate" in climb
|
||||
assert climb["aggregate"]["wrong_total"] == 0
|
||||
assert climb.get("content_replay_checksum"), "content-level replay checksum must be present (Claim B)"
|
||||
# Lived flag path exercised
|
||||
assert climb.get("proposal_flag", {}).get("only_with_flag") is True
|
||||
# Semantic determine(rule='direct') on positives exercised in at least one climb
|
||||
sem = sum(
|
||||
(climb.get(k, {}) or {}).get("semantic_positives_determined_direct", 0)
|
||||
for k in ("is_a_climb", "less_than_climb", "before_event_climb")
|
||||
)
|
||||
assert sem >= 1, "expected at least one positive scored via direct determine (Claim B)"
|
||||
|
|
|
|||
Loading…
Reference in a new issue