feat: create dedicated CLOSE Flywheel Regression Surface (Claim-B Level) (#793)
- Makefile: new 'test-close-flywheel' target (clearly named heavy surface; runs full Claim-B yardstick + anti embedding; explicitly not in fast/slow/full). - docs/testing-lanes.md: new prominent 'Dedicated CLOSE Flywheel Regression Surface (Claim-B Level)' section with purpose, invocation (make primary), full Claim-B capabilities list, runtime (~60s+ heavy), hermeticity guarantees, Engineering Pillars alignment (Mechanical Sympathy / Semantic Rigor / Third Door per Whitepaper), and all cross-refs (ratifs, contract, anti demo, etc.). - Cleanly embed / polish (additive, hermetic, building on #792): anti_regression/run_demo.py (comments + RESULT label the surface + make target), test_anti_regression_demo.py (updated docstring), docs/evals/anti_regression_demo.md (how-to + complementary section updated for surface participation). - Supporting: contract.md (dedicated surface note + ratif links), runtime_contracts.md (tightened ref). Ratified first (docs/analysis/close-flywheel-dedicated-regression-surface-ratification-2026-06-16.md). Strictly in scope. Preserves all invariants (wrong_total=0, determinism, proposal-only, etc.). No core, no CLI additions, no fast/CI inclusion, no teaching logic changes. See ratification for pillars justification and 'why only correct path'.
This commit is contained in:
Shay
GitHub
parent
b2b4d79bc0
commit
9736dcc03f
8 changed files with 191 additions and 42 deletions
17
Makefile
17
Makefile
|
|
@ -15,7 +15,7 @@
|
|||
# uv run python -m pytest tests/test_derived_close_proposals.py tests/test_architectural_invariants.py -q
|
||||
# See docs/testing-lanes.md "Recommended determinism / teaching regression invocation".
|
||||
|
||||
.PHONY: test-fast test-slow test-full
|
||||
.PHONY: test-fast test-slow test-full test-close-flywheel
|
||||
|
||||
# Fast dev lane — excludes the slow soak/bench/proof/eval-matrix registry.
|
||||
test-fast:
|
||||
|
|
@ -28,3 +28,18 @@ test-slow:
|
|||
# Full lane — everything except known-failing quarantine (what CI runs).
|
||||
test-full:
|
||||
uv run pytest -m "not quarantine" -q
|
||||
|
||||
# Dedicated CLOSE Flywheel Regression Surface (Claim-B Level)
|
||||
# High-signal, intentional regression target for heavier determinism
|
||||
# regressions and teaching/anti-regression verification flows (not fast
|
||||
# local dev or CI). Exercises the full hardened Claim-B yardstick:
|
||||
# - lived ChatRuntime.idle_tick() + IdleTickResult.derived_close_proposals_emitted
|
||||
# - semantic determine() + Determined(..., rule='direct') asserts
|
||||
# - content_replay_checksum (canonical closures + proposal bodies)
|
||||
# - wrong_total=0, 1/5/8 growth, full determinism, hermeticity
|
||||
# See docs/testing-lanes.md "Dedicated CLOSE Flywheel Regression Surface..."
|
||||
# Explicitly heavy (real runtimes; ~60s+); composable via make; respects
|
||||
# Mechanical Sympathy / Semantic Rigor / Third Door.
|
||||
test-close-flywheel:
|
||||
uv run python -m evals.close_derived_climb
|
||||
uv run python -m pytest tests/test_derived_close_proposals.py tests/test_architectural_invariants.py tests/test_anti_regression_demo.py -q
|
||||
|
|
|
|||
|
|
@ -0,0 +1,87 @@
|
|||
# Ratification: Create a Dedicated CLOSE Flywheel Regression Surface (Claim-B Level)
|
||||
|
||||
**Date:** 2026-06-16
|
||||
**Author:** Grok (following user brief)
|
||||
**Branch:** feat/create-dedicated-close-flywheel-regression-surface (freshly created from clean main)
|
||||
**Base:** Clean fetched main @b2b4d79b (includes post-#792 merge of prior integration work; working tree clean)
|
||||
**Brief reference:** "Title: Create a Dedicated CLOSE Flywheel Regression Surface (Claim-B Level)" (strict "ratify first" workflow; build on #792)
|
||||
|
||||
**Context:**
|
||||
PR #791 hardened `evals/close_derived_climb` to full Claim B (lived `ChatRuntime.idle_tick()` + `IdleTickResult.derived_close_proposals_emitted`, `determine()` + `Determined(..., rule='direct')` semantic asserts on positives, `content_replay_checksum` over canonical closures with structure_key/Derivation/premise keys + proposal bodies; preserved 1/5/8 growth, wrong_total=0, determinism, SPECULATIVE/proposal_only, all invariants).
|
||||
|
||||
PR #792 (the immediate "Integrate..." follow-up) promoted the yardstick via docs recommendation in testing-lanes.md + Makefile comments + hermetic embedding into `evals/anti_regression/run_demo.py` (and its contract test) so `core demo anti-regression` now executes and reports the Claim-B signals. This made it "recurring" in documented determinism reruns and the anti-regression teaching demo flow.
|
||||
|
||||
This brief asks for the *next* step: elevate from "recommended invocation" to a **dedicated, clearly named, intentional regression surface** for the CLOSE flywheel at Claim-B level. It must be positioned for heavier determinism regressions and teaching/anti-regression verification (not fast local dev or CI), align with the project's Engineering Pillars (Mechanical Sympathy, Semantic Rigor, Third Door per Whitepaper.md), respect hermeticity/composable lanes, and stay strictly in scope.
|
||||
|
||||
The yardstick already exists and is proven (via `python -m evals.close_derived_climb`, the contract tests, and the #792 embedding). The task is naming, documenting, and surfacing it as a coherent target.
|
||||
|
||||
## Ratification Decision (chosen approach, justified before any implementation)
|
||||
|
||||
**Before any implementation code or edits to source/docs (other than creation of this artifact), the single correct path is:**
|
||||
|
||||
Introduce a **dedicated CLOSE Flywheel regression surface** via a minimal, composable, explicitly heavy "lane" construct that reifies the full Claim-B yardstick invocation as a first-class, intentional target. Concretely:
|
||||
|
||||
1. **Define the dedicated surface (clearly named invocation for heavier flows):**
|
||||
- Add a new explicit make target `test-close-flywheel` (plus .PHONY) in `Makefile`. The target runs the complete Claim-B yardstick:
|
||||
```
|
||||
uv run python -m evals.close_derived_climb
|
||||
uv run python -m pytest tests/test_derived_close_proposals.py tests/test_architectural_invariants.py tests/test_anti_regression_demo.py -q
|
||||
```
|
||||
(The inclusion of the anti test ensures the #792 hermetic embedding participates.)
|
||||
- Name it intentionally ("CLOSE Flywheel" / "test-close-flywheel"). Document in the target comment and help text its purpose as the high-signal Claim-B regression surface.
|
||||
- **Positioning:** Explicitly heavy / opt-in only. Do *not* wire it into `test-fast`, `test-slow`, `test-full`, any `-m` marker, `core test --suite`, CI workflows, or generic pytest collection. It lives alongside the existing lanes as a composable heavy-verification tool (for use after CLOSE-related changes in determinism reruns or teaching/anti-regression verification). This satisfies "not fast local or CI runs" and out-of-scope constraints while giving a "clearly named and documented way to invoke".
|
||||
|
||||
2. **Integrate into existing high-value flows (clean, hermetic, additive, building directly on #792):**
|
||||
- The #792 embedding of the yardstick execution inside `evals/anti_regression/run_demo.py` (the teaching anti-regression demo) + pinning in its test is the primary "high-value flow" integration point. It already pulls the full Claim-B (lived flag, semantic determine, content checksum) into the demo that exercises reviewed teaching gates.
|
||||
- Make small, purely additive, hermetic polish if needed to "cleanly embed" (e.g. enhance comments/labels in the embedding and RESULT output to explicitly reference "CLOSE Flywheel Regression Surface (Claim-B)" and the new make target; ensure the `close_derived_climb` report field and human output continue to surface the key signals; no behavior, dataclass, or logic changes).
|
||||
- Optionally add cross-references or a one-line hermetic mention in one related teaching verification path (e.g. a comment or docstring in `tests/test_reviewed_teaching_loop.py` or `docs/evals/anti_regression_demo.md` "How to reproduce" section) that operators running teaching anti-regression verification should also consider `make test-close-flywheel`. Keep strictly additive/hermetic — no new calls that could introduce side effects or non-determinism.
|
||||
- The surface "integrates" by *naming and elevating* the existing #792 embedding as part of the dedicated target (the anti test is part of what the make target runs).
|
||||
|
||||
3. **Update documentation (prominently and completely):**
|
||||
- In `docs/testing-lanes.md`: Elevate or replace the prior "# Recommended determinism / teaching regression invocation..." section with a new top-level or clearly headed section titled **"Dedicated CLOSE Flywheel Regression Surface (Claim-B Level)"** (or equivalent). Include:
|
||||
- Purpose: high-signal, intentional regression target for the full lived CLOSE flywheel (autonomous derived-fact growth + gated proposal emission) at Claim-B strength.
|
||||
- Invocation: `make test-close-flywheel` (primary named surface) or the equivalent `uv run python -m ...` commands.
|
||||
- Claim-B capabilities: exact list (real idle_tick + IdleTickResult.derived_close_proposals_emitted for proposal flag; explicit determine() asserts with rule='direct' for semantic_positives_determined_direct; content_replay_checksum on canonical closures + proposal bodies; retained wrong_total=0, 1/5/8 growth, determinism, etc.).
|
||||
- Expected runtime characteristics: heavyweight (~60s+ on 10-core mac, driven by multiple real ChatRuntime turns + climbs to fixed point; comparable to other proof-scale inner-loop fixtures); explicitly for heavier determinism regressions and teaching/anti-regression verification flows.
|
||||
- Hermeticity guarantees: fresh per-run ChatRuntime (no_load_state), internal TemporaryDirectory only for proposal sink during flag test, zero writes to engine_state/ / active teaching corpus / shared proposal sinks / evals reports; preserves all invariants (wrong_total=0, replayability, SPECULATIVE-only, proposal-only boundaries, INV-21/29/30/31, versor_condition, etc.).
|
||||
- Alignment with Engineering Pillars (Whitepaper.md §IV): Mechanical Sympathy (respects cost model by keeping it out of fast paths and default CI; explicit opt-in for heavy work), Semantic Rigor (precise named surface with non-negotiable Claim-B contract and capabilities; no fuzzy or approximate inclusion), Third Door (neither pollutes existing generic suites/CI nor adds new CLI commands/heavy infra; instead a minimal composable make lane + docs reification built from first principles of the project's lane model).
|
||||
- References: this ratification artifact, the #791 Claim-B hardening ratification, the #792 integration ratification, `evals/close_derived_climb/contract.md`, `docs/evals/anti_regression_demo.md`, `docs/runtime_contracts.md`, prior analysis docs on CLOSE, the anti-regression test, and the make target.
|
||||
- Update `Makefile` comments (already partially present from #792) to reference the new dedicated target and surface name.
|
||||
- Update supporting docs for accuracy and refs: `docs/evals/anti_regression_demo.md` (how-to and falsifiable claims sections to note the surface/make target and that the demo participates in it), `evals/close_derived_climb/contract.md` (add "Dedicated surface" note + link to testing-lanes section + this ratif), and any cross-refs in `docs/runtime_contracts.md` if the determination surface mention needs tightening. Keep changes minimal and reference-focused.
|
||||
- The surface name and description make the "what/why/how/hermeticity" first-class and auditable.
|
||||
|
||||
4. **Preserve all invariants and boundaries (non-negotiable):**
|
||||
- The make target and any doc updates are pure invocation + description. They call only already-shipped hermetic entrypoints (`python -m evals...` which asserts internally, and the anti demo which already embeds without mutation).
|
||||
- No edits to core/ (chat/runtime.py, generate/*, session/*, vault/*, etc.), no changes to proposal review/teaching logic, FrameVerdict, RuntimeConfig, closed-world reasoning, or any serving/ratification paths.
|
||||
- All prior guarantees (wrong_total=0 across scenarios, content/replay checksum stability, lived flag isolation only when flag enabled, semantic direct rule, proposal bodies with status="proposal_only" + requires_review + epistemic="speculative", no corpus mutation, byte-identical active state in anti demo, etc.) remain enforced by the yardstick and #792 embedding.
|
||||
- Hermeticity per testing-lanes.md rules is upheld (and documented as a property of the surface).
|
||||
|
||||
**Why this is the only correct path (and must not be broadened):**
|
||||
|
||||
- It directly satisfies the Objective ("clear, intentional, and high-signal regression surface" that "properly exercises the hardened Claim-B yardstick") and every Success Criterion while obeying every Constraint, Out-of-Scope item, and the Engineering Pillars.
|
||||
- **Mechanical Sympathy:** Explicitly treats the yardstick as heavyweight real-runtime work (multiple ChatRuntime + idle_tick to fixed point + climbs). Does not fight the cost model by sneaking it into fast lanes, generic suites, or CI.
|
||||
- **Semantic Rigor:** Gives the Claim-B behaviors a *precise name and contract* ("Dedicated CLOSE Flywheel Regression Surface (Claim-B Level)") with enumerated capabilities, runtime expectations, and hermeticity guarantees. No vague "also run this sometimes."
|
||||
- **Third Door:** Rejects the two obvious doors (1. add to existing fast/full/slow or a generic "determinism" suite, which would violate positioning + out-of-scope + mechanical sympathy; 2. create new CLI command / heavy test infrastructure like a new core subcommand or pytest plugin or broad suite wiring). Instead: a minimal named make target (already the project's composable lane mechanism per testing-lanes + Makefile) + authoritative documentation. Composable, intentional, built from the project's own lane philosophy.
|
||||
- Strictly inside Scope: defines the named surface (make target + docs section), integrates cleanly/additively into anti-regression (the high-value teaching path, building on #792 embedding), updates the required docs with all specified content (purpose, Claim-B list, runtime, hermeticity, pillar alignment, ratif/contract refs), preserves every invariant (no core/teaching/FrameVerdict changes).
|
||||
- Avoids every Out-of-Scope item: no fast/generic suite inclusion, no "on every push/CI", no new CLI commands, no modification of proposal review/teaching logic/closed-world, no broad refactoring.
|
||||
- Highest long-term value with lowest risk: the surface is discoverable and intentional exactly where heavy CLOSE flywheel verification belongs (post-change determinism reruns + teaching anti-regression flows). The #792 embedding already provides the "integrate into existing high-value flows"; this task names and elevates the whole thing.
|
||||
- Any other path (new `core test --suite close-flywheel` entry in cli.py + test_cli_test_suites.py, adding the evals to an existing suite, creating a scripts/ wrapper as "heavy infra", modifying conftest markers, running inside test-full, or changes that touch teaching code) would either violate the "Creating new CLI commands or heavy test infrastructure", "not ... on every push/CI", "stay strictly within Scope", "Prioritize clarity... over maximal... inclusion", or "No changes to core... proposal review" constraints — or would dilute the intentionality that the brief demands.
|
||||
- Ratifying this exact path first ensures implementation cannot drift; every edit will be traceable to "the dedicated surface is the make target + the named section in testing-lanes.md that exercises the full yardstick (including via the anti demo)".
|
||||
|
||||
**References (to be linked in PR):**
|
||||
- This ratification: `docs/analysis/close-flywheel-dedicated-regression-surface-ratification-2026-06-16.md`
|
||||
- #791 Claim-B hardening: `docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md`
|
||||
- #792 integration: `docs/analysis/integrate-hardened-close-yardstick-determinism-teaching-regression-ratification-2026-06-16.md` + PR #792
|
||||
- Yardstick: `evals/close_derived_climb/{contract.md, runner.py, __main__.py}`
|
||||
- Lanes + hermeticity: `docs/testing-lanes.md` (will host the dedicated surface section) + `Makefile`
|
||||
- Anti-regression (the integration flow): `evals/anti_regression/run_demo.py` + `tests/test_anti_regression_demo.py` + `docs/evals/anti_regression_demo.md`
|
||||
- Pillars: `docs/Whitepaper.md` §"IV. The Three Engineering Pillars"
|
||||
- Related: `docs/runtime_contracts.md`, inner-loop determinism discussion in testing-lanes.md, `tests/test_derived_close_proposals.py`, `tests/test_architectural_invariants.py`
|
||||
|
||||
**Ratification Status:** COMPLETE AND LOCKED. This artifact was written on the clean branch *before any search_replace, write (other than this ratif), or other implementation edit to Makefile, testing-lanes.md, anti_regression code/docs, contract.md, or any other file*. All prior steps were git + read-only exploration (list_dir, read_file, grep).
|
||||
|
||||
Implementation will now proceed *strictly* to the four Scope bullets using exactly the approach above. After edits: full verification (re-run the yardstick, new make target, anti demo, derived contract tests, confirm green + invariants + hermeticity + pillar alignment), then PR with the mandated elements.
|
||||
|
||||
The surface will embody Mechanical Sympathy (heavy, explicit), Semantic Rigor (precisely named Claim-B contract), and Third Door (composable lane via existing mechanisms, not the obvious two paths).
|
||||
|
||||
(End of ratification. No implementation code has been written.)
|
||||
|
|
@ -41,15 +41,18 @@ A proposal that fails any one of these never reaches the next.
|
|||
|
||||
## Complementary CLOSE flywheel protection (Claim B)
|
||||
|
||||
This demo is part of the anti-regression / teaching demonstration surfaces. As of the post-#791 integration, `core demo anti-regression` (and `tests/test_anti_regression_demo.py`) also executes the hardened `evals/close_derived_climb` yardstick. This adds recurring execution of the lived-runtime CLOSE autonomous-growth path (real `idle_tick()` + `IdleTickResult.derived_close_proposals_emitted` gating, explicit `determine(..., rule='direct')` semantic asserts on materialized derived facts, and `content_replay_checksum` over canonical closures + proposal bodies) with its own invariants (wrong_total=0, proposal-only/SPECULATIVE, hermetic).
|
||||
This demo is part of the anti-regression / teaching demonstration surfaces and participates in the **Dedicated CLOSE Flywheel Regression Surface (Claim-B Level)** (see `make test-close-flywheel` and `docs/testing-lanes.md` "Dedicated CLOSE Flywheel Regression Surface...").
|
||||
|
||||
As of the post-#791 / #792 work, `core demo anti-regression` (and `tests/test_anti_regression_demo.py`) executes the hardened `evals/close_derived_climb` yardstick. This adds the full Claim-B lived-runtime CLOSE autonomous-growth path (real `idle_tick()` + `IdleTickResult.derived_close_proposals_emitted` gating, explicit `determine(..., rule='direct')` semantic asserts on materialized derived facts, and `content_replay_checksum` over canonical closures + proposal bodies) with its own invariants (wrong_total=0, proposal-only/SPECULATIVE, hermetic, determinism).
|
||||
|
||||
See:
|
||||
- `evals/close_derived_climb/contract.md`
|
||||
- `docs/testing-lanes.md` (Recommended determinism / teaching regression invocation)
|
||||
- `docs/testing-lanes.md` (Dedicated CLOSE Flywheel Regression Surface section + pillar alignment)
|
||||
- `docs/analysis/close-flywheel-dedicated-regression-surface-ratification-2026-06-16.md`
|
||||
- `docs/analysis/integrate-hardened-close-yardstick-determinism-teaching-regression-ratification-2026-06-16.md`
|
||||
- `docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md`
|
||||
|
||||
The three reviewed-teaching gates (S1–S3) and the CLOSE derived-fact growth gates are complementary anti-regression surfaces; both must hold.
|
||||
The three reviewed-teaching gates (S1–S3) and the CLOSE derived-fact growth gates are complementary anti-regression surfaces; both must hold. Running the dedicated surface also verifies the integrated embedding.
|
||||
|
||||
## The synthetic regression in S2
|
||||
|
||||
|
|
@ -145,9 +148,15 @@ Scenes 1 and 3 both use the real production replay function.
|
|||
```bash
|
||||
core demo anti-regression # human output (preamble + scenes + result)
|
||||
core demo anti-regression --json # machine-readable DemoReport
|
||||
python -m pytest tests/test_anti_regression_demo.py -q # ~10s
|
||||
python -m pytest tests/test_anti_regression_demo.py -q # ~60s+ (includes CLOSE yardstick)
|
||||
```
|
||||
|
||||
The demo participates in the Dedicated CLOSE Flywheel Regression Surface (Claim-B Level):
|
||||
```bash
|
||||
make test-close-flywheel
|
||||
```
|
||||
See `docs/testing-lanes.md` "Dedicated CLOSE Flywheel Regression Surface..." for the full surface definition, Claim-B capabilities, runtime, hermeticity, and pillar alignment.
|
||||
|
||||
## Falsifiable claims
|
||||
|
||||
If any of these stops holding, the demo's headline no longer holds:
|
||||
|
|
|
|||
|
|
@ -101,7 +101,8 @@ the user-facing `surface` is the rendered determination. The hardened
|
|||
CLOSE derived-climb yardstick (`evals/close_derived_climb`) now exercises
|
||||
this path explicitly via `determine()` asserts with `rule='direct'` on
|
||||
materialized derived facts (see "semantic_positives_determined_direct" in
|
||||
its contract and the integration ratification in docs/analysis/).
|
||||
its contract and the Dedicated CLOSE Flywheel Regression Surface in
|
||||
docs/testing-lanes.md + docs/analysis/close-flywheel-dedicated-regression-surface-ratification-2026-06-16.md).
|
||||
(`generate.determine.render_determination`) — the engine answers *from what it
|
||||
accrued in the conversation*. The basis is rendered **honestly**: SPECULATIVE
|
||||
grounds (today's only case) read as "as I was told", never "verified"; D0 only
|
||||
|
|
|
|||
|
|
@ -86,34 +86,64 @@ it is deferred, not done here.
|
|||
shared/session-scoped warm-runtime fixture for read-only tests would cut this
|
||||
further.
|
||||
|
||||
## Recommended determinism / teaching regression invocation (post-Claim-B hardening of CLOSE yardstick)
|
||||
## Dedicated CLOSE Flywheel Regression Surface (Claim-B Level)
|
||||
|
||||
After any change touching CLOSE flywheel, idle_tick, realize_derived, consolidate_determinations, vault recall of realized facts, determine(), or the derived close proposal bridge, run the hardened yardstick as part of your determinism regression and anti-regression verification:
|
||||
This is the **clearly named, intentional, high-signal regression surface** for the CLOSE flywheel at full Claim-B strength. It is positioned exclusively for heavier determinism regressions and teaching/anti-regression verification flows — **not** for fast local development or default CI runs.
|
||||
|
||||
### Invocation (the dedicated surface)
|
||||
```bash
|
||||
uv run python -m evals.close_derived_climb
|
||||
uv run python -m pytest tests/test_derived_close_proposals.py tests/test_architectural_invariants.py -q
|
||||
make test-close-flywheel
|
||||
```
|
||||
|
||||
(Also available via `core demo anti-regression` which now embeds the yardstick — see below.)
|
||||
Or the equivalent explicit commands (the make target is the canonical named surface):
|
||||
```bash
|
||||
uv run python -m evals.close_derived_climb
|
||||
uv run python -m pytest tests/test_derived_close_proposals.py tests/test_architectural_invariants.py tests/test_anti_regression_demo.py -q
|
||||
```
|
||||
|
||||
This is the canonical "standard verification story" invocation for the CLOSE autonomous growth surface. It is the direct follow-up to the Claim-B hardening (#791) and makes the improved measurement recurring rather than isolated.
|
||||
The inclusion of the anti-regression test ensures the hermetic embedding of the yardstick (from #792) participates, so the surface exercises both the direct yardstick and the integrated teaching demo path.
|
||||
|
||||
**What the hardened yardstick now exercises and measures (Claim B):**
|
||||
- Real `ChatRuntime.idle_tick()` + `IdleTickResult.derived_close_proposals_emitted` (proposal flag gating via the lived runtime path, not a simulation).
|
||||
- Explicit `determine()` calls on the post-fixed-point positive probes, asserting `Determined(True, rule='direct')` ("semantic_positives_determined_direct").
|
||||
- `content_replay_checksum` covering canonical closure sets (with structure_key, Derivation, and premise_structure_keys) and full proposal bodies for exact-trajectory fidelity.
|
||||
- Retained Claim A guarantees: strict/monotone growth (1/5/8), wrong_total == 0, negatives and excluded predicates refused, full determinism, hermetic (no serving, no ratification, SPECULATIVE/proposal_only only, all INV-30/31 etc. preserved).
|
||||
### Purpose
|
||||
Provide a coherent, auditable regression target that exercises the full lived CLOSE flywheel (comprehend → realize → determine → CLOSE consolidate → proposal emission under the review_derived_close_proposals flag → measured climb) using the hardened Claim-B yardstick (`evals/close_derived_climb`). This surface makes the improved measurement (post-#791) a first-class, recurring part of heavy verification without polluting fast or generic lanes.
|
||||
|
||||
See:
|
||||
- `evals/close_derived_climb/contract.md` (metrics, scenarios, "no side effects")
|
||||
- `docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md` (the hardening ratification)
|
||||
- `docs/analysis/integrate-hardened-close-yardstick-determinism-teaching-regression-ratification-2026-06-16.md` (this integration ratification + "why only correct path")
|
||||
- `docs/runtime_contracts.md` (determination surface contract exercised by the semantic asserts)
|
||||
- `docs/evals/anti_regression_demo.md` (the anti-regression demo now runs the yardstick)
|
||||
- `tests/test_anti_regression_demo.py` (contract test that pins the embedding)
|
||||
- `Makefile` (comments under test lanes point here)
|
||||
### What the surface exercises and measures (Claim B)
|
||||
- Real `ChatRuntime.idle_tick()` + `IdleTickResult.derived_close_proposals_emitted` (proposal flag gating via the *lived* runtime path, not simulation).
|
||||
- Explicit `determine()` calls on post-fixed-point positive probes, asserting `Determined(True, rule='direct')` ("semantic_positives_determined_direct").
|
||||
- `content_replay_checksum` covering canonical closure sets (structure_key + Derivation with premise_structure_keys) and proposal bodies for exact-trajectory fidelity.
|
||||
- Retained Claim A guarantees: strict/monotone growth (1→5→8 on is-a + relational-transitive scenarios), `wrong_total == 0` (negatives and excluded predicates refused), full determinism and replayability, hermetic execution (no serving, no ratification, SPECULATIVE-only realization, proposal-only boundaries, all INV-21/29/30/31 etc. preserved).
|
||||
|
||||
The yardstick itself remains hermetic per the rules in this document (fresh runtimes, internal temps only for proposal sink during flag test). It introduces no new writers to engine_state/, teaching/proposals/, or evals reports.
|
||||
Scenarios: is-a (member/subset) climb, less_than relational climb, before_event temporal climb, parent/sibling negatives refused.
|
||||
|
||||
This integration (documentation promotion into the lanes + hermetic execution inside the anti-regression demo) is the highest-leverage way to ensure the project actually benefits from the hardened Claim-B measurement surface on every relevant regression run.
|
||||
### Expected runtime characteristics
|
||||
Heavyweight (~60s+ on 10-core macOS with CORE_BACKEND=numpy; driven by multiple real `ChatRuntime` constructions + idle_tick to fixed point + climbs). Comparable to other proof-scale fixtures (e.g. the 975s inner-loop phase2 outlier). Intended for deliberate, post-change verification in heavier determinism reruns and teaching/anti-regression flows — not for rapid iteration or every push.
|
||||
|
||||
### Hermeticity guarantees
|
||||
- Fresh `ChatRuntime(no_load_state=True)` per scenario.
|
||||
- Internal `TemporaryDirectory` only for proposal sink isolation during the flag test (DEFAULT_SINK patch is restored).
|
||||
- Zero writes to `engine_state/`, active teaching corpus, `teaching/proposals/`, or shared evals reports.
|
||||
- All existing anti-regression demo guarantees (active corpus byte-identical pre/post) continue to hold.
|
||||
- Complies with the hermeticity rules in this document (see "Follow-ups (separate PRs)" and xdist notes). The surface itself introduces no new race surfaces.
|
||||
|
||||
### Alignment with Engineering Pillars (Whitepaper.md §IV)
|
||||
- **Mechanical Sympathy**: Understands and respects the cost model. The surface does real runtime work (ChatRuntime turns, consolidation, derivation). It is kept out of fast paths, generic suites, and CI so the machine is not forced to pay the price for every build.
|
||||
- **Semantic Rigor**: The surface has a precise, non-negotiable name and contract. "CLOSE Flywheel (Claim-B)" means exactly the lived behaviors listed above — no approximations, no "good enough" inclusion, no silent embedding of the yardstick into unrelated lanes. Every term (IdleTickResult, rule='direct', content_replay_checksum, proposal_only, etc.) retains its defined meaning.
|
||||
- **Third Door**: The world offered two obvious doors — (1) add the yardstick to fast/full/slow or a generic "determinism" suite (violates positioning, mechanical sympathy, and out-of-scope rules), or (2) invent a new CLI command or heavy test infrastructure. This surface takes the third door: a minimal, composable `make test-close-flywheel` target (using the project's existing lane mechanism) + authoritative documentation that elevates the yardstick and the #792 embedding into a named, intentional regression surface built from first principles.
|
||||
|
||||
See the ratification for the full justification of why this (and only this) approach satisfies the brief while aligning with the pillars:
|
||||
- `docs/analysis/close-flywheel-dedicated-regression-surface-ratification-2026-06-16.md`
|
||||
|
||||
### References
|
||||
- Yardstick contract + implementation: `evals/close_derived_climb/contract.md` (run with `uv run python -m ...`; metrics, scenarios, "no side effects")
|
||||
- Claim-B hardening: `docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md` (#791)
|
||||
- Prior integration (foundation for the embedding): `docs/analysis/integrate-hardened-close-yardstick-determinism-teaching-regression-ratification-2026-06-16.md` (#792)
|
||||
- Anti-regression demo (the primary high-value teaching/anti-regression flow that now participates in the surface): `docs/evals/anti_regression_demo.md`, `evals/anti_regression/run_demo.py`, `tests/test_anti_regression_demo.py`
|
||||
- Determination surface exercised by the semantic asserts: `docs/runtime_contracts.md`
|
||||
- `Makefile` (the `test-close-flywheel` target is the named entry point)
|
||||
- Related heavy determinism context: the "Finding: the 975s `test_inner_loop_phase2` outlier" and inner-loop suite discussion above
|
||||
|
||||
The surface remains hermetic and additive. All prior invariants and the #792 embedding are unchanged.
|
||||
|
||||
### How the surface builds on prior work
|
||||
The #792 "recommended invocation" and hermetic embedding into the anti-regression demo provided the recurring protection. This dedicated surface gives that work a clear name, a primary make target, full pillar-aligned documentation, and explicit positioning as the heavy CLOSE flywheel regression lane. The anti-regression test is deliberately included in the surface so that running `make test-close-flywheel` also verifies the integrated teaching path.
|
||||
|
||||
(Operators doing heavy CLOSE-related work after #788/#789/#791 should run this surface as part of their determinism and teaching verification.)
|
||||
|
|
|
|||
|
|
@ -424,12 +424,13 @@ def run_demo(*, emit_json: bool = False) -> dict[str, Any]:
|
|||
and s3.outcome == "pending_awaiting_operator"
|
||||
)
|
||||
|
||||
# Execute the hardened CLOSE derived-climb yardstick (Claim B) here.
|
||||
# This integrates the yardstick into the anti-regression / teaching demo
|
||||
# flows. The call is hermetic (climb uses only fresh runtimes + internal
|
||||
# temps for proposal sink isolation during its flag test; zero production
|
||||
# writes). Key Claim-B signals (lived IdleTickResult, semantic determine
|
||||
# rule=direct, content_replay_checksum) are captured for the report.
|
||||
# Execute the hardened CLOSE derived-climb yardstick (Claim B) here as part
|
||||
# of the Dedicated CLOSE Flywheel Regression Surface (Claim-B Level).
|
||||
# This cleanly embeds (additive, hermetic) the full surface into the
|
||||
# anti-regression / teaching demo flows (building on #792). The call uses
|
||||
# only fresh runtimes + internal temps; zero production writes.
|
||||
# See docs/testing-lanes.md "Dedicated CLOSE Flywheel..." and
|
||||
# `make test-close-flywheel`.
|
||||
close_derived_climb = run_close_derived_climb()
|
||||
|
||||
report = DemoReport(
|
||||
|
|
@ -446,14 +447,16 @@ def run_demo(*, emit_json: bool = False) -> dict[str, Any]:
|
|||
_say("═" * 72)
|
||||
_say(f" all three gates held : {report.all_gates_held}")
|
||||
_say(f" active corpus byte-eq : {report.active_corpus_byte_identical}")
|
||||
# CLOSE flywheel (Claim B) — now part of this anti-regression demo.
|
||||
# Exercises real idle_tick + IdleTickResult.derived_close_proposals_emitted,
|
||||
# semantic determine() with rule='direct', and content-level replay checksum.
|
||||
# CLOSE Flywheel Regression Surface (Claim-B Level) — executed here as
|
||||
# part of the anti-regression demo (see `make test-close-flywheel` and
|
||||
# docs/testing-lanes.md "Dedicated CLOSE Flywheel...").
|
||||
# Exercises lived idle_tick + IdleTickResult flag, semantic determine
|
||||
# (rule='direct'), and content_replay_checksum.
|
||||
climb = report.close_derived_climb or {}
|
||||
agg = climb.get("aggregate", {})
|
||||
propf = climb.get("proposal_flag", {})
|
||||
_say(
|
||||
f" CLOSE derived climb (Claim B): wrong_total={agg.get('wrong_total')}, "
|
||||
f" CLOSE Flywheel Regression Surface (Claim B): wrong_total={agg.get('wrong_total')}, "
|
||||
f"proposals_only_with_flag={propf.get('only_with_flag')}, "
|
||||
f"content_replay_checksum={(climb.get('content_replay_checksum') or '')[:12]}..."
|
||||
)
|
||||
|
|
|
|||
|
|
@ -28,4 +28,6 @@ This lane measures the monotone growth in directly-answerable set enabled by:
|
|||
Run: uv run python -m evals.close_derived_climb
|
||||
Replays the exact trajectories (aggregates + full content) for audit. Now qualifies as full lived-runtime Claim B yardstick per post-merge hardening audit.
|
||||
|
||||
Integrated into the project's standard determinism regression and teaching/anti-regression surfaces (see docs/testing-lanes.md "Recommended determinism / teaching regression invocation", docs/evals/anti_regression_demo.md, and the integration ratification docs/analysis/integrate-hardened-close-yardstick-determinism-teaching-regression-ratification-2026-06-16.md). The prior Claim-B hardening ratification is at docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md. Cross-references runtime determination surface in docs/runtime_contracts.md.
|
||||
**Dedicated surface:** `make test-close-flywheel` (or the equivalent python -m + pytest commands). This is the clearly named, intentional Claim-B regression surface for heavier determinism regressions and teaching/anti-regression verification flows. See the full definition, capabilities, runtime characteristics, hermeticity guarantees, and Engineering Pillars alignment in `docs/testing-lanes.md` "Dedicated CLOSE Flywheel Regression Surface (Claim-B Level)".
|
||||
|
||||
Integrated into the project's standard determinism regression and teaching/anti-regression surfaces via the anti-regression demo embedding (see docs/evals/anti_regression_demo.md and `tests/test_anti_regression_demo.py`). See ratification docs/analysis/close-flywheel-dedicated-regression-surface-ratification-2026-06-16.md (this task), docs/analysis/integrate-hardened-close-yardstick-determinism-teaching-regression-ratification-2026-06-16.md (#792), and docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md (#791). Cross-references runtime determination surface in docs/runtime_contracts.md.
|
||||
|
|
|
|||
|
|
@ -64,10 +64,12 @@ def test_active_corpus_never_touched_across_full_demo() -> None:
|
|||
|
||||
|
||||
def test_close_derived_climb_yardstick_runs_as_part_of_anti_regression_demo() -> None:
|
||||
"""Integration pin: the hardened CLOSE yardstick (Claim B) is now executed
|
||||
inside the anti-regression demo flow. Asserts key invariants from the
|
||||
lived flag + semantic determine + content checksum surface without
|
||||
affecting the three reviewed-teaching gate claims."""
|
||||
"""Integration pin (Dedicated CLOSE Flywheel Regression Surface, Claim B):
|
||||
the full hardened yardstick is executed inside the anti-regression demo flow
|
||||
(via `make test-close-flywheel`). Asserts key invariants from the lived flag
|
||||
+ semantic determine + content checksum surface without affecting the three
|
||||
reviewed-teaching gate claims. See docs/testing-lanes.md dedicated surface
|
||||
section and the ratification."""
|
||||
report = run_demo(emit_json=True)
|
||||
assert "close_derived_climb" in report
|
||||
climb = report["close_derived_climb"] or {}
|
||||
|
|
|
|||
Loading…
Reference in a new issue