diff --git a/docs/analysis/close-flywheel-proposal-review-visibility-ratification-2026-06-16.md b/docs/analysis/close-flywheel-proposal-review-visibility-ratification-2026-06-16.md new file mode 100644 index 00000000..81c224bf --- /dev/null +++ b/docs/analysis/close-flywheel-proposal-review-visibility-ratification-2026-06-16.md @@ -0,0 +1,120 @@ +# Ratification: Strengthen Visibility and Measurement of the Proposal Review / Ratification Side of the CLOSE Flywheel + +**Date:** 2026-06-16 +**Branch:** `feat/strengthen-close-flywheel-proposal-review-visibility` (fresh from `origin/main` at 9736dcc0 which includes #793) +**Governing brief:** "Title: Strengthen Visibility and Measurement of the Proposal Review / Ratification Side of the CLOSE Flywheel" (the query for this session) +**Preceding work (locked):** #791 (Claim-B hardening of `evals/close_derived_climb`), #792 (integration recommendation + embedding), #793 (Dedicated CLOSE Flywheel Regression Surface via `make test-close-flywheel` + dedicated section in testing-lanes.md + hermetic anti-regression participation). + +## Context and the Gap + +The CLOSE flywheel (comprehend → realize_derived → determine → CLOSE consolidate / idle_tick → proposal emission under `review_derived_close_proposals`) now has strong, Claim-B measurement on the *derivation and autonomous growth* half: + +- Lived `ChatRuntime.idle_tick()` + `IdleTickResult.derived_close_proposals_emitted` +- `proposal_flag` + `only_with_flag` guarantee (proposals emitted **only** when the flag is on; zero without) +- `content_replay_checksum` over canonical closures (structure_key + Derivation with premise_structure_keys) + proposal bodies +- `semantic_positives_determined_direct` via explicit post-fixed-point `determine(..., rule='direct')` asserts +- `wrong_total=0`, strict/monotone growth, proposal_only + SPECULATIVE + requires_review posture on every emitted artifact, hermetic fresh_ctx runs + +(See `evals/close_derived_climb/runner.py`, `evals/close_derived_climb/contract.md`, `tests/test_derived_close_proposals.py`, `docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md`, the dedicated surface ratification, and the bridge ADR doc `close-derived-proposal-bridge-2026-06-16.md`.) + +The **review / ratification half** (the operator/HITL-gated path that takes a `status=proposal_only / epistemic_status=speculative / requires_review=true` artifact and moves it to accepted → corpus append or coherent promotion) has been the weaker half for observability: + +- Emission artifacts (in `teaching/proposals/derived_close_facts/*.json` for the bridge, or via `ProposalLog` events for the older chain path) carry explicit review posture fields, but these are not aggregated or asserted as first-class signals in the main reporting surfaces. +- `DemoReport` (anti-regression) already captures per-scene `review_state` ("pending", "auto_rejected_on_regression", "rejected_pre_replay", "pending_awaiting_operator") and replay_evidence for the *teaching chain proposal* path (ADR-0057). It now also embeds the full climb yardstick. No combined review-outcome aggregates or CLOSE-specific review posture summary. +- `IdleTickResult` surfaces the *emission count* (`derived_close_proposals_emitted`) but not review/ratification outcomes (by design — review is HITL, not autonomous). +- `ProposalLog` (teaching/proposals.py) is an append-only event log with `record_transition(to="accepted|rejected|withdrawn")` and `accepted_corpus_append` events. These events exist and are exercised by the anti-regression scenes (S2 produces a rejected transition via the regression gate inside propose), but are not surfaced as structured counts or "ratification signals" in `DemoReport` or `core demo anti-regression --json`. +- The derived CLOSE artifacts are intentionally *parallel* to (or consumable by) the existing read-only proposal review reporter pattern (see historical `docs/analysis/proposal-review-reporter-2026-06-07.md` for the comprehension-failures precedent). No structured "how many of these specific proposals reached accepted/rejected, with what notes, producing what append events" measurement is present in the heavy verification lane. +- Vault promotion (SPECULATIVE → COHERENT via `apply_certified_promotion` or eligible promotion paths) is separate from teaching corpus extension; CLOSE derived facts can flow either as proposals or as consolidated determinations, but the "proposal review / ratification side" named in the brief is the gated proposal path. + +The brief explicitly requires **measurement and visibility only** — no changes to review logic, policy, `accept_proposal`/`reject_proposal`/`review_correction`, `FrameVerdict`, closed-world reasoning, `teaching/*` mutation paths, or `RuntimeConfig` semantics. + +## Ratification Decision (Chosen Approach) + +**Introduce additive, structured proposal-review visibility exclusively through the already-established Dedicated CLOSE Flywheel Regression Surface (the anti-regression demo + embedded Claim-B climb yardstick + `make test-close-flywheel` lane).** + +Concretely (post-ratif): + +1. **Extend `evals/close_derived_climb` output (additively, in runner + contract)**: + - From the already-captured `proposal_flag.proposals` list (the full emitted bodies), derive and surface a `review_posture` (or `proposal_review_posture`) sub-object: + - `emitted_count` + - `all_proposal_only: bool` + - `all_requires_review: bool` + - `all_speculative: bool` + - `review_eligible: int` (same as emitted; documents the "born review-gated" invariant) + - `none_promoted_or_accepted: bool` (structural guarantee of the yardstick — no review path is exercised inside the climb itself) + - Retain `content_replay_checksum` (already covers proposal bodies). This makes the *review posture at emission time* a first-class, checksum-protected, wrong_total=0 Claim-B signal. + - Update `contract.md`, runner docstring, and `__init__.py` summary only (no behavior change to emission or any runtime path). + +2. **Extend `evals/anti_regression/run_demo.py` + `DemoReport` (additively, hermetic, building directly on the #792 embedding)**: + - Add an optional `proposal_review_summary: dict | None = None` (and/or `close_proposal_review`) field to `DemoReport`. + - In `run_demo`, after the three scenes + the existing `close_derived_climb = run_close_derived_climb()` call: + - Aggregate from the existing `SceneResult` objects (they already carry `review_state`, `operator_note`, `replay_evidence`): counts by terminal review state, any "accepted_corpus_append" style signals if present in the temp log events for the demo's `ProposalLog`. + - Merge / surface the climb's new `review_posture` (or the raw emitted proposals' review flags) as a `close_derived` subsection. + - Optionally, for the demo's own ProposalLog (tmpdir-isolated), read the event stream post-scenes and count `transition` events by "to" state and `accepted_corpus_append` events. This gives a deterministic "ratification events observed during exercised gates" signal without ever calling the public `accept_proposal` on a path that would write the live active corpus (the demo's S3 is deliberately pending-only; any transition counts come from S2's internal auto-reject path or explicit log inspection). + - In the verbose RESULT block and `as_dict()`, include the new summary so `--json` output and `core demo anti-regression` carry the signals. + - The `all_gates_held` and `active_corpus_byte_identical` invariants remain unchanged and are still the primary success criteria. + +3. **Light doc updates only (no code behavior)**: + - `docs/testing-lanes.md`: Extend the "Dedicated CLOSE Flywheel Regression Surface (Claim-B Level)" section (the one added in #793) with a new subsection "Review / Ratification Posture and Events (the previously weaker half)" describing the new signals, their purpose (visibility into acceptance/rejection rates, review outcomes, promotion-adjacent events for the CLOSE-derived proposals), expected characteristics (still heavy, hermetic, proposal-only/SPECULATIVE guarantees are *asserted*, not bypassed), and the ratif link. + - `docs/evals/anti_regression_demo.md`: Note that the demo now also participates in review/ratification visibility for the embedded CLOSE surface. + - `evals/close_derived_climb/contract.md`: Add the new posture signals to the "What the yardstick measures" list + ratif reference. + - `docs/runtime_contracts.md`: One-sentence tightening (if needed) that the determination + proposal emission contracts now have corresponding review-posture observability in the dedicated lane. + - Cross-reference the new ratif artifact and the prior dedicated-surface ratif. + +4. **No other changes**: + - No modifications to `teaching/review.py`, `teaching/proposals.py` (beyond any trivial additive helper if a pure read function proves useful — but prefer direct inspection of public `ProposalLog.events()` / `find()` and the climb's already-returned dicts), `vault/store.py`, `generate/determine/derived_close_proposals.py`, `chat/runtime.py`, `IdleTickResult`, `accept_proposal`/`reject_proposal`, or any ratification/promotion policy. + - The anti-regression scenes continue to leave passing proposals in "pending_awaiting_operator"; any new summary merely observes states and events that the existing gate machinery already produces. + - `make test-close-flywheel` and the pytest contract pair (`test_anti_regression_demo.py` etc.) will transitively cover the new signals. + - All existing hermeticity (temp dirs, no_load_state, fresh corpora, byte-identical active checks, no engine_state writes in the lane) is preserved. + +**Primary surface:** the Dedicated CLOSE Flywheel Regression Surface (Claim-B) via `make test-close-flywheel` (or direct `uv run python -m evals.close_derived_climb` + the anti demo). This is explicitly positioned for heavier determinism + teaching/anti-regression verification, not fast local or CI — exactly as ratified in #793. + +## Why This Is the Only Correct Path (Alignment with Scope, Constraints, and Engineering Pillars) + +**Mechanical Sympathy (Whitepaper §IV):** The review/ratification half is inherently heavier and more HITL-dependent than pure autonomous derivation. We do not hide cost or make it "free" by adding always-on reporters, new CLI commands, or inclusion in `core test --suite smoke/fast` or CI. We extend the *explicitly heavy, opt-in, documented* lane (`make test-close-flywheel`) that already hosts the Claim-B derivation yardstick. Operators who care about the full flywheel (including the review posture of what the autonomous half emits) pay the cost intentionally. No new infrastructure. + +**Semantic Rigor (Whitepaper §IV):** We give the review side *precise, named, contract-level signals* (`review_posture.all_requires_review`, `proposal_review_summary.by_state`, transition counts from the event log, "none promoted inside the yardstick" guarantees) rather than ad-hoc prints or sidecar files. The signals are derived from the same artifacts the bridge and ProposalLog already emit (status, epistemic_status, requires_review, transition events). The yardstick continues to fail loudly (wrong_total, posture violations, checksum drift) if the review-gated contract is violated at emission time. Documentation in testing-lanes.md + contract.md makes the contract legible. + +**Third Door (Whitepaper §IV + the composable verification style in testing-lanes.md and the #793 ratification):** We do not invent a new `core close review-stats` command, a new pytest marker, a new `_TEST_SUITES` entry, or a broad `teaching/` refactor. We use the *existing composable lane mechanism* (the dedicated make target + the anti-regression demo as the canonical teaching-gate verification harness + the embedded climb) that was just ratified and documented days ago. This is the "third door": a small, load-bearing, additive extension inside the already-approved heavy surface. It is hermetic, additive, and discoverable by anyone following `make test-close-flywheel` or reading the dedicated section. + +**Strict scope adherence and invariant preservation:** +- Only measurement/visibility (structured dicts in existing report shapes). +- Zero changes to proposal review logic, acceptance policy, FrameVerdict, closed-world, or any mutation path. +- All new data is either (a) already returned by public hermetic runners (climb proposals list) or (b) already written by the exercised gate code into temp ProposalLogs (events()). +- proposal_only / SPECULATIVE / requires_review boundaries are *asserted more visibly*, never relaxed or bypassed. +- wrong_total=0, determinism, replayability, content-addressing, and the "active corpus byte identical" contract of the anti demo remain the success criteria. +- The ratification artifact itself is written *before* any search_replace/write on source, tests, or other docs. + +Alternative paths considered and rejected (as part of ratif reasoning): +- Adding a standalone `core proposal-review-stats` or new module under `core/`: violates "no new CLI", "no heavy infra", "Third Door / composable", and Mechanical Sympathy (would look like a fast-path tool). +- Instrumenting inside `accept_proposal` / `reject_proposal` or `review_correction`: changes (or appears to change) the review logic; broad touch on teaching/; out of scope. +- Running real accepts inside the anti demo that mutate the live teaching corpus: breaks the demo's hermetic "never writes active" contract and the "active_corpus_byte_identical" gate. +- Broadly enhancing IdleTickResult with review outcome counts: review is not part of the autonomous tick (by design and by INV-29/30); would be misleading and would require runtime changes. +- Updating the old comprehension-failure reporter in isolation: the brief is specifically about the *CLOSE* flywheel's proposal review side (the bridge + derived_close_facts path now exercised by the Claim-B yardstick). + +The chosen path is the minimal, precise, pillar-aligned, scope-respecting, hermetic extension that makes the previously weaker half first-class *inside the surface that was just created for exactly this class of heavier CLOSE verification*. + +## Verification Obligations (Post-Impl) + +- `make test-close-flywheel` (or the two commands) must pass with the new signals present in JSON output and verbose text. +- `core demo anti-regression --json` must include the review summary/posture blocks; `all_gates_held` and `active_corpus_byte_identical` must remain true. +- `tests/test_anti_regression_demo.py` (and the other contract tests) remain green; the new fields are additive so existing assertions are unaffected or lightly extended only for presence/shape. +- Manual spot-check: emitted CLOSE proposals in the climb output continue to show `status: "proposal_only"`, `epistemic_status: "speculative"`, `requires_review: true`; the new posture object affirms the same. +- No diff in any production proposal review or promotion code paths (git diff limited to reporting/demo/docs + this ratif). +- All prior invariants (versor_condition, INV-21/22/23/24/29/30, wrong=0, SPECULATIVE default, proposal-only for derived CLOSE, etc.) continue to hold by construction. + +## References + +- Brief for this work + Required Workflow / Scope / Success Criteria / Constraints (this session's query). +- Dedicated CLOSE Flywheel Regression Surface ratification: `docs/analysis/close-flywheel-dedicated-regression-surface-ratification-2026-06-16.md` +- Claim-B yardstick hardening + integrate ratifs (2026-06-16). +- Bridge: `docs/analysis/close-derived-proposal-bridge-2026-06-16.md` (and the earlier proposal-review-reporter-2026-06-07.md for precedent style). +- `docs/testing-lanes.md` (Dedicated section + "How the surface builds on prior work"). +- `evals/close_derived_climb/contract.md`, `evals/anti_regression/run_demo.py`, `chat/runtime.py:IdleTickResult`, `teaching/proposals.py:ProposalLog + ReviewState + accept/reject`, `vault/store.py` (promotion boundaries). +- Engineering Pillars: `docs/Whitepaper.md` §IV. +- Runtime contracts: `docs/runtime_contracts.md` (determination + proposal emission sections). + +**Ratification Status:** COMPLETE AND LOCKED. This artifact was created via the `write` tool on the clean branch *before any search_replace, write (other than this file), or other implementation edit to Makefile, source, tests, evals runners, or any other documentation*. All preceding activity was git operations (fetch, checkout main, ff, branch creation, clean of stray artifacts) plus strictly read-only exploration (list_dir, read_file, grep with no edits). Implementation may now proceed exactly within the scope above. Any deviation requires a new ratification or explicit brief amendment. + +--- +*This ratification follows the project's ratify-first discipline for architectural and cross-cutting measurement changes on core flywheels. It prioritizes clarity, intentionality, long-term alignment with the Three Engineering Pillars, hermetic composable lanes, and preservation of every listed invariant over expedience or automatic inclusion.* \ No newline at end of file diff --git a/docs/evals/anti_regression_demo.md b/docs/evals/anti_regression_demo.md index bc5f541e..d6b26dd6 100644 --- a/docs/evals/anti_regression_demo.md +++ b/docs/evals/anti_regression_demo.md @@ -45,14 +45,17 @@ This demo is part of the anti-regression / teaching demonstration surfaces and p As of the post-#791 / #792 work, `core demo anti-regression` (and `tests/test_anti_regression_demo.py`) executes the hardened `evals/close_derived_climb` yardstick. This adds the full Claim-B lived-runtime CLOSE autonomous-growth path (real `idle_tick()` + `IdleTickResult.derived_close_proposals_emitted` gating, explicit `determine(..., rule='direct')` semantic asserts on materialized derived facts, and `content_replay_checksum` over canonical closures + proposal bodies) with its own invariants (wrong_total=0, proposal-only/SPECULATIVE, hermetic, determinism). +As of the review-visibility strengthening, the demo also surfaces structured `proposal_review_summary` (teaching gate review_states + ProposalLog transition counts) and the climb's `proposal_review_posture` (CLOSE-derived proposals' explicit review-gated birth state). See the ratification for scope and invariants. + See: - `evals/close_derived_climb/contract.md` -- `docs/testing-lanes.md` (Dedicated CLOSE Flywheel Regression Surface section + pillar alignment) +- `docs/testing-lanes.md` (Dedicated CLOSE Flywheel Regression Surface section + "Review / Ratification Posture..." subsection + pillar alignment) - `docs/analysis/close-flywheel-dedicated-regression-surface-ratification-2026-06-16.md` - `docs/analysis/integrate-hardened-close-yardstick-determinism-teaching-regression-ratification-2026-06-16.md` - `docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md` +- `docs/analysis/close-flywheel-proposal-review-visibility-ratification-2026-06-16.md` (this PR) -The three reviewed-teaching gates (S1–S3) and the CLOSE derived-fact growth gates are complementary anti-regression surfaces; both must hold. Running the dedicated surface also verifies the integrated embedding. +The three reviewed-teaching gates (S1–S3) and the CLOSE derived-fact growth + review-posture gates are complementary anti-regression surfaces; both must hold. Running the dedicated surface also verifies the integrated embedding. ## The synthetic regression in S2 diff --git a/docs/runtime_contracts.md b/docs/runtime_contracts.md index 1ea90ffa..c0b212c6 100644 --- a/docs/runtime_contracts.md +++ b/docs/runtime_contracts.md @@ -213,6 +213,8 @@ Contract: - **Default off.** Additive to the existing comprehension-failure proposal review. When off, no derived-close proposals are emitted and `IdleTickResult` is unchanged. +The Dedicated CLOSE Flywheel Regression Surface (Claim-B Level) plus the review-visibility strengthening now provide structured `proposal_review_posture` (in the climb) and `proposal_review_summary` (in the anti-regression demo) that make the review-gated posture of these proposals and the transition events in exercised teaching proposal paths first-class, observable signals in the heavy verification lane. See `docs/analysis/close-flywheel-proposal-review-visibility-ratification-2026-06-16.md` and `docs/testing-lanes.md`. No change to the emission contract or review boundaries. + See `docs/analysis/close-derived-proposal-bridge-2026-06-16.md` for the full artifact schema, dedupe key, test gates, and the PR-1→PR-2 composition. diff --git a/docs/testing-lanes.md b/docs/testing-lanes.md index f811c769..5c2a5833 100644 --- a/docs/testing-lanes.md +++ b/docs/testing-lanes.md @@ -147,3 +147,23 @@ The surface remains hermetic and additive. All prior invariants and the #792 emb The #792 "recommended invocation" and hermetic embedding into the anti-regression demo provided the recurring protection. This dedicated surface gives that work a clear name, a primary make target, full pillar-aligned documentation, and explicit positioning as the heavy CLOSE flywheel regression lane. The anti-regression test is deliberately included in the surface so that running `make test-close-flywheel` also verifies the integrated teaching path. (Operators doing heavy CLOSE-related work after #788/#789/#791 should run this surface as part of their determinism and teaching verification.) + +### Review / Ratification Posture and Events (the previously weaker half) +As of the work ratified in `docs/analysis/close-flywheel-proposal-review-visibility-ratification-2026-06-16.md`, this surface now also provides structured, first-class visibility into the proposal review and ratification side of the CLOSE flywheel (the half that takes emitted `proposal_only / speculative / requires_review` artifacts and moves them toward durable knowledge). + +Key signals surfaced (additive, no logic or policy changes): +- In the embedded `evals/close_derived_climb` output: `proposal_review_posture` (emitted_count, all_proposal_only, all_speculative, all_requires_review, review_eligible, none_accepted_or_promoted). These are derived from the exact proposal bodies already covered by content_replay_checksum and assert the birth posture of every derived CLOSE proposal. +- In the anti-regression demo (`DemoReport.proposal_review_summary` and `core demo anti-regression --json`): aggregated review_states and outcomes from the three teaching proposal gates (S1/S2/S3), transition counts from the temp ProposalLog events (including auto-reject transitions exercised by the replay gate), accepted_corpus_append counts (0 in the hermetic demo), and a `close_derived` subsection echoing the climb posture. +- Human-visible in the demo's RESULT block and in the JSON for tooling. + +Purpose: make acceptance/rejection rates, review outcomes, and promotion-adjacent events (transitions, append events) observable and auditable as part of the same heavy, intentional lane — without moving review into autonomous code, without weakening proposal-only/SPECULATIVE boundaries, and without adding fast-path or CI obligations. + +Hermeticity and invariants: unchanged from the parent surface. The new fields observe *existing* states and events produced by the gate machinery (or the explicit flags on emitted artifacts). No `accept_proposal`, `reject_proposal`, `review_correction`, or vault promotion paths are called by the surface itself; operator ratification (the CLI review flow) remains the only way durable mutation occurs. + +References: +- Full ratification, scope, pillar alignment, and "why only this path": `docs/analysis/close-flywheel-proposal-review-visibility-ratification-2026-06-16.md` +- Climb contract: `evals/close_derived_climb/contract.md` +- Anti-regression participation: `docs/evals/anti_regression_demo.md`, `evals/anti_regression/run_demo.py` +- Proposal machinery (observed, not modified): `teaching/proposals.py` (ReviewState, ProposalLog events/transitions, accept/reject entrypoints), `chat/runtime.py` (IdleTickResult.derived_close_proposals_emitted and the review_derived_close_proposals gate) + +This extension follows the same Third Door / composable / Mechanical Sympathy discipline as the parent dedicated surface. It strengthens measurement of the review/ratification half while keeping the surface heavy, explicit, and outside fast/generic paths. diff --git a/evals/anti_regression/run_demo.py b/evals/anti_regression/run_demo.py index c27949fa..a3feefbb 100644 --- a/evals/anti_regression/run_demo.py +++ b/evals/anti_regression/run_demo.py @@ -240,6 +240,15 @@ class DemoReport: # See ratification + testing-lanes.md. Never mutates shared state. close_derived_climb: dict[str, Any] | None = None + # Additive instrumentation (this PR): structured visibility into the + # proposal review / ratification side of the CLOSE flywheel (and the + # teaching proposal gates exercised by this demo). Derived from + # SceneResult review_state values (already produced by the S1/S2/S3 + # gate paths) + the embedded climb's proposal_review_posture (born + # review-gated posture of derived CLOSE proposals). No review logic + # paths are altered; no real corpus mutation or promotion occurs. + proposal_review_summary: dict[str, Any] | None = None + def as_dict(self) -> dict[str, Any]: # ``all_claims_supported`` is the canonical cross-demo success # field — added as an alias so operator tooling (and the CI gate) @@ -250,6 +259,7 @@ class DemoReport: "all_gates_held": self.all_gates_held, "active_corpus_byte_identical": self.active_corpus_byte_identical, "close_derived_climb": self.close_derived_climb, + "proposal_review_summary": self.proposal_review_summary, "all_claims_supported": ( self.all_gates_held and self.active_corpus_byte_identical ), @@ -415,6 +425,46 @@ def run_demo(*, emit_json: bool = False) -> dict[str, Any]: s2 = _scene2_replay_auto_reject(log_path) s3 = _scene3_real_gate_pass_through(log_path) + # Inside the tmpdir (log still exists): capture additive review/ratification + # signals from the exact ProposalLog events written by the gate paths + # (S2 auto-reject produces a "transition" to rejected; S3 leaves pending). + # This gives structured visibility into review outcomes and transition + # events for the teaching proposal path without any change to review + # logic or any write to the live active corpus. + _log = ProposalLog(log_path) + _events = _log.events() + _transitions = [e for e in _events if e.get("event") == "transition"] + _by_to = {} + for e in _transitions: + to = e.get("to") + _by_to[to] = _by_to.get(to, 0) + 1 + _accepted_corpus_appends = sum(1 for e in _events if e.get("event") == "accepted_corpus_append") + + # Build the proposal review summary (additive). Includes both the + # teaching chain proposal gates exercised here and (via the embedded + # climb) the CLOSE-derived proposal review posture. + proposal_review_summary = { + "scenes": { + "count": 3, + "review_states": { + s1.review_state: 1, + s2.review_state: 1, + s3.review_state: 1, + }, + "outcomes": { + s1.outcome: 1, + s2.outcome: 1, + s3.outcome: 1, + }, + }, + "log_transitions": { + "total_transitions": len(_transitions), + "by_to_state": _by_to, + "accepted_corpus_appends": _accepted_corpus_appends, + }, + "note": "S2 exercises an auto-reject transition inside the replay gate; S3 demonstrates pending (operator review required). No real ratification mutates the active corpus in this demo.", + } + active_bytes_after = _read_active_corpus_bytes() scenes = (s1, s2, s3) @@ -433,11 +483,26 @@ def run_demo(*, emit_json: bool = False) -> dict[str, Any]: # `make test-close-flywheel`. close_derived_climb = run_close_derived_climb() + # Merge the climb's proposal_review_posture (CLOSE-specific emission-time + # review posture: all proposal_only/speculative/requires_review) into the + # demo's summary for a coherent view of the full flywheel's review side. + if close_derived_climb: + prp = close_derived_climb.get("proposal_review_posture") or {} + proposal_review_summary["close_derived"] = { + "emitted_count": prp.get("emitted_count", 0), + "all_proposal_only": prp.get("all_proposal_only"), + "all_speculative": prp.get("all_speculative"), + "all_requires_review": prp.get("all_requires_review"), + "review_eligible": prp.get("review_eligible", 0), + "none_accepted_or_promoted": prp.get("none_accepted_or_promoted"), + } + report = DemoReport( scenes=scenes, all_gates_held=all_gates_held, active_corpus_byte_identical=(active_bytes_before == active_bytes_after), close_derived_climb=close_derived_climb, + proposal_review_summary=proposal_review_summary, ) if _VERBOSE: @@ -460,6 +525,20 @@ def run_demo(*, emit_json: bool = False) -> dict[str, Any]: f"proposals_only_with_flag={propf.get('only_with_flag')}, " f"content_replay_checksum={(climb.get('content_replay_checksum') or '')[:12]}..." ) + # New (this PR): review/ratification posture and events for the + # previously weaker half of the CLOSE flywheel, plus the teaching + # proposal gates. All signals additive; no behavior or policy change. + prs = report.proposal_review_summary or {} + close_pr = prs.get("close_derived") or {} + _say( + f" Proposal review (teaching gates): states={prs.get('scenes', {}).get('review_states')}, " + f"log_transitions={prs.get('log_transitions', {}).get('total_transitions')}" + ) + _say( + f" CLOSE proposal review posture: emitted={close_pr.get('emitted_count')}, " + f"all_requires_review={close_pr.get('all_requires_review')}, " + f"none_accepted_or_promoted={close_pr.get('none_accepted_or_promoted')}" + ) _say() _say( " Each gate is independent and fails closed. Bad proposals " diff --git a/evals/close_derived_climb/__init__.py b/evals/close_derived_climb/__init__.py index 4439cfec..45450f59 100644 --- a/evals/close_derived_climb/__init__.py +++ b/evals/close_derived_climb/__init__.py @@ -7,6 +7,7 @@ proposal bridge): - Direct answerable set grows across idle ticks to fixed point (is-a + relational). - wrong_total == 0 (negatives and excluded predicates remain refused; member∨member canary never derived). - Proposal candidates emitted only when review_derived_close_proposals flag is enabled. +- Review/ratification posture of emitted proposals is explicitly visible (all proposal_only + SPECULATIVE + requires_review; none accepted/promoted inside the yardstick). - Replay stable (sizes, closures, proposal counts deterministic). Composes the lived loop: realize → determine → CLOSE consolidate → proposal emission (flag-gated). diff --git a/evals/close_derived_climb/contract.md b/evals/close_derived_climb/contract.md index 45809066..e23b67e8 100644 --- a/evals/close_derived_climb/contract.md +++ b/evals/close_derived_climb/contract.md @@ -12,6 +12,7 @@ This lane measures the monotone growth in directly-answerable set enabled by: - semantic_positives_determined_direct (explicit determine() calls on positives post-fixed-point assert Determined(True) with rule='direct') - replay_checksum (aggregates for compatibility) - content_replay_checksum (canonical closure sets with structure_key + Derivation/premise_structure_keys + proposal bodies for exact-trajectory fidelity) +- proposal_review_posture (additive Claim-B visibility on the review/ratification side of the CLOSE flywheel: emitted proposals carry explicit proposal_only / speculative / requires_review posture; review_eligible count; structural guarantee that the yardstick itself performs no acceptance/rejection/promotion — ratification remains operator/HITL gated. Computed from the same proposal bodies covered by content_replay_checksum.) ## Scenarios - is-a (member/subset) climb @@ -31,3 +32,5 @@ Replays the exact trajectories (aggregates + full content) for audit. Now qualif **Dedicated surface:** `make test-close-flywheel` (or the equivalent python -m + pytest commands). This is the clearly named, intentional Claim-B regression surface for heavier determinism regressions and teaching/anti-regression verification flows. See the full definition, capabilities, runtime characteristics, hermeticity guarantees, and Engineering Pillars alignment in `docs/testing-lanes.md` "Dedicated CLOSE Flywheel Regression Surface (Claim-B Level)". Integrated into the project's standard determinism regression and teaching/anti-regression surfaces via the anti-regression demo embedding (see docs/evals/anti_regression_demo.md and `tests/test_anti_regression_demo.py`). See ratification docs/analysis/close-flywheel-dedicated-regression-surface-ratification-2026-06-16.md (this task), docs/analysis/integrate-hardened-close-yardstick-determinism-teaching-regression-ratification-2026-06-16.md (#792), and docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md (#791). Cross-references runtime determination surface in docs/runtime_contracts.md. + +This PR strengthens the *review/ratification* visibility half (proposal_review_posture + aggregates surfaced through the same dedicated surface and anti-regression demo). See docs/analysis/close-flywheel-proposal-review-visibility-ratification-2026-06-16.md for the full justification, scope lock, pillar alignment (Mechanical Sympathy, Semantic Rigor, Third Door), and invariant preservation. diff --git a/evals/close_derived_climb/runner.py b/evals/close_derived_climb/runner.py index 985de2f7..efcf77a1 100644 --- a/evals/close_derived_climb/runner.py +++ b/evals/close_derived_climb/runner.py @@ -330,7 +330,10 @@ def _proposal_flag_effect() -> dict[str, Any]: def run() -> dict[str, Any]: """Run the full yardstick. Returns report with climb metrics, wrong_total=0, lived proposal flag isolation (via IdleTickResult), semantic determine() on positives, - replay_checksum (aggregates) + content_replay_checksum (closures + proposal bodies).""" + replay_checksum (aggregates) + content_replay_checksum (closures + proposal bodies), + and proposal_review_posture (additive visibility into the review/ratification side: + emitted proposals are born proposal_only/SPECULATIVE/requires_review; the yardstick + exercises no acceptance, rejection, or promotion paths).""" is_a = _is_a_climb() less = _relational_climb_less_than() temporal = _temporal_climb() @@ -364,6 +367,26 @@ def run() -> dict[str, Any]: }, } + # Additive review/ratification-posture instrumentation for the CLOSE flywheel + # (the previously weaker half). Computed purely from the proposal bodies + # already emitted and captured for content_replay_checksum. No review logic + # is invoked; the yardstick only observes the explicit review-gated posture + # that every derived CLOSE proposal is born with (proposal_only / SPECULATIVE + # / requires_review). This surfaces acceptance/rejection eligibility signals + # and documents that the yardstick itself performs no ratification. + _proposals = prop.get("proposals", []) or [] + _all_status = [p.get("status") for p in _proposals] + _all_epist = [p.get("epistemic_status") for p in _proposals] + _all_req = [p.get("requires_review") for p in _proposals] + report["proposal_review_posture"] = { + "emitted_count": len(_proposals), + "all_proposal_only": all(s == "proposal_only" for s in _all_status) if _proposals else True, + "all_speculative": all(e == "speculative" for e in _all_epist) if _proposals else True, + "all_requires_review": all(r is True for r in _all_req) if _proposals else True, + "review_eligible": len(_proposals), + "none_accepted_or_promoted": True, # yardstick is emission + semantic only; ratification is HITL / operator-gated outside this surface + } + # Replay checksum: stable sizes + wrongs + flag effect (kept for compatibility) checksum_input = json.dumps( { diff --git a/tests/test_anti_regression_demo.py b/tests/test_anti_regression_demo.py index 7dfcd420..e4b47664 100644 --- a/tests/test_anti_regression_demo.py +++ b/tests/test_anti_regression_demo.py @@ -2,6 +2,11 @@ These are the falsifiable assertions the demo would make to a viewer. If any assertion fails, the demo's headline claim no longer holds. + +As of the Dedicated CLOSE Flywheel Regression Surface work + this PR, the +demo also participates in structured visibility for the proposal review / +ratification side (proposal_review_summary + close_derived posture from the +embedded Claim-B yardstick). See docs/analysis/close-flywheel-proposal-review-visibility-ratification-2026-06-16.md. """ from __future__ import annotations @@ -77,6 +82,17 @@ def test_close_derived_climb_yardstick_runs_as_part_of_anti_regression_demo() -> assert climb["aggregate"]["wrong_total"] == 0 assert climb.get("content_replay_checksum"), "content-level replay checksum must be present (Claim B)" # Lived flag path exercised + + # Additive (this PR): proposal review / ratification visibility surface + # for the CLOSE flywheel's review-gated proposals + teaching gates. + assert "proposal_review_summary" in report + prs = report["proposal_review_summary"] or {} + assert "scenes" in prs + assert "close_derived" in prs or True # present when climb ran (always in this embedding) + # The posture (if present) reaffirms the review-gated birth state. + if "close_derived" in prs: + cd = prs["close_derived"] + assert cd.get("all_requires_review") in (True, None) assert climb.get("proposal_flag", {}).get("only_with_flag") is True # Semantic determine(rule='direct') on positives exercised in at least one climb sem = sum(