7a20356ab6
- Semantic: determine() asserts (rule=direct) on positives post-FP in climbs - Lived flag: _proposal_flag_effect now uses real idle_tick + IdleTickResult.derived_close_proposals_emitted (temp sink isolation) - Checksum: added content_replay_checksum (canonical closures + proposal bodies) - Docs: contract.md + module docstrings updated for accuracy Ratification artifact created before impl (see docs/analysis/...-ratification-...md). All original metrics, wrong_total=0, invariants preserved. Closes the 3 gaps from post-merge audit for Claim B.
33 lines
No EOL
3.9 KiB
Markdown
33 lines
No EOL
3.9 KiB
Markdown
# Ratification: Harden evals/close_derived_climb Yardstick Toward Full Lived-Runtime Claim B
|
||
|
||
**Date:** 2026-06-16
|
||
**Author:** Grok (per brief)
|
||
**Context:** Post-merge hardening audit (on main cf1e7371) of the CLOSE flywheel yardstick established that it delivers strong lived-runtime proof of **Claim A** (direct closure-growth via real `ChatRuntime.idle_tick()` + `consolidate_once` + `realize_derived`, with strict/monotone growth 1→5→8, `wrong_total=0`, boundaries preserved) but only partial evidence for **Claim B** (full lived flywheel yardstick).
|
||
|
||
**The three gaps identified (verbatim from audit):**
|
||
1. `_proposal_flag_effect()` uses explicit simulation (direct `consolidate_once` + `emit`) instead of real `ChatRuntime.idle_tick()`.
|
||
2. Positive growth metrics are scored via vault-direct `recall_realized` counts (`_count_answerable`) rather than semantic `determine()` calls on the grown facts.
|
||
3. `replay_checksum` only hashes aggregate sizes + wrong_total + flag-isolation and does not match its own documentation (which claims coverage of "closure sets" and "exact trajectories").
|
||
|
||
All core mechanics, invariants (INV-21/29/30/31), `wrong_total=0` behavior, SPECULATIVE-only, proposal-only boundary, and determinism are already solid.
|
||
|
||
## Ratification Decision
|
||
**The single recommended (and only correct) path is exactly the four targeted, minimal high-leverage improvements specified in the brief:**
|
||
|
||
1. **Semantic Answerability**: After reaching fixed point in the climb scenarios, explicitly call `determine()` (or `_ask_rel` equivalent) on the positive probe queries and assert `Determined(True)` with `rule='direct'`.
|
||
2. **Lived Flag Path**: Replace or augment `_proposal_flag_effect` with a version that uses real `ChatRuntime` + `idle_tick()` (flag enabled) and captures `IdleTickResult.derived_close_proposals_emitted`. Remove any dependence on direct simulation for the yardstick’s reported "proposals_only_with_flag" metric.
|
||
3. **Checksum Fidelity**: Extend `replay_checksum` (or add a parallel `content_replay_checksum`) to include canonical representations of actual closure sets (structure_key + `Derivation` with premise keys) and proposal bodies. Align implementation with existing module/contract documentation.
|
||
4. **Documentation**: Update `evals/close_derived_climb/contract.md` and relevant module docstrings to accurately reflect what the yardstick now measures (lived flag via `IdleTickResult`, semantic via `determine()`, content-level replay) versus prior claims.
|
||
|
||
**Why this is the only correct path (and must not be broadened or altered):**
|
||
- It directly, precisely, and minimally closes each of the three gaps from the audit without introducing new scenarios, metrics, harnesses, or capabilities.
|
||
- It makes *the yardstick code itself* (not external tests or side harnesses) exercise and report the full lived flywheel behaviors required for Claim B.
|
||
- It is the smallest set of changes that allow the yardstick to pass the original audit checklist under *both* Claim A *and* full Claim B criteria.
|
||
- Any other approach (e.g., a separate "Claim B" runner, larger refactor of recall/determine paths, adding new proposal families, or non-minimal content in checksum) would violate the "minimal + high-leverage", "do not broaden scope", and "preserve all existing invariants/behavior" constraints.
|
||
- Ratifying this exact path ensures subsequent implementation stays focused, the PR can reference this artifact as the justified direction, and no unrelated work is mixed in.
|
||
|
||
**Ratification Status:** Explicitly ratified as the sole authorized direction before any implementation code is written. Implementation will now proceed strictly to these four items (plus the required workflow steps: branch already created, PR at end with mandated description elements).
|
||
|
||
This ratification artifact will be referenced in the eventual PR description.
|
||
|
||
(End of ratification. No implementation code has been written to the yardstick yet.) |