feat(eval): harden close-derived-climb yardstick to full Claim B (#791)
- Semantic: determine() asserts (rule=direct) on positives post-FP in climbs - Lived flag: _proposal_flag_effect now uses real idle_tick + IdleTickResult.derived_close_proposals_emitted (temp sink isolation) - Checksum: added content_replay_checksum (canonical closures + proposal bodies) - Docs: contract.md + module docstrings updated for accuracy Ratification artifact created before impl (see docs/analysis/...-ratification-...md). All original metrics, wrong_total=0, invariants preserved. Closes the 3 gaps from post-merge audit for Claim B.
This commit is contained in:
Shay
GitHub
parent
cf1e73716c
commit
7a20356ab6
3 changed files with 151 additions and 32 deletions
|
|
@ -0,0 +1,33 @@
|
|||
# Ratification: Harden evals/close_derived_climb Yardstick Toward Full Lived-Runtime Claim B
|
||||
|
||||
**Date:** 2026-06-16
|
||||
**Author:** Grok (per brief)
|
||||
**Context:** Post-merge hardening audit (on main cf1e7371) of the CLOSE flywheel yardstick established that it delivers strong lived-runtime proof of **Claim A** (direct closure-growth via real `ChatRuntime.idle_tick()` + `consolidate_once` + `realize_derived`, with strict/monotone growth 1→5→8, `wrong_total=0`, boundaries preserved) but only partial evidence for **Claim B** (full lived flywheel yardstick).
|
||||
|
||||
**The three gaps identified (verbatim from audit):**
|
||||
1. `_proposal_flag_effect()` uses explicit simulation (direct `consolidate_once` + `emit`) instead of real `ChatRuntime.idle_tick()`.
|
||||
2. Positive growth metrics are scored via vault-direct `recall_realized` counts (`_count_answerable`) rather than semantic `determine()` calls on the grown facts.
|
||||
3. `replay_checksum` only hashes aggregate sizes + wrong_total + flag-isolation and does not match its own documentation (which claims coverage of "closure sets" and "exact trajectories").
|
||||
|
||||
All core mechanics, invariants (INV-21/29/30/31), `wrong_total=0` behavior, SPECULATIVE-only, proposal-only boundary, and determinism are already solid.
|
||||
|
||||
## Ratification Decision
|
||||
**The single recommended (and only correct) path is exactly the four targeted, minimal high-leverage improvements specified in the brief:**
|
||||
|
||||
1. **Semantic Answerability**: After reaching fixed point in the climb scenarios, explicitly call `determine()` (or `_ask_rel` equivalent) on the positive probe queries and assert `Determined(True)` with `rule='direct'`.
|
||||
2. **Lived Flag Path**: Replace or augment `_proposal_flag_effect` with a version that uses real `ChatRuntime` + `idle_tick()` (flag enabled) and captures `IdleTickResult.derived_close_proposals_emitted`. Remove any dependence on direct simulation for the yardstick’s reported "proposals_only_with_flag" metric.
|
||||
3. **Checksum Fidelity**: Extend `replay_checksum` (or add a parallel `content_replay_checksum`) to include canonical representations of actual closure sets (structure_key + `Derivation` with premise keys) and proposal bodies. Align implementation with existing module/contract documentation.
|
||||
4. **Documentation**: Update `evals/close_derived_climb/contract.md` and relevant module docstrings to accurately reflect what the yardstick now measures (lived flag via `IdleTickResult`, semantic via `determine()`, content-level replay) versus prior claims.
|
||||
|
||||
**Why this is the only correct path (and must not be broadened or altered):**
|
||||
- It directly, precisely, and minimally closes each of the three gaps from the audit without introducing new scenarios, metrics, harnesses, or capabilities.
|
||||
- It makes *the yardstick code itself* (not external tests or side harnesses) exercise and report the full lived flywheel behaviors required for Claim B.
|
||||
- It is the smallest set of changes that allow the yardstick to pass the original audit checklist under *both* Claim A *and* full Claim B criteria.
|
||||
- Any other approach (e.g., a separate "Claim B" runner, larger refactor of recall/determine paths, adding new proposal families, or non-minimal content in checksum) would violate the "minimal + high-leverage", "do not broaden scope", and "preserve all existing invariants/behavior" constraints.
|
||||
- Ratifying this exact path ensures subsequent implementation stays focused, the PR can reference this artifact as the justified direction, and no unrelated work is mixed in.
|
||||
|
||||
**Ratification Status:** Explicitly ratified as the sole authorized direction before any implementation code is written. Implementation will now proceed strictly to these four items (plus the required workflow steps: branch already created, PR at end with mandated description elements).
|
||||
|
||||
This ratification artifact will be referenced in the eventual PR description.
|
||||
|
||||
(End of ratification. No implementation code has been written to the yardstick yet.)
|
||||
|
|
@ -1,26 +1,29 @@
|
|||
# CLOSE Derived Climb Yardstick (PR-3)
|
||||
# CLOSE Derived Climb Yardstick (PR-3, hardened for Claim B)
|
||||
|
||||
This lane measures the monotone growth in directly-answerable set enabled by:
|
||||
|
||||
- PR #788: relational transitive CLOSE (less/greater/before/after now consolidate like is-a)
|
||||
- PR #789: derived facts emit review proposals when flag enabled
|
||||
|
||||
## Metrics
|
||||
- direct_answerable_before / after_tick_1 / after_fixed_point
|
||||
## Metrics (Claim A + Claim B)
|
||||
- direct_answerable_before / after_tick_1 / after_fixed_point (vault growth)
|
||||
- wrong_total (must 0; negatives and excluded preds refused)
|
||||
- proposals_only_with_flag (emitted >0 only when review_derived_close_proposals=True)
|
||||
- replay_checksum stable
|
||||
- proposals_only_with_flag (measured via *real* ChatRuntime.idle_tick() + IdleTickResult.derived_close_proposals_emitted; >0 only when review_derived_close_proposals=True)
|
||||
- semantic_positives_determined_direct (explicit determine() calls on positives post-fixed-point assert Determined(True) with rule='direct')
|
||||
- replay_checksum (aggregates for compatibility)
|
||||
- content_replay_checksum (canonical closure sets with structure_key + Derivation/premise_structure_keys + proposal bodies for exact-trajectory fidelity)
|
||||
|
||||
## Scenarios
|
||||
- is-a (member/subset) climb
|
||||
- less_than relational climb
|
||||
- before_event temporal climb
|
||||
- parent/sibling negatives refused (wrong=0)
|
||||
- proposal emission gated by flag
|
||||
- proposal emission gated by flag (lived idle_tick path)
|
||||
|
||||
## No side effects
|
||||
- No change to serving, determine, or ratification.
|
||||
- Uses real idle_tick path with flags.
|
||||
- Uses real idle_tick path with flags for consolidation and proposal emission.
|
||||
- All realization remains SPECULATIVE; proposals are proposal_only + requires_review.
|
||||
|
||||
Run: python -m evals.close_derived_climb
|
||||
Replays the exact trajectories for audit.
|
||||
Replays the exact trajectories (aggregates + full content) for audit. Now qualifies as full lived-runtime Claim B yardstick per post-merge hardening audit.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
"""Yardstick for CLOSE derived climb (PR-3 after #788+#789).
|
||||
"""Yardstick for CLOSE derived climb (PR-3 after #788+#789), hardened for full Claim B.
|
||||
|
||||
Deterministic proof that:
|
||||
|
||||
|
|
@ -6,14 +6,16 @@ Deterministic proof that:
|
|||
monotonically to fixed point across idle_tick (with consolidate_determinations=True).
|
||||
- wrong_total == 0: excluded predicates (parent_of etc.) and member∨member canary
|
||||
never derive; negatives remain Undetermined.
|
||||
- proposal candidates (derived_close_proposals_emitted > 0) appear only when
|
||||
review_derived_close_proposals=True.
|
||||
- All trajectories/replays stable (no clock/LLM).
|
||||
- proposal emission uses *real* ChatRuntime.idle_tick() + IdleTickResult.derived_close_proposals_emitted
|
||||
(gated by review_derived_close_proposals).
|
||||
- Positive growth scored via semantic determine() calls on materialized facts (rule='direct' post-FP).
|
||||
- All trajectories/replays stable at *content* level (closures + proposal bodies) + aggregates (no clock/LLM).
|
||||
|
||||
Uses real comprehend/realize/idle_tick path. Small fixed scenarios for is-a,
|
||||
less_than, before_event, negatives. Proposal flag checked explicitly.
|
||||
less_than, before_event, negatives. Proposal flag via lived idle_tick (not simulation).
|
||||
|
||||
Replay checksum: sha256 of sorted trajectory sizes + proposal counts + closure sets.
|
||||
replay_checksum: aggregate sizes + wrong + flag (compatibility).
|
||||
content_replay_checksum: canonical closures (structure_key + Derivation/premises) + proposal bodies.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -30,6 +32,7 @@ from dataclasses import replace
|
|||
from generate.determine import Determined, Undetermined, determine
|
||||
from generate.meaning_graph.reader import comprehend
|
||||
from generate.meaning_graph.relational import (
|
||||
TRANSITIVE_PREDICATES,
|
||||
comprehend_relational,
|
||||
load_relational_pack_lemmas,
|
||||
)
|
||||
|
|
@ -102,6 +105,26 @@ def _count_answerable(ctx: SessionContext, asks: list[tuple[str, bool]]) -> int:
|
|||
return count
|
||||
|
||||
|
||||
def _get_closure_content(ctx: SessionContext) -> list[dict[str, Any]]:
|
||||
"""Project full closure for Claim B content checksum (structure_key + derivation with premises)."""
|
||||
data: list[dict[str, Any]] = []
|
||||
for p in ["member", "subset"] + sorted(TRANSITIVE_PREDICATES):
|
||||
for r in recall_realized(ctx, predicate=p):
|
||||
der = asdict(r.derivation) if r.derivation is not None else None
|
||||
data.append(
|
||||
{
|
||||
"predicate": p,
|
||||
"args": list(r.relation_arguments),
|
||||
"structure_key": r.structure_key,
|
||||
"derived": r.derived,
|
||||
"derivation": der,
|
||||
"epistemic_status": getattr(r, "epistemic_status", None),
|
||||
}
|
||||
)
|
||||
data.sort(key=lambda x: (x["predicate"], tuple(x["args"])))
|
||||
return data
|
||||
|
||||
|
||||
# --- Scenarios ---
|
||||
|
||||
def _is_a_climb() -> dict[str, Any]:
|
||||
|
|
@ -139,6 +162,17 @@ def _is_a_climb() -> dict[str, Any]:
|
|||
if "kingdom" in [r.relation_arguments[1] for r in recall_realized(ctx, subject="rex", predicate="member")]:
|
||||
wrong_total += 1
|
||||
|
||||
# Semantic Answerability (Claim B): explicitly call determine on positives post-fixed-point
|
||||
# and assert Determined(True) with rule='direct' (the materialized direct path).
|
||||
positive_queries = [q for q in queries if "kingdom" not in q[0].lower()]
|
||||
for q, is_rel in positive_queries:
|
||||
det = _ask(ctx, q) if not is_rel else _ask_rel(ctx, q)
|
||||
assert isinstance(det, Determined) and getattr(det, "answer", False) and getattr(det, "rule", None) == "direct", (
|
||||
f"Claim B semantic answerability failed for {q}: got {det}"
|
||||
)
|
||||
|
||||
closure = _get_closure_content(ctx)
|
||||
|
||||
return {
|
||||
"scenario": "is_a_climb",
|
||||
"before": before,
|
||||
|
|
@ -148,6 +182,8 @@ def _is_a_climb() -> dict[str, Any]:
|
|||
"at_fixed_point": (fp.facts_consolidated == 0),
|
||||
"wrong_total": wrong_total,
|
||||
"proposals_emitted": 0, # separate flag run below
|
||||
"closure": closure,
|
||||
"semantic_positives_determined_direct": len(positive_queries),
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -176,12 +212,23 @@ def _relational_climb_less_than() -> dict[str, Any]:
|
|||
if isinstance(neg, Determined):
|
||||
wrong += 1
|
||||
|
||||
# Semantic Answerability (Claim B)
|
||||
for q, is_rel in queries:
|
||||
det = _ask_rel(ctx, q)
|
||||
assert isinstance(det, Determined) and getattr(det, "answer", False) and getattr(det, "rule", None) == "direct", (
|
||||
f"Claim B semantic answerability failed for {q}: got {det}"
|
||||
)
|
||||
|
||||
closure = _get_closure_content(ctx)
|
||||
|
||||
return {
|
||||
"scenario": "less_than_climb",
|
||||
"before": before,
|
||||
"after_tick_1": after1,
|
||||
"after_fixed_point": after_fp,
|
||||
"wrong_total": wrong,
|
||||
"closure": closure,
|
||||
"semantic_positives_determined_direct": len(queries),
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -199,11 +246,22 @@ def _temporal_climb() -> dict[str, Any]:
|
|||
rt.idle_tick()
|
||||
after = _count_answerable(ctx, queries)
|
||||
|
||||
# Semantic Answerability (Claim B)
|
||||
for q, is_rel in queries:
|
||||
det = _ask_rel(ctx, q)
|
||||
assert isinstance(det, Determined) and getattr(det, "answer", False) and getattr(det, "rule", None) == "direct", (
|
||||
f"Claim B semantic answerability failed for {q}: got {det}"
|
||||
)
|
||||
|
||||
closure = _get_closure_content(ctx)
|
||||
|
||||
return {
|
||||
"scenario": "before_event_climb",
|
||||
"before": before,
|
||||
"after_fixed_point": after,
|
||||
"wrong_total": 0,
|
||||
"closure": closure,
|
||||
"semantic_positives_determined_direct": len(queries),
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -231,36 +289,48 @@ def _negatives_refused() -> dict[str, Any]:
|
|||
|
||||
|
||||
def _proposal_flag_effect() -> dict[str, Any]:
|
||||
"""Measure proposal bridge: call emit only when "enabled" (simulating the runtime flag).
|
||||
The runtime flag in idle_tick gates the call to this bridge."""
|
||||
from generate.determine.consolidate import consolidate_once
|
||||
from generate.determine.derived_close_proposals import emit_derived_close_proposals
|
||||
"""Measure proposal bridge via *lived* runtime flag path.
|
||||
Uses real ChatRuntime.idle_tick() + IdleTickResult.derived_close_proposals_emitted
|
||||
(the runtime flag in idle_tick gates the call to the bridge)."""
|
||||
import generate.determine.derived_close_proposals as dcp
|
||||
import tempfile
|
||||
|
||||
# "without" : do not call emit
|
||||
no_flag = 0
|
||||
# "without" via real idle_tick with flag off (must emit 0)
|
||||
ctx_off, rt_off = _fresh_ctx(consolidate=True, proposals=False)
|
||||
_tell_rel(ctx_off, "A is less than B.")
|
||||
_tell_rel(ctx_off, "B is less than C.")
|
||||
res_off = rt_off.idle_tick()
|
||||
emitted_without = res_off.derived_close_proposals_emitted
|
||||
|
||||
# "with" : consolidate then emit (as the bridge does when flag on)
|
||||
ctx, _ = _fresh_ctx(consolidate=True, proposals=True)
|
||||
_tell_rel(ctx, "A is less than B.")
|
||||
_tell_rel(ctx, "B is less than C.")
|
||||
consolidate_once(ctx)
|
||||
# "with" via real idle_tick with flag on (captures from IdleTickResult; temp sink for isolation + content)
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
sink = Path(tmp)
|
||||
counts = emit_derived_close_proposals(ctx, sink=sink)
|
||||
with_flag = counts.get("emitted", 0)
|
||||
orig_sink = dcp.DEFAULT_SINK
|
||||
dcp.DEFAULT_SINK = sink
|
||||
try:
|
||||
ctx_on, rt_on = _fresh_ctx(consolidate=True, proposals=True)
|
||||
_tell_rel(ctx_on, "A is less than B.")
|
||||
_tell_rel(ctx_on, "B is less than C.")
|
||||
res_on = rt_on.idle_tick()
|
||||
emitted_with = res_on.derived_close_proposals_emitted
|
||||
# capture full bodies for content checksum (lived emission path)
|
||||
proposals = [json.loads(p.read_text()) for p in sorted(sink.glob("*.json"))]
|
||||
finally:
|
||||
dcp.DEFAULT_SINK = orig_sink
|
||||
|
||||
return {
|
||||
"scenario": "proposal_flag",
|
||||
"emitted_without_flag": no_flag,
|
||||
"emitted_with_flag": with_flag,
|
||||
"only_with_flag": with_flag > 0 and no_flag == 0,
|
||||
"emitted_without_flag": emitted_without,
|
||||
"emitted_with_flag": emitted_with,
|
||||
"only_with_flag": emitted_with > 0 and emitted_without == 0,
|
||||
"proposals": proposals,
|
||||
}
|
||||
|
||||
|
||||
def run() -> dict[str, Any]:
|
||||
"""Run the full yardstick. Returns report with climb metrics, wrong_total=0,
|
||||
proposal flag isolation, replay checksum."""
|
||||
lived proposal flag isolation (via IdleTickResult), semantic determine() on positives,
|
||||
replay_checksum (aggregates) + content_replay_checksum (closures + proposal bodies)."""
|
||||
is_a = _is_a_climb()
|
||||
less = _relational_climb_less_than()
|
||||
temporal = _temporal_climb()
|
||||
|
|
@ -294,7 +364,7 @@ def run() -> dict[str, Any]:
|
|||
},
|
||||
}
|
||||
|
||||
# Replay checksum: stable sizes + wrongs + flag effect
|
||||
# Replay checksum: stable sizes + wrongs + flag effect (kept for compatibility)
|
||||
checksum_input = json.dumps(
|
||||
{
|
||||
"sizes": [report["aggregate"][k] for k in ("direct_answerable_before", "direct_answerable_after_tick_1", "direct_answerable_after_fixed_point")],
|
||||
|
|
@ -305,6 +375,19 @@ def run() -> dict[str, Any]:
|
|||
).encode()
|
||||
report["replay_checksum"] = hashlib.sha256(checksum_input).hexdigest()
|
||||
|
||||
# Parallel full-content checksum for Claim B fidelity (canonical closures + proposal bodies)
|
||||
# This aligns the yardstick with its documentation ("closure sets", "exact trajectories").
|
||||
content = {
|
||||
"closures": {
|
||||
"is_a": is_a.get("closure", []),
|
||||
"less_than": less.get("closure", []),
|
||||
"before_event": temporal.get("closure", []),
|
||||
},
|
||||
"proposals": prop.get("proposals", []),
|
||||
}
|
||||
content_input = json.dumps(content, sort_keys=True).encode()
|
||||
report["content_replay_checksum"] = hashlib.sha256(content_input).hexdigest()
|
||||
|
||||
return report
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue