core-labs/core
core-labs/core
1
0
Fork 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity
core/evals/close_derived_climb/contract.md
Shay f03979f025 feat: integrate hardened CLOSE yardstick into determinism & teaching regression surfaces 
- docs/testing-lanes.md + Makefile: recommended determinism regression invocation (uv run python -m evals.close_derived_climb + contract pytest) as part of standard verification story / rerun flows.
- evals/anti_regression/run_demo.py + tests/test_anti_regression_demo.py: hermetic embedding of the yardstick (lived IdleTickResult flag, semantic determine rule=direct, content_replay_checksum) into the core anti-regression / teaching demo flow (core demo anti-regression now carries + reports the Claim-B signals; active corpus remains untouched).
- Supporting docs: contract.md (uv + refs), anti_regression_demo.md (complementary note), runtime_contracts.md (determination surface cross-ref).
- New ratification artifact (pre-impl) + all success criteria met.

See ratification for chosen approach + why only correct path.
All existing tests/invariants green; hermeticity preserved; no core/engine changes.
2026-06-16 17:25:05 -07:00

2.2 KiB
Raw Blame History

CLOSE Derived Climb Yardstick (PR-3, hardened for Claim B)

This lane measures the monotone growth in directly-answerable set enabled by:

  • PR #788: relational transitive CLOSE (less/greater/before/after now consolidate like is-a)
  • PR #789: derived facts emit review proposals when flag enabled

Metrics (Claim A + Claim B)

  • direct_answerable_before / after_tick_1 / after_fixed_point (vault growth)
  • wrong_total (must 0; negatives and excluded preds refused)
  • proposals_only_with_flag (measured via real ChatRuntime.idle_tick() + IdleTickResult.derived_close_proposals_emitted; >0 only when review_derived_close_proposals=True)
  • semantic_positives_determined_direct (explicit determine() calls on positives post-fixed-point assert Determined(True) with rule='direct')
  • replay_checksum (aggregates for compatibility)
  • content_replay_checksum (canonical closure sets with structure_key + Derivation/premise_structure_keys + proposal bodies for exact-trajectory fidelity)

Scenarios

  • is-a (member/subset) climb
  • less_than relational climb
  • before_event temporal climb
  • parent/sibling negatives refused (wrong=0)
  • proposal emission gated by flag (lived idle_tick path)

No side effects

  • No change to serving, determine, or ratification.
  • Uses real idle_tick path with flags for consolidation and proposal emission.
  • All realization remains SPECULATIVE; proposals are proposal_only + requires_review.

Run: uv run python -m evals.close_derived_climb Replays the exact trajectories (aggregates + full content) for audit. Now qualifies as full lived-runtime Claim B yardstick per post-merge hardening audit.

Integrated into the project's standard determinism regression and teaching/anti-regression surfaces (see docs/testing-lanes.md "Recommended determinism / teaching regression invocation", docs/evals/anti_regression_demo.md, and the integration ratification docs/analysis/integrate-hardened-close-yardstick-determinism-teaching-regression-ratification-2026-06-16.md). The prior Claim-B hardening ratification is at docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md. Cross-references runtime determination surface in docs/runtime_contracts.md.