core-labs/core
core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity
core/evals/close_derived_climb/contract.md
Shay bad7c9f79d feat: create dedicated CLOSE Flywheel Regression Surface (Claim-B Level) 
- Makefile: new 'test-close-flywheel' target (clearly named heavy surface; runs full Claim-B yardstick + anti embedding; explicitly not in fast/slow/full).
- docs/testing-lanes.md: new prominent 'Dedicated CLOSE Flywheel Regression Surface (Claim-B Level)' section with purpose, invocation (make primary), full Claim-B capabilities list, runtime (~60s+ heavy), hermeticity guarantees, Engineering Pillars alignment (Mechanical Sympathy / Semantic Rigor / Third Door per Whitepaper), and all cross-refs (ratifs, contract, anti demo, etc.).
- Cleanly embed / polish (additive, hermetic, building on #792): anti_regression/run_demo.py (comments + RESULT label the surface + make target), test_anti_regression_demo.py (updated docstring), docs/evals/anti_regression_demo.md (how-to + complementary section updated for surface participation).
- Supporting: contract.md (dedicated surface note + ratif links), runtime_contracts.md (tightened ref).

Ratified first (docs/analysis/close-flywheel-dedicated-regression-surface-ratification-2026-06-16.md). Strictly in scope. Preserves all invariants (wrong_total=0, determinism, proposal-only, etc.). No core, no CLI additions, no fast/CI inclusion, no teaching logic changes.

See ratification for pillars justification and 'why only correct path'.
2026-06-16 17:50:37 -07:00

2.7 KiB
Raw Blame History

CLOSE Derived Climb Yardstick (PR-3, hardened for Claim B)

This lane measures the monotone growth in directly-answerable set enabled by:

  • PR #788: relational transitive CLOSE (less/greater/before/after now consolidate like is-a)
  • PR #789: derived facts emit review proposals when flag enabled

Metrics (Claim A + Claim B)

  • direct_answerable_before / after_tick_1 / after_fixed_point (vault growth)
  • wrong_total (must 0; negatives and excluded preds refused)
  • proposals_only_with_flag (measured via real ChatRuntime.idle_tick() + IdleTickResult.derived_close_proposals_emitted; >0 only when review_derived_close_proposals=True)
  • semantic_positives_determined_direct (explicit determine() calls on positives post-fixed-point assert Determined(True) with rule='direct')
  • replay_checksum (aggregates for compatibility)
  • content_replay_checksum (canonical closure sets with structure_key + Derivation/premise_structure_keys + proposal bodies for exact-trajectory fidelity)

Scenarios

  • is-a (member/subset) climb
  • less_than relational climb
  • before_event temporal climb
  • parent/sibling negatives refused (wrong=0)
  • proposal emission gated by flag (lived idle_tick path)

No side effects

  • No change to serving, determine, or ratification.
  • Uses real idle_tick path with flags for consolidation and proposal emission.
  • All realization remains SPECULATIVE; proposals are proposal_only + requires_review.

Run: uv run python -m evals.close_derived_climb Replays the exact trajectories (aggregates + full content) for audit. Now qualifies as full lived-runtime Claim B yardstick per post-merge hardening audit.

Dedicated surface: make test-close-flywheel (or the equivalent python -m + pytest commands). This is the clearly named, intentional Claim-B regression surface for heavier determinism regressions and teaching/anti-regression verification flows. See the full definition, capabilities, runtime characteristics, hermeticity guarantees, and Engineering Pillars alignment in docs/testing-lanes.md "Dedicated CLOSE Flywheel Regression Surface (Claim-B Level)".

Integrated into the project's standard determinism regression and teaching/anti-regression surfaces via the anti-regression demo embedding (see docs/evals/anti_regression_demo.md and tests/test_anti_regression_demo.py). See ratification docs/analysis/close-flywheel-dedicated-regression-surface-ratification-2026-06-16.md (this task), docs/analysis/integrate-hardened-close-yardstick-determinism-teaching-regression-ratification-2026-06-16.md (#792), and docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md (#791). Cross-references runtime determination surface in docs/runtime_contracts.md.