core-labs/core
core-labs/core
1
0
Fork 0
Code Issues 11 Pull requests 2 Projects Releases Packages Wiki Activity Actions
core/evals/close_derived_climb/contract.md
Shay cf1e73716c
feat(eval): yardstick for CLOSE derived climb (#790) 
* feat(eval): add yardstick lane for CLOSE derived climb (is-a + relational + flag + wrong=0)

* fix(eval): correct proposal flag measurement in yardstick (direct emit for isolation; runner now reports True)

* fix(eval): measure CLOSE climb from live queries

* fix(eval): require strict CLOSE climb growth
2026-06-16 16:09:56 -07:00

899 B
Raw Blame History

CLOSE Derived Climb Yardstick (PR-3)

This lane measures the monotone growth in directly-answerable set enabled by:

  • PR #788: relational transitive CLOSE (less/greater/before/after now consolidate like is-a)
  • PR #789: derived facts emit review proposals when flag enabled

Metrics

  • direct_answerable_before / after_tick_1 / after_fixed_point
  • wrong_total (must 0; negatives and excluded preds refused)
  • proposals_only_with_flag (emitted >0 only when review_derived_close_proposals=True)
  • replay_checksum stable

Scenarios

  • is-a (member/subset) climb
  • less_than relational climb
  • before_event temporal climb
  • parent/sibling negatives refused (wrong=0)
  • proposal emission gated by flag

No side effects

  • No change to serving, determine, or ratification.
  • Uses real idle_tick path with flags.

Run: python -m evals.close_derived_climb Replays the exact trajectories for audit.