core/evals
Shay a8c12670ec fix(capability): correct discourse_planner flag catalog + commit-independent public_demo pin
Two pre-existing latent issues fixed:

1. discourse_planner flag catalog drift (test_flag_report failure)

   On 2026-05-21 the discourse_planner default was flipped to True
   after byte-equality verification (per inline comment in
   core/config.py:130-138), but the capability flag catalog at
   core/capability/reporting.py was not updated — it still claimed
   "flag_shipped_default_off". The test
   test_flag_report_tracks_default_off_flags_without_enabling_them
   correctly caught the inconsistency; it had been failing across
   every commit since ADR-0092 first ran the suite.

   Fix:
   - New "flag_shipped_default_on" state in _FLAG_CATALOG, added
     to flag_report() grouped output
   - discourse_planner moved from default_off → default_on
   - Test renamed to test_flag_report_classification_matches_actual_defaults,
     enforces BOTH directions of the contract (catalog claim must
     match DEFAULT_CONFIG value)
   - New test test_flag_catalog_state_is_consistent_with_default_config
     cross-checks every catalog entry against DEFAULT_CONFIG;
     catches future drift before it lands

2. public_demo lane SHA shifted every commit

   Each commit advances the showcase's generated_at_revision field
   (git HEAD SHA). _strip_volatile in the lane runner was stripping
   wall-clock and per-run paths but NOT generated_at_revision, so
   the byte-equality case's details.sha256 changed with every commit
   even when underlying demos produced identical content. That made
   the pin a "did this run today" check rather than a "did the code
   produce the right artifact" check — exactly the failure mode
   the verifier was supposed to prevent.

   Fix:
   - Add generated_at_revision to _VOLATILE_KEYS in the public_demo
     runner. Lane's invariant is "same code → same SHA," not
     "same HEAD → same SHA"; HEAD belongs in the showcase output
     (operators need it) but not in the lane's equality projection.
   - Pin refreshed once to capture the now-commit-independent SHA;
     subsequent commits won't shift it unless underlying demo content
     actually changes.

After fix:
- Capability tests: 6/6 passing (was 4/5 with discourse_planner failing)
- Lane SHAs: 6/6 match pinned values; public_demo pin will now survive
  routine code changes
- Smoke 67/67, cognition eval byte-identical 100/100/100/100

This is the single known pre-existing test failure cleaned up.
2026-05-21 20:53:15 -07:00
..
adversarial_identity
anchor_lens_tour
anti_regression chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
articulation
articulation_of_status
audit_tour
calibration
classical_literature_ood
cognition
cold_start_grounding
compositionality
compound_intent_decomposition
contradiction_detection
conversation
conversational_thread_coherence
cross_domain_transfer
demo_composition feat(demos): implement ADR-0098 — Demo Composition Contract 2026-05-21 19:02:29 -07:00
deterministic_fluency
discourse_paragraph
domain_contract_validation feat(capability): implement ADR-0093 — Domain Pack Contract v1 wired in 2026-05-21 18:33:23 -07:00
elementary_mathematics_ood
english_fluency_ood
fabrication_control feat(evals): implement ADR-0096 — Fabrication-Control Eval Lane 2026-05-21 18:44:25 -07:00
forward_semantic_control
foundational_biology_ood
foundational_physics_ood
frontier_compare feat(frontier): add replay variability suite and token-cost telemetry (#66) 2026-05-20 15:04:34 -07:00
grammatical_coverage
hebrew_fluency
identity_divergence
industry_demos evals/industry_demos: add run_all.py suite runner 2026-05-21 08:23:29 -07:00
inference_closure
introspection
koine_greek_fluency
lab research(evals): phi separation probe for ADR-0081 follow-up (#57) 2026-05-20 12:34:59 -07:00
learning_loop chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
long_context_cost chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
miner_loop_closure feat(teaching): implement ADR-0095 — Miner-Sourced Teaching Proposals 2026-05-21 18:18:51 -07:00
monotonic_learning
multi_agent_composition
multi_sentence_response
multi_step_reasoning
orthogonality_tour
prompt_diversity feat(adr-0085): gloss-aware CAUSE composer — explanation frame from glosses (#70) 2026-05-20 15:55:08 -07:00
provenance
public_demo fix(capability): correct discourse_planner flag catalog + commit-independent public_demo pin 2026-05-21 20:53:15 -07:00
realizer_guard chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
refusal_calibration
register_diagnostics feat(evals): per-intent register-firing diagnostic + CI gate + tests (#103) 2026-05-21 07:05:23 -07:00
register_tour
reports
results feat: ADR-0086 + ADR-0087 + 100-register catalog — cognition lane closure 2026-05-21 00:08:12 -07:00
reviewer_registry feat(capability): implement ADR-0092 — Reviewer Registry v1 2026-05-21 18:01:24 -07:00
sample_efficiency
self_consistency_over_time
symbolic_logic
teaching_injection_resistance
walkthrough_chain
warmed_session_consistency
zero_code_domain_acquisition
__init__.py
_parallel.py
baseline_runner.py
CLAIMS.md chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
cognition_cases.jsonl
framework.py
holdout_runner.py
metrics.py
parallel.py
run_cognition_eval.py