core/evals
Shay cc6f13a939
feat(ADR-0167/W3-A): e2e determinism + cognition regression — LexicalClaim slice closed (#357)
Wave 3, closes the LexicalClaim slice of ADR-0167.  After this PR the
math reader's refusal taxonomy is evidence, not terminus: lexical
refusals flow through audit row → typed evidence → dedup signature →
HITL ratification (W2-D) → pack write → next-audit-pass-resolves.

Deliverables
------------
- tests/test_math_evidence_e2e.py (new, 7 tests):
  * test_full_pipeline_from_audit_to_evidence
  * test_e2e_replay_equivalence
  * test_lexical_ratification_advances_unknown_word_row (case 0040 'sees')
  * test_e2e_determinism_across_processes
  * test_cognition_teaching_corridor_unaffected
  * test_evidence_dedup_via_claim_signature
  * test_audit_artifact_round_trip_with_signatures
- evals/gsm8k_math/train_sample/v1/audit_brief_11.md: Post-W2 baseline
  table + cognition regression line + case 0050 hazard status + pointer
  to the new e2e regression module.
- tests/test_candidate_domain_partition.py: minimal allowlist patch to
  test_existing_cognition_tests_untouched so that future ADR-0167 PRs
  can add their own evidence test files without tripping a structurally
  brittle hard-coded whitelist (W2-C partition risk; recorded in PR body).

Hard constraints held
---------------------
- wrong == 0: case 0050 hazard still refuses at sentence_index 0
  after the tmpdir-pack 'sees' ratification; no admission introduced.
- Cognition regression: zero modifications to cognition test bodies;
  only the W2-C whitelist assertion was loosened.
- Determinism: in-process and cross-process evidence_hash byte-identical.
- No real-pack mutation: a per-test digest fixture asserts
  language_packs/data/en_core_math_v1/ is byte-identical before and
  after each test.

Out of scope
------------
- Frame/Composition/Reference/Slot ratification handlers (follow-up ADRs).
- Workbench v1 wiring of math candidates (ADR-0167 §Q4).
- Auto-ratification — HITL only, forever.
- The two partition risks Gemini flagged in W2-C (cognition pack indexing,
  replay-gate default) remain follow-up.

With this PR merged the engine can ratify math-domain lexical claims
from its own refusal evidence through the existing HITL teaching
corridor — the thesis claim of ADR-0167 becomes a concrete green test.
2026-05-27 07:27:24 -07:00
..
adversarial_identity
anchor_lens_tour
anti_regression feat(epistemic): Phase 3 state tagging spine (#220) 2026-05-24 11:26:06 -07:00
articulation
articulation_of_status
audit_tour
calibration
classical_literature_ood
cognition
cold_start_grounding fix(quarantine): drain all 60 quarantined tests — QUARANTINE=∅ (#267) 2026-05-25 11:22:12 -07:00
compositionality
compound_intent_decomposition
contemplation_quality feat(W-025): contemplation quality eval lane (ADR-0159) (#286) 2026-05-25 20:38:52 -07:00
contradiction_detection
conversation
conversational_thread_coherence
cross_domain_transfer
curriculum_loop_closure
demo_composition fix(W-015): replace _slerp_toward with rotor-geodesic anchor pull (#255) 2026-05-24 20:05:25 -07:00
deterministic_fluency
discourse_paragraph
domain_contract_validation chore(ci): re-pin drifted lane SHAs + refresh canonical reports (#229) 2026-05-24 14:25:11 -07:00
elementary_mathematics_ood
english_fluency_ood
fabrication_control chore(ci): re-pin drifted lane SHAs + refresh canonical reports (#229) 2026-05-24 14:25:11 -07:00
forward_semantic_control
foundational_biology_ood
foundational_physics_ood
frontier_compare
grammatical_coverage
gsm8k_math feat(ADR-0167/W3-A): e2e determinism + cognition regression — LexicalClaim slice closed (#357) 2026-05-27 07:27:24 -07:00
gsm8k_parser_dev
hebrew_fluency
identity_divergence
industry_demos
inference_closure
introspection
koine_greek_fluency
lab
learning_arc fix(W-025): polish contemplation-quality eval lane follow-ups (#290) 2026-05-26 09:39:18 -07:00
learning_loop
long_context_cost
math_bounded_grammar/v1
math_capability_axes feat(ADR-0163.D.4): question grammar extension — mass nouns, comparatives, pronoun-entity resolution (#310) 2026-05-26 16:19:37 -07:00
math_expert_claims/v1 fix(quarantine): drain all 60 quarantined tests — QUARANTINE=∅ (#267) 2026-05-25 11:22:12 -07:00
math_symbolic_equivalence
math_teaching_corpus/v1
miner_loop_closure
monotonic_learning
multi_agent_composition
multi_sentence_response
multi_step_reasoning
obligation_2_ood_ratio feat(ADR-0114a.2): OOD-ratio auditor — Obligation #2 wired for B3, ratio=1.00 (#193) 2026-05-23 16:25:28 -07:00
obligation_5_perturbation feat(ADR-0114a.5): reasoning-isolation perturbation suite — Obligation #5 wired for B3, PASSING 130/130 preserving, 68/68 breaking (#191) 2026-05-23 16:07:59 -07:00
obligation_6_depth_curve feat(ADR-0114a.6): depth-curve auditor — Obligation #6 wired for B3 (assertion holds, coverage gap named) (#190) 2026-05-23 16:19:58 -07:00
obligation_8_adversarial feat(ADR-0114a.8): adversarial auditor — Obligation #8 wired, PASSING; surfaces 2 known parser-layer gaps (#192) 2026-05-23 16:11:37 -07:00
obligation_10_pack_provenance
orthogonality_tour
prompt_diversity
provenance
public_demo chore(ratify): accept four Phase C round-2 recognizers (round 2) (#309) 2026-05-26 16:03:01 -07:00
realizer_guard
refusal_calibration
refusal_taxonomy feat(ADR-0163.B.2): four new exemplar corpora — discrete_count_statement, multiplicative_aggregation, currency_amount, plus temporal_aggregation v2 widening (#306) 2026-05-26 14:36:59 -07:00
register_diagnostics
register_tour
reports
results
reviewer_registry
sample_efficiency
self_consistency_over_time
symbolic_logic
teaching_injection_resistance
walkthrough_chain
warmed_session_consistency
zero_code_domain_acquisition
__init__.py
_parallel.py
baseline_runner.py
CLAIMS.md
cognition_cases.jsonl
framework.py
holdout_runner.py
metrics.py
parallel.py
run_cognition_eval.py feat(W-008): L10 Shape B hybrid engine-state persistence (#271) 2026-05-25 11:45:54 -07:00