core/evals
Shay 35a29ed2de
fix(tests): G2 comparative-counter excludes recognizer-path refusals + refresh report.json (#375)
The G.2 test \`_comparative_clause_refusal_count\` reads \`report.json\`
and counts refusals whose reason quotes a statement clause containing
comparative anchors ("more/less than", "twice as many", etc.). After
#359's wrong=0 fix, the candidate-graph emits two refusal-reason
families that both quote a statement:

1. "no admissible candidate for statement: '...'" — parser-path
   refusal (the comparative-parse-failure family this metric tracks).
2. "recognizer matched but produced no injection for statement:
   '...'" — recognizer-path refusal; the quoted statement may
   incidentally contain comparative anchors but the refusal cause is
   the missing injector, NOT the comparative parse.

The pre-#359 counter only saw family (1) reasons; post-#359 it
over-counts whenever a recognizer-path refusal quotes a statement
containing comparative anchors. This was the test failure A2's PR
(#369) and the cleanup PR (#373) both surfaced.

## Fix

Filter the counter to exclude family (2) explicitly. Recognizer-path
refusals are tracked separately by the recognizer-wiring test suite;
they don't belong in the G.2 metric.

Result on current main:
- total statements with comparative anchors in refusal reasons: 2
- parser-path: 1 (case 0009, the legitimate G.2-tracked refusal)
- recognizer-path: 1 (filtered out — incidental anchor in #359-format reason)
- G.2 metric correctly reports 1 < baseline 2 → assertion passes

## Also: refresh report.json

The checked-in \`report.json\` was generated pre-#359 with the legacy
refusal-reason format. The runner now emits the new format on every
run; checking in the current output makes the baseline reproducible
and clears the CI friction that A2 originally flagged.

## Test plan

- tests/test_adr_0131_G2_comparatives.py: 25 passed (was 24 pass / 1 fail)
- tests/test_adr_0131_G4_multi_clause.py + G5_aggregate + S1_rate_events: 105 passed
- tests/test_brief_11b_audit_artifact + step2_lexicon + recognizer_skip + brief_11_audit + wiring + partition + adr_0163_d2: 89 passed
- Total: 219 passed

## Hard invariants

- No runtime change
- wrong=0 invariant preserved
- ADR-0166: no new eval lanes
- No teaching-store / pack mutation
2026-05-27 11:26:25 -07:00
..
adversarial_identity
anchor_lens_tour
anti_regression
articulation
articulation_of_status
audit_tour
calibration
classical_literature_ood
cognition
cold_start_grounding
compositionality
compound_intent_decomposition
contemplation_quality
contradiction_detection
conversation
conversational_thread_coherence
cross_domain_transfer
curriculum_loop_closure
demo_composition
deterministic_fluency
discourse_paragraph
domain_contract_validation
elementary_mathematics_ood
english_fluency_ood
fabrication_control
forward_semantic_control
foundational_biology_ood
foundational_physics_ood
frontier_compare
grammatical_coverage
gsm8k_math fix(tests): G2 comparative-counter excludes recognizer-path refusals + refresh report.json (#375) 2026-05-27 11:26:25 -07:00
gsm8k_parser_dev
hebrew_fluency
identity_divergence
industry_demos
inference_closure
introspection
koine_greek_fluency
lab
learning_arc
learning_loop
long_context_cost
math_bounded_grammar/v1
math_capability_axes
math_expert_claims/v1
math_symbolic_equivalence
math_teaching_corpus/v1
miner_loop_closure
monotonic_learning
multi_agent_composition
multi_sentence_response
multi_step_reasoning
obligation_2_ood_ratio
obligation_5_perturbation
obligation_6_depth_curve
obligation_8_adversarial
obligation_10_pack_provenance
orthogonality_tour
prompt_diversity
provenance
public_demo
realizer_guard
refusal_calibration
refusal_taxonomy
register_diagnostics
register_tour
reports
results
reviewer_registry
sample_efficiency
self_consistency_over_time
symbolic_logic
teaching_injection_resistance
walkthrough_chain
warmed_session_consistency
zero_code_domain_acquisition
__init__.py
_parallel.py
baseline_runner.py
CLAIMS.md
cognition_cases.jsonl
framework.py
holdout_runner.py
metrics.py
parallel.py
run_cognition_eval.py