6a4d356ce9
Builds the corpus from the ADR-0163-F2 spec: 30 hand-curated, real-sourced cases across the proven misfire categories (disguised-polarity, pseudo-accumulation/ fractions, multi-referent H1, multi-actor-pronoun ADR-0174, distractor-quantity, temporal-scope H3, comparative-referent H2, unit-confuser) + genuine-positive minimal-pair twins. Schema carries category/surface_trap/expected/pair_id/source. The runner scores OPPOSITE to a coverage lane: the bar is `wrong` -> 0 (a confuser *answered* is a defect regardless of value) plus pair-consistency (solving a twin but answering its confuser = a surface-matching tell). It runs the realistic sealed attempt (accumulation -> multiplicative -> chain, first to resolve). Honest measured baseline (the probe's whole point — these are the defects the templated corpus hid): 30 cases -> 7 solved / 15 refused / 7 WRONG / 1 spurious; 4 pair-tells (0001/0003/0014/0020). Wrong by category: disguised-polarity 2 (buys-a-toy-for-30 -> +30), pseudo-accumulation 2 (the 0002 cable/fraction), distractor-quantity 2, temporal-scope 1 (before-giving -> gave the now-value). Per the overfitting lesson, the composers are NOT reactively patched to pass the probe (that is the trap). The baseline is pinned as a no-regression gate (wrong <= 7, pair-tells <= 4, positives keep solving); future fixes must be GENERAL mechanisms validated on train_sample, driving wrong down. Sealed: serving 3/47/0 byte-identical (lane-SHA 8/8, claims OK); architectural invariants green. |
||
|---|---|---|
| .. | ||
| adversarial_identity | ||
| anchor_lens_tour | ||
| anti_regression | ||
| articulation | ||
| articulation_of_status | ||
| audit_tour | ||
| calibration | ||
| classical_literature_ood | ||
| cognition | ||
| cold_start_grounding | ||
| compositionality | ||
| compound_intent_decomposition | ||
| contemplation_quality | ||
| contradiction_detection | ||
| conversation | ||
| conversational_thread_coherence | ||
| cross_domain_transfer | ||
| curriculum_loop_closure | ||
| demo_composition | ||
| deterministic_fluency | ||
| discourse_paragraph | ||
| domain_contract_validation | ||
| elementary_mathematics_ood | ||
| english_fluency_ood | ||
| fabrication_control | ||
| flywheel_demo | ||
| forward_semantic_control | ||
| foundational_biology_ood | ||
| foundational_physics_ood | ||
| frontier_compare | ||
| grammatical_coverage | ||
| gsm8k_math | ||
| gsm8k_parser_dev | ||
| hebrew_fluency | ||
| identity_divergence | ||
| industry_demos | ||
| inference_closure | ||
| introspection | ||
| koine_greek_fluency | ||
| lab | ||
| learning_arc | ||
| learning_loop | ||
| long_context_cost | ||
| math_bounded_grammar/v1 | ||
| math_capability_axes | ||
| math_expert_claims/v1 | ||
| math_symbolic_equivalence | ||
| math_teaching_corpus/v1 | ||
| miner_loop_closure | ||
| monotonic_learning | ||
| multi_agent_composition | ||
| multi_sentence_response | ||
| multi_step_reasoning | ||
| obligation_2_ood_ratio | ||
| obligation_5_perturbation | ||
| obligation_6_depth_curve | ||
| obligation_8_adversarial | ||
| obligation_10_pack_provenance | ||
| orthogonality_tour | ||
| prompt_diversity | ||
| provenance | ||
| public_demo | ||
| realizer_guard | ||
| refusal_calibration | ||
| refusal_taxonomy | ||
| register_diagnostics | ||
| register_tour | ||
| reports | ||
| results | ||
| reviewer_registry | ||
| sample_efficiency | ||
| self_consistency_over_time | ||
| symbolic_logic | ||
| teaching_injection_resistance | ||
| walkthrough_chain | ||
| warmed_session_consistency | ||
| zero_code_domain_acquisition | ||
| __init__.py | ||
| _parallel.py | ||
| baseline_runner.py | ||
| CLAIMS.md | ||
| cognition_cases.jsonl | ||
| framework.py | ||
| holdout_runner.py | ||
| metrics.py | ||
| parallel.py | ||
| run_cognition_eval.py | ||