d90887b80f
ADR-0175 Phase 2 — a NEW lane (evals/gsm8k_math/practice/v1/), separate from the
wrong=0-pinned serving runner which is NOT modified. Runs the 50 cases in
practice mode: scores correct/wrong/refused as practice metrics, feeds per-class
counts into the Phase 1 ledger, diagnoses every refusal (§8), emits an
elimination record per wrong.
- classify_operation: gold-derived primary op class {multiplicative,divisive,
additive} from <<a*b=c>> calc annotations (Tier-1 checkable in practice).
- diagnose_refusal (§8): skill_gap / knowledge_gap / genuine_ambiguity router.
- EliminationRecord (§9): wrong attempt gold caught -> pruning signal.
- PracticeReport: counts + per-class ledger + diagnoses + eliminations; as_dict.
- run_practice(cases, scorer=...): injectable scorer for tests; defaults to the
candidate-graph scorer (read-only — never alters serving).
Live result mirrors serving (3 correct / 0 wrong / 47 refused of 50) because the
engine still refuses rather than guesses — attempts/eliminations go live in
Phase 3. But the diagnosis is already actionable: 35 skill_gap / 12 knowledge_gap
/ 0 genuine_ambiguity — 74% of refusals are skill gaps (Phase 3's search target),
quantifying the skill-vs-knowledge split.
Invariants: #1 seal (serving still 3/47/0; no generate/chat import of the lane),
#3 determinism (report byte-identical across runs). Elimination + wrong-tolerance
paths unit-tested via injected scorer (no live wrongs yet).
Verified: Phase 1+2 53/53, serving train_sample tests 4/4 (seal), smoke 67/67,
ruff clean.
|
||
|---|---|---|
| .. | ||
| adversarial_identity | ||
| anchor_lens_tour | ||
| anti_regression | ||
| articulation | ||
| articulation_of_status | ||
| audit_tour | ||
| calibration | ||
| classical_literature_ood | ||
| cognition | ||
| cold_start_grounding | ||
| compositionality | ||
| compound_intent_decomposition | ||
| contemplation_quality | ||
| contradiction_detection | ||
| conversation | ||
| conversational_thread_coherence | ||
| cross_domain_transfer | ||
| curriculum_loop_closure | ||
| demo_composition | ||
| deterministic_fluency | ||
| discourse_paragraph | ||
| domain_contract_validation | ||
| elementary_mathematics_ood | ||
| english_fluency_ood | ||
| fabrication_control | ||
| flywheel_demo | ||
| forward_semantic_control | ||
| foundational_biology_ood | ||
| foundational_physics_ood | ||
| frontier_compare | ||
| grammatical_coverage | ||
| gsm8k_math | ||
| gsm8k_parser_dev | ||
| hebrew_fluency | ||
| identity_divergence | ||
| industry_demos | ||
| inference_closure | ||
| introspection | ||
| koine_greek_fluency | ||
| lab | ||
| learning_arc | ||
| learning_loop | ||
| long_context_cost | ||
| math_bounded_grammar/v1 | ||
| math_capability_axes | ||
| math_expert_claims/v1 | ||
| math_symbolic_equivalence | ||
| math_teaching_corpus/v1 | ||
| miner_loop_closure | ||
| monotonic_learning | ||
| multi_agent_composition | ||
| multi_sentence_response | ||
| multi_step_reasoning | ||
| obligation_2_ood_ratio | ||
| obligation_5_perturbation | ||
| obligation_6_depth_curve | ||
| obligation_8_adversarial | ||
| obligation_10_pack_provenance | ||
| orthogonality_tour | ||
| prompt_diversity | ||
| provenance | ||
| public_demo | ||
| realizer_guard | ||
| refusal_calibration | ||
| refusal_taxonomy | ||
| register_diagnostics | ||
| register_tour | ||
| reports | ||
| results | ||
| reviewer_registry | ||
| sample_efficiency | ||
| self_consistency_over_time | ||
| symbolic_logic | ||
| teaching_injection_resistance | ||
| walkthrough_chain | ||
| warmed_session_consistency | ||
| zero_code_domain_acquisition | ||
| __init__.py | ||
| _parallel.py | ||
| baseline_runner.py | ||
| CLAIMS.md | ||
| cognition_cases.jsonl | ||
| framework.py | ||
| holdout_runner.py | ||
| metrics.py | ||
| parallel.py | ||
| run_cognition_eval.py | ||