core/evals
Shay 7451e7cd74
feat(adr-0177-cp2a): cue-precision ledger training + measurement (+ unit hygiene) (#461)
CP-2a populates the CP-1 ledger from gold-labelled candidate readings and reports
per-pattern reliability — the measurement the cue-precision thesis rests on. Plus
the function-word unit filter, whose value this measurement makes concrete (clean
unit_shape labelling).

What landed (all sealed; serving 3/47/0 byte-identical):
- generate/cue_precision/trainer.py — train_from_cases(cases, enumerators): folds
  gold-labelled candidate chains into the ledger via record_case. Decoupled (the
  candidate enumerators are injected, so the package still imports nothing from
  search). candidates_for dedupes a reading shared by two enumerators.
- generate/derivation/multistep.py — extracted the enumeration half of search_chain
  into public candidate_chains(problem_text); search_chain now delegates (verified
  byte-identical: ms3 tests + practice counts unchanged). CP-2 needs the readings
  the search weighs, not just the one it resolves.
- generate/derivation/extract.py — function-word unit filter (_NON_UNIT_WORDS):
  blanks spurious function-word units ($0.75 each -> "", 3/4 of -> "") that
  corrupt same-unit detection and unit_shape. Closed lexeme set, ADR-0165-safe.
- evals/gsm8k_math/practice/v1/cue_precision_report.py — trains over 200 sealed
  cases (50 train_sample + 150 ADR-0163-F additive) with the real enumerators and
  prints the per-pattern reliability table.
- tests/test_adr_0177_cp2a_training.py — trainer obligations (credit/dedupe/
  determinism/empty) via synthetic enumerators; real-measurement well-formedness;
  search_chain parity.

Load-bearing finding (recorded in ADR-0177): over 200 cases EVERY (cue,op,unit_shape)
pattern floors at ~0.0 reliability (best: for-multiply-cross_unit 0.0116 at 2/34).
The blunt product/sum-of-all readings are almost always wrong vs gold, so the
conservative floor correctly trusts nothing. => CP-2b (trust reliable cues) is
blocked on candidate GENERATION, not the ledger: candidate readings must get less
crude (clause/referent structure, ADR-0178 GB-3b) before any cue earns reliability.
Cue-precision and compositional structure are coupled; structure comes first.

Verification: 107 targeted tests green (CP-2a/CP-1/extract/ms3/GB-1/2/3/MS-1/2) +
architectural invariants; serving CLAIMS.md sha unchanged; practice 4/1/45 and
0/1/149 unchanged. Inert: trains/reports only, consulted by no search/gate.
2026-05-29 10:21:58 -07:00
..
adversarial_identity
anchor_lens_tour
anti_regression feat(epistemic): Phase 3 state tagging spine (#220) 2026-05-24 11:26:06 -07:00
articulation
articulation_of_status
audit_tour
calibration
classical_literature_ood
cognition
cold_start_grounding fix(quarantine): drain all 60 quarantined tests — QUARANTINE=∅ (#267) 2026-05-25 11:22:12 -07:00
compositionality
compound_intent_decomposition
contemplation_quality feat(W-025): contemplation quality eval lane (ADR-0159) (#286) 2026-05-25 20:38:52 -07:00
contradiction_detection
conversation
conversational_thread_coherence
cross_domain_transfer
curriculum_loop_closure feat(evals): ADR-0105 — sealed holdout encryption via age (#108) 2026-05-22 10:09:43 -07:00
demo_composition fix(W-015): replace _slerp_toward with rotor-geodesic anchor pull (#255) 2026-05-24 20:05:25 -07:00
deterministic_fluency
discourse_paragraph
domain_contract_validation chore(ci): re-pin drifted lane SHAs + refresh canonical reports (#229) 2026-05-24 14:25:11 -07:00
elementary_mathematics_ood feat(capability): ADR-0110 promote mathematics_logic to expert_demo (#118) 2026-05-22 12:59:23 -07:00
english_fluency_ood
fabrication_control chore(ci): re-pin drifted lane SHAs + refresh canonical reports (#229) 2026-05-24 14:25:11 -07:00
flywheel_demo feat(demo): core demo flywheel — public-proof reproduction of the loop 2026-05-27 21:33:54 -07:00
forward_semantic_control
foundational_biology_ood
foundational_physics_ood feat: ADR-0111 physics expert-demo promotion (second successful) 2026-05-22 14:37:36 -07:00
frontier_compare feat(contemplation): land ADR-0080 phase 1 (#119) 2026-05-22 13:10:03 -07:00
grammatical_coverage
gsm8k_math feat(adr-0177-cp2a): cue-precision ledger training + measurement (+ unit hygiene) (#461) 2026-05-29 10:21:58 -07:00
gsm8k_parser_dev feat: add ADR-0125 perturbation suite 2026-05-22 17:12:33 -07:00
hebrew_fluency feat(packs): ADR-0103 — attach hebrew_fluency + koine_greek_fluency lanes to ADR-0102 (#106) 2026-05-22 09:43:46 -07:00
identity_divergence
industry_demos
inference_closure feat: ADR-0122 systems_software audit-passed deferred (lane-shape mismatch) 2026-05-22 16:31:59 -07:00
introspection
koine_greek_fluency feat(packs): ADR-0103 — attach hebrew_fluency + koine_greek_fluency lanes to ADR-0102 (#106) 2026-05-22 09:43:46 -07:00
lab
learning_arc fix(W-025): polish contemplation-quality eval lane follow-ups (#290) 2026-05-26 09:39:18 -07:00
learning_loop
long_context_cost
math_bounded_grammar/v1 feat(ADR-0131.3): bounded-grammar word-problem benchmark — lane PASSED 50/50 (#180) 2026-05-23 11:27:04 -07:00
math_capability_axes feat(ADR-0163.D.4): question grammar extension — mass nouns, comparatives, pronoun-entity resolution (#310) 2026-05-26 16:19:37 -07:00
math_expert_claims/v1 fix(quarantine): drain all 60 quarantined tests — QUARANTINE=∅ (#267) 2026-05-25 11:22:12 -07:00
math_symbolic_equivalence feat(ADR-0131.1.F): frontier-baseline comparison harness for B1 (#178) 2026-05-23 12:14:06 -07:00
math_teaching_corpus/v1 feat(ADR-0131.2.B): B2 teaching-corpus enrichment — load-bearing gate (#177) 2026-05-23 11:29:48 -07:00
miner_loop_closure
monotonic_learning
multi_agent_composition
multi_sentence_response
multi_step_reasoning
obligation_2_ood_ratio feat(ADR-0114a.2): OOD-ratio auditor — Obligation #2 wired for B3, ratio=1.00 (#193) 2026-05-23 16:25:28 -07:00
obligation_5_perturbation feat(ADR-0114a.5): reasoning-isolation perturbation suite — Obligation #5 wired for B3, PASSING 130/130 preserving, 68/68 breaking (#191) 2026-05-23 16:07:59 -07:00
obligation_6_depth_curve feat(ADR-0114a.6): depth-curve auditor — Obligation #6 wired for B3 (assertion holds, coverage gap named) (#190) 2026-05-23 16:19:58 -07:00
obligation_8_adversarial feat(ADR-0114a.8): adversarial auditor — Obligation #8 wired, PASSING; surfaces 2 known parser-layer gaps (#192) 2026-05-23 16:11:37 -07:00
obligation_10_pack_provenance feat(ADR-0114a.10): pack-provenance auditor — Obligation #10 wired for B3, PASSING 2026-05-23 15:44:53 -07:00
orthogonality_tour
prompt_diversity
provenance
public_demo chore(ratify): accept four Phase C round-2 recognizers (round 2) (#309) 2026-05-26 16:03:01 -07:00
realizer_guard
refusal_calibration
refusal_taxonomy feat(ADR-0163.B.2): four new exemplar corpora — discrete_count_statement, multiplicative_aggregation, currency_amount, plus temporal_aggregation v2 widening (#306) 2026-05-26 14:36:59 -07:00
register_diagnostics
register_tour
reports
results
reviewer_registry
sample_efficiency
self_consistency_over_time
symbolic_logic feat: ADR-0122 systems_software audit-passed deferred (lane-shape mismatch) 2026-05-22 16:31:59 -07:00
teaching_injection_resistance
walkthrough_chain
warmed_session_consistency
zero_code_domain_acquisition
__init__.py
_parallel.py
baseline_runner.py
CLAIMS.md
cognition_cases.jsonl
framework.py feat(evals): ADR-0105 — sealed holdout encryption via age (#108) 2026-05-22 10:09:43 -07:00
holdout_runner.py feat(evals): ADR-0105 — sealed holdout encryption via age (#108) 2026-05-22 10:09:43 -07:00
metrics.py
parallel.py
run_cognition_eval.py feat(W-008): L10 Shape B hybrid engine-state persistence (#271) 2026-05-25 11:45:54 -07:00