7451e7cd74
CP-2a populates the CP-1 ledger from gold-labelled candidate readings and reports per-pattern reliability — the measurement the cue-precision thesis rests on. Plus the function-word unit filter, whose value this measurement makes concrete (clean unit_shape labelling). What landed (all sealed; serving 3/47/0 byte-identical): - generate/cue_precision/trainer.py — train_from_cases(cases, enumerators): folds gold-labelled candidate chains into the ledger via record_case. Decoupled (the candidate enumerators are injected, so the package still imports nothing from search). candidates_for dedupes a reading shared by two enumerators. - generate/derivation/multistep.py — extracted the enumeration half of search_chain into public candidate_chains(problem_text); search_chain now delegates (verified byte-identical: ms3 tests + practice counts unchanged). CP-2 needs the readings the search weighs, not just the one it resolves. - generate/derivation/extract.py — function-word unit filter (_NON_UNIT_WORDS): blanks spurious function-word units ($0.75 each -> "", 3/4 of -> "") that corrupt same-unit detection and unit_shape. Closed lexeme set, ADR-0165-safe. - evals/gsm8k_math/practice/v1/cue_precision_report.py — trains over 200 sealed cases (50 train_sample + 150 ADR-0163-F additive) with the real enumerators and prints the per-pattern reliability table. - tests/test_adr_0177_cp2a_training.py — trainer obligations (credit/dedupe/ determinism/empty) via synthetic enumerators; real-measurement well-formedness; search_chain parity. Load-bearing finding (recorded in ADR-0177): over 200 cases EVERY (cue,op,unit_shape) pattern floors at ~0.0 reliability (best: for-multiply-cross_unit 0.0116 at 2/34). The blunt product/sum-of-all readings are almost always wrong vs gold, so the conservative floor correctly trusts nothing. => CP-2b (trust reliable cues) is blocked on candidate GENERATION, not the ledger: candidate readings must get less crude (clause/referent structure, ADR-0178 GB-3b) before any cue earns reliability. Cue-precision and compositional structure are coupled; structure comes first. Verification: 107 targeted tests green (CP-2a/CP-1/extract/ms3/GB-1/2/3/MS-1/2) + architectural invariants; serving CLAIMS.md sha unchanged; practice 4/1/45 and 0/1/149 unchanged. Inert: trains/reports only, consulted by no search/gate.
68 lines
3 KiB
Python
68 lines
3 KiB
Python
"""ADR-0177 CP-2a — populate the cue-precision ledger from gold-labelled cases.
|
|
|
|
The CP-1 ledger (:mod:`generate.cue_precision.ledger`) is the mechanism; this is
|
|
the **training step** that gives it signal. For each ``(problem_text, gold)`` case
|
|
it gathers the candidate readings the search would consider, labels each by gold,
|
|
and folds the per-step ``(cue, op, unit_shape)`` credit into the ledger
|
|
(``record_case``). The result is the per-pattern reliability table — the
|
|
*measurement* CP-2b/CP-3 will consult before trusting a cue.
|
|
|
|
Decoupled by construction: the candidate *enumerators* are injected (callables
|
|
``problem_text -> Iterable[GroundedDerivation]``), so this module imports nothing
|
|
from :mod:`generate.derivation.search` / ``multistep`` and stays as inert and
|
|
replay-stable as CP-1. The eval side (:mod:`evals.gsm8k_math...`) wires the real
|
|
enumerators to the real cases.
|
|
|
|
wrong=0 posture: training reads gold (Tier-1, available in the sealed practice
|
|
regime only) and writes counts. It changes no search/gate behaviour — the ledger
|
|
is still consulted by nobody until CP-2b. Serving stays ``3/47/0``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from collections.abc import Callable, Iterable
|
|
|
|
from generate.cue_precision.ledger import CuePrecisionLedger
|
|
from generate.derivation.model import GroundedDerivation
|
|
|
|
# A candidate enumerator turns a problem into the readings the search considers.
|
|
CandidateEnumerator = Callable[[str], Iterable[GroundedDerivation]]
|
|
|
|
# A training case: the problem text and its gold numeric answer.
|
|
TrainingCase = tuple[str, float]
|
|
|
|
|
|
def candidates_for(
|
|
problem_text: str, enumerators: Iterable[CandidateEnumerator]
|
|
) -> tuple[GroundedDerivation, ...]:
|
|
"""The deduplicated union of every enumerator's candidates, in stable order.
|
|
|
|
A reading produced by two enumerators is counted **once** per case (per-step
|
|
credit already counts each pattern occurrence within a chain; double-counting
|
|
the whole chain across enumerators would inflate the same evidence twice).
|
|
Dedup preserves first-seen order, so the fold is deterministic.
|
|
"""
|
|
seen: dict[GroundedDerivation, None] = {}
|
|
for enumerate_candidates in enumerators:
|
|
for candidate in enumerate_candidates(problem_text):
|
|
seen.setdefault(candidate, None)
|
|
return tuple(seen)
|
|
|
|
|
|
def train_from_cases(
|
|
cases: Iterable[TrainingCase],
|
|
enumerators: Iterable[CandidateEnumerator],
|
|
) -> CuePrecisionLedger:
|
|
"""Fold every case's gold-labelled candidates into a fresh ledger.
|
|
|
|
Deterministic in ``cases`` order, ``enumerators`` order, and each enumerator's
|
|
own candidate order. A case with no candidate contributes nothing (no refusal
|
|
penalty — the ledger only labels readings, ADR-0177).
|
|
"""
|
|
enumerator_tuple = tuple(enumerators)
|
|
ledger = CuePrecisionLedger()
|
|
for problem_text, gold in cases:
|
|
candidates = candidates_for(problem_text, enumerator_tuple)
|
|
if candidates:
|
|
ledger = ledger.record_case(candidates, float(gold))
|
|
return ledger
|