872ed3b52d
ADR-0175 Phase 3b — the first live attempt generator. Runs only in the sealed
practice lane, only on cases the engine refused; every proposal is gated by the
Phase 3a self-verification gate.
generate/derivation/:
- extract.py: extract_quantities() — lexeme-level (number + unit word; ADR-0165).
- search.py: search_multiplicative() — one in-clause product candidate per
sentence with >=2 quantities + a present multiplicative cue; gated by
select_self_verified. Per-sentence scope + multi-candidate disagreement give
the uniqueness gate real teeth (two qualifying sentences -> refuse). The cue
set {each,every,for,per,times} is an explicit PROVISIONAL hypothesis the
practice loop refines, not a claimed-correct grammar.
evals/gsm8k_math/practice/v1/search_runner.py: search_augmented_scorer +
build_search_report — base scorer, then a practice-only attempt on refusals.
MEASUREMENT (the deliverable, per the breadth-of-impact test):
practice with search: correct=4 wrong=9 refused=37 (baseline 3/0/47)
- Flips +1 (0021, the clean in-clause aggregate) and its renumbered/reworded
variants (ADR-0114a perturbation guard) -> a real capability, not memorisation.
- 9 wrong attempts -> elimination records (§9), the learning signal. The naive
full-product cue model over-attempts; the eliminations are exactly the signal
that refines it.
HONEST FINDING: self-verification (grounding ∧ cue ∧ unit ∧ uniqueness) is
NECESSARY but NOT SUFFICIENT — 9/13 self-verified attempts were wrong vs gold.
The gap is cue PRECISION / which-quantities-compose (the knowledge axis), not
'can we multiply' (skill). This is why the search runs sealed: gold catches the
9, and case 0050 (canary) attempted-and-failed IN PRACTICE without touching
serving -> validates the seal.
Invariants: #1 seal (serving still 3/47/0; 0050 refuses in serving; no
generate/chat import of the lane), #3 determinism. Serving wrong=0 untouched.
Verified: 3a+3b 31/31; ruff clean; serving lane 4/4; smoke 67/67.
32 lines
944 B
Python
32 lines
944 B
Python
"""ADR-0175 Phase 3 — grounded derivation search + self-verification gate.
|
|
|
|
Phase 3a (this surface): the self-verification gate — grounded operands ∧
|
|
grounded operation cues ∧ unit consistency ∧ uniqueness. The wrong=0-critical
|
|
guard that keeps the (Phase 3b) bounded search honest.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from generate.derivation.extract import extract_quantities
|
|
from generate.derivation.model import GroundedDerivation, Quantity, Step, VALID_OPS
|
|
from generate.derivation.search import MULTIPLICATIVE_CUES, search_multiplicative
|
|
from generate.derivation.verify import (
|
|
Resolution,
|
|
SelfVerification,
|
|
select_self_verified,
|
|
self_verifies,
|
|
)
|
|
|
|
__all__ = [
|
|
"GroundedDerivation",
|
|
"MULTIPLICATIVE_CUES",
|
|
"Quantity",
|
|
"Resolution",
|
|
"SelfVerification",
|
|
"Step",
|
|
"VALID_OPS",
|
|
"extract_quantities",
|
|
"search_multiplicative",
|
|
"select_self_verified",
|
|
"self_verifies",
|
|
]
|