872ed3b52d
ADR-0175 Phase 3b — the first live attempt generator. Runs only in the sealed
practice lane, only on cases the engine refused; every proposal is gated by the
Phase 3a self-verification gate.
generate/derivation/:
- extract.py: extract_quantities() — lexeme-level (number + unit word; ADR-0165).
- search.py: search_multiplicative() — one in-clause product candidate per
sentence with >=2 quantities + a present multiplicative cue; gated by
select_self_verified. Per-sentence scope + multi-candidate disagreement give
the uniqueness gate real teeth (two qualifying sentences -> refuse). The cue
set {each,every,for,per,times} is an explicit PROVISIONAL hypothesis the
practice loop refines, not a claimed-correct grammar.
evals/gsm8k_math/practice/v1/search_runner.py: search_augmented_scorer +
build_search_report — base scorer, then a practice-only attempt on refusals.
MEASUREMENT (the deliverable, per the breadth-of-impact test):
practice with search: correct=4 wrong=9 refused=37 (baseline 3/0/47)
- Flips +1 (0021, the clean in-clause aggregate) and its renumbered/reworded
variants (ADR-0114a perturbation guard) -> a real capability, not memorisation.
- 9 wrong attempts -> elimination records (§9), the learning signal. The naive
full-product cue model over-attempts; the eliminations are exactly the signal
that refines it.
HONEST FINDING: self-verification (grounding ∧ cue ∧ unit ∧ uniqueness) is
NECESSARY but NOT SUFFICIENT — 9/13 self-verified attempts were wrong vs gold.
The gap is cue PRECISION / which-quantities-compose (the knowledge axis), not
'can we multiply' (skill). This is why the search runs sealed: gold catches the
9, and case 0050 (canary) attempted-and-failed IN PRACTICE without touching
serving -> validates the seal.
Invariants: #1 seal (serving still 3/47/0; 0050 refuses in serving; no
generate/chat import of the lane), #3 determinism. Serving wrong=0 untouched.
Verified: 3a+3b 31/31; ruff clean; serving lane 4/4; smoke 67/67.
39 lines
1.5 KiB
Python
39 lines
1.5 KiB
Python
"""ADR-0175 Phase 3b — lexeme-level quantity extraction.
|
|
|
|
Pulls ``(value, unit, source_token)`` triples from a problem using a single
|
|
orthographic pattern: a number immediately followed by a unit word. Per
|
|
ADR-0165 this is a *lexeme* pattern ("what this piece looks like: a number, a
|
|
unit word") — not a grammar template ("how words combine to mean X"). The
|
|
*combining* is the search's job (search.py) gated by self-verification.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import re
|
|
from typing import Final
|
|
|
|
from generate.derivation.model import Quantity
|
|
|
|
# Number (int or decimal) immediately followed by a unit word. Lexeme-level.
|
|
_QTY_RE: Final[re.Pattern[str]] = re.compile(
|
|
r"(?<![\w.])(\d+(?:\.\d+)?)\s+([a-zA-Z]+)"
|
|
)
|
|
|
|
|
|
def extract_quantities(problem_text: str) -> tuple[Quantity, ...]:
|
|
"""Extract ``(value, unit, source_token)`` quantities in left-to-right order.
|
|
|
|
Deterministic. ``source_token`` is the surface number string (used by the
|
|
self-verification gate to prove the value is grounded in the text). Units
|
|
are lowercased; the value's surface token is preserved verbatim.
|
|
"""
|
|
out: list[Quantity] = []
|
|
for match in _QTY_RE.finditer(problem_text):
|
|
value_token = match.group(1)
|
|
unit = match.group(2).lower()
|
|
try:
|
|
value = float(value_token)
|
|
except ValueError: # pragma: no cover - regex guarantees numeric
|
|
continue
|
|
out.append(Quantity(value=value, unit=unit, source_token=value_token))
|
|
return tuple(out)
|