fix(kernel): ground scalar spans and tighten morphology labels
Add extract_scalar_candidates() with exact source spans and problem_text provenance while keeping canonicalize_scalar() as the detached pack helper. Morphology labels now emit missing_* only when substrate frame/unit lookups actually fail, not on mere trigger-surface presence.
This commit is contained in:
Shay
parent
12a53648f6
commit
1e4fcff102
5 changed files with 442 additions and 54 deletions
|
|
@ -28,7 +28,28 @@ Seven new Python modules have been introduced to establish the substrate layer:
|
|||
|
||||
## 4. Scalar Equivalence Coverage
|
||||
|
||||
Exposed via `language_packs/scalar_equivalence.py`:
|
||||
### Canonicalize vs grounded extraction
|
||||
|
||||
`language_packs/scalar_equivalence.py` exposes two complementary APIs:
|
||||
|
||||
- **`canonicalize_scalar(surface)`** — pack-level helper for detached surface strings.
|
||||
Returns a `ScalarCandidate` with canonical `Fraction`, source kind, entry id, and
|
||||
hazards. Provenance and span fields remain `None` (no problem-text grounding).
|
||||
- **`extract_scalar_candidates(text)`** — text-level extraction for ProblemFrame
|
||||
substrate facts. Every emitted candidate carries:
|
||||
- `source_surface` — exact substring from the original text
|
||||
- `source_span` — `(start, end)` character offsets (Python slice semantics)
|
||||
- `provenance_kind = "problem_text"`
|
||||
- `canonical` — exact `Fraction`
|
||||
- `source` — scalar resolution kind (`fraction_word`, `decimal`, etc.)
|
||||
- `hazards` — ambiguity hazard IDs
|
||||
|
||||
Pack/world/derived values (`classify_dimension`, detached `canonicalize_scalar`)
|
||||
do not masquerade as `problem_text`. Multiple scalars are emitted in deterministic
|
||||
left-to-right span order. Unsupported forms (`.5`, `1 / 2`) are omitted from
|
||||
extraction and flagged separately by the morphology atlas.
|
||||
|
||||
Exposed scalar surfaces via the facade:
|
||||
- **Word forms:** `half`, `one half`, `one-half`, `third`, `one third`, `two thirds`, `quarter`, `one quarter`, `three quarters`.
|
||||
- **Symbols:** Unicode symbols (`½`, `¼`, `¾`, `⅓`, `⅔`).
|
||||
- **Mixed numbers / slash forms:** `1/2`, `3/4`, `1 1/2`.
|
||||
|
|
@ -97,6 +118,28 @@ Exposed via `scripts/gsm8k_substrate_morphology.py`:
|
|||
`missing_scalar_equivalence`, `missing_unit_dimension`, `missing_process_frame`, `missing_part_whole_frame`, `missing_container_frame`, `missing_temporal_frame`, `missing_route_frame`, `missing_question_target`, `blocked_ambiguity_hazard`, `blocked_provenance_gap`.
|
||||
- Exposes a deterministic function `classify_missing_substrate` and a CLI interface to batch-process cases.
|
||||
|
||||
### Corrected label semantics (post-patch)
|
||||
|
||||
`missing_*` labels now mean **substrate lookup failure**, not mere trigger-surface
|
||||
presence. For frame-backed categories, the classifier checks
|
||||
`generate/process_frames.lookup_frame` before emitting a label:
|
||||
|
||||
- Text containing **give** does **not** receive `missing_process_frame` when the
|
||||
`transfer` frame is registered.
|
||||
- Text containing **box** does **not** receive `missing_container_frame` when
|
||||
`container_packing` is registered.
|
||||
- Similarly for **split** / `partition`, **drive** / `travel`, and other
|
||||
registered frame triggers.
|
||||
|
||||
Labels that remain trigger-based (not frame lookup):
|
||||
|
||||
- `missing_scalar_equivalence` — unsupported numeric surfaces (`.5`, `1 / 2`, etc.)
|
||||
- `missing_unit_dimension` — unknown unit-like nouns after digits
|
||||
- `missing_temporal_frame` — time surfaces with no registered process frame
|
||||
- `missing_question_target`, `blocked_ambiguity_hazard`, `blocked_provenance_gap`
|
||||
|
||||
All labels are deterministic and sorted.
|
||||
|
||||
---
|
||||
|
||||
## 11. Serving Integration Status
|
||||
|
|
@ -116,7 +159,28 @@ Documented and explicitly refused surfaces in the scalar facade:
|
|||
|
||||
## 13. Validation
|
||||
|
||||
Verification has been completed across multiple lanes:
|
||||
### Post-patch verification (2026-06-18)
|
||||
|
||||
After grounding scalar spans and tightening morphology label semantics:
|
||||
|
||||
1. **`git diff --check origin/main...HEAD`** — no whitespace errors.
|
||||
2. **Kernel substrate unit tests** — all pass:
|
||||
- `tests/test_kernel_facts.py`
|
||||
- `tests/test_language_packs_scalar_equivalence.py` (includes span/provenance extraction)
|
||||
- `tests/test_language_packs_unit_dimensions.py`
|
||||
- `tests/test_ambiguity_hazards.py`
|
||||
- `tests/test_process_frames.py`
|
||||
- `tests/test_problem_frame_skeleton.py`
|
||||
- `tests/test_gsm8k_morphology_missing_kernel_labels.py`
|
||||
3. **Capability safety** — ADR-0128 numeric format and math candidate graph sprint tests pass.
|
||||
4. **Evaluation scores (unchanged):**
|
||||
- `train_sample`: 30 correct / 20 refused / 0 wrong — `wrong_ids: []`
|
||||
- `holdout_dev`: 5 correct / 495 refused / 0 wrong — `wrong_ids: []`
|
||||
5. **Smoke suite:** `core test --suite smoke -q` green.
|
||||
|
||||
### Initial tranche verification
|
||||
|
||||
Verification from the initial implementation:
|
||||
1. **Unit tests:** 7 new test files containing 33 unit test cases verify all primitives, facade mappings, hazard lookups, and schemas. All 33 pass.
|
||||
2. **Capability safety:** 227 existing tests pass, confirming zero regression.
|
||||
3. **Evaluation scores:**
|
||||
|
|
|
|||
|
|
@ -6,12 +6,17 @@ Thin facade over ``numerics_loader.py`` exposing canonical ``Fraction``
|
|||
values for scalar surfaces. Respects ADR-0128 boundaries: if a surface
|
||||
is refused by the underlying pack, this facade does not silently broaden it.
|
||||
|
||||
``canonicalize_scalar`` is the pack-level helper for detached surfaces.
|
||||
``extract_scalar_candidates`` is the text-level API that grounds spans in
|
||||
problem text with ``problem_text`` provenance.
|
||||
|
||||
The facade MAY emit ``ScalarCandidate`` records.
|
||||
It MAY NOT solve problems, bind base quantities, choose operations,
|
||||
or infer final answers.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from fractions import Fraction
|
||||
|
||||
|
|
@ -23,6 +28,7 @@ from language_packs.numerics_loader import (
|
|||
lookup_fraction,
|
||||
lookup_multiplier,
|
||||
match_number_format,
|
||||
number_format_entries,
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -34,6 +40,8 @@ _UNSUPPORTED_SURFACES: tuple[str, ...] = (
|
|||
"1 / 2", # spaces around slash — not a single token
|
||||
)
|
||||
|
||||
PROVENANCE_PROBLEM_TEXT: str = "problem_text"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Source category constants — closed set per Tranche 1 spec.
|
||||
|
|
@ -62,6 +70,32 @@ _FORMAT_ID_TO_SOURCE: dict[str, str] = {
|
|||
"signed_integer": SOURCE_DECIMAL,
|
||||
}
|
||||
|
||||
# Format search priority — longer / more specific patterns first.
|
||||
_FORMAT_SEARCH_ORDER: tuple[str, ...] = (
|
||||
"mixed_number",
|
||||
"slash_fraction",
|
||||
"percentage",
|
||||
"decimal",
|
||||
"thousand_separated",
|
||||
"signed_integer",
|
||||
)
|
||||
|
||||
_UNICODE_FRACTION_SYMBOLS: frozenset[str] = frozenset({"½", "¼", "¾", "⅓", "⅔"})
|
||||
|
||||
_WORD_SCALAR_PATTERNS: tuple[re.Pattern[str], ...] = (
|
||||
re.compile(r"three\s+quarters?", re.IGNORECASE),
|
||||
re.compile(r"two\s+thirds?", re.IGNORECASE),
|
||||
re.compile(r"one\s+quarter", re.IGNORECASE),
|
||||
re.compile(r"one\s+third", re.IGNORECASE),
|
||||
re.compile(r"one\s+half", re.IGNORECASE),
|
||||
re.compile(r"one-half", re.IGNORECASE),
|
||||
re.compile(r"one-third", re.IGNORECASE),
|
||||
re.compile(r"three-quarters?", re.IGNORECASE),
|
||||
re.compile(r"\bhalf\b", re.IGNORECASE),
|
||||
re.compile(r"\bquarter\b", re.IGNORECASE),
|
||||
re.compile(r"\bthird\b", re.IGNORECASE),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ScalarCandidate — frozen, immutable result record.
|
||||
|
|
@ -75,12 +109,18 @@ class ScalarCandidate:
|
|||
``entry_id`` is the ``en_numerics_v1`` entry id when the value came from
|
||||
a pack-backed lookup, ``None`` for purely format-parsed values.
|
||||
``hazards`` carries hazard IDs from the ambiguity registry.
|
||||
|
||||
Grounding fields are populated only by :func:`extract_scalar_candidates`.
|
||||
Detached :func:`canonicalize_scalar` results leave them ``None``.
|
||||
"""
|
||||
surface: str
|
||||
canonical: Fraction
|
||||
source: str
|
||||
entry_id: str | None
|
||||
hazards: tuple[str, ...]
|
||||
source_surface: str | None = None
|
||||
source_span: tuple[int, int] | None = None
|
||||
provenance_kind: str | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -97,10 +137,14 @@ def _collect_hazard_ids(surface: str) -> tuple[str, ...]:
|
|||
def _fraction_entry_to_candidate(
|
||||
surface: str,
|
||||
entry: FractionEntry,
|
||||
*,
|
||||
source_surface: str | None = None,
|
||||
source_span: tuple[int, int] | None = None,
|
||||
provenance_kind: str | None = None,
|
||||
) -> ScalarCandidate:
|
||||
"""Convert a ``FractionEntry`` to a ``ScalarCandidate``."""
|
||||
canonical = Fraction(entry.numerator, entry.denominator)
|
||||
if entry.morphology == "fraction-symbol" or surface in {"½", "¼", "¾", "⅓", "⅔"}:
|
||||
if entry.morphology == "fraction-symbol" or surface in _UNICODE_FRACTION_SYMBOLS:
|
||||
source = SOURCE_FRACTION_SYMBOL
|
||||
else:
|
||||
source = _MORPHOLOGY_TO_SOURCE.get(entry.morphology, SOURCE_FRACTION_WORD)
|
||||
|
|
@ -110,12 +154,19 @@ def _fraction_entry_to_candidate(
|
|||
source=source,
|
||||
entry_id=entry.entry_id,
|
||||
hazards=_collect_hazard_ids(surface),
|
||||
source_surface=source_surface,
|
||||
source_span=source_span,
|
||||
provenance_kind=provenance_kind,
|
||||
)
|
||||
|
||||
|
||||
def _parsed_number_to_candidate(
|
||||
surface: str,
|
||||
parsed: ParsedNumber,
|
||||
*,
|
||||
source_surface: str | None = None,
|
||||
source_span: tuple[int, int] | None = None,
|
||||
provenance_kind: str | None = None,
|
||||
) -> ScalarCandidate | None:
|
||||
"""Convert a ``ParsedNumber`` to a ``ScalarCandidate``.
|
||||
|
||||
|
|
@ -168,12 +219,19 @@ def _parsed_number_to_candidate(
|
|||
source=source,
|
||||
entry_id=None,
|
||||
hazards=_collect_hazard_ids(surface),
|
||||
source_surface=source_surface,
|
||||
source_span=source_span,
|
||||
provenance_kind=provenance_kind,
|
||||
)
|
||||
|
||||
|
||||
def _multiplier_entry_to_candidate(
|
||||
surface: str,
|
||||
entry: MultiplierEntry,
|
||||
*,
|
||||
source_surface: str | None = None,
|
||||
source_span: tuple[int, int] | None = None,
|
||||
provenance_kind: str | None = None,
|
||||
) -> ScalarCandidate:
|
||||
"""Convert a ``MultiplierEntry`` to a ``ScalarCandidate``.
|
||||
|
||||
|
|
@ -187,6 +245,120 @@ def _multiplier_entry_to_candidate(
|
|||
source=SOURCE_MULTIPLIER,
|
||||
entry_id=entry.entry_id,
|
||||
hazards=_collect_hazard_ids(surface),
|
||||
source_surface=source_surface,
|
||||
source_span=source_span,
|
||||
provenance_kind=provenance_kind,
|
||||
)
|
||||
|
||||
|
||||
def _strip_regex_anchors(regex: str) -> str:
|
||||
body = regex
|
||||
if body.startswith("^"):
|
||||
body = body[1:]
|
||||
if body.endswith("$"):
|
||||
body = body[:-1]
|
||||
return body
|
||||
|
||||
|
||||
def _format_search_patterns() -> tuple[re.Pattern[str], ...]:
|
||||
by_id = {fmt.format_id: fmt for fmt in number_format_entries()}
|
||||
patterns: list[re.Pattern[str]] = []
|
||||
for format_id in _FORMAT_SEARCH_ORDER:
|
||||
fmt = by_id.get(format_id)
|
||||
if fmt is None:
|
||||
continue
|
||||
body = _strip_regex_anchors(fmt.regex)
|
||||
# Allow trailing sentence punctuation; refuse embedded spaced slashes.
|
||||
patterns.append(
|
||||
re.compile(
|
||||
r"(?<!\S)" + body + r"(?=\s|$|[.,;:!?)\]}\"'])"
|
||||
)
|
||||
)
|
||||
return tuple(patterns)
|
||||
|
||||
|
||||
def _is_rejected_span(text: str, start: int, end: int, surface: str) -> bool:
|
||||
"""Reject ambiguous spans such as ``1`` in ``1 / 2`` or ``.5`` tokenisations."""
|
||||
if surface in _UNSUPPORTED_SURFACES:
|
||||
return True
|
||||
if surface.startswith("."):
|
||||
return True
|
||||
if start > 0 and text[start - 1] == ".":
|
||||
return True
|
||||
if re.match(r"\s+/\s+\d", text[end:]):
|
||||
return True
|
||||
if re.search(r"/\s+$", text[:start]):
|
||||
return True
|
||||
if re.match(r"\.\d+\b", text[start:end] if start == 0 else text[max(0, start - 1):end]):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _select_non_overlapping(
|
||||
spans: list[tuple[int, int, str]],
|
||||
) -> list[tuple[int, int, str]]:
|
||||
"""Greedy longest-leftmost selection, then return in span order."""
|
||||
ranked = sorted(spans, key=lambda item: (item[0], -(item[1] - item[0])))
|
||||
selected: list[tuple[int, int, str]] = []
|
||||
occupied: list[tuple[int, int]] = []
|
||||
|
||||
for start, end, surface in ranked:
|
||||
overlaps = any(not (end <= occ_start or start >= occ_end) for occ_start, occ_end in occupied)
|
||||
if overlaps:
|
||||
continue
|
||||
selected.append((start, end, surface))
|
||||
occupied.append((start, end))
|
||||
|
||||
return sorted(selected, key=lambda item: item[0])
|
||||
|
||||
|
||||
def _discover_scalar_spans(text: str) -> list[tuple[int, int, str]]:
|
||||
spans: list[tuple[int, int, str]] = []
|
||||
|
||||
for pattern in _format_search_patterns():
|
||||
for match in pattern.finditer(text):
|
||||
start, end = match.start(), match.end()
|
||||
surface = match.group(0)
|
||||
if _is_rejected_span(text, start, end, surface):
|
||||
continue
|
||||
spans.append((start, end, surface))
|
||||
|
||||
for symbol in _UNICODE_FRACTION_SYMBOLS:
|
||||
start = 0
|
||||
while True:
|
||||
idx = text.find(symbol, start)
|
||||
if idx < 0:
|
||||
break
|
||||
spans.append((idx, idx + len(symbol), symbol))
|
||||
start = idx + len(symbol)
|
||||
|
||||
for pattern in _WORD_SCALAR_PATTERNS:
|
||||
for match in pattern.finditer(text):
|
||||
spans.append((match.start(), match.end(), match.group(0)))
|
||||
|
||||
return _select_non_overlapping(spans)
|
||||
|
||||
|
||||
def _grounded_candidate(
|
||||
text: str,
|
||||
start: int,
|
||||
end: int,
|
||||
source_surface: str,
|
||||
detached: ScalarCandidate,
|
||||
) -> ScalarCandidate:
|
||||
if text[start:end] != source_surface:
|
||||
raise ValueError(
|
||||
f"source span [{start}:{end}] does not slice source_surface {source_surface!r}"
|
||||
)
|
||||
return ScalarCandidate(
|
||||
surface=detached.surface,
|
||||
canonical=detached.canonical,
|
||||
source=detached.source,
|
||||
entry_id=detached.entry_id,
|
||||
hazards=detached.hazards,
|
||||
source_surface=source_surface,
|
||||
source_span=(start, end),
|
||||
provenance_kind=PROVENANCE_PROBLEM_TEXT,
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -203,7 +375,7 @@ def canonicalize_scalar(surface: str) -> ScalarCandidate | None:
|
|||
3. Try ``lookup_multiplier`` (handles ``half`` as multiplier).
|
||||
|
||||
Returns ``None`` if the surface is unsupported or refused by the
|
||||
underlying pack.
|
||||
underlying pack. Detached results do not carry span/provenance fields.
|
||||
"""
|
||||
if not surface:
|
||||
return None
|
||||
|
|
@ -232,6 +404,34 @@ def canonicalize_scalar(surface: str) -> ScalarCandidate | None:
|
|||
return None
|
||||
|
||||
|
||||
def extract_scalar_candidates(text: str) -> tuple[ScalarCandidate, ...]:
|
||||
"""Extract grounded scalar candidates from problem text.
|
||||
|
||||
Every emitted candidate carries:
|
||||
|
||||
* ``source_surface`` — exact substring from *text*
|
||||
* ``source_span`` — ``(start, end)`` character offsets
|
||||
* ``provenance_kind`` — ``"problem_text"``
|
||||
* ``canonical`` — exact ``Fraction``
|
||||
* ``source`` — scalar resolution kind
|
||||
* ``hazards`` — ambiguity hazard IDs
|
||||
|
||||
Candidates are returned in deterministic span order (left-to-right).
|
||||
Unsupported surfaces (``.5``, ``1 / 2``, etc.) are omitted.
|
||||
"""
|
||||
if not text:
|
||||
return ()
|
||||
|
||||
candidates: list[ScalarCandidate] = []
|
||||
for start, end, source_surface in _discover_scalar_spans(text):
|
||||
detached = canonicalize_scalar(source_surface)
|
||||
if detached is None:
|
||||
continue
|
||||
candidates.append(_grounded_candidate(text, start, end, source_surface, detached))
|
||||
|
||||
return tuple(candidates)
|
||||
|
||||
|
||||
def is_supported_scalar(surface: str) -> bool:
|
||||
"""Return ``True`` if the surface can be canonicalized."""
|
||||
return canonicalize_scalar(surface) is not None
|
||||
|
|
@ -243,4 +443,4 @@ def list_unsupported_surfaces() -> tuple[str, ...]:
|
|||
These are surfaces that look numeric but are explicitly refused
|
||||
by the pack or this facade due to tokenisation or ambiguity issues.
|
||||
"""
|
||||
return _UNSUPPORTED_SURFACES
|
||||
return _UNSUPPORTED_SURFACES
|
||||
|
|
@ -2,6 +2,10 @@
|
|||
"""Classify GSM8K problems by missing substrate category.
|
||||
|
||||
Tranche 1 — broad base-layer foundations.
|
||||
|
||||
Labels are semantically honest: ``missing_*`` categories fire only when a
|
||||
needed substrate lookup actually fails, not merely because a trigger
|
||||
surface appears in the text.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
|
@ -14,7 +18,60 @@ from typing import Sequence
|
|||
from language_packs.scalar_equivalence import list_unsupported_surfaces
|
||||
from language_packs.unit_dimensions import classify_dimension
|
||||
from language_packs.loader import lookup_container
|
||||
from generate.process_frames import all_frames
|
||||
from generate.process_frames import all_frames, lookup_frame
|
||||
|
||||
|
||||
_PROCESS_FRAME_NAMES: frozenset[str] = frozenset({"transfer", "consumption", "transaction"})
|
||||
_CONTAINER_FRAME_NAMES: frozenset[str] = frozenset({"container_packing"})
|
||||
_PARTITION_FRAME_NAMES: frozenset[str] = frozenset({"partition"})
|
||||
_TRAVEL_FRAME_NAMES: frozenset[str] = frozenset({"travel"})
|
||||
|
||||
_TEMPORAL_SURFACE_TRIGGERS: tuple[str, ...] = (
|
||||
"hour", "hours", "minute", "minutes", "second", "seconds",
|
||||
"day", "days", "week", "weeks", "month", "months", "year", "years",
|
||||
)
|
||||
|
||||
_AMBIGUITY_HAZARD_SURFACES: tuple[str, ...] = (
|
||||
"half", "quarter", "third", "percent", "percentage points", "times",
|
||||
"more than", "less than", "of", "per", "each", "some", "remaining",
|
||||
"left", "total", "altogether",
|
||||
)
|
||||
|
||||
|
||||
def _surface_in_text(surface: str, text_lower: str) -> bool:
|
||||
"""Return True when *surface* appears as a token/phrase in *text_lower*."""
|
||||
token = surface.lower()
|
||||
padded = f" {text_lower} "
|
||||
return (
|
||||
f" {token} " in padded
|
||||
or text_lower.startswith(f"{token} ")
|
||||
or text_lower.endswith(f" {token}")
|
||||
or text_lower == token
|
||||
)
|
||||
|
||||
|
||||
def _frame_triggers(frame_names: frozenset[str]) -> tuple[str, ...]:
|
||||
triggers: list[str] = []
|
||||
for frame in all_frames():
|
||||
if frame.name in frame_names:
|
||||
triggers.extend(frame.trigger_surfaces)
|
||||
return tuple(triggers)
|
||||
|
||||
|
||||
def _missing_frame_for_triggers(
|
||||
text_lower: str,
|
||||
triggers: Sequence[str],
|
||||
frame_names: frozenset[str],
|
||||
) -> bool:
|
||||
"""True when text contains category triggers but none resolve to a frame."""
|
||||
saw_trigger = False
|
||||
for trigger in triggers:
|
||||
if not _surface_in_text(trigger, text_lower):
|
||||
continue
|
||||
saw_trigger = True
|
||||
if any(frame.name in frame_names for frame in lookup_frame(trigger)):
|
||||
return False
|
||||
return saw_trigger
|
||||
|
||||
|
||||
def classify_missing_substrate(problem_text: str) -> tuple[str, ...]:
|
||||
|
|
@ -22,21 +79,18 @@ def classify_missing_substrate(problem_text: str) -> tuple[str, ...]:
|
|||
|
||||
Inspects problem text using substrate facades to identify gaps.
|
||||
"""
|
||||
labels = set()
|
||||
labels: set[str] = set()
|
||||
text_lower = problem_text.lower()
|
||||
|
||||
# 1. missing_scalar_equivalence
|
||||
# If the text has unsupported surfaces like ".5" or "1 / 2"
|
||||
for unsup in list_unsupported_surfaces():
|
||||
if unsup in text_lower:
|
||||
if unsup in problem_text or unsup in text_lower:
|
||||
labels.add("missing_scalar_equivalence")
|
||||
|
||||
# Look for digit-slash-digit with spaces
|
||||
if re.search(r"\b\d+\s+/\s+\d+\b", problem_text) or re.search(r"\b\.\d+\b", problem_text):
|
||||
labels.add("missing_scalar_equivalence")
|
||||
|
||||
# 2. missing_unit_dimension
|
||||
# Extract words following digits (e.g. "5 widgets")
|
||||
matches = re.findall(r"\b\d+(?:\.\d+)?\s+([a-zA-Z]+)\b", problem_text)
|
||||
for word in matches:
|
||||
word_lower = word.lower()
|
||||
|
|
@ -48,31 +102,42 @@ def classify_missing_substrate(problem_text: str) -> tuple[str, ...]:
|
|||
if classify_dimension(word_lower) is None and lookup_container(word_lower) is None:
|
||||
labels.add("missing_unit_dimension")
|
||||
|
||||
# 3. missing_process_frame
|
||||
has_triggers = False
|
||||
for frame in all_frames():
|
||||
for trigger in frame.trigger_surfaces:
|
||||
if f" {trigger} " in f" {text_lower} " or text_lower.startswith(trigger) or text_lower.endswith(trigger):
|
||||
has_triggers = True
|
||||
break
|
||||
if has_triggers:
|
||||
if "give" in text_lower or "gave" in text_lower or "gives" in text_lower:
|
||||
labels.add("missing_process_frame")
|
||||
# 3. missing_process_frame — only when process triggers fail lookup
|
||||
if _missing_frame_for_triggers(
|
||||
text_lower,
|
||||
_frame_triggers(_PROCESS_FRAME_NAMES),
|
||||
_PROCESS_FRAME_NAMES,
|
||||
):
|
||||
labels.add("missing_process_frame")
|
||||
|
||||
# 4. missing_part_whole_frame
|
||||
if any(w in text_lower for w in ["split", "divide", "share", "partition", "rest of", "portion"]):
|
||||
# 4. missing_part_whole_frame — partition triggers must fail lookup
|
||||
if _missing_frame_for_triggers(
|
||||
text_lower,
|
||||
_frame_triggers(_PARTITION_FRAME_NAMES),
|
||||
_PARTITION_FRAME_NAMES,
|
||||
):
|
||||
labels.add("missing_part_whole_frame")
|
||||
|
||||
# 5. missing_container_frame
|
||||
if any(w in text_lower for w in ["box", "pack", "bag", "fill", "contain", "crate", "carton"]):
|
||||
# 5. missing_container_frame — container triggers must fail lookup
|
||||
if _missing_frame_for_triggers(
|
||||
text_lower,
|
||||
_frame_triggers(_CONTAINER_FRAME_NAMES),
|
||||
_CONTAINER_FRAME_NAMES,
|
||||
):
|
||||
labels.add("missing_container_frame")
|
||||
|
||||
# 6. missing_temporal_frame
|
||||
if any(w in text_lower for w in ["hour", "minute", "day", "week", "month", "year", "work", "earn", "salary", "wage"]):
|
||||
labels.add("missing_temporal_frame")
|
||||
# 6. missing_temporal_frame — temporal surfaces with no registered frame
|
||||
for trigger in _TEMPORAL_SURFACE_TRIGGERS:
|
||||
if _surface_in_text(trigger, text_lower) and not lookup_frame(trigger):
|
||||
labels.add("missing_temporal_frame")
|
||||
break
|
||||
|
||||
# 7. missing_route_frame
|
||||
if any(w in text_lower for w in ["drive", "walk", "run", "travel", "miles per hour", "mph", "trip", "journey"]):
|
||||
# 7. missing_route_frame — travel triggers must fail lookup
|
||||
if _missing_frame_for_triggers(
|
||||
text_lower,
|
||||
_frame_triggers(_TRAVEL_FRAME_NAMES),
|
||||
_TRAVEL_FRAME_NAMES,
|
||||
):
|
||||
labels.add("missing_route_frame")
|
||||
|
||||
# 8. missing_question_target
|
||||
|
|
@ -80,13 +145,10 @@ def classify_missing_substrate(problem_text: str) -> tuple[str, ...]:
|
|||
labels.add("missing_question_target")
|
||||
|
||||
# 9. blocked_ambiguity_hazard
|
||||
for hazard_surf in [
|
||||
"half", "quarter", "third", "percent", "percentage points", "times",
|
||||
"more than", "less than", "of", "per", "each", "some", "remaining",
|
||||
"left", "total", "altogether"
|
||||
]:
|
||||
if f" {hazard_surf} " in f" {text_lower} " or text_lower.startswith(hazard_surf) or text_lower.endswith(hazard_surf):
|
||||
for hazard_surf in _AMBIGUITY_HAZARD_SURFACES:
|
||||
if _surface_in_text(hazard_surf, text_lower):
|
||||
labels.add("blocked_ambiguity_hazard")
|
||||
break
|
||||
|
||||
# 10. blocked_provenance_gap
|
||||
if "leap year" in text_lower or "calendar" in text_lower or "world fact" in text_lower:
|
||||
|
|
@ -148,4 +210,4 @@ def main() -> None:
|
|||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
main()
|
||||
|
|
@ -1,8 +1,6 @@
|
|||
"""Tests for scripts/gsm8k_substrate_morphology.py."""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from scripts.gsm8k_substrate_morphology import classify_missing_substrate
|
||||
|
||||
|
||||
|
|
@ -20,27 +18,27 @@ def test_classify_missing_substrate_labels() -> None:
|
|||
|
||||
labels = classify_missing_substrate("There are 10 bloops in the box.")
|
||||
assert "missing_unit_dimension" in labels
|
||||
assert "missing_container_frame" in labels # "box" trigger
|
||||
assert "missing_container_frame" not in labels # "box" resolves to container_packing
|
||||
|
||||
# 3. missing_process_frame
|
||||
# 3. registered process frame must not be labeled missing
|
||||
labels = classify_missing_substrate("John decides to give away his cards.")
|
||||
assert "missing_process_frame" in labels
|
||||
assert "missing_process_frame" not in labels
|
||||
|
||||
# 4. missing_part_whole_frame
|
||||
# 4. registered partition frame must not be labeled missing
|
||||
labels = classify_missing_substrate("Mary wants to split the money.")
|
||||
assert "missing_part_whole_frame" in labels
|
||||
assert "missing_part_whole_frame" not in labels
|
||||
|
||||
# 5. missing_container_frame
|
||||
# 5. registered container frame must not be labeled missing
|
||||
labels = classify_missing_substrate("Pack 10 apples into a bag.")
|
||||
assert "missing_container_frame" in labels
|
||||
assert "missing_container_frame" not in labels
|
||||
|
||||
# 6. missing_temporal_frame
|
||||
# 6. missing_temporal_frame for uncovered time surfaces
|
||||
labels = classify_missing_substrate("John worked for 5 hours to earn money.")
|
||||
assert "missing_temporal_frame" in labels
|
||||
|
||||
# 7. missing_route_frame
|
||||
# 7. registered travel frame must not be labeled missing
|
||||
labels = classify_missing_substrate("They will drive a distance of 50 miles.")
|
||||
assert "missing_route_frame" in labels
|
||||
assert "missing_route_frame" not in labels
|
||||
|
||||
# 8. missing_question_target
|
||||
labels = classify_missing_substrate("Calculate the total amount.")
|
||||
|
|
@ -55,6 +53,12 @@ def test_classify_missing_substrate_labels() -> None:
|
|||
assert "blocked_provenance_gap" in labels
|
||||
|
||||
|
||||
def test_registered_frames_suppress_missing_labels() -> None:
|
||||
labels = classify_missing_substrate("John gives 3 apples from the box.")
|
||||
assert "missing_process_frame" not in labels
|
||||
assert "missing_container_frame" not in labels
|
||||
|
||||
|
||||
def test_deterministic_and_sorted() -> None:
|
||||
problem = "John decides to split 5 bloops into boxes during a leap year."
|
||||
labels1 = classify_missing_substrate(problem)
|
||||
|
|
@ -62,8 +66,7 @@ def test_deterministic_and_sorted() -> None:
|
|||
|
||||
assert labels1 == labels2
|
||||
assert list(labels1) == sorted(labels1)
|
||||
# Check that multiple labels are correctly triggered
|
||||
assert "missing_unit_dimension" in labels1 # "bloops"
|
||||
assert "missing_part_whole_frame" in labels1 # "split"
|
||||
assert "missing_container_frame" in labels1 # "boxes"
|
||||
assert "blocked_provenance_gap" in labels1 # "leap year"
|
||||
assert "missing_unit_dimension" in labels1
|
||||
assert "missing_part_whole_frame" not in labels1
|
||||
assert "missing_container_frame" not in labels1
|
||||
assert "blocked_provenance_gap" in labels1
|
||||
|
|
@ -5,11 +5,14 @@ from fractions import Fraction
|
|||
import pytest
|
||||
|
||||
from language_packs.scalar_equivalence import (
|
||||
PROVENANCE_PROBLEM_TEXT,
|
||||
canonicalize_scalar,
|
||||
extract_scalar_candidates,
|
||||
is_supported_scalar,
|
||||
list_unsupported_surfaces,
|
||||
ScalarCandidate,
|
||||
)
|
||||
from language_packs.unit_dimensions import classify_dimension
|
||||
|
||||
|
||||
def test_fraction_words_canonicalization() -> None:
|
||||
|
|
@ -117,3 +120,59 @@ def test_is_supported_scalar() -> None:
|
|||
assert not is_supported_scalar(".5")
|
||||
assert not is_supported_scalar("1 / 2")
|
||||
assert not is_supported_scalar("random_string")
|
||||
|
||||
|
||||
def test_detached_canonicalize_has_no_problem_text_provenance() -> None:
|
||||
cand = canonicalize_scalar("half")
|
||||
assert cand is not None
|
||||
assert cand.provenance_kind is None
|
||||
assert cand.source_span is None
|
||||
assert cand.source_surface is None
|
||||
|
||||
|
||||
def test_extract_scalar_candidates_source_span_slices_text() -> None:
|
||||
text = "She ate half of a 1/2 pizza and saved 50%."
|
||||
candidates = extract_scalar_candidates(text)
|
||||
assert len(candidates) == 3
|
||||
for cand in candidates:
|
||||
assert cand.source_span is not None
|
||||
start, end = cand.source_span
|
||||
assert text[start:end] == cand.source_surface
|
||||
assert cand.provenance_kind == PROVENANCE_PROBLEM_TEXT
|
||||
|
||||
|
||||
def test_extract_scalar_candidates_provenance_is_problem_text() -> None:
|
||||
text = "The team used 0.5 of the budget."
|
||||
candidates = extract_scalar_candidates(text)
|
||||
assert len(candidates) == 1
|
||||
assert candidates[0].provenance_kind == PROVENANCE_PROBLEM_TEXT
|
||||
assert candidates[0].source_surface == "0.5"
|
||||
assert candidates[0].canonical == Fraction(1, 2)
|
||||
|
||||
|
||||
def test_pack_level_values_do_not_masquerade_as_problem_text() -> None:
|
||||
unit = classify_dimension("dollar")
|
||||
assert unit is not None
|
||||
assert unit.provenance_kind == "kernel_unit"
|
||||
|
||||
cand = canonicalize_scalar("half")
|
||||
assert cand is not None
|
||||
assert cand.provenance_kind is None
|
||||
|
||||
|
||||
def test_extract_scalar_candidates_deterministic_span_order() -> None:
|
||||
text = "A 1/2 share and 50% tip and half portion."
|
||||
first = extract_scalar_candidates(text)
|
||||
second = extract_scalar_candidates(text)
|
||||
assert first == second
|
||||
spans = [c.source_span for c in first]
|
||||
assert spans == sorted(spans)
|
||||
assert [c.source_surface for c in first] == ["1/2", "50%", "half"]
|
||||
|
||||
|
||||
def test_extract_scalar_candidates_skip_unsupported_forms() -> None:
|
||||
text = "The value is .5 and also 1 / 2 parts."
|
||||
assert extract_scalar_candidates(text) == ()
|
||||
assert "missing_scalar_equivalence" not in {
|
||||
c.source_surface for c in extract_scalar_candidates("supported 1/2 only")
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue