feat(gsm8k): Gate A1 multiplicative comparative recognizer injection (#805)
* feat(gsm8k): Gate A1 multiplicative comparative recognizer injection Add COMPARATIVE_WITH_UNIT matcher/injector emitting compare_multiplicative for the closed v1 template family (twice/thrice/N-times/half/quarter/third). DCS yields comparative surfaces instead of detection-only fallback. Includes ratified exemplar corpus + accepted recognizer proposal, 19 unit tests, and live frontier proof that comparative_with_unit no-injection = 0. wrong=0 preserved; no report.json rebaseline. * fix(gsm8k): tighten Gate A1 N-times factors and confuser tests Restrict comparative matcher N-times factors to plain digits and single-word cardinals; refuse money, slash-fraction, hyphenated, and indefinite surfaces. Strengthen confuser tests to assert empty injection for any recognizer category; add graph-level refusal checks. Add Gate A1 lookback doc and EOF hygiene fixes. * docs(analysis): pin Gate A1 lookback head SHA * chore(gsm8k): fix Gate A1 file endings
This commit is contained in:
Shay
GitHub
parent
fa7819a2e7
commit
bb0830046a
10 changed files with 664 additions and 1 deletions
|
|
@ -0,0 +1,130 @@
|
|||
# GSM8K Workstream A Gate A1 — multiplicative comparative injection lookback
|
||||
|
||||
**Date:** 2026-06-17 (post-implementation evidence closure)
|
||||
**Branch:** `feat/gsm8k-workstream-a-gate-a1-comparative-multiplicative-injection`
|
||||
**Head (implementation + patch):** `64ab58a7fa01ccfb1c707573b1be044296f5fe38`
|
||||
**Base implementation commit:** `e578ec72`
|
||||
**Governing ratification:** `docs/analysis/gsm8k-workstream-a-gate-a1-comparative-multiplicative-ratification-2026-06-17.md` (merged #803)
|
||||
**Scope:** **First subfamily only** — multiplicative entity comparison with explicit same-sentence reference. Additive comparative (Gate A2) deferred.
|
||||
|
||||
## What shipped
|
||||
|
||||
Gate A1 closes the `COMPARATIVE_WITH_UNIT` recognizer-anchor injector frontier for the closed v1 template family:
|
||||
|
||||
- `twice/thrice/<N> times/half/a quarter/a third as many <unit> as <Reference>`
|
||||
- Emits existing `CandidateOperation(kind="compare_multiplicative", operand=Comparison(...))`
|
||||
- DCS yield guard routes comparative surfaces away from detection-only discrete-count fallback
|
||||
- N-times factor narrowness: plain digit or single-word cardinal only (no money, slash-fraction, hyphenated, indefinite)
|
||||
|
||||
### Exact semantic path
|
||||
|
||||
1. **Matcher** (`generate/recognizer_match.py`): `_match_comparative_with_unit` + `_parse_comparative_v1_count_factor` on the N-times branch.
|
||||
2. **Injector** (`generate/recognizer_anchor_inject.py`): `inject_comparative_multiplicative` reuses `_build_compare_multiplicative` + `roundtrip_admissible`.
|
||||
3. **Registry** (`teaching/admissibility_exemplars/comparative_with_unit_v1.jsonl` + accepted proposal `bec14058…`).
|
||||
4. **DCS yield** (`_match_discrete_count_statement`): returns `None` when `_is_comparative_multiplicative_v1_surface` holds.
|
||||
|
||||
No solver semantic changes. No `report.json` rebaseline. No sealed-lane movement.
|
||||
|
||||
## Changed files (implementation + patch)
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `generate/recognizer_match.py` | Matcher + v1 count-factor narrowness + DCS yield |
|
||||
| `generate/recognizer_anchor_inject.py` | Injector + `_INJECTORS` registration |
|
||||
| `teaching/recognizer_synthesis.py` | `_synthesize_comparative_with_unit` |
|
||||
| `teaching/exemplar_ingest.py` | Validator + supported category |
|
||||
| `teaching/admissibility_exemplars/comparative_with_unit_v1.jsonl` | 12 exemplars |
|
||||
| `teaching/proposals/proposals.jsonl` | Created + accepted recognizer proposal |
|
||||
| `teaching/cognition_chains/cognition_chains_v1.jsonl` | Corpus append from accept |
|
||||
| `tests/test_recognizer_comparative_inject.py` | Unit + confuser + graph tests |
|
||||
| `tests/test_gsm8k_frontier_report.py` | Live `comparative_with_unit` no-injection = 0 |
|
||||
|
||||
## Tests run (patch)
|
||||
|
||||
```bash
|
||||
git diff --check origin/main...HEAD
|
||||
.venv/bin/python -m pytest tests/test_recognizer_comparative_inject.py -q
|
||||
.venv/bin/python -m pytest tests/test_gsm8k_frontier_report.py -q
|
||||
.venv/bin/python -m pytest tests/test_candidate_graph_recognizer_wiring.py -q
|
||||
.venv/bin/python -m pytest tests/test_candidate_graph_completeness_guard.py -q
|
||||
.venv/bin/python -m pytest tests/test_adr_0131_G2_comparatives.py -q
|
||||
.venv/bin/python -m pytest tests/test_recognizer_anchor_inject.py -q
|
||||
.venv/bin/python -m pytest tests/test_math_candidate_graph_rate_injection.py -q
|
||||
.venv/bin/python -m core test --suite smoke -q
|
||||
```
|
||||
|
||||
## Measurement truth (pinned vs live)
|
||||
|
||||
### Pinned committed artifact (unchanged)
|
||||
|
||||
`evals/gsm8k_math/train_sample/v1/report.json` remains the **Inc1-era / pre-Gate-A1** artifact:
|
||||
|
||||
- **6 correct / 44 refused / 0 wrong**
|
||||
- No `comparative_with_unit` category in pinned no-injection bucket (category did not exist on serving path; comparative-bearing stmts appeared under `discrete_count_statement` no-injection)
|
||||
|
||||
**Not rebaselined** — intentional per ratification §9.
|
||||
|
||||
### Live ephemeral runner (current code)
|
||||
|
||||
`build_report(cases)` on Gate A1 head (ephemeral; no `report.json` write):
|
||||
|
||||
| Metric | Before Gate A1 (main @ ed2d04c9) | After Gate A1 |
|
||||
|--------|----------------------------------|---------------|
|
||||
| correct | 6 | 6 |
|
||||
| refused | 44 | 44 |
|
||||
| wrong | 0 | 0 |
|
||||
| `comparative_with_unit` no-injection | N/A (not on serving path) | **0** |
|
||||
| total `recognized_no_injection` | 31 | **31** |
|
||||
|
||||
Live `recognized_no_injection_by_category` (post-Gate A1):
|
||||
|
||||
- `discrete_count_statement: 19`
|
||||
- `temporal_aggregation: 2`
|
||||
- `multiplicative_aggregation: 3`
|
||||
- `descriptive_setup_no_quantity: 4`
|
||||
- `currency_amount: 3`
|
||||
|
||||
**Interpretation:** injector frontier for `comparative_with_unit` is closed (mirror Inc3 rate lesson). Aggregate proxy unchanged; refusal family reclassification expected over time as comparative-bearing stmts move off DCS misroutes.
|
||||
|
||||
## wrong=0
|
||||
|
||||
- Live ephemeral runner: `wrong: 0`
|
||||
- Proposal replay at accept: `wrong_count_delta: 0`
|
||||
- Confuser suite: money/slash-fraction/indefinite N-times factors refuse at matcher; any recognizer match must emit `inject_from_match() == ()`
|
||||
|
||||
## Explicit non-changes
|
||||
|
||||
- No `report.json` rebaseline
|
||||
- No additive comparative (`compare_additive`) — Gate A2
|
||||
- No `double` / `one-third` / hyphenated N-times factors
|
||||
- No sealed-lane movement
|
||||
- No solver `_apply_compare_multiplicative` semantic change
|
||||
- No `determine()` / FrameVerdict / CLOSE interaction
|
||||
|
||||
## Known caveats
|
||||
|
||||
1. **Not full Gate A1 family** — multiplicative entity-comparison subfamily only; additive deferred to Gate A2.
|
||||
2. **No guaranteed correct-count lift** — monotonic contract holds (`correct>=6`, `refused<=44`); primary deliverable is injector closure + reclassification visibility.
|
||||
3. **DCS/parser path for bare `Jerry has 3 times`** — standalone no-reference surface may still interact with parser initials on some question shapes; Gate A1 injector refuses; completeness guard covers multi-clause confabulation class. Not claimed solved in this slice.
|
||||
4. **Matcher regex vs factor narrowness** — `_COMPARE_MULT_NTIMES_RE` still uses broad `_VALUE` at template level; extraction refuses unsafe factors via `_parse_comparative_v1_count_factor`.
|
||||
5. **Pinned frontier tests** — historical `report.json` fixture unchanged; live behavior tested ephemerally.
|
||||
|
||||
## Deferred work (Gate A1b / A2)
|
||||
|
||||
- `double`, `one-third`, `as much`, unit ellipsis
|
||||
- Additive comparative injection
|
||||
- Cross-sentence / pronoun reference (ADR-0138)
|
||||
- Nested comparative composition
|
||||
- Hyphenated N-times cardinals (e.g. `twenty-five times`) if ratified separately
|
||||
- `report.json` rebaseline only via separate ratified PR
|
||||
|
||||
## Loop-closure criterion
|
||||
|
||||
Gate A1 implementation loop is **closed** when:
|
||||
|
||||
1. Ratification #803 merged (**done**)
|
||||
2. Test doctrine #804 merged (**done**)
|
||||
3. Matcher + injector + exemplars + accepted proposal on serving path (**done**)
|
||||
4. `comparative_with_unit` live no-injection = 0 (**done**)
|
||||
5. `wrong=0` on exercised lanes (**done**)
|
||||
6. This lookback doc lands with patch (**this PR**)
|
||||
|
|
@ -48,7 +48,11 @@ from __future__ import annotations
|
|||
from typing import Mapping, Union
|
||||
|
||||
from evals.refusal_taxonomy.shape_categories import ShapeCategory
|
||||
from generate.math_candidate_parser import CandidateInitial, CandidateOperation
|
||||
from generate.math_candidate_parser import (
|
||||
CandidateInitial,
|
||||
CandidateOperation,
|
||||
_build_compare_multiplicative,
|
||||
)
|
||||
from generate.math_problem_graph import (
|
||||
InitialPossession,
|
||||
MathGraphError,
|
||||
|
|
@ -56,6 +60,7 @@ from generate.math_problem_graph import (
|
|||
Quantity,
|
||||
Rate,
|
||||
)
|
||||
from generate.math_roundtrip import roundtrip_admissible
|
||||
from generate.recognizer_match import (
|
||||
RecognizerMatch,
|
||||
extract_proper_noun_subject,
|
||||
|
|
@ -714,6 +719,66 @@ def inject_rate_with_currency(
|
|||
return tuple(out)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gate A1 — comparative_with_unit → compare_multiplicative (Workstream A)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def inject_comparative_multiplicative(
|
||||
match: RecognizerMatch,
|
||||
sentence: str,
|
||||
) -> tuple[InjectorEmission, ...]:
|
||||
"""Narrow injector for ShapeCategory.COMPARATIVE_WITH_UNIT.
|
||||
|
||||
Emits ``CandidateOperation(kind="compare_multiplicative")`` only when
|
||||
the matcher published a fully grounded comparative anchor and
|
||||
:func:`roundtrip_admissible` accepts the construction.
|
||||
"""
|
||||
if not match.parsed_anchors or len(match.parsed_anchors) != 1:
|
||||
return ()
|
||||
|
||||
anchor = match.parsed_anchors[0]
|
||||
if not isinstance(anchor, dict):
|
||||
return ()
|
||||
if anchor.get("kind") != "comparative_multiplicative":
|
||||
return ()
|
||||
|
||||
actor_token = anchor.get("actor_token")
|
||||
reference_token = anchor.get("reference_actor_token")
|
||||
unit_token = anchor.get("unit_token")
|
||||
factor_token = anchor.get("factor_token")
|
||||
matched_verb = anchor.get("matched_verb")
|
||||
direction = anchor.get("direction")
|
||||
factor = anchor.get("factor")
|
||||
|
||||
if not all(
|
||||
isinstance(v, str) and v
|
||||
for v in (actor_token, reference_token, unit_token, factor_token, matched_verb, direction)
|
||||
):
|
||||
return ()
|
||||
if not isinstance(factor, (int, float)) or factor <= 0:
|
||||
return ()
|
||||
|
||||
# Narrow actor binding (mirror rate v1): ProperName subject only.
|
||||
actor = extract_proper_noun_subject(sentence)
|
||||
if not actor or actor != actor_token:
|
||||
return ()
|
||||
|
||||
cand = _build_compare_multiplicative(
|
||||
actor_raw=actor_token,
|
||||
factor=float(factor),
|
||||
matched_verb=matched_verb,
|
||||
matched_value_token=factor_token,
|
||||
unit_raw=unit_token,
|
||||
reference_raw=reference_token,
|
||||
source=sentence,
|
||||
direction=direction,
|
||||
)
|
||||
if cand is None or not roundtrip_admissible(cand):
|
||||
return ()
|
||||
return (cand,)
|
||||
|
||||
|
||||
_INJECTORS: Mapping[ShapeCategory, "type"] = {
|
||||
ShapeCategory.DISCRETE_COUNT_STATEMENT: inject_discrete_count_statement, # type: ignore[dict-item]
|
||||
# WAVE-A — multiplicative_aggregation now has a per-category
|
||||
|
|
@ -729,6 +794,10 @@ _INJECTORS: Mapping[ShapeCategory, "type"] = {
|
|||
# frontier for the currency-per-unit surfaces without touching
|
||||
# sealed lanes or any other category.
|
||||
ShapeCategory.RATE_WITH_CURRENCY: inject_rate_with_currency, # type: ignore[dict-item]
|
||||
# Gate A1 (Workstream A) — comparative_with_unit emits
|
||||
# CandidateOperation(kind="compare_multiplicative") for the closed
|
||||
# v1 multiplicative entity-comparison template family.
|
||||
ShapeCategory.COMPARATIVE_WITH_UNIT: inject_comparative_multiplicative, # type: ignore[dict-item]
|
||||
# All other recognizer categories continue to route to the
|
||||
# empty-tuple fallback (explicit "recognizer matched but produced
|
||||
# no injection" refusal in the candidate-graph). That is the
|
||||
|
|
@ -764,4 +833,5 @@ __all__ = [
|
|||
"inject_from_match",
|
||||
"inject_discrete_count_statement",
|
||||
"inject_rate_with_currency",
|
||||
"inject_comparative_multiplicative",
|
||||
]
|
||||
|
|
|
|||
|
|
@ -810,6 +810,10 @@ def _match_discrete_count_statement(
|
|||
return None
|
||||
if _has_temporal_quantifier(padded):
|
||||
return None
|
||||
# Gate A1 — yield comparative multiplicative surfaces to
|
||||
# COMPARATIVE_WITH_UNIT instead of detection-only DCS fallback.
|
||||
if _is_comparative_multiplicative_v1_surface(statement):
|
||||
return None
|
||||
|
||||
anchor = _try_extract_discrete_count_anchor(statement, padded, spec)
|
||||
if anchor is not None:
|
||||
|
|
@ -1831,6 +1835,140 @@ def _match_currency_amount(
|
|||
return (tuple(), "amount")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gate A1 — comparative_with_unit → compare_multiplicative (Workstream A)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
from generate.math_candidate_parser import ( # noqa: E402
|
||||
_ANCHOR_TO_FACTOR,
|
||||
_COMPARE_MULT_ANCHOR_RE,
|
||||
_COMPARE_MULT_NTIMES_RE,
|
||||
_is_indefinite_quantifier,
|
||||
)
|
||||
from generate.math_roundtrip import WORD_NUMBERS # noqa: E402
|
||||
|
||||
_DEFERRED_COMPARATIVE_FACTOR_SURFACES: Final[frozenset[str]] = frozenset({
|
||||
"double", "triple", "quadruple", "one-third",
|
||||
})
|
||||
|
||||
_COMPARATIVE_V1_CURRENCY_PREFIXES: Final[frozenset[str]] = frozenset({
|
||||
"$", "£", "€", "¥", "¢", "₱",
|
||||
})
|
||||
|
||||
|
||||
def _parse_comparative_v1_count_factor(value_raw: str) -> float | None:
|
||||
"""Gate A1 v1 N-times factor narrowness: plain digit or single-word cardinal.
|
||||
|
||||
Refuses money, slash-fraction, hyphenated, decimal, and indefinite
|
||||
quantifier surfaces that the broader parser ``_VALUE`` slot admits.
|
||||
"""
|
||||
t = value_raw.strip()
|
||||
if not t:
|
||||
return None
|
||||
if _is_indefinite_quantifier(t):
|
||||
return None
|
||||
if t[0] in _COMPARATIVE_V1_CURRENCY_PREFIXES:
|
||||
return None
|
||||
if "/" in t or "-" in t or "." in t:
|
||||
return None
|
||||
if t.isdigit():
|
||||
v = int(t)
|
||||
return float(v) if v > 0 else None
|
||||
lower = t.lower()
|
||||
if lower in WORD_NUMBERS:
|
||||
v = WORD_NUMBERS[lower]
|
||||
return float(v) if v > 0 else None
|
||||
return None
|
||||
|
||||
|
||||
def _is_comparative_multiplicative_v1_surface(statement: str) -> bool:
|
||||
"""True when *statement* matches the Gate A1 closed comparative template."""
|
||||
s = statement.strip()
|
||||
if _COMPARE_MULT_ANCHOR_RE.match(s) is not None:
|
||||
return True
|
||||
return _COMPARE_MULT_NTIMES_RE.match(s) is not None
|
||||
|
||||
|
||||
def _try_extract_comparative_multiplicative_anchor(
|
||||
statement: str,
|
||||
spec: Mapping[str, Any],
|
||||
) -> Mapping[str, Any] | None:
|
||||
"""Extract one comparative_multiplicative anchor when narrowness holds."""
|
||||
s = statement.strip()
|
||||
observed_anchors = set(spec.get("observed_factor_anchors") or ())
|
||||
allows_numeric = bool(spec.get("allows_numeric_factor"))
|
||||
|
||||
m = _COMPARE_MULT_ANCHOR_RE.match(s)
|
||||
if m is not None:
|
||||
anchor_word = m.group("anchor").lower()
|
||||
if anchor_word in _DEFERRED_COMPARATIVE_FACTOR_SURFACES:
|
||||
return None
|
||||
if anchor_word not in observed_anchors:
|
||||
return None
|
||||
factor, direction = _ANCHOR_TO_FACTOR[anchor_word]
|
||||
actor_token = m.group("actor")
|
||||
unit_token = m.group("unit")
|
||||
reference_token = m.group("reference")
|
||||
phrase = f"{anchor_word} as many {unit_token}"
|
||||
return {
|
||||
"kind": "comparative_multiplicative",
|
||||
"actor_token": actor_token,
|
||||
"reference_actor_token": reference_token,
|
||||
"unit_token": unit_token,
|
||||
"factor_token": anchor_word,
|
||||
"factor": factor,
|
||||
"direction": direction,
|
||||
"matched_verb": anchor_word,
|
||||
"comparator_phrase": phrase,
|
||||
}
|
||||
|
||||
m = _COMPARE_MULT_NTIMES_RE.match(s)
|
||||
if m is not None:
|
||||
if not allows_numeric:
|
||||
return None
|
||||
value_raw = m.group("value")
|
||||
factor = _parse_comparative_v1_count_factor(value_raw)
|
||||
if factor is None:
|
||||
return None
|
||||
actor_token = m.group("actor")
|
||||
unit_token = m.group("unit")
|
||||
reference_token = m.group("reference")
|
||||
phrase = f"{value_raw} times as many {unit_token}"
|
||||
return {
|
||||
"kind": "comparative_multiplicative",
|
||||
"actor_token": actor_token,
|
||||
"reference_actor_token": reference_token,
|
||||
"unit_token": unit_token,
|
||||
"factor_token": value_raw,
|
||||
"factor": factor,
|
||||
"direction": "times",
|
||||
"matched_verb": "times",
|
||||
"comparator_phrase": phrase,
|
||||
}
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _match_comparative_with_unit(
|
||||
statement: str, spec: Mapping[str, Any]
|
||||
) -> tuple[tuple[Mapping[str, Any], ...], Literal["compare"]] | None:
|
||||
"""Gate A1 — multiplicative entity comparison with explicit reference.
|
||||
|
||||
Strict match only: returns populated anchors on full v1 template
|
||||
match, ``None`` otherwise (no detection-only fallback).
|
||||
"""
|
||||
if spec.get("anchor_kind") != "comparative_multiplicative":
|
||||
return None
|
||||
anchor = _try_extract_comparative_multiplicative_anchor(statement, spec)
|
||||
if anchor is None:
|
||||
return None
|
||||
cmin = int(spec.get("anchor_count_min", 1))
|
||||
cmax = int(spec.get("anchor_count_max", 1))
|
||||
if not (cmin <= 1 <= cmax):
|
||||
return None
|
||||
return ((anchor,), "compare")
|
||||
|
||||
|
||||
_MATCHERS: Final[dict[ShapeCategory, Any]] = {
|
||||
ShapeCategory.DESCRIPTIVE_SETUP_NO_QUANTITY: _match_descriptive_setup_no_quantity,
|
||||
ShapeCategory.TEMPORAL_AGGREGATION: _match_temporal_aggregation,
|
||||
|
|
@ -1838,6 +1976,7 @@ _MATCHERS: Final[dict[ShapeCategory, Any]] = {
|
|||
ShapeCategory.DISCRETE_COUNT_STATEMENT: _match_discrete_count_statement,
|
||||
ShapeCategory.MULTIPLICATIVE_AGGREGATION: _match_multiplicative_aggregation,
|
||||
ShapeCategory.CURRENCY_AMOUNT: _match_currency_amount,
|
||||
ShapeCategory.COMPARATIVE_WITH_UNIT: _match_comparative_with_unit,
|
||||
}
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,12 @@
|
|||
{"exemplar_id": "cwu-v1-0001", "shape_category": "comparative_with_unit", "statement": "Alice has twice as many apples as Bob.", "expected_graph": {"subject": "Alice", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Alice", "factor_token": "twice", "factor_kind": "anchor", "direction": "times", "unit_token": "apples", "reference_actor_token": "Bob"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
||||
{"exemplar_id": "cwu-v1-0002", "shape_category": "comparative_with_unit", "statement": "Jerry has thrice as many apples as Tom.", "expected_graph": {"subject": "Jerry", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Jerry", "factor_token": "thrice", "factor_kind": "anchor", "direction": "times", "unit_token": "apples", "reference_actor_token": "Tom"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
||||
{"exemplar_id": "cwu-v1-0003", "shape_category": "comparative_with_unit", "statement": "Brooke has three times as many jumping jacks as Sidney.", "expected_graph": {"subject": "Brooke", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Brooke", "factor_token": "three", "factor_kind": "numeric", "direction": "times", "unit_token": "jumping jacks", "reference_actor_token": "Sidney"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3, "train_case_id": "gsm8k-train-sample-v1-0024"}}
|
||||
{"exemplar_id": "cwu-v1-0004", "shape_category": "comparative_with_unit", "statement": "Dana has 4 times as many pencils as Eli.", "expected_graph": {"subject": "Dana", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Dana", "factor_token": "4", "factor_kind": "numeric", "direction": "times", "unit_token": "pencils", "reference_actor_token": "Eli"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
||||
{"exemplar_id": "cwu-v1-0005", "shape_category": "comparative_with_unit", "statement": "Alice has half as many apples as Bob.", "expected_graph": {"subject": "Alice", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Alice", "factor_token": "half", "factor_kind": "anchor", "direction": "fraction", "unit_token": "apples", "reference_actor_token": "Bob"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
||||
{"exemplar_id": "cwu-v1-0006", "shape_category": "comparative_with_unit", "statement": "Alice has a quarter as many apples as Bob.", "expected_graph": {"subject": "Alice", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Alice", "factor_token": "quarter", "factor_kind": "anchor", "direction": "fraction", "unit_token": "apples", "reference_actor_token": "Bob"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
||||
{"exemplar_id": "cwu-v1-0007", "shape_category": "comparative_with_unit", "statement": "Alice has a third as many apples as Bob.", "expected_graph": {"subject": "Alice", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Alice", "factor_token": "third", "factor_kind": "anchor", "direction": "fraction", "unit_token": "apples", "reference_actor_token": "Bob"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
||||
{"exemplar_id": "cwu-v1-0008", "shape_category": "comparative_with_unit", "statement": "Mason collected twice as many shells as Nora.", "expected_graph": {"subject": "Mason", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Mason", "factor_token": "twice", "factor_kind": "anchor", "direction": "times", "unit_token": "shells", "reference_actor_token": "Nora"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
||||
{"exemplar_id": "cwu-v1-0009", "shape_category": "comparative_with_unit", "statement": "Ivan has 3 times as many cards as Jerry.", "expected_graph": {"subject": "Ivan", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Ivan", "factor_token": "3", "factor_kind": "numeric", "direction": "times", "unit_token": "cards", "reference_actor_token": "Jerry"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
||||
{"exemplar_id": "cwu-v1-0010", "shape_category": "comparative_with_unit", "statement": "Kira gained twice as many points as Leo.", "expected_graph": {"subject": "Kira", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Kira", "factor_token": "twice", "factor_kind": "anchor", "direction": "times", "unit_token": "points", "reference_actor_token": "Leo"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
||||
{"exemplar_id": "cwu-v1-0011", "shape_category": "comparative_with_unit", "statement": "Nina studied three times as many pages as Omar.", "expected_graph": {"subject": "Nina", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Nina", "factor_token": "three", "factor_kind": "numeric", "direction": "times", "unit_token": "pages", "reference_actor_token": "Omar"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
||||
{"exemplar_id": "cwu-v1-0012", "shape_category": "comparative_with_unit", "statement": "Paula has half as many marbles as Quinn.", "expected_graph": {"subject": "Paula", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Paula", "factor_token": "half", "factor_kind": "anchor", "direction": "fraction", "unit_token": "marbles", "reference_actor_token": "Quinn"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
||||
|
|
@ -24,3 +24,4 @@
|
|||
{"chain_id":"admissibility_multiplicative_aggregation_recognizes_c70663bd722672930b6e2f24596fbab28e94135aeffd05e7037fc6f35e5702f7","connective":"recognizes","domains_object_k":1,"domains_subject_k":2,"intent":"admissibility","object":"c70663bd722672930b6e2f24596fbab28e94135aeffd05e7037fc6f35e5702f7","provenance":"adr-0057:discovery_promoted:2026-05-27","subject":"multiplicative_aggregation"}
|
||||
{"chain_id":"admissibility_currency_amount_recognizes_25df92963a15941294c232f37c91987bb35f91e8f3a483f950da43f84c2b7684","connective":"recognizes","domains_object_k":1,"domains_subject_k":2,"intent":"admissibility","object":"25df92963a15941294c232f37c91987bb35f91e8f3a483f950da43f84c2b7684","provenance":"adr-0057:discovery_promoted:2026-05-27","subject":"currency_amount"}
|
||||
{"chain_id":"admissibility_temporal_aggregation_recognizes_9684dd780b4d0d387facdce18b474e09413d671b0d4cb944c2754ba2a0bb6208","connective":"recognizes","domains_object_k":1,"domains_subject_k":2,"intent":"admissibility","object":"9684dd780b4d0d387facdce18b474e09413d671b0d4cb944c2754ba2a0bb6208","provenance":"adr-0057:discovery_promoted:2026-05-27","subject":"temporal_aggregation"}
|
||||
{"chain_id":"admissibility_comparative_with_unit_recognizes_f3be480f69b85cff21ff6525d769a92fa21f0ef89dfb5e3af076265b90d5883d","connective":"recognizes","domains_object_k":1,"domains_subject_k":2,"intent":"admissibility","object":"f3be480f69b85cff21ff6525d769a92fa21f0ef89dfb5e3af076265b90d5883d","provenance":"adr-0057:discovery_promoted:2026-06-17","subject":"comparative_with_unit"}
|
||||
|
|
|
|||
|
|
@ -61,6 +61,8 @@ _SUPPORTED_CATEGORIES: frozenset[ShapeCategory] = frozenset({
|
|||
ShapeCategory.DISCRETE_COUNT_STATEMENT,
|
||||
ShapeCategory.MULTIPLICATIVE_AGGREGATION,
|
||||
ShapeCategory.CURRENCY_AMOUNT,
|
||||
# Gate A1 (Workstream A) — multiplicative comparative injection.
|
||||
ShapeCategory.COMPARATIVE_WITH_UNIT,
|
||||
})
|
||||
|
||||
|
||||
|
|
@ -266,6 +268,45 @@ def _validate_multiplicative_aggregation(ctx: str, graph: Mapping[str, Any]) ->
|
|||
raise ExemplarIngestError(f"{ctx} outcome must be 'admissible'")
|
||||
|
||||
|
||||
def _validate_comparative_with_unit(ctx: str, graph: Mapping[str, Any]) -> None:
|
||||
anchors = graph["quantity_anchors"]
|
||||
if not isinstance(anchors, list) or not anchors:
|
||||
raise ExemplarIngestError(f"{ctx} comparative_with_unit needs ≥1 anchor")
|
||||
for a in anchors:
|
||||
if not isinstance(a, Mapping):
|
||||
raise ExemplarIngestError(f"{ctx} anchor must be a mapping")
|
||||
_require_keys(ctx, a, frozenset({
|
||||
"kind",
|
||||
"subject_role",
|
||||
"factor_token",
|
||||
"factor_kind",
|
||||
"direction",
|
||||
"unit_token",
|
||||
"reference_actor_token",
|
||||
}))
|
||||
if a["kind"] != "comparative_multiplicative":
|
||||
raise ExemplarIngestError(
|
||||
f"{ctx} anchor kind must be 'comparative_multiplicative'"
|
||||
)
|
||||
if a["factor_kind"] not in {"anchor", "numeric"}:
|
||||
raise ExemplarIngestError(
|
||||
f"{ctx} factor_kind {a['factor_kind']!r} must be 'anchor' or 'numeric'"
|
||||
)
|
||||
if a["direction"] not in {"times", "fraction"}:
|
||||
raise ExemplarIngestError(
|
||||
f"{ctx} direction {a['direction']!r} must be 'times' or 'fraction'"
|
||||
)
|
||||
for fld in (
|
||||
"subject_role", "factor_token", "unit_token", "reference_actor_token",
|
||||
):
|
||||
if not isinstance(a[fld], str) or not a[fld]:
|
||||
raise ExemplarIngestError(f"{ctx} {fld} must be non-empty str")
|
||||
if graph["graph_intent"] != "compare":
|
||||
raise ExemplarIngestError(f"{ctx} graph_intent must be 'compare'")
|
||||
if graph["outcome"] != "admissible":
|
||||
raise ExemplarIngestError(f"{ctx} outcome must be 'admissible'")
|
||||
|
||||
|
||||
def _validate_currency_amount(ctx: str, graph: Mapping[str, Any]) -> None:
|
||||
anchors = graph["quantity_anchors"]
|
||||
if not isinstance(anchors, list) or not anchors:
|
||||
|
|
@ -306,6 +347,7 @@ _CATEGORY_VALIDATORS = {
|
|||
ShapeCategory.DISCRETE_COUNT_STATEMENT: _validate_discrete_count_statement,
|
||||
ShapeCategory.MULTIPLICATIVE_AGGREGATION: _validate_multiplicative_aggregation,
|
||||
ShapeCategory.CURRENCY_AMOUNT: _validate_currency_amount,
|
||||
ShapeCategory.COMPARATIVE_WITH_UNIT: _validate_comparative_with_unit,
|
||||
}
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -78,3 +78,7 @@
|
|||
{"event":"transition","note":"WAVE-A re-seed with extract_values=True","proposal_id":"rat1-seed-4dc30608fb783bc7","review_date":"2026-05-27","to":"accepted"}
|
||||
{"event":"created","proposal":{"claim_domain":"factual","evidence":[],"polarity":"affirms","proposal_id":"rat1-seed-8c3d568c7f90771c","proposed_chain":{"connective":"ratifies","intent":"recognizer_spec_seed","object":"multiplicative_aggregate","recognizer_spec":{"canonical_pattern":{"anchor_kind":"multiplicative_aggregate","extract_values":true,"graph_intent":"aggregate","observed_units":["apple","apples","basket","baskets","book","books","ounce","ounces","strawberries","strawberry"],"outcome":"admissible","shape_category":"multiplicative_aggregation"},"coverage":{},"exemplar_count":0,"exemplar_digest":"8c3d568c7f90771c533e507e614b5385719420198941654a1305628c7b2d81c8","shape_category":"multiplicative_aggregation"},"subject":"multiplicative_aggregation"},"source":{"emitted_at_revision":"flywheel-demo","kind":"exemplar_corpus","source_id":"8c3d568c7f90771c533e507e614b5385719420198941654a1305628c7b2d81c8"}}}
|
||||
{"event":"transition","note":"flywheel-demo seed","proposal_id":"rat1-seed-8c3d568c7f90771c","review_date":"2026-05-27","to":"accepted"}
|
||||
{"event":"created","proposal":{"claim_domain":"factual","evidence":[{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:cwu-v1-0001","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:cwu-v1-0002","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:gsm8k-train-sample-v1-0024","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:cwu-v1-0004","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:cwu-v1-0005","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:cwu-v1-0006","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:cwu-v1-0007","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:cwu-v1-0008","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:cwu-v1-0009","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:cwu-v1-0010","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:cwu-v1-0011","source":"corpus"},{"epistemic_status":"coherent","polarity":"affirms","ref":"exemplar:cwu-v1-0012","source":"corpus"}],"operator_note":"","polarity":"affirms","proposal_id":"bec14058b9afbb76216414e903106ae9","proposed_chain":{"connective":"recognizes","intent":"admissibility","object":"f3be480f69b85cff21ff6525d769a92fa21f0ef89dfb5e3af076265b90d5883d","recognizer_spec":{"canonical_pattern":{"allows_numeric_factor":true,"anchor_count_max":1,"anchor_count_min":1,"anchor_kind":"comparative_multiplicative","graph_intent":"compare","observed_factor_anchors":["half","quarter","third","thrice","twice"],"outcome":"admissible","shape_category":"comparative_with_unit","unresolved_notes":[]},"coverage":{"anchors_comparative_multiplicative":12,"factor:half":2,"factor:numeric":4,"factor:quarter":1,"factor:third":1,"factor:thrice":1,"factor:twice":3},"exemplar_count":12,"exemplar_digest":"4891b1ea35c17dcf92f089ebf0dbd6b4075ad0fe340ab31b36e24f7e4f30fadf","shape_category":"comparative_with_unit"},"subject":"comparative_with_unit"},"provenance":null,"replay_evidence":null,"review_state":"pending","source":{"emitted_at_revision":"ed2d04c99e90f96d04689d0d94c825fc9a48d5b1","kind":"exemplar_corpus","source_id":"4891b1ea35c17dcf92f089ebf0dbd6b4075ad0fe340ab31b36e24f7e4f30fadf"},"source_candidate_id":"3973a72c777f9418df3f7605c7e98e9c4a4b873ce3df111737227c4ea7afd0ed"}}
|
||||
{"event":"replay","proposal_id":"bec14058b9afbb76216414e903106ae9","replay_evidence":{"baseline":{"intent_accuracy":1.0,"surface_groundedness":1.0,"term_capture_rate":1.0,"versor_closure_rate":1.0},"candidate":{"intent_accuracy":1.0,"surface_groundedness":1.0,"term_capture_rate":1.0,"versor_closure_rate":1.0},"capability_axes":{"G1_verb_classes":{"correct":20,"refused":0,"wrong":0},"G2_comparatives":{"correct":29,"refused":0,"wrong":0},"G3_numerics":{"correct":20,"refused":6,"wrong":0},"G4_multi_clause":{"correct":32,"refused":0,"wrong":0},"G5_aggregate":{"correct":20,"refused":0,"wrong":0},"S1_rate_events":{"correct":20,"refused":0,"wrong":0}},"gsm8k_train_sample":{"correct":6,"refused":44,"wrong":0},"regressed_metrics":[],"replay_equivalent":true,"wrong_count_delta":0}}
|
||||
{"event":"transition","note":"Gate A1 ratification 2026-06-17","proposal_id":"bec14058b9afbb76216414e903106ae9","to":"accepted"}
|
||||
{"chain_id":"admissibility_comparative_with_unit_recognizes_f3be480f69b85cff21ff6525d769a92fa21f0ef89dfb5e3af076265b90d5883d","event":"accepted_corpus_append","proposal_id":"bec14058b9afbb76216414e903106ae9","provenance":{"adr_id":"adr-0057","raw":"adr-0057:discovery_promoted:2026-06-17","review_date":"2026-06-17","source":"discovery_promoted"}}
|
||||
|
|
|
|||
|
|
@ -347,6 +347,48 @@ def _synthesize_multiplicative_aggregation(
|
|||
return canonical_pattern, coverage
|
||||
|
||||
|
||||
def _synthesize_comparative_with_unit(
|
||||
corpus: ExemplarCorpus,
|
||||
) -> tuple[Mapping[str, Any], Mapping[str, int]]:
|
||||
"""Gate A1 — multiplicative entity comparison seeds."""
|
||||
exemplars = corpus.exemplars
|
||||
factor_anchors: list[str] = []
|
||||
anchor_counts: list[int] = []
|
||||
coverage_factor: dict[str, int] = {}
|
||||
has_numeric = False
|
||||
|
||||
for ex in exemplars:
|
||||
anchors = ex.expected_graph["quantity_anchors"]
|
||||
anchor_counts.append(len(anchors))
|
||||
for a in anchors:
|
||||
fk = a.get("factor_kind", "anchor")
|
||||
if fk == "numeric":
|
||||
has_numeric = True
|
||||
coverage_factor["numeric"] = coverage_factor.get("numeric", 0) + 1
|
||||
else:
|
||||
token = a["factor_token"]
|
||||
factor_anchors.append(token)
|
||||
coverage_factor[token] = coverage_factor.get(token, 0) + 1
|
||||
|
||||
canonical_pattern: dict[str, Any] = {
|
||||
"shape_category": ShapeCategory.COMPARATIVE_WITH_UNIT.value,
|
||||
"graph_intent": "compare",
|
||||
"outcome": "admissible",
|
||||
"anchor_kind": "comparative_multiplicative",
|
||||
"observed_factor_anchors": _sorted_unique(factor_anchors),
|
||||
"allows_numeric_factor": has_numeric,
|
||||
"anchor_count_min": min(anchor_counts),
|
||||
"anchor_count_max": max(anchor_counts),
|
||||
"unresolved_notes": _collect_author_notes(exemplars),
|
||||
}
|
||||
coverage: dict[str, int] = {
|
||||
"anchors_comparative_multiplicative": sum(anchor_counts),
|
||||
}
|
||||
for token, n in sorted(coverage_factor.items()):
|
||||
coverage[f"factor:{token}"] = n
|
||||
return canonical_pattern, coverage
|
||||
|
||||
|
||||
def _synthesize_currency_amount(
|
||||
corpus: ExemplarCorpus,
|
||||
) -> tuple[Mapping[str, Any], Mapping[str, int]]:
|
||||
|
|
@ -398,6 +440,7 @@ _SYNTHESIZERS = {
|
|||
ShapeCategory.DISCRETE_COUNT_STATEMENT: _synthesize_discrete_count_statement,
|
||||
ShapeCategory.MULTIPLICATIVE_AGGREGATION: _synthesize_multiplicative_aggregation,
|
||||
ShapeCategory.CURRENCY_AMOUNT: _synthesize_currency_amount,
|
||||
ShapeCategory.COMPARATIVE_WITH_UNIT: _synthesize_comparative_with_unit,
|
||||
}
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -107,6 +107,35 @@ def test_post_inc3_live_runner_has_zero_rate_no_injection():
|
|||
assert cats.get("rate_with_currency", 0) == 0
|
||||
|
||||
|
||||
def test_post_gate_a1_live_runner_has_zero_comparative_no_injection():
|
||||
"""Live train_sample: comparative_with_unit bucket closed at injector."""
|
||||
import re
|
||||
from collections import Counter
|
||||
|
||||
from evals.gsm8k_math.train_sample.v1.runner import build_report
|
||||
from tests.gsm8k_train_sample_baseline import assert_monotonic_serving_counts
|
||||
|
||||
cases_path = _REPO_ROOT / "evals/gsm8k_math/train_sample/v1/cases.jsonl"
|
||||
cases = [
|
||||
json.loads(line)
|
||||
for line in cases_path.read_text(encoding="utf-8").splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
report = build_report(cases)
|
||||
assert_monotonic_serving_counts(report["counts"])
|
||||
|
||||
cats: Counter[str] = Counter()
|
||||
for row in report["per_case"]:
|
||||
reason = row.get("reason", "")
|
||||
if "produced no injection" not in reason:
|
||||
continue
|
||||
m = re.search(r"category=(\w+)", reason)
|
||||
if m:
|
||||
cats[m.group(1)] += 1
|
||||
|
||||
assert cats.get("comparative_with_unit", 0) == 0
|
||||
|
||||
|
||||
def test_classify_and_extract_category_logic():
|
||||
"""Unit the internal classification on the exact reason strings the graph emits."""
|
||||
# We exercise via the public analyze path with a tiny synthetic report
|
||||
|
|
|
|||
193
tests/test_recognizer_comparative_inject.py
Normal file
193
tests/test_recognizer_comparative_inject.py
Normal file
|
|
@ -0,0 +1,193 @@
|
|||
"""Gate A1 — comparative_with_unit recognizer-anchor injection tests.
|
||||
|
||||
Mirrors the Inc2 rate injector ladder: unit confusers, live-registry dispatch,
|
||||
half/quarter/third serving proof, and DCS yield for comparative surfaces.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import types
|
||||
|
||||
import pytest
|
||||
|
||||
from evals.refusal_taxonomy.shape_categories import ShapeCategory
|
||||
from generate.math_candidate_graph import parse_and_solve
|
||||
from generate.math_candidate_parser import CandidateOperation
|
||||
from generate.math_problem_graph import Comparison
|
||||
from generate.math_roundtrip import roundtrip_admissible
|
||||
from generate.recognizer_anchor_inject import (
|
||||
inject_comparative_multiplicative,
|
||||
inject_from_match,
|
||||
)
|
||||
from generate.recognizer_match import RecognizerMatch, match
|
||||
from generate.recognizer_registry import load_ratified_registry
|
||||
|
||||
|
||||
def _stub_recognizer(category: ShapeCategory) -> types.SimpleNamespace:
|
||||
return types.SimpleNamespace(shape_category=category, canonical_pattern={})
|
||||
|
||||
|
||||
def _make_match(anchor: dict) -> RecognizerMatch:
|
||||
return RecognizerMatch(
|
||||
recognizer=_stub_recognizer(ShapeCategory.COMPARATIVE_WITH_UNIT),
|
||||
category=ShapeCategory.COMPARATIVE_WITH_UNIT,
|
||||
outcome="admissible",
|
||||
graph_intent="compare",
|
||||
parsed_anchors=(anchor,),
|
||||
)
|
||||
|
||||
|
||||
def _anchor(
|
||||
*,
|
||||
actor: str = "Alice",
|
||||
reference: str = "Bob",
|
||||
unit: str = "apples",
|
||||
factor_token: str = "twice",
|
||||
factor: float = 2.0,
|
||||
direction: str = "times",
|
||||
matched_verb: str = "twice",
|
||||
) -> dict:
|
||||
return {
|
||||
"kind": "comparative_multiplicative",
|
||||
"actor_token": actor,
|
||||
"reference_actor_token": reference,
|
||||
"unit_token": unit,
|
||||
"factor_token": factor_token,
|
||||
"factor": factor,
|
||||
"direction": direction,
|
||||
"matched_verb": matched_verb,
|
||||
"comparator_phrase": f"{factor_token} as many {unit}",
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"sentence,actor,reference,unit,factor_token,factor,direction,matched_verb",
|
||||
[
|
||||
("Alice has twice as many apples as Bob.", "Alice", "Bob", "apples", "twice", 2.0, "times", "twice"),
|
||||
("Jerry has thrice as many apples as Tom.", "Jerry", "Tom", "apples", "thrice", 3.0, "times", "thrice"),
|
||||
("Dana has 4 times as many pencils as Eli.", "Dana", "Eli", "pencils", "4", 4.0, "times", "times"),
|
||||
("Alice has half as many apples as Bob.", "Alice", "Bob", "apples", "half", 0.5, "fraction", "half"),
|
||||
("Alice has a quarter as many apples as Bob.", "Alice", "Bob", "apples", "quarter", 0.25, "fraction", "quarter"),
|
||||
("Alice has a third as many apples as Bob.", "Alice", "Bob", "apples", "third", 1.0 / 3.0, "fraction", "third"),
|
||||
],
|
||||
)
|
||||
def test_positive_surfaces_emit_compare_multiplicative(
|
||||
sentence, actor, reference, unit, factor_token, factor, direction, matched_verb
|
||||
):
|
||||
emitted = inject_comparative_multiplicative(_make_match(_anchor(
|
||||
actor=actor,
|
||||
reference=reference,
|
||||
unit=unit,
|
||||
factor_token=factor_token,
|
||||
factor=factor,
|
||||
direction=direction,
|
||||
matched_verb=matched_verb,
|
||||
)), sentence)
|
||||
assert len(emitted) == 1
|
||||
cand = emitted[0]
|
||||
assert isinstance(cand, CandidateOperation)
|
||||
assert cand.op.kind == "compare_multiplicative"
|
||||
assert isinstance(cand.op.operand, Comparison)
|
||||
assert cand.op.operand.factor == factor
|
||||
assert cand.op.operand.direction == direction
|
||||
assert cand.matched_value_token == factor_token
|
||||
assert cand.matched_verb == matched_verb
|
||||
assert roundtrip_admissible(cand) is True
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"sentence",
|
||||
[
|
||||
"Jerry has 3 times as many apples.",
|
||||
"Jerry has twice as many apples.",
|
||||
"Jerry has 3 times more apples than Bob.",
|
||||
"Alice has 3 more apples than Bob.",
|
||||
"He has twice as many apples as Bob.",
|
||||
"Alice lost twice as many apples as Bob.",
|
||||
"Alice has one-third as many apples as Bob.",
|
||||
"Alice has double as many apples as Bob.",
|
||||
"Jerry has 3 times",
|
||||
"Alice has $2 times as many apples as Bob.",
|
||||
"Alice has 3/4 times as many apples as Bob.",
|
||||
"Alice has some times as many apples as Bob.",
|
||||
"Alice has twenty-five times as many apples as Bob.",
|
||||
],
|
||||
)
|
||||
def test_confuser_surfaces_refuse_injection(sentence: str):
|
||||
registry = load_ratified_registry()
|
||||
m = match(sentence, registry)
|
||||
if m is None:
|
||||
return
|
||||
assert inject_from_match(m, sentence, sealed=False) == ()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"sentence",
|
||||
[
|
||||
"Alice has $2 times as many apples as Bob.",
|
||||
"Alice has 3/4 times as many apples as Bob.",
|
||||
"Alice has some times as many apples as Bob.",
|
||||
"Alice has twenty-five times as many apples as Bob.",
|
||||
],
|
||||
)
|
||||
def test_ntimes_factor_confusers_do_not_match_comparative(sentence: str):
|
||||
registry = load_ratified_registry()
|
||||
m = match(sentence, registry)
|
||||
assert m is None or m.category is not ShapeCategory.COMPARATIVE_WITH_UNIT
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"text",
|
||||
[
|
||||
"Jerry has 3 times as many apples. How many apples does Jerry have?",
|
||||
"Alice has $2 times as many apples as Bob. How many apples does Alice have?",
|
||||
"Alice has 3 more apples than Bob. How many apples does Alice have?",
|
||||
],
|
||||
)
|
||||
def test_graph_confusers_refuse_without_compare_lift(text: str):
|
||||
res = parse_and_solve(text, sealed=False)
|
||||
assert res.answer is None
|
||||
assert res.refusal_reason is not None
|
||||
|
||||
|
||||
def test_unknown_actor_refuses():
|
||||
emitted = inject_comparative_multiplicative(
|
||||
_make_match(_anchor(actor="fish")),
|
||||
"fish have twice as many apples as Bob.",
|
||||
)
|
||||
assert emitted == ()
|
||||
|
||||
|
||||
def test_dispatch_table_routes_comparative_with_unit():
|
||||
registry = load_ratified_registry()
|
||||
stmt = "Alice has twice as many apples as Bob."
|
||||
m = match(stmt, registry)
|
||||
assert m is not None
|
||||
assert m.category is ShapeCategory.COMPARATIVE_WITH_UNIT
|
||||
emitted = inject_from_match(m, stmt, sealed=False)
|
||||
assert len(emitted) == 1
|
||||
assert roundtrip_admissible(emitted[0]) is True
|
||||
|
||||
|
||||
def test_dcs_yields_comparative_not_initial_times():
|
||||
registry = load_ratified_registry()
|
||||
stmt = "Jerry has 3 times as many apples as Tom."
|
||||
m = match(stmt, registry)
|
||||
assert m is not None
|
||||
assert m.category is ShapeCategory.COMPARATIVE_WITH_UNIT
|
||||
emitted = inject_from_match(m, stmt, sealed=False)
|
||||
assert len(emitted) == 1
|
||||
assert emitted[0].op.kind == "compare_multiplicative"
|
||||
|
||||
|
||||
def test_matched_tokens_ground_in_source_sentence():
|
||||
sentence = "Nina studied three times as many pages as Omar."
|
||||
registry = load_ratified_registry()
|
||||
m = match(sentence, registry)
|
||||
assert m is not None
|
||||
emitted = inject_from_match(m, sentence, sealed=False)
|
||||
assert len(emitted) == 1
|
||||
c = emitted[0]
|
||||
assert c.matched_actor_token in sentence
|
||||
assert c.matched_reference_actor_token in sentence
|
||||
assert c.matched_unit_token in sentence
|
||||
Loading…
Reference in a new issue