fix(gsm8k): tighten Gate A1 N-times factors and confuser tests
Restrict comparative matcher N-times factors to plain digits and single-word cardinals; refuse money, slash-fraction, hyphenated, and indefinite surfaces. Strengthen confuser tests to assert empty injection for any recognizer category; add graph-level refusal checks. Add Gate A1 lookback doc and EOF hygiene fixes.
This commit is contained in:
parent
e578ec72f0
commit
64ab58a7fa
4 changed files with 199 additions and 8 deletions
|
|
@ -0,0 +1,130 @@
|
||||||
|
# GSM8K Workstream A Gate A1 — multiplicative comparative injection lookback
|
||||||
|
|
||||||
|
**Date:** 2026-06-17 (post-implementation evidence closure)
|
||||||
|
**Branch:** `feat/gsm8k-workstream-a-gate-a1-comparative-multiplicative-injection`
|
||||||
|
**Head (implementation + patch):** `11abb9d572e26cb8feb57063827eda39f9279bd7`
|
||||||
|
**Base implementation commit:** `e578ec72`
|
||||||
|
**Governing ratification:** `docs/analysis/gsm8k-workstream-a-gate-a1-comparative-multiplicative-ratification-2026-06-17.md` (merged #803)
|
||||||
|
**Scope:** **First subfamily only** — multiplicative entity comparison with explicit same-sentence reference. Additive comparative (Gate A2) deferred.
|
||||||
|
|
||||||
|
## What shipped
|
||||||
|
|
||||||
|
Gate A1 closes the `COMPARATIVE_WITH_UNIT` recognizer-anchor injector frontier for the closed v1 template family:
|
||||||
|
|
||||||
|
- `twice/thrice/<N> times/half/a quarter/a third as many <unit> as <Reference>`
|
||||||
|
- Emits existing `CandidateOperation(kind="compare_multiplicative", operand=Comparison(...))`
|
||||||
|
- DCS yield guard routes comparative surfaces away from detection-only discrete-count fallback
|
||||||
|
- N-times factor narrowness: plain digit or single-word cardinal only (no money, slash-fraction, hyphenated, indefinite)
|
||||||
|
|
||||||
|
### Exact semantic path
|
||||||
|
|
||||||
|
1. **Matcher** (`generate/recognizer_match.py`): `_match_comparative_with_unit` + `_parse_comparative_v1_count_factor` on the N-times branch.
|
||||||
|
2. **Injector** (`generate/recognizer_anchor_inject.py`): `inject_comparative_multiplicative` reuses `_build_compare_multiplicative` + `roundtrip_admissible`.
|
||||||
|
3. **Registry** (`teaching/admissibility_exemplars/comparative_with_unit_v1.jsonl` + accepted proposal `bec14058…`).
|
||||||
|
4. **DCS yield** (`_match_discrete_count_statement`): returns `None` when `_is_comparative_multiplicative_v1_surface` holds.
|
||||||
|
|
||||||
|
No solver semantic changes. No `report.json` rebaseline. No sealed-lane movement.
|
||||||
|
|
||||||
|
## Changed files (implementation + patch)
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|------|------|
|
||||||
|
| `generate/recognizer_match.py` | Matcher + v1 count-factor narrowness + DCS yield |
|
||||||
|
| `generate/recognizer_anchor_inject.py` | Injector + `_INJECTORS` registration |
|
||||||
|
| `teaching/recognizer_synthesis.py` | `_synthesize_comparative_with_unit` |
|
||||||
|
| `teaching/exemplar_ingest.py` | Validator + supported category |
|
||||||
|
| `teaching/admissibility_exemplars/comparative_with_unit_v1.jsonl` | 12 exemplars |
|
||||||
|
| `teaching/proposals/proposals.jsonl` | Created + accepted recognizer proposal |
|
||||||
|
| `teaching/cognition_chains/cognition_chains_v1.jsonl` | Corpus append from accept |
|
||||||
|
| `tests/test_recognizer_comparative_inject.py` | Unit + confuser + graph tests |
|
||||||
|
| `tests/test_gsm8k_frontier_report.py` | Live `comparative_with_unit` no-injection = 0 |
|
||||||
|
|
||||||
|
## Tests run (patch)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git diff --check origin/main...HEAD
|
||||||
|
.venv/bin/python -m pytest tests/test_recognizer_comparative_inject.py -q
|
||||||
|
.venv/bin/python -m pytest tests/test_gsm8k_frontier_report.py -q
|
||||||
|
.venv/bin/python -m pytest tests/test_candidate_graph_recognizer_wiring.py -q
|
||||||
|
.venv/bin/python -m pytest tests/test_candidate_graph_completeness_guard.py -q
|
||||||
|
.venv/bin/python -m pytest tests/test_adr_0131_G2_comparatives.py -q
|
||||||
|
.venv/bin/python -m pytest tests/test_recognizer_anchor_inject.py -q
|
||||||
|
.venv/bin/python -m pytest tests/test_math_candidate_graph_rate_injection.py -q
|
||||||
|
.venv/bin/python -m core test --suite smoke -q
|
||||||
|
```
|
||||||
|
|
||||||
|
## Measurement truth (pinned vs live)
|
||||||
|
|
||||||
|
### Pinned committed artifact (unchanged)
|
||||||
|
|
||||||
|
`evals/gsm8k_math/train_sample/v1/report.json` remains the **Inc1-era / pre-Gate-A1** artifact:
|
||||||
|
|
||||||
|
- **6 correct / 44 refused / 0 wrong**
|
||||||
|
- No `comparative_with_unit` category in pinned no-injection bucket (category did not exist on serving path; comparative-bearing stmts appeared under `discrete_count_statement` no-injection)
|
||||||
|
|
||||||
|
**Not rebaselined** — intentional per ratification §9.
|
||||||
|
|
||||||
|
### Live ephemeral runner (current code)
|
||||||
|
|
||||||
|
`build_report(cases)` on Gate A1 head (ephemeral; no `report.json` write):
|
||||||
|
|
||||||
|
| Metric | Before Gate A1 (main @ ed2d04c9) | After Gate A1 |
|
||||||
|
|--------|----------------------------------|---------------|
|
||||||
|
| correct | 6 | 6 |
|
||||||
|
| refused | 44 | 44 |
|
||||||
|
| wrong | 0 | 0 |
|
||||||
|
| `comparative_with_unit` no-injection | N/A (not on serving path) | **0** |
|
||||||
|
| total `recognized_no_injection` | 31 | **31** |
|
||||||
|
|
||||||
|
Live `recognized_no_injection_by_category` (post-Gate A1):
|
||||||
|
|
||||||
|
- `discrete_count_statement: 19`
|
||||||
|
- `temporal_aggregation: 2`
|
||||||
|
- `multiplicative_aggregation: 3`
|
||||||
|
- `descriptive_setup_no_quantity: 4`
|
||||||
|
- `currency_amount: 3`
|
||||||
|
|
||||||
|
**Interpretation:** injector frontier for `comparative_with_unit` is closed (mirror Inc3 rate lesson). Aggregate proxy unchanged; refusal family reclassification expected over time as comparative-bearing stmts move off DCS misroutes.
|
||||||
|
|
||||||
|
## wrong=0
|
||||||
|
|
||||||
|
- Live ephemeral runner: `wrong: 0`
|
||||||
|
- Proposal replay at accept: `wrong_count_delta: 0`
|
||||||
|
- Confuser suite: money/slash-fraction/indefinite N-times factors refuse at matcher; any recognizer match must emit `inject_from_match() == ()`
|
||||||
|
|
||||||
|
## Explicit non-changes
|
||||||
|
|
||||||
|
- No `report.json` rebaseline
|
||||||
|
- No additive comparative (`compare_additive`) — Gate A2
|
||||||
|
- No `double` / `one-third` / hyphenated N-times factors
|
||||||
|
- No sealed-lane movement
|
||||||
|
- No solver `_apply_compare_multiplicative` semantic change
|
||||||
|
- No `determine()` / FrameVerdict / CLOSE interaction
|
||||||
|
|
||||||
|
## Known caveats
|
||||||
|
|
||||||
|
1. **Not full Gate A1 family** — multiplicative entity-comparison subfamily only; additive deferred to Gate A2.
|
||||||
|
2. **No guaranteed correct-count lift** — monotonic contract holds (`correct>=6`, `refused<=44`); primary deliverable is injector closure + reclassification visibility.
|
||||||
|
3. **DCS/parser path for bare `Jerry has 3 times`** — standalone no-reference surface may still interact with parser initials on some question shapes; Gate A1 injector refuses; completeness guard covers multi-clause confabulation class. Not claimed solved in this slice.
|
||||||
|
4. **Matcher regex vs factor narrowness** — `_COMPARE_MULT_NTIMES_RE` still uses broad `_VALUE` at template level; extraction refuses unsafe factors via `_parse_comparative_v1_count_factor`.
|
||||||
|
5. **Pinned frontier tests** — historical `report.json` fixture unchanged; live behavior tested ephemerally.
|
||||||
|
|
||||||
|
## Deferred work (Gate A1b / A2)
|
||||||
|
|
||||||
|
- `double`, `one-third`, `as much`, unit ellipsis
|
||||||
|
- Additive comparative injection
|
||||||
|
- Cross-sentence / pronoun reference (ADR-0138)
|
||||||
|
- Nested comparative composition
|
||||||
|
- Hyphenated N-times cardinals (e.g. `twenty-five times`) if ratified separately
|
||||||
|
- `report.json` rebaseline only via separate ratified PR
|
||||||
|
|
||||||
|
## Loop-closure criterion
|
||||||
|
|
||||||
|
Gate A1 implementation loop is **closed** when:
|
||||||
|
|
||||||
|
1. Ratification #803 merged (**done**)
|
||||||
|
2. Test doctrine #804 merged (**done**)
|
||||||
|
3. Matcher + injector + exemplars + accepted proposal on serving path (**done**)
|
||||||
|
4. `comparative_with_unit` live no-injection = 0 (**done**)
|
||||||
|
5. `wrong=0` on exercised lanes (**done**)
|
||||||
|
6. This lookback doc lands with patch (**this PR**)
|
||||||
|
|
@ -1844,13 +1844,42 @@ from generate.math_candidate_parser import ( # noqa: E402
|
||||||
_COMPARE_MULT_ANCHOR_RE,
|
_COMPARE_MULT_ANCHOR_RE,
|
||||||
_COMPARE_MULT_NTIMES_RE,
|
_COMPARE_MULT_NTIMES_RE,
|
||||||
_is_indefinite_quantifier,
|
_is_indefinite_quantifier,
|
||||||
_resolve_value,
|
|
||||||
)
|
)
|
||||||
|
from generate.math_roundtrip import WORD_NUMBERS # noqa: E402
|
||||||
|
|
||||||
_DEFERRED_COMPARATIVE_FACTOR_SURFACES: Final[frozenset[str]] = frozenset({
|
_DEFERRED_COMPARATIVE_FACTOR_SURFACES: Final[frozenset[str]] = frozenset({
|
||||||
"double", "triple", "quadruple", "one-third",
|
"double", "triple", "quadruple", "one-third",
|
||||||
})
|
})
|
||||||
|
|
||||||
|
_COMPARATIVE_V1_CURRENCY_PREFIXES: Final[frozenset[str]] = frozenset({
|
||||||
|
"$", "£", "€", "¥", "¢", "₱",
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_comparative_v1_count_factor(value_raw: str) -> float | None:
|
||||||
|
"""Gate A1 v1 N-times factor narrowness: plain digit or single-word cardinal.
|
||||||
|
|
||||||
|
Refuses money, slash-fraction, hyphenated, decimal, and indefinite
|
||||||
|
quantifier surfaces that the broader parser ``_VALUE`` slot admits.
|
||||||
|
"""
|
||||||
|
t = value_raw.strip()
|
||||||
|
if not t:
|
||||||
|
return None
|
||||||
|
if _is_indefinite_quantifier(t):
|
||||||
|
return None
|
||||||
|
if t[0] in _COMPARATIVE_V1_CURRENCY_PREFIXES:
|
||||||
|
return None
|
||||||
|
if "/" in t or "-" in t or "." in t:
|
||||||
|
return None
|
||||||
|
if t.isdigit():
|
||||||
|
v = int(t)
|
||||||
|
return float(v) if v > 0 else None
|
||||||
|
lower = t.lower()
|
||||||
|
if lower in WORD_NUMBERS:
|
||||||
|
v = WORD_NUMBERS[lower]
|
||||||
|
return float(v) if v > 0 else None
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def _is_comparative_multiplicative_v1_surface(statement: str) -> bool:
|
def _is_comparative_multiplicative_v1_surface(statement: str) -> bool:
|
||||||
"""True when *statement* matches the Gate A1 closed comparative template."""
|
"""True when *statement* matches the Gate A1 closed comparative template."""
|
||||||
|
|
@ -1898,10 +1927,8 @@ def _try_extract_comparative_multiplicative_anchor(
|
||||||
if not allows_numeric:
|
if not allows_numeric:
|
||||||
return None
|
return None
|
||||||
value_raw = m.group("value")
|
value_raw = m.group("value")
|
||||||
if _is_indefinite_quantifier(value_raw):
|
factor = _parse_comparative_v1_count_factor(value_raw)
|
||||||
return None
|
if factor is None:
|
||||||
rv = _resolve_value(value_raw)
|
|
||||||
if rv is None or rv.value <= 0:
|
|
||||||
return None
|
return None
|
||||||
actor_token = m.group("actor")
|
actor_token = m.group("actor")
|
||||||
unit_token = m.group("unit")
|
unit_token = m.group("unit")
|
||||||
|
|
@ -1913,7 +1940,7 @@ def _try_extract_comparative_multiplicative_anchor(
|
||||||
"reference_actor_token": reference_token,
|
"reference_actor_token": reference_token,
|
||||||
"unit_token": unit_token,
|
"unit_token": unit_token,
|
||||||
"factor_token": value_raw,
|
"factor_token": value_raw,
|
||||||
"factor": float(rv.value),
|
"factor": factor,
|
||||||
"direction": "times",
|
"direction": "times",
|
||||||
"matched_verb": "times",
|
"matched_verb": "times",
|
||||||
"comparator_phrase": phrase,
|
"comparator_phrase": phrase,
|
||||||
|
|
|
||||||
|
|
@ -9,4 +9,4 @@
|
||||||
{"exemplar_id": "cwu-v1-0009", "shape_category": "comparative_with_unit", "statement": "Ivan has 3 times as many cards as Jerry.", "expected_graph": {"subject": "Ivan", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Ivan", "factor_token": "3", "factor_kind": "numeric", "direction": "times", "unit_token": "cards", "reference_actor_token": "Jerry"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
{"exemplar_id": "cwu-v1-0009", "shape_category": "comparative_with_unit", "statement": "Ivan has 3 times as many cards as Jerry.", "expected_graph": {"subject": "Ivan", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Ivan", "factor_token": "3", "factor_kind": "numeric", "direction": "times", "unit_token": "cards", "reference_actor_token": "Jerry"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
||||||
{"exemplar_id": "cwu-v1-0010", "shape_category": "comparative_with_unit", "statement": "Kira gained twice as many points as Leo.", "expected_graph": {"subject": "Kira", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Kira", "factor_token": "twice", "factor_kind": "anchor", "direction": "times", "unit_token": "points", "reference_actor_token": "Leo"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
{"exemplar_id": "cwu-v1-0010", "shape_category": "comparative_with_unit", "statement": "Kira gained twice as many points as Leo.", "expected_graph": {"subject": "Kira", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Kira", "factor_token": "twice", "factor_kind": "anchor", "direction": "times", "unit_token": "points", "reference_actor_token": "Leo"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
||||||
{"exemplar_id": "cwu-v1-0011", "shape_category": "comparative_with_unit", "statement": "Nina studied three times as many pages as Omar.", "expected_graph": {"subject": "Nina", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Nina", "factor_token": "three", "factor_kind": "numeric", "direction": "times", "unit_token": "pages", "reference_actor_token": "Omar"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
{"exemplar_id": "cwu-v1-0011", "shape_category": "comparative_with_unit", "statement": "Nina studied three times as many pages as Omar.", "expected_graph": {"subject": "Nina", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Nina", "factor_token": "three", "factor_kind": "numeric", "direction": "times", "unit_token": "pages", "reference_actor_token": "Omar"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
||||||
{"exemplar_id": "cwu-v1-0012", "shape_category": "comparative_with_unit", "statement": "Paula has half as many marbles as Quinn.", "expected_graph": {"subject": "Paula", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Paula", "factor_token": "half", "factor_kind": "anchor", "direction": "fraction", "unit_token": "marbles", "reference_actor_token": "Quinn"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
{"exemplar_id": "cwu-v1-0012", "shape_category": "comparative_with_unit", "statement": "Paula has half as many marbles as Quinn.", "expected_graph": {"subject": "Paula", "quantity_anchors": [{"kind": "comparative_multiplicative", "subject_role": "Paula", "factor_token": "half", "factor_kind": "anchor", "direction": "fraction", "unit_token": "marbles", "reference_actor_token": "Quinn"}], "graph_intent": "compare", "outcome": "admissible"}, "provenance": {"source": "gate_a1_seed", "author": "Grok (Gate A1)", "round": 1, "category_rank": 3}}
|
||||||
|
|
|
||||||
|
|
@ -11,6 +11,7 @@ import types
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from evals.refusal_taxonomy.shape_categories import ShapeCategory
|
from evals.refusal_taxonomy.shape_categories import ShapeCategory
|
||||||
|
from generate.math_candidate_graph import parse_and_solve
|
||||||
from generate.math_candidate_parser import CandidateOperation
|
from generate.math_candidate_parser import CandidateOperation
|
||||||
from generate.math_problem_graph import Comparison
|
from generate.math_problem_graph import Comparison
|
||||||
from generate.math_roundtrip import roundtrip_admissible
|
from generate.math_roundtrip import roundtrip_admissible
|
||||||
|
|
@ -106,16 +107,49 @@ def test_positive_surfaces_emit_compare_multiplicative(
|
||||||
"Alice has one-third as many apples as Bob.",
|
"Alice has one-third as many apples as Bob.",
|
||||||
"Alice has double as many apples as Bob.",
|
"Alice has double as many apples as Bob.",
|
||||||
"Jerry has 3 times",
|
"Jerry has 3 times",
|
||||||
|
"Alice has $2 times as many apples as Bob.",
|
||||||
|
"Alice has 3/4 times as many apples as Bob.",
|
||||||
|
"Alice has some times as many apples as Bob.",
|
||||||
|
"Alice has twenty-five times as many apples as Bob.",
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
def test_confuser_surfaces_refuse_injection(sentence: str):
|
def test_confuser_surfaces_refuse_injection(sentence: str):
|
||||||
registry = load_ratified_registry()
|
registry = load_ratified_registry()
|
||||||
m = match(sentence, registry)
|
m = match(sentence, registry)
|
||||||
if m is None or m.category is not ShapeCategory.COMPARATIVE_WITH_UNIT:
|
if m is None:
|
||||||
return
|
return
|
||||||
assert inject_from_match(m, sentence, sealed=False) == ()
|
assert inject_from_match(m, sentence, sealed=False) == ()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"sentence",
|
||||||
|
[
|
||||||
|
"Alice has $2 times as many apples as Bob.",
|
||||||
|
"Alice has 3/4 times as many apples as Bob.",
|
||||||
|
"Alice has some times as many apples as Bob.",
|
||||||
|
"Alice has twenty-five times as many apples as Bob.",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_ntimes_factor_confusers_do_not_match_comparative(sentence: str):
|
||||||
|
registry = load_ratified_registry()
|
||||||
|
m = match(sentence, registry)
|
||||||
|
assert m is None or m.category is not ShapeCategory.COMPARATIVE_WITH_UNIT
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"text",
|
||||||
|
[
|
||||||
|
"Jerry has 3 times as many apples. How many apples does Jerry have?",
|
||||||
|
"Alice has $2 times as many apples as Bob. How many apples does Alice have?",
|
||||||
|
"Alice has 3 more apples than Bob. How many apples does Alice have?",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_graph_confusers_refuse_without_compare_lift(text: str):
|
||||||
|
res = parse_and_solve(text, sealed=False)
|
||||||
|
assert res.answer is None
|
||||||
|
assert res.refusal_reason is not None
|
||||||
|
|
||||||
|
|
||||||
def test_unknown_actor_refuses():
|
def test_unknown_actor_refuses():
|
||||||
emitted = inject_comparative_multiplicative(
|
emitted = inject_comparative_multiplicative(
|
||||||
_make_match(_anchor(actor="fish")),
|
_make_match(_anchor(actor="fish")),
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue