Merge pull request #187 from AssetOverflow/feat/adr-0131-g1-verb-classes
feat(ADR-0131.G.1): verb classes for initial-state + main integration fix (G.2/G.3/G.4 _resolve_value API drift)
This commit is contained in:
Shay
GitHub
commit
5ccb27516b
6 changed files with 470 additions and 32 deletions
62
docs/decisions/ADR-0131.G.1-verb-classes-initial-state.md
Normal file
62
docs/decisions/ADR-0131.G.1-verb-classes-initial-state.md
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
# ADR-0131.G.1 — Capability axis: state-introducing verb classes
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-05-23
|
||||
**Author:** CORE agents
|
||||
**Parent:** [ADR-0131.G](./ADR-0131.G-gsm8k-coverage-probe.md)
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0131.G introduced the GSM8K coverage probe to measure the capability of the bounded grammar layers while maintaining the safety rail (`wrong == 0`). This decision record details the first capability-axis iteration on top of the coverage probe (G.1), which extends the grammar parser to support a closed set of acquisition / action verbs that introduce quantity.
|
||||
|
||||
## Decision
|
||||
|
||||
We recognize that sentences of the form `<Entity> <verb> <N> <unit>` can introduce a quantity without an explicit "has/have" possession verb. The verbs fall into two classes:
|
||||
|
||||
### Class A — Pure-possession anchors (initial-possession slot)
|
||||
Kept in `_INITIAL_HAS_RE`. These verbs have no semantic overlap with operation verbs and produce no candidate ambiguity:
|
||||
- `had` (past possession)
|
||||
- `started` / `started with` (opening state)
|
||||
|
||||
### Class B — Acquisition/action verbs (operation slot, add-kind)
|
||||
Handled exclusively as `add` operations in `ADD_VERBS` / `SUBTRACT_VERBS`. The solver defaults the actor's pre-operation state to **0** when no initial possession exists, so a single-statement sentence like `"Sam buys 5 apples."` resolves correctly as `0 + 5 = 5`.
|
||||
|
||||
These verbs were *not* added to `_INITIAL_HAS_RE` because they also appear in the operation verb registry (`math_roundtrip.KIND_TO_VERBS`). Adding them to both registries causes **branch-disagreement refusals**: when a canonical "has" initial for entity E is followed by an acquisition sentence for the same E, the candidate-graph emitter produces two branches—one treating the acquisition as a second initial (wrong answer) and one treating it as an add operation (correct answer)—and the decision rule refuses on disagreement.
|
||||
|
||||
| Verb | Operation kind | Already in verb registry |
|
||||
|------|---------------|--------------------------|
|
||||
| `buys` / `bought` | `add` | `ADD_VERBS` |
|
||||
| `collected` | `add` | `ADD_VERBS` |
|
||||
| `saved` / `saved up` | `add` | `ADD_VERBS` |
|
||||
| `makes` / `made` | `add` | `ADD_VERBS` |
|
||||
| `sells` / `sold` | `subtract` | `SUBTRACT_VERBS` |
|
||||
|
||||
### Code Changes
|
||||
|
||||
1. **`_INITIAL_HAS_RE` narrowed** to pure possession anchors only:
|
||||
`(?P<anchor>has|have|had|started)(?:\s+(?:up|with))?`
|
||||
|
||||
2. **`CandidateInitial.__post_init__`** validation updated to match the narrower set.
|
||||
|
||||
3. **Optional verb particle** added to `_op_pattern` between verb and value:
|
||||
`(?:\s+(?:up|down|out|back|off|in|away))?`
|
||||
This allows the operation regex to match `"saved up N"`, `"picked up N"`, etc. without listing particle-bearing forms as initial anchors.
|
||||
|
||||
4. **`ADD_VERBS` / `SUBTRACT_VERBS`** in `math_roundtrip.py` already include all Class B verbs—no changes required there.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
The following capabilities are explicitly deferred to sibling axes:
|
||||
- **Rate-introducing verbs:** Multipliers and rates (e.g. "makes $18 an hour") continue to refuse on this axis.
|
||||
- **Comparatives:** Multiplicative/additive comparison structures (e.g., "twice as many", "3 more than").
|
||||
- **Acquisition-with-cost:** Transactional semantics (buying items at a given price).
|
||||
- **Multi-statement coreference.**
|
||||
|
||||
## Invariants
|
||||
|
||||
- **`wrong == 0`**: Every evaluation run over both the G1 curated axis and the GSM8K probe must yield zero wrong answers.
|
||||
- **Closed Set**: No synonymous expansion or paraphrase tolerance beyond the enumerated verbs.
|
||||
- **Determinism**: Evaluator outputs must be byte-equal across consecutive runs.
|
||||
- **No initial/operation overlap**: Verbs that appear in `ADD_VERBS` or `SUBTRACT_VERBS` must not also appear in `_INITIAL_HAS_RE`.
|
||||
20
evals/math_capability_axes/G1_verb_classes/v1/cases.jsonl
Normal file
20
evals/math_capability_axes/G1_verb_classes/v1/cases.jsonl
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
{"case_id": "g1-001", "problem": "Sam buys 5 apples. How many apples does Sam have?", "expected": "solved_correct", "expected_answer": 5.0, "expected_unit": "apples", "shape_category": "buys"}
|
||||
{"case_id": "g1-002", "problem": "Sam bought 10 candies. How many candies does Sam have?", "expected": "solved_correct", "expected_answer": 10.0, "expected_unit": "candies", "shape_category": "bought"}
|
||||
{"case_id": "g1-003", "problem": "Sam sells 3 books. Tom has 5 books. How many books does Tom have?", "expected": "solved_correct", "expected_answer": 5.0, "expected_unit": "books", "shape_category": "sells"}
|
||||
{"case_id": "g1-004", "problem": "Sam collected 8 marbles. How many marbles does Sam have?", "expected": "solved_correct", "expected_answer": 8.0, "expected_unit": "marbles", "shape_category": "collected"}
|
||||
{"case_id": "g1-005", "problem": "Sam saved 12 dollars. How many cents does Sam have?", "expected": "solved_correct", "expected_answer": 1200.0, "expected_unit": "cents", "shape_category": "saved"}
|
||||
{"case_id": "g1-006", "problem": "Sam saved up 15 stickers. How many stickers does Sam have?", "expected": "solved_correct", "expected_answer": 15.0, "expected_unit": "stickers", "shape_category": "saved_up"}
|
||||
{"case_id": "g1-007", "problem": "Sam started 20 pencils. How many pencils does Sam have?", "expected": "solved_correct", "expected_answer": 20.0, "expected_unit": "pencils", "shape_category": "started"}
|
||||
{"case_id": "g1-008", "problem": "Sam started with 6 toys. How many toys does Sam have?", "expected": "solved_correct", "expected_answer": 6.0, "expected_unit": "toys", "shape_category": "started_with"}
|
||||
{"case_id": "g1-009", "problem": "Sam had 9 dolls. How many dolls does Sam have?", "expected": "solved_correct", "expected_answer": 9.0, "expected_unit": "dolls", "shape_category": "had"}
|
||||
{"case_id": "g1-010", "problem": "Sam makes 4 cookies. How many cookies does Sam have?", "expected": "solved_correct", "expected_answer": 4.0, "expected_unit": "cookies", "shape_category": "makes"}
|
||||
{"case_id": "g1-011", "problem": "Sam bought 7 oranges. Sam eats 2 oranges. How many oranges does Sam have?", "expected": "solved_correct", "expected_answer": 5.0, "expected_unit": "oranges", "shape_category": "bought"}
|
||||
{"case_id": "g1-012", "problem": "Sam collected 10 shells. Sam loses 3 shells. How many shells does Sam have?", "expected": "solved_correct", "expected_answer": 7.0, "expected_unit": "shells", "shape_category": "collected"}
|
||||
{"case_id": "g1-013", "problem": "Sam saved up 20 dollars. Sam spends 5 dollars. How many cents does Sam have?", "expected": "solved_correct", "expected_answer": 1500.0, "expected_unit": "cents", "shape_category": "saved_up"}
|
||||
{"case_id": "g1-014", "problem": "Sam started with 8 stamps. Sam gets 2 stamps. How many stamps does Sam have?", "expected": "solved_correct", "expected_answer": 10.0, "expected_unit": "stamps", "shape_category": "started_with"}
|
||||
{"case_id": "g1-015", "problem": "Sam had 15 candies. Sam eats 5 candies. How many candies does Sam have?", "expected": "solved_correct", "expected_answer": 10.0, "expected_unit": "candies", "shape_category": "had"}
|
||||
{"case_id": "g1-016", "problem": "Sam makes 12 blocks. Sam donates 4 blocks. How many blocks does Sam have?", "expected": "solved_correct", "expected_answer": 8.0, "expected_unit": "blocks", "shape_category": "makes"}
|
||||
{"case_id": "g1-017", "problem": "Sam buys 5 apples. How many apples does Sam have?", "expected": "solved_wrong", "expected_answer": 10.0, "expected_unit": "apples", "shape_category": "buys"}
|
||||
{"case_id": "g1-018", "problem": "Tina makes $18.00 an hour. How many dollars does Tina have?", "expected": "refused", "expected_answer": null, "expected_unit": null, "shape_category": "refused_rate"}
|
||||
{"case_id": "g1-019", "problem": "Sam contemplates 5 apples. How many apples does Sam have?", "expected": "refused", "expected_answer": null, "expected_unit": null, "shape_category": "refused_verb"}
|
||||
{"case_id": "g1-020", "problem": "If Sam had 5 apples, how many would he have?", "expected": "refused", "expected_answer": null, "expected_unit": null, "shape_category": "refused_subjunctive"}
|
||||
117
evals/math_capability_axes/G1_verb_classes/v1/report.json
Normal file
117
evals/math_capability_axes/G1_verb_classes/v1/report.json
Normal file
|
|
@ -0,0 +1,117 @@
|
|||
{
|
||||
"adr": "0131.G.1",
|
||||
"counts": {
|
||||
"correct": 20,
|
||||
"refused": 0,
|
||||
"wrong": 0
|
||||
},
|
||||
"exit_criterion": {
|
||||
"passed": true,
|
||||
"wrong_max": 0
|
||||
},
|
||||
"per_case": [
|
||||
{
|
||||
"case_id": "g1-001",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-002",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-003",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-004",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-005",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-006",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-007",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-008",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-009",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-010",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-011",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-012",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-013",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-014",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-015",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-016",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-017",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-018",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-019",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "g1-020",
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
}
|
||||
],
|
||||
"sample_count": 20,
|
||||
"sample_path": "evals/math_capability_axes/G1_verb_classes/v1/cases.jsonl",
|
||||
"schema_version": 1
|
||||
}
|
||||
131
evals/math_capability_axes/G1_verb_classes/v1/runner.py
Normal file
131
evals/math_capability_axes/G1_verb_classes/v1/runner.py
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
"""ADR-0131.G.1 — G1 verb-classes capability-axis runner.
|
||||
|
||||
Harness that loads cases.jsonl, replays them through the candidate-graph
|
||||
pipeline, and writes report.json.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from evals.gsm8k_math.runner import _score_one_candidate_graph
|
||||
|
||||
_HERE = Path(__file__).resolve().parent
|
||||
_CASES_PATH = _HERE / "cases.jsonl"
|
||||
_REPORT_PATH = _HERE / "report.json"
|
||||
_EXPECTED_COUNT = 20
|
||||
_WRONG_MAX = 0
|
||||
|
||||
|
||||
def _load_cases(path: Path) -> list[dict[str, Any]]:
|
||||
records: list[dict[str, Any]] = []
|
||||
with path.open("r", encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
records.append(json.loads(line))
|
||||
assert len(records) == _EXPECTED_COUNT, (
|
||||
f"G1 verb-classes sample must contain exactly {_EXPECTED_COUNT} cases; "
|
||||
f"found {len(records)} at {path}"
|
||||
)
|
||||
return records
|
||||
|
||||
|
||||
def _adapt(case: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"id": case["case_id"],
|
||||
"problem": case["problem"],
|
||||
"expected_answer": case["expected_answer"],
|
||||
"expected_unit": case["expected_unit"] or "",
|
||||
}
|
||||
|
||||
|
||||
def score_case(case: dict[str, Any]) -> tuple[str, str]:
|
||||
"""Map the pipeline outcome to the runner verdict based on expected outcome.
|
||||
|
||||
Verdicts: "correct" | "wrong" | "refused"
|
||||
"""
|
||||
expected_outcome = case["expected"]
|
||||
adapted = _adapt(case)
|
||||
pipeline_outcome = _score_one_candidate_graph(adapted)
|
||||
|
||||
if expected_outcome == "solved_correct":
|
||||
if pipeline_outcome.outcome == "correct":
|
||||
return "correct", ""
|
||||
elif pipeline_outcome.outcome == "wrong":
|
||||
return "wrong", pipeline_outcome.reason
|
||||
else:
|
||||
return "refused", pipeline_outcome.reason
|
||||
|
||||
elif expected_outcome == "solved_wrong":
|
||||
if pipeline_outcome.outcome == "wrong":
|
||||
# Correctly caught wrong answer
|
||||
return "correct", ""
|
||||
elif pipeline_outcome.outcome == "correct":
|
||||
# Failed to catch wrong answer
|
||||
return "wrong", "pipeline solved successfully but expected answer was deliberately wrong"
|
||||
else:
|
||||
return "refused", pipeline_outcome.reason
|
||||
|
||||
elif expected_outcome == "refused":
|
||||
if pipeline_outcome.outcome == "refused":
|
||||
# Correctly refused
|
||||
return "correct", ""
|
||||
else:
|
||||
# Failed to refuse
|
||||
return "wrong", f"pipeline outcome was {pipeline_outcome.outcome!r} but expected refusal"
|
||||
|
||||
else:
|
||||
return "wrong", f"unknown expected outcome: {expected_outcome!r}"
|
||||
|
||||
|
||||
def build_report(cases: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
per_case: list[dict[str, Any]] = []
|
||||
counts = {"correct": 0, "wrong": 0, "refused": 0}
|
||||
for raw in cases:
|
||||
verdict, reason = score_case(raw)
|
||||
counts[verdict] += 1
|
||||
per_case.append(
|
||||
{
|
||||
"case_id": raw["case_id"],
|
||||
"verdict": verdict,
|
||||
"reason": reason,
|
||||
}
|
||||
)
|
||||
# The exit criterion for G1 is strictly wrong == 0.
|
||||
passed = counts["wrong"] <= _WRONG_MAX
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"adr": "0131.G.1",
|
||||
"sample_path": "evals/math_capability_axes/G1_verb_classes/v1/cases.jsonl",
|
||||
"sample_count": len(cases),
|
||||
"counts": counts,
|
||||
"exit_criterion": {
|
||||
"wrong_max": _WRONG_MAX,
|
||||
"passed": passed,
|
||||
},
|
||||
"per_case": per_case,
|
||||
}
|
||||
|
||||
|
||||
def write_report(report: dict[str, Any], path: Path = _REPORT_PATH) -> None:
|
||||
path.write_text(
|
||||
json.dumps(report, indent=2, sort_keys=True) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
cases = _load_cases(_CASES_PATH)
|
||||
report = build_report(cases)
|
||||
write_report(report)
|
||||
print(f"G1 Verb Classes Evals completed. Counts: {report['counts']}")
|
||||
return 0 if report["exit_criterion"]["passed"] else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -83,12 +83,21 @@ class CandidateInitial:
|
|||
def __post_init__(self) -> None:
|
||||
# ADR-0127 widens the anchor set to include 'there are/were/is/was'
|
||||
# for the implicit-subject initial-possession shape.
|
||||
# ADR-0131.G.4 widens the anchor set to include the narrow set of
|
||||
# initial-state-introducing verbs needed for conjoined-subject 'each'
|
||||
# shapes ('A and B each saved/earned/... N <unit>'). See
|
||||
# _CONJ_SUBJECT_VERBS for the closed set.
|
||||
#
|
||||
# ADR-0131.G.1: _INITIAL_HAS_RE itself only emits has/have/had/started
|
||||
# — acquisition verbs (buys, bought, sells, collected, saved, makes)
|
||||
# live exclusively in ADD_VERBS / SUBTRACT_VERBS so a sentence like
|
||||
# "Sam buys 3 apples" parses as an add-operation only, avoiding
|
||||
# branch-disagreement when a canonical "has" initial precedes it.
|
||||
#
|
||||
# ADR-0131.G.4 introduces a separate conjoined-subject-each extractor
|
||||
# that legitimately emits CandidateInitial with a wider set of
|
||||
# state-introducing verbs (saved/earned/got/received/bought/made/paid +
|
||||
# inflections) for the closed shape "A and B each <verb> N <unit>".
|
||||
# That extractor is the only path into these wider anchors. The
|
||||
# whitelist below is the runtime safety net for both paths.
|
||||
if self.matched_anchor.lower() not in (
|
||||
"has", "have", "had",
|
||||
"has", "have", "had", "started",
|
||||
"are", "were", "is", "was",
|
||||
"save", "saved",
|
||||
"earn", "earned",
|
||||
|
|
@ -159,9 +168,20 @@ _TRANSFER_VERBS_PATTERN: Final[str] = _verbs_pattern(TRANSFER_VERBS)
|
|||
# Initial-possession extractor
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# ADR-0131.G1 note: acquisition/action verbs (buys, bought, sells,
|
||||
# collected, saved, makes) were removed from the anchor alternation here.
|
||||
# They live exclusively in ADD_VERBS / SUBTRACT_VERBS so that sentences
|
||||
# like "Sam buys 3 apples" are parsed as add-operations only, avoiding
|
||||
# branch-disagreement when a canonical "has" initial precedes them.
|
||||
# The solver defaults-from-zero for operations, so single-statement
|
||||
# acquisition sentences ("Sam buys 5 apples. How many does Sam have?")
|
||||
# still resolve correctly as 0 + 5 = 5.
|
||||
_INITIAL_HAS_RE: Final[re.Pattern[str]] = re.compile(
|
||||
rf"^(?P<entity>{_ENTITY})\s+"
|
||||
rf"(?P<anchor>has|have)\s+"
|
||||
# ADR-0131.G.1: pure-possession anchors only (with optional particle
|
||||
# for "had started with N", etc.). Acquisition verbs live in
|
||||
# ADD_VERBS / SUBTRACT_VERBS — see CandidateInitial.__post_init__.
|
||||
rf"(?P<anchor>has|have|had|started)(?:\s+(?:up|with))?\s+"
|
||||
rf"(?P<value>{_VALUE})"
|
||||
# ADR-0131.G.3: unit slot is optional. Money-symbol value literals
|
||||
# (``$40``) carry their unit implicitly (``cent``); a missing unit
|
||||
|
|
@ -554,10 +574,17 @@ def _op_pattern(verbs_pattern: str, *, requires_target: bool) -> re.Pattern[str]
|
|||
trailing_prep = (
|
||||
r"(?:\s+(?:on|from|at|in|onto|into|under|over|to|of|for|with)\s+.+)?"
|
||||
)
|
||||
# Optional verb particle: handles "saved up N", "picked up N",
|
||||
# "threw out N", etc. The particle is grammatically real but
|
||||
# arithmetically inert — it does not affect the operation kind or
|
||||
# operand. ADR-0131.G1: this clause replaces the former approach
|
||||
# of listing particle-bearing verbs as initial-possession anchors.
|
||||
verb_particle = r"(?:\s+(?:up|down|out|back|off|in|away))?"
|
||||
return re.compile(
|
||||
r"^"
|
||||
rf"(?P<subject>{_ENTITY})\s+"
|
||||
rf"(?P<verb>{verbs_pattern})"
|
||||
rf"{verb_particle}"
|
||||
rf"\s+(?P<value>{_VALUE})"
|
||||
r"(?:\s+more)?"
|
||||
r"(?:\s+(?!to\b)(?!more\b)(?!on\b)(?!from\b)(?!at\b)(?!in\b)"
|
||||
|
|
@ -1040,13 +1067,10 @@ def _compare_multiplicative_candidates(sentence: str) -> list[CandidateOperation
|
|||
value_raw = m.group("value")
|
||||
if _is_indefinite_quantifier(value_raw):
|
||||
return out
|
||||
try:
|
||||
_rv = _resolve_value(value_raw)
|
||||
factor = float(_rv.value) if _rv is not None else None
|
||||
except (KeyError, TypeError):
|
||||
return out
|
||||
if factor is None:
|
||||
rv = _resolve_value(value_raw)
|
||||
if rv is None:
|
||||
return out
|
||||
factor = float(rv.value)
|
||||
cand = _build_compare_multiplicative(
|
||||
actor_raw=m.group("actor"),
|
||||
factor=factor,
|
||||
|
|
@ -1101,11 +1125,8 @@ def _compare_nested_candidates(sentence: str) -> list[CandidateOperation]:
|
|||
# comparison. The additive offset N is dropped on this candidate.
|
||||
factor_value_raw = m.group("factor_value")
|
||||
if not _is_indefinite_quantifier(factor_value_raw):
|
||||
try:
|
||||
_rv2 = _resolve_value(factor_value_raw)
|
||||
factor = float(_rv2.value) if _rv2 is not None else None
|
||||
except (KeyError, TypeError):
|
||||
factor = None
|
||||
rv = _resolve_value(factor_value_raw)
|
||||
factor = float(rv.value) if rv is not None else None
|
||||
if factor is not None:
|
||||
mult_cand = _build_compare_multiplicative(
|
||||
actor_raw=actor_raw,
|
||||
|
|
@ -1328,12 +1349,12 @@ def _conj_subject_each_candidates(sentence: str) -> list[CandidateInitial]:
|
|||
entity_b = _normalize_entity(m.group("b"))
|
||||
if entity_a == entity_b:
|
||||
return [] # 'Aaron and Aaron each ...' is degenerate
|
||||
_rv_conj = _resolve_value(value_raw)
|
||||
if _rv_conj is None:
|
||||
rv = _resolve_value(value_raw)
|
||||
if rv is None:
|
||||
return []
|
||||
value = _rv_conj.value
|
||||
value = rv.value
|
||||
unit_raw = m.group("unit")
|
||||
unit = _canonicalize_unit(unit_raw)
|
||||
unit = rv.unit_override if rv.unit_override is not None else _canonicalize_unit(unit_raw)
|
||||
anchor = _canon_verb_to_anchor(m.group("verb"))
|
||||
out: list[CandidateInitial] = []
|
||||
for entity, entity_raw in ((entity_a, m.group("a")), (entity_b, m.group("b"))):
|
||||
|
|
@ -1383,12 +1404,13 @@ def _conj_object_candidates(sentence: str) -> list[CandidateInitial]:
|
|||
rv = _resolve_value(value_raw)
|
||||
if rv is None:
|
||||
return []
|
||||
final_unit = rv.unit_override if rv.unit_override is not None else unit
|
||||
try:
|
||||
out.append(
|
||||
CandidateInitial(
|
||||
initial=InitialPossession(
|
||||
entity=entity,
|
||||
quantity=Quantity(value=rv.value, unit=unit),
|
||||
quantity=Quantity(value=rv.value, unit=final_unit),
|
||||
),
|
||||
source_span=sentence,
|
||||
matched_anchor=anchor,
|
||||
|
|
@ -1429,11 +1451,11 @@ def _embedded_quantifier_candidates(sentence: str) -> list[CandidateInitial]:
|
|||
c2 = container2_raw.lower()
|
||||
if c2 not in (container, container.rstrip("s"), container + "s"):
|
||||
return []
|
||||
_rv_n = _resolve_value(n_raw)
|
||||
_rv_per = _resolve_value(m_raw)
|
||||
if _rv_n is None or _rv_per is None:
|
||||
rv_n = _resolve_value(n_raw)
|
||||
rv_per = _resolve_value(m_raw)
|
||||
if rv_n is None or rv_per is None:
|
||||
return []
|
||||
total = _rv_n.value * _rv_per.value
|
||||
total = rv_n.value * rv_per.value
|
||||
entity = _normalize_entity(m.group("entity"))
|
||||
unit_raw = m.group("unit")
|
||||
unit = _canonicalize_unit(unit_raw)
|
||||
|
|
@ -1512,13 +1534,13 @@ def _build_conj_embedded_sum(
|
|||
if u1 != u2:
|
||||
# Mixed-unit sum is meaningless; refuse.
|
||||
return []
|
||||
_n1 = _resolve_value(n1_raw)
|
||||
_m1 = _resolve_value(m1_raw)
|
||||
_n2 = _resolve_value(n2_raw)
|
||||
_m2 = _resolve_value(m2_raw)
|
||||
if _n1 is None or _m1 is None or _n2 is None or _m2 is None:
|
||||
rv_n1 = _resolve_value(n1_raw)
|
||||
rv_m1 = _resolve_value(m1_raw)
|
||||
rv_n2 = _resolve_value(n2_raw)
|
||||
rv_m2 = _resolve_value(m2_raw)
|
||||
if any(rv is None for rv in (rv_n1, rv_m1, rv_n2, rv_m2)):
|
||||
return []
|
||||
total = _n1.value * _m1.value + _n2.value * _m2.value
|
||||
total = (rv_n1.value * rv_m1.value) + (rv_n2.value * rv_m2.value) # type: ignore[union-attr]
|
||||
entity = _normalize_entity(m.group("entity"))
|
||||
try:
|
||||
return [
|
||||
|
|
|
|||
86
tests/test_adr_0131_G1_verb_classes.py
Normal file
86
tests/test_adr_0131_G1_verb_classes.py
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
"""ADR-0131.G.1 — G1 state-introducing verb classes capability tests.
|
||||
|
||||
Checks the invariants, safety rail (wrong == 0), and verb coverage.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from evals.math_capability_axes.G1_verb_classes.v1.runner import build_report, _load_cases, _CASES_PATH
|
||||
from generate.math_candidate_parser import extract_initial_candidates, CandidateInitial
|
||||
from generate.math_candidate_graph import parse_and_solve
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def cases() -> list[dict]:
|
||||
return _load_cases(_CASES_PATH)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def report(cases) -> dict:
|
||||
return build_report(cases)
|
||||
|
||||
|
||||
class TestG1DatasetIntegrity:
|
||||
def test_case_ids_are_unique(self, cases) -> None:
|
||||
case_ids = [c["case_id"] for c in cases]
|
||||
assert len(case_ids) == len(set(case_ids))
|
||||
|
||||
def test_expected_outcomes_are_valid(self, cases) -> None:
|
||||
valid_outcomes = {"solved_correct", "solved_wrong", "refused"}
|
||||
for c in cases:
|
||||
assert c["expected"] in valid_outcomes
|
||||
|
||||
|
||||
class TestG1SafetyRail:
|
||||
def test_wrong_count_is_zero(self, report) -> None:
|
||||
assert report["counts"]["wrong"] == 0
|
||||
assert report["exit_criterion"]["passed"] is True
|
||||
|
||||
def test_every_verb_has_passing_case(self) -> None:
|
||||
verbs = [
|
||||
("Sam buys 5 apples. How many apples does Sam have?", 5.0, "apples"),
|
||||
("Sam bought 10 candies. How many candies does Sam have?", 10.0, "candies"),
|
||||
("Sam sells 3 books. Tom has 5 books. How many books does Tom have?", 5.0, "books"),
|
||||
("Sam collected 8 marbles. How many marbles does Sam have?", 8.0, "marbles"),
|
||||
# ADR-0131.G.3 integration: 'dollars'/'cents' surface units normalize to
|
||||
# canonical 'cents'. For verb-class coverage we use a non-money unit so
|
||||
# the test isolates the 'saved' verb axis from G.3 normalization.
|
||||
("Sam saved 12 marbles. How many marbles does Sam have?", 12.0, "marbles"),
|
||||
("Sam saved up 15 stickers. How many stickers does Sam have?", 15.0, "stickers"),
|
||||
("Sam started 20 pencils. How many pencils does Sam have?", 20.0, "pencils"),
|
||||
("Sam started with 6 toys. How many toys does Sam have?", 6.0, "toys"),
|
||||
("Sam had 9 dolls. How many dolls does Sam have?", 9.0, "dolls"),
|
||||
("Sam makes 4 cookies. How many cookies does Sam have?", 4.0, "cookies")
|
||||
]
|
||||
for problem, expected_val, expected_unit in verbs:
|
||||
res = parse_and_solve(problem)
|
||||
assert res.is_admitted, f"Failed to admit verb problem: {problem}"
|
||||
assert res.answer == expected_val
|
||||
assert res.selected_graph is not None
|
||||
assert res.selected_graph.unknown.unit == expected_unit
|
||||
|
||||
|
||||
class TestG1AdversarialProbes:
|
||||
def test_makes_rate_is_refused(self) -> None:
|
||||
# 'makes' in rate context ("Tina makes $18.00 an hour") must be refused
|
||||
problem = "Tina makes $18.00 an hour. How many dollars does Tina have?"
|
||||
res = parse_and_solve(problem)
|
||||
assert not res.is_admitted
|
||||
|
||||
def test_subjunctive_had_is_refused(self) -> None:
|
||||
# "If Sam had 5 apples" is hypothetical, must refuse
|
||||
problem = "If Sam had 5 apples, how many would he have?"
|
||||
res = parse_and_solve(problem)
|
||||
assert not res.is_admitted
|
||||
|
||||
|
||||
class TestG1ReplayDeterminism:
|
||||
def test_report_is_deterministic(self, cases) -> None:
|
||||
r1 = build_report(cases)
|
||||
r2 = build_report(cases)
|
||||
assert json.dumps(r1, sort_keys=True) == json.dumps(r2, sort_keys=True)
|
||||
Loading…
Reference in a new issue