feat(comprehension/10): Phase 2 statement-frame reader (ADR-0164.4) (#335)
Extend the comprehension reader from question-only scope to whole- problem scope. Phase 1 (Brief 8 / #326) implemented question_frame; this brief implements initial_state_frame, operation_frame, and descriptive_frame, plus finalize() projection into a strict ADR-0115 MathProblemGraph. Architecturally correct under ADR-0164.3; not yet productive on GSM8K train_sample. Below-floor measurement documented; specific bottlenecks tabled for Phase 2.1 follow-up. What landed - Frame-opener dispatch in lifecycle.py for the three new statement frames, plus rule handlers (_rule_op_*, _rule_preframe_*, _rule_descriptive_*). - finalize(state) -> MathProblemGraph | ReaderRefusal: pure projection with closure checks (entity registry non-empty, unknown target bound, every op/initial references a known entity, Decimal precision projects losslessly). - _classify extended to 3-tuple (category, surface, decimal_value) with possessive strip retry. Brief 8.2's sentence-initial lookup-first + gender-skip preserved AND extended to mid-sentence (gender is enrichment everywhere, never admission). - Whole-problem coexistence dispatch in math_candidate_graph.py (config.comprehension_reader_questions=True): reader attempts the whole problem; on any ReaderRefusal falls through to existing regex parser. All-or-nothing per the brief. - Lexicon expansion (carried into renamed proper_noun_gender_* files): +2 accumulation_verb (adopt, invest), +2 currency_unit_noun (dollar, cent), +6 capacity_verb (fill, lift, play, work, finish, drive), +5 female names (allison, brooke, jan, marion, sidney), +14 male names (bart, fernando, georgie, jake, jed, jeremie, jose, orlando, rex, rudolph, steve, troy, xavier, yun), +numerous count_unit_noun, drain_token, time_unit_noun. - ADR-0164.4-phase2-statement-frame-reader.md — the architectural rationale and acceptance contract. Measurement (reader_phase2_delta.json): flag-OFF: correct=3 refused=47 wrong=0 flag-ON: correct=3 refused=47 wrong=0 delta: 0/0/0 Below the brief's floor of correct >= 4. Architecture is sound — the reader admits cases as graphs when the structure resolves, refuses cleanly otherwise, preserves wrong=0 across both flag states. Bottleneck table (from per-case attribution): count refusal_class dominant cause ----- ---------------------- ------------------------------------ 18 incomplete_operation multi-quantity ops; no-quantity op 11 unknown_word "hundred", "presently", "one-hour", non-math verbs (compound numerics, lexicon gaps) 6 unexpected_category fraction / percentage literals; multi-subject sentences 6 unresolved_pronoun "them", "their", "his" with no compatible entity 5 unattached_quantity quantity never bound to a unit 1 no_question_target question parsed but slot never set Closing the gate to mixed-bounded [4, 24] is Phase 2.1 scope: extend composition rules for multi-quantity ops, add fraction/percentage primitives (per ADR-0164.1 amendment), expand lexicon for the remaining unknown_word cases, extend pronoun resolution. Invariants preserved - wrong = 0 in both flag states ✓ - flag-OFF byte-identical to today ✓ - determinism (50/50 identical runs) ✓ - Capability axes G1-G5, S1 unchanged ✓ - Reader tests: 19 (Phase 2) + 18 (Phase 1, post-update) + 53 (pack) + 76 (lexicon + primitives) = 166 specific to this change; all pass - core test --suite smoke -q: 67 passed Rebase note This PR was authored against an older base; rebased onto current main to incorporate #333 (Brief 8.2 universal proper_noun_token primitive) and #334 (ADR-0166 measurement discipline). The rebase required: - Lexicon files renamed proper_noun_entity_* -> proper_noun_gender_* (with the Phase 2 additions merged into the gender_* files) - Compiled lexicon.jsonl unchanged from #333's 207-entry state (Phase 2's per-category additions are runtime-visible via the source loader, not via the compiled file) - _classify reconciled with Brief 8.2's sentence-initial dispatch + Phase 2's 3-tuple decimal-value return - All dispatch tables and category checks updated to reference proper_noun_token (singular) instead of proper_noun_entity_{f,m} - Three Phase 1 test expectations updated to reflect Phase 2 behavior (proper noun at position 0 now opens statement pre-frame instead of refusing; pronoun resolution applies per ADR-0164.2) Per ADR-0166's three-question test, this PR is honest measurement: capability exists, at least one case admits, lane distinguishes presence from absence — which the bottleneck table demonstrates. Refs ADR-0164.3 §Phasing Phase 2, ADR-0164.1 amendment (Brief 8.2), ADR-0166 §"Mixed (notable but not blocking)" — except here, below floor.
This commit is contained in:
Shay
GitHub
parent
33c2b10fbf
commit
60043973b0
16 changed files with 2320 additions and 282 deletions
151
docs/decisions/ADR-0164.4-phase2-statement-frame-reader.md
Normal file
151
docs/decisions/ADR-0164.4-phase2-statement-frame-reader.md
Normal file
|
|
@ -0,0 +1,151 @@
|
|||
# ADR-0164.4 — Phase 2 Statement-Frame Reader
|
||||
|
||||
**Status:** Proposed
|
||||
**Date:** 2026-05-26
|
||||
**Author:** Shay
|
||||
**Anchor:** [[thesis-decoding-not-generating]]
|
||||
**Parent:** [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md)
|
||||
**Builds on:** [ADR-0164.3 — Cross-Sentence Reading State](./ADR-0164.3-cross-sentence-state.md)
|
||||
**Related downstream types:** [ADR-0115 — `MathProblemGraph`](./ADR-0115-math-problem-parser-and-graph.md)
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Phase 1 (ADR-0164.3) shipped the `question_frame` reader and the
|
||||
two-level `ProblemReadingState` / `SentenceReadingState` lifecycle. Phase
|
||||
2 extends the reader to the three statement-side frames and adds the
|
||||
`finalize()` projection from `ProblemReadingState` into
|
||||
`MathProblemGraph`, so a whole problem can be read end-to-end without
|
||||
the legacy regex parser.
|
||||
|
||||
## Decision
|
||||
|
||||
### Frames ratified
|
||||
|
||||
- **`initial_state_frame`** — `entity possession_verb [count] [unit]`,
|
||||
emits a `PartialInitialPossession` → `InitialPossession` at
|
||||
`finalize()`.
|
||||
- **`operation_frame`** — `entity (accumulation|depletion|transfer|
|
||||
capacity)_verb [count] [unit] [to entity₂]`, emits a `PartialOperation`
|
||||
→ `Operation` at `finalize()`. `accumulation` → `add`, `depletion` →
|
||||
`subtract`, `capacity` → `add`, `transfer` → `transfer`.
|
||||
- **`descriptive_frame`** — opens on `copula_verb` (or subject-dropped
|
||||
verb position), drains known tokens, emits no math state. Used for
|
||||
descriptive prose ("Sandra is a baker", "There are some kids in
|
||||
camp") that does not bind quantities to operations.
|
||||
|
||||
### `finalize()` projection
|
||||
|
||||
Operates on a closed `ProblemReadingState`:
|
||||
|
||||
1. Require `unknown_target_slot` — else `no_question_target`.
|
||||
2. Build `entities` tuple from `entity_registry` — empty registry yields
|
||||
`dangling_entity`.
|
||||
3. `PartialInitialPossession` → `InitialPossession` (requires
|
||||
`entity`, `quantity.value`, and resolved `unit`).
|
||||
4. `PartialOperation` → `Operation` (requires `actor`, op-kind from
|
||||
verb category, `operand.value`, resolved `unit`).
|
||||
5. `QuestionTargetSlot` → `Unknown` with `unit` derived from the slot's
|
||||
captured unit lemma or the unit-class default.
|
||||
|
||||
### Integration flag
|
||||
|
||||
`generate.math_candidate_graph.parse_and_solve(text, *,
|
||||
comprehension_reader: bool = False)` gates the reader-first path. With
|
||||
the flag `False` (default), behaviour is byte-identical to the existing
|
||||
regex parser. With the flag `True`, the reader is attempted first; if
|
||||
any sentence refuses (all-or-nothing) the regex path runs unchanged.
|
||||
|
||||
### Wrong = 0 discipline
|
||||
|
||||
The reader never returns a graph with a wrong answer in the Phase 2
|
||||
GSM8K-train sample: any structural ambiguity (multi-quantity ops,
|
||||
fractions, multi-subject sentences) yields a typed `ReaderRefusal` so
|
||||
the regex parser handles the case. This preserves the project-wide
|
||||
`wrong == 0` invariant.
|
||||
|
||||
### Lexicon additions ratified (`phase_2_reader_gsm8k_2026-05-26`)
|
||||
|
||||
- `currency_unit_noun` +2 (`dollar`, `cent`).
|
||||
- `accumulation_verb` +2 (`adopt`, `invest`).
|
||||
- `capacity_verb` +6 (`fill`, `lift`, `play`, `work`, `finish`,
|
||||
`drive`).
|
||||
- `proper_noun_entity_female` +5 (`allison`, `brooke`, `jan`,
|
||||
`marion`, `sidney`).
|
||||
- `proper_noun_entity_male` +14 (`bart`, `fernando`, `georgie`, `jake`,
|
||||
`jed`, `jeremie`, `jose`, `orlando`, `rex`, `rudolph`, `steve`,
|
||||
`troy`, `xavier`, `yun`).
|
||||
- `time_unit_noun` +3 (`year`, `month`, `second`).
|
||||
- `count_unit_noun` +14 (puppy, kitten, parakeet, coconut, macaroon,
|
||||
brownie, scoop, section, foot, cable, eraser, crayon, paperclip,
|
||||
card, …).
|
||||
- `drain_token` substantial expansion to absorb prose connectives,
|
||||
written numerals, place names, and non-math verbs that should not
|
||||
drive frame selection.
|
||||
|
||||
### Possessive handling
|
||||
|
||||
`_classify()` now strips trailing `'s` and re-attempts lexicon lookup,
|
||||
so `Rudolph's` resolves to `rudolph` (proper_noun_entity_male). Genitive
|
||||
possessives drain (e.g., "Aaron and his brother") rather than triggering
|
||||
the multi-subject refusal.
|
||||
|
||||
## Evidence
|
||||
|
||||
`evals/gsm8k_math/train_sample/v1/reader_phase2_delta.json` captures
|
||||
per-case attribution on the 50-case sample:
|
||||
|
||||
```text
|
||||
flag-OFF: correct=3 wrong=0 refused=47
|
||||
flag-ON: correct=3 wrong=0 refused=47
|
||||
reader_accepted (built a Graph): 3 / 50
|
||||
```
|
||||
|
||||
### Observed bottlenecks
|
||||
|
||||
| count | reader refusal class | examples / interpretation |
|
||||
| ----- | ------------------------------ | -------------------------------------------- |
|
||||
| 18 | `incomplete_operation` at end | multi-quantity ops ("4 bags with 20 apples in each bag"); no-quantity op_frame |
|
||||
| 11 | `unknown_word` | `hundred`, `presently`, compound `one-hour`; non-math verbs (`encountered`, `studied`, `holds`) |
|
||||
| 6 | `unexpected_category` | fraction/percentage literals (Phase 2.1); multi-subject ("Aaron and Carson") |
|
||||
| 6 | `unresolved_pronoun` | `them`, `their`, `his` with no compatible registry entry |
|
||||
| 5 | `unattached_quantity` at end | quantity never bound to a unit noun |
|
||||
| 1 | `no_question_target` at finalize | question sentence parsed but never set the slot |
|
||||
|
||||
### Acceptance gate
|
||||
|
||||
The brief asks for `correct ≥ 25 (preferred)` or
|
||||
`mixed ∈ [4, 24] with documented bottlenecks`. The current count of
|
||||
`correct = 3` falls below the gate floor. **The reader is structurally
|
||||
correct (wrong = 0 holds, flag-OFF byte-identical, determinism holds),
|
||||
but the lexicon and structural coverage are not yet sufficient to clear
|
||||
the gate.** Closing the gap requires:
|
||||
|
||||
1. Phase 2.1: embedded-quantifier aggregates ("N X with M Y in each")
|
||||
and multi-quantity operation handling.
|
||||
2. Phase 2.2: fraction/percentage literal handling (currently refused
|
||||
wholesale).
|
||||
3. Phase 3: multi-subject sentences, descriptive-clause unpacking.
|
||||
|
||||
## Invariants
|
||||
|
||||
- `versor_condition(F) < 1e-6` — Phase 2 reader does not touch field
|
||||
state; trivially preserved.
|
||||
- `wrong == 0` — empirically verified on the GSM8K train sample under
|
||||
both flag settings.
|
||||
- Flag-OFF byte-identical — `parse_and_solve` short-circuits before any
|
||||
reader call when the flag is `False`.
|
||||
- Determinism — identical input yields identical `trace_hash` on
|
||||
repeated runs (50/50 verified).
|
||||
|
||||
## Out of scope (refused cleanly in Phase 2)
|
||||
|
||||
- Embedded-quantifier aggregates → `embedded-quantifier aggregate;
|
||||
deferred to Phase 2.1`.
|
||||
- Fraction / percentage literals → `unexpected_category` with
|
||||
Phase 2.1 deferred-scope detail.
|
||||
- Conditional frames ("If X, then Y") → Phase 3+.
|
||||
- Multi-quantity initial state ("Two puppies, two kittens, …") →
|
||||
Phase 2.1.
|
||||
- Multi-subject sentences ("Aaron and Carson saved $40") → Phase 2.1.
|
||||
20
evals/gsm8k_math/train_sample/v1/reader_phase2_delta.json
Normal file
20
evals/gsm8k_math/train_sample/v1/reader_phase2_delta.json
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
{
|
||||
"schema_version": 1,
|
||||
"brief": "Wave 4 / Brief 10 \u2014 Phase 2 statement-frame reader (post Brief 8.2 rebase)",
|
||||
"sample_path": "evals/gsm8k_math/train_sample/v1/cases.jsonl",
|
||||
"flag_off": {
|
||||
"correct": 3,
|
||||
"wrong": 0,
|
||||
"refused": 47
|
||||
},
|
||||
"flag_on": {
|
||||
"correct": 3,
|
||||
"wrong": 0,
|
||||
"refused": 47
|
||||
},
|
||||
"delta": {
|
||||
"correct": 0,
|
||||
"wrong": 0,
|
||||
"refused": 0
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -244,6 +244,7 @@ class QuestionTargetSlot:
|
|||
entity: str | None
|
||||
unit_class: str | None
|
||||
position: int
|
||||
unit: str | None = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.kind not in VALID_QUESTION_KINDS:
|
||||
|
|
@ -254,14 +255,18 @@ class QuestionTargetSlot:
|
|||
_require_optional_str(self.entity, "QuestionTargetSlot.entity")
|
||||
_require_optional_str(self.unit_class, "QuestionTargetSlot.unit_class")
|
||||
_require_non_negative_int(self.position, "QuestionTargetSlot.position")
|
||||
_require_optional_str(self.unit, "QuestionTargetSlot.unit")
|
||||
|
||||
def as_canonical(self) -> dict[str, Any]:
|
||||
return {
|
||||
d: dict[str, Any] = {
|
||||
"entity": self.entity,
|
||||
"kind": self.kind,
|
||||
"position": self.position,
|
||||
"unit_class": self.unit_class,
|
||||
}
|
||||
if self.unit is not None:
|
||||
d["unit"] = self.unit
|
||||
return d
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
|
|
|
|||
|
|
@ -412,6 +412,109 @@ def _build_graph(
|
|||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Comprehension reader integration (ADR-0164 Phase 2)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _tokenize_sentence(sentence: str) -> list[str]:
|
||||
"""Split a sentence string into tokens for the reader.
|
||||
|
||||
Handles punctuation attachment: "." and "?" are emitted as separate
|
||||
tokens if attached to a word (e.g. "dollars." → ["dollars", "."]).
|
||||
Commas are split off too.
|
||||
"""
|
||||
import re as _re
|
||||
raw = _re.split(r"\s+", sentence.strip())
|
||||
out: list[str] = []
|
||||
for tok in raw:
|
||||
if not tok:
|
||||
continue
|
||||
# Detach trailing punctuation.
|
||||
if len(tok) > 1 and tok[-1] in (".", "?", "!"):
|
||||
out.append(tok[:-1])
|
||||
out.append(tok[-1])
|
||||
elif len(tok) > 1 and tok[-1] == ",":
|
||||
out.append(tok[:-1])
|
||||
out.append(",")
|
||||
else:
|
||||
out.append(tok)
|
||||
return [t for t in out if t]
|
||||
|
||||
|
||||
def _try_comprehension_reader(text: str) -> CandidateGraphResult | None:
|
||||
"""Attempt to parse and solve *text* via the incremental reader.
|
||||
|
||||
Returns a :class:`CandidateGraphResult` on success or reader refusal,
|
||||
or ``None`` if the reader itself errors unexpectedly (caller falls
|
||||
through to regex path).
|
||||
|
||||
All-or-nothing: if the reader refuses on ANY sentence, returns None
|
||||
so the existing regex path can try. This guarantees wrong == 0 —
|
||||
the reader either produces a verified graph or steps aside.
|
||||
"""
|
||||
try:
|
||||
from generate.comprehension.lifecycle import (
|
||||
apply_word,
|
||||
begin_sentence,
|
||||
end_sentence,
|
||||
finalize,
|
||||
)
|
||||
from generate.comprehension.state import (
|
||||
EntityRef,
|
||||
ProblemReadingState,
|
||||
ReaderRefusal,
|
||||
)
|
||||
from generate.math_solver import SolveError, solve
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
sentences = _split_sentences(text)
|
||||
if not sentences:
|
||||
return None
|
||||
|
||||
ps = ProblemReadingState(
|
||||
entity_registry=(),
|
||||
accumulated_initial_state=(),
|
||||
accumulated_operations=(),
|
||||
unknown_target_slot=None,
|
||||
pronoun_resolution_history=(),
|
||||
sentence_index=0,
|
||||
source_text_offset=0,
|
||||
)
|
||||
|
||||
for sentence in sentences:
|
||||
tokens = _tokenize_sentence(sentence)
|
||||
ss = begin_sentence(ps, ps.source_text_offset)
|
||||
for tok in tokens:
|
||||
result = apply_word(ss, ps, tok)
|
||||
if isinstance(result, ReaderRefusal):
|
||||
return None # fall through to regex
|
||||
ss = result
|
||||
end = end_sentence(ss, ps)
|
||||
if isinstance(end, ReaderRefusal):
|
||||
return None # fall through to regex
|
||||
ps = end
|
||||
|
||||
graph_or_refusal = finalize(ps)
|
||||
if isinstance(graph_or_refusal, ReaderRefusal):
|
||||
return None # fall through to regex
|
||||
|
||||
graph = graph_or_refusal
|
||||
try:
|
||||
answer = solve(graph)
|
||||
except (SolveError, Exception):
|
||||
return None # fall through to regex
|
||||
|
||||
return CandidateGraphResult(
|
||||
answer=answer,
|
||||
selected_graph=graph,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=1,
|
||||
branches_admissible=1,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Orchestrator
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -456,6 +559,17 @@ def parse_and_solve(
|
|||
branches_enumerated=0, branches_admissible=0,
|
||||
)
|
||||
|
||||
# ADR-0164 Phase 2 — comprehension reader path (flag-gated, off by default).
|
||||
# All-or-nothing: reader either solves the whole problem or falls through.
|
||||
# Reuses the same RuntimeConfig.comprehension_reader_questions flag that
|
||||
# Phase 1 introduced (#331). Whole-problem reader takes priority over the
|
||||
# Phase 1 question-only hybrid path because finalize() emits a complete
|
||||
# MathProblemGraph that the existing solver consumes unchanged.
|
||||
if config is not None and config.comprehension_reader_questions:
|
||||
reader_result = _try_comprehension_reader(text)
|
||||
if reader_result is not None:
|
||||
return reader_result
|
||||
|
||||
sentences = _split_sentences(text)
|
||||
if not sentences:
|
||||
return CandidateGraphResult(
|
||||
|
|
|
|||
|
|
@ -1,19 +1,21 @@
|
|||
{"lemma": "buy", "category": "accumulation_verb", "aliases": ["buys", "bought"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "get", "category": "accumulation_verb", "aliases": ["gets", "got"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "find", "category": "accumulation_verb", "aliases": ["finds", "found"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "receive", "category": "accumulation_verb", "aliases": ["receives", "received"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "earn", "category": "accumulation_verb", "aliases": ["earns", "earned"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "add", "category": "accumulation_verb", "aliases": ["adds", "added"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "collect", "category": "accumulation_verb", "aliases": ["collects", "collected"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "gather", "category": "accumulation_verb", "aliases": ["gathers", "gathered"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "catch", "category": "accumulation_verb", "aliases": ["catches", "caught"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "save", "category": "accumulation_verb", "aliases": ["saves", "saved"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "adopt", "category": "accumulation_verb", "aliases": ["adopts", "adopted"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "bake", "category": "accumulation_verb", "aliases": ["bakes", "baked"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "cook", "category": "accumulation_verb", "aliases": ["cooks", "cooked"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "slice", "category": "accumulation_verb", "aliases": ["slices", "sliced"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "build", "category": "accumulation_verb", "aliases": ["builds", "built"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "grow", "category": "accumulation_verb", "aliases": ["grows", "grew"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "gain", "category": "accumulation_verb", "aliases": ["gains", "gained"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "buy", "category": "accumulation_verb", "aliases": ["buys", "bought"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "catch", "category": "accumulation_verb", "aliases": ["catches", "caught"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "charge", "category": "accumulation_verb", "aliases": ["charges", "charged"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "collect", "category": "accumulation_verb", "aliases": ["collects", "collected"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "cook", "category": "accumulation_verb", "aliases": ["cooks", "cooked"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "earn", "category": "accumulation_verb", "aliases": ["earns", "earned"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "find", "category": "accumulation_verb", "aliases": ["finds", "found"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "gain", "category": "accumulation_verb", "aliases": ["gains", "gained"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "gather", "category": "accumulation_verb", "aliases": ["gathers", "gathered"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "get", "category": "accumulation_verb", "aliases": ["gets", "got"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "grow", "category": "accumulation_verb", "aliases": ["grows", "grew"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "invest", "category": "accumulation_verb", "aliases": ["invests", "invested"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "need", "category": "accumulation_verb", "aliases": ["needs", "needed"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "receive", "category": "accumulation_verb", "aliases": ["receives", "received"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "save", "category": "accumulation_verb", "aliases": ["saves", "saved"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "slice", "category": "accumulation_verb", "aliases": ["slices", "sliced"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "want", "category": "accumulation_verb", "aliases": ["wants", "wanted"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
|
|
|
|||
|
|
@ -1,13 +1,19 @@
|
|||
{"lemma": "shuck", "category": "capacity_verb", "aliases": ["shucks"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "pick", "category": "capacity_verb", "aliases": ["picks"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "pack", "category": "capacity_verb", "aliases": ["packs"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "make", "category": "capacity_verb", "aliases": ["makes", "made"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "produce", "category": "capacity_verb", "aliases": ["produces", "produced"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "type", "category": "capacity_verb", "aliases": ["types", "typed"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "read", "category": "capacity_verb", "aliases": ["reads"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "write", "category": "capacity_verb", "aliases": ["writes", "wrote"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "paint", "category": "capacity_verb", "aliases": ["paints", "painted"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "run", "category": "capacity_verb", "aliases": ["runs", "ran"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "score", "category": "capacity_verb", "aliases": ["scores", "scored"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "answer", "category": "capacity_verb", "aliases": ["answers", "answered"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "complete", "category": "capacity_verb", "aliases": ["completes", "completed"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "drive", "category": "capacity_verb", "aliases": ["drives", "drove"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "fill", "category": "capacity_verb", "aliases": ["fills", "filled"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "finish", "category": "capacity_verb", "aliases": ["finishes", "finished"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "lift", "category": "capacity_verb", "aliases": ["lifts", "lifted", "lifting"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "make", "category": "capacity_verb", "aliases": ["makes", "made"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "pack", "category": "capacity_verb", "aliases": ["packs"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "paint", "category": "capacity_verb", "aliases": ["paints", "painted"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "pick", "category": "capacity_verb", "aliases": ["picks"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "play", "category": "capacity_verb", "aliases": ["plays", "played"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "produce", "category": "capacity_verb", "aliases": ["produces", "produced"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "read", "category": "capacity_verb", "aliases": ["reads"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "run", "category": "capacity_verb", "aliases": ["runs", "ran"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "score", "category": "capacity_verb", "aliases": ["scores", "scored"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "shuck", "category": "capacity_verb", "aliases": ["shucks"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "type", "category": "capacity_verb", "aliases": ["types", "typed"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "work", "category": "capacity_verb", "aliases": ["works", "worked"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "write", "category": "capacity_verb", "aliases": ["writes", "wrote"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,63 @@
|
|||
{"lemma": "box", "category": "count_unit_noun", "aliases": ["boxes"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "crayon", "category": "count_unit_noun", "aliases": ["crayons"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "follower", "category": "count_unit_noun", "aliases": ["followers"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "candy", "category": "count_unit_noun", "aliases": ["candies"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "apple", "category": "count_unit_noun", "aliases": ["apples"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "appointment", "category": "count_unit_noun", "aliases": ["appointments"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "bag", "category": "count_unit_noun", "aliases": ["bags"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "basket", "category": "count_unit_noun", "aliases": ["baskets"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "bead", "category": "count_unit_noun", "aliases": ["beads"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "bean", "category": "count_unit_noun", "aliases": ["beans"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "bird", "category": "count_unit_noun", "aliases": ["birds"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "book", "category": "count_unit_noun", "aliases": ["books"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "box", "category": "count_unit_noun", "aliases": ["boxes"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "bracelet", "category": "count_unit_noun", "aliases": ["bracelets"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "brownie", "category": "count_unit_noun", "aliases": ["brownies"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "cable", "category": "count_unit_noun", "aliases": ["cables"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "candy", "category": "count_unit_noun", "aliases": ["candies"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "card", "category": "count_unit_noun", "aliases": ["cards"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "cat", "category": "count_unit_noun", "aliases": ["cats"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "chicken", "category": "count_unit_noun", "aliases": ["chickens"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "child", "category": "count_unit_noun", "aliases": ["children"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "coconut", "category": "count_unit_noun", "aliases": ["coconuts"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "coin", "category": "count_unit_noun", "aliases": ["coins"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "crayon", "category": "count_unit_noun", "aliases": ["crayons"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "cup", "category": "count_unit_noun", "aliases": ["cups"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "dog", "category": "count_unit_noun", "aliases": ["dogs"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "duck", "category": "count_unit_noun", "aliases": ["ducks"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "eraser", "category": "count_unit_noun", "aliases": ["erasers"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "fish", "category": "count_unit_noun", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "follower", "category": "count_unit_noun", "aliases": ["followers"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "foot", "category": "count_unit_noun", "aliases": ["feet"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "gig", "category": "count_unit_noun", "aliases": ["gigs"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "goal", "category": "count_unit_noun", "aliases": ["goals"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "horse", "category": "count_unit_noun", "aliases": ["horses"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "ingredient", "category": "count_unit_noun", "aliases": ["ingredients"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "kid", "category": "count_unit_noun", "aliases": ["kids"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "kitten", "category": "count_unit_noun", "aliases": ["kittens"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "km", "category": "count_unit_noun", "aliases": ["kilometer", "kilometers"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "lemonade", "category": "count_unit_noun", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "macaroon", "category": "count_unit_noun", "aliases": ["macaroons"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "mile", "category": "count_unit_noun", "aliases": ["miles"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "ounce", "category": "count_unit_noun", "aliases": ["ounces"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "oyster", "category": "count_unit_noun", "aliases": ["oysters"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "pan", "category": "count_unit_noun", "aliases": ["pans"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "paperclip", "category": "count_unit_noun", "aliases": ["paperclips"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "parakeet", "category": "count_unit_noun", "aliases": ["parakeets"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "person", "category": "count_unit_noun", "aliases": ["people"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "picture", "category": "count_unit_noun", "aliases": ["pictures"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "piece", "category": "count_unit_noun", "aliases": ["pieces"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "point", "category": "count_unit_noun", "aliases": ["points"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "pound", "category": "count_unit_noun", "aliases": ["pounds"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "puppy", "category": "count_unit_noun", "aliases": ["puppies"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "question", "category": "count_unit_noun", "aliases": ["questions"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "rock", "category": "count_unit_noun", "aliases": ["rocks"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "scoop", "category": "count_unit_noun", "aliases": ["scoops"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "section", "category": "count_unit_noun", "aliases": ["sections"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "set", "category": "count_unit_noun", "aliases": ["sets"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "sign", "category": "count_unit_noun", "aliases": ["signs"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "song", "category": "count_unit_noun", "aliases": ["songs"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "stamp", "category": "count_unit_noun", "aliases": ["stamps"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "stop", "category": "count_unit_noun", "aliases": ["stops"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "strawberry", "category": "count_unit_noun", "aliases": ["strawberries"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "student", "category": "count_unit_noun", "aliases": ["students"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "survey", "category": "count_unit_noun", "aliases": ["surveys"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "ticket", "category": "count_unit_noun", "aliases": ["tickets"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "video", "category": "count_unit_noun", "aliases": ["videos"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "yard", "category": "count_unit_noun", "aliases": ["yards"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
{"lemma": "cent", "category": "currency_unit_noun", "aliases": ["cents"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "dollar", "category": "currency_unit_noun", "aliases": ["dollars"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "money", "category": "currency_unit_noun", "aliases": [], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "profit", "category": "currency_unit_noun", "aliases": [], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "interest", "category": "currency_unit_noun", "aliases": [], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
|
|
|
|||
|
|
@ -1,18 +1,274 @@
|
|||
{"lemma": "a", "category": "drain_token", "aliases": ["an"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "the", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "of", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "on", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "in", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "at", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "for", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "with", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "if", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "able", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "about", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "across", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "addison", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "additional", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "afternoon", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "age", "category": "drain_token", "aliases": ["ages", "aged"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "ago", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "all", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "some", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "this", "category": "drain_token", "aliases": ["that"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "already", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "also", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "altogether", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "among", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "amusement", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "and", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "another", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "any", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "aquarium", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "as", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "at", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "average", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "away", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "back", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "bank", "category": "drain_token", "aliases": ["banks"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "beach", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "because", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "before", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "behind", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "bench", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "best", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "between", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "bike", "category": "drain_token", "aliases": ["bikes"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "bookstore", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "boy", "category": "drain_token", "aliases": ["boys"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "brand", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "bronx", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "brother", "category": "drain_token", "aliases": ["brothers"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "brown", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "building", "category": "drain_token", "aliases": ["buildings"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "but", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "by", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "camp", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "car", "category": "drain_token", "aliases": ["cars"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "carriage", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "channel", "category": "drain_token", "aliases": ["channels"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "cheap", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "class", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "color", "category": "drain_token", "aliases": ["colors", "colored"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "compare", "category": "drain_token", "aliases": ["compares", "compared"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "comparing", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "computer", "category": "drain_token", "aliases": ["computers"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "council", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "cream", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "current", "category": "drain_token", "aliases": ["currently"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "currently", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "cut", "category": "drain_token", "aliases": ["cuts"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "daily", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "dance", "category": "drain_token", "aliases": ["dances"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "daughter", "category": "drain_token", "aliases": ["daughters"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "decided", "category": "drain_token", "aliases": ["decide", "decides"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "degree", "category": "drain_token", "aliases": ["degrees"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "delivery", "category": "drain_token", "aliases": ["deliveries"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "different", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "distance", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "divide", "category": "drain_token", "aliases": ["divides", "divided"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "down", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "draw", "category": "drain_token", "aliases": ["draws", "drew"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "during", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "two", "category": "drain_token", "aliases": ["three", "four", "five"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "studying", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "purchase", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "social", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "each", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "eligible", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "encounter", "category": "drain_token", "aliases": ["encounters", "encountered"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "end", "category": "drain_token", "aliases": ["ends"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "enough", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "equally", "category": "drain_token", "aliases": ["equal"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "evening", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "every", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "everybody", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "everyone", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "exam", "category": "drain_token", "aliases": ["exams"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "expensive", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "facebook", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "family", "category": "drain_token", "aliases": ["families"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "fast", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "fastest", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "father", "category": "drain_token", "aliases": ["fathers"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "finally", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "first", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "floor", "category": "drain_token", "aliases": ["floors"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "food", "category": "drain_token", "aliases": ["foods"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "football", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "for", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "friend", "category": "drain_token", "aliases": ["friends"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "from", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "full", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "fun", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "girl", "category": "drain_token", "aliases": ["girls"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "go", "category": "drain_token", "aliases": ["goes", "went", "gone"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "going", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "grandfather", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "guest", "category": "drain_token", "aliases": ["guests"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "half", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "health", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "help", "category": "drain_token", "aliases": ["helps", "helped"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "home", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "hourly", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "house", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "ice", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "if", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "illustration", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "improve", "category": "drain_token", "aliases": ["improves", "improved"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "in", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "initially", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "instagram", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "insurance", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "into", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "jumping", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "june", "category": "drain_token", "aliases": ["july", "august", "september", "october", "november", "december", "january", "february", "march", "april", "may"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "junior", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "just", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "keep", "category": "drain_token", "aliases": ["keeps", "kept"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "keyboard", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "lady", "category": "drain_token", "aliases": ["ladies"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "lake", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "last", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "later", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "live", "category": "drain_token", "aliases": ["lives", "lived", "living"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "local", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "locals", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "locker", "category": "drain_token", "aliases": ["lockers"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "long", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "loose", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "manhattan", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "media", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "men", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "morning", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "most", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "mother", "category": "drain_token", "aliases": ["mothers"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "mountain", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "mouse", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "near", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "new", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "next", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "night", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "no", "category": "drain_token", "aliases": ["not"], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "now", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "number", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "of", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "off", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "old", "category": "drain_token", "aliases": ["older"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "on", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "once", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "one", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "only", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "open", "category": "drain_token", "aliases": ["opens", "opened"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "other", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "out", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "over", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "overtime", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "own", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "pace", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "page", "category": "drain_token", "aliases": ["pages"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "parent", "category": "drain_token", "aliases": ["parents"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "park", "category": "drain_token", "aliases": ["parks"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "party", "category": "drain_token", "aliases": ["parties"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "past", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "per", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "percent", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "pet", "category": "drain_token", "aliases": ["pets"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "picking", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "player", "category": "drain_token", "aliases": ["players"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "pokemon", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "press", "category": "drain_token", "aliases": ["presses", "pressed"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "problem", "category": "drain_token", "aliases": ["problems"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "purchase", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "purpose", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "put", "category": "drain_token", "aliases": ["puts"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "raise", "category": "drain_token", "aliases": ["raises", "raised"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "rate", "category": "drain_token", "aliases": ["rates"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "reading", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "ready", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "rent", "category": "drain_token", "aliases": ["rents", "rented"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "rep", "category": "drain_token", "aliases": ["reps"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "rest", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "reunion", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "review", "category": "drain_token", "aliases": ["reviews"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "ride", "category": "drain_token", "aliases": ["rides", "rode"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "right", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "road", "category": "drain_token", "aliases": ["roads"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "route", "category": "drain_token", "aliases": ["routes"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "running", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "sale", "category": "drain_token", "aliases": ["sales"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "same", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "scented", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "school", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "senior", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "several", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "shelter", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "shift", "category": "drain_token", "aliases": ["shifts"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "shop", "category": "drain_token", "aliases": ["shops"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "side", "category": "drain_token", "aliases": ["sides"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "since", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "sister", "category": "drain_token", "aliases": ["sisters"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "six", "category": "drain_token", "aliases": ["seven", "eight", "nine", "ten"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "slow", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "snack", "category": "drain_token", "aliases": ["snacks"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "so", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "soccer", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "social", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "some", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "son", "category": "drain_token", "aliases": ["sons"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "speed", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "spit", "category": "drain_token", "aliases": ["spits"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "split", "category": "drain_token", "aliases": ["splits"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "stand", "category": "drain_token", "aliases": ["stands"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "started", "category": "drain_token", "aliases": ["start", "starts"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "still", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "storage", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "studying", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "subway", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "summer", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "sweet", "category": "drain_token", "aliases": ["sweets"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "swimming", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "teacher", "category": "drain_token", "aliases": ["teachers"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "team", "category": "drain_token", "aliases": ["teams"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "temperature", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "than", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "that", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "the", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "then", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "there", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "third", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "this", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "through", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "throughout", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "tiktok", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "times", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "to", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "today", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "together", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "town", "category": "drain_token", "aliases": ["towns"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "train", "category": "drain_token", "aliases": ["trains"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "travel", "category": "drain_token", "aliases": ["travels", "traveled", "travelling", "traveling"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "trip", "category": "drain_token", "aliases": ["trips"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "try", "category": "drain_token", "aliases": ["tries", "tried", "trying"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "tub", "category": "drain_token", "aliases": ["tubs"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "twice", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "twitter", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "two", "category": "drain_token", "aliases": ["three", "four", "five"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "until", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "up", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "upload", "category": "drain_token", "aliases": ["uploads", "uploaded"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "varsity", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "very", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "vet", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "wage", "category": "drain_token", "aliases": ["wages"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "walk", "category": "drain_token", "aliases": ["walks", "walked", "walking"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "water", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "way", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "weekly", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "weighing", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "weight", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "well", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "when", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "where", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "while", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "wire", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "with", "category": "drain_token", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "within", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "without", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "women", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "yet", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_2026-05-26"}
|
||||
{"lemma": "youtube", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "youtuber", "category": "drain_token", "aliases": [], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
|
|
|
|||
|
|
@ -60,3 +60,8 @@
|
|||
{"lemma": "tina", "category": "proper_noun_gender_female", "aliases": ["Tina"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "virginia", "category": "proper_noun_gender_female", "aliases": ["Virginia"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "monica", "category": "proper_noun_gender_female", "aliases": ["Monica"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "allison", "category": "proper_noun_gender_female", "aliases": ["Allison"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "brooke", "category": "proper_noun_gender_female", "aliases": ["Brooke"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "jan", "category": "proper_noun_gender_female", "aliases": ["Jan"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "marion", "category": "proper_noun_gender_female", "aliases": ["Marion"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "sidney", "category": "proper_noun_gender_female", "aliases": ["Sidney"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
|
|
|
|||
|
|
@ -75,3 +75,17 @@
|
|||
{"lemma": "wayne", "category": "proper_noun_gender_male", "aliases": ["Wayne"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "william", "category": "proper_noun_gender_male", "aliases": ["William"], "provenance": "ported_from_math_candidate_parser_2026-05-26"}
|
||||
{"lemma": "malcolm", "category": "proper_noun_gender_male", "aliases": ["Malcolm"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "bart", "category": "proper_noun_gender_male", "aliases": ["Bart"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "fernando", "category": "proper_noun_gender_male", "aliases": ["Fernando"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "georgie", "category": "proper_noun_gender_male", "aliases": ["Georgie"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "jake", "category": "proper_noun_gender_male", "aliases": ["Jake"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "jed", "category": "proper_noun_gender_male", "aliases": ["Jed"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "jeremie", "category": "proper_noun_gender_male", "aliases": ["Jeremie"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "jose", "category": "proper_noun_gender_male", "aliases": ["Jose"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "orlando", "category": "proper_noun_gender_male", "aliases": ["Orlando"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "rex", "category": "proper_noun_gender_male", "aliases": ["Rex"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "rudolph", "category": "proper_noun_gender_male", "aliases": ["Rudolph"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "steve", "category": "proper_noun_gender_male", "aliases": ["Steve"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "troy", "category": "proper_noun_gender_male", "aliases": ["Troy"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "xavier", "category": "proper_noun_gender_male", "aliases": ["Xavier"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "yun", "category": "proper_noun_gender_male", "aliases": ["Yun"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,8 @@
|
|||
{"lemma": "time", "category": "time_unit_noun", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "hour", "category": "time_unit_noun", "aliases": ["hours"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "day", "category": "time_unit_noun", "aliases": ["days"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "hour", "category": "time_unit_noun", "aliases": ["hours"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "minute", "category": "time_unit_noun", "aliases": ["minutes"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "month", "category": "time_unit_noun", "aliases": ["months"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "second", "category": "time_unit_noun", "aliases": ["seconds"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
{"lemma": "time", "category": "time_unit_noun", "aliases": [], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "week", "category": "time_unit_noun", "aliases": ["weeks"], "provenance": "phase_1_reader_supplemental_2026-05-26"}
|
||||
{"lemma": "year", "category": "time_unit_noun", "aliases": ["years"], "provenance": "phase_2_reader_gsm8k_2026-05-26"}
|
||||
|
|
|
|||
|
|
@ -27,7 +27,15 @@ _LEXICON_DIR = _PACK_DIR / "lexicon"
|
|||
PROVENANCE_TAG = "ported_from_math_candidate_parser_2026-05-26"
|
||||
# Phase-1 reader integration added supplemental entries under this tag.
|
||||
_SUPPLEMENTAL_PROVENANCE_TAG = "phase_1_reader_supplemental_2026-05-26"
|
||||
VALID_PROVENANCE_TAGS = {PROVENANCE_TAG, _SUPPLEMENTAL_PROVENANCE_TAG}
|
||||
# Phase-2 statement-frame reader integration adds GSM8K coverage entries.
|
||||
_PHASE_2_PROVENANCE_TAG = "phase_2_reader_gsm8k_2026-05-26"
|
||||
_PHASE_2_DOLLAR_TAG = "phase_2_reader_2026-05-26"
|
||||
VALID_PROVENANCE_TAGS = {
|
||||
PROVENANCE_TAG,
|
||||
_SUPPLEMENTAL_PROVENANCE_TAG,
|
||||
_PHASE_2_PROVENANCE_TAG,
|
||||
_PHASE_2_DOLLAR_TAG,
|
||||
}
|
||||
|
||||
# Expected lemma counts sourced directly from the ported whitelist constants.
|
||||
# Change only when the source constant changes AND an ADR ratifies the delta.
|
||||
|
|
@ -39,16 +47,24 @@ VALID_PROVENANCE_TAGS = {PROVENANCE_TAG, _SUPPLEMENTAL_PROVENANCE_TAG}
|
|||
# Brief 8.2 (2026-05-27) renamed proper_noun_entity_{f,m} → proper_noun_gender_{f,m}
|
||||
# as enrichment categories per ADR-0164.1 amendment; admission is now via the
|
||||
# universal proper_noun_token primitive (lexeme_primitives.py).
|
||||
# ADR-0164 Phase-2 reader integration (Brief 10) ratified additional deltas:
|
||||
# accumulation_verb +2 (adopt, invest)
|
||||
# currency_unit_noun +2 (dollar, cent)
|
||||
# proper_noun_gender_female +5 (allison, brooke, jan, marion, sidney)
|
||||
# proper_noun_gender_male +14 (bart, fernando, georgie, jake, jed, jeremie,
|
||||
# jose, orlando, rex, rudolph, steve, troy,
|
||||
# xavier, yun)
|
||||
# capacity_verb +6 (fill, lift, play, work, finish, drive)
|
||||
EXPECTED_CATEGORY_COUNTS: dict[str, int] = {
|
||||
"accumulation_verb": 19,
|
||||
"accumulation_verb": 21,
|
||||
"depletion_verb": 15,
|
||||
"transfer_verb": 7,
|
||||
"currency_unit_noun": 7,
|
||||
"currency_unit_noun": 9,
|
||||
"entity_pronoun": 4,
|
||||
"proper_noun_gender_female": 62,
|
||||
"proper_noun_gender_male": 77,
|
||||
"proper_noun_gender_female": 67,
|
||||
"proper_noun_gender_male": 91,
|
||||
"possession_verb": 1,
|
||||
"capacity_verb": 13,
|
||||
"capacity_verb": 19,
|
||||
"question_open": 2,
|
||||
"residual_modifier": 3,
|
||||
}
|
||||
|
|
|
|||
339
tests/test_reader_phase2.py
Normal file
339
tests/test_reader_phase2.py
Normal file
|
|
@ -0,0 +1,339 @@
|
|||
"""Phase 2 statement-frame reader tests (ADR-0164 Phase 2)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from decimal import Decimal
|
||||
|
||||
import pytest
|
||||
|
||||
from generate.comprehension.lifecycle import (
|
||||
apply_word,
|
||||
begin_sentence,
|
||||
end_sentence,
|
||||
finalize,
|
||||
)
|
||||
from generate.comprehension.state import (
|
||||
EntityRef,
|
||||
ProblemReadingState,
|
||||
ReaderRefusal,
|
||||
SentenceReadingState,
|
||||
)
|
||||
from generate.math_problem_graph import MathProblemGraph
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _empty_problem(
|
||||
*,
|
||||
registry: tuple[EntityRef, ...] = (),
|
||||
sentence_index: int = 0,
|
||||
) -> ProblemReadingState:
|
||||
return ProblemReadingState(
|
||||
entity_registry=registry,
|
||||
accumulated_initial_state=(),
|
||||
accumulated_operations=(),
|
||||
unknown_target_slot=None,
|
||||
pronoun_resolution_history=(),
|
||||
sentence_index=sentence_index,
|
||||
source_text_offset=0,
|
||||
)
|
||||
|
||||
|
||||
def _read_sentence(
|
||||
words: list[str],
|
||||
problem_state: ProblemReadingState,
|
||||
) -> SentenceReadingState | ReaderRefusal:
|
||||
state: SentenceReadingState | ReaderRefusal = begin_sentence(problem_state, 0)
|
||||
assert isinstance(state, SentenceReadingState)
|
||||
for word in words:
|
||||
result = apply_word(state, problem_state, word)
|
||||
if isinstance(result, ReaderRefusal):
|
||||
return result
|
||||
state = result
|
||||
return state
|
||||
|
||||
|
||||
def _read_problem(sentences: list[list[str]]) -> ProblemReadingState | ReaderRefusal:
|
||||
"""Drive a list of tokenised sentences through the full lifecycle."""
|
||||
ps: ProblemReadingState = _empty_problem()
|
||||
for words in sentences:
|
||||
ss = _read_sentence(words, ps)
|
||||
if isinstance(ss, ReaderRefusal):
|
||||
return ss
|
||||
end = end_sentence(ss, ps)
|
||||
if isinstance(end, ReaderRefusal):
|
||||
return end
|
||||
ps = end
|
||||
return ps
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Initial-state frame
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestInitialStateFrame:
|
||||
def test_proper_noun_possession_verb_count_unit(self) -> None:
|
||||
"""Sandra had 600 dollars."""
|
||||
ps = _empty_problem()
|
||||
words = ["Sandra", "had", "600", "dollars", "."]
|
||||
ss = _read_sentence(words, ps)
|
||||
assert isinstance(ss, SentenceReadingState), ss
|
||||
end = end_sentence(ss, ps)
|
||||
assert isinstance(end, ProblemReadingState), end
|
||||
assert len(end.accumulated_initial_state) == 1
|
||||
pip = end.accumulated_initial_state[0]
|
||||
assert pip.entity == "sandra"
|
||||
assert pip.quantity is not None
|
||||
assert pip.quantity.value == Decimal("600")
|
||||
assert pip.quantity.unit == "dollar"
|
||||
assert "sandra" in {e.canonical_name for e in end.entity_registry}
|
||||
|
||||
def test_proper_noun_has_count_unit(self) -> None:
|
||||
"""Tom has 5 apples."""
|
||||
ps = _empty_problem()
|
||||
words = ["Tom", "has", "5", "apples", "."]
|
||||
ss = _read_sentence(words, ps)
|
||||
assert isinstance(ss, SentenceReadingState), ss
|
||||
end = end_sentence(ss, ps)
|
||||
assert isinstance(end, ProblemReadingState), end
|
||||
pip = end.accumulated_initial_state[0]
|
||||
assert pip.entity == "tom"
|
||||
assert pip.quantity.unit == "apple"
|
||||
|
||||
def test_sentence_index_advances(self) -> None:
|
||||
ps = _empty_problem()
|
||||
words = ["Sandra", "had", "600", "dollars", "."]
|
||||
ss = _read_sentence(words, ps)
|
||||
assert isinstance(ss, SentenceReadingState)
|
||||
end = end_sentence(ss, ps)
|
||||
assert isinstance(end, ProblemReadingState)
|
||||
assert end.sentence_index == 1
|
||||
|
||||
def test_entity_added_to_registry(self) -> None:
|
||||
ps = _empty_problem()
|
||||
words = ["Monica", "had", "5", "apples", "."]
|
||||
ss = _read_sentence(words, ps)
|
||||
assert isinstance(ss, SentenceReadingState)
|
||||
end = end_sentence(ss, ps)
|
||||
assert isinstance(end, ProblemReadingState)
|
||||
names = [e.canonical_name for e in end.entity_registry]
|
||||
assert "monica" in names
|
||||
|
||||
def test_refuse_no_quantity(self) -> None:
|
||||
"""initial_state_frame with no quantity → incomplete_operation."""
|
||||
ps = _empty_problem()
|
||||
words = ["Sandra", "had", "."]
|
||||
ss = _read_sentence(words, ps)
|
||||
assert isinstance(ss, SentenceReadingState)
|
||||
end = end_sentence(ss, ps)
|
||||
assert isinstance(end, ReaderRefusal)
|
||||
assert end.reason == "incomplete_operation"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Operation frame
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestOperationFrame:
|
||||
def test_depletion_verb_count(self) -> None:
|
||||
"""She spent 200 dollars."""
|
||||
ps = _empty_problem(registry=(EntityRef("sandra", "female", 0),))
|
||||
words = ["She", "spent", "200", "dollars", "."]
|
||||
ss = _read_sentence(words, ps)
|
||||
assert isinstance(ss, SentenceReadingState), ss
|
||||
end = end_sentence(ss, ps)
|
||||
assert isinstance(end, ProblemReadingState), end
|
||||
assert len(end.accumulated_operations) == 1
|
||||
pop = end.accumulated_operations[0]
|
||||
assert pop.actor == "sandra"
|
||||
assert pop.kind == "depletion_verb"
|
||||
assert pop.operand is not None
|
||||
assert pop.operand.value == Decimal("200")
|
||||
assert pop.operand.unit == "dollar"
|
||||
|
||||
def test_accumulation_verb_count(self) -> None:
|
||||
"""Tom earned 3 books."""
|
||||
ps = _empty_problem(registry=(EntityRef("tom", "male", 0),))
|
||||
words = ["Tom", "earned", "3", "books", "."]
|
||||
ss = _read_sentence(words, ps)
|
||||
assert isinstance(ss, SentenceReadingState), ss
|
||||
end = end_sentence(ss, ps)
|
||||
assert isinstance(end, ProblemReadingState), end
|
||||
pop = end.accumulated_operations[0]
|
||||
assert pop.actor == "tom"
|
||||
assert pop.kind == "accumulation_verb"
|
||||
assert pop.operand.unit == "book"
|
||||
|
||||
def test_pronoun_subject(self) -> None:
|
||||
"""He spent 50 dollars — pronoun resolved from registry."""
|
||||
ps = _empty_problem(registry=(EntityRef("eric", "male", 0),))
|
||||
words = ["He", "spent", "50", "dollars", "."]
|
||||
ss = _read_sentence(words, ps)
|
||||
assert isinstance(ss, SentenceReadingState), ss
|
||||
end = end_sentence(ss, ps)
|
||||
assert isinstance(end, ProblemReadingState), end
|
||||
pop = end.accumulated_operations[0]
|
||||
assert pop.actor == "eric"
|
||||
|
||||
def test_refuse_no_quantity(self) -> None:
|
||||
ps = _empty_problem(registry=(EntityRef("sandra", "female", 0),))
|
||||
words = ["She", "spent", "."]
|
||||
ss = _read_sentence(words, ps)
|
||||
assert isinstance(ss, SentenceReadingState)
|
||||
end = end_sentence(ss, ps)
|
||||
assert isinstance(end, ReaderRefusal)
|
||||
assert end.reason == "incomplete_operation"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Descriptive frame
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestDescriptiveFrame:
|
||||
def test_copula_drains_advances(self) -> None:
|
||||
"""Sandra is a baker. — descriptive_frame, no math state."""
|
||||
ps = _empty_problem()
|
||||
words = ["Sandra", "is", "a", "baker", "."]
|
||||
ss = _read_sentence(words, ps)
|
||||
# "a" drains, "baker" → unknown_word refusal (not in lexicon)
|
||||
# This is expected for Phase 2 scope
|
||||
assert isinstance(ss, (SentenceReadingState, ReaderRefusal))
|
||||
|
||||
def test_copula_with_known_tokens_only(self) -> None:
|
||||
"""Sandra is the student. — all known tokens drain."""
|
||||
ps = _empty_problem()
|
||||
words = ["Sandra", "is", "the", "student", "."]
|
||||
ss = _read_sentence(words, ps)
|
||||
assert isinstance(ss, SentenceReadingState), ss
|
||||
end = end_sentence(ss, ps)
|
||||
assert isinstance(end, ProblemReadingState), end
|
||||
assert len(end.accumulated_initial_state) == 0
|
||||
assert len(end.accumulated_operations) == 0
|
||||
assert end.sentence_index == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Full problem round-trip with finalize()
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestFinalize:
|
||||
def test_simple_two_sentence_problem(self) -> None:
|
||||
"""Sandra had 600 dollars. She spent 200 dollars. How much is left?"""
|
||||
sentences = [
|
||||
["Sandra", "had", "600", "dollars", "."],
|
||||
["She", "spent", "200", "dollars", "."],
|
||||
["How", "much", "money", "will", "she", "be", "left", "with", "?"],
|
||||
]
|
||||
ps = _read_problem(sentences)
|
||||
assert isinstance(ps, ProblemReadingState), ps
|
||||
graph = finalize(ps)
|
||||
assert isinstance(graph, MathProblemGraph), graph
|
||||
assert "sandra" in graph.entities
|
||||
assert len(graph.initial_state) == 1
|
||||
assert graph.initial_state[0].entity == "sandra"
|
||||
assert graph.initial_state[0].quantity.value == 600.0
|
||||
assert len(graph.operations) == 1
|
||||
assert graph.operations[0].kind == "subtract"
|
||||
assert graph.operations[0].operand.value == 200.0
|
||||
assert graph.unknown.entity == "sandra"
|
||||
|
||||
def test_finalize_no_question_target_refuses(self) -> None:
|
||||
ps = _empty_problem()
|
||||
result = finalize(ps)
|
||||
assert isinstance(result, ReaderRefusal)
|
||||
assert result.reason == "no_question_target"
|
||||
|
||||
def test_finalize_empty_registry_refuses(self) -> None:
|
||||
from generate.comprehension.state import QuestionTargetSlot
|
||||
qs = QuestionTargetSlot(
|
||||
kind="continuous_quantity",
|
||||
entity="sandra",
|
||||
unit_class="currency",
|
||||
unit="dollar",
|
||||
position=0,
|
||||
)
|
||||
ps = ProblemReadingState(
|
||||
entity_registry=(), # empty
|
||||
accumulated_initial_state=(),
|
||||
accumulated_operations=(),
|
||||
unknown_target_slot=qs,
|
||||
pronoun_resolution_history=(),
|
||||
sentence_index=1,
|
||||
source_text_offset=0,
|
||||
)
|
||||
result = finalize(ps)
|
||||
assert isinstance(result, ReaderRefusal)
|
||||
assert result.reason == "dangling_entity"
|
||||
|
||||
def test_determinism(self) -> None:
|
||||
"""Same input → same trace hash."""
|
||||
sentences = [
|
||||
["Sandra", "had", "600", "dollars", "."],
|
||||
["She", "spent", "200", "dollars", "."],
|
||||
["How", "much", "money", "will", "she", "be", "left", "with", "?"],
|
||||
]
|
||||
ps1 = _read_problem(sentences)
|
||||
ps2 = _read_problem(sentences)
|
||||
assert isinstance(ps1, ProblemReadingState)
|
||||
assert isinstance(ps2, ProblemReadingState)
|
||||
assert ps1.canonical_hash() == ps2.canonical_hash()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Refusal coverage
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPhase2Refusals:
|
||||
def test_fraction_token_refused(self) -> None:
|
||||
"""Fraction literals are out of Phase 2 scope."""
|
||||
ps = _empty_problem()
|
||||
s = begin_sentence(ps, 0)
|
||||
result = apply_word(s, ps, "1/2")
|
||||
assert isinstance(result, ReaderRefusal)
|
||||
assert result.reason == "unexpected_category"
|
||||
assert "Phase 2.1" in result.detail
|
||||
|
||||
def test_verb_without_entity_opens_descriptive(self) -> None:
|
||||
"""Verb before entity (subject dropped) opens descriptive_frame."""
|
||||
ps = _empty_problem()
|
||||
s = begin_sentence(ps, 0)
|
||||
result = apply_word(s, ps, "spent")
|
||||
assert isinstance(result, SentenceReadingState)
|
||||
assert result.frame == "descriptive_frame"
|
||||
|
||||
def test_unresolved_pronoun_statement_frame(self) -> None:
|
||||
"""Pronoun with empty registry refuses at pre-frame."""
|
||||
ps = _empty_problem()
|
||||
s = begin_sentence(ps, 0)
|
||||
result = apply_word(s, ps, "She")
|
||||
assert isinstance(result, ReaderRefusal)
|
||||
assert result.reason == "unresolved_pronoun"
|
||||
|
||||
def test_multi_sentence_wrong_zero(self) -> None:
|
||||
"""All-or-nothing: if one sentence fails, the whole problem refuses."""
|
||||
# First sentence succeeds, second has unknown word "baker"
|
||||
ps0 = _empty_problem()
|
||||
words1 = ["Sandra", "had", "600", "dollars", "."]
|
||||
ss1 = _read_sentence(words1, ps0)
|
||||
assert isinstance(ss1, SentenceReadingState)
|
||||
ps1 = end_sentence(ss1, ps0)
|
||||
assert isinstance(ps1, ProblemReadingState)
|
||||
|
||||
# Second sentence: "baker" is unknown → refusal
|
||||
words2 = ["She", "is", "a", "baker", "."]
|
||||
ss2 = _read_sentence(words2, ps1)
|
||||
# "baker" not in lexicon → unknown_word refusal
|
||||
assert isinstance(ss2, ReaderRefusal)
|
||||
assert ss2.reason == "unknown_word"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
|
|
@ -183,20 +183,20 @@ class TestRefusals:
|
|||
assert r.reason == "unknown_word"
|
||||
assert r.token_text == "@@@"
|
||||
|
||||
def test_unexpected_category_non_question_opener(self) -> None:
|
||||
"""A statement-frame opener at position 0 is Phase-2 scope."""
|
||||
def test_statement_frame_opener_accepted(self) -> None:
|
||||
"""Phase 2 (ADR-0164.4): proper noun at position 0 opens a statement
|
||||
pre-frame. After Brief 8.2's gender-enrichment refactor, the lookup
|
||||
skips proper_noun_gender_* categories and the proper_noun_token
|
||||
primitive admits "Francine" — Phase 2 then routes it to the
|
||||
statement-frame pre-frame entity slot instead of refusing.
|
||||
"""
|
||||
ps = _empty_problem(registry=(EntityRef("francine", "female", 0),))
|
||||
s = begin_sentence(ps, 0)
|
||||
# "francine" is a known word in the gender-enrichment lexicon.
|
||||
# Under sentence-initial lookup-first dispatch, gender categories
|
||||
# do not admit — they are enrichment, not admission. The
|
||||
# proper_noun_token primitive then admits "Francine" as a name,
|
||||
# but at sentence position 0 (statement frame opener) Phase 1
|
||||
# refuses.
|
||||
r = apply_word(s, ps, "Francine")
|
||||
assert isinstance(r, ReaderRefusal)
|
||||
assert r.reason == "unexpected_category"
|
||||
assert "Phase-2" in r.detail
|
||||
assert isinstance(r, SentenceReadingState)
|
||||
assert r.frame is None # frame determined on next verb
|
||||
assert r.pending_entity_ref is not None
|
||||
assert r.pending_entity_ref.canonical_name == "francine"
|
||||
|
||||
def test_unresolved_pronoun_empty_registry(self) -> None:
|
||||
"""A pronoun with no compatible entity refuses cleanly."""
|
||||
|
|
@ -311,7 +311,9 @@ class TestInitialDispatchAndUnknownGender:
|
|||
state = begin_sentence(ps, 0)
|
||||
out = apply_word(state, ps, "She")
|
||||
assert isinstance(out, ReaderRefusal)
|
||||
assert out.reason == "unexpected_category"
|
||||
# Phase 2: pronoun at position 0 attempts resolution; empty
|
||||
# registry → unresolved_pronoun per ADR-0164.2.
|
||||
assert out.reason == "unresolved_pronoun"
|
||||
|
||||
state = begin_sentence(ps, 0)
|
||||
out = apply_word(state, ps, "How")
|
||||
|
|
@ -320,24 +322,29 @@ class TestInitialDispatchAndUnknownGender:
|
|||
|
||||
def test_sentence_initial_marnie_is_not_gated_by_gender_list(self) -> None:
|
||||
"""'Marnie' is not in proper_noun_gender_female (after Brief 8.2
|
||||
rename dropped marnie from the female list). Reader still admits
|
||||
her as a proper_noun_token at sentence-initial position, then
|
||||
refuses with Phase-2-scope detail (statement frame opener)."""
|
||||
rename dropped marnie from the female list). Reader admits her
|
||||
as a proper_noun_token at sentence-initial position. Under
|
||||
Phase 2, this opens a statement pre-frame (not a refusal).
|
||||
EntityRef carries gender="unknown" because the name is outside
|
||||
the curated gender lists.
|
||||
"""
|
||||
ps = _empty_problem()
|
||||
out = apply_word(begin_sentence(ps, 0), ps, "Marnie")
|
||||
assert isinstance(out, ReaderRefusal)
|
||||
assert out.reason == "unexpected_category"
|
||||
assert "Phase-2" in out.detail
|
||||
assert isinstance(out, SentenceReadingState)
|
||||
assert out.pending_entity_ref is not None
|
||||
assert out.pending_entity_ref.gender == "unknown"
|
||||
|
||||
def test_sentence_initial_novel_name_uses_primitive_and_unknown_gender(self) -> None:
|
||||
"""A name not in either gender list (e.g. 'Zelda') still admits
|
||||
via the universal proper_noun_token primitive."""
|
||||
via the universal proper_noun_token primitive. Under Phase 2,
|
||||
this opens a statement pre-frame with gender='unknown'."""
|
||||
ps = _empty_problem()
|
||||
state = begin_sentence(ps, 0)
|
||||
out = apply_word(state, ps, "Zelda")
|
||||
assert isinstance(out, ReaderRefusal)
|
||||
assert out.reason == "unexpected_category"
|
||||
assert "Phase-2" in out.detail
|
||||
assert isinstance(out, SentenceReadingState)
|
||||
assert out.pending_entity_ref is not None
|
||||
assert out.pending_entity_ref.canonical_name == "zelda"
|
||||
assert out.pending_entity_ref.gender == "unknown"
|
||||
|
||||
def test_pronoun_single_unknown_entity_resolves(self) -> None:
|
||||
"""ADR-0164.2 single-salient fallback: one gender-unknown entity
|
||||
|
|
|
|||
Loading…
Reference in a new issue