60043973b0
Extend the comprehension reader from question-only scope to whole- problem scope. Phase 1 (Brief 8 / #326) implemented question_frame; this brief implements initial_state_frame, operation_frame, and descriptive_frame, plus finalize() projection into a strict ADR-0115 MathProblemGraph. Architecturally correct under ADR-0164.3; not yet productive on GSM8K train_sample. Below-floor measurement documented; specific bottlenecks tabled for Phase 2.1 follow-up. What landed - Frame-opener dispatch in lifecycle.py for the three new statement frames, plus rule handlers (_rule_op_*, _rule_preframe_*, _rule_descriptive_*). - finalize(state) -> MathProblemGraph | ReaderRefusal: pure projection with closure checks (entity registry non-empty, unknown target bound, every op/initial references a known entity, Decimal precision projects losslessly). - _classify extended to 3-tuple (category, surface, decimal_value) with possessive strip retry. Brief 8.2's sentence-initial lookup-first + gender-skip preserved AND extended to mid-sentence (gender is enrichment everywhere, never admission). - Whole-problem coexistence dispatch in math_candidate_graph.py (config.comprehension_reader_questions=True): reader attempts the whole problem; on any ReaderRefusal falls through to existing regex parser. All-or-nothing per the brief. - Lexicon expansion (carried into renamed proper_noun_gender_* files): +2 accumulation_verb (adopt, invest), +2 currency_unit_noun (dollar, cent), +6 capacity_verb (fill, lift, play, work, finish, drive), +5 female names (allison, brooke, jan, marion, sidney), +14 male names (bart, fernando, georgie, jake, jed, jeremie, jose, orlando, rex, rudolph, steve, troy, xavier, yun), +numerous count_unit_noun, drain_token, time_unit_noun. - ADR-0164.4-phase2-statement-frame-reader.md — the architectural rationale and acceptance contract. Measurement (reader_phase2_delta.json): flag-OFF: correct=3 refused=47 wrong=0 flag-ON: correct=3 refused=47 wrong=0 delta: 0/0/0 Below the brief's floor of correct >= 4. Architecture is sound — the reader admits cases as graphs when the structure resolves, refuses cleanly otherwise, preserves wrong=0 across both flag states. Bottleneck table (from per-case attribution): count refusal_class dominant cause ----- ---------------------- ------------------------------------ 18 incomplete_operation multi-quantity ops; no-quantity op 11 unknown_word "hundred", "presently", "one-hour", non-math verbs (compound numerics, lexicon gaps) 6 unexpected_category fraction / percentage literals; multi-subject sentences 6 unresolved_pronoun "them", "their", "his" with no compatible entity 5 unattached_quantity quantity never bound to a unit 1 no_question_target question parsed but slot never set Closing the gate to mixed-bounded [4, 24] is Phase 2.1 scope: extend composition rules for multi-quantity ops, add fraction/percentage primitives (per ADR-0164.1 amendment), expand lexicon for the remaining unknown_word cases, extend pronoun resolution. Invariants preserved - wrong = 0 in both flag states ✓ - flag-OFF byte-identical to today ✓ - determinism (50/50 identical runs) ✓ - Capability axes G1-G5, S1 unchanged ✓ - Reader tests: 19 (Phase 2) + 18 (Phase 1, post-update) + 53 (pack) + 76 (lexicon + primitives) = 166 specific to this change; all pass - core test --suite smoke -q: 67 passed Rebase note This PR was authored against an older base; rebased onto current main to incorporate #333 (Brief 8.2 universal proper_noun_token primitive) and #334 (ADR-0166 measurement discipline). The rebase required: - Lexicon files renamed proper_noun_entity_* -> proper_noun_gender_* (with the Phase 2 additions merged into the gender_* files) - Compiled lexicon.jsonl unchanged from #333's 207-entry state (Phase 2's per-category additions are runtime-visible via the source loader, not via the compiled file) - _classify reconciled with Brief 8.2's sentence-initial dispatch + Phase 2's 3-tuple decimal-value return - All dispatch tables and category checks updated to reference proper_noun_token (singular) instead of proper_noun_entity_{f,m} - Three Phase 1 test expectations updated to reflect Phase 2 behavior (proper noun at position 0 now opens statement pre-frame instead of refusing; pronoun resolution applies per ADR-0164.2) Per ADR-0166's three-question test, this PR is honest measurement: capability exists, at least one case admits, lane distinguishes presence from absence — which the bottleneck table demonstrates. Refs ADR-0164.3 §Phasing Phase 2, ADR-0164.1 amendment (Brief 8.2), ADR-0166 §"Mixed (notable but not blocking)" — except here, below floor.
380 lines
15 KiB
Python
380 lines
15 KiB
Python
"""Tests for the Phase-1 question-frame reader lifecycle (ADR-0164.3)."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from generate.comprehension.lifecycle import (
|
|
apply_word,
|
|
begin_sentence,
|
|
end_sentence,
|
|
)
|
|
from generate.comprehension.state import (
|
|
EntityRef,
|
|
ProblemReadingState,
|
|
ReaderRefusal,
|
|
SentenceReadingState,
|
|
)
|
|
|
|
|
|
def _empty_problem(
|
|
*,
|
|
registry: tuple[EntityRef, ...] = (),
|
|
sentence_index: int = 0,
|
|
) -> ProblemReadingState:
|
|
return ProblemReadingState(
|
|
entity_registry=registry,
|
|
accumulated_initial_state=(),
|
|
accumulated_operations=(),
|
|
unknown_target_slot=None,
|
|
pronoun_resolution_history=(),
|
|
sentence_index=sentence_index,
|
|
source_text_offset=0,
|
|
)
|
|
|
|
|
|
def _read_sentence(
|
|
words: list[str],
|
|
problem_state: ProblemReadingState,
|
|
) -> SentenceReadingState | ReaderRefusal:
|
|
"""Drive a full sentence through apply_word. Returns final state or refusal."""
|
|
state: SentenceReadingState | ReaderRefusal = begin_sentence(problem_state, 0)
|
|
assert isinstance(state, SentenceReadingState)
|
|
for word in words:
|
|
result = apply_word(state, problem_state, word)
|
|
if isinstance(result, ReaderRefusal):
|
|
return result
|
|
state = result
|
|
return state
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Determinism
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestDeterminism:
|
|
def test_apply_word_byte_equal_outputs(self) -> None:
|
|
ps = _empty_problem(registry=(EntityRef("monica", "female", 0),))
|
|
s1 = begin_sentence(ps, 0)
|
|
s2 = begin_sentence(ps, 0)
|
|
r1 = apply_word(s1, ps, "How")
|
|
r2 = apply_word(s2, ps, "How")
|
|
assert isinstance(r1, SentenceReadingState)
|
|
assert isinstance(r2, SentenceReadingState)
|
|
assert r1.canonical_bytes() == r2.canonical_bytes()
|
|
assert r1.canonical_hash() == r2.canonical_hash()
|
|
|
|
def test_full_sentence_byte_equal(self) -> None:
|
|
ps = _empty_problem(registry=(EntityRef("monica", "female", 0),))
|
|
words = ["How", "much", "time", "did", "she", "spend", "?"]
|
|
a = _read_sentence(words, ps)
|
|
b = _read_sentence(words, ps)
|
|
assert isinstance(a, SentenceReadingState)
|
|
assert isinstance(b, SentenceReadingState)
|
|
assert a.canonical_bytes() == b.canonical_bytes()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Five GSM8K target question sentences
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestGSM8KQuestions:
|
|
def test_0007_how_many_more_boxes(self) -> None:
|
|
ps = _empty_problem(
|
|
registry=(EntityRef("francine_and_friend", "unknown", 0),)
|
|
)
|
|
words = [
|
|
"How", "many", "more", "boxes", "do", "they", "need",
|
|
"if", "Francine", "has", "a", "total", "of", "85", "crayons", "?",
|
|
]
|
|
state = _read_sentence(words, ps)
|
|
assert isinstance(state, SentenceReadingState), state
|
|
end = end_sentence(state, ps)
|
|
assert isinstance(end, ProblemReadingState), end
|
|
target = end.unknown_target_slot
|
|
assert target is not None
|
|
assert target.entity == "francine_and_friend"
|
|
assert target.unit_class == "count"
|
|
assert target.kind == "difference"
|
|
|
|
def test_0017_how_much_cost_him(self) -> None:
|
|
ps = _empty_problem(
|
|
registry=(
|
|
EntityRef("eric", "male", 0),
|
|
EntityRef("house", "neuter", 1),
|
|
)
|
|
)
|
|
words = ["How", "much", "will", "it", "cost", "him", "?"]
|
|
state = _read_sentence(words, ps)
|
|
assert isinstance(state, SentenceReadingState), state
|
|
end = end_sentence(state, ps)
|
|
assert isinstance(end, ProblemReadingState), end
|
|
target = end.unknown_target_slot
|
|
assert target is not None
|
|
assert target.entity == "eric"
|
|
assert target.unit_class == "currency"
|
|
assert target.kind == "continuous_quantity"
|
|
|
|
def test_0027_how_many_followers_malcolm(self) -> None:
|
|
ps = _empty_problem()
|
|
words = [
|
|
"How", "many", "followers", "does", "Malcolm", "have",
|
|
"on", "all", "his", "social", "media", "?",
|
|
]
|
|
state = _read_sentence(words, ps)
|
|
assert isinstance(state, SentenceReadingState), state
|
|
end = end_sentence(state, ps)
|
|
assert isinstance(end, ProblemReadingState), end
|
|
target = end.unknown_target_slot
|
|
assert target is not None
|
|
assert target.entity == "Malcolm"
|
|
assert target.unit_class == "count"
|
|
assert target.kind == "discrete_quantity"
|
|
# Proper-noun entity entered the registry.
|
|
names = [e.canonical_name for e in end.entity_registry]
|
|
assert "Malcolm" in names
|
|
|
|
def test_0036_how_much_time_studying(self) -> None:
|
|
ps = _empty_problem(registry=(EntityRef("monica", "female", 0),))
|
|
words = [
|
|
"How", "much", "time", "did", "she", "spend", "studying",
|
|
"in", "total", "during", "the", "five", "days", "?",
|
|
]
|
|
state = _read_sentence(words, ps)
|
|
assert isinstance(state, SentenceReadingState), state
|
|
end = end_sentence(state, ps)
|
|
assert isinstance(end, ProblemReadingState), end
|
|
target = end.unknown_target_slot
|
|
assert target is not None
|
|
assert target.entity == "monica"
|
|
assert target.unit_class == "time"
|
|
assert target.kind == "continuous_quantity"
|
|
|
|
def test_0043_how_much_money_left(self) -> None:
|
|
ps = _empty_problem(registry=(EntityRef("sandra", "female", 0),))
|
|
words = [
|
|
"How", "much", "money", "will", "she", "be", "left",
|
|
"with", "after", "the", "purchase", "?",
|
|
]
|
|
state = _read_sentence(words, ps)
|
|
assert isinstance(state, SentenceReadingState), state
|
|
end = end_sentence(state, ps)
|
|
assert isinstance(end, ProblemReadingState), end
|
|
target = end.unknown_target_slot
|
|
assert target is not None
|
|
assert target.entity == "sandra"
|
|
assert target.unit_class == "currency"
|
|
assert target.kind == "continuous_quantity"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Refusal modes
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestRefusals:
|
|
def test_unknown_word(self) -> None:
|
|
ps = _empty_problem()
|
|
s = begin_sentence(ps, 0)
|
|
r = apply_word(s, ps, "@@@")
|
|
assert isinstance(r, ReaderRefusal)
|
|
assert r.reason == "unknown_word"
|
|
assert r.token_text == "@@@"
|
|
|
|
def test_statement_frame_opener_accepted(self) -> None:
|
|
"""Phase 2 (ADR-0164.4): proper noun at position 0 opens a statement
|
|
pre-frame. After Brief 8.2's gender-enrichment refactor, the lookup
|
|
skips proper_noun_gender_* categories and the proper_noun_token
|
|
primitive admits "Francine" — Phase 2 then routes it to the
|
|
statement-frame pre-frame entity slot instead of refusing.
|
|
"""
|
|
ps = _empty_problem(registry=(EntityRef("francine", "female", 0),))
|
|
s = begin_sentence(ps, 0)
|
|
r = apply_word(s, ps, "Francine")
|
|
assert isinstance(r, SentenceReadingState)
|
|
assert r.frame is None # frame determined on next verb
|
|
assert r.pending_entity_ref is not None
|
|
assert r.pending_entity_ref.canonical_name == "francine"
|
|
|
|
def test_unresolved_pronoun_empty_registry(self) -> None:
|
|
"""A pronoun with no compatible entity refuses cleanly."""
|
|
ps = _empty_problem() # empty registry
|
|
s = begin_sentence(ps, 0)
|
|
s = apply_word(s, ps, "How")
|
|
assert isinstance(s, SentenceReadingState)
|
|
s = apply_word(s, ps, "much")
|
|
assert isinstance(s, SentenceReadingState)
|
|
s = apply_word(s, ps, "money")
|
|
assert isinstance(s, SentenceReadingState)
|
|
s = apply_word(s, ps, "will")
|
|
assert isinstance(s, SentenceReadingState)
|
|
r = apply_word(s, ps, "she")
|
|
assert isinstance(r, ReaderRefusal)
|
|
assert r.reason == "unresolved_pronoun"
|
|
assert r.token_text == "she"
|
|
|
|
def test_unfinished_frame_on_end(self) -> None:
|
|
ps = _empty_problem()
|
|
s = begin_sentence(ps, 0)
|
|
# No words applied → frame is still None.
|
|
end = end_sentence(s, ps)
|
|
assert isinstance(end, ReaderRefusal)
|
|
assert end.reason == "unfinished_frame"
|
|
|
|
def test_unattached_quantity_on_end(self) -> None:
|
|
"""A SentenceReadingState with frame set but pending_quantities
|
|
non-empty refuses with unattached_quantity at end_sentence."""
|
|
from decimal import Decimal
|
|
|
|
from generate.comprehension.state import QuantityRef
|
|
|
|
ps = _empty_problem()
|
|
# Construct a hand-built state to isolate the rule.
|
|
pending = QuantityRef(
|
|
value=Decimal("18"),
|
|
unit=None,
|
|
unit_class="pending",
|
|
owner_entity=None,
|
|
mention_position=0,
|
|
)
|
|
state = SentenceReadingState(
|
|
entities=(),
|
|
quantities=(),
|
|
operations=(),
|
|
frame="question_frame",
|
|
pending_quantities=(pending,),
|
|
token_index=2,
|
|
)
|
|
end = end_sentence(state, ps)
|
|
assert isinstance(end, ReaderRefusal)
|
|
assert end.reason == "unattached_quantity"
|
|
|
|
def test_incomplete_operation_no_unit(self) -> None:
|
|
"""question_frame closes with no unit_class on the target → refuse."""
|
|
ps = _empty_problem(registry=(EntityRef("monica", "female", 0),))
|
|
s = begin_sentence(ps, 0)
|
|
# Only "How" then "?" — no unit was ever set.
|
|
s = apply_word(s, ps, "How")
|
|
assert isinstance(s, SentenceReadingState)
|
|
s = apply_word(s, ps, "?")
|
|
assert isinstance(s, SentenceReadingState)
|
|
end = end_sentence(s, ps)
|
|
assert isinstance(end, ReaderRefusal)
|
|
assert end.reason == "incomplete_operation"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Lifecycle invariants
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestLifecycleInvariants:
|
|
def test_problem_state_preserved_when_sentence_introduces_no_entity(self) -> None:
|
|
"""begin → apply_word(*) → end_sentence preserves the registry
|
|
when the sentence only references existing entities."""
|
|
registry = (EntityRef("monica", "female", 0),)
|
|
ps = _empty_problem(registry=registry)
|
|
words = ["How", "much", "time", "did", "she", "spend", "?"]
|
|
state = _read_sentence(words, ps)
|
|
assert isinstance(state, SentenceReadingState)
|
|
end = end_sentence(state, ps)
|
|
assert isinstance(end, ProblemReadingState)
|
|
assert end.entity_registry == registry
|
|
|
|
def test_sentence_index_advances(self) -> None:
|
|
ps = _empty_problem(registry=(EntityRef("monica", "female", 0),))
|
|
words = ["How", "much", "time", "did", "she", "spend", "?"]
|
|
state = _read_sentence(words, ps)
|
|
assert isinstance(state, SentenceReadingState)
|
|
end = end_sentence(state, ps)
|
|
assert isinstance(end, ProblemReadingState)
|
|
assert end.sentence_index == 1
|
|
|
|
|
|
class TestInitialDispatchAndUnknownGender:
|
|
"""ADR-0164.1 amendment (Brief 8.2): sentence-initial lookup-first +
|
|
universal proper_noun_token primitive + unknown-gender pronoun
|
|
resolution via single-salient fallback (ADR-0164.2)."""
|
|
|
|
def test_sentence_initial_common_words_lookup_first(self) -> None:
|
|
"""Sentence-initial 'The'/'She'/'How' resolve via lexicon, not
|
|
the proper_noun_token primitive. 'The' drains; 'She' refuses
|
|
(pronoun outside question_frame); 'How' opens question_frame."""
|
|
ps = _empty_problem()
|
|
state = begin_sentence(ps, 0)
|
|
out = apply_word(state, ps, "The")
|
|
assert isinstance(out, SentenceReadingState)
|
|
assert out.lookback[-1].category == "drain_token"
|
|
|
|
state = begin_sentence(ps, 0)
|
|
out = apply_word(state, ps, "She")
|
|
assert isinstance(out, ReaderRefusal)
|
|
# Phase 2: pronoun at position 0 attempts resolution; empty
|
|
# registry → unresolved_pronoun per ADR-0164.2.
|
|
assert out.reason == "unresolved_pronoun"
|
|
|
|
state = begin_sentence(ps, 0)
|
|
out = apply_word(state, ps, "How")
|
|
assert isinstance(out, SentenceReadingState)
|
|
assert out.frame == "question_frame"
|
|
|
|
def test_sentence_initial_marnie_is_not_gated_by_gender_list(self) -> None:
|
|
"""'Marnie' is not in proper_noun_gender_female (after Brief 8.2
|
|
rename dropped marnie from the female list). Reader admits her
|
|
as a proper_noun_token at sentence-initial position. Under
|
|
Phase 2, this opens a statement pre-frame (not a refusal).
|
|
EntityRef carries gender="unknown" because the name is outside
|
|
the curated gender lists.
|
|
"""
|
|
ps = _empty_problem()
|
|
out = apply_word(begin_sentence(ps, 0), ps, "Marnie")
|
|
assert isinstance(out, SentenceReadingState)
|
|
assert out.pending_entity_ref is not None
|
|
assert out.pending_entity_ref.gender == "unknown"
|
|
|
|
def test_sentence_initial_novel_name_uses_primitive_and_unknown_gender(self) -> None:
|
|
"""A name not in either gender list (e.g. 'Zelda') still admits
|
|
via the universal proper_noun_token primitive. Under Phase 2,
|
|
this opens a statement pre-frame with gender='unknown'."""
|
|
ps = _empty_problem()
|
|
state = begin_sentence(ps, 0)
|
|
out = apply_word(state, ps, "Zelda")
|
|
assert isinstance(out, SentenceReadingState)
|
|
assert out.pending_entity_ref is not None
|
|
assert out.pending_entity_ref.canonical_name == "zelda"
|
|
assert out.pending_entity_ref.gender == "unknown"
|
|
|
|
def test_pronoun_single_unknown_entity_resolves(self) -> None:
|
|
"""ADR-0164.2 single-salient fallback: one gender-unknown entity
|
|
resolves the pronoun cleanly."""
|
|
ps = _empty_problem(registry=(EntityRef("Zelda", "unknown", 0),))
|
|
state = _read_sentence(
|
|
["How", "much", "money", "will", "she", "earn", "?"], ps
|
|
)
|
|
assert isinstance(state, SentenceReadingState), state
|
|
assert state.question_target is not None
|
|
assert state.question_target.entity == "Zelda"
|
|
|
|
def test_pronoun_two_unknown_entities_refuses_ambiguous(self) -> None:
|
|
"""ADR-0164.2: two gender-unknown entities + no recency
|
|
disambiguation → ambiguous_pronoun_referent."""
|
|
ps = _empty_problem(
|
|
registry=(
|
|
EntityRef("Zelda", "unknown", 0),
|
|
EntityRef("Marnie", "unknown", 1),
|
|
)
|
|
)
|
|
state = begin_sentence(ps, 0)
|
|
assert isinstance(state, SentenceReadingState)
|
|
for token in ["How", "much", "money", "will"]:
|
|
state = apply_word(state, ps, token)
|
|
assert isinstance(state, SentenceReadingState)
|
|
out = apply_word(state, ps, "she")
|
|
assert isinstance(out, ReaderRefusal)
|
|
assert out.reason == "ambiguous_pronoun_referent"
|
|
|
|
|
|
if __name__ == "__main__":
|
|
pytest.main([__file__, "-v"])
|