96b9b942e6
Disciplined Path β (field decode α was empirically falsified). Reads S-P-O
structure SYMBOLICALLY from the token sequence via domain-agnostic templates
keyed on FUNCTION WORDS + ORDER, mints content as MeaningGraph entities/relations,
parse-or-refuse (wrong=0 at the comprehension layer).
Templates (set_membership): 'X is a Y' -> member; 'all Xs are Ys' -> subset;
'is X a Y?' / 'are all Xs Ys?' -> queries; definite-NP ('the X is a Y');
conservative singularization incl. irregulars (people->person), refusing
unknown morphology rather than guessing.
End-to-end: prose -> comprehend -> project -> INDEPENDENT set_membership oracle
-> answer vs gold. Result on the real v1 lane: 8 correct / 0 wrong / 0 refused
(full coverage, wrong=0). Cross-content generality (animals/professions/geography)
asserted. 14 reader unit tests + 3 end-to-end tests, each bites under its
violation. Additive only; invariants + capability index green.
31 lines
1.2 KiB
Python
31 lines
1.2 KiB
Python
"""Phase 2a-r2 — end-to-end: the comprehension reader scored on set_membership.
|
|
|
|
prose -> comprehend -> project -> INDEPENDENT oracle -> answer vs gold.
|
|
The load-bearing invariant: wrong == 0 (the reader refuses rather than emit a
|
|
structure that yields a wrong answer). Coverage (correct/total) is reported but
|
|
is allowed to be partial — refusal is honest, a wrong commit is not.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from evals.comprehension.set_membership_runner import run
|
|
|
|
|
|
def test_comprehension_set_membership_wrong_is_zero() -> None:
|
|
report = run()
|
|
assert report["wrong"] == 0, report["wrongs"]
|
|
|
|
|
|
def test_comprehension_set_membership_has_real_coverage() -> None:
|
|
report = run()
|
|
# Real, non-trivial comprehension — not just refuse-everything.
|
|
assert report["correct"] > 0
|
|
assert report["correct"] + report["refused"] == report["total"]
|
|
|
|
|
|
def test_comprehension_set_membership_full_coverage() -> None:
|
|
# This increment covers the whole v1 lane (member / subset / both query forms,
|
|
# definite-NP, irregular plurals) with zero wrong commits.
|
|
report = run()
|
|
assert report["refused"] == 0
|
|
assert report["correct"] == report["total"]
|