a005a92fed
The binding-graph's FIRST comprehension consumer (doctrine-aligned: quantities live in binding_graph, NOT the MeaningGraph). generate/quantitative_comprehension.py reads arithmetic prose into SymbolBinding/BoundFact/BoundEquation and runs the REAL check_admissibility (shell -> verify -> rebuild with the actual UnitProof) — there is NO stamped "admitted": an equation is admitted only if its operand units verify. Then to_relational_metric projects the binding-graph to the independent relational_metric oracle for the verdict. Templates (digits only; non-digit quantity REFUSES): "<X> has <N> <unit>" -> BoundFact(X = N) "<Y> has <N> more <unit> than <X>" -> BoundEquation(Y = X + N) op=add "<Y> has <N> fewer <unit> than <X>" -> BoundEquation(Y = X - N) op=subtract "How many <unit> does <Y> have" -> ask Y "How many <unit> do <X> and <Y> have"-> total = X + Y; ask total Unit modelling (honest, not faked): a noun the closed en_units_v1 pack knows is used verbatim (dollars -> dollar/money); an UNKNOWN sortal noun (stickers, coins) is a count of discrete objects -> the existing 'item' lemma (dimension count). So admissibility stays a REAL check: count+count admits, count+money (a mixed-unit sum) REFUSES with unit_mismatch — verified to bite. comprehension_relational_metric: 15/15 wrong=0 (full coverage). Located OUTSIDE generate/meaning_graph (it targets binding_graph, not the MeaningGraph) so INV-28 neutrality stays intact; oracle imports none of the SUT (new INV-25 lane). Capability index breadth 7->8, score 0.928622 -> 0.937258, wrong_total 0, digest 50e0675b… Tests: reader templates + count/known-unit modelling + admissibility-bite (mixed unit refuses) + non-digit refusal; end-to-end full-coverage wrong=0; arithmetic added to the structure-preservation generative panel (projected relations+query == ground truth); capability breadth 7->8; INV-25 arithmetic lane. 93 targeted + 90 smoke green; lane SHAs 8/9 (sole miss = public_demo env flake; deductive_logic + math_teaching unchanged -> no GSM8K coupling).
30 lines
1.1 KiB
Python
30 lines
1.1 KiB
Python
"""Phase 2b — end-to-end: the comprehension reader scored on relational_metric.
|
|
|
|
prose -> comprehend_quantitative -> binding_graph -> to_relational_metric ->
|
|
INDEPENDENT arithmetic oracle -> answer vs gold. The load-bearing invariant:
|
|
wrong == 0. This is the binding-graph's first comprehension consumer; quantities
|
|
are admissibility-checked (never stamped), then projected to the oracle.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from evals.comprehension.relational_metric_runner import run
|
|
|
|
|
|
def test_comprehension_relational_metric_wrong_is_zero() -> None:
|
|
report = run()
|
|
assert report["wrong"] == 0, report["wrongs"]
|
|
|
|
|
|
def test_comprehension_relational_metric_has_real_coverage() -> None:
|
|
report = run()
|
|
assert report["correct"] > 0
|
|
assert report["correct"] + report["refused"] == report["total"]
|
|
|
|
|
|
def test_comprehension_relational_metric_full_coverage() -> None:
|
|
# The whole 15-case lane reads end-to-end (fact / more_than / fewer_than /
|
|
# sum_of, single + chained, count nouns -> item dimension, dollars -> money).
|
|
report = run()
|
|
assert report["refused"] == 0
|
|
assert report["correct"] == report["total"]
|