a005a92fed
The binding-graph's FIRST comprehension consumer (doctrine-aligned: quantities live in binding_graph, NOT the MeaningGraph). generate/quantitative_comprehension.py reads arithmetic prose into SymbolBinding/BoundFact/BoundEquation and runs the REAL check_admissibility (shell -> verify -> rebuild with the actual UnitProof) — there is NO stamped "admitted": an equation is admitted only if its operand units verify. Then to_relational_metric projects the binding-graph to the independent relational_metric oracle for the verdict. Templates (digits only; non-digit quantity REFUSES): "<X> has <N> <unit>" -> BoundFact(X = N) "<Y> has <N> more <unit> than <X>" -> BoundEquation(Y = X + N) op=add "<Y> has <N> fewer <unit> than <X>" -> BoundEquation(Y = X - N) op=subtract "How many <unit> does <Y> have" -> ask Y "How many <unit> do <X> and <Y> have"-> total = X + Y; ask total Unit modelling (honest, not faked): a noun the closed en_units_v1 pack knows is used verbatim (dollars -> dollar/money); an UNKNOWN sortal noun (stickers, coins) is a count of discrete objects -> the existing 'item' lemma (dimension count). So admissibility stays a REAL check: count+count admits, count+money (a mixed-unit sum) REFUSES with unit_mismatch — verified to bite. comprehension_relational_metric: 15/15 wrong=0 (full coverage). Located OUTSIDE generate/meaning_graph (it targets binding_graph, not the MeaningGraph) so INV-28 neutrality stays intact; oracle imports none of the SUT (new INV-25 lane). Capability index breadth 7->8, score 0.928622 -> 0.937258, wrong_total 0, digest 50e0675b… Tests: reader templates + count/known-unit modelling + admissibility-bite (mixed unit refuses) + non-digit refusal; end-to-end full-coverage wrong=0; arithmetic added to the structure-preservation generative panel (projected relations+query == ground truth); capability breadth 7->8; INV-25 arithmetic lane. 93 targeted + 90 smoke green; lane SHAs 8/9 (sole miss = public_demo env flake; deductive_logic + math_teaching unchanged -> no GSM8K coupling).
75 lines
2.4 KiB
Python
75 lines
2.4 KiB
Python
"""Score the general comprehension reader on the relational_metric gold lane.
|
|
|
|
prose -> comprehend_quantitative() -> binding_graph -> to_relational_metric() ->
|
|
independent arithmetic oracle -> answer vs gold. This is the binding-graph's first
|
|
comprehension consumer: quantities live in the binding-graph (admissibility-checked,
|
|
never stamped), then project to the relational_metric oracle for the verdict.
|
|
|
|
A refusal (unreadable prose, admissibility refusal, unprojectable, or an
|
|
OracleError on the projection) is NOT a wrong; only a committed integer answer that
|
|
disagrees with gold is wrong (must stay 0).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import sys
|
|
from typing import Any
|
|
|
|
from evals.relational_metric.oracle import OracleError, oracle_answer
|
|
from evals.relational_metric.runner import _load_cases
|
|
from generate.meaning_graph.reader import Refusal
|
|
from generate.quantitative_comprehension import comprehend_quantitative, to_relational_metric
|
|
|
|
|
|
def run() -> dict[str, Any]:
|
|
cases = _load_cases()
|
|
correct = wrong = refused = 0
|
|
wrongs: list[dict[str, Any]] = []
|
|
|
|
for case in cases:
|
|
comp = comprehend_quantitative(case["text"])
|
|
if isinstance(comp, Refusal):
|
|
refused += 1
|
|
continue
|
|
projected = to_relational_metric(comp)
|
|
if projected is None:
|
|
refused += 1
|
|
continue
|
|
relations, query = projected
|
|
try:
|
|
got = oracle_answer(relations, query)
|
|
except OracleError:
|
|
refused += 1
|
|
continue
|
|
if got == case.get("gold"):
|
|
correct += 1
|
|
else:
|
|
wrong += 1
|
|
wrongs.append(
|
|
{"id": case.get("id"), "got": got, "gold": case.get("gold"), "text": case["text"]}
|
|
)
|
|
|
|
return {
|
|
"domain": "comprehension_relational_metric",
|
|
"total": len(cases),
|
|
"correct": correct,
|
|
"wrong": wrong,
|
|
"refused": refused,
|
|
"wrongs": wrongs,
|
|
"counts": {"correct": correct, "wrong": wrong, "refused": refused},
|
|
}
|
|
|
|
|
|
def main() -> int:
|
|
report = run()
|
|
print(json.dumps({k: v for k, v in report.items() if k != "wrongs"}, indent=2, sort_keys=True))
|
|
if report["wrong"]:
|
|
print("WRONG > 0 — comprehension produced a wrong committed answer:", file=sys.stderr)
|
|
print(json.dumps(report["wrongs"], indent=2), file=sys.stderr)
|
|
return 1
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|