core-labs/core
core-labs/core
1
0
Fork 0
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity

test(comprehension): expose R1 answer lane

Browse source
This commit is contained in:
Shay 2026-06-06 18:28:49 -07:00
parent c364d434da
commit 88d8950d3e
1 changed files with 4 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
evals/setup_oracle/__init__.py
View file

@ -9,11 +9,11 @@ closes that gap: it compares the reader's comprehended STRUCTURE (the relations
binding-graph reader). A wrong setup is a first-class failure even when its answer is right. binding-graph reader). A wrong setup is a first-class failure even when its answer is right.
`setup_wrong` is the load-bearing, wrong=0-critical count: a reading that misrepresents `setup_wrong` is the load-bearing, wrong=0-critical count: a reading that misrepresents
the problem. v1 grades structure (facts + equations + question target/state/form); unit the problem. The R1 answer lane (PR-6b) runs only after setup is correct; unsupported
modelling stays covered by the admissibility tests (a documented signature extension). fixtures remain refusals and setup-wrong fixtures are never answer-scored.
""" """
from evals.setup_oracle.runner import run, run_r1 from evals.setup_oracle.runner import run, run_r1, run_r1_answers
from evals.setup_oracle.signature import ( from evals.setup_oracle.signature import (
gold_unknown_signature, gold_unknown_signature,
reader_symbol_units, reader_symbol_units,
@ -29,5 +29,6 @@ __all__ = [
"relation_signature", "relation_signature",
"run", "run",
"run_r1", "run_r1",
"run_r1_answers",
"symbol_unit_signature", "symbol_unit_signature",
] ]