59974865ef
Two coupled, additive, off-serving changes toward the typed math-comprehension organ. No serving path touched; the relational_metric answer lane stays 15/15 wrong=0. PR-1 — QuantQuery → BoundUnknown. comprehend_quantitative now emits the question target as a BoundUnknown INSIDE the binding-graph (symbol_id, state_index="terminal", question_form "count"|"total", expected_unit), so the graph is a real question-bearing mathematical object and its canonical serialization carries the target. The external QuantQuery is RETAINED, consistent-by-construction, so the two consumers (to_relational_metric, realize/quantitative) are byte-identical; a follow-up rewires them onto graph.unknowns and drops the duplicate field. Setup-oracle lane (evals/setup_oracle) — grade the READING, not the answer. The relational_metric lane scores answers, which can bless a semantically-wrong derivation that coincidentally lands on the right number (the exact hazard the held-out measurements + the 2/87 resolve_pooled probe exposed). The setup-oracle compares the reader's comprehended STRUCTURE — a span-free signature of facts + typed equations + the BoundUnknown target — against the INDEPENDENT gold structure (the relational_metric cases' own relations/query, authored separately from the binding-graph reader). A structural mismatch is setup_wrong, the wrong=0-critical count, even when the answer would be right. v1 grades structure (units deferred — covered by admissibility). The reader reads all 15 cases with the gold structure (setup_wrong=0); a meaningful-fail test proves the oracle catches a right-answer/wrong-structure reading (it is not decoration). `python -m evals.setup_oracle` exits nonzero iff setup_wrong > 0. This is the measurement rig BEFORE investing in frame families: setup_wrong=0 is the gate; serving must not move while setup_wrong > 0. It is the first milestone of the math-comprehension organ, not a path to "solve GSM8K". Verified: setup-oracle 15/15 setup_correct wrong=0; quantitative + setup-oracle unit tests (17); realize-binding-graph + binding-graph + architectural invariants (183).
76 lines
3.1 KiB
Python
76 lines
3.1 KiB
Python
"""Span-free structural signatures for the setup-oracle.
|
|
|
|
A *signature* is a deterministic, order-independent, span-free projection of the
|
|
mathematical SETUP — what the reader claims the problem says, stripped of input
|
|
offsets and surface tokens. Two readings are setup-equivalent iff their signatures
|
|
are equal. Used to compare the reader's comprehended structure against the
|
|
independent gold structure (the relational_metric cases' own `relations`/`query`).
|
|
|
|
v1 grades: facts (entity, value), equations (the typed relation shape), and the
|
|
question target (symbol, state-index, question-form). Unit modelling is intentionally
|
|
NOT in the signature yet — it is covered by the admissibility tests, and a future
|
|
extension adds an expected-unit axis once the gold carries it.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import Any
|
|
|
|
from generate.binding_graph.model import SemanticSymbolicBindingGraph
|
|
|
|
|
|
def relation_signature(relations: list[dict[str, Any]]) -> tuple[tuple, ...]:
|
|
"""Canonicalize a list of relations (the ``to_relational_metric`` / gold shape)
|
|
into a sorted, order-independent tuple of typed relation tuples."""
|
|
out: list[tuple] = []
|
|
for r in relations:
|
|
kind = r["kind"]
|
|
if kind == "fact":
|
|
out.append(("fact", r["entity"], int(r["value"])))
|
|
elif kind in ("more_than", "fewer_than"):
|
|
out.append((kind, r["entity"], r["ref"], int(r["delta"])))
|
|
elif kind == "sum_of":
|
|
out.append(("sum_of", r["entity"], tuple(sorted(r["parts"]))))
|
|
else: # an unknown relation kind is itself a structural difference, not a crash
|
|
out.append(("unhandled_kind", kind, r.get("entity", "")))
|
|
return tuple(sorted(out, key=repr))
|
|
|
|
|
|
def gold_unknown_signature(
|
|
relations: list[dict[str, Any]], query: dict[str, Any]
|
|
) -> tuple[str, str, str]:
|
|
"""The expected question-target signature, derived from the INDEPENDENT gold.
|
|
|
|
A query whose target is an aggregate (the gold contains a ``sum_of`` producing it)
|
|
is a ``total`` form; otherwise a ``count``. All current cases ask the terminal state.
|
|
"""
|
|
form = "total" if any(r["kind"] == "sum_of" for r in relations) else "count"
|
|
return (query["entity"], "terminal", form)
|
|
|
|
|
|
def _state_token(state_index: Any) -> str:
|
|
if isinstance(state_index, str):
|
|
return state_index
|
|
# An Operation state-index (ADR-0135) — name it by its operation index.
|
|
return f"op{getattr(state_index, 'operation_index', '?')}"
|
|
|
|
|
|
def reader_unknown_signature(graph: SemanticSymbolicBindingGraph) -> tuple[str, str, str]:
|
|
"""The reader's question-target signature from ``graph.unknowns`` (PR-1).
|
|
|
|
A graph that does not carry exactly one unknown is itself a structural defect — it
|
|
is reported as a distinguished ``MALFORMED`` signature so it can never silently
|
|
compare equal to a well-formed gold target.
|
|
"""
|
|
unknowns = graph.unknowns
|
|
if len(unknowns) != 1:
|
|
return ("MALFORMED", str(len(unknowns)), "")
|
|
u = unknowns[0]
|
|
return (u.symbol_id, _state_token(u.state_index), u.question_form)
|
|
|
|
|
|
__all__ = [
|
|
"gold_unknown_signature",
|
|
"reader_unknown_signature",
|
|
"relation_signature",
|
|
]
|