core/evals
Shay 59974865ef feat(comprehension): question target in the graph (PR-1) + setup-oracle lane (grade the reading)
Two coupled, additive, off-serving changes toward the typed math-comprehension organ.
No serving path touched; the relational_metric answer lane stays 15/15 wrong=0.

PR-1 — QuantQuery → BoundUnknown. comprehend_quantitative now emits the question
target as a BoundUnknown INSIDE the binding-graph (symbol_id, state_index="terminal",
question_form "count"|"total", expected_unit), so the graph is a real question-bearing
mathematical object and its canonical serialization carries the target. The external
QuantQuery is RETAINED, consistent-by-construction, so the two consumers
(to_relational_metric, realize/quantitative) are byte-identical; a follow-up rewires
them onto graph.unknowns and drops the duplicate field.

Setup-oracle lane (evals/setup_oracle) — grade the READING, not the answer. The
relational_metric lane scores answers, which can bless a semantically-wrong derivation
that coincidentally lands on the right number (the exact hazard the held-out
measurements + the 2/87 resolve_pooled probe exposed). The setup-oracle compares the
reader's comprehended STRUCTURE — a span-free signature of facts + typed equations +
the BoundUnknown target — against the INDEPENDENT gold structure (the relational_metric
cases' own relations/query, authored separately from the binding-graph reader). A
structural mismatch is setup_wrong, the wrong=0-critical count, even when the answer
would be right. v1 grades structure (units deferred — covered by admissibility). The
reader reads all 15 cases with the gold structure (setup_wrong=0); a meaningful-fail
test proves the oracle catches a right-answer/wrong-structure reading (it is not
decoration). `python -m evals.setup_oracle` exits nonzero iff setup_wrong > 0.

This is the measurement rig BEFORE investing in frame families: setup_wrong=0 is the
gate; serving must not move while setup_wrong > 0. It is the first milestone of the
math-comprehension organ, not a path to "solve GSM8K".

Verified: setup-oracle 15/15 setup_correct wrong=0; quantitative + setup-oracle unit
tests (17); realize-binding-graph + binding-graph + architectural invariants (183).
2026-06-06 16:40:15 -07:00
..
adversarial_identity
anchor_lens_tour
anti_regression
articulation
articulation_of_status
audio_sensorium
audit_tour
calibration
capability_index
classical_literature_ood
cognition
cold_start_grounding
compositionality
compound_intent_decomposition
comprehension
contemplation_quality
contradiction_detection
conversation
conversational_thread_coherence
cross_domain_transfer
curriculum_loop_closure
deductive_logic
demo_composition
determination_closure
determination_estimation
deterministic_fluency
dimensional
discourse_paragraph
domain_contract_validation
edge_budget
elementary_mathematics_ood
english_fluency_ood
environment_falsification
event_vision_sensorium
fabrication_control
flywheel_demo
forward_semantic_control
foundational_biology_ood
foundational_physics_ood
frontier_compare
grammatical_coverage
gsm8k_math
gsm8k_parser_dev
hebrew_fluency
identity_divergence
industry_demos
inference_closure
introspection
koine_greek_fluency
l10_continuity
lab
learning_arc
learning_loop
long_context_cost
math_bounded_grammar/v1
math_capability_axes
math_expert_claims/v1
math_symbolic_equivalence
math_teaching_corpus/v1
miner_loop_closure
monotonic_learning
multi_agent_composition
multi_sentence_response
multi_step_reasoning
obligation_2_ood_ratio
obligation_5_perturbation
obligation_6_depth_curve
obligation_8_adversarial
obligation_10_pack_provenance
orthogonality_tour
prompt_diversity
propositional_logic
provenance
public_demo
realizer_guard
refusal_calibration
refusal_taxonomy
register_diagnostics
register_tour
relational
relational_metric
reports
results
reviewer_registry
sample_efficiency
self_consistency_over_time
sensorimotor_sensorium
sensorium
set_membership
setup_oracle feat(comprehension): question target in the graph (PR-1) + setup-oracle lane (grade the reading) 2026-06-06 16:40:15 -07:00
syllogism
symbolic_logic
teaching_injection_resistance
total_ordering
vision_sensorium
walkthrough_chain
warmed_session_consistency
zero_code_domain_acquisition
__init__.py
_parallel.py
baseline_runner.py
CLAIMS.md
cognition_cases.jsonl
framework.py
holdout_runner.py
metrics.py
parallel.py
run_cognition_eval.py