core/evals
Shay 4ddaef4d00 feat(constraint): two-category reader — prose to ConstraintProblem (R2 C5-C9)
generate/constraint_comprehension/reader.py: read_constraint_problem comprehends two-category constraint prose into a typed ConstraintProblem via five recognizers — C5 category-pair (from coefficient clauses; >2 -> too_many_categories), C6 coefficient (Each <cat> holds/has/costs/is-worth <N> <unit>; shared measured unit), C7 total-count (collective-unit sentence -> x+y=N; absent -> missing_total_count), C8 weighted-total (measured-unit sentence -> ax+by=T; absent -> missing_weighted_total), C9 query-target (How many <cat> -> the asked unknown). Refuses, never mis-assembles.

Reconciles the sketch's 'no equal coefficients' note: equal coefficients are a SOLVER refusal (indistinguishable_weights), not a reader one — the reader reads r2-010 setup_correct and the solver refuses it. run_reader lane + 'reader' CLI subcommand. R2 reader: setup_correct 10 / setup_wrong 0 / refused_correct 3; R1 7/0/3 and 15-case 15/0/0 unchanged. Off-serving. 8 reader tests incl. full read->solve->verify chain + meaningful-fail guards.
2026-06-07 07:34:55 -07:00
..
adversarial_identity
anchor_lens_tour
anti_regression
articulation
articulation_of_status
audio_sensorium
audit_tour
calibration
capability_index
classical_literature_ood
cognition
cold_start_grounding
compositionality
compound_intent_decomposition
comprehension
constraint_oracle
contemplation_quality
contradiction_detection
conversation
conversational_thread_coherence
cross_domain_transfer
curriculum_loop_closure
deductive_logic
demo_composition
determination_closure
determination_estimation
deterministic_fluency
dimensional
discourse_paragraph
domain_contract_validation
edge_budget
elementary_mathematics_ood
english_fluency_ood
environment_falsification
event_vision_sensorium
fabrication_control
flywheel_demo
forward_semantic_control
foundational_biology_ood
foundational_physics_ood
frontier_compare
grammatical_coverage
gsm8k_math
gsm8k_parser_dev
hebrew_fluency
identity_divergence
industry_demos
inference_closure
introspection
koine_greek_fluency
l10_continuity
lab
learning_arc
learning_loop
long_context_cost
math_bounded_grammar/v1
math_capability_axes
math_expert_claims/v1
math_symbolic_equivalence
math_teaching_corpus/v1
miner_loop_closure
monotonic_learning
multi_agent_composition
multi_sentence_response
multi_step_reasoning
obligation_2_ood_ratio
obligation_5_perturbation
obligation_6_depth_curve
obligation_8_adversarial
obligation_10_pack_provenance
orthogonality_tour
prompt_diversity
propositional_logic
provenance
public_demo
realizer_guard
refusal_calibration
refusal_taxonomy
register_diagnostics
register_tour
relational
relational_metric
reports
results
reviewer_registry
sample_efficiency
self_consistency_over_time
sensorimotor_sensorium
sensorium
set_membership
setup_oracle
syllogism
symbolic_logic
teaching_injection_resistance
total_ordering
vision_sensorium
walkthrough_chain
warmed_session_consistency
zero_code_domain_acquisition
__init__.py
_parallel.py
baseline_runner.py
CLAIMS.md
cognition_cases.jsonl
framework.py
holdout_runner.py
metrics.py
parallel.py
run_cognition_eval.py