core/evals
Shay e831ed2615 feat(comprehend): complete 3-domain comprehension organ (syllogism + total_ordering)
Phase 2a r2/r3/r4 of the redefined plan: the general comprehension reader now
reads THREE independent-gold reasoning domains end-to-end (prose -> MeaningGraph
-> projection -> independent oracle -> answer vs gold), all wrong=0, and all
three are wired into the capability index.

reader.py — new domain-agnostic templates (function words + order; parse-or-refuse):
  - categorical E/I/O: "no Xs are Ys"->disjoint, "some Xs are Ys"->intersects,
    "some Xs are not Ys"->some_not  (A "all Xs are Ys"->subset already existed)
  - "therefore <categorical>" -> conclusion QUERY (same neutral predicate vocab)
  - comparative facts: "<X> [is] <comp> [than] <Y>" -> less(...), closed
    less/greater comparator lexicon, elided-copula support
  - sort query ("sort ascending|descending", "... order from <low> to <high>")
    and compare query ("compare <X> with <Y>")
  - clause-splitting on commas / leading and|or for multi-clause sentences

projectors.py — to_syllogism (premises + validity conclusion, finite-model size 3)
  and to_total_ordering (less-facts + sort/compare). Both return None when nothing
  is honestly askable of their oracle (caller treats as refusal).

capability_index — wire 3 comprehension lanes into ADAPTERS; re-freeze baseline
  breadth 3->6, capability_score 0.919641->0.814356 (geomean falls BY DESIGN as
  honest partial-coverage domains join; wrong_total stays 0). digest 0a98b9b4...

Scores: set_membership 8/8, syllogism 6/8, total_ordering 4/8 — all wrong=0.

Multi-word NP handling is DEFERRED on purpose, not missed: the gold lanes
canonicalize multi-word NPs three contradictory ways ("North station"->"north",
"Level one"->"level_one", "metal objects"->"metal"), so no single general rule is
wrong=0-safe. The reader refuses multi-word NPs until the gold lanes carry a
canonicalization contract. Every refusal is a genuine harder phenomenon
(multi-word NP, adjectival predicate, trailing tokens) — never a readable case
silently dropped.

Tests: reader templates, projector unit tests, syllogism/total_ordering
end-to-end wrong=0 with pinned counts, capability breadth 3->6. 138 targeted +
87 smoke green. Lane SHAs 8/9 (sole miss = public_demo env wall-clock flake).
2026-06-05 21:02:43 -07:00
..
adversarial_identity
anchor_lens_tour
anti_regression feat(epistemic): Phase 3 state tagging spine (#220) 2026-05-24 11:26:06 -07:00
articulation
articulation_of_status
audio_sensorium feat(adr-0181-p4): audio compiler eval gate lane (sensorium/audio) (#470) 2026-05-29 11:20:31 -07:00
audit_tour
calibration
capability_index feat(comprehend): complete 3-domain comprehension organ (syllogism + total_ordering) 2026-06-05 21:02:43 -07:00
classical_literature_ood
cognition
cold_start_grounding fix(quarantine): drain all 60 quarantined tests — QUARANTINE=∅ (#267) 2026-05-25 11:22:12 -07:00
compositionality
compound_intent_decomposition
comprehension feat(comprehend): complete 3-domain comprehension organ (syllogism + total_ordering) 2026-06-05 21:02:43 -07:00
contemplation_quality feat(W-025): contemplation quality eval lane (ADR-0159) (#286) 2026-05-25 20:38:52 -07:00
contradiction_detection
conversation
conversational_thread_coherence
cross_domain_transfer
curriculum_loop_closure
deductive_logic test(l10): add independent-gold adversarial logic fixtures 2026-06-05 09:08:23 -07:00
demo_composition fix(W-015): replace _slerp_toward with rotor-geodesic anchor pull (#255) 2026-05-24 20:05:25 -07:00
deterministic_fluency
dimensional feat: dimensional-reasoning lane — 3rd diversity-panel domain 2026-06-04 16:38:56 -07:00
discourse_paragraph
domain_contract_validation chore(ci): re-pin drifted lane SHAs + refresh canonical reports (#229) 2026-05-24 14:25:11 -07:00
elementary_mathematics_ood
english_fluency_ood
fabrication_control chore(ci): re-pin drifted lane SHAs + refresh canonical reports (#229) 2026-05-24 14:25:11 -07:00
flywheel_demo feat(demo): core demo flywheel — public-proof reproduction of the loop 2026-05-27 21:33:54 -07:00
forward_semantic_control
foundational_biology_ood
foundational_physics_ood
frontier_compare
grammatical_coverage
gsm8k_math feat: add gsm8k r1 reconstruction 2026-06-04 13:25:11 -07:00
gsm8k_parser_dev
hebrew_fluency
identity_divergence
industry_demos
inference_closure
introspection
koine_greek_fluency
l10_continuity feat(persistence): Shape B+ Phase D+E — opt-in lived-state persistence; reboot transparent 2026-06-05 13:17:30 -07:00
lab
learning_arc fix(W-025): polish contemplation-quality eval lane follow-ups (#290) 2026-05-26 09:39:18 -07:00
learning_loop
long_context_cost
math_bounded_grammar/v1
math_capability_axes chore(evals): refresh stale committed reports 2026-06-03 01:01:50 -07:00
math_expert_claims/v1 docs(claims): ADR-0200 reconciliation — expert claim to audit-passed truth 2026-06-02 10:06:16 -07:00
math_symbolic_equivalence
math_teaching_corpus/v1
miner_loop_closure
monotonic_learning
multi_agent_composition
multi_sentence_response
multi_step_reasoning
obligation_2_ood_ratio feat(ADR-0114a.2): OOD-ratio auditor — Obligation #2 wired for B3, ratio=1.00 (#193) 2026-05-23 16:25:28 -07:00
obligation_5_perturbation
obligation_6_depth_curve feat(ADR-0114a.6): depth-curve auditor — Obligation #6 wired for B3 (assertion holds, coverage gap named) (#190) 2026-05-23 16:19:58 -07:00
obligation_8_adversarial
obligation_10_pack_provenance
orthogonality_tour
prompt_diversity
provenance
public_demo chore(ratify): accept four Phase C round-2 recognizers (round 2) (#309) 2026-05-26 16:03:01 -07:00
realizer_guard
refusal_calibration feat(identity): L11 identity continuity — same identity across reboot, not just same bytes 2026-06-05 13:52:57 -07:00
refusal_taxonomy fix(refusal-taxonomy): parse new refusal-reason format; reconcile to reader gains 2026-06-03 01:49:41 -07:00
register_diagnostics
register_tour
relational_metric feat(field-wedge): ablation verdict — field is decoration on additive (C3) (Phase W.2) 2026-06-04 19:44:22 -07:00
reports
results
reviewer_registry
sample_efficiency
self_consistency_over_time
sensorimotor_sensorium Add sensorium eval and governance runway 2026-06-03 20:53:05 -07:00
sensorium Add sensorium eval and governance runway 2026-06-03 20:53:05 -07:00
set_membership evals: add staged independent gold lanes 2026-06-05 16:03:26 -07:00
syllogism evals: add staged independent gold lanes 2026-06-05 16:03:26 -07:00
symbolic_logic
teaching_injection_resistance
total_ordering evals: add staged independent gold lanes 2026-06-05 16:03:26 -07:00
vision_sensorium Add vision evidence and sensorimotor contracts 2026-06-03 20:27:46 -07:00
walkthrough_chain
warmed_session_consistency
zero_code_domain_acquisition
__init__.py
_parallel.py
baseline_runner.py
CLAIMS.md
cognition_cases.jsonl
framework.py
holdout_runner.py
metrics.py
parallel.py
run_cognition_eval.py feat(W-008): L10 Shape B hybrid engine-state persistence (#271) 2026-05-25 11:45:54 -07:00