core/evals
Shay 4b9404a88e
feat(adr-0085): gloss-aware CAUSE composer — explanation frame from glosses (#70)
The original "Why does light exist?" complaint that motivated ADR-0084
was specifically about CAUSE-intent surfaces. ADR-0084 (substrate) +
PR #65 (content) already moved DEFINITION/RECALL to gloss-grounded
surfaces ("Light is visible medium that reveal truth."). But CAUSE
still dispatched through the chain-walk path:

  Before: light — teaching-grounded (cognition_chains_v1):
            cognition.illumination; logos.core.
            light reveals truth (cognition.truth).
            No session evidence yet.

  After:  Light exists as visible medium that reveal truth.
          pack-grounded (en_core_cognition_v1).

The chain-walk is structurally correct but the wrong SHAPE for a why-
question — it's a graph traversal, not an explanation. ADR-0085 fixes
the shape using the same gloss material that DEFINITION/RECALL already
consume, with no new content authoring.

Additive composer
  chat/pack_grounding.py:gloss_aware_cause_surface()
  - Resolves gloss via lexicon-residency-checked resolve_gloss().
  - Frames POS-aware:
      NOUN -> "{Lemma} exists as {gloss}."
      VERB -> "To {lemma} is to {gloss}."
      ADJ  -> "To be {lemma} is to {gloss}."
      *    -> falls back to _frame_gloss (predicate-identity).
  - Threads anchor lens via the existing helper (ADR-0073c parity).
  - Returns None when no gloss exists — runtime falls through to the
    existing chain-walk path. Additive: no CAUSE case loses its surface.

Runtime dispatch
  chat/runtime.py — IntentTag.CAUSE tries gloss path FIRST under the
  flag; falls through to teaching_grounded_surface* on None.
  Unconditional fallback — never silent.

Opt-in flag
  core/config.py — RuntimeConfig.gloss_aware_cause: bool = False
  Default off preserves pre-ADR-0085 chain-walk surfaces byte-
  identically (null-drop invariant, CI-pinned).

Prompt-diversity classifier update
  evals/prompt_diversity/runner.py — _CAUSE_MARKERS widened with the
  explanation-frame markers ("exists as", "is to", "to be", "is for",
  "purpose of") plus bare-form predicates ("reveal" alongside
  "reveals"). Neither composer path is penalised on shape_fit just on
  inflection grounds.

v1/public lift (flag OFF vs ON, 26 cases)
  intent_accuracy        : 65.4% -> 65.4%   ( — )
  versor_closure_rate    : 100.0% -> 100.0% ( — )
  response_shape_fit     : 57.7% -> 57.7%   ( — , both frames recognized)
  audit_in_surface_rate  : 42.3% -> 42.3%   ( — , envelope ADR's job)
  gloss_quote_rate       : 11.5% -> 23.1%   (+11.5pp, structural lift)

Tests (15)
  - 5 pure composer (NOUN/VERB frame, unknown/empty None, no chain-
    walk artifacts in surface)
  - 5 runtime dispatch (flag-off chain-walk, flag-on gloss, parametrized
    across glossed subjects, VERIFICATION unchanged under flag, no-
    gloss fallback engages)
  - 5 cognition lane invariance (aggregate metrics byte-identical
    under both flag states; surfaces deliberately shift on the 2 CAUSE
    cases with glossed subjects — the structural-change-vs-metric-
    invariance both-sides invariant)

Lanes
  smoke 67/0, cognition 120/0/1 skipped, packs 6/0, teaching 17/0,
  runtime 19/0. core eval cognition byte-identical 100/91.7/100/100
  under both flag states.

Scope limits (per ADR §Scope limits)
  - CAUSE only; VERIFICATION still chain-walks (different shape).
  - English pilot only; Greek/Hebrew packs not opted into definitional
    layer yet (ADR-0084 scope limit).
  - Single-lemma subjects; compound/anaphoric fall through.
  - Opt-in until cognition holdout confirms the lift transfers off-
    fixture. Future PR flips default on.

Out of scope
  - Surface-vs-envelope cleanup ("pack-grounded (...)" still leaks).
  - Predicate licensing (ADR-0086).
  - Content style pass (bare lemma forms in glosses — separate brief).
2026-05-20 15:55:08 -07:00
..
adversarial_identity docs(identity): empirical finding — fix #3 needs upstream ingest-gate work 2026-05-16 14:23:20 -07:00
anchor_lens_tour parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
anti_regression chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
articulation feat(demo): core demo articulation — discourse-planner spine, end-to-end 2026-05-19 13:41:24 -07:00
articulation_of_status feat(epistemic): realizer-side closure — refusal_calibration + articulation_of_status graduate 2026-05-17 10:12:59 -07:00
audit_tour feat(adr-0042): audit-tour demo — pack-layer story in four scenes 2026-05-17 22:06:45 -07:00
calibration fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
classical_literature_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
cognition parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
cold_start_grounding feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
compositionality feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
compound_intent_decomposition feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
contradiction_detection feat(epistemic): contradiction coherence checker — CONTESTED transitions wired, last Tier 4.5 row closes 2026-05-17 10:36:48 -07:00
conversation feat(demo): humanise teaching-grounded surface for layperson display 2026-05-19 14:14:02 -07:00
conversational_thread_coherence feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
cross_domain_transfer feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
deterministic_fluency feat(evals): deterministic_fluency lane — six structural predicates 2026-05-19 07:16:44 -07:00
discourse_paragraph feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
elementary_mathematics_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
english_fluency_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
forward_semantic_control docs(cli): self-explanatory demos — preambles + per-directory READMEs 2026-05-17 16:39:50 -07:00
foundational_biology_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
foundational_physics_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
frontier_compare feat(frontier): add replay variability suite and token-cost telemetry (#66) 2026-05-20 15:04:34 -07:00
grammatical_coverage fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
hebrew_fluency feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep 2026-05-16 20:59:31 -07:00
identity_divergence feat(adr-0043): Phase-2 pack measurements — claims → numbers 2026-05-17 22:19:24 -07:00
industry_demos fix(adr-0046): make forward-graph-constraint branch mergeable 2026-05-18 05:57:46 -07:00
inference_closure feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
introspection feat(phase3): core/cognition/explain.py — close Gap 3 introspection 2026-05-16 15:09:48 -07:00
koine_greek_fluency feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep 2026-05-16 20:59:31 -07:00
lab research(evals): phi separation probe for ADR-0081 follow-up (#57) 2026-05-20 12:34:59 -07:00
learning_loop chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
long_context_cost chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
monotonic_learning feat(evals): v3 lanes — monotonic-learning passes, adversarial-identity reveals gap 2026-05-16 13:42:47 -07:00
multi_agent_composition fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
multi_sentence_response feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
multi_step_reasoning feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
orthogonality_tour parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
prompt_diversity feat(adr-0085): gloss-aware CAUSE composer — explanation frame from glosses (#70) 2026-05-20 15:55:08 -07:00
provenance feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
realizer_guard chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
refusal_calibration feat(adr-0043): Phase-2 pack measurements — claims → numbers 2026-05-17 22:19:24 -07:00
register_tour feat(register): ADR-0077 — substantive register knobs + layering boundary (R6) 2026-05-19 23:39:11 -07:00
reports feat(adr-0023): Forward Semantic Control proof evidence — Accepted 2026-05-17 12:55:19 -07:00
results chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
sample_efficiency feat(phase4): sample-efficiency v1 — first quantitative-curve lane 2026-05-16 15:39:28 -07:00
self_consistency_over_time feat(runtime+evals): warm-path pack grounding + three long-span lanes 2026-05-19 08:26:38 -07:00
symbolic_logic feat(evals): v2 lanes for calibration and symbolic-logic 2026-05-16 13:17:41 -07:00
teaching_injection_resistance feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants 2026-05-17 07:27:41 -07:00
walkthrough_chain feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
warmed_session_consistency feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
zero_code_domain_acquisition
__init__.py
_parallel.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
baseline_runner.py feat(evals): frontier structural-zero baselines for Phase 2 v1 lanes 2026-05-16 12:45:28 -07:00
CLAIMS.md chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
cognition_cases.jsonl
framework.py fix(evals): introspect run_lane signature before passing workers kwarg 2026-05-20 05:59:51 -07:00
holdout_runner.py
metrics.py
parallel.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
run_cognition_eval.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00