core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/evals
History
Shay 7cc2888ed2 feat(coherence): ADR-0075 — realizer slot-type guard (C1) 
C1 coherence floor: a deterministic verifier that runs on every
candidate surface produced by the truth path, before assignment to
ChatResponse.surface.  Rejects illegal articulations and routes them
to a bounded disclosure string — admission control with a
deterministic fallback, not normalization.

Active rules (R1 deferred during ratification — see ADR):
  R2_aux_neg_requires_verb     — "<aux> not <wrong-POS>"  rejected
  R3_be_neg_requires_predicate — "<be>  not <verb>"       rejected

Fail-open on unknown POS, fail-closed on explicit wrong POS.
Cognition eval byte-identical (100/91.7/100/100).

Original bug class — "Light reveals truth, right?" → "Right does not
thought." — now routes to "I do not have a reviewed articulation for
that yet." with grounding_source=none, walk_surface preserving the
rejected candidate, and telemetry carrying R2_aux_neg_requires_verb.

Files:
  generate/realizer_guard.py            NEW — pure verifier
  chat/runtime.py                       hook on stub + main paths
  chat/telemetry.py                     serialize guard fields
  core/physics/identity.py              TurnEvent +2 fields
  evals/realizer_guard/run_holdout.py   NEW — 6-prompt cluster
  tests/test_realizer_guard_*.py        NEW — 46 tests (unit/seam/holdout)
  docs/decisions/ADR-0075-*.md          NEW — ratified

Invariants pinned:
  invariant_realizer_no_illegal_articulation
  invariant_realizer_guard_byte_identity_on_currently_passing_cases

Lanes (excluding 1 pre-existing TestDemoPreambles failure unrelated
to C1, already present at 4426f38):
  smoke 67/67  cognition 120/120(+1s)  teaching 17/17
  packs 6/6   runtime 19/19   algebra 132/132   full 2792/2793
2026-05-19 22:35:09 -07:00
..
adversarial_identity docs(identity): empirical finding — fix #3 needs upstream ingest-gate work 2026-05-16 14:23:20 -07:00
anchor_lens_tour feat(anchor_lens): ADR-0073d — L1.4 telemetry, CLI flag, tour demo 2026-05-19 20:21:41 -07:00
anti_regression feat(adr-0057): anti-regression demo — three-gate defense against learning harm 2026-05-18 10:52:23 -07:00
articulation feat(demo): core demo articulation — discourse-planner spine, end-to-end 2026-05-19 13:41:24 -07:00
articulation_of_status feat(epistemic): realizer-side closure — refusal_calibration + articulation_of_status graduate 2026-05-17 10:12:59 -07:00
audit_tour feat(adr-0042): audit-tour demo — pack-layer story in four scenes 2026-05-17 22:06:45 -07:00
calibration fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
classical_literature_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
cognition feat(adr-0060): correction acknowledgement carries corrected-topic lemma 2026-05-18 14:14:27 -07:00
cold_start_grounding feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
compositionality feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
compound_intent_decomposition feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
contradiction_detection feat(epistemic): contradiction coherence checker — CONTESTED transitions wired, last Tier 4.5 row closes 2026-05-17 10:36:48 -07:00
conversation feat(demo): humanise teaching-grounded surface for layperson display 2026-05-19 14:14:02 -07:00
conversational_thread_coherence feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
cross_domain_transfer feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
deterministic_fluency feat(evals): deterministic_fluency lane — six structural predicates 2026-05-19 07:16:44 -07:00
discourse_paragraph feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
elementary_mathematics_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
english_fluency_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
forward_semantic_control docs(cli): self-explanatory demos — preambles + per-directory READMEs 2026-05-17 16:39:50 -07:00
foundational_biology_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
foundational_physics_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
grammatical_coverage fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
hebrew_fluency feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep 2026-05-16 20:59:31 -07:00
identity_divergence feat(adr-0043): Phase-2 pack measurements — claims → numbers 2026-05-17 22:19:24 -07:00
industry_demos fix(adr-0046): make forward-graph-constraint branch mergeable 2026-05-18 05:57:46 -07:00
inference_closure feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
introspection feat(phase3): core/cognition/explain.py — close Gap 3 introspection 2026-05-16 15:09:48 -07:00
koine_greek_fluency feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep 2026-05-16 20:59:31 -07:00
learning_loop feat(adr-0064): cross-pack teaching chains + relations_chains_v1 seed (Phase 1.3+1.4) 2026-05-18 16:04:20 -07:00
long_context_cost feat(adr-0044, adr-0045): domain ethics pack + long-context comparison 2026-05-17 22:31:47 -07:00
monotonic_learning feat(evals): v3 lanes — monotonic-learning passes, adversarial-identity reveals gap 2026-05-16 13:42:47 -07:00
multi_agent_composition fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
multi_sentence_response feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
multi_step_reasoning feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
orthogonality_tour feat(demo): ADR-0074 — orthogonality tour (anchor-lens × register) 2026-05-19 20:33:33 -07:00
provenance feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
realizer_guard feat(coherence): ADR-0075 — realizer slot-type guard (C1) 2026-05-19 22:35:09 -07:00
refusal_calibration feat(adr-0043): Phase-2 pack measurements — claims → numbers 2026-05-17 22:19:24 -07:00
register_tour feat(register): R5 — operator-visible register telemetry + tour demo 2026-05-19 19:03:07 -07:00
reports feat(adr-0023): Forward Semantic Control proof evidence — Accepted 2026-05-17 12:55:19 -07:00
results chore(evals): refresh pack-measurements artifact to current runtime 2026-05-19 19:16:33 -07:00
sample_efficiency feat(phase4): sample-efficiency v1 — first quantitative-curve lane 2026-05-16 15:39:28 -07:00
self_consistency_over_time feat(runtime+evals): warm-path pack grounding + three long-span lanes 2026-05-19 08:26:38 -07:00
symbolic_logic feat(evals): v2 lanes for calibration and symbolic-logic 2026-05-16 13:17:41 -07:00
teaching_injection_resistance feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants 2026-05-17 07:27:41 -07:00
walkthrough_chain feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
warmed_session_consistency feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
zero_code_domain_acquisition feat began creation of zero code domain acquisition. did not complete yet. 2026-05-16 06:31:01 -07:00
__init__.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
baseline_runner.py feat(evals): frontier structural-zero baselines for Phase 2 v1 lanes 2026-05-16 12:45:28 -07:00
CLAIMS.md feat(bench): bench cost — $/1000 turns + latency, with disclosed assumptions 2026-05-17 10:53:08 -07:00
cognition_cases.jsonl feat: vault recall index, Rust versor parity, cognitive pack expansion 2026-05-15 15:34:39 -07:00
framework.py feat(adr-0054): vault recall indexing/batching + holdout split wired 2026-05-18 07:58:57 -07:00
holdout_runner.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
metrics.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
parallel.py feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
run_cognition_eval.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00