core-labs/core
core-labs/core
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
core/evals
History
Shay 3bda4313c9 feat: ADR-0119.5 — adversarial generation (closes ADR-0114a Obligation #8) 
Phase 5.5 of ADR-0119. Adversarial case generator + scoring CLI;
discharges the last remaining ADR-0114a obligation.

Numbers
  adversarial suite: 38 cases × 12 families
  per-family: every family produces wrong == 0
  overall: correct 5, wrong 0, refused 33

Families
  conditional_phrasing       (4)  "If/When/Suppose ..."
  compound_questions         (3)  multiple ?
  undefined_entity_question  (3)  question references unknown entity
  unknown_verb               (5)  "polishes", "admires", etc.
  empty_or_whitespace        (3)  empty input
  no_question                (3)  statement-only
  numbers_spelled_out        (3)  "five", "ten"
  passive_voice              (3)  "X are bought by Y"
  red_herring_numbers        (3)  digits in name positions, mid-quantity
  question_only              (2)  no preceding statements
  mid_sentence_punctuation   (2)  embedded ? or !
  subtle_in_grammar          (4)  IN-grammar; runner must produce correct
                                  (gate-sanity: not trivially "refuse all")

The subtle_in_grammar family is the load-bearing sanity check —
proves the gate isn't trivially satisfied by refusing everything.

ADR-0114a obligation status

  10 of 10 discharged on main:
    #1  fab_control lane (0119.1); GSM8K test pending (0119.7)
    #2  ADR-0118a
    #3  ADR-0117
    #4  ADR-0116 + ADR-0119.3
    #5  ADR-0125
    #6  ADR-0119.6 harness; ε threshold to ADR-0120
    #7  ADR-0119.4
    #8  THIS ADR
    #9  ADR-0116/0117/0118/0119.3
    #10 ADR-0116

Phase 5 remaining: 5.7 (sealed GSM8K test, real corpus) and 5.8
(overall lane gate). After those, ADR-0120 (first expert promotion
contract) can compose all ten obligations.

Tests: 18 new + 25 prior Phase 5 = 43 green; 67/67 smoke.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 18:11:36 -07:00
..
adversarial_identity docs(identity): empirical finding — fix #3 needs upstream ingest-gate work 2026-05-16 14:23:20 -07:00
anchor_lens_tour parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
anti_regression chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
articulation feat(demo): core demo articulation — discourse-planner spine, end-to-end 2026-05-19 13:41:24 -07:00
articulation_of_status feat(epistemic): realizer-side closure — refusal_calibration + articulation_of_status graduate 2026-05-17 10:12:59 -07:00
audit_tour feat(adr-0042): audit-tour demo — pack-layer story in four scenes 2026-05-17 22:06:45 -07:00
calibration fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
classical_literature_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
cognition parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
cold_start_grounding feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
compositionality feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
compound_intent_decomposition feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
contradiction_detection feat(epistemic): contradiction coherence checker — CONTESTED transitions wired, last Tier 4.5 row closes 2026-05-17 10:36:48 -07:00
conversation feat(demo): humanise teaching-grounded surface for layperson display 2026-05-19 14:14:02 -07:00
conversational_thread_coherence feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
cross_domain_transfer feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
curriculum_loop_closure feat(evals): ADR-0105 — sealed holdout encryption via age (#108) 2026-05-22 10:09:43 -07:00
demo_composition feat(demos): implement ADR-0098 — Demo Composition Contract 2026-05-21 19:02:29 -07:00
deterministic_fluency feat(evals): deterministic_fluency lane — six structural predicates 2026-05-19 07:16:44 -07:00
discourse_paragraph feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
domain_contract_validation feat(capability): implement ADR-0093 — Domain Pack Contract v1 wired in 2026-05-21 18:33:23 -07:00
elementary_mathematics_ood feat(capability): ADR-0110 promote mathematics_logic to expert_demo (#118) 2026-05-22 12:59:23 -07:00
english_fluency_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
fabrication_control feat: ADR-0119.1 — seal fabrication_control holdout with age encryption (Obligation #1) 2026-05-22 17:22:46 -07:00
forward_semantic_control docs(cli): self-explanatory demos — preambles + per-directory READMEs 2026-05-17 16:39:50 -07:00
foundational_biology_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
foundational_physics_ood feat: ADR-0111 physics expert-demo promotion (second successful) 2026-05-22 14:37:36 -07:00
frontier_compare feat(contemplation): land ADR-0080 phase 1 (#119) 2026-05-22 13:10:03 -07:00
grammatical_coverage fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
gsm8k_math feat: ADR-0119.5 — adversarial generation (closes ADR-0114a Obligation #8) 2026-05-22 18:11:36 -07:00
gsm8k_parser_dev feat: add ADR-0125 perturbation suite 2026-05-22 17:12:33 -07:00
hebrew_fluency feat(packs): ADR-0103 — attach hebrew_fluency + koine_greek_fluency lanes to ADR-0102 (#106) 2026-05-22 09:43:46 -07:00
identity_divergence feat(adr-0043): Phase-2 pack measurements — claims → numbers 2026-05-17 22:19:24 -07:00
industry_demos evals/industry_demos: add run_all.py suite runner 2026-05-21 08:23:29 -07:00
inference_closure feat: ADR-0122 systems_software audit-passed deferred (lane-shape mismatch) 2026-05-22 16:31:59 -07:00
introspection feat(phase3): core/cognition/explain.py — close Gap 3 introspection 2026-05-16 15:09:48 -07:00
koine_greek_fluency feat(packs): ADR-0103 — attach hebrew_fluency + koine_greek_fluency lanes to ADR-0102 (#106) 2026-05-22 09:43:46 -07:00
lab research(evals): phi separation probe for ADR-0081 follow-up (#57) 2026-05-20 12:34:59 -07:00
learning_loop chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
long_context_cost chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
miner_loop_closure feat(teaching): implement ADR-0095 — Miner-Sourced Teaching Proposals 2026-05-21 18:18:51 -07:00
monotonic_learning feat(evals): v3 lanes — monotonic-learning passes, adversarial-identity reveals gap 2026-05-16 13:42:47 -07:00
multi_agent_composition fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
multi_sentence_response feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
multi_step_reasoning feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
orthogonality_tour parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
prompt_diversity feat(adr-0085): gloss-aware CAUSE composer — explanation frame from glosses (#70) 2026-05-20 15:55:08 -07:00
provenance feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
public_demo feat: ADR-0111 physics expert-demo promotion (second successful) 2026-05-22 14:37:36 -07:00
realizer_guard chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
refusal_calibration feat(adr-0043): Phase-2 pack measurements — claims → numbers 2026-05-17 22:19:24 -07:00
register_diagnostics feat(evals): per-intent register-firing diagnostic + CI gate + tests (#103) 2026-05-21 07:05:23 -07:00
register_tour feat(register): ADR-0077 — substantive register knobs + layering boundary (R6) 2026-05-19 23:39:11 -07:00
reports feat(adr-0023): Forward Semantic Control proof evidence — Accepted 2026-05-17 12:55:19 -07:00
results feat: ADR-0086 + ADR-0087 + 100-register catalog — cognition lane closure 2026-05-21 00:08:12 -07:00
reviewer_registry feat(capability): implement ADR-0092 — Reviewer Registry v1 2026-05-21 18:01:24 -07:00
sample_efficiency feat(phase4): sample-efficiency v1 — first quantitative-curve lane 2026-05-16 15:39:28 -07:00
self_consistency_over_time feat(runtime+evals): warm-path pack grounding + three long-span lanes 2026-05-19 08:26:38 -07:00
symbolic_logic feat: ADR-0122 systems_software audit-passed deferred (lane-shape mismatch) 2026-05-22 16:31:59 -07:00
teaching_injection_resistance feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants 2026-05-17 07:27:41 -07:00
walkthrough_chain feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
warmed_session_consistency feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
zero_code_domain_acquisition feat began creation of zero code domain acquisition. did not complete yet. 2026-05-16 06:31:01 -07:00
__init__.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
_parallel.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
baseline_runner.py feat(evals): frontier structural-zero baselines for Phase 2 v1 lanes 2026-05-16 12:45:28 -07:00
CLAIMS.md chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
cognition_cases.jsonl feat: vault recall index, Rust versor parity, cognitive pack expansion 2026-05-15 15:34:39 -07:00
framework.py feat(evals): ADR-0105 — sealed holdout encryption via age (#108) 2026-05-22 10:09:43 -07:00
holdout_runner.py feat(evals): ADR-0105 — sealed holdout encryption via age (#108) 2026-05-22 10:09:43 -07:00
metrics.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
parallel.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
run_cognition_eval.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00