core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/evals
History
Shay 36aad75202 feat(cli): ADR-0024 chain test-suite aliases + core demo subcommand 
Two layers of CLI surface so reviewers / investors / industry
observers can run the ADR-0024 chain evidence end-to-end without
typing test file paths or hunting for runner scripts.

Layer 1 — test-suite aliases:
  core test --suite refusal     (Phase 2 typed refusals)
  core test --suite margin      (Phase 3 / ADR-0026 ranked-with-margin)
  core test --suite rotor       (Phase 4 / ADR-0025 rotor admissibility)
  core test --suite inner-loop  (ADR-0024 inner-loop, all 4 sub-tests)
  core test --suite phase5      (stratified mechanism-isolation)
  core test --suite phase6      (3-condition comparative demo)
  core test --suite adr-0024    (full chain, 98 tests, ~2 min)

Layer 2 — `core demo` subcommand:
  core demo phase5              stratified pass/refuse table + per-family
                                breakdown (5 families, both modes)
  core demo phase6              three head-to-head verdicts vs baseline
                                (replay determinism / traced rejection /
                                coherent refusal)
  core demo all                 both phases + combined summary
  core demo list-results        index every JSON report in the central
                                results directory with headline metrics

All demo runs:
  - Write fresh JSON to evals/forward_semantic_control/results/
  - Refresh the results/index.json manifest so reviewers see every
    available report in one place
  - Accept --json for machine-readable output

Central results directory: evals/forward_semantic_control/results/
  phase2_inner_loop_report.json
  phase3_v2_report.json
  phase4_characterization_*.json
  phase5_report.json
  phase5_benign_inner_loop_report.json
  phase6_demo_report.json
  index.json (auto-generated manifest)

Files:
  core/cli.py                   — +9 suite aliases, +cmd_demo (3 targets +
                                  list-results), +index manifest writer
  tests/test_cli_demo.py        — 14 contract tests pinning Layer 1 + 2
  evals/forward_semantic_control/results/index.json — auto-generated

Tests: 1099 passed, 2 skipped (+14 from Phase 6 baseline).
2026-05-17 16:21:37 -07:00
..
adversarial_identity docs(identity): empirical finding — fix #3 needs upstream ingest-gate work 2026-05-16 14:23:20 -07:00
articulation_of_status feat(epistemic): realizer-side closure — refusal_calibration + articulation_of_status graduate 2026-05-17 10:12:59 -07:00
calibration fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
classical_literature_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
cognition feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
compositionality feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
contradiction_detection feat(epistemic): contradiction coherence checker — CONTESTED transitions wired, last Tier 4.5 row closes 2026-05-17 10:36:48 -07:00
cross_domain_transfer feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
discourse_paragraph feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
elementary_mathematics_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
english_fluency_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
forward_semantic_control feat(cli): ADR-0024 chain test-suite aliases + core demo subcommand 2026-05-17 16:21:37 -07:00
foundational_biology_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
foundational_physics_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
grammatical_coverage fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
hebrew_fluency feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep 2026-05-16 20:59:31 -07:00
identity_divergence feat(evals): identity-divergence lane v1 - 93 curriculum events, two axis profiles (Precision/Generosity), divergence/coherence/causal metrics (all pass) 2026-05-16 06:48:13 -07:00
inference_closure feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
introspection feat(phase3): core/cognition/explain.py — close Gap 3 introspection 2026-05-16 15:09:48 -07:00
koine_greek_fluency feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep 2026-05-16 20:59:31 -07:00
long_context_cost feat(phase4): long-context-cost lane + ADR-0019 Stage 1 vault recall vectorisation 2026-05-16 16:39:30 -07:00
monotonic_learning feat(evals): v3 lanes — monotonic-learning passes, adversarial-identity reveals gap 2026-05-16 13:42:47 -07:00
multi_agent_composition fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
multi_step_reasoning feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
provenance feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
refusal_calibration feat(epistemic): realizer-side closure — refusal_calibration + articulation_of_status graduate 2026-05-17 10:12:59 -07:00
reports feat(adr-0023): Forward Semantic Control proof evidence — Accepted 2026-05-17 12:55:19 -07:00
sample_efficiency feat(phase4): sample-efficiency v1 — first quantitative-curve lane 2026-05-16 15:39:28 -07:00
symbolic_logic feat(evals): v2 lanes for calibration and symbolic-logic 2026-05-16 13:17:41 -07:00
teaching_injection_resistance feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants 2026-05-17 07:27:41 -07:00
zero_code_domain_acquisition feat began creation of zero code domain acquisition. did not complete yet. 2026-05-16 06:31:01 -07:00
__init__.py
baseline_runner.py feat(evals): frontier structural-zero baselines for Phase 2 v1 lanes 2026-05-16 12:45:28 -07:00
CLAIMS.md feat(bench): bench cost — $/1000 turns + latency, with disclosed assumptions 2026-05-17 10:53:08 -07:00
cognition_cases.jsonl
framework.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
holdout_runner.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
metrics.py
parallel.py feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
run_cognition_eval.py