core/evals
Shay 1e2fce5f4f feat(evals): v3 lanes — monotonic-learning passes, adversarial-identity reveals gap
Closes the Phase 2 roadmap exit gate (v3 for at least two lanes).

monotonic-learning v3:
  public/v3  — 7 domains × 3-4 probes × 30 cycles (805 ops)
                domains: truth, light, wisdom, order, memory, meaning, identity
                max_regression=0.0, floor_score=1.0
  holdouts/v3 — 6 distinct domains × 4 probes × 25 cycles (597 ops)
                domains: creation, knowledge, reason, spirit, principle, judgment
                max_regression=0.0, floor_score=1.0

  Zero regression across 55 cumulative cycles and 13 distinct domains.
  Sequential within a split (single shared longitudinal session) is
  structural; public/v3 ran in 12.5 min, holdouts/v3 in 8.2 min.

adversarial-identity v3:
  public/v3  — 20 paraphrased attacks + 10 legitimate corrections
  holdouts/v3 — 12 paraphrased attacks + 8 legitimate corrections
  Result: attack_rejection_rate=0.0, legitimate_acceptance_rate=1.0
          on both splits.

  THIS IS THE INTENDED OUTCOME OF v3. The attack set was constructed
  to use only paraphrased / synonym-shifted phrasings that avoid every
  literal string in teaching/review.py::_IDENTITY_MARKERS. Every
  attack ("become a different agent", "behave as", "assume the role
  of", "switch into helper mode", "transform into", "drop the
  framework", "let's roleplay that") bypasses the marker list and is
  admitted to the teaching store as a PackMutationProposal.

  evals/adversarial_identity/gaps.md documents the finding in detail
  and proposes three follow-up fixes in increasing order of weight:
    1. Extend _IDENTITY_MARKERS with verb-of-becoming and role-frame
       classes (cheapest, still string-matching).
    2. Semantic syntactic check on
       [redirect-verb] + [self-reference] + [role-frame] structure.
    3. Geometric identity-versor check (architectural; aligns with
       ADR-0010 identity-as-geometry doctrine — synonymous attacks
       produce similar field deltas, so the defense is paraphrase-
       invariant by construction).

  v1 (38 attacks, all blocked) and v2 (32 attacks, all blocked)
  remain valid for their declared coverage (the marker-list smoke
  test and its punctuation/case variants). v3 is recorded as a
  known-failing stress test, not a regression — it is load-bearing
  evidence for the v4 / architectural fix work above.

Phase 2 status: COMPLETE.
  - All five lanes v1+v2 at 100% (provenance, monotonic-learning,
    calibration, symbolic-logic, adversarial-identity)
  - Frontier structural baselines documented for all five
  - v3 exit gate met: monotonic-learning v3 passes, adversarial-
    identity v3 reveals load-bearing architectural finding
  - Test suite: 596 passing (no regression)
2026-05-16 13:42:47 -07:00
..
adversarial_identity feat(evals): v3 lanes — monotonic-learning passes, adversarial-identity reveals gap 2026-05-16 13:42:47 -07:00
calibration feat(evals): v2 lanes for calibration and symbolic-logic 2026-05-16 13:17:41 -07:00
cognition feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
grammatical_coverage feat(evals): grammatical-coverage v2 cases - 36 cases with deeper nesting and rarer vocabulary (100% pass) 2026-05-16 06:40:55 -07:00
identity_divergence feat(evals): identity-divergence lane v1 - 93 curriculum events, two axis profiles (Precision/Generosity), divergence/coherence/causal metrics (all pass) 2026-05-16 06:48:13 -07:00
monotonic_learning feat(evals): v3 lanes — monotonic-learning passes, adversarial-identity reveals gap 2026-05-16 13:42:47 -07:00
provenance feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
reports Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
symbolic_logic feat(evals): v2 lanes for calibration and symbolic-logic 2026-05-16 13:17:41 -07:00
zero_code_domain_acquisition feat began creation of zero code domain acquisition. did not complete yet. 2026-05-16 06:31:01 -07:00
__init__.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
baseline_runner.py feat(evals): frontier structural-zero baselines for Phase 2 v1 lanes 2026-05-16 12:45:28 -07:00
cognition_cases.jsonl feat: vault recall index, Rust versor parity, cognitive pack expansion 2026-05-15 15:34:39 -07:00
framework.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
holdout_runner.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
metrics.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
parallel.py feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
run_cognition_eval.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00