core/evals
Shay 819c8b81ac feat(phase3): compositionality, multi-step-reasoning, introspection, cross-domain-transfer v1
Spreads the four remaining Phase 3 lanes to map the full reasoning-
depth surface alongside inference-closure (already landed at e509e0d).
Each lane is a v1 honest probe per the roadmap; engineering work
follows once the full surface is visible.

Results across all five Phase 3 lanes:

  lane                      split        primary signal  foundation
  inference-closure         public/v1    0.0             1.0 / 1.0
  inference-closure         holdouts/v1  0.0             1.0 / 1.0
  compositionality          public/v1   0.0625 (1/16)   1.0 / 1.0
  compositionality          holdouts/v1  0.0             1.0 / 1.0
  multi-step-reasoning      public/v1    0.0             1.0 / 1.0
  multi-step-reasoning      holdouts/v1  0.0             1.0 / 1.0
  introspection             public/v1    0.0 (no api)    n/a
  introspection             holdouts/v1  0.0             n/a
  cross-domain-transfer     public/v1    0.0             1.0 / 1.0
  cross-domain-transfer     holdouts/v1  0.0             1.0 / 1.0

Foundation guarantees (storage + replay) intact across every lane
that has them. The reasoning-depth signal is uniformly zero. The
five lanes triangulate four architectural gaps:

  Gap 1. generate/graph_planner.py has no transitive composition.
  Gap 2. field/propagate.py has no derivable-but-not-asserted recall.
  Gap 3. core/cognition/explain.py module does not exist.
  Gap 4. no structural-pattern recogniser (cross-subdomain transfer).

Gaps 1, 2, 4 cluster on the same code surface and may close together
as a single bounded PR. Gap 3 is independent module-creation work.

Lane scaffolding mirrors inference-closure (contract.md, runner.py,
dev + public/v1 + holdouts/v1 cases.jsonl, baselines/v1_structural_zero.json,
gaps.md). All runners are parallel-safe and use the standard
run_lane(cases, *, config, workers) interface.

Per-lane gaps.md records the engineering shape for v2 plus future
directions worth not forgetting:
  - compositionality/gaps.md: metaphor is compositionality with
    selective property transfer; building it is correctly downstream
    of closing this lane.
  - cross-domain-transfer/gaps.md: metaphor + narrative as
    cross-domain operators; narrative requires the Agency open-scope
    decision to pin first.
  - introspection/gaps.md: explain API is also the substrate for
    first-person narrative self-account.

Recommended v2 sequence in docs/PROGRESS.md:
  1. Pin Agency + Tool-use open-scope decisions (deadline: before
     Phase 3 engineering).
  2. Engineer Gaps 1 + 2 as one bounded PR.
  3. Engineer Gap 3 independently.
  4. Re-author cross-domain-transfer v2 with matched-control
     contract refinement.

Phase 3 v1 exit: 0/5 lanes passing, which is the expected v1 floor.
CLI suites smoke / cognition / teaching pass; no regression on
Phase 2.
2026-05-16 14:48:36 -07:00
..
adversarial_identity docs(identity): empirical finding — fix #3 needs upstream ingest-gate work 2026-05-16 14:23:20 -07:00
calibration feat(evals): v2 lanes for calibration and symbolic-logic 2026-05-16 13:17:41 -07:00
cognition feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
compositionality feat(phase3): compositionality, multi-step-reasoning, introspection, cross-domain-transfer v1 2026-05-16 14:48:36 -07:00
cross_domain_transfer feat(phase3): compositionality, multi-step-reasoning, introspection, cross-domain-transfer v1 2026-05-16 14:48:36 -07:00
grammatical_coverage feat(evals): grammatical-coverage v2 cases - 36 cases with deeper nesting and rarer vocabulary (100% pass) 2026-05-16 06:40:55 -07:00
identity_divergence feat(evals): identity-divergence lane v1 - 93 curriculum events, two axis profiles (Precision/Generosity), divergence/coherence/causal metrics (all pass) 2026-05-16 06:48:13 -07:00
inference_closure feat(phase3): inference-closure lane v1 — foundation OK, no operator 2026-05-16 14:33:08 -07:00
introspection feat(phase3): compositionality, multi-step-reasoning, introspection, cross-domain-transfer v1 2026-05-16 14:48:36 -07:00
monotonic_learning feat(evals): v3 lanes — monotonic-learning passes, adversarial-identity reveals gap 2026-05-16 13:42:47 -07:00
multi_step_reasoning feat(phase3): compositionality, multi-step-reasoning, introspection, cross-domain-transfer v1 2026-05-16 14:48:36 -07:00
provenance feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
reports Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
symbolic_logic feat(evals): v2 lanes for calibration and symbolic-logic 2026-05-16 13:17:41 -07:00
zero_code_domain_acquisition feat began creation of zero code domain acquisition. did not complete yet. 2026-05-16 06:31:01 -07:00
__init__.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
baseline_runner.py feat(evals): frontier structural-zero baselines for Phase 2 v1 lanes 2026-05-16 12:45:28 -07:00
cognition_cases.jsonl feat: vault recall index, Rust versor parity, cognitive pack expansion 2026-05-15 15:34:39 -07:00
framework.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
holdout_runner.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
metrics.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
parallel.py feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
run_cognition_eval.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00