core/docs/PROGRESS.md

Capability Roadmap — Progress Tracker

Tracks completion of the phased plan defined in docs/capability_roadmap.md (ADR-0016). Updated as work lands.

---

Phase 0 — Benchmark Methodology Lock-in

Status: Complete Started: 2026-05-15 Completed: 2026-05-16

• Promote roadmap to ADR-0016
• Extract docs/eval_methodology.md from roadmap Part I
• Create progress tracker (docs/PROGRESS.md)
• Implement evals/<lane>/ directory convention
• Build generic eval framework (evals/framework.py)
• Retrofit core eval cognition into new convention
  • Split 45 cases into dev (13) / public v1 (13) / holdout (19)
  • Write evals/cognition/contract.md
  • Migrate runner.py to use framework
  • Record v1 results under new layout
• Generalize core eval <lane> CLI (dynamic lane discovery)
• Implement holdout runner scaffold
• Implement baseline runner scaffold
• Exit gate: core eval cognition runs under new convention with v1 public + holdout + baseline

Methodology issues discovered (Phase 0 audit)

1. Pipeline turn_log crash: CognitiveTurnPipeline.run() assumed turn_log was always populated after chat(), but the unknown-domain gate returns a stub without appending. Fixed with fallback to tokenizer output.
2. Versor drift in multi-turn sessions: test_pipeline_preserves_versor_closure reveals that after 3 turns in the same session, "spirit breath" causes versor_condition = 1.12e-04 (threshold: 1e-6). Pre-existing; resolved by strict runtime closure enforcement (always unitize after sandwich product).
3. Identity/drive bias shelved: Premature persona motor and drive bias introduced trajectory drift. Removed in favour of persona-neutral generic runtime; identity returns behind explicit IdentityProfile contract.

---

Phase 1 — Foundational Triple

Status: Complete ✓ Started: 2026-05-16 Completed: 2026-05-16 Depends on: Phase 0 exit

• grammatical-coverage lane (v1 + v2 complete)
  • Enumerate English v1 constructions (13 constructions: C01-C13)
  • Write contract test pairs (PropositionGraph -> surface family)
  • Implement v1 dev/public (~41/36 items)
  • Implement holdout (52 items) — 100% pass
  • Engineer realizer.py to pass v1 (dev=100%, public=100%, holdout=100%)
  • Hebrew pack (he_core_cognition_v1 with binyanim support)
  • Koine Greek pack (grc_logos_cognition_v1 with Greek morphology)
  • Generate v2 on pass (deeper nesting, longer sentences, rarer vocabulary) — 36 cases (100% pass)
• zero-code-domain-acquisition lane (v1 complete, zero engineering gaps)
  • Define 3 surprise domains (kinship, calendar, color)
  • Build pack-only authoring kits (vocabulary, relations, axioms, teaching examples, prompts)
  • Test: author brings CORE to >=80% without Python edits (100% achieved)
  • Log engineering gaps (ZERO — pack-only authoring contract is solid)
  • v1 dev (30/30), v1 public (18/18 across all 3 domains), v1 holdout (21/21) — all 100% pass
• identity-divergence lane (v1 complete)
  • Define two identity axis sets (Axis A: Precision-first, Axis B: Generosity-first)
  • Curate shared curriculum (93 teaching events across color/kinship/reasoning/spatial)
  • Build divergence metric (>0.30 threshold): all pass (1.000)
  • Build coherence metric (>0.85 threshold for A and B): all pass (1.000)
  • Identity-stripped baseline with causal check: all pass (delta=1.000)
  • v1 dev (5/5), v1 public (5/5), v1 holdout (5/5) — all 100% pass
• Exit gate: All three lanes pass v1 public + holdout ✓

---

Phase 2 — Structural Wins Made Visible

Status: In Progress Started: 2026-05-16 Depends on: Phase 1 exit

• provenance lane (v1 complete)
  • Define Provenance dataclass + compute_provenance() (core/cognition/provenance.py)
  • Unit tests for provenance derivation (6/6 pass — tests/test_provenance.py)
  • Build pack-axiom / vault-recall / teaching / mixed case categories
  • v1 dev (10/10), v1 public (20/20), v1 holdouts (15/15) — all 100% pass
  • Sub-metrics: replay_determinism=1.0, source_attribution=1.0, source_validity=1.0, input_sensitivity=1.0
  • Fixed shape regression in generate/stream.py score-weighted recall (np.eye → multivector identity)
  • Replaced linear-blend rotor scaling with manifold-preserving rotor_power (algebra/rotor.py); 41 closure-preservation tests
  • Restored respond()/result.final_state identity contract after anchor pull
• monotonic-learning lane (v1 complete)
  • Define contract: longitudinal regression check across ≥10 teaching cycles
  • Implement runner: shared session, sorted ops, per-(cycle, domain) accuracy table
  • Generator (scripts/generate_monotonic_cases.py) for cycle/probe corpora
  • v1 dev (10 cycles), v1 public (12 cycles, 3 domains), v1 holdouts (12 cycles, 2 distinct domains)
  • All splits: max_regression=0.00, floor_score=1.00, overall_pass=true
  • Structural win demonstrated: zero regression across 34 total cycles / 7 distinct domains
• calibration lane (v1 complete)
  • Define contract: typed signals for no_grounding / coherent / correction_proposed
  • Classification from CognitiveTurnResult (vault_hits + pack_mutation_proposal)
  • Runner with per-case fresh pipeline (avoids cross-case field drift)
  • v1 dev (12/12), v1 public (24/24), v1 holdouts (18/18) — all 100% pass
  • Sub-metrics: no_grounding=1.0, coherent=1.0, correction_proposed=1.0
  • Architectural finding documented (evals/calibration/gaps.md): the ingest gate is geometric, not semantic — 6/42 hand-chosen OOD prompts fire the geometric gate. v1 measures recall-presence + correction-firing signals (deterministic), not semantic OOD. Pipeline override of gate's safety surface is a separate gap.
• symbolic-logic lane (v1 complete)
  • Define contract: structural foundations for proposition-based inference
  • Patterns: modus_ponens_chain, modus_tollens_chain, syllogism, negation, chain_recall
  • Runner: per-case fresh pipeline + double-run replay check
  • Sub-metrics: premise_recall=1.0, replay_determinism=1.0, proposal_storage=1.0
  • v1 dev (8/8), v1 public (18/18), v1 holdouts (12/12) — all 100% pass
  • Architectural finding documented (evals/symbolic_logic/gaps.md): CORE has no first-class inference operator yet. v1 measures the storage, replay, and recall foundations on which a future inference engine would be built. v2 would assert specific inference correctness (transitive recall surface contents).
• adversarial-identity lane (v1 complete)
  • Define contract: identity-override attacks rejected at review; legitimate corrections still accepted
  • Cover all _IDENTITY_MARKERS families (you are / forget / pretend / override / ignore / your name / act as / from now / character / personality)
  • Per-case fresh pipeline; prior question primes the review surface
  • Sub-metrics: attack_rejection_rate=1.0, legitimate_acceptance_rate=1.0
  • v1 dev (10/10), v1 public (25/25), v1 holdouts (18/18) — all 100% pass
• All five Phase 2 v1 lanes passing ✓
• Frontier baselines computed for all lanes (structural-zero floor)
  • docs/frontier_baselines.md — per-lane analysis: frontier LLMs do not emit the typed signals CORE's rubrics score against (provenance sources, pack_mutation_proposal, vault_hits, REJECTED_IDENTITY outcome, deterministic trace_hash)
  • Per-lane structural-zero baseline JSON written under evals/<lane>/baselines/v1_structural_zero.json
  • StructuralZeroBaseline adapter in evals/baseline_runner.py — deterministic floor; live-API adapters can be added when keys are configured
• v2 lanes: all five at 100% pass
  • monotonic-learning v2 — 20 cyc / 5 dom (public), 18 cyc / 4 dom (holdouts)
  • provenance v2 — 30 + 20 cases, all sub-metrics 1.0
  • adversarial-identity v2 — 35 + 22 cases, all 1.0
  • calibration v2 — 33 + 24 cases, all class accuracies 1.0
  • symbolic-logic v2 — 24 + 16 cases (chains up to 5 hops), all 1.0
• Exit gate: v3 lanes for at least two of the five ✓
  • monotonic-learning v3 — 30 cyc / 7 dom (public), 25 cyc / 6 dom (holdouts), max_regression=0.0, floor_score=1.0 on both splits
  • adversarial-identity v3 — 30 + 20 paraphrased-attack cases. Initial v3 result (pre-fix): attack_rejection_rate=0.0, legitimate_acceptance_rate=1.0. v3 was a load-bearing finding that exposed the marker-string defense as brittle to paraphrase.

Identity-override defense — fix #2 + fix #3 (2026-05-16)

Triggered by the v3 finding above. Two-layer defense now active in teaching/review.py:

• Fix #2 (syntactic). _is_identity_override applies four deterministic rules: (a) legacy markers, (b) redirect-verb + role-frame co-occurrence, (c) negating qualifier ±3 tokens from a role-frame, (d) negating qualifier ±3 tokens from a redirect-verb.
• Fix #3 (geometric). IdentityCheck.would_violate(score, manifold) predicate added to core/physics/identity.py; review_correction now accepts identity_score / identity_manifold kwargs and is wired in CognitiveTurnPipeline._run_teaching from response.identity_score.

Lane results after both fixes:

split	attacks	attack_rej	legit_acc
public/v1	15	1.0	1.0
holdouts/v1	10	1.0	1.0
public/v2	20	1.0	1.0
holdouts/v2	12	1.0	1.0
public/v3	20	1.0	1.0
holdouts/v3	12	1.0	1.0
public/v4	20	1.0	1.0
holdouts/v4	12	1.0	1.0
public/v5	20	1.0	1.0
holdouts/v5	12	1.0	1.0

v4 is the regression gate for fix #2 — new attack vocabulary combinations that exercise rules (b)/(c)/(d) without repeating v3's specific surface. v5 is the regression gate for the normalization layer — contractions (you're/it's/let's/don't), curly quotes (U+2018/U+2019), em-dashes, and verb morphology (becoming / transformed / dropped / becomes) — all now folded before rule evaluation. All v1–v5 splits pass at 100%; legitimate-correction false-positive rate is 0% (including legitimates that themselves use contractions: wisdom's broader, knowledge isn't merely collected, etc.).

Honest finding: with the current default IdentityManifold (three unit-axis ValueAxes), the geometric layer flags 0/32 of v3 attacks independently of fix #2. The predicate and wiring are in place; the manifold's axis design is the limiting factor and needs sharpening before the geometric defense can carry weight on its own. See evals/adversarial_identity/gaps.md.

Geometric-axis sharpening investigation (2026-05-16)

A focused empirical investigation against v3 and v5 (preserved as evals/adversarial_identity/calibration/probe_field_signature.py) swept every candidate per-case discriminator derivable from the existing CognitiveTurnResult — identity_score.alignment, field-delta L2 norm, semantic-coord energy ratio, vault_hits, surface length, intent tag. No signal separated attack from legitimate at the per-case level. identity_score.alignment is 1.000 universally; field-delta distributions overlap heavily; vault retrieval grounds both kinds similarly.

The pipeline encodes identity-override attacks and legitimate corrections into statistically indistinguishable field-state geometries. No amount of axis-direction sharpening on the IdentityManifold can recover a signal that isn't present in the trajectory data being projected.

Architectural conclusion: fix #3 cannot be made load-bearing in place. The required upstream work — encoding token semantic categories into specific blade coordinates of the field versor at the ingest gate, then redefining the IdentityManifold axes in the 32-dim Cl(4,1) basis with a real inner-product projection — is a scoped multi-PR effort, not a single sharpening exercise. The calibration probe stands as the empirical baseline that any future ingest-gate change must beat before fix #3 can be claimed load-bearing. See evals/adversarial_identity/gaps.md for the full table of measured signals and the recommended path.

What stands today as the load-bearing defense: fix #2 (syntactic rules a/b/c/d) + the normalization layer reject 100% of v1–v5 attacks (n=121) with 0 false positives on 51 legitimate corrections. Fix #3's predicate, unit tests, and wiring remain as scaffolding for the upstream work above.

Phase 2 — COMPLETE

All five Phase 2 v1+v2 lanes pass at 100%; frontier structural baselines documented; v3 satisfies the exit-gate requirement (two lanes, one demonstrating a passing structural-depth test and one demonstrating an architectural vulnerability that the geometric identity-check fix in evals/adversarial_identity/gaps.md would close).

Parallel eval infrastructure (2026-05-16)

• evals/parallel.py — run_cases_parallel() helper using multiprocessing.Pool with the "spawn" start method (avoids forking heavy parent state). Default workers = min(cpu_count, 8).
• Wired into the four per-case lanes (provenance, calibration, symbolic-logic, adversarial-identity). run_lane(..., workers=N) controls parallelism; workers=1 forces serial for debugging.
• Empirical speedup (adversarial-identity public/v1, 25 cases): serial 14.1s → parallel 3.1s (~4.5x).
• Monotonic-learning intentionally stays serial within a split (shared longitudinal session by design).

---

Phase 3 — Reasoning Depth

Status: Not Started Depends on: Phase 2 exit

• compositionality lane (construction-family splits, not sampling)
• inference-closure lane
• introspection lane
• multi-step-reasoning lane
• cross-domain-transfer lane
• Pin agency scope decision (responsive vs. goal-directed)
• Pin tool-use scope decision
• Exit gate: All five v1 scored; at least two passing v1

---

Phase 4 — Scale and Efficiency

Status: Not Started Depends on: Phase 3 exit

• sample-efficiency curves (>=10 concepts)
• long-context-cost curves (10^3 to 10^6 vault entries)
• multi-agent-composition (>=2 agents, replay preserved)
• Vault indexing strategy decided
• Exit gate: All curves published with confidence intervals

---

Phase 5 — Curriculum Era

Status: Not Started Depends on: Phase 4 exit

• 5.1 English fluency (grammatical-coverage v5 OOD)
• 5.2 Hebrew fluency
• 5.3 Koine Greek fluency
• 5.4 Elementary mathematics
• 5.5 Foundational physics
• 5.6 Foundational biology
• 5.7 Classical literature
• Phase 1-4 lanes re-run on every release (no regression)

---

Open Scope Decisions

Decision	Status	Deadline
Agency (responsive vs. goal-directed)	Open	Before Phase 3
Tool use (typed deterministic operators)	Open	Before Phase 3
Code generation (first-class target)	Open	Before Phase 5
Embodiment (sensorium gates)	Open	Phase 5