core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/docs/PROGRESS.md
Shay c4f056c44c feat(evals): frontier structural-zero baselines for Phase 2 v1 lanes 
Records the architectural floor for frontier-LLM performance on each
Phase 2 v1 lane.

The baseline is structural: every lane's scoring rubric measures a
property that frontier LLMs do not architecturally emit (Provenance
typed sources, pack_mutation_proposal, vault_hits, REJECTED_IDENTITY
outcome, deterministic trace_hash). The frontier score on each of
those sub-metrics is 0.0 by construction, not by failure — even a
live-API run would still record 0.0 on these typed-signal checks
because the evidence is absent regardless of prose quality.

Artifacts:
  docs/frontier_baselines.md
    Full per-lane analysis: what each sub-metric scores, why the
    frontier value is 0, and where a live-API baseline would or
    would not add information.

  evals/<lane>/baselines/v1_structural_zero.json (× 5)
    Per-lane baseline records in the same shape as lane reports.
    Encodes 0.0 / None on each sub-metric with rationale.

  evals/baseline_runner.py
    Adds StructuralZeroBaseline adapter conforming to the
    BaselineModel protocol — a real, non-stub adapter that returns
    the deterministic floor. Live-API adapters (Anthropic, OpenAI)
    can be wired alongside when API keys are configured; the
    structural floor remains the comparison baseline.

Across 5 lanes / 14 typed-signal sub-metrics:
  CORE v1:            1.0 (each)
  frontier structural: 0.0 (each)

The gap is "CORE measures a property frontier output does not
expose", not "CORE outperforms on a shared benchmark". v2 lanes may
add content-level sub-metrics where direct comparison via live-API
runs becomes meaningful.
2026-05-16 12:45:28 -07:00

9.6 KiB
Raw Blame History

Capability Roadmap — Progress Tracker

Tracks completion of the phased plan defined in docs/capability_roadmap.md (ADR-0016). Updated as work lands.

Phase 0 — Benchmark Methodology Lock-in

Status: Complete Started: 2026-05-15 Completed: 2026-05-16

  • Promote roadmap to ADR-0016
  • Extract docs/eval_methodology.md from roadmap Part I
  • Create progress tracker (docs/PROGRESS.md)
  • Implement evals/<lane>/ directory convention
  • Build generic eval framework (evals/framework.py)
  • Retrofit core eval cognition into new convention
    • Split 45 cases into dev (13) / public v1 (13) / holdout (19)
    • Write evals/cognition/contract.md
    • Migrate runner.py to use framework
    • Record v1 results under new layout
  • Generalize core eval <lane> CLI (dynamic lane discovery)
  • Implement holdout runner scaffold
  • Implement baseline runner scaffold
  • Exit gate: core eval cognition runs under new convention with v1 public + holdout + baseline

Methodology issues discovered (Phase 0 audit)

  1. Pipeline turn_log crash: CognitiveTurnPipeline.run() assumed turn_log was always populated after chat(), but the unknown-domain gate returns a stub without appending. Fixed with fallback to tokenizer output.
  2. Versor drift in multi-turn sessions: test_pipeline_preserves_versor_closure reveals that after 3 turns in the same session, "spirit breath" causes versor_condition = 1.12e-04 (threshold: 1e-6). Pre-existing; resolved by strict runtime closure enforcement (always unitize after sandwich product).
  3. Identity/drive bias shelved: Premature persona motor and drive bias introduced trajectory drift. Removed in favour of persona-neutral generic runtime; identity returns behind explicit IdentityProfile contract.

Phase 1 — Foundational Triple

Status: Complete ✓ Started: 2026-05-16 Completed: 2026-05-16 Depends on: Phase 0 exit

  • grammatical-coverage lane (v1 + v2 complete)
    • Enumerate English v1 constructions (13 constructions: C01-C13)
    • Write contract test pairs (PropositionGraph -> surface family)
    • Implement v1 dev/public (~41/36 items)
    • Implement holdout (52 items) — 100% pass
    • Engineer realizer.py to pass v1 (dev=100%, public=100%, holdout=100%)
    • Hebrew pack (he_core_cognition_v1 with binyanim support)
    • Koine Greek pack (grc_logos_cognition_v1 with Greek morphology)
    • Generate v2 on pass (deeper nesting, longer sentences, rarer vocabulary) — 36 cases (100% pass)
  • zero-code-domain-acquisition lane (v1 complete, zero engineering gaps)
    • Define 3 surprise domains (kinship, calendar, color)
    • Build pack-only authoring kits (vocabulary, relations, axioms, teaching examples, prompts)
    • Test: author brings CORE to >=80% without Python edits (100% achieved)
    • Log engineering gaps (ZERO — pack-only authoring contract is solid)
    • v1 dev (30/30), v1 public (18/18 across all 3 domains), v1 holdout (21/21) — all 100% pass
  • identity-divergence lane (v1 complete)
    • Define two identity axis sets (Axis A: Precision-first, Axis B: Generosity-first)
    • Curate shared curriculum (93 teaching events across color/kinship/reasoning/spatial)
    • Build divergence metric (>0.30 threshold): all pass (1.000)
    • Build coherence metric (>0.85 threshold for A and B): all pass (1.000)
    • Identity-stripped baseline with causal check: all pass (delta=1.000)
    • v1 dev (5/5), v1 public (5/5), v1 holdout (5/5) — all 100% pass
  • Exit gate: All three lanes pass v1 public + holdout ✓

Phase 2 — Structural Wins Made Visible

Status: In Progress Started: 2026-05-16 Depends on: Phase 1 exit

  • provenance lane (v1 complete)
    • Define Provenance dataclass + compute_provenance() (core/cognition/provenance.py)
    • Unit tests for provenance derivation (6/6 pass — tests/test_provenance.py)
    • Build pack-axiom / vault-recall / teaching / mixed case categories
    • v1 dev (10/10), v1 public (20/20), v1 holdouts (15/15) — all 100% pass
    • Sub-metrics: replay_determinism=1.0, source_attribution=1.0, source_validity=1.0, input_sensitivity=1.0
    • Fixed shape regression in generate/stream.py score-weighted recall (np.eye → multivector identity)
    • Replaced linear-blend rotor scaling with manifold-preserving rotor_power (algebra/rotor.py); 41 closure-preservation tests
    • Restored respond()/result.final_state identity contract after anchor pull
  • monotonic-learning lane (v1 complete)
    • Define contract: longitudinal regression check across ≥10 teaching cycles
    • Implement runner: shared session, sorted ops, per-(cycle, domain) accuracy table
    • Generator (scripts/generate_monotonic_cases.py) for cycle/probe corpora
    • v1 dev (10 cycles), v1 public (12 cycles, 3 domains), v1 holdouts (12 cycles, 2 distinct domains)
    • All splits: max_regression=0.00, floor_score=1.00, overall_pass=true
    • Structural win demonstrated: zero regression across 34 total cycles / 7 distinct domains
  • calibration lane (v1 complete)
    • Define contract: typed signals for no_grounding / coherent / correction_proposed
    • Classification from CognitiveTurnResult (vault_hits + pack_mutation_proposal)
    • Runner with per-case fresh pipeline (avoids cross-case field drift)
    • v1 dev (12/12), v1 public (24/24), v1 holdouts (18/18) — all 100% pass
    • Sub-metrics: no_grounding=1.0, coherent=1.0, correction_proposed=1.0
    • Architectural finding documented (evals/calibration/gaps.md): the ingest gate is geometric, not semantic — 6/42 hand-chosen OOD prompts fire the geometric gate. v1 measures recall-presence + correction-firing signals (deterministic), not semantic OOD. Pipeline override of gate's safety surface is a separate gap.
  • symbolic-logic lane (v1 complete)
    • Define contract: structural foundations for proposition-based inference
    • Patterns: modus_ponens_chain, modus_tollens_chain, syllogism, negation, chain_recall
    • Runner: per-case fresh pipeline + double-run replay check
    • Sub-metrics: premise_recall=1.0, replay_determinism=1.0, proposal_storage=1.0
    • v1 dev (8/8), v1 public (18/18), v1 holdouts (12/12) — all 100% pass
    • Architectural finding documented (evals/symbolic_logic/gaps.md): CORE has no first-class inference operator yet. v1 measures the storage, replay, and recall foundations on which a future inference engine would be built. v2 would assert specific inference correctness (transitive recall surface contents).
  • adversarial-identity lane (v1 complete)
    • Define contract: identity-override attacks rejected at review; legitimate corrections still accepted
    • Cover all _IDENTITY_MARKERS families (you are / forget / pretend / override / ignore / your name / act as / from now / character / personality)
    • Per-case fresh pipeline; prior question primes the review surface
    • Sub-metrics: attack_rejection_rate=1.0, legitimate_acceptance_rate=1.0
    • v1 dev (10/10), v1 public (25/25), v1 holdouts (18/18) — all 100% pass
  • All five Phase 2 v1 lanes passing ✓
  • Frontier baselines computed for all lanes (structural-zero floor)
    • docs/frontier_baselines.md — per-lane analysis: frontier LLMs do not emit the typed signals CORE's rubrics score against (provenance sources, pack_mutation_proposal, vault_hits, REJECTED_IDENTITY outcome, deterministic trace_hash)
    • Per-lane structural-zero baseline JSON written under evals/<lane>/baselines/v1_structural_zero.json
    • StructuralZeroBaseline adapter in evals/baseline_runner.py — deterministic floor; live-API adapters can be added when keys are configured
  • v2 lanes (provenance, monotonic-learning, calibration, symbolic-logic, adversarial-identity)
  • Exit gate: All five v1+v2 with baselines; at least two have v3

Phase 3 — Reasoning Depth

Status: Not Started Depends on: Phase 2 exit

  • compositionality lane (construction-family splits, not sampling)
  • inference-closure lane
  • introspection lane
  • multi-step-reasoning lane
  • cross-domain-transfer lane
  • Pin agency scope decision (responsive vs. goal-directed)
  • Pin tool-use scope decision
  • Exit gate: All five v1 scored; at least two passing v1

Phase 4 — Scale and Efficiency

Status: Not Started Depends on: Phase 3 exit

  • sample-efficiency curves (>=10 concepts)
  • long-context-cost curves (10^3 to 10^6 vault entries)
  • multi-agent-composition (>=2 agents, replay preserved)
  • Vault indexing strategy decided
  • Exit gate: All curves published with confidence intervals

Phase 5 — Curriculum Era

Status: Not Started Depends on: Phase 4 exit

  • 5.1 English fluency (grammatical-coverage v5 OOD)
  • 5.2 Hebrew fluency
  • 5.3 Koine Greek fluency
  • 5.4 Elementary mathematics
  • 5.5 Foundational physics
  • 5.6 Foundational biology
  • 5.7 Classical literature
  • Phase 1-4 lanes re-run on every release (no regression)

Open Scope Decisions

Decision Status Deadline
Agency (responsive vs. goal-directed) Open Before Phase 3
Tool use (typed deterministic operators) Open Before Phase 3
Code generation (first-class target) Open Before Phase 5
Embodiment (sensorium gates) Open Phase 5