core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/docs/decisions/ADR-0016-capability-roadmap.md
Shay 1e01f7794e feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 
Implement the eval infrastructure defined in ADR-0016 before building new
eval lanes. This establishes the discipline that governs the entire
capability roadmap.

- Generic eval framework (evals/framework.py): lane discovery, versioned
  scoring, result persistence
- Cognition lane retrofitted into new convention: 45 cases split into
  stratified dev (13) / public v1 (13) / holdout (19) sets with contract,
  runner, and recorded results
- Generalized `core eval <lane>` CLI: dynamic lane discovery, --list,
  --version, --split, --save, --json flags
- Holdout runner scaffold: plaintext fallback, encryption interface ready
- Baseline runner scaffold: pluggable frontier model interface
- Fix: CognitiveTurnPipeline.run() crashed on turn_log[-1] when the
  unknown-domain gate returned a stub without appending to turn_log
- ADR-0016, eval_methodology.md, PROGRESS.md, capability gates session log

Phase 0 exit audit found two methodology issues:
1. Pipeline turn_log crash (fixed here)
2. Versor drift in multi-turn sessions (pre-existing, under investigation)
2026-05-15 22:36:53 -07:00

2.7 KiB
Raw Blame History

ADR-0016 — Capability Roadmap and Eval Methodology

Status: Accepted Date: 2026-05-15 Authors: Joshua Shay Derived from: docs/sessions/SESSION-2026-05-15-capability-gates.md

Context

CORE needs a falsifiable framework for measuring progress toward its design goals. Without one, "are we there yet" remains a subjective question and progress drifts toward vibes-based evaluation.

The 2026-05-15 session deliberated on:

  • What fluency means for the three foundational languages (English, Hebrew, Koine Greek) and when engineering ends and curriculum begins.
  • What additional capability dimensions matter for an AGI-aspiring architecture.
  • How CORE's structural properties compare to modern transformer architectures.
  • How to build honest benchmarks that resist overfitting.

Decision

Adopt the Verifiable Competence Benchmark framework defined in docs/capability_roadmap.md as the governing plan for CORE's capability development. The framework consists of:

  1. Benchmark Discipline (Part I) — five rules that govern every eval lane: three-set splits, versioned difficulty escalation, adversarial regeneration on pass, frontier baseline tracking, and honest reporting.

  2. Six Phases (Part II):

    • Phase 0: Methodology lock-in (eval infrastructure)
    • Phase 1: Foundational Triple (fluency, domain acquisition, identity)
    • Phase 2: Structural Wins (provenance, monotonic learning, calibration, symbolic logic, adversarial identity)
    • Phase 3: Reasoning Depth (compositionality, inference closure, introspection, multi-step reasoning, cross-domain transfer)
    • Phase 4: Scale and Efficiency (sample efficiency, vault cost curves, multi-agent composition)
    • Phase 5: Curriculum Era (open-ended domain acquisition)

  3. Eval contract template — every lane lives in evals/<lane>/ with: contract.md, dev/, public/v1/, holdouts/, runner.py, baselines/, results/.

  4. Open scope decisions to be pinned before Phase 3:

    • Agency (responsive vs. goal-directed)
    • Tool use (typed deterministic operators)
    • Code generation (first-class articulation target)

Consequences

  • Every new eval lane must follow the convention or it does not merge.
  • The existing core eval cognition is retrofitted as the first lane under the new convention (Phase 0 forcing function).
  • Progress is tracked in docs/PROGRESS.md with evidence links.
  • The roadmap itself is versioned; amendments are dated, never silently rewritten.

References

  • docs/capability_roadmap.md — full roadmap
  • docs/sessions/SESSION-2026-05-15-capability-gates.md — deliberation log
  • docs/eval_methodology.md — extracted Part I (benchmark discipline contract)