core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/docs/adr/ADR-0016-capability-roadmap.md

2.7 KiB
Raw Blame History

ADR-0016 — Capability Roadmap and Eval Methodology

Status: Accepted Date: 2026-05-15 Authors: Joshua Shay Derived from: docs/sessions/SESSION-2026-05-15-capability-gates.md

Context

CORE needs a falsifiable framework for measuring progress toward its design goals. Without one, "are we there yet" remains a subjective question and progress drifts toward vibes-based evaluation.

The 2026-05-15 session deliberated on:

  • What fluency means for the three foundational languages (English, Hebrew, Koine Greek) and when engineering ends and curriculum begins.
  • What additional capability dimensions matter for an AGI-aspiring architecture.
  • How CORE's structural properties compare to modern transformer architectures.
  • How to build honest benchmarks that resist overfitting.

Decision

Adopt the Verifiable Competence Benchmark framework defined in docs/capability_roadmap.md as the governing plan for CORE's capability development. The framework consists of:

  1. Benchmark Discipline (Part I) — five rules that govern every eval lane: three-set splits, versioned difficulty escalation, adversarial regeneration on pass, frontier baseline tracking, and honest reporting.

  2. Six Phases (Part II):

    • Phase 0: Methodology lock-in (eval infrastructure)
    • Phase 1: Foundational Triple (fluency, domain acquisition, identity)
    • Phase 2: Structural Wins (provenance, monotonic learning, calibration, symbolic logic, adversarial identity)
    • Phase 3: Reasoning Depth (compositionality, inference closure, introspection, multi-step reasoning, cross-domain transfer)
    • Phase 4: Scale and Efficiency (sample efficiency, vault cost curves, multi-agent composition)
    • Phase 5: Curriculum Era (open-ended domain acquisition)

  3. Eval contract template — every lane lives in evals/<lane>/ with: contract.md, dev/, public/v1/, holdouts/, runner.py, baselines/, results/.

  4. Open scope decisions to be pinned before Phase 3:

    • Agency (responsive vs. goal-directed)
    • Tool use (typed deterministic operators)
    • Code generation (first-class articulation target)

Consequences

  • Every new eval lane must follow the convention or it does not merge.
  • The existing core eval cognition is retrofitted as the first lane under the new convention (Phase 0 forcing function).
  • Progress is tracked in docs/PROGRESS.md with evidence links.
  • The roadmap itself is versioned; amendments are dated, never silently rewritten.

References

  • docs/capability_roadmap.md — full roadmap
  • docs/sessions/SESSION-2026-05-15-capability-gates.md — deliberation log
  • docs/eval_methodology.md — extracted Part I (benchmark discipline contract)