64 lines
2.7 KiB
Markdown
64 lines
2.7 KiB
Markdown
# ADR-0016 — Capability Roadmap and Eval Methodology
|
|
|
|
**Status:** Accepted
|
|
**Date:** 2026-05-15
|
|
**Authors:** Joshua Shay
|
|
**Derived from:** `docs/sessions/SESSION-2026-05-15-capability-gates.md`
|
|
|
|
## Context
|
|
|
|
CORE needs a falsifiable framework for measuring progress toward its design
|
|
goals. Without one, "are we there yet" remains a subjective question and
|
|
progress drifts toward vibes-based evaluation.
|
|
|
|
The 2026-05-15 session deliberated on:
|
|
|
|
- What fluency means for the three foundational languages (English, Hebrew,
|
|
Koine Greek) and when engineering ends and curriculum begins.
|
|
- What additional capability dimensions matter for an AGI-aspiring architecture.
|
|
- How CORE's structural properties compare to modern transformer architectures.
|
|
- How to build honest benchmarks that resist overfitting.
|
|
|
|
## Decision
|
|
|
|
Adopt the Verifiable Competence Benchmark framework defined in
|
|
`docs/capability_roadmap.md` as the governing plan for CORE's capability
|
|
development. The framework consists of:
|
|
|
|
1. **Benchmark Discipline (Part I)** — five rules that govern every eval lane:
|
|
three-set splits, versioned difficulty escalation, adversarial regeneration
|
|
on pass, frontier baseline tracking, and honest reporting.
|
|
|
|
2. **Six Phases (Part II):**
|
|
- Phase 0: Methodology lock-in (eval infrastructure)
|
|
- Phase 1: Foundational Triple (fluency, domain acquisition, identity)
|
|
- Phase 2: Structural Wins (provenance, monotonic learning, calibration,
|
|
symbolic logic, adversarial identity)
|
|
- Phase 3: Reasoning Depth (compositionality, inference closure,
|
|
introspection, multi-step reasoning, cross-domain transfer)
|
|
- Phase 4: Scale and Efficiency (sample efficiency, vault cost curves,
|
|
multi-agent composition)
|
|
- Phase 5: Curriculum Era (open-ended domain acquisition)
|
|
|
|
3. **Eval contract template** — every lane lives in `evals/<lane>/` with:
|
|
`contract.md`, `dev/`, `public/v1/`, `holdouts/`, `runner.py`, `baselines/`,
|
|
`results/`.
|
|
|
|
4. **Open scope decisions** to be pinned before Phase 3:
|
|
- Agency (responsive vs. goal-directed)
|
|
- Tool use (typed deterministic operators)
|
|
- Code generation (first-class articulation target)
|
|
|
|
## Consequences
|
|
|
|
- Every new eval lane must follow the convention or it does not merge.
|
|
- The existing `core eval cognition` is retrofitted as the first lane under the
|
|
new convention (Phase 0 forcing function).
|
|
- Progress is tracked in `docs/PROGRESS.md` with evidence links.
|
|
- The roadmap itself is versioned; amendments are dated, never silently rewritten.
|
|
|
|
## References
|
|
|
|
- `docs/capability_roadmap.md` — full roadmap
|
|
- `docs/sessions/SESSION-2026-05-15-capability-gates.md` — deliberation log
|
|
- `docs/eval_methodology.md` — extracted Part I (benchmark discipline contract)
|