1e01f7794e
Implement the eval infrastructure defined in ADR-0016 before building new eval lanes. This establishes the discipline that governs the entire capability roadmap. - Generic eval framework (evals/framework.py): lane discovery, versioned scoring, result persistence - Cognition lane retrofitted into new convention: 45 cases split into stratified dev (13) / public v1 (13) / holdout (19) sets with contract, runner, and recorded results - Generalized `core eval <lane>` CLI: dynamic lane discovery, --list, --version, --split, --save, --json flags - Holdout runner scaffold: plaintext fallback, encryption interface ready - Baseline runner scaffold: pluggable frontier model interface - Fix: CognitiveTurnPipeline.run() crashed on turn_log[-1] when the unknown-domain gate returned a stub without appending to turn_log - ADR-0016, eval_methodology.md, PROGRESS.md, capability gates session log Phase 0 exit audit found two methodology issues: 1. Pipeline turn_log crash (fixed here) 2. Versor drift in multi-turn sessions (pre-existing, under investigation)
64 lines
2.7 KiB
Markdown
64 lines
2.7 KiB
Markdown
# ADR-0016 — Capability Roadmap and Eval Methodology
|
|
|
|
**Status:** Accepted
|
|
**Date:** 2026-05-15
|
|
**Authors:** Joshua Shay
|
|
**Derived from:** `docs/sessions/SESSION-2026-05-15-capability-gates.md`
|
|
|
|
## Context
|
|
|
|
CORE needs a falsifiable framework for measuring progress toward its design
|
|
goals. Without one, "are we there yet" remains a subjective question and
|
|
progress drifts toward vibes-based evaluation.
|
|
|
|
The 2026-05-15 session deliberated on:
|
|
|
|
- What fluency means for the three foundational languages (English, Hebrew,
|
|
Koine Greek) and when engineering ends and curriculum begins.
|
|
- What additional capability dimensions matter for an AGI-aspiring architecture.
|
|
- How CORE's structural properties compare to modern transformer architectures.
|
|
- How to build honest benchmarks that resist overfitting.
|
|
|
|
## Decision
|
|
|
|
Adopt the Verifiable Competence Benchmark framework defined in
|
|
`docs/capability_roadmap.md` as the governing plan for CORE's capability
|
|
development. The framework consists of:
|
|
|
|
1. **Benchmark Discipline (Part I)** — five rules that govern every eval lane:
|
|
three-set splits, versioned difficulty escalation, adversarial regeneration
|
|
on pass, frontier baseline tracking, and honest reporting.
|
|
|
|
2. **Six Phases (Part II):**
|
|
- Phase 0: Methodology lock-in (eval infrastructure)
|
|
- Phase 1: Foundational Triple (fluency, domain acquisition, identity)
|
|
- Phase 2: Structural Wins (provenance, monotonic learning, calibration,
|
|
symbolic logic, adversarial identity)
|
|
- Phase 3: Reasoning Depth (compositionality, inference closure,
|
|
introspection, multi-step reasoning, cross-domain transfer)
|
|
- Phase 4: Scale and Efficiency (sample efficiency, vault cost curves,
|
|
multi-agent composition)
|
|
- Phase 5: Curriculum Era (open-ended domain acquisition)
|
|
|
|
3. **Eval contract template** — every lane lives in `evals/<lane>/` with:
|
|
`contract.md`, `dev/`, `public/v1/`, `holdouts/`, `runner.py`, `baselines/`,
|
|
`results/`.
|
|
|
|
4. **Open scope decisions** to be pinned before Phase 3:
|
|
- Agency (responsive vs. goal-directed)
|
|
- Tool use (typed deterministic operators)
|
|
- Code generation (first-class articulation target)
|
|
|
|
## Consequences
|
|
|
|
- Every new eval lane must follow the convention or it does not merge.
|
|
- The existing `core eval cognition` is retrofitted as the first lane under the
|
|
new convention (Phase 0 forcing function).
|
|
- Progress is tracked in `docs/PROGRESS.md` with evidence links.
|
|
- The roadmap itself is versioned; amendments are dated, never silently rewritten.
|
|
|
|
## References
|
|
|
|
- `docs/capability_roadmap.md` — full roadmap
|
|
- `docs/sessions/SESSION-2026-05-15-capability-gates.md` — deliberation log
|
|
- `docs/eval_methodology.md` — extracted Part I (benchmark discipline contract)
|