- grammatical-coverage holdout v1: 52 cases across all 13 constructions, 100% pass
- zero-code-domain-acquisition lane: contract + 3 surprise domains (kinship,
calendar, color) with vocabulary, relations, axioms, teaching examples,
and dev prompts; pack closure verified for all three domains
- he_core_cognition_v1: 20 entries in Hebrew script with morphology decomposition
(triliteral roots, binyanim, aspect/person/gender/number); depth_root role
with fail_closed OOV policy
- grc_logos_cognition_v1: 20 entries in polytonic Greek with morphology
decomposition (stems, prefix/suffix chains, declension class, tense/voice/
mood/person); depth_relation role with fail_closed OOV policy
Establish the grammatical-coverage eval lane with 13 English v1
constructions (simple declarative, negation, conjunction, disjunction,
embedded clause, relative clause, quantification, tense, aspect).
- contract.md with scoring rubric and pass thresholds
- runner.py conforming to framework interface
- dev set: 41 cases (baseline: 24.4%, only C01/C10 pass)
- public v1: 36 cases (baseline: 19.4%, only C01/C10 pass)
- holdout and realizer engineering are next
The realizer currently handles only simple present-tense SVO declaratives.
Negation, conjunction, embedding, quantification, tense, and aspect all
need engineering work.
The top-level --version flag (bool) collided with eval's --version argument
(string). Rename the top-level dest to print_version so both coexist.
Also mark Phase 0 exit gate as complete in PROGRESS.md:
- v1 public: 13/13 (100% all metrics)
- holdout: 19/19 (unsealed plaintext, encryption deferred)
- baseline: scaffold with pluggable BaselineModel protocol
Implement the eval infrastructure defined in ADR-0016 before building new
eval lanes. This establishes the discipline that governs the entire
capability roadmap.
- Generic eval framework (evals/framework.py): lane discovery, versioned
scoring, result persistence
- Cognition lane retrofitted into new convention: 45 cases split into
stratified dev (13) / public v1 (13) / holdout (19) sets with contract,
runner, and recorded results
- Generalized `core eval <lane>` CLI: dynamic lane discovery, --list,
--version, --split, --save, --json flags
- Holdout runner scaffold: plaintext fallback, encryption interface ready
- Baseline runner scaffold: pluggable frontier model interface
- Fix: CognitiveTurnPipeline.run() crashed on turn_log[-1] when the
unknown-domain gate returned a stub without appending to turn_log
- ADR-0016, eval_methodology.md, PROGRESS.md, capability gates session log
Phase 0 exit audit found two methodology issues:
1. Pipeline turn_log crash (fixed here)
2. Versor drift in multi-turn sessions (pre-existing, under investigation)
