core/evals/capability_index
Shay 514c6c52ca feat(evals): AGI-roadmap Phase 1 — cross-domain capability index (the MEASURE yardstick)
The instrument that gates every later "more capable" claim and makes "general,
not narrow" a number. evals/capability_index/ composes the self-loading
independent-gold reasoning lanes (deductive_logic, dimensional, relational_metric)
into one report with honest, un-gameable axes:

- accuracy (of committed answers; wrong stays 0 in assert mode),
- coverage (attempted-not-refused),
- coverage_geomean — the headline: geometric mean of per-domain coverage, which is
  0 if ANY domain has zero coverage, so a narrow per-domain win cannot move it; it
  rises only when breadth rises,
- capability_score = coverage_geomean × accuracy, HARD-GATED to 0 if any domain
  committed a wrong answer (assert-mode invariant),
- a deterministic digest (the replayable baseline the autonomous loop must climb).

Baseline (today): score 0.9196, accuracy 1.0, breadth 3, wrong_total 0 — high
because all three composed lanes are formal/structured; when comprehension-gated
NL domains join, the geomean will honestly drop to expose the breadth gap (the
instrument working). Adapters surface any lane that fails to run as not_covered —
no silent drop (proven: it caught a deductive-report shape mismatch mid-build).

Pure aggregation + the geomean anti-gaming property + the wrong=0 hard gate are
unit-tested; a real-composition integration test asserts wrong=0 + breadth=3.
10 tests + 52 architectural invariants pass. Additive (new evals/ package).
Part of docs/analysis/AGI-candidacy-autonomous-improvement-roadmap-2026-06-05.md (Phase 1).
2026-06-05 15:17:46 -07:00
..
__init__.py feat(evals): AGI-roadmap Phase 1 — cross-domain capability index (the MEASURE yardstick) 2026-06-05 15:17:46 -07:00
__main__.py feat(evals): AGI-roadmap Phase 1 — cross-domain capability index (the MEASURE yardstick) 2026-06-05 15:17:46 -07:00
adapters.py feat(evals): AGI-roadmap Phase 1 — cross-domain capability index (the MEASURE yardstick) 2026-06-05 15:17:46 -07:00
index.py feat(evals): AGI-roadmap Phase 1 — cross-domain capability index (the MEASURE yardstick) 2026-06-05 15:17:46 -07:00