core-labs/core - Forgejo: Beyond coding. We forge.

core-labs/core

Fork 0

Commit graph

Author	SHA1	Message	Date
Shay	6b25069da8	feat(adr-0054): vault recall indexing/batching + holdout split wired Two doctrine-aligned CLAUDE.md items closed together. Part 1 — vault indexing + batching (item #4): - VaultStore lazy _matrix_cache (invalidated on store / reproject / eviction); vault_recall(prebuilt_matrix=...) skips deque→ndarray rebuild on hot path - New vault_recall_batch + VaultStore.recall_batch — B queries scored in one component-serial sweep, bit-identical to per-query vault_recall (3 seeds × 7 queries × N=137 parity test) - No approximation, no hot-path repair, scoring arithmetic unchanged Part 2 — holdout split wired: - LaneInfo.holdout_cases_path resolves plaintext holdouts in fixed priority; sealed (.age) holdouts stay in holdout_runner - framework.run_lane(split="holdout") + argparse --split choices - First official cognition holdout numbers: 19 cases, intent 100%, surface 94.7%, term_capture 70.8%, versor 100% — single miss is predicted correction_truth_040 (ADR-0053 scope-limit) Tests: 21 new vault tests + 10 new framework tests. Lanes: smoke 67, cognition 121, runtime 19, teaching 17, packs 6, algebra 132 — all green. versor_condition < 1e-6 invariant preserved.	2026-05-18 07:58:57 -07:00
Shay	1e01f7794e	feat(evals): Phase 0 — benchmark methodology lock-in and eval framework Implement the eval infrastructure defined in ADR-0016 before building new eval lanes. This establishes the discipline that governs the entire capability roadmap. - Generic eval framework (evals/framework.py): lane discovery, versioned scoring, result persistence - Cognition lane retrofitted into new convention: 45 cases split into stratified dev (13) / public v1 (13) / holdout (19) sets with contract, runner, and recorded results - Generalized `core eval <lane>` CLI: dynamic lane discovery, --list, --version, --split, --save, --json flags - Holdout runner scaffold: plaintext fallback, encryption interface ready - Baseline runner scaffold: pluggable frontier model interface - Fix: CognitiveTurnPipeline.run() crashed on turn_log[-1] when the unknown-domain gate returned a stub without appending to turn_log - ADR-0016, eval_methodology.md, PROGRESS.md, capability gates session log Phase 0 exit audit found two methodology issues: 1. Pipeline turn_log crash (fixed here) 2. Versor drift in multi-turn sessions (pre-existing, under investigation)	2026-05-15 22:36:53 -07:00

Author

SHA1

Message

Date

Shay

6b25069da8

feat(adr-0054): vault recall indexing/batching + holdout split wired

Two doctrine-aligned CLAUDE.md items closed together.

Part 1 — vault indexing + batching (item #4):
- VaultStore lazy _matrix_cache (invalidated on store / reproject /
  eviction); vault_recall(prebuilt_matrix=...) skips deque→ndarray
  rebuild on hot path
- New vault_recall_batch + VaultStore.recall_batch — B queries
  scored in one component-serial sweep, bit-identical to per-query
  vault_recall (3 seeds × 7 queries × N=137 parity test)
- No approximation, no hot-path repair, scoring arithmetic
  unchanged

Part 2 — holdout split wired:
- LaneInfo.holdout_cases_path resolves plaintext holdouts in fixed
  priority; sealed (.age) holdouts stay in holdout_runner
- framework.run_lane(split="holdout") + argparse --split choices
- First official cognition holdout numbers: 19 cases, intent 100%,
  surface 94.7%, term_capture 70.8%, versor 100% — single miss is
  predicted correction_truth_040 (ADR-0053 scope-limit)

Tests: 21 new vault tests + 10 new framework tests. Lanes: smoke
67, cognition 121, runtime 19, teaching 17, packs 6, algebra 132 —
all green. versor_condition < 1e-6 invariant preserved.

2026-05-18 07:58:57 -07:00

Shay

1e01f7794e

feat(evals): Phase 0 — benchmark methodology lock-in and eval framework

Implement the eval infrastructure defined in ADR-0016 before building new
eval lanes. This establishes the discipline that governs the entire
capability roadmap.

- Generic eval framework (evals/framework.py): lane discovery, versioned
  scoring, result persistence
- Cognition lane retrofitted into new convention: 45 cases split into
  stratified dev (13) / public v1 (13) / holdout (19) sets with contract,
  runner, and recorded results
- Generalized `core eval <lane>` CLI: dynamic lane discovery, --list,
  --version, --split, --save, --json flags
- Holdout runner scaffold: plaintext fallback, encryption interface ready
- Baseline runner scaffold: pluggable frontier model interface
- Fix: CognitiveTurnPipeline.run() crashed on turn_log[-1] when the
  unknown-domain gate returned a stub without appending to turn_log
- ADR-0016, eval_methodology.md, PROGRESS.md, capability gates session log

Phase 0 exit audit found two methodology issues:
1. Pipeline turn_log crash (fixed here)
2. Versor drift in multi-turn sessions (pre-existing, under investigation)

2026-05-15 22:36:53 -07:00

2 commits