core-labs/core - Forgejo: Beyond coding. We forge.

core-labs/core

Fork 0

Commit graph

Author	SHA1	Message	Date
Shay	257fd4503d	feat(evals): ADR-0105 — sealed holdout encryption via age (#108 ) * feat(evals): add pyrage dependency * feat(evals): add sealed holdout path resolution * feat(evals): implement sealed holdout decryption * feat(evals): add sealed holdout CLI * test(evals): add sealed holdout encryption tests * docs(decisions): add ADR-0105 sealed holdout encryption * feat(evals): route holdout split through sealed decryptor * docs(decisions): add ADR-0105 index entry * chore: restore project description * fix(evals): use pyrage Identity.from_str and pin curriculum SHA - holdout_runner: pyrage exposes Identity.from_str, not from_file; parse identity file by line and pass list[Identity] into decrypt(). Restores PR 108's sealed-holdout test suite to green. - verify_lane_shas: realign curriculum_loop_closure pin with the actual deterministic runner output (carryover from PR 107).	2026-05-22 10:09:43 -07:00
Shay	1e01f7794e	feat(evals): Phase 0 — benchmark methodology lock-in and eval framework Implement the eval infrastructure defined in ADR-0016 before building new eval lanes. This establishes the discipline that governs the entire capability roadmap. - Generic eval framework (evals/framework.py): lane discovery, versioned scoring, result persistence - Cognition lane retrofitted into new convention: 45 cases split into stratified dev (13) / public v1 (13) / holdout (19) sets with contract, runner, and recorded results - Generalized `core eval <lane>` CLI: dynamic lane discovery, --list, --version, --split, --save, --json flags - Holdout runner scaffold: plaintext fallback, encryption interface ready - Baseline runner scaffold: pluggable frontier model interface - Fix: CognitiveTurnPipeline.run() crashed on turn_log[-1] when the unknown-domain gate returned a stub without appending to turn_log - ADR-0016, eval_methodology.md, PROGRESS.md, capability gates session log Phase 0 exit audit found two methodology issues: 1. Pipeline turn_log crash (fixed here) 2. Versor drift in multi-turn sessions (pre-existing, under investigation)	2026-05-15 22:36:53 -07:00

Author

SHA1

Message

Date

Shay

257fd4503d

feat(evals): ADR-0105 — sealed holdout encryption via age (#108 )

* feat(evals): add pyrage dependency

* feat(evals): add sealed holdout path resolution

* feat(evals): implement sealed holdout decryption

* feat(evals): add sealed holdout CLI

* test(evals): add sealed holdout encryption tests

* docs(decisions): add ADR-0105 sealed holdout encryption

* feat(evals): route holdout split through sealed decryptor

* docs(decisions): add ADR-0105 index entry

* chore: restore project description

* fix(evals): use pyrage Identity.from_str and pin curriculum SHA

- holdout_runner: pyrage exposes Identity.from_str, not from_file; parse
  identity file by line and pass list[Identity] into decrypt(). Restores
  PR 108's sealed-holdout test suite to green.
- verify_lane_shas: realign curriculum_loop_closure pin with the actual
  deterministic runner output (carryover from PR 107).

2026-05-22 10:09:43 -07:00

Shay

1e01f7794e

feat(evals): Phase 0 — benchmark methodology lock-in and eval framework

Implement the eval infrastructure defined in ADR-0016 before building new
eval lanes. This establishes the discipline that governs the entire
capability roadmap.

- Generic eval framework (evals/framework.py): lane discovery, versioned
  scoring, result persistence
- Cognition lane retrofitted into new convention: 45 cases split into
  stratified dev (13) / public v1 (13) / holdout (19) sets with contract,
  runner, and recorded results
- Generalized `core eval <lane>` CLI: dynamic lane discovery, --list,
  --version, --split, --save, --json flags
- Holdout runner scaffold: plaintext fallback, encryption interface ready
- Baseline runner scaffold: pluggable frontier model interface
- Fix: CognitiveTurnPipeline.run() crashed on turn_log[-1] when the
  unknown-domain gate returned a stub without appending to turn_log
- ADR-0016, eval_methodology.md, PROGRESS.md, capability gates session log

Phase 0 exit audit found two methodology issues:
1. Pipeline turn_log crash (fixed here)
2. Versor drift in multi-turn sessions (pre-existing, under investigation)

2026-05-15 22:36:53 -07:00

2 commits