Shay
|
075169c33c
|
feat(evals): v2 lanes — monotonic-learning + provenance
monotonic-learning v2:
public/v2 — 5 domains × 3-4 probes × 20 cycles (377 ops)
domains: truth, light, wisdom, order, memory
max_regression=0.0, floor_score=1.0
holdouts/v2 — 4 distinct domains × 3-4 probes × 18 cycles (284 ops)
domains: creation, knowledge, reason, spirit
max_regression=0.0, floor_score=1.0
Demonstrates the structural claim (zero regression on prior domains
as new ones accumulate) at substantially deeper cycle count and
broader domain breadth than v1.
provenance v2:
public/v2 — 30 cases across pack_axiom, vault_recall, teaching, mixed
deeper priming (3-5 turns), mixed-kind cases combining
pack + vault + teaching sources in one probe
source_attribution=1.0, source_validity=1.0,
replay_determinism=1.0, input_sensitivity=1.0
holdouts/v2 — 20 cases on distinct vocabulary
all sub-metrics 1.0
Generator: scripts/generate_monotonic_cases.py extended with three
extra domain probe sets (order, memory, reason, spirit) and split
definitions for v2.
|
2026-05-16 13:03:28 -07:00 |
|
Shay
|
2e4e45b49b
|
feat(evals): provenance lane v1 — replay determinism + source back-pointers
Phase 2's first lane: every articulated claim must back-point to one of
{pack axiom, vault entry, teaching event}, and replay must reproduce the
trace bit-for-bit.
Components:
- core/cognition/provenance.py: Provenance dataclass + compute_provenance()
deriving sources from a CognitiveTurnResult. Pack source = non-UNKNOWN
intent.tag (pack-defined intent rule matched); vault source = vault_hits
count; teaching source = pack_mutation_proposal.proposal_id.
- evals/provenance/{contract.md, runner.py, dev/, public/v1/, holdouts/v1/}:
45 cases across pack_axiom / vault_recall / teaching / mixed categories.
- tests/test_provenance.py: 6 unit tests covering all source-kind profiles.
Sub-metrics (all four must pass):
- replay_determinism: same input + fresh runtime -> same trace_hash
- input_sensitivity: distinct prompts -> distinct trace_hashes
- source_attribution: every expected source kind present in Provenance
- source_validity: every cited source resolves to a real artefact
Results:
- dev: 10/10 (all sub-metrics 1.0)
- public/v1: 20/20 (all sub-metrics 1.0)
- holdouts/v1: 15/15 (all sub-metrics 1.0)
PROGRESS.md updated to mark Phase 2 in progress with provenance v1 complete.
|
2026-05-16 11:45:00 -07:00 |
|