2e4e45b49b
Phase 2's first lane: every articulated claim must back-point to one of
{pack axiom, vault entry, teaching event}, and replay must reproduce the
trace bit-for-bit.
Components:
- core/cognition/provenance.py: Provenance dataclass + compute_provenance()
deriving sources from a CognitiveTurnResult. Pack source = non-UNKNOWN
intent.tag (pack-defined intent rule matched); vault source = vault_hits
count; teaching source = pack_mutation_proposal.proposal_id.
- evals/provenance/{contract.md, runner.py, dev/, public/v1/, holdouts/v1/}:
45 cases across pack_axiom / vault_recall / teaching / mixed categories.
- tests/test_provenance.py: 6 unit tests covering all source-kind profiles.
Sub-metrics (all four must pass):
- replay_determinism: same input + fresh runtime -> same trace_hash
- input_sensitivity: distinct prompts -> distinct trace_hashes
- source_attribution: every expected source kind present in Provenance
- source_validity: every cited source resolves to a real artefact
Results:
- dev: 10/10 (all sub-metrics 1.0)
- public/v1: 20/20 (all sub-metrics 1.0)
- holdouts/v1: 15/15 (all sub-metrics 1.0)
PROGRESS.md updated to mark Phase 2 in progress with provenance v1 complete.
10 lines
1.5 KiB
JSON
10 lines
1.5 KiB
JSON
{"id": "PROV-DEV-001", "category": "pack_axiom", "prime": [], "prompt": "What is truth?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-DEV-002", "category": "pack_axiom", "prime": [], "prompt": "Why does light reveal?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-DEV-003", "category": "pack_axiom", "prime": [], "prompt": "Compare knowledge and wisdom", "expected_sources": ["pack"]}
|
|
{"id": "PROV-DEV-004", "category": "pack_axiom", "prime": [], "prompt": "How do I learn truth?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-DEV-005", "category": "vault_recall", "prime": ["What is logos?"], "prompt": "What is logos?", "expected_sources": ["pack", "vault"]}
|
|
{"id": "PROV-DEV-006", "category": "vault_recall", "prime": ["What is wisdom?", "What is knowledge?"], "prompt": "Compare wisdom and knowledge", "expected_sources": ["pack", "vault"]}
|
|
{"id": "PROV-DEV-007", "category": "teaching", "prime": ["What is truth?"], "prompt": "No, that's not quite right.", "expected_sources": ["pack", "teaching"]}
|
|
{"id": "PROV-DEV-008", "category": "teaching", "prime": ["What is wisdom?"], "prompt": "Actually wisdom is applied knowledge.", "expected_sources": ["pack", "teaching"]}
|
|
{"id": "PROV-DEV-009", "category": "mixed", "prime": ["What is light?", "What is truth?"], "prompt": "Actually light is also revelation.", "expected_sources": ["pack", "vault", "teaching"]}
|
|
{"id": "PROV-DEV-010", "category": "mixed", "prime": ["What is creation?"], "prompt": "Why does creation matter?", "expected_sources": ["pack", "vault"]}
|