2e4e45b49b
Phase 2's first lane: every articulated claim must back-point to one of
{pack axiom, vault entry, teaching event}, and replay must reproduce the
trace bit-for-bit.
Components:
- core/cognition/provenance.py: Provenance dataclass + compute_provenance()
deriving sources from a CognitiveTurnResult. Pack source = non-UNKNOWN
intent.tag (pack-defined intent rule matched); vault source = vault_hits
count; teaching source = pack_mutation_proposal.proposal_id.
- evals/provenance/{contract.md, runner.py, dev/, public/v1/, holdouts/v1/}:
45 cases across pack_axiom / vault_recall / teaching / mixed categories.
- tests/test_provenance.py: 6 unit tests covering all source-kind profiles.
Sub-metrics (all four must pass):
- replay_determinism: same input + fresh runtime -> same trace_hash
- input_sensitivity: distinct prompts -> distinct trace_hashes
- source_attribution: every expected source kind present in Provenance
- source_validity: every cited source resolves to a real artefact
Results:
- dev: 10/10 (all sub-metrics 1.0)
- public/v1: 20/20 (all sub-metrics 1.0)
- holdouts/v1: 15/15 (all sub-metrics 1.0)
PROGRESS.md updated to mark Phase 2 in progress with provenance v1 complete.
15 lines
2.3 KiB
JSON
15 lines
2.3 KiB
JSON
{"id": "PROV-H1-001", "category": "pack_axiom", "prime": [], "prompt": "What is knowledge?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-H1-002", "category": "pack_axiom", "prime": [], "prompt": "What is distinction?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-H1-003", "category": "pack_axiom", "prime": [], "prompt": "Why does learning matter?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-H1-004", "category": "pack_axiom", "prime": [], "prompt": "How do I distinguish truth?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-H1-005", "category": "pack_axiom", "prime": [], "prompt": "Compare knowledge and understanding", "expected_sources": ["pack"]}
|
|
{"id": "PROV-H1-006", "category": "pack_axiom", "prime": [], "prompt": "Does wisdom require knowledge?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-H1-007", "category": "vault_recall", "prime": ["What is knowledge?"], "prompt": "What is knowledge?", "expected_sources": ["pack", "vault"]}
|
|
{"id": "PROV-H1-008", "category": "vault_recall", "prime": ["What is distinction?"], "prompt": "Why does distinction matter?", "expected_sources": ["pack", "vault"]}
|
|
{"id": "PROV-H1-009", "category": "vault_recall", "prime": ["What is understanding?", "What is wisdom?"], "prompt": "Compare understanding and wisdom", "expected_sources": ["pack", "vault"]}
|
|
{"id": "PROV-H1-010", "category": "vault_recall", "prime": ["What is correction?"], "prompt": "Is correction necessary?", "expected_sources": ["pack", "vault"]}
|
|
{"id": "PROV-H1-011", "category": "teaching", "prime": ["What is knowledge?"], "prompt": "No, knowledge alone is not wisdom.", "expected_sources": ["pack", "teaching"]}
|
|
{"id": "PROV-H1-012", "category": "teaching", "prime": ["What is distinction?"], "prompt": "Actually distinction requires comparison.", "expected_sources": ["pack", "teaching"]}
|
|
{"id": "PROV-H1-013", "category": "teaching", "prime": ["What is correction?"], "prompt": "No, correction needs review first.", "expected_sources": ["pack", "teaching"]}
|
|
{"id": "PROV-H1-014", "category": "mixed", "prime": ["What is light?", "What is creation?"], "prompt": "Actually light is part of creation.", "expected_sources": ["pack", "vault", "teaching"]}
|
|
{"id": "PROV-H1-015", "category": "mixed", "prime": ["What is wisdom?"], "prompt": "How do I cultivate wisdom?", "expected_sources": ["pack", "vault"]}
|