2e4e45b49b
Phase 2's first lane: every articulated claim must back-point to one of
{pack axiom, vault entry, teaching event}, and replay must reproduce the
trace bit-for-bit.
Components:
- core/cognition/provenance.py: Provenance dataclass + compute_provenance()
deriving sources from a CognitiveTurnResult. Pack source = non-UNKNOWN
intent.tag (pack-defined intent rule matched); vault source = vault_hits
count; teaching source = pack_mutation_proposal.proposal_id.
- evals/provenance/{contract.md, runner.py, dev/, public/v1/, holdouts/v1/}:
45 cases across pack_axiom / vault_recall / teaching / mixed categories.
- tests/test_provenance.py: 6 unit tests covering all source-kind profiles.
Sub-metrics (all four must pass):
- replay_determinism: same input + fresh runtime -> same trace_hash
- input_sensitivity: distinct prompts -> distinct trace_hashes
- source_attribution: every expected source kind present in Provenance
- source_validity: every cited source resolves to a real artefact
Results:
- dev: 10/10 (all sub-metrics 1.0)
- public/v1: 20/20 (all sub-metrics 1.0)
- holdouts/v1: 15/15 (all sub-metrics 1.0)
PROGRESS.md updated to mark Phase 2 in progress with provenance v1 complete.
20 lines
2.8 KiB
JSON
20 lines
2.8 KiB
JSON
{"id": "PROV-V1-001", "category": "pack_axiom", "prime": [], "prompt": "What is light?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-V1-002", "category": "pack_axiom", "prime": [], "prompt": "What is wisdom?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-V1-003", "category": "pack_axiom", "prime": [], "prompt": "What is creation?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-V1-004", "category": "pack_axiom", "prime": [], "prompt": "Why does word matter?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-V1-005", "category": "pack_axiom", "prime": [], "prompt": "Why does correction help?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-V1-006", "category": "pack_axiom", "prime": [], "prompt": "How do I find wisdom?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-V1-007", "category": "pack_axiom", "prime": [], "prompt": "Compare light and darkness", "expected_sources": ["pack"]}
|
|
{"id": "PROV-V1-008", "category": "pack_axiom", "prime": [], "prompt": "Compare truth and falsehood", "expected_sources": ["pack"]}
|
|
{"id": "PROV-V1-009", "category": "pack_axiom", "prime": [], "prompt": "Is wisdom valuable?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-V1-010", "category": "pack_axiom", "prime": [], "prompt": "Is truth absolute?", "expected_sources": ["pack"]}
|
|
{"id": "PROV-V1-011", "category": "vault_recall", "prime": ["What is wisdom?"], "prompt": "What is wisdom?", "expected_sources": ["pack", "vault"]}
|
|
{"id": "PROV-V1-012", "category": "vault_recall", "prime": ["What is light?"], "prompt": "Why does light reveal?", "expected_sources": ["pack", "vault"]}
|
|
{"id": "PROV-V1-013", "category": "vault_recall", "prime": ["What is creation?", "What is word?"], "prompt": "Compare creation and word", "expected_sources": ["pack", "vault"]}
|
|
{"id": "PROV-V1-014", "category": "vault_recall", "prime": ["What is logos?", "What is dabar?"], "prompt": "Compare logos and dabar", "expected_sources": ["pack", "vault"]}
|
|
{"id": "PROV-V1-015", "category": "vault_recall", "prime": ["What is truth?"], "prompt": "Is truth coherent?", "expected_sources": ["pack", "vault"]}
|
|
{"id": "PROV-V1-016", "category": "teaching", "prime": ["What is truth?"], "prompt": "No, that is incomplete.", "expected_sources": ["pack", "teaching"]}
|
|
{"id": "PROV-V1-017", "category": "teaching", "prime": ["What is creation?"], "prompt": "Actually creation includes word.", "expected_sources": ["pack", "teaching"]}
|
|
{"id": "PROV-V1-018", "category": "teaching", "prime": ["What is light?"], "prompt": "No, that misses revelation.", "expected_sources": ["pack", "teaching"]}
|
|
{"id": "PROV-V1-019", "category": "mixed", "prime": ["What is wisdom?", "What is knowledge?"], "prompt": "Actually wisdom is more than knowledge.", "expected_sources": ["pack", "vault", "teaching"]}
|
|
{"id": "PROV-V1-020", "category": "mixed", "prime": ["What is light?", "What is truth?"], "prompt": "Why does light relate to truth?", "expected_sources": ["pack", "vault"]}
|