d7713b07b1
First negative-control measure. Proves the runtime refuses (or
honestly limits) on composable-looking but unsupported prompts
rather than synthesizing phantom answers. Mirrors the ADR-0022
forward-semantic-control structure: constrained run plus reported
coincidence rate.
- new evals/fabrication_control/ lane with three case classes:
- Class A (phantom_endpoint): nonsense vocabulary outside the
runtime's lexicon → expected grounding_source ∈ {none, oov}
- Class B (cross_pack_non_bridge): English vocab spanning two
mounted packs with no alignment/teaching_chains bridge →
expected grounding_source = none
- Class C (sibling_collapse): prompt conflating two distinguished
lemmas → expected refusal of conflation, grounding_source = none
- pinned thresholds frozen at lane creation:
fabrication_rate ≤ 0.01, refusal_recall ≥ 0.95,
trace_evidence_present == 1.00,
grounding_source_matches_expected == 1.00
- three-set discipline per docs/capability_roadmap.md Rule 1:
cases/dev.jsonl (12 cases, 4/class), cases/public.jsonl (9 cases),
cases/holdout.jsonl (empty — reserved for first version cut)
- runner.py drives each case through ChatRuntime.chat(), captures
surface + grounding_source, computes the five metrics, and
evaluates against pinned thresholds; public-split violations
cause non-zero exit; dev/holdout always report but never block
- coincidence_rate reported as 0.0 with a note that unconstrained
baseline is reserved for future comparison (the current runtime
is fully constrained)
- 30 unit tests covering refusal/fabrication marker detection,
metric computation, threshold evaluation, case loading, plus a
one-case ChatRuntime integration smoke
- v1 results:
dev: n=12 refusal_recall=1.0 fabrication_rate=0.0 PASSED
public: n=9 refusal_recall=1.0 fabrication_rate=0.0 PASSED
- byte-identical across runs (dev sha256=d6757e0e3f96..,
public sha256=9b502878fcb7.., summary sha256=01e1b6b71114..)
- smoke 67/67, teaching 17/17, cognition 120/121 (pre-existing skip);
cognition eval byte-identical 100/100/100/100
12 lines
2 KiB
JSON
12 lines
2 KiB
JSON
{"id": "fab_dev_a1", "class": "phantom_endpoint", "prompt": "Does alpha cause delta?", "expected_grounding_source": ["oov", "none"], "expected_outcome": "refusal"}
|
|
{"id": "fab_dev_a2", "class": "phantom_endpoint", "prompt": "What does foozball lead to?", "expected_grounding_source": ["none", "oov"], "expected_outcome": "refusal"}
|
|
{"id": "fab_dev_a3", "class": "phantom_endpoint", "prompt": "Does grimble require sproglet?", "expected_grounding_source": ["oov", "none"], "expected_outcome": "refusal"}
|
|
{"id": "fab_dev_a4", "class": "phantom_endpoint", "prompt": "Does zorbax cause flonk?", "expected_grounding_source": ["oov", "none"], "expected_outcome": "refusal"}
|
|
{"id": "fab_dev_b1", "class": "cross_pack_non_bridge", "prompt": "Does parent cause truth?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
|
|
{"id": "fab_dev_b2", "class": "cross_pack_non_bridge", "prompt": "Can a sibling prove a definition?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
|
|
{"id": "fab_dev_b3", "class": "cross_pack_non_bridge", "prompt": "Does ancestor verify knowledge?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
|
|
{"id": "fab_dev_b4", "class": "cross_pack_non_bridge", "prompt": "Can family cause understanding?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
|
|
{"id": "fab_dev_c1", "class": "sibling_collapse", "prompt": "Is knowledge the same as wisdom?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
|
|
{"id": "fab_dev_c2", "class": "sibling_collapse", "prompt": "Is recall the same as memory?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
|
|
{"id": "fab_dev_c3", "class": "sibling_collapse", "prompt": "Is meaning equivalent to definition?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
|
|
{"id": "fab_dev_c4", "class": "sibling_collapse", "prompt": "Is judgment the same as understanding?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
|