a447dce5d1
INV-25 makes the GSM8K lesson structural: no capability claim is valid unless its gold is computed by a procedure sharing no code with the system under test. Three meaningfully-failing checks (proven able to fail): 25a oracle imports no SUT module (AST); 25b every committed deductive gold is reproduced by the independent oracle AND matched by the engine; 25c an unsound engine disagrees on committed cases. SHA-pin the deductive lane (deductive_logic_v1, dev+holdout+external 716/716, wrong=0, refused=0) via a deterministic --report writer + run_as_module (the lane dir's local generate.py shadows the package in script mode) + CLAIMS regen. Fix drift the review surfaced: contract.md + pivot doc claimed an 8,000/7,340 fuzz and an 'external mirror' -> corrected to the real gated 3,000-case fuzz (2,796 definite) and 'hand-authored, NOT a published-benchmark mirror'; pivot doc GSM8K holdout 0->5 (all 5 are R1 reconstruction; composer scored 0). Validated: smoke 78, deductive 23, proof 29, invariants 44, lane-sha+claims 9.
73 lines
1.3 KiB
JSON
73 lines
1.3 KiB
JSON
{
|
|
"adr": "ADR-0206",
|
|
"aggregate": {
|
|
"correct": 716,
|
|
"n": 716,
|
|
"refused": 0,
|
|
"wrong": 0
|
|
},
|
|
"all_correct": true,
|
|
"lane": "deductive_logic",
|
|
"refused_is_zero": true,
|
|
"schema_version": 1,
|
|
"splits": {
|
|
"dev": {
|
|
"all_cases_correct": true,
|
|
"by_gold": {
|
|
"entailed": 37,
|
|
"refuted": 37,
|
|
"unknown": 126
|
|
},
|
|
"correct_by_gold": {
|
|
"entailed": 37,
|
|
"refuted": 37,
|
|
"unknown": 126
|
|
},
|
|
"counts": {
|
|
"correct": 200,
|
|
"refused": 0,
|
|
"wrong": 0
|
|
},
|
|
"n": 200
|
|
},
|
|
"external_v1": {
|
|
"all_cases_correct": true,
|
|
"by_gold": {
|
|
"entailed": 8,
|
|
"refuted": 4,
|
|
"unknown": 4
|
|
},
|
|
"correct_by_gold": {
|
|
"entailed": 8,
|
|
"refuted": 4,
|
|
"unknown": 4
|
|
},
|
|
"counts": {
|
|
"correct": 16,
|
|
"refused": 0,
|
|
"wrong": 0
|
|
},
|
|
"n": 16
|
|
},
|
|
"holdout_v1": {
|
|
"all_cases_correct": true,
|
|
"by_gold": {
|
|
"entailed": 117,
|
|
"refuted": 110,
|
|
"unknown": 273
|
|
},
|
|
"correct_by_gold": {
|
|
"entailed": 117,
|
|
"refuted": 110,
|
|
"unknown": 273
|
|
},
|
|
"counts": {
|
|
"correct": 500,
|
|
"refused": 0,
|
|
"wrong": 0
|
|
},
|
|
"n": 500
|
|
}
|
|
},
|
|
"wrong_is_zero": true
|
|
}
|