core/scripts
Shay 25580e18b0
fix(adr-0191): candidate-graph completeness guard — real-corpus wrong 5→0 (#496)
* fix(adr-0191): candidate-graph completeness guard — real-corpus wrong 5->0

The candidate-graph reader (serving) checked grounding + round-trip but had
no completeness obligation, so problems whose later clauses failed to parse
emitted a partial reading. Over the full 7,473-question real GSM8K train
split this confabulated 5 answers (wrong!=0) the 47-case train_sample hid;
2 were regressions from #488.

Add the missing admissibility leg (mirrors the derivation reader's verify.py):
every source quantity (all statements + question) must be consumed by the
chosen reading, else refuse. Refusal-only -> cannot create a wrong answer.
Number-sense is pack-authoritative (en_numerics_v1 parse_compound_cardinal +
lookup_multiplier + all 6 currency symbols) so it never disagrees with the
engine; aggregating initials expose consumed_value_tokens provenance.

Evidence: real-corpus wrong 5->0, correct held at 4; train_sample byte-
identical 4/46/0; G1-G5+S1+G3.1 green; smoke 67 passed; math_teaching_corpus
lane byte-identical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(adr-0191): committed full-corpus GSM8K microscope (standing wrong=0 + coverage instrument)

Promotes the throwaway tmp/ microscope that found the 5 confabulations into a
committed tool. Runs the canonical serving reader over any GSM8K corpus and
reports failures-first: correct/wrong/refused, every wrong answer by name,
refusal families, and the no-injection per-category coverage map that ranks
which injector to build next by real frequency.

Default corpus is the committed 47-case train_sample (always available);
--corpus path runs the full real split. This is the ADR-0191 follow-up: re-run
after every capability PR, not just train_sample — a flip is only real if it
does not widen the confabulation surface.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-30 15:45:07 -07:00
..
__init__.py
build_refusal_taxonomy_cases.py feat(ADR-0163.A): refusal taxonomy lane — shape categorization of GSM8K admissibility gaps (#297) 2026-05-26 11:27:11 -07:00
calibrate_ratification_threshold.py chore(ratifier): calibrate default ratification threshold 0.0 → 0.5 (#86) 2026-05-20 19:59:25 -07:00
dump-api-schemas.py feat(W-028): chat surface + trace drawer (#303) 2026-05-26 13:22:11 -07:00
dump-enums.py feat(workbench/W-031): replay theater (ADR-0160 §Phase 6, deterministic-evidence UX) (#328) 2026-05-26 20:56:31 -07:00
generate_claims.py perf(tests): extract math_teaching_corpus lane from pytest into CI lane SHAs (-9m suite time) (#261) 2026-05-25 05:42:12 -07:00
generate_discourse_paragraph.py feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer 2026-05-16 21:53:46 -07:00
generate_en_numerics_v1.py feat(packs): ADR-0128.1+0128.2 — en_numerics_v1 + loader 2026-05-23 07:02:09 -07:00
generate_english_fluency_ood.py fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
generate_grammatical_v2.py
generate_identity_curriculum.py
generate_identity_test_cases.py
generate_monotonic_cases.py
generate_phase5_domain_lanes.py
generate_phase5_language_lanes.py
gsm8k_microscope.py fix(adr-0191): candidate-graph completeness guard — real-corpus wrong 5→0 (#496) 2026-05-30 15:45:07 -07:00
publish_pack_measurements.py chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
ratify_anchor_lens_packs.py chore(packs): re-seal register packs (ISSUED_AT bump + missing seals) 2026-05-20 07:39:38 -07:00
ratify_ethics_pack.py feat(packs): expansion round 2 — ethics ×3, anchor-lens ×3, relations-v3, register ×2 (#48) 2026-05-20 07:18:35 -07:00
ratify_identity_packs.py feat(identity-packs): ADR-0027 Phase 5 — ratify all three v1 packs 2026-05-17 19:31:55 -07:00
ratify_register_packs.py fix(packs/register): remove bare '.' closings + ratify gate refuses them 2026-05-21 06:57:43 -07:00
ratify_rhetorical_style_packs.py feat(adr-0087): rhetorical-style pack substrate (loader + default_unstyled_v1) (#74) 2026-05-20 16:19:36 -07:00
ratify_safety_pack.py feat(safety): ADR-0029 — always-loaded, never-replaceable safety pack 2026-05-17 19:56:29 -07:00
review_trace.py
run_examples.py
run_pulse.py
scaffold_phase5_domain_lanes.py
score_phase5_holdouts.py
seal_gsm8k_test.py feat: ADR-0119.7 — seal GSM8K test as gsm8k_math holdout (Phase 5 substrate complete) 2026-05-22 20:08:35 -07:00
seal_holdouts.py feat(evals): ADR-0105 — sealed holdout encryption via age (#108) 2026-05-22 10:09:43 -07:00
verify_definitional_closure.py feat(packs): ADR-0085 content style pass v2 — 3sg + plural agreement (+ closure infra) (#100) 2026-05-20 23:12:28 -07:00
verify_lane_shas.py chore(ratify): accept four Phase C round-2 recognizers (round 2) (#309) 2026-05-26 16:03:01 -07:00