Shay
5cbd782e7b
Merge pull request #148 from AssetOverflow/feat/adr-0119.5-adversarial
...
feat: ADR-0119.5 — adversarial generation (closes ADR-0114a Obligation #8 )
2026-05-22 19:39:00 -07:00
Shay
3bda4313c9
feat: ADR-0119.5 — adversarial generation (closes ADR-0114a Obligation #8 )
...
Phase 5.5 of ADR-0119. Adversarial case generator + scoring CLI;
discharges the last remaining ADR-0114a obligation.
Numbers
adversarial suite: 38 cases × 12 families
per-family: every family produces wrong == 0
overall: correct 5, wrong 0, refused 33
Families
conditional_phrasing (4) "If/When/Suppose ..."
compound_questions (3) multiple ?
undefined_entity_question (3) question references unknown entity
unknown_verb (5) "polishes", "admires", etc.
empty_or_whitespace (3) empty input
no_question (3) statement-only
numbers_spelled_out (3) "five", "ten"
passive_voice (3) "X are bought by Y"
red_herring_numbers (3) digits in name positions, mid-quantity
question_only (2) no preceding statements
mid_sentence_punctuation (2) embedded ? or !
subtle_in_grammar (4) IN-grammar; runner must produce correct
(gate-sanity: not trivially "refuse all")
The subtle_in_grammar family is the load-bearing sanity check —
proves the gate isn't trivially satisfied by refusing everything.
ADR-0114a obligation status
10 of 10 discharged on main:
#1 fab_control lane (0119.1); GSM8K test pending (0119.7)
#2 ADR-0118a
#3 ADR-0117
#4 ADR-0116 + ADR-0119.3
#5 ADR-0125
#6 ADR-0119.6 harness; ε threshold to ADR-0120
#7 ADR-0119.4
#8 THIS ADR
#9 ADR-0116/0117/0118/0119.3
#10 ADR-0116
Phase 5 remaining: 5.7 (sealed GSM8K test, real corpus) and 5.8
(overall lane gate). After those, ADR-0120 (first expert promotion
contract) can compose all ten obligations.
Tests: 18 new + 25 prior Phase 5 = 43 green; 67/67 smoke.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 18:11:36 -07:00
Shay
78312b3151
chore: ADR-0119.4 + ADR-0119.6 cleanup — typed refusals + numeric/freshness asserts
...
Audit follow-ups from #145/#146 merge review. Five small fixes; no
behavior change on the green path, but failure modes are now explicit
rather than silent.
ADR-0119.6 depth_curve.py
- Add DepthCurveError typed exception
- Raise on case_id missing from lane_report (was: silent → "refused")
- Raise on depth >= 9 (was: silent new bucket key)
- Two new tests pin both refusals
- Removed stale sys.path hack at module top
ADR-0119.4 frontier-baseline tests
- Assert comparison_v1.json's core_measurement reports wrong == 0
(the load-bearing differentiator named in the disclaimer; a
tampered file with wrong > 0 was previously syntactically valid
and would have passed all old assertions)
- Assert frontier citations are dated 2023 or later (freshness
guard; older citations should be refreshed before ADR-0120
gates anything for `expert` promotion)
Tests
- tests/test_adr_0119_6_depth_curve.py: 7 → 9
- tests/test_adr_0119_4_frontier_baseline.py: 5 → 7
- 29/29 across runner + depth-curve + frontier suites; 67/67 smoke
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 17:47:42 -07:00
Shay
0ffbd5c40a
merge origin/main and resolve README conflicts
2026-05-22 17:38:21 -07:00
Shay
9288688640
feat: ADR-0119.3 — gsm8k_math lane runner (Phase 5.3) ( #145 )
...
Composes the Phases 1-4 pipeline (parser → solver → verifier →
realizer) into a per-case scoring decision: correct / wrong /
refused.
Outcome categorization (ADR-0114a Obligation #4 ):
parser ParseError → refused
solver SolveError → refused
verifier verdict failed → wrong
realizer error → wrong
answer/unit mismatch → wrong
all match → correct
`wrong == 0` is the load-bearing gate. The lane's overall_pass
holds only if wrong == 0 AND correct + refused == total.
Initial measurement on the Phase 5.2 corpus:
dev (50) : 50 correct, 0 wrong, 0 refused, overall_pass=True
public (150) : 150 correct, 0 wrong, 0 refused, overall_pass=True
Every correct case carries a trace_hash (64-char SHA-256) and
realized prose — full audit trail per case, consumable by ADR-0119.4
(frontier comparison), ADR-0119.6 (depth curve), and ADR-0120
(eventual expert-tier gate).
Tests: 13/13 green; 443 total green across runner + realizer +
solver + verifier; 67/67 smoke green.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 17:37:54 -07:00
Shay
a65040cb73
merge origin/main and resolve conflicts
2026-05-22 17:37:13 -07:00
Shay
51d3a73589
feat: ADR-0119.6 — depth-curve measurement harness (ADR-0114a Obligation #6 )
2026-05-22 17:33:58 -07:00
Shay
c21068ed3e
feat: ADR-0119.4 — frontier-baseline comparison (ADR-0114a Obligation #7 )
2026-05-22 17:33:28 -07:00
Shay
1a37f97c4f
feat: ADR-0119.2 — author 200 grade-school math problems for the GSM8K eval lane (dev + public)
2026-05-22 17:28:00 -07:00
