Audit follow-ups from #145/#146 merge review. Five small fixes; no
behavior change on the green path, but failure modes are now explicit
rather than silent.
ADR-0119.6 depth_curve.py
- Add DepthCurveError typed exception
- Raise on case_id missing from lane_report (was: silent → "refused")
- Raise on depth >= 9 (was: silent new bucket key)
- Two new tests pin both refusals
- Removed stale sys.path hack at module top
ADR-0119.4 frontier-baseline tests
- Assert comparison_v1.json's core_measurement reports wrong == 0
(the load-bearing differentiator named in the disclaimer; a
tampered file with wrong > 0 was previously syntactically valid
and would have passed all old assertions)
- Assert frontier citations are dated 2023 or later (freshness
guard; older citations should be refreshed before ADR-0120
gates anything for `expert` promotion)
Tests
- tests/test_adr_0119_6_depth_curve.py: 7 → 9
- tests/test_adr_0119_4_frontier_baseline.py: 5 → 7
- 29/29 across runner + depth-curve + frontier suites; 67/67 smoke
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Umbrella ADR for Phase 5 of ADR-0114. Decomposes the eval-lane work
into eight sub-phases (5.1..5.8) that ship under their own ADRs.
Sub-phase scope:
5.1 — sealed-holdout encryption (one lane, fabrication_control)
5.2 — CORE-original GSM8K-style corpus (dev + public, 200 cases)
5.3 — lane runner (parser → solver → verifier → realizer)
5.4 — frontier-baseline comparison (Obligation #7)
5.5 — adversarial generation; misparse rate zero (Obligation #8)
5.6 — depth-curve measurement harness (Obligation #6, measurement)
5.7 — sealed GSM8K test as the holdout (Obligation #1, lane-side)
5.8 — overall lane gate; new gsm8k_capability_shape
Roadmap only — no code lands with this ADR. Each sub-phase ships
independently and discharges a specific ADR-0114a obligation.
Critical invariant: dev + public splits are CORE-original; the real
GSM8K test set enters only via the encrypted holdout under 5.7.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 4 of the ADR-0114 GSM8K-math roadmap. Consumes a SolutionTrace
and emits one sentence per step plus setup + answer sentences. Pure
function; same trace → byte-equal RealizedTrace.
What landed
generate/math_realizer.py
- realize(initial_state, trace) -> RealizedTrace
- Frozen RealizedTrace dataclass with canonical_bytes() + as_prose()
- Per-kind sentence rules (add / subtract / transfer / multiply×2 /
multiply×3 / multiply-general / divide)
- Singular/plural surface rule matches parser canonicalization
- Typed RealizerError on unrecognized step kinds
tests/test_math_realizer.py — 60 cases pinning five invariants:
1. All 50 dev-set cases realize without error
2. Determinism: byte-equal RealizedTrace across two calls
3. Setup sentence count == initial_state count
4. Step sentence count == operation count
5. Answer sentence contains the resolved value + unit
ADR-0114a obligation discharge update
ADR-0118 hardens determinism (#9) across a third layer (realizer)
and makes #3 / #10 human-inspectable via the prose surface. No
obligation is directly newly discharged by ADR-0118; it's substrate
for ADR-0119 GSM8K eval lane.
Round-trippability of the prose through the parser is explicitly
out of scope for this phase. The trace is the verifiable artifact
(ADR-0117); the prose is human-readable documentation.
Tests: 60 new realizer cases; 546 total green across realizer +
parser + solver + verifier + OOD; 67/67 smoke green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 3 of the ADR-0114 expert-capability roadmap. Re-applies every
step of a SolutionTrace from the input graph's initial state and
asserts byte-equal reproduction of answer_value. Pure function; same
(graph, trace) → byte-equal VerifierVerdict.
Why this is distinct from the solver
ADR-0116's solver enforces correctness at construction. ADR-0117's
verifier is a SECOND, INDEPENDENT implementation that re-derives
every value the trace claims. The verifier does NOT call solve(). It
re-implements the operation semantics from ADR-0116 directly inside
_verify_step. If the solver had a bug or was tampered with after the
fact, the verifier catches it.
Six checks per verdict (named, ordered, audit-logged):
1. graph_canonical_hash_matches
2. pack_id_matches
3. pack_lemmas_resolve
4. step_pack_lemma_ids_match_bindings
5. step_replay_matches_before_after
6. answer_value_reproduces
Seven named tamper classes all caught:
- mutated before_value / after_value / operand of any step
- mutated pack_lemma_id of any step
- mutated graph_canonical_hash
- mutated answer_value
- mutated pack_id
- mutated target_before / target_after of transfer step
ADR-0114a obligation update
#3 Replay-equal trace — now discharged at VERIFIER FIDELITY
(was solver-only under ADR-0116). A third party with only
(graph, trace, pack) can reproduce the answer byte-equal.
Five of ten obligations now load-bearing: #3, #4, #9, #10 plus
in-flight #2 (Codex's ADR-0118a OOD generator).
Tests: 62/62 verifier suite green; 67/67 smoke green; existing
solver + parser + schema suites unaffected.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 2 of the ADR-0114 expert-capability roadmap. Consumes the
MathProblemGraph from Phase 1 and emits a SolutionTrace — ordered
operation applications ending at a numeric answer, byte-deterministic
across runs, with each step's operation bound to a pack-resolved
lemma identifier.
What landed
generate/math_solver.py
- solve(graph) -> SolutionTrace; pure function, no I/O, no globals
- SolutionStep dataclass with before/after values per step (for
verifier replay; ADR-0117 hardens)
- SolutionTrace with canonical_bytes() byte-deterministic JSON
- SolveError typed refusal: missing pack, division by zero,
unknown-references-nothing
language_packs/data/en_arithmetic_v1/
- 5 operator lemmas: add / subtract / multiply / divide / transfer
- role=operational_base (vocabulary-only; no domain claim)
- SHA-256-anchored lexicon + glosses; manifest carries
provenance=adr-0116:operator_seed:2026-05-22
tests/test_math_solver.py — 109 cases pinning five invariants:
1. Phase 2 exit criterion: ≥ 0.80 on parser-correct dev set
(current: 50/50 = 1.00)
2. Determinism: two solves produce byte-equal trace
3. Trace replay reproduces answer_value (verifier rehearsal)
4. Typed refusal on under-determined inputs
5. Every step.pack_lemma_id resolves to a real lexicon entry
in en_arithmetic_v1
ADR-0114a obligation discharge
Four of ten anti-overfitting obligations now have load-bearing
implementations in code:
#3 replay-equal trace — discharged (solver-layer)
#4 typed refusal — discharged (solver-layer)
#9 determinism — discharged (solver-layer)
#10 operation provenance via pack — DISCHARGED IN FULL
Removing the en_arithmetic_v1 pack now breaks every solve loudly.
The "operations bind to concepts, not hardcoded strings" claim is
architecturally true, not rhetorical.
Tests: 109/109 green on solver suite; 67/67 smoke suite green;
parser + schema suites still green from prior phases.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Documentation-only amendment to ADR-0114. Locks in the 10-point
falsifiable proof framework that any future `expert` ledger-tier
promotion (ADR-0120+) MUST require.
The obligations:
1. Sealed-holdout discipline
2. OOD surface variation ≥ 0.95 of public
3. Every correct answer ships with replay-equal trace
4. Refusal is first-class; misparse rate zero; zero `wrong` answers
5. Reasoning-isolation perturbation suite (invariance + predictable change)
6. Compositional-depth curve flat within documented ε
7. Frontier-baseline comparison on identical items, published
8. Adversarial generation; misparse rate zero
9. Determinism across release boundaries
10. Operation provenance via the pack (not hardcoded strings)
Each obligation is load-bearing and falsifiable: a domain that
cannot satisfy any one stays at `audit-passed`. ADR-0114a binds
ADR-0116..ADR-0120 to carry the obligations into implementation;
ADR-0120 finally invokes all ten as hard gates.
The audit-passed tier (ADR-0106/0109/0113) is unaffected. The two
tiers measure orthogonal properties: audit-passed verifies CORE
claim-shape compliance (transformer-unreachable invariants); expert
verifies capability with anti-overfitting proof.
No code change. Pure forward contract for the next phase of work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes Phase 1.3 of the ADR-0114 expert-capability roadmap. Turns a
grade-school word problem into a typed MathProblemGraph deterministically
(no LLM, no sampling). Same input string always produces the same
graph; unsupported constructions raise ParseError rather than guessing.
What the parser handles
Initial possession: "<E> has <N> <unit>."
Add verbs: buys, gets, finds, receives, earns, adds
(+ "<N> more" / unit elision via state.last_unit)
Subtract verbs: eats, loses, sells, donates, uses, spends, drops, removes
Transfer verbs: gives, sends, hands, passes, mails (with target)
Multiply (scalar): "X doubles <obj>" / "X triples <obj>"
Divide (split): "X splits {them|his Y|N Y} evenly into M groups [and keeps one]"
Compound sentences: "X buys 5, then donates 3."
Sentence opener: "Then X eats 1." (inherits subject + unit)
Pronoun anaphora: he/she/it → last-introduced singular subject
Object pronoun: them/these/those → state.last_unit
Trailing PP: "finds 7 buttons on the floor" — discarded
Singular→plural: "Iris has 1 coin" → canonical unit "coins"
Questions:
"How many <unit> does <E> have [left|now|in total|altogether]?"
"How many <unit> do they have [in total|altogether|left|now]?"
What it explicitly rejects
- Conditional / time-modal ("If X had ...")
- Compound questions (two unknowns)
- Multiple "?" sentences
- Questions referencing entities never introduced
- Empty / whitespace-only input
Verification
- tests/test_math_parser.py: 20 cases (5 byte-equal parametrized
+ 5 determinism parametrized + 1 exit-criterion gate + 6 typed-
refusal + 2 purity + 1 type check)
- tests/test_math_problem_graph.py: 26 schema cases still green
- On the 5 seed cases: 5/5 = 100% byte-equal
- On Codex's PR #128 50-case dev set (locally tested):
49/50 = 98% byte-equal. Single failure (gpd-021) is a case-
quality issue, not a parser limit; feedback filed on #128 to
rewrite (mixed units + metaphor not in pattern registry).
- Phase 1.3 exit criterion (≥ 0.90): met.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First Phase of ADR-0114's expert-capability roadmap. Decomposed into four
sub-phases so each lands as its own auditable step:
1.1 schema + 5 seed cases + invariants ← this commit
1.2 45 more dev-set cases ← delegated (Codex)
1.3 the parser itself ← exit: ≥0.90 on dev set
1.4 runtime binding ← if non-trivial
What landed
- generate/math_problem_graph.py — typed dataclasses (Quantity,
InitialPossession, Operation, Unknown, MathProblemGraph) + frozen
validation + canonical_bytes() byte-deterministic serialization +
graph_from_dict roundtrip.
- evals/gsm8k_parser_dev/cases.jsonl — 5 seed cases (gpd-001..005)
covering single-add, single-subtract, multi-step, two-entity
transfer, and multi-entity sum constructions. Every case carries a
ground_truth_graph and the documented patterns it exercises.
- evals/gsm8k_parser_dev/README.md — authoring contract: schema,
pattern registry, canonicalization rules, Phase 1.1 scope boundary,
hand-solving rubric, distribution target for the remaining 45
cases. This is the spec Phase 1.2 authors work against.
- tests/test_math_problem_graph.py — 26 cases pinning four invariants:
round-trip byte equality, canonical_bytes() determinism, schema
rejection of malformed graphs, and ground_truth_graph ↔
expected_answer agreement (a hand-solver inside the test module
falsifies mis-authored cases).
Why this is sticky
The Phase 1.1 schema is load-bearing for Phase 1.2 (the 45 authored
cases will be written against it) AND Phase 1.3 (the parser will be
graded byte-equal against ground-truth graphs in this schema). Changing
the schema after Phase 1.2 lands requires an amendment ADR + rewriting
authored cases. The schema choices here are intentionally conservative.
Tests: 26/26 new; 67/67 smoke green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The word "expert" in the previous status name implied raw-capability parity
with frontier LLMs on the same benchmark — which the gate does NOT verify.
What the gate actually verifies is CORE *claim-shape compliance*:
* signed digest (replay-reproducible from on-disk lane results)
* replay determinism (same inputs → byte-equal trace_hash)
* typed refusal (fabrication refused, not paraphrased)
* exact recall (no ANN, no cosine, no attention bottleneck)
* grounding-source provenance
These are claim shapes a transformer LLM cannot structurally produce
regardless of raw accuracy. A frontier LLM might score higher on the
same benchmark but cannot pass this contract.
Rename scope (semantics only, per ADR-0113):
status string "expert-demo" → "audit-passed"
predicate key predicates.expert_demo → predicates.audit_passed
reason key expert_demo_reason → audit_passed_reason
YAML key expert_demo_claims → audit_passed_claims
CLI command core demo expert → core demo audit-passed
output dir evals/expert_demos/ → evals/audit_passed/
artifact filenames expert_demo.{json,html} → audit_passed.{json,html}
HTML title CORE Expert-Demo: X → CORE Audit-Passed: X
Internal Python identifiers (module/file/function/class names like
`expert_demo.py`, `evaluate_expert_demo`, `ExpertDemoClaim`,
`expert_demo_claim_for`) are deliberately kept to minimize churn. ADR
file titles (ADR-0106..0112) preserved as historical record.
`expert` namespace reserved for ADR-0114+: an actual capability tier
above `audit-passed` backed by a public benchmark with a stated
threshold. ADR-0114 proposes the first such target — GSM8K-math —
laying out a falsifiable 7-phase arc (parser → solver → verifier →
stepped-realizer → eval lane → first `expert` ledger tier promotion).
Tests: 184 directly-affected tests green (140 capability/expert-demo
suite + 34 demo/audit-tour + 10 correction-cue). Smoke suite 67/67.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes the asymmetry between the `expert-demo` ledger status (audit
artifact only) and the actual `core demo` surface (runnable
walkthroughs producing HTML + JSON). Until this commit the word
"demo" in `expert-demo` was aspirational; now it corresponds to
something a reader can open.
What it does
- Reads the signed expert_demo_claims entry from docs/reviewers.yaml
- Loads latest on-disk result files for each attached lane × split
- Re-derives the evidence-bundle digest and asserts byte-for-byte
match against the signed claim_digest — this is the load-bearing
audit step, now exercised at two independent enforcement points
(ledger gate + showcase)
- Runs each lane's metrics through the ADR-0109 lane-shape registry
and surfaces the verdict
- Picks the first three cases from each split verbatim (deterministic
by file order) and renders them as HTML for inspection
- Emits expert_demo.json (canonical bytes, deterministic) + expert_demo.html
Surface
core demo expert --domain mathematics_logic
core demo expert --domain physics
# → evals/expert_demos/<domain>/latest/expert_demo.{json,html}
Read-only by construction: cannot mutate docs/reviewers.yaml or any
lane result file. Tested. Unpromoted domains raise ValueError —
no silent fallback, no "preview" mode that fakes a showcase.
Generated artifacts are gitignored — the inputs they derive from are
already committed, so duplicating the renders would just churn the
tree.
Tests: 16 new cases pinning all five ADR-0112 invariants. Smoke suite
still 67/67 green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ADR-0111 landed via PR #123 (merge commit 4b5e6b7). The structural docs
(README, ADR index, reviewers.yaml, CLAIMS.md) were updated in that PR.
The narrative docs that reference the worked-promotion arc by ordinal
("first promotion / one demonstrated domain") needed a follow-up sweep:
- docs/PROGRESS.md — ADR-0100 footnote, ADR-0111 row, ledger table,
Open-within-Phase-6 changed from "second" to "third" promotion
- docs/runtime_contracts.md — narrative line on contract demonstration
- docs/Whitepaper.md §XIII — chain range, demonstration paragraph,
"two domains demonstrated" line, full-chain pointer
- docs/capability_roadmap.md — Phase 6 worked-promotion narrative step 5
and exit-criteria checklist (5th item now ☑, 6th open as "third")
docs/Yellowpaper.md is unchanged: its expert-demo section is the formal
contract spec, domain-agnostic. docs/eval_methodology.md unchanged for
the same reason. docs/sessions/SESSION-2026-05-22-contract-layer-arc.md
is left as the historical snapshot of the original session.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
