Merge pull request 'feat(adr-0250): instrument routes Tier-2 (50/50) + ADR-0250 — Tier-2 arc closure' (#74) from feat/adr-0250-instrument-adr into main
Reviewed-on: #74 --- The Tier-2 arc, complete Four PRs, each smoke-gated, one per phase: ┌────────────┬───────────────────────┬──────────────────────────────────────────────────────────┐ │ PR │ Phase │ Result │ ├────────────┼───────────────────────┼──────────────────────────────────────────────────────────┤ │ #71 ✓ │ Design spike │ multi-register model, atomicity, chain-of-custody design │ ├────────────┼───────────────────────┼──────────────────────────────────────────────────────────┤ │ #72 ✓ │ T2a executor │ 18/18 single-entity, wrong=0 │ ├────────────┼───────────────────────┼──────────────────────────────────────────────────────────┤ │ #73 ✓ │ 2b summation │ full holdout 50/50, wrong=0 │ ├────────────┼───────────────────────┼──────────────────────────────────────────────────────────┤ │ #74 (open) │ Instrument + ADR-0250 │ coverage recorded 26→50; ADR proposed │ └────────────┴───────────────────────┴──────────────────────────────────────────────────────────┘ The headline, from the top Across ADR-0249 and ADR-0250, the reader→Hamiltonian compiler went from ingesting almost nothing to solving the entire real GSM8K dev holdout — 50/50, wrong = 0 — by chained certified relaxation on the Cl(4,1) substrate: - 26 Tier-1 single-accumulator arithmetic - 18 T2a multi-register (multi-entity, coupled-translator transfers, conservation-gated) - 6 certified summation over registers ("altogether") Every step is certified, the transfers are conservation-gated (relative, hard-reject), the whole thing is transactionally atomic and tamper-evidently recorded, and it's all off-serving with no gate activated. Against a symbolic evaluator it's honest PARITY — the field matches arithmetic, it doesn't beat it — and the deliverable is exactly what the plan asked for: a decidable, real-data measurement with recorded coverage, not an inflated claim.
This commit is contained in:
commit
09257ed32f
4 changed files with 226 additions and 36 deletions
95
docs/adr/ADR-0250-tier2-multi-entity-arithmetic.md
Normal file
95
docs/adr/ADR-0250-tier2-multi-entity-arithmetic.md
Normal file
|
|
@ -0,0 +1,95 @@
|
||||||
|
# ADR-0250: Tier-2 Multi-Entity Arithmetic — the Multi-Register Extension
|
||||||
|
|
||||||
|
**Status**: **Proposed**
|
||||||
|
**Date**: 2026-07-18
|
||||||
|
**Authors**: Joshua Shay + multi-model R&D (implemented Fable 5)
|
||||||
|
**Depends on**: ADR-0249 (reader→Hamiltonian compiler — quantity kernel, relation compiler, turn programs), ADR-0243/0244/0245 (relaxation, content-addressing, f64)
|
||||||
|
**Design record**: `docs/research/tier2-multi-entity-arithmetic-spike-2026-07-18.md`
|
||||||
|
**Acceptance evidence**: `docs/handoff/ADR-0250-Acceptance-Evidence.md`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. What this ADR is
|
||||||
|
|
||||||
|
ADR-0249 closed Tier-1 (single-accumulator affine arithmetic) and recorded the 24 multi-entity
|
||||||
|
GSM8K dev-holdout cases as the Tier-2 frontier. This ADR crosses that frontier — and with it the
|
||||||
|
reader→Hamiltonian compiler solves the **entire** real GSM8K dev holdout (50/50) at wrong=0. It
|
||||||
|
extends the substrate mechanism only; no serving change, no gate activated, no manifold mutation.
|
||||||
|
|
||||||
|
## 2. The multi-register model
|
||||||
|
|
||||||
|
Multi-entity state is a **product of independent conformal lines**: one register (null point /
|
||||||
|
its own relaxation) per entity. This respects the Cl(4,1) boundary exactly — no dimensional
|
||||||
|
inflation (no `Cl(8,2)` to pack two points into one multivector), so the f64 rounding floor
|
||||||
|
(~6e-11) stays the only residual and every op is a native, exact versor action on a pure state.
|
||||||
|
Per-register ops (add/subtract/multiply/divide by constants) reuse the Tier-1 transport unchanged.
|
||||||
|
|
||||||
|
## 3. Transfers + the conservation pin
|
||||||
|
|
||||||
|
A constant-operand transfer ("actor gives `k` to target", ADR-0116 decomposition) is a **coupled
|
||||||
|
pair of translators** — `T₋ₖ` on the actor register, `T₊ₖ` on the target — acting on disjoint
|
||||||
|
registers, so exactness across the independent null points is inherited from ADR-0249 P1.
|
||||||
|
|
||||||
|
**Conservation pin (hard-reject, relative).** `Σ decode(after) ≡ Σ decode(before)` within
|
||||||
|
`|Δ| ≤ rtol·max(1,|before|)`. The tolerance is *relative* because decoded-quantity error scales
|
||||||
|
with magnitude (found on a case chaining to 222); an absolute tolerance is brittle, while a
|
||||||
|
genuine non-conserving transfer is off by a whole `k` and is still caught. A conservation failure
|
||||||
|
is an algebraic failure, not a scope miss — fail closed.
|
||||||
|
|
||||||
|
## 4. Transactional atomicity
|
||||||
|
|
||||||
|
Registers are an **immutable snapshot**. A transfer runs **prepare → validate → commit**: both
|
||||||
|
candidate states are relaxed into locals, validated (both converged AND conservation), then — only
|
||||||
|
on full success — a *new* register mapping is produced. A failure anywhere means the new mapping is
|
||||||
|
never built, so the original stands untouched: no partial-mutation window, no rollback. Records
|
||||||
|
append only on commit; any abort raises a typed `MultiRegisterError` and aborts the whole program
|
||||||
|
(fail-closed — a partial answer is never emitted, preserving wrong=0).
|
||||||
|
|
||||||
|
## 5. Certified summation + chain of custody
|
||||||
|
|
||||||
|
A total-like unknown (`unknown.entity is None`, "how many altogether") is the **explicit typed
|
||||||
|
signal** for a certified summation over all registers — never inferred from an operation pattern.
|
||||||
|
Per the summation ruling it stays in the substrate: each addition is a certified relaxation turn
|
||||||
|
(`T_v` on the running accumulator), never a Python `sum()`.
|
||||||
|
|
||||||
|
The operand of each summation turn is the decode of a certified register state, bound in the record
|
||||||
|
by `operand_source_digest` (the `psi_digest` byte convention). Deterministic re-execution
|
||||||
|
reproduces the state and its decode, so the Python layer cannot tamper with the intermediate value
|
||||||
|
— the chain of custody holds with no trusted-Python assumption. The field is added conditionally to
|
||||||
|
the record payload, so non-summation record digests are unchanged.
|
||||||
|
|
||||||
|
## 6. Honest measurement (done-when)
|
||||||
|
|
||||||
|
On the sealed real GSM8K dev holdout (50 cases): **50/50 solved, wrong=0** — Tier-1 (26) + T2a
|
||||||
|
single-entity (18) + summation (6). The instrument's `arithmetic-chain` domain routes each case to
|
||||||
|
the right tier and compares the corridor to a symbolic fold of the *same* compiled program: PARITY
|
||||||
|
(delta 0 — the field matches arithmetic, it does not beat it). The deliverable is a decidable
|
||||||
|
measurement + real-holdout coverage, per the honest-NULL protocol.
|
||||||
|
|
||||||
|
## 7. Scope + recorded frontier
|
||||||
|
|
||||||
|
Tier-2a: single-accumulator-per-entity affine + constant-operand transfers, positive scale,
|
||||||
|
matching units. 2b: the certified summation turn (real, exercised by the 6 total-like cases) and
|
||||||
|
the derived-operand-transfer path ("half of X" — designed + guarded, **0 on this holdout**). Still
|
||||||
|
refused and recorded: non-affine kinds (rate/comparison/fraction/partition — 0 on this holdout),
|
||||||
|
derived-operand transfers, and >5-atom deduction — never silently dropped.
|
||||||
|
|
||||||
|
## 8. Governance
|
||||||
|
|
||||||
|
Off-serving (A-04): `evals/multi_register_program.py` + the instrument domain; never imported by
|
||||||
|
`chat/runtime.py`. No gate activated; I-03 untouched; ratified contracts reused
|
||||||
|
(`compile_quadratic_well`, `quantity_kernel`, the Ring-2 chain pattern, `RelaxationCertificate`).
|
||||||
|
Local-first CI: smoke green in-worktree at every phase.
|
||||||
|
|
||||||
|
## 9. What this ADR does NOT claim
|
||||||
|
|
||||||
|
- Not a serving change; the symbolic `generate/` path remains the serve path.
|
||||||
|
- Not a lift over symbolic solvers on arithmetic (PARITY is the honest result).
|
||||||
|
- Not derived-operand-transfer coverage (0 on this holdout; designed + guarded only).
|
||||||
|
- Not an activation of any admission gate (ADR-0246 §3.7 stays uncalibrated).
|
||||||
|
|
||||||
|
## 10. Ruling record (Shay)
|
||||||
|
|
||||||
|
_Awaiting ratification._ The design rulings (multi-register product-of-lines; relative
|
||||||
|
conservation hard-reject; prepare→validate→commit atomicity; certified summation turn; ship 2b
|
||||||
|
designed+guarded) are recorded RESOLVED in the spike; acceptance is Shay's alone (no self-Accept).
|
||||||
59
docs/handoff/ADR-0250-Acceptance-Evidence.md
Normal file
59
docs/handoff/ADR-0250-Acceptance-Evidence.md
Normal file
|
|
@ -0,0 +1,59 @@
|
||||||
|
# ADR-0250 — Acceptance Evidence
|
||||||
|
|
||||||
|
**Status**: Evidence pack for Shay's ruling (no self-Accept)
|
||||||
|
**Date**: 2026-07-18
|
||||||
|
**ADR**: `docs/adr/ADR-0250-tier2-multi-entity-arithmetic.md`
|
||||||
|
**Design record**: `docs/research/tier2-multi-entity-arithmetic-spike-2026-07-18.md`
|
||||||
|
|
||||||
|
Tier-2 extends the reader→Hamiltonian compiler (ADR-0249) to multi-entity arithmetic. With it,
|
||||||
|
the corridor solves the **entire** real GSM8K dev holdout. Built in smoke-gated phases, each
|
||||||
|
merged under its own PR off `forgejo/main`.
|
||||||
|
|
||||||
|
## Phases merged
|
||||||
|
|
||||||
|
| Phase | PR | Merge commit | What landed |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Design spike | #71 | `97b2f8b7` | multi-register model, atomicity, chain-of-custody design; probe of the 24 refused cases |
|
||||||
|
| T2a executor | #72 | `efd5335e` | `evals/multi_register_program.py` — multi-register + transfers + conservation; 18/18 single-entity |
|
||||||
|
| 2b summation | #73 | `3c3dcdbe` | certified summation turn + `operand_source_digest`; full holdout 50/50 |
|
||||||
|
| Instrument + ADR | this PR | — | `arithmetic-chain` domain routes Tier-2; ADR-0250 + this evidence |
|
||||||
|
|
||||||
|
Smoke (`uv run core test --suite smoke -q`) green at every phase (176 passed).
|
||||||
|
|
||||||
|
## Load-bearing result
|
||||||
|
|
||||||
|
**Real GSM8K dev holdout (sealed, 50 cases):**
|
||||||
|
|
||||||
|
| Tier | Cases | Corridor |
|
||||||
|
|---|---|---|
|
||||||
|
| Tier-1 single-accumulator (ADR-0249) | 26 | correct, wrong=0 |
|
||||||
|
| T2a single-entity multi-register | 18 | correct, wrong=0 |
|
||||||
|
| 2b certified summation ("altogether") | 6 | correct, wrong=0 |
|
||||||
|
| **Total** | **50 / 50** | **wrong = 0** |
|
||||||
|
|
||||||
|
Corridor vs symbolic fold of the same compiled program: **PARITY** (delta 0) — the field matches
|
||||||
|
arithmetic, it does not beat it. The instrument's `arithmetic-chain` domain now records 50/50
|
||||||
|
coverage, 0 refused.
|
||||||
|
|
||||||
|
## Verified in-tree
|
||||||
|
|
||||||
|
- **Transfer conservation + per-register exactness**: Ruth 36→31, Sara 19→24, sum conserved;
|
||||||
|
200 random transfers exact + conserving to ~6e-11.
|
||||||
|
- **Relative conservation pin**: `gma-050` (chains to 222) tripped an absolute 1e-6 tolerance;
|
||||||
|
the relative check `|Δ| ≤ rtol·max(1,|before|)` fixes it and still catches real violations.
|
||||||
|
- **Chain of custody**: summation records carry `operand_source_digest`; tamper on any non-terminal
|
||||||
|
record breaks the successor link (`verify_multi_register_chain`).
|
||||||
|
|
||||||
|
## Design decisions applied (honest register)
|
||||||
|
|
||||||
|
- **Atomicity via immutability**: prepare→validate→commit; no partial-mutation window, no rollback.
|
||||||
|
- **Summation stays in the substrate** (ruling #1): each addition is a certified turn, never a
|
||||||
|
Python `sum()`; the total is grounded in `RelaxationCertificate`s.
|
||||||
|
- **The summation signal is the typed `unknown.entity is None`**, not an inferred op pattern.
|
||||||
|
|
||||||
|
## What remains open (not claimed here)
|
||||||
|
|
||||||
|
- Derived-operand transfers ("half of X") — designed + guarded, 0 on this holdout.
|
||||||
|
- Non-affine kinds (rate/comparison/fraction/partition) — 0 on this holdout, refused.
|
||||||
|
- >5-atom deduction via ROBDD-partitioned turn programs — future arc.
|
||||||
|
- ADR-0246 §3.7 admission-policy calibration — unchanged; no gate activated.
|
||||||
|
|
@ -66,6 +66,11 @@ from evals.turn_program import (
|
||||||
compile_turn_program,
|
compile_turn_program,
|
||||||
execute_turn_program,
|
execute_turn_program,
|
||||||
)
|
)
|
||||||
|
from evals.multi_register_program import (
|
||||||
|
MultiRegisterError,
|
||||||
|
compile_multi_register_program,
|
||||||
|
execute_multi_register_program,
|
||||||
|
)
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"DomainOutcome",
|
"DomainOutcome",
|
||||||
|
|
@ -486,14 +491,51 @@ def _symbolic_fold(seed: float, steps) -> float:
|
||||||
return answer
|
return answer
|
||||||
|
|
||||||
|
|
||||||
def run_arithmetic_chain_domain() -> DomainOutcome:
|
def _symbolic_multi_register(program) -> float:
|
||||||
"""Corridor turn-program executor vs symbolic fold on real GSM8K problems.
|
"""Baseline for Tier-2: solve the SAME multi-register program arithmetically.
|
||||||
|
|
||||||
Both paths consume the graph's `ground_truth_graph` compiled to a turn
|
Mirror of the corridor executor with Python arithmetic instead of
|
||||||
program; non-Tier-1 graphs (multi-entity, transfer, rate, …) are refused by
|
relaxation — per-register affine folds, transfers move `operand` between
|
||||||
BOTH and recorded as the composition frontier — never silently dropped. The
|
registers, a `None` answer entity sums all registers.
|
||||||
expected honest verdict is PARITY with wrong=0: the field does not beat
|
"""
|
||||||
arithmetic at arithmetic; the result is real-holdout coverage, not a lift.
|
registers = {entity: seed for entity, seed in program.seeds}
|
||||||
|
for turn in program.turns:
|
||||||
|
if turn.kind == "transfer":
|
||||||
|
registers[turn.actor] -= turn.operand
|
||||||
|
registers[turn.target] += turn.operand
|
||||||
|
else:
|
||||||
|
registers[turn.actor] = turn.scale * registers[turn.actor] + turn.offset
|
||||||
|
if program.answer_entity is None:
|
||||||
|
return float(sum(registers.values()))
|
||||||
|
return registers[program.answer_entity]
|
||||||
|
|
||||||
|
|
||||||
|
def _corridor_and_baseline(graph):
|
||||||
|
"""Route a graph to Tier-1 (single-accumulator) or Tier-2 (multi-register),
|
||||||
|
returning (corridor_answer, baseline_answer) or None if genuinely un-ingestible.
|
||||||
|
Both paths consume the identical compiled program; the corridor relaxes, the
|
||||||
|
baseline folds arithmetically."""
|
||||||
|
try:
|
||||||
|
program = compile_turn_program(graph)
|
||||||
|
return execute_turn_program(program).answer, _symbolic_fold(program.seed, program.steps)
|
||||||
|
except TurnProgramError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
program = compile_multi_register_program(graph)
|
||||||
|
except MultiRegisterError:
|
||||||
|
return None
|
||||||
|
return execute_multi_register_program(program).answer, _symbolic_multi_register(program)
|
||||||
|
|
||||||
|
|
||||||
|
def run_arithmetic_chain_domain() -> DomainOutcome:
|
||||||
|
"""Corridor executor vs symbolic fold on real GSM8K problems (Tier-1 + Tier-2).
|
||||||
|
|
||||||
|
Each `ground_truth_graph` is routed to the single-accumulator turn program or
|
||||||
|
the multi-register program; both corridor and baseline consume that same
|
||||||
|
compiled program. Anything neither tier ingests (derived-operand transfers,
|
||||||
|
>5-atom shapes) is refused by BOTH and recorded, never silently dropped. The
|
||||||
|
expected honest verdict is PARITY with wrong=0: the field matches arithmetic,
|
||||||
|
it does not beat it; the result is real-holdout coverage.
|
||||||
"""
|
"""
|
||||||
corridor_correct = corridor_wrong = corridor_refused = 0
|
corridor_correct = corridor_wrong = corridor_refused = 0
|
||||||
baseline_correct = baseline_wrong = baseline_refused = 0
|
baseline_correct = baseline_wrong = baseline_refused = 0
|
||||||
|
|
@ -508,23 +550,15 @@ def run_arithmetic_chain_domain() -> DomainOutcome:
|
||||||
graph = graph_from_dict(graph_dict) if graph_dict else None
|
graph = graph_from_dict(graph_dict) if graph_dict else None
|
||||||
except (MathGraphError, KeyError, TypeError, ValueError):
|
except (MathGraphError, KeyError, TypeError, ValueError):
|
||||||
graph = None
|
graph = None
|
||||||
try:
|
answers = _corridor_and_baseline(graph) if graph is not None else None
|
||||||
program = compile_turn_program(graph) if graph is not None else None
|
if answers is None:
|
||||||
except TurnProgramError as refusal:
|
|
||||||
corridor_refused += 1
|
corridor_refused += 1
|
||||||
baseline_refused += 1
|
baseline_refused += 1
|
||||||
rows.append({"case_id": case_id, "ingested": False, "refusal": refusal.reason})
|
rows.append({"case_id": case_id, "ingested": False})
|
||||||
continue
|
|
||||||
if program is None:
|
|
||||||
corridor_refused += 1
|
|
||||||
baseline_refused += 1
|
|
||||||
rows.append({"case_id": case_id, "ingested": False, "refusal": "no_graph"})
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
|
corridor_answer, baseline_answer = answers
|
||||||
gold = None if gold_raw is None else float(gold_raw)
|
gold = None if gold_raw is None else float(gold_raw)
|
||||||
corridor_answer = execute_turn_program(program).answer
|
|
||||||
baseline_answer = _symbolic_fold(program.seed, program.steps)
|
|
||||||
|
|
||||||
corridor_ok = gold is not None and abs(corridor_answer - gold) < 1e-4
|
corridor_ok = gold is not None and abs(corridor_answer - gold) < 1e-4
|
||||||
baseline_ok = gold is not None and abs(baseline_answer - gold) < 1e-4
|
baseline_ok = gold is not None and abs(baseline_answer - gold) < 1e-4
|
||||||
corridor_correct += int(corridor_ok)
|
corridor_correct += int(corridor_ok)
|
||||||
|
|
@ -553,10 +587,10 @@ def run_arithmetic_chain_domain() -> DomainOutcome:
|
||||||
baseline_wrong=baseline_wrong,
|
baseline_wrong=baseline_wrong,
|
||||||
baseline_refused=baseline_refused,
|
baseline_refused=baseline_refused,
|
||||||
notes=(
|
notes=(
|
||||||
f"Real GSM8K dev holdout: {ingested}/{len(cases)} problems are Tier-1 "
|
f"Real GSM8K dev holdout: {ingested}/{len(cases)} problems ingestible "
|
||||||
f"affine single-accumulator ingestible and solved wrong=0; the rest are "
|
f"(Tier-1 single-accumulator + Tier-2 multi-entity/transfer/summation) and "
|
||||||
f"refused (multi-entity/transfer/rate/…) — the recorded Tier-2 frontier.",
|
f"solved wrong=0. Derived-operand transfers / >5-atom shapes remain the frontier.",
|
||||||
"Baseline is a symbolic fold of the SAME compiled turn program; PARITY "
|
"Baseline is a symbolic fold of the SAME compiled program; PARITY "
|
||||||
"with wrong=0 is the honest outcome — the field matches arithmetic, it "
|
"with wrong=0 is the honest outcome — the field matches arithmetic, it "
|
||||||
"does not beat it. The result is real-holdout coverage, not a lift.",
|
"does not beat it. The result is real-holdout coverage, not a lift.",
|
||||||
),
|
),
|
||||||
|
|
@ -579,11 +613,12 @@ def run_generalized_lift_instrument() -> LiftInstrumentReport:
|
||||||
wrong_zero_guard_held=(propositional.corridor_wrong == 0 and arithmetic.corridor_wrong == 0),
|
wrong_zero_guard_held=(propositional.corridor_wrong == 0 and arithmetic.corridor_wrong == 0),
|
||||||
honest_null=all(o.delta_correct <= 0 for o in outcomes),
|
honest_null=all(o.delta_correct <= 0 for o in outcomes),
|
||||||
scope_limitations=(
|
scope_limitations=(
|
||||||
"The reader→Hamiltonian compiler now exists for the affine "
|
"The reader→Hamiltonian compiler now solves the FULL real GSM8K dev "
|
||||||
"single-accumulator arithmetic subset (ADR-0249): on the real GSM8K "
|
"holdout wrong=0 (Tier-1 single-accumulator, ADR-0249; Tier-2 "
|
||||||
"dev holdout it solves the Tier-1 subset wrong=0 (see the "
|
"multi-entity + constant-operand transfers + certified summation, "
|
||||||
"arithmetic-chain domain). Non-affine / multi-entity GSM8K problems "
|
"ADR-0250) — see the arithmetic-chain domain. Remaining general "
|
||||||
"(transfer, rate, comparison, fraction, partition) remain the "
|
"frontier: derived-operand transfers ('half of X') and non-affine "
|
||||||
"recorded Tier-2 frontier — refused, never silently dropped.",
|
"kinds (rate/comparison/fraction/partition), both 0 on this holdout; "
|
||||||
|
"and >5-atom deduction. Refused, never silently dropped.",
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -27,7 +27,8 @@ def test_domain_solves_real_gsm8k_wrong_zero(domain) -> None:
|
||||||
assert domain.domain_id == "arithmetic-chain"
|
assert domain.domain_id == "arithmetic-chain"
|
||||||
assert domain.corridor_wrong == 0 # wrong=0 on real GSM8K, not just templates
|
assert domain.corridor_wrong == 0 # wrong=0 on real GSM8K, not just templates
|
||||||
assert domain.corridor_correct > 0 # the corridor ingests and solves real problems
|
assert domain.corridor_correct > 0 # the corridor ingests and solves real problems
|
||||||
assert domain.corridor_refused > 0 # and honestly refuses what it can't ingest
|
# Tier-2 (ADR-0250) closed the multi-entity frontier: the full holdout ingests now.
|
||||||
|
assert domain.corridor_refused == 0
|
||||||
|
|
||||||
|
|
||||||
def test_every_ingested_case_is_correct(domain) -> None:
|
def test_every_ingested_case_is_correct(domain) -> None:
|
||||||
|
|
@ -50,10 +51,10 @@ def test_coverage_is_recorded_not_dropped(domain) -> None:
|
||||||
ingested = domain.corridor_correct + domain.corridor_wrong
|
ingested = domain.corridor_correct + domain.corridor_wrong
|
||||||
assert ingested + domain.corridor_refused == domain.n_cases
|
assert ingested + domain.corridor_refused == domain.n_cases
|
||||||
assert domain.n_cases > 0
|
assert domain.n_cases > 0
|
||||||
# A coverage tracker against the sealed dev holdout (50 cases, 26 Tier-1).
|
# Coverage tracker against the sealed dev holdout: Tier-1 (26) + Tier-2 (24) = 50/50.
|
||||||
assert domain.corridor_correct == 26
|
assert domain.corridor_correct == 50
|
||||||
assert domain.corridor_refused == 24
|
assert domain.corridor_refused == 0
|
||||||
assert any("Tier-1" in note for note in domain.notes)
|
assert any("Tier-2" in note for note in domain.notes)
|
||||||
|
|
||||||
|
|
||||||
def test_domain_joins_full_report_and_scope_corrected() -> None:
|
def test_domain_joins_full_report_and_scope_corrected() -> None:
|
||||||
|
|
@ -61,8 +62,8 @@ def test_domain_joins_full_report_and_scope_corrected() -> None:
|
||||||
ids = [o.domain_id for o in report.outcomes]
|
ids = [o.domain_id for o in report.outcomes]
|
||||||
assert "arithmetic-chain" in ids
|
assert "arithmetic-chain" in ids
|
||||||
assert report.wrong_zero_guard_held # deductive + arithmetic both wrong=0
|
assert report.wrong_zero_guard_held # deductive + arithmetic both wrong=0
|
||||||
# The stale "no compiler exists" scope note must be gone; the compiler exists now.
|
# The scope note reflects the full-holdout closure; the stale "no compiler" line is gone.
|
||||||
joined = " ".join(report.scope_limitations)
|
joined = " ".join(report.scope_limitations)
|
||||||
assert "now exists" in joined
|
assert "FULL real GSM8K dev holdout" in joined
|
||||||
assert "no reader-to-Hamiltonian compiler exists" not in joined
|
assert "no reader-to-Hamiltonian compiler exists" not in joined
|
||||||
json.dumps(report.as_dict()) # JSON-safe artifact
|
json.dumps(report.as_dict()) # JSON-safe artifact
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue