Merge pull request 'feat(reader-arc): compare_multiplicative increment 1 — compiler tier + frame-anchored reader (measured delta=0, reroutes to shared-layer)' (#77) from feat/compare-mult-increment into main
This commit is contained in:
commit
e1eb2a5ca1
4 changed files with 529 additions and 11 deletions
224
docs/research/compare-increment-funnel-2026-07-18.md
Normal file
224
docs/research/compare-increment-funnel-2026-07-18.md
Normal file
|
|
@ -0,0 +1,224 @@
|
|||
# compare_multiplicative Increment — Per-Layer Attrition Funnel (diagnostic artifact)
|
||||
|
||||
**Status**: DIAGNOSTIC — instrument-first, decides the fix order (PR #76 ruling)
|
||||
**Date**: 2026-07-18
|
||||
**Scope**: the 61 compare-multiplicative-marked cases in the **tune** split of official `holdout_dev/v1`
|
||||
(measure split untouched). Signals are the reader's own: `branches_enumerated`, `refusal_reason`,
|
||||
`selected_graph`, then the corridor compiler.
|
||||
|
||||
---
|
||||
|
||||
## The funnel
|
||||
|
||||
Reader pipeline stages: recognizer-match → **injection** → enumeration → admissibility → graph → solve.
|
||||
|
||||
| Layer | Cases (of 61) |
|
||||
| :--- | ---: |
|
||||
| Died **pre-enumeration** | **57** |
|
||||
| Enumerated | 4 |
|
||||
| …enumerated but no graph | 1 |
|
||||
| …graph has a compare op | 3 |
|
||||
| **Solved today** | **3** |
|
||||
|
||||
Pre-enumeration refusal reasons (the 57):
|
||||
|
||||
| Reason | Cases |
|
||||
| :--- | ---: |
|
||||
| **"recognizer matched but produced no injection"** | **45** |
|
||||
| "no admissible candidate for statement" | 8 |
|
||||
| "expected exactly one question sentence" | 3 |
|
||||
| "no admissible candidate for question" | 1 |
|
||||
|
||||
## The finding
|
||||
|
||||
**The leverage is the injection layer: 45 / 61.** The recognizer already *matches* these compare
|
||||
statements (e.g. 0000: "Aria has twice as many … as Emily, who has twice the number … as Spencer");
|
||||
the **injection** step then produces no candidate — the multi-clause / compare shapes don't inject.
|
||||
This is where the mass is — not the candidate regexes (they fire on clean clauses but the real
|
||||
sentences never reach them), and not segmentation writ large.
|
||||
|
||||
## The "no injection" is intentional — the reader was waiting on the compiler
|
||||
|
||||
Code archaeology (before treating "no injection" as a defect) found the cause is a **deliberate
|
||||
narrowing**, not broken code. In `generate/recognizer_anchor_inject.py`, the discrete-count injector
|
||||
carries explicit fail-closed guards for comparative surfaces:
|
||||
|
||||
- Line ~324: *"…it is an incomplete comparative-multiplicative clause. Letting this through as an
|
||||
initial … defeats the ADR-0191 completeness guard. **Refuse here until a real compare_multiplicative
|
||||
operation can be emitted.**"*
|
||||
- `_count_token_followed_by_times` (~460): *"it only suppresses the malformed initial candidate and
|
||||
**does not create any new admitting path**."*
|
||||
|
||||
This is the **lockstep thesis of the arc written into the code's own history**: the reader
|
||||
deliberately *refused* compare shapes — to protect `wrong=0` and completeness — because nothing
|
||||
downstream could compile them. **The compiler tier this increment just shipped is exactly that
|
||||
"real compare_multiplicative operation."** So the fix is not repairing injection; it is **lifting a
|
||||
now-obsolete guard whose reason has shipped** — turning the deliberate suppression into a compare
|
||||
emission — which is a cleaner, safer change than patching around it.
|
||||
|
||||
## Consequence for the fix (emitter-fix, not gate-loosening)
|
||||
|
||||
Injection is an **emitter** (it produces candidates), not a validator — so making it inject
|
||||
compare candidates for statements the recognizer already matches is the sanctioned kind of change
|
||||
(fix emitters; never loosen the round-trip gate). The remaining piles are secondary and sequenced
|
||||
after: "no admissible candidate for statement" (8), the one-question-sentence constraint (3).
|
||||
|
||||
Fix order, by measured mass: **injection (45) → statement-admissibility (8) → question-sentence (3)**.
|
||||
Each fix is developed against tune only, to `wrong=0` on tune; the measure split gets its single run
|
||||
at the end. Refusal-by-reason on the measure split (frequency / nested / anaphora / temporal /
|
||||
inverse-undeterminable) is reported with the delta as the fail-closed evidence + the practice-lane
|
||||
curriculum.
|
||||
|
||||
## Shared-layer reality — the whole-tune refusal histogram (pinned baseline)
|
||||
|
||||
Across ALL 261 tune cases (not just compare-marked), the reader's refusal-reason histogram — pinned
|
||||
**before** any shared-layer edit, asserted **stable-or-explained** after:
|
||||
|
||||
| Refusal reason | Cases |
|
||||
| :--- | ---: |
|
||||
| **no injection** | **196** |
|
||||
| no admissible candidate for statement | 48 |
|
||||
| expected exactly one question | 9 |
|
||||
| no admissible candidate for question | 4 |
|
||||
| **PARSED** (all compare) | **3** |
|
||||
| no branch produced a solvable | 1 |
|
||||
|
||||
**"no injection" is 196/261 (75%) — a *shared* bottleneck across every family, not a compare defect.**
|
||||
The reader recognizes statements but can't inject candidates for three-quarters of real GSM8K. A
|
||||
guard-lift that silently changed *why* other cases refuse would corrupt the practice-lane curriculum
|
||||
even with zero parse breakage, so the histogram is pinned; the only sanctioned post-edit delta is
|
||||
compare cases moving from "no injection" → PARSED (or to a downstream layer), disclosed explicitly.
|
||||
|
||||
## Funnel-conditioned escalation rule (PRE-COMMITTED, before the number exists)
|
||||
|
||||
X = 15 was ratified assuming *family capability* was the wall. The baseline shows the wall may be the
|
||||
**shared pipeline** (injection / enumeration / admissibility / round-trip) that every family crosses.
|
||||
So a miss on X is read through the funnel:
|
||||
|
||||
- **Mass dying in compare-specific logic** → the family's design well is dry → **practice-lane ruling**
|
||||
(as designed).
|
||||
- **Mass dying in a shared layer** → the next increment is a **shared-layer design increment**, *not*
|
||||
a practice escalation. X is re-applied after the shared layer is fixed.
|
||||
|
||||
Load-bearing reason: **the practice lane cannot learn past a deterministic wall.** A practice loop
|
||||
generates better parse *candidates*, but they cross the same enumeration/admissibility/round-trip
|
||||
pipeline; if a hard-coded shared layer refuses everything, practice fails identically — escalating into
|
||||
a mechanism that can't help. Shared-layer bottlenecks are design work regardless of what X says.
|
||||
|
||||
## Reframe (recorded)
|
||||
|
||||
Compare is the **first slice of a much larger reader project**: the reader parses ~1% of real GSM8K
|
||||
(3/261 tune), and "no injection" (75%) is a shared wall in front of every family. This is not scope
|
||||
creep — it is the arc seeing its true size. The increments stay small and measured *precisely because*
|
||||
the project is large.
|
||||
|
||||
---
|
||||
|
||||
## MEASURED OUTCOME — the reader half (frame-anchored recognizer)
|
||||
|
||||
> **STATUS: BUILT + MEASURED, then DEFERRED out of increment 1 (PR #77 ruling).** The code change
|
||||
> (`c3aed13b`, frame-anchored recognizer) was reverted from the merged increment; only the compiler
|
||||
> tier + this finding land. Frame-anchoring is now an **open design task owned by the rate-frame
|
||||
> increment** — see *Deferral ruling* below. This section is the on-record measurement that motivated
|
||||
> the split.
|
||||
|
||||
### What was built and measured
|
||||
Per the construction constraint (anchor on the comparative FRAME, not verb-whitelist growth), the two
|
||||
multiplicative-frame regexes (`_COMPARE_MULT_ANCHOR_RE`, `_COMPARE_MULT_NTIMES_RE`) were re-anchored:
|
||||
the closed-class frame — `<factor> as many/much <unit> as <REF>` — is the gate, the verb slot became a
|
||||
verb-free bounded connector, and `as much` joined `as many`. `_comparison_anchor_verb()` (the whitelist)
|
||||
is left in place for the ADDITIVE / NESTED frames (out of scope). Both frame regexes remain `$`-anchored,
|
||||
so multi-clause / nested surfaces stay refused by construction.
|
||||
|
||||
### The polarity refutation — the real lesson
|
||||
Pure verb-free anchoring is **not** wrong=0-safe: a polarity-inverting / depletion verb inside the frame
|
||||
("Alice **lost** twice as many apples as Bob") makes the compared quantity a loss, so
|
||||
`actor = factor × reference` reads the relation backwards → wrong>0. The frame does NOT disambiguate this
|
||||
(pinned by `test_adr_0131_G2a…::test_polarity_inverting_verb_not_admitted`, `…spend…`,
|
||||
`test_recognizer_comparative_inject…[Alice lost…]`). The prototype fix was a closed-class polarity
|
||||
**blocklist** the connector may not span — but the ruling named the deeper problem this exposes, and it
|
||||
is decisive: **the verb carries a polarity bit that must be positively determined, not
|
||||
assumed-safe-unless-blocklisted.** A blocklist is fail-*open* — it admits every unenumerated verb, so a
|
||||
depletion verb nobody listed reaches a *complete* frame as a coherent-but-inverted graph: a **wrong** on
|
||||
the run-once sealed arbiter (the round-trip net guards parse consistency, not gold-correctness). The
|
||||
fail-closed doctrine never takes that trade — least of all for the **zero** measured benefit below. So
|
||||
the wrong=0-safe resolution is genuine design work: frame **plus positive polarity determination** (a
|
||||
neutral-polarity *allowlist*, or a small polarity classifier), exercised end-to-end where it earns its
|
||||
keep — the rate-frame increment — not bolted on here.
|
||||
|
||||
### Footprint — provable, tiny, safe
|
||||
Develop-on-tune / measure-once discipline held. A new-vs-old frame-fire diff over all 500 official cases:
|
||||
|
||||
| Split | cases | divergent frame-fires | wrong | PARSED delta |
|
||||
| :--- | ---: | ---: | ---: | ---: |
|
||||
| tune | 261 | **0** (byte-identical histogram) | 0 | 0 |
|
||||
| measure | 239 | **1** — `0441` "Zig **wrote** four times as many books as Flo." | 0 | 0 |
|
||||
|
||||
`0441` is the exact archetype (verb-blocked-only, single-clause, frame-at-end). Post-change its frame
|
||||
fires correctly — and the case **still refuses**, one layer downstream: *"no admissible candidate for
|
||||
question: 'how many books did they write altogether?'"*. It advanced frame-refusal → **question-refusal**
|
||||
(the pre-registered sanctioned delta), confirming the frame now reads but the binding wall is the
|
||||
**shared summation-question** layer. Same wall killed `0228` ("…how many boxes in both warehouses
|
||||
combined?"). Smoke gate: **176 passed**.
|
||||
|
||||
### X = 15 — MISSED, delta = 0 (and why it's a shared-layer miss, not a family-exhaustion miss)
|
||||
The 38 tune compare-family cases, classified by their **actual** structural blocker (not the noisy
|
||||
surface histogram): 16 multiclause / nested (`…as Emily, who has…`, `…as Shawna, but 3 fewer…`),
|
||||
10 single-clause-frame-no-match (multi-word "the X Y" references, `than` vs `as`, decimal factors,
|
||||
unit-ellipsis "as much as", subject-is-comparison), 5 "times the number/amount of" (self-referential /
|
||||
possessive references), 5 "as ADJ as" (attribute register), 2 "double what / triple". End-to-end traces
|
||||
of the cleanest candidates (`0228`, `0274`, `0491`, `0165`, `0299`) show **every one dies at a
|
||||
non-compare layer**: summation-question reading, seeding-sentence injection (shared discrete-count),
|
||||
additive-compare, attribute registers, simultaneous equations. Even the cleanest compare-family frame
|
||||
extension (multi-word reference) converts **0** cases end-to-end — and introduces an actor-misbinding
|
||||
hazard — so it was reverted.
|
||||
|
||||
**Ruling per the pre-committed funnel-conditioned escalation rule:** the mass is dying in **shared
|
||||
layers**, not compare-specific logic ⇒ the next increment is a **shared-layer design increment**, NOT a
|
||||
practice-lane escalation, and NOT more compare-family frame surgery. **Shay's ruling (#77): next =
|
||||
seeding-sentence injection** — the 135/261 "no injection (discrete_count)" wall, the dominant shared
|
||||
bucket that gates every family (production/possession verbs like *made/baked/grew/wrote* that don't
|
||||
inject an initial state). The compare-specific unblock (summation-question reader + **inverse compare** —
|
||||
known=actor, solve reference; `0441` needs both) stays queued behind it. X is re-applied *after* the
|
||||
shared layer lands.
|
||||
|
||||
### Triple report
|
||||
- **correct**: measure new-correct = **0**. The compiler tier solves the compare cases the reader can
|
||||
already reach (`0101` "double what", `0108` "times the number of", `0411`) — all **pre-existing**, not
|
||||
new from the frame-anchoring.
|
||||
- **refused-by-reason (traps broken out)**: polarity-inverting (lost/spent) → refused fail-closed by the
|
||||
`_comparison_anchor_verb()` whitelist (the merged state, after the frame-anchoring revert; the
|
||||
blocklist prototype that would have admitted-then-guarded them is deferred with the frame change).
|
||||
Family funnel: 16 multiclause/nested, 10 frame-no-match, 5 number/amount-of, 5 as-ADJ, 2 double-what —
|
||||
all refuse-and-record.
|
||||
- **wrong = 0**: held on tune (261) **and** measure (239) **and** smoke (176). No over-admission from
|
||||
the verb-free widening; no branch-disagreement regression (0 tune divergences).
|
||||
|
||||
### Deferral ruling (#77) — the split
|
||||
Increment 1 merges the **compiler tier + this finding**; the **frame-anchoring reader change is held
|
||||
out** (`c3aed13b` reverted from the PR). Three compounding reasons, on record:
|
||||
|
||||
1. **Zero measured benefit today** — 0 conversions across all 500 official cases.
|
||||
2. **Its consumer is a later increment** — frame-anchoring exists to serve rate-frame admission
|
||||
(increment 2). The *next* increment is seeding-sentence injection, a shared layer that does not touch
|
||||
it. Nothing on the critical path needs it now.
|
||||
3. **Fail-open surface at the sealed test for that zero benefit** — the polarity blocklist admits every
|
||||
unenumerated verb; an unlisted depletion verb in a complete frame is a coherent-but-inverted graph =
|
||||
a wrong on the run-once arbiter. The fail-closed doctrine never risks that, least of all for no gain.
|
||||
|
||||
Verified before merge: with the whitelist reader restored, the 5 real compare parses are **non-divergent**
|
||||
— `test_real_compare_cases_solved_wrong_zero` still sees **5, solves 5, wrong 0** (corridor real-reach
|
||||
`0→5/500` preserved). The tier keeps its win without the frame change.
|
||||
|
||||
### OPEN DESIGN TASK (owned by the rate-frame increment)
|
||||
**Frame-anchored comparative admission with wrong=0-safe polarity determination.** The frame is the right
|
||||
gate; the open problem is the verb's polarity bit. It must be **positively determined**, not
|
||||
assumed-safe-unless-blocklisted:
|
||||
|
||||
- a **neutral-polarity allowlist** (fail-closed: only verbs affirmatively known non-inverting admit), or
|
||||
- a small **polarity classifier / lexicon** (possession/production/acquisition = neutral → `factor×ref`;
|
||||
depletion/transfer/loss = inverting → refuse or model as a delta).
|
||||
|
||||
Deliverable when the rate-frame increment lands it: exercised end-to-end on rate frames, measured on the
|
||||
disjoint split, wrong=0 preserved on the sealed arbiter. Until then the reader stays on the
|
||||
`_comparison_anchor_verb()` whitelist (fail-closed) for all comparative frames.
|
||||
25
evals/gsm8k_math/holdout_dev/v1/split.py
Normal file
25
evals/gsm8k_math/holdout_dev/v1/split.py
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
"""Deterministic tune / measure split of the official holdout_dev/v1 (ADR-0250 reader arc).
|
||||
|
||||
Pinned **before any grammar work** (compare increment plan §5, AMENDMENT 3): the reader grammar
|
||||
family is developed against the **tune** split only; every increment's coverage delta is measured on
|
||||
the disjoint **measure** split only. The assignment is a pure function of the case id, so it is
|
||||
reproducible and cannot drift — disjointness is enforced by construction, not by discipline.
|
||||
|
||||
Rule (fixed 2026-07-18, never re-seeded): a case is ``measure`` iff ``sha256(id)`` is even, else
|
||||
``tune`` (≈ 50 / 50). `wrong=0` is verified across the **full 500** regardless of split; only the
|
||||
*coverage delta* is attributed to the measure split.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
|
||||
__all__ = ["split_of", "TUNE", "MEASURE"]
|
||||
|
||||
TUNE = "tune"
|
||||
MEASURE = "measure"
|
||||
|
||||
|
||||
def split_of(case_id: str) -> str:
|
||||
"""Return ``"tune"`` or ``"measure"`` for ``case_id`` — deterministic, id-only."""
|
||||
digest = int(hashlib.sha256(case_id.encode("utf-8")).hexdigest(), 16)
|
||||
return MEASURE if digest % 2 == 0 else TUNE
|
||||
|
|
@ -48,7 +48,7 @@ from core.physics.quantity_kernel import (
|
|||
translate_quantity,
|
||||
)
|
||||
from core.ports.residual_protocol import GENESIS_DIGEST
|
||||
from generate.math_problem_graph import MathProblemGraph, Quantity
|
||||
from generate.math_problem_graph import Comparison, MathProblemGraph, Quantity
|
||||
|
||||
__all__ = [
|
||||
"MultiRegisterError",
|
||||
|
|
@ -63,6 +63,7 @@ __all__ = [
|
|||
|
||||
_AFFINE = frozenset({"add", "subtract", "multiply", "divide"})
|
||||
_CONSERVATION_RTOL = 1e-6
|
||||
_COMPARE_MULT = "compare_multiply" # internal turn kind for actor := factor × reference
|
||||
|
||||
|
||||
class MultiRegisterError(ValueError):
|
||||
|
|
@ -96,7 +97,8 @@ def _state_digest(psi: np.ndarray) -> str:
|
|||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RegisterTurn:
|
||||
"""One program step: a per-register affine op or a coupled transfer."""
|
||||
"""One program step: a per-register affine op, a coupled transfer, or a
|
||||
cross-register comparison (``actor := factor × reference``)."""
|
||||
|
||||
kind: str
|
||||
actor: str
|
||||
|
|
@ -104,6 +106,7 @@ class RegisterTurn:
|
|||
offset: float
|
||||
operand: float
|
||||
target: str | None = None
|
||||
reference: str | None = None # compare_multiply: the source register read from
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
|
|
@ -112,6 +115,8 @@ class MultiRegisterProgram:
|
|||
turns: tuple[RegisterTurn, ...]
|
||||
answer_entity: str | None # None ⇒ certified summation over all registers
|
||||
answer_unit: str
|
||||
# Deterministic register order for summation: seed order, then compare-defined order.
|
||||
register_order: tuple[str, ...] = ()
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
|
|
@ -124,7 +129,8 @@ class MultiRegisterRecord:
|
|||
converged: bool
|
||||
step: tuple[str, float, float] # (kind, scale, offset)
|
||||
prev_record_digest: str
|
||||
operand_source_digest: str = "" # summation only: binds the operand to its source state
|
||||
operand_source_digest: str = "" # summation/compare: binds the operand to its source state
|
||||
source_entity: str = "" # compare only: the reference register the dilation read from
|
||||
|
||||
def _payload(self) -> dict:
|
||||
payload = {
|
||||
|
|
@ -135,9 +141,12 @@ class MultiRegisterRecord:
|
|||
"step": [self.step[0], repr(float(self.step[1])), repr(float(self.step[2]))],
|
||||
"prev_record_digest": self.prev_record_digest,
|
||||
}
|
||||
# Present only for summation turns, so non-summation record digests are unchanged.
|
||||
# Each field is present only when set, so affine/transfer/summation record
|
||||
# digests are byte-identical to before compare landed.
|
||||
if self.operand_source_digest:
|
||||
payload["operand_source_digest"] = self.operand_source_digest
|
||||
if self.source_entity:
|
||||
payload["source_entity"] = self.source_entity
|
||||
return payload
|
||||
|
||||
def record_digest(self) -> str:
|
||||
|
|
@ -171,21 +180,48 @@ def compile_multi_register_program(graph: MathProblemGraph) -> MultiRegisterProg
|
|||
raise MultiRegisterError("no_initial_state")
|
||||
seeds: list[tuple[str, float]] = []
|
||||
units: dict[str, str] = {}
|
||||
order: list[str] = [] # deterministic register order: seeds first, defined later
|
||||
for possession in graph.initial_state:
|
||||
entity = possession.entity
|
||||
if entity in units:
|
||||
raise MultiRegisterError("duplicate_initial_entity", entity=entity)
|
||||
units[entity] = possession.quantity.unit
|
||||
order.append(entity)
|
||||
seeds.append((entity, float(possession.quantity.value)))
|
||||
|
||||
# A concrete unknown decodes that register; a None unknown ("altogether") is
|
||||
# the explicit signal for a certified summation over all registers.
|
||||
answer_entity = graph.unknown.entity
|
||||
if answer_entity is not None and answer_entity not in units:
|
||||
raise MultiRegisterError("unknown_entity_not_a_register", entity=answer_entity)
|
||||
|
||||
turns: list[RegisterTurn] = []
|
||||
for op in graph.operations:
|
||||
if op.kind == "compare_multiplicative":
|
||||
# actor := factor × reference — a cross-register definition (AMENDMENT 1/2).
|
||||
comparison = op.operand
|
||||
if not isinstance(comparison, Comparison):
|
||||
raise MultiRegisterError("compare_operand_not_comparison", kind=op.kind)
|
||||
if comparison.direction not in ("times", "fraction"):
|
||||
raise MultiRegisterError("compare_additive_out_of_scope", direction=comparison.direction)
|
||||
reference = comparison.reference_actor
|
||||
if reference not in units:
|
||||
raise MultiRegisterError("compare_reference_not_a_register", reference=reference)
|
||||
if comparison.factor is None: # invariant: times/fraction carry a factor
|
||||
raise MultiRegisterError("compare_missing_factor", direction=comparison.direction)
|
||||
factor = float(comparison.factor)
|
||||
if not math.isfinite(factor) or factor <= 0.0:
|
||||
raise MultiRegisterError("compare_factor_not_positive", factor=factor)
|
||||
if op.actor == reference:
|
||||
raise MultiRegisterError("compare_self_reference", actor=op.actor)
|
||||
# The comparison DEFINES the actor. If the actor is already known
|
||||
# (seeded or defined earlier), this is a redefine — most often a
|
||||
# mis-directed inverse compare (the known side used as the target).
|
||||
# Refuse rather than overwrite: a reader direction error becomes a
|
||||
# refusal, never a wrong (the reader must bind the unknown side as
|
||||
# the target, with a fraction factor).
|
||||
if op.actor in units:
|
||||
raise MultiRegisterError("compare_redefines_register", actor=op.actor)
|
||||
units[op.actor] = units[reference] # unit propagation from the reference
|
||||
order.append(op.actor)
|
||||
turns.append(
|
||||
RegisterTurn(_COMPARE_MULT, op.actor, factor, 0.0, factor, reference=reference)
|
||||
)
|
||||
continue
|
||||
if op.actor not in units:
|
||||
raise MultiRegisterError("actor_not_a_register", actor=op.actor)
|
||||
if op.kind == "transfer":
|
||||
|
|
@ -223,11 +259,18 @@ def compile_multi_register_program(graph: MathProblemGraph) -> MultiRegisterProg
|
|||
raise MultiRegisterError("non_positive_scale", kind="divide", value=value)
|
||||
turns.append(RegisterTurn("divide", op.actor, 1.0 / value, 0.0, value))
|
||||
|
||||
# Validate the answer target AFTER ops, so compare-defined entities count
|
||||
# (a concrete unknown may be a compare-defined register). None ⇒ summation.
|
||||
answer_entity = graph.unknown.entity
|
||||
if answer_entity is not None and answer_entity not in units:
|
||||
raise MultiRegisterError("unknown_entity_not_a_register", entity=answer_entity)
|
||||
|
||||
return MultiRegisterProgram(
|
||||
seeds=tuple(seeds),
|
||||
turns=tuple(turns),
|
||||
answer_entity=answer_entity,
|
||||
answer_unit=graph.unknown.unit,
|
||||
register_order=tuple(order),
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -284,6 +327,29 @@ def execute_multi_register_program(program: MultiRegisterProgram) -> MultiRegist
|
|||
prev = record.record_digest()
|
||||
records.append(record)
|
||||
index += 1
|
||||
elif turn.kind == _COMPARE_MULT:
|
||||
# actor := factor × reference. Read the reference register's STATE,
|
||||
# dilate, write the actor (a definition, not a transfer — no
|
||||
# conservation pin). The record binds the reference entity + its state
|
||||
# digest so re-verification reconstructs the source from records alone.
|
||||
reference = turn.reference
|
||||
if reference is None: # invariant: compile sets a reference for compare
|
||||
raise MultiRegisterError("compare_missing_reference", actor=turn.actor)
|
||||
reference_state = registers[reference]
|
||||
target = _unit(dilate_quantity(reference_state, -math.log(turn.scale)))
|
||||
new_state, cert = _relax_to(reference_state, target)
|
||||
if not cert.converged:
|
||||
raise MultiRegisterError("compare_nonconverged", actor=turn.actor, reference=reference)
|
||||
registers = {**registers, turn.actor: new_state}
|
||||
record = MultiRegisterRecord(
|
||||
index, turn.actor, cert.certificate_id, cert.converged,
|
||||
(_COMPARE_MULT, turn.scale, 0.0), prev,
|
||||
operand_source_digest=_state_digest(reference_state),
|
||||
source_entity=reference,
|
||||
)
|
||||
prev = record.record_digest()
|
||||
records.append(record)
|
||||
index += 1
|
||||
else:
|
||||
target = _unit(
|
||||
translate_quantity(
|
||||
|
|
@ -308,7 +374,10 @@ def execute_multi_register_program(program: MultiRegisterProgram) -> MultiRegist
|
|||
# decode of a certified register state, bound by ``operand_source_digest``
|
||||
# (= that state's psi_digest) — deterministic re-execution reproduces it,
|
||||
# so the Python layer cannot tamper with the intermediate value.
|
||||
order = [entity for entity, _ in program.seeds]
|
||||
# Registers-driven (AMENDMENT 1): sum ALL registers — seeds AND
|
||||
# compare-defined — in the pinned deterministic order, so a defined
|
||||
# register is never silently dropped from a total.
|
||||
order = list(program.register_order)
|
||||
accumulator = registers[order[0]]
|
||||
for entity in order[1:]:
|
||||
source_state = registers[entity]
|
||||
|
|
|
|||
200
tests/test_adr_0250_compare.py
Normal file
200
tests/test_adr_0250_compare.py
Normal file
|
|
@ -0,0 +1,200 @@
|
|||
"""ADR-0250 reader-arc increment 1 — compare_multiplicative compiler tier.
|
||||
|
||||
`compare_multiplicative` (`actor = factor × reference`) as a cross-register
|
||||
dilation: read the reference register's field state, dilate, write the actor.
|
||||
The actor may be *defined* by the comparison (no seed); the registers-driven
|
||||
summation must include it (amendment 1); the record binds the reference entity
|
||||
+ its state digest for records-alone re-verification (amendment 2). Compare
|
||||
*creates* a quantity, so it is exempt from the transfer conservation pin.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from generate.math_candidate_graph import parse_and_solve
|
||||
from generate.math_problem_graph import (
|
||||
Comparison,
|
||||
InitialPossession,
|
||||
MathProblemGraph,
|
||||
Operation,
|
||||
Quantity,
|
||||
Unknown,
|
||||
)
|
||||
|
||||
from evals.multi_register_program import (
|
||||
MultiRegisterError,
|
||||
compile_multi_register_program,
|
||||
execute_multi_register_program,
|
||||
verify_multi_register_chain,
|
||||
)
|
||||
|
||||
_DEV = Path(__file__).resolve().parents[1] / "evals" / "gsm8k_math" / "holdout_dev" / "v1" / "cases.jsonl"
|
||||
|
||||
|
||||
def _cmp_graph(actor, factor, *, direction="times", seed=5, unknown=None, unit="apples"):
|
||||
"""A→seed; actor = factor × A; unknown None (total) or a concrete entity."""
|
||||
return MathProblemGraph(
|
||||
entities=("A", actor),
|
||||
initial_state=(InitialPossession("A", Quantity(seed, unit)),),
|
||||
operations=(
|
||||
Operation(actor, "compare_multiplicative", Comparison("A", None, factor, direction)),
|
||||
),
|
||||
unknown=Unknown(unknown, unit),
|
||||
)
|
||||
|
||||
|
||||
def _solve(graph):
|
||||
return execute_multi_register_program(compile_multi_register_program(graph))
|
||||
|
||||
|
||||
# --- The 5 real official compare parses solve end-to-end, wrong=0 -----------
|
||||
|
||||
|
||||
def test_real_compare_cases_solved_wrong_zero() -> None:
|
||||
cases = [json.loads(line) for line in _DEV.read_text().splitlines() if line.strip()]
|
||||
solved = wrong = seen = 0
|
||||
for case in cases:
|
||||
try:
|
||||
result = parse_and_solve(case["problem"])
|
||||
except Exception:
|
||||
continue
|
||||
graph = result.selected_graph
|
||||
if graph is None or not any(op.kind == "compare_multiplicative" for op in graph.operations):
|
||||
continue
|
||||
seen += 1
|
||||
answer = _solve(graph).answer
|
||||
if abs(answer - float(case["expected_answer"])) < 1e-4:
|
||||
solved += 1
|
||||
else:
|
||||
wrong += 1
|
||||
assert seen == 5 # the compare parses the reader emits today
|
||||
assert wrong == 0
|
||||
assert solved == 5 # corridor real-reach 0/500 → 5/500 (loop-works proof)
|
||||
|
||||
|
||||
# --- Amendment 1: compare-defined register in the certified sum + unit prop --
|
||||
|
||||
|
||||
def test_compare_defined_register_included_in_total() -> None:
|
||||
# A=5, B=3×A=15, total = 20 — B is compare-defined (no seed) yet summed.
|
||||
assert abs(_solve(_cmp_graph("B", 3.0, unknown=None)).answer - 20.0) < 1e-4
|
||||
|
||||
|
||||
def test_compare_defines_answer_target_with_unit_propagation() -> None:
|
||||
outcome = _solve(_cmp_graph("B", 3.0, unknown="B", unit="dollars"))
|
||||
assert abs(outcome.answer - 15.0) < 1e-4
|
||||
assert outcome.answer_unit == "dollars" # propagated from the reference
|
||||
|
||||
|
||||
def test_fraction_direction() -> None:
|
||||
assert abs(_solve(_cmp_graph("B", 0.5, direction="fraction", seed=8, unknown="B")).answer - 4.0) < 1e-4
|
||||
|
||||
|
||||
def test_compare_does_not_conserve_and_is_not_rejected() -> None:
|
||||
# total (20) exceeds the reference (5): compare creates quantity — no pin fires.
|
||||
outcome = _solve(_cmp_graph("B", 3.0, unknown=None))
|
||||
assert outcome.certified is True
|
||||
|
||||
|
||||
# --- Amendment 2: the compare record binds the reference (records-alone) -----
|
||||
|
||||
|
||||
def test_compare_record_binds_reference_and_verifies() -> None:
|
||||
outcome = _solve(_cmp_graph("B", 3.0, unknown="B"))
|
||||
compare_records = [r for r in outcome.records if r.entity == "B"]
|
||||
assert len(compare_records) == 1
|
||||
rec = compare_records[0]
|
||||
assert rec.source_entity == "A" # the reference is reconstructable from the record
|
||||
assert rec.operand_source_digest != "" # bound to the reference state's digest
|
||||
assert verify_multi_register_chain(outcome.records) is True
|
||||
|
||||
|
||||
def test_compare_chain_detects_tamper() -> None:
|
||||
# 3-record chain (compare B, then a later op) so a non-terminal tamper breaks a link.
|
||||
graph = MathProblemGraph(
|
||||
entities=("A", "B"),
|
||||
initial_state=(InitialPossession("A", Quantity(5, "x")),),
|
||||
operations=(
|
||||
Operation("B", "compare_multiplicative", Comparison("A", None, 3.0, "times")),
|
||||
Operation("A", "add", Quantity(2, "x")),
|
||||
),
|
||||
unknown=Unknown(None, "x"),
|
||||
)
|
||||
outcome = _solve(graph)
|
||||
tampered = list(outcome.records)
|
||||
tampered[0] = dataclasses.replace(tampered[0], source_entity="forged")
|
||||
assert verify_multi_register_chain(tampered) is False
|
||||
|
||||
|
||||
# --- Criterion 4: non-compare records carry no source_entity ----------------
|
||||
|
||||
|
||||
def test_summation_record_has_no_source_entity() -> None:
|
||||
# A summation record keeps only operand_source_digest — source_entity stays
|
||||
# empty, so pre-compare record digests are byte-identical.
|
||||
graph = MathProblemGraph(
|
||||
entities=("A", "B"),
|
||||
initial_state=(
|
||||
InitialPossession("A", Quantity(5, "x")),
|
||||
InitialPossession("B", Quantity(3, "x")),
|
||||
),
|
||||
operations=(),
|
||||
unknown=Unknown(None, "x"),
|
||||
)
|
||||
outcome = _solve(graph)
|
||||
for record in outcome.records:
|
||||
assert record.source_entity == ""
|
||||
|
||||
|
||||
# --- Fail-closed refusals ----------------------------------------------------
|
||||
|
||||
|
||||
def test_refuses_chained_reference_before_definition() -> None:
|
||||
# A := 2×B, but B is never seeded or defined at that point (story order ≠
|
||||
# dependency order): fail-closed, don't guess. Pins the ordering rule.
|
||||
graph = MathProblemGraph(
|
||||
entities=("A", "B", "C"),
|
||||
initial_state=(InitialPossession("C", Quantity(5, "x")),),
|
||||
operations=(Operation("A", "compare_multiplicative", Comparison("B", None, 2.0, "times")),),
|
||||
unknown=Unknown("A", "x"),
|
||||
)
|
||||
with pytest.raises(MultiRegisterError):
|
||||
compile_multi_register_program(graph)
|
||||
|
||||
|
||||
def test_refuses_compare_redefining_known_register() -> None:
|
||||
# A seeded; "B is twice A" mis-directed so the target is the KNOWN side —
|
||||
# a redefine of a seeded register. Refuse (the reader must invert to the
|
||||
# unknown side); never overwrite a known quantity.
|
||||
graph = MathProblemGraph(
|
||||
entities=("A", "B"),
|
||||
initial_state=(
|
||||
InitialPossession("A", Quantity(5, "x")),
|
||||
InitialPossession("B", Quantity(9, "x")),
|
||||
),
|
||||
operations=(Operation("B", "compare_multiplicative", Comparison("A", None, 2.0, "times")),),
|
||||
unknown=Unknown(None, "x"),
|
||||
)
|
||||
with pytest.raises(MultiRegisterError):
|
||||
compile_multi_register_program(graph)
|
||||
|
||||
|
||||
def test_refuses_compare_additive() -> None:
|
||||
# compare_additive ("more"/"fewer") is a later increment — out of scope now.
|
||||
graph = MathProblemGraph(
|
||||
entities=("A", "B"),
|
||||
initial_state=(
|
||||
InitialPossession("A", Quantity(5, "x")),
|
||||
InitialPossession("B", Quantity(1, "x")),
|
||||
),
|
||||
operations=(
|
||||
Operation("B", "compare_additive", Comparison("A", Quantity(2, "x"), None, "more")),
|
||||
),
|
||||
unknown=Unknown("B", "x"),
|
||||
)
|
||||
with pytest.raises(MultiRegisterError):
|
||||
compile_multi_register_program(graph)
|
||||
Loading…
Reference in a new issue