Merge pull request #157 from AssetOverflow/feat/adr-0123-substrate

feat(parser): ADR-0123 surface (lands #156 onto main)
This commit is contained in:
Shay 2026-05-23 02:09:01 -07:00 committed by GitHub
commit 79e786c727
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 1162 additions and 3 deletions

View file

@ -0,0 +1,502 @@
# ADR-0123 — Comparison-Phrasing Realizer (surface increment on the ADR-0123 substrate)
**Status:** Accepted (surface increment; substrate landed in PR #155)
**Date:** 2026-05-23
**Author:** CORE agents + reviewers
**Depends on:**
- ADR-0115 (parser substrate),
- ADR-0116 (deterministic solver),
- ADR-0117 (`SolutionTrace` verifier),
- ADR-0118 (stepped realizer),
- ADR-0119 (+ all 8 sub-phases),
- ADR-0121 (math `expert` promotion deferred),
- the ADR-0123 **substrate commit** (`feat/adr-0123-substrate`, commit
`c9bd5d4`): `Comparison` dataclass + `compare_additive` /
`compare_multiplicative` operation kinds + parser patterns +
solver / verifier wiring + `en_arithmetic_v1:compare_additive`
(en-arith-006) + `en_arithmetic_v1:compare_multiplicative`
(en-arith-007) pack lemmas.
**Supersedes:** none
> Disambiguation: this is **ADR-0123-parser-comparison-phrasing**, the
> parser-arc ADR. It is distinct from
> `ADR-0123-symbolic-logic-shape-remap.md` (the lane-shape governance
> ADR that happens to share the number). The disambiguation pattern
> follows the same convention as the audit-passed / parser-rate split
> (e.g., `ADR-0122-systems-software-audit-passed-deferred.md` vs the
> parser-rate work pinned in a parallel PR). Two ADRs with the same
> number, distinct slugs, distinct subject-matter streams.
---
## Context
ADR-0121 deferred the first `expert` promotion with named blocker
`correct_rate = 0/1319` on sealed GSM8K. ADR-0121 §"What would
unlock the promotion" enumerates a parser-expansion arc of 48
construction classes. **Comparison phrasing** is the second class
in that arc (the first was rate / per-unit, currently in flight as
a parallel parser-rate ADR; the two are non-blocking siblings).
PR #155 (`feat/adr-0123-substrate`, commit `c9bd5d4`) shipped the
*substrate* — the typed graph operand (`Comparison`), the two new
operation kinds (`compare_additive`, `compare_multiplicative`),
the parser patterns for the four canonical English surfaces
(`N more / N fewer / twice / N times / half`), the solver
handlers, the verifier replay-equality extensions, and the two
pack lemmas. What the substrate did *not* ship — by deliberate
scope discipline mirroring ADR-0119's sub-phase decomposition — is
the **surface realization** of comparison steps in
`generate/math_realizer.py`. Without realizer phrasing, a problem
that solves successfully under the new operation kinds still
raises `RealizerError("unknown operation_kind 'compare_additive'")`
when the stepped explanation is requested.
This ADR closes that one-line architectural gap: the surface
sentence templates for the four comparison shapes, wired into
`_step_sentence` so the full
`parse_problem → solve → verify → realize` pipeline operates on
comparison problems end-to-end.
---
## Decision
Extend `generate/math_realizer.py` with **two helper functions**
(`_compare_additive_sentence`, `_compare_multiplicative_sentence`)
plus a **dispatch branch** in `_step_sentence` that consumes the
new operation kinds. The signature of `_step_sentence` widens by
one optional parameter (`entity_units`) carrying a
`{entity_name: unit}` mapping derived once-per-trace from
`graph_initial_state` in `realize()`. This is the load-bearing
plumbing change: the multiplicative comparison helper needs the
reference actor's unit at render time, and the substrate
deliberately does not carry that on `Comparison` itself when
`factor` is set (the substrate's solver derives it from
in-flight state, which the realizer cannot reach without
re-running the solver).
### Realizer-level additions (`generate/math_realizer.py`)
1. **`_compare_additive_sentence(step: SolutionStep) -> str`** —
renders an additive-comparison step as one of two templates,
selected by `step.operand.direction`:
```
direction='more':
"<actor> has <delta.value> more <delta.unit> than <ref>,
giving <actor> a total of <after> <delta.unit>."
direction='fewer':
"<actor> has <delta.value> fewer <delta.unit> than <ref>,
leaving <actor> with a total of <after> <delta.unit>."
```
The two-clause shape — *comparison clause* + *resolved-state
clause* — is pinned as a structural invariant. `delta.value`
and `after_value` pluralize independently via the existing
`_unit_surface` helper, so "1 more apple" and the resolved
state "3 apples" coexist without forcing one count's
plurality onto the other.
2. **`_compare_multiplicative_sentence(step: SolutionStep,
entity_units: dict[str, str]) -> str`** — renders a
multiplicative comparison step as one of three templates,
selected by `step.operand.direction` and `factor`:
```
direction='times':
"<actor> has <factor> times as many <unit> as <ref>,
giving <actor> a total of <after> <unit>."
direction='fraction' (factor == 0.5):
"<actor> has half as many <unit> as <ref>,
giving <actor> a total of <after> <unit>."
direction='fraction' (other factor):
"<actor> has <factor> as many <unit> as <ref>,
giving <actor> a total of <after> <unit>."
```
`unit` is resolved via `entity_units[reference_actor]` — the
initial-state lookup is sufficient because the substrate's
solver refuses multi-unit reference actors (`SolveError("…is
ambiguous: reference actor … holds quantities in multiple
units…")`) and refuses to overwrite a comparison actor's
existing state. Both refusals guarantee that the reference's
initial unit and the comparison-time unit are the same string.
3. **`_step_sentence` dispatch widened** with two prepended
branches:
```python
if step.operation_kind == "compare_additive":
return _compare_additive_sentence(step)
if step.operation_kind == "compare_multiplicative":
if entity_units is None:
raise RealizerError(...)
return _compare_multiplicative_sentence(step, entity_units)
```
The pre-existing `add` / `subtract` / `transfer` / `multiply`
/ `divide` branches are unchanged, byte-identically. The new
branches sit at the top because the substrate's solver already
refuses ambiguous comparisons; the realizer's job is to
*render* what survived solver refusal, not to re-validate.
4. **`realize()` builds `entity_units` once** from
`graph_initial_state`:
```python
entity_units = {p.entity: p.quantity.unit for p in graph_initial_state}
```
and threads it through to every `_step_sentence` call. The
add/subtract/transfer/multiply/divide branches accept the
parameter and ignore it (default `None`), preserving the
existing behavior on pre-comparison traces.
### Refusal discipline (load-bearing)
The helpers raise `RealizerError` on every shape the substrate
already refuses *plus* one shape the substrate cannot see:
| Refusal | Substrate or Realizer |
|---|---|
| operand not a `Comparison` | realizer (defensive — substrate guarantees this via `Operation.__post_init__`) |
| `delta is None` in additive branch (multiplicative shape leaked) | realizer |
| `factor is None` in multiplicative branch (additive shape leaked) | realizer |
| `direction not in {'more','fewer'}` for additive | realizer |
| `direction not in {'times','fraction'}` for multiplicative | realizer |
| `actor == reference_actor` (self-comparison) | both (substrate at parse time; realizer at render time as defense in depth) |
| `reference_actor` not in `entity_units` | realizer (multiplicative only — substrate guarantees this for parsed problems but does not for hand-constructed graphs) |
ADR-0114a Obligation #4 (`wrong == 0`) holds by construction —
the new branches only fire when the substrate has already
emitted a successful step; if the substrate refused, no step
exists to render.
### What this ADR does NOT touch
- `generate/math_problem_graph.py``Comparison` is already
shipped by the substrate; not modified.
- `generate/math_parser.py``_try_comparison_declaration` is
already shipped by the substrate; not modified.
- `generate/math_solver.py``_apply_compare_additive` /
`_apply_compare_multiplicative` are already shipped; not
modified.
- `generate/math_verifier.py` — comparison-step verification is
already shipped; not modified.
- `language_packs/data/en_arithmetic_v1/*``en-arith-006`
(compare_additive) and `en-arith-007` (compare_multiplicative)
are already shipped at manifest version 1.1.0; this ADR adds
no further pack vocabulary.
The scope discipline matches ADR-0119's sub-phase decomposition
exactly: substrate ships first (PR #155), surface ships second
(this PR), each with its own re-measurable invariants.
---
## Anti-overfit re-measurement (load-bearing — per ADR-0121)
This ADR ships **only** when every measurement below holds.
### 1. Sealed-GSM8K correct_rate + wrong count
Run `evals/gsm8k_math/runner.py` against the decrypted sealed
holdout (1319 cases). **Pass condition**: `wrong == 0` (the
absolute discipline). The `correct_rate > 0.0` lift gate is
**deferred** — the substrate ADR pre-measured zero sealed lift
(every comparison-matching sealed case also requires aggregation
/ rate / conditional structure not yet in the parser), and the
realizer surface cannot create matches the parser refuses.
### 2. ADR-0118a OOD re-measurement
Run the OOD perturbation suite (`evals/gsm8k_parser_dev/ood_score.py`).
**Pass condition**: OOD/public ratio remains ≥ 0.95. Adding two
realizer branches and threading one extra parameter must not move
this number.
### 3. ADR-0125 perturbation re-measurement
Run the invariance perturbation suite. **Pass condition**:
invariance-preserving rate = 1.0; invariance-breaking rate = 1.0.
### 4. ADR-0119.5 adversarial re-measurement
Run `evals/gsm8k_math/adversarial/`. **Pass condition**:
`wrong == 0` across all 38 cases × 12 families.
### 5. ADR-0119.7 sealed-seal integrity
The sealed holdout `cases.jsonl.age` file is **not modified**.
SHA-256 digest unchanged.
### 6. ADR-0117 replay-equality
Runner remains deterministic — same case set → byte-equal
`LaneReport.canonical_bytes()`. The realizer change extends to
new step kinds but does not modify the existing kinds'
rendering, so prior traces re-render byte-identically.
### 7. Substrate measurement preservation
Every invariant the substrate ADR pinned (parser, solver,
verifier, pack-binding) continues to hold byte-identically. The
substrate's test suite re-runs cleanly under this PR.
### 8. ADR-0118 stepped-realizer preservation
The pre-existing add/subtract/transfer/multiply/divide step
sentences re-render byte-identically. ADR-0118's pinning of
those templates is not weakened.
---
## Invariants
### `adr_0123_realize_compare_additive_more_canonical`
`parse_problem("Alice has 5 apples. Bob has 3 more apples than
Alice. How many apples does Bob have?") → solve() → realize()`
produces prose containing **all** of:
- `"3 more apples than Alice"`
- `"Bob a total of 8 apples"`
- the final `"Bob has 8 apples."` answer sentence.
### `adr_0123_realize_compare_additive_fewer_canonical`
`"Anna has 10 flowers. Mary has 5 fewer flowers than Anna. How
many flowers does Mary have?"` → prose containing:
- `"5 fewer flowers than Anna"`
- `"Mary with a total of 5 flowers"`
### `adr_0123_realize_compare_multiplicative_twice_canonical`
`"Carla has 7 marbles. Ben has twice as many marbles as Carla.
How many marbles does Ben have?"` → prose containing:
- `"2 times as many marbles as Carla"`
- `"Ben a total of 14 marbles"`
### `adr_0123_realize_compare_multiplicative_n_times_canonical`
`"Tom has 3 cookies. Sara has 4 times as many cookies as Tom.
How many cookies does Sara have?"` → prose containing:
- `"4 times as many cookies as Tom"`
- `"Sara a total of 12 cookies"`
### `adr_0123_realize_compare_fraction_half_canonical`
`"Tom has 8 cookies. Lisa has half as many cookies as Tom. How
many cookies does Lisa have?"` → prose containing:
- `"half as many cookies as Tom"` (the literal word "half", not
"0.5 as many")
- `"Lisa a total of 4 cookies"`
### `adr_0123_realize_byte_deterministic`
Two `realize()` calls on the same `(graph_initial_state, trace)`
produce byte-equal `RealizedTrace.canonical_bytes()`. Pinned for
both additive and multiplicative cases.
### `adr_0123_realize_singular_plural_independence`
A `compare_additive` step with `delta.value=1` and `after_value=5`
renders `"1 more apple"` (singular delta clause) and
`"5 apples"` (plural resolved state). Symmetric: `delta.value=3`
and `after_value=1` renders `"3 more apples"` and
`"1 apple"`.
### `adr_0123_realize_refuses_non_comparison_operand`
A hand-constructed `SolutionStep` with `operation_kind="compare_additive"`
but `operand=Quantity(...)` raises `RealizerError("requires a
Comparison operand")`.
### `adr_0123_realize_refuses_missing_delta`
A `Comparison(direction='more', delta=None, factor=2.0)`
operand on a `compare_additive` step raises `RealizerError`
matching `/requires Comparison.delta/`.
### `adr_0123_realize_refuses_missing_factor`
A `Comparison(direction='times', delta=Quantity(3,'a'),
factor=None)` operand on a `compare_multiplicative` step raises
`RealizerError` matching `/requires Comparison.factor/`.
### `adr_0123_realize_refuses_missing_entity_units`
`_compare_multiplicative_sentence(step, entity_units={})` for a
reference actor not in the map raises `RealizerError` matching
`/initial state/`. This catches hand-constructed graph traces
that omit the reference actor from initial state.
### `adr_0123_realize_pre_comparison_traces_byte_identical`
A trace containing only add/subtract/transfer/multiply/divide
steps renders **byte-identically** to its pre-this-PR rendering
on the substrate branch. The realizer change does not modify
the prior templates.
### `adr_0123_sealed_correct_rate_zero_at_landing`
`run_lane(sealed_cases).metrics["correct_rate"] == 0.0` at the
time of landing. Inherits the substrate ADR's deferral mechanic:
the multi-construction barrier (every sealed comparison-matching
case combines with at least one other class not yet in the
parser) holds at the surface layer too — comparison alone
matches zero sealed cases. The test fails (correctly) only when
a future composition ADR finally lifts the number above 0.
### `adr_0123_sealed_wrong_zero_holds`
`run_lane(sealed_cases).metrics["wrong"] == 0`. Inherits the
substrate's wrong-zero discipline. The realizer cannot create
new misparses; it only renders successful traces.
### `adr_0123_adr_0118_stepped_realizer_unchanged`
ADR-0118's canonical realization tests pin the
add/subtract/transfer surfaces. They continue to pass
byte-identically.
---
## Measurement (at landing)
| Metric | Pre-ADR (substrate tip) | Post-ADR (this branch) | Gate | Pass? |
|---|---|---|---|---|
| `parse_problem → solve → realize` on 4 canonical comparison shapes | refuses at realize() with `unknown operation_kind` | **all four render** (more, fewer, twice/N times, half) | end-to-end pipeline | ✓ |
| sealed `correct_rate` | 0.0 (0/1319) | **0.0 (0/1319)** | deferred (see Decision) | ✓ (deferred) |
| sealed `wrong` | 0 | **0** | must remain 0 | ✓ |
| public `correct_rate` | 1.0 (150/150) | unchanged | ≥ 0.95 | ✓ |
| OOD/public ratio | 1.00 | unchanged | ≥ 0.95 | ✓ |
| perturbation invariance-preserving | 1.0 | unchanged | 1.0 | ✓ |
| perturbation invariance-breaking | 1.0 | unchanged | 1.0 | ✓ |
| adversarial `wrong` | 0 | **0** | 0 | ✓ |
| sealed seal SHA-256 | (pinned by ADR-0119.7) | unchanged | byte-equal | ✓ |
| ADR-0118 stepped-realizer canonical surfaces | pinned templates | unchanged | byte-equal | ✓ |
**Honest finding:** the realizer surface closes the last
architectural gap in the comparison-phrasing class. A problem
that the substrate's solver evaluates successfully now produces
show-your-work prose — without this ADR, every successful
comparison solve raises `RealizerError` at the explanation
step. The lift gate stays at zero because the parser only
recognizes the four canonical comparison surfaces in isolation,
not in composition with rate / aggregation / unit conversion
(the multi-construction barrier the substrate ADR documented).
---
## Out of scope
- **Composed rate × comparison constructions** ("A watermelon
costs three times what each pepper costs") — composition ADR.
- **Comparative ratio phrasing beyond half** (`"X has 2/3 as
many as Y"`, `"X has 75% of Y"`) — percentage / fraction ADR
(the third foundational class).
- **Multi-step comparison chains** ("A has 3 more than B; B has
twice as many as C") — composition ADR.
- **Comparative superlatives** ("X has the most apples", "Y has
more than anyone else") — out of arc.
- **Negative-direction additive with non-positive delta**
refused by substrate at construction time; realizer inherits.
- **Round-trip equality between realized prose and re-parsed
graph** — deferred to a future ADR. The current realizer
surfaces are *human-readable*; they are not yet a strict
fixed point of the parse → realize → parse → realize loop.
ADR-0118 holds this distinction for its own operation kinds;
ADR-0123 inherits it.
---
## What this proves (and what it doesn't)
### Proves
- The full `parse_problem → solve → verify → realize` pipeline
now operates end-to-end on the four canonical comparison
surfaces (`N more`, `N fewer`, `twice`/`N times`, `half`).
Before this ADR, the pipeline crashed at the last step.
- The wrong-zero discipline (ADR-0114a Obligation #4) holds
against an expanded grammar surface. Adding two realizer
branches did not introduce a single new misparse on any of
the existing eval lanes.
- ADR-0118's pinned templates re-render byte-identically. The
realizer change is purely additive at the dispatch layer.
### Does NOT prove
- That comparison problems will eventually lift sealed
`correct_rate`. They won't, in isolation — the multi-
construction barrier documented in the substrate ADR and
inherited here is the load-bearing reason. The cumulative
lift signal arrives after the 3rd or 4th foundational class
composes.
- That the chosen prose templates are the *best* templates
for downstream consumers. They are deterministic, structurally
invariant, and human-readable. If a future composition ADR
finds them ambiguous (e.g., the parser misparses its own
realizer output during round-trip), the templates get revised
at that point.
- That the `fraction` direction with non-`0.5` factors is
well-tested. The substrate parser only emits `factor=0.5` for
`fraction`; the realizer's fall-through template
(`"<factor> as many ..."`) exists for future
parser extensions but is not exercised by any parsed problem
today.
---
## Consequences
- The realizer covers all six operation kinds the substrate
emits (add/subtract/transfer/multiply/divide/compare_*).
ADR-0114a Obligation #5 (realizer coverage parity) holds
across the full graph vocabulary.
- ADR-0121's deferral remains in place — surface-layer ADRs
cannot move the sealed `correct_rate` gate because they only
render what the parser+solver already produce.
- Substrate measurements continue to hold byte-identically.
The realizer change is fully additive at the dispatch layer.
- The parser-expansion arc gains its second class **fully
end-to-end** (substrate + surface). Per ADR-0121's revised
sequencing, no lift signal is expected until at least the
3rd or 4th class lands. The next ADR is percentage /
fraction.
- The substrate-then-surface decomposition pattern (PR #155
this PR) is reusable for future parser-expansion classes.
Substrate ADRs ship the typed graph operand + parser/solver/
verifier wiring; surface ADRs ship the realizer phrasing.
Each measures independently.
---
## Why this ADR is small on purpose
ADR-0114a's honest-fitting discipline rewards narrow expansions
that each get fully re-measured across all anti-overfit lanes.
The substrate ADR shipped 4,798 lines of new code (parser
patterns + solver handlers + verifier extensions + pack lemmas
+ tests for adjacent ADRs that were caught in the same merge);
this ADR ships ~140 lines of realizer code and one ADR doc.
The substrate-then-surface split exists for two reasons:
1. **Independent bisection.** If a regression appears on OOD or
perturbation, the bisection points at one of: (a) the
substrate's parser/solver changes, or (b) this PR's
realizer phrasing. Bundling them into one PR loses the
bisection signal.
2. **Independent reviewability.** A reviewer who knows the
parser/solver subsystem need not also be a realizer expert,
and vice versa. Each PR has a tractable diff for a single
reviewer to load into working memory.
This is the same load-bearing rule as ADR-0119's sub-phase
decomposition and the substrate ADR's own scope discipline,
applied one level finer.

View file

@ -58,6 +58,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt
| [ADR-0120](ADR-0120-expert-promotion-contract.md) | First `expert` Promotion Contract (composes ADR-0114a 10/10) | Proposed (2026-05-23) |
| [ADR-0121](ADR-0121-mathematics-logic-expert-deferred.md) | `mathematics_logic` `expert` Promotion — Deferred (first attempt) | Accepted (2026-05-23) |
| [ADR-0122](ADR-0122-parser-rate-per-unit.md) | Parser Expansion: Rate / Per-Unit Reasoning (substrate-only; lift deferred) | Accepted (2026-05-22) |
| [ADR-0123](ADR-0123-parser-comparison-phrasing.md) | Comparison-Phrasing Realizer (surface increment on the ADR-0123 substrate; `_compare_additive_sentence` + `_compare_multiplicative_sentence`) | Accepted (2026-05-23) |
---
@ -111,6 +112,7 @@ The ADR-0091..0114 slate is fully accepted (0091..0113) plus one proposed-roadma
- First `expert` Promotion Contract (composes all 10 ADR-0114a obligations + correct_rate ≥ 0.60 floor + depth-curve ε=0.05 + signed expert_claims; proposed; ADR-0121 the first worked attempt) — ADR-0120
- First `expert` Promotion Attempt — `mathematics_logic` — DEFERRED (mirrors ADR-0107 → ADR-0110 pattern for audit-passed; all 10 obligations pass; correct_rate gate refuses honestly at 0/1319; parser-expansion arc is the named unlock; `wrong == 0` discipline holds against external benchmark) — ADR-0121
- Parser-Expansion Arc — first class shipped (rate/per-unit) as **substrate-only with lift deferred** (`Rate` dataclass + `apply_rate` operation kind + parser/solver/verifier/realizer + `en_arithmetic_v1:apply_rate` pack lemma; 41 invariants pinned; sealed `correct_rate` stays at 0/1319 with `wrong == 0`; multi-construction barrier documented — every real GSM8K rate problem combines rate with ≥1 other class, so per-ADR lift signal is corrected to cumulative-after-3rd-or-4th-class) — ADR-0122
- Parser-Expansion Arc — comparison-phrasing class shipped **fully end-to-end** as substrate (PR #155 `feat/adr-0123-substrate`: `Comparison` dataclass + `compare_additive`/`compare_multiplicative` operation kinds + parser patterns for `N more`/`N fewer`/`twice`/`N times`/`half` + solver/verifier wiring + `en_arithmetic_v1:compare_additive` + `en_arithmetic_v1:compare_multiplicative` pack lemmas) **+ surface** (this ADR: `_compare_additive_sentence` + `_compare_multiplicative_sentence` realizer helpers wired into `_step_sentence`; `parse_problem → solve → realize` operates end-to-end on all four comparison shapes); sealed `correct_rate` stays at 0/1319 with `wrong == 0` (multi-construction barrier holds; cumulative lift signal expected after the 3rd/4th class lands) — ADR-0123
- **Phase 5 complete (2026-05-22):** All ADR-0119 sub-phases (5.1..5.8) landed; ADR-0114a 10/10 obligations discharged for the gsm8k_math lane on main; first honest CORE-vs-real-GSM8K measurement published (0/1319 correct, 0 wrong, 1319 refused); ADR-0120 (first `expert` promotion contract) is the next gate.
ADR-0080 has also landed: Contemplation Loop Phase 1 adds a read-only frontier-compare miner that emits `SPECULATIVE` findings only.

View file

@ -39,7 +39,7 @@ import json
from dataclasses import dataclass
from typing import Any
from generate.math_problem_graph import Rate
from generate.math_problem_graph import Comparison, Rate
from generate.math_solver import SolutionStep, SolutionTrace
@ -104,9 +104,19 @@ def realize(graph_initial_state: tuple, trace: SolutionTrace) -> RealizedTrace:
for p in graph_initial_state
)
# ADR-0123: the multiplicative comparison helper needs the
# reference actor's unit, which is not stored on Comparison when
# factor is set. We derive it deterministically from initial
# state — the only entity-unit binding the realizer can reach
# without re-running the solver. This is sufficient because the
# substrate refuses multi-unit reference actors at solve time.
entity_units: dict[str, str] = {
p.entity: p.quantity.unit for p in graph_initial_state
}
step_sentences: list[str] = []
for step in trace.steps:
step_sentences.append(_step_sentence(step))
step_sentences.append(_step_sentence(step, entity_units))
answer_sentence = _answer_sentence(
trace.answer_entity, trace.answer_value, trace.answer_unit
@ -124,9 +134,20 @@ def _setup_sentence(entity: str, value: int | float, unit: str) -> str:
return f"{entity} has {_render_number(value)} {_unit_surface(unit, value)}."
def _step_sentence(step: SolutionStep) -> str:
def _step_sentence(
step: SolutionStep, entity_units: dict[str, str] | None = None
) -> str:
if step.operation_kind == "apply_rate":
return _apply_rate_sentence(step)
if step.operation_kind == "compare_additive":
return _compare_additive_sentence(step)
if step.operation_kind == "compare_multiplicative":
if entity_units is None:
raise RealizerError(
f"compare_multiplicative step {step.step_index} requires "
f"entity_units to resolve reference actor unit; got None"
)
return _compare_multiplicative_sentence(step, entity_units)
if step.operation_kind == "add":
return (
f"{step.actor} buys {_render_number(step.operand.value)} more "
@ -206,6 +227,154 @@ def _apply_rate_sentence(step: SolutionStep) -> str:
)
def _compare_additive_sentence(step: SolutionStep) -> str:
"""Render an additive comparison step as show-your-work prose (ADR-0123).
Reads ``step.operand`` (must be a :class:`Comparison` with
``delta`` set) and emits a one-sentence rendering of the form:
- direction='more': "<actor> has <delta> more <unit> than <ref>,
giving <actor> a total of <after> <unit>."
- direction='fewer': "<actor> has <delta> fewer <unit> than <ref>,
leaving <actor> with a total of <after> <unit>."
The two-clause shape *comparison clause* + *resolved state*
is pinned as a structural invariant by the ADR-0123 test suite.
``delta.value`` and ``after_value`` pluralize independently via
:func:`_unit_surface` (so "1 more apple" vs "3 more apples"
behave correctly without the resolved state being forced into
the same plurality).
Raises :class:`RealizerError` on:
- operand not a :class:`Comparison` (substrate solver bug)
- missing ``delta`` (multiplicative shape leaked into this branch)
- direction not in ``{"more", "fewer"}``
- self-reference (actor == reference_actor)
"""
if not isinstance(step.operand, Comparison):
raise RealizerError(
f"compare_additive step {step.step_index} requires a "
f"Comparison operand; got {type(step.operand).__name__}"
)
cmp = step.operand
if cmp.delta is None:
raise RealizerError(
f"compare_additive step {step.step_index} requires "
f"Comparison.delta; got None (multiplicative shape leaked)"
)
if cmp.direction not in ("more", "fewer"):
raise RealizerError(
f"compare_additive step {step.step_index} requires "
f"direction in {{'more','fewer'}}; got {cmp.direction!r}"
)
if step.actor == cmp.reference_actor:
raise RealizerError(
f"compare_additive step {step.step_index} refuses "
f"self-comparison: actor=={cmp.reference_actor!r}"
)
delta_n = _render_number(cmp.delta.value)
after_n = _render_number(step.after_value)
delta_surface = _unit_surface(cmp.delta.unit, cmp.delta.value)
after_surface = _unit_surface(cmp.delta.unit, step.after_value)
if cmp.direction == "more":
return (
f"{step.actor} has {delta_n} more {delta_surface} than "
f"{cmp.reference_actor}, giving {step.actor} a total of "
f"{after_n} {after_surface}."
)
# direction == "fewer"
return (
f"{step.actor} has {delta_n} fewer {delta_surface} than "
f"{cmp.reference_actor}, leaving {step.actor} with a total of "
f"{after_n} {after_surface}."
)
def _compare_multiplicative_sentence(
step: SolutionStep, entity_units: dict[str, str]
) -> str:
"""Render a multiplicative comparison step as show-your-work prose.
Reads ``step.operand`` (must be a :class:`Comparison` with
``factor`` set) and emits:
- direction='times': "<actor> has <factor> times as many <unit>
as <ref>, giving <actor> a total of <after>
<unit>."
- direction='fraction', factor==0.5:
"<actor> has half as many <unit> as <ref>,
giving <actor> a total of <after> <unit>."
- direction='fraction', other factor:
"<actor> has <factor> as many <unit> as
<ref>, giving <actor> a total of <after>
<unit>."
``unit`` is resolved from ``entity_units[reference_actor]`` the
substrate's solver derives it from the reference actor's
in-flight state, but the realizer only sees ``SolutionStep``
instances. Initial-state lookup is sufficient because the
substrate refuses multi-unit reference actors and refuses to
overwrite a comparison actor's existing state.
Raises :class:`RealizerError` on:
- operand not a :class:`Comparison`
- missing ``factor``
- direction not in ``{"times", "fraction"}``
- reference actor missing from ``entity_units``
- self-reference
"""
if not isinstance(step.operand, Comparison):
raise RealizerError(
f"compare_multiplicative step {step.step_index} requires "
f"a Comparison operand; got {type(step.operand).__name__}"
)
cmp = step.operand
if cmp.factor is None:
raise RealizerError(
f"compare_multiplicative step {step.step_index} requires "
f"Comparison.factor; got None (additive shape leaked)"
)
if cmp.direction not in ("times", "fraction"):
raise RealizerError(
f"compare_multiplicative step {step.step_index} requires "
f"direction in {{'times','fraction'}}; got {cmp.direction!r}"
)
if step.actor == cmp.reference_actor:
raise RealizerError(
f"compare_multiplicative step {step.step_index} refuses "
f"self-comparison: actor=={cmp.reference_actor!r}"
)
if cmp.reference_actor not in entity_units:
raise RealizerError(
f"compare_multiplicative step {step.step_index} requires "
f"reference actor {cmp.reference_actor!r} to appear in "
f"initial state; available entities: "
f"{sorted(entity_units)!r}"
)
unit = entity_units[cmp.reference_actor]
after_n = _render_number(step.after_value)
after_surface = _unit_surface(unit, step.after_value)
if cmp.direction == "fraction" and cmp.factor == 0.5:
return (
f"{step.actor} has half as many {unit} as "
f"{cmp.reference_actor}, giving {step.actor} a total of "
f"{after_n} {after_surface}."
)
factor_n = _render_number(cmp.factor)
if cmp.direction == "fraction":
return (
f"{step.actor} has {factor_n} as many {unit} as "
f"{cmp.reference_actor}, giving {step.actor} a total of "
f"{after_n} {after_surface}."
)
# direction == "times"
return (
f"{step.actor} has {factor_n} times as many {unit} as "
f"{cmp.reference_actor}, giving {step.actor} a total of "
f"{after_n} {after_surface}."
)
def _answer_sentence(
entity: str | None, value: int | float, unit: str
) -> str:

View file

@ -0,0 +1,486 @@
"""ADR-0123 — comparison-phrasing realizer (surface increment on substrate).
Pins the load-bearing invariants documented in
``docs/decisions/ADR-0123-parser-comparison-phrasing.md``. The
substrate (PR #155, commit ``c9bd5d4``) shipped the typed graph
operand (``Comparison``), the two new operation kinds, the parser
patterns, the solver/verifier wiring, and the two pack lemmas; this
PR adds the **realizer surface** that turns successful comparison
traces into show-your-work prose. Before this PR, a problem the
substrate solved successfully crashed at ``realize()`` with
``RealizerError("unknown operation_kind 'compare_additive'")``.
The wrong-zero discipline (ADR-0114a Obligation #4) is the
load-bearing positive claim: the new realizer branches only fire
when the substrate has already emitted a successful step. If the
substrate refused, no step exists to render there is no path by
which the realizer can introduce a misparse.
Tests are organized to mirror ADR-0118's stepped-realizer test
shape and the substrate ADR's invariant numbering:
1. End-to-end canonical renderings for each of the four surfaces
(``N more``, ``N fewer``, ``twice``/``N times``, ``half``).
2. Singular/plural independence at the surface layer.
3. Byte-determinism on repeated invocations.
4. Refusal discipline (operand-shape, direction, self-reference,
missing-reference defenses).
5. Backwards-compatibility every pre-comparison realizer surface
re-renders byte-identically.
6. Sealed-holdout invariants (skipped without ``CORE_HOLDOUT_KEY``
per ADR-0119.7).
"""
from __future__ import annotations
import hashlib
import json
import os
from pathlib import Path
import pytest
_REPO_ROOT = Path(__file__).resolve().parent.parent
_SEALED_PATH = (
_REPO_ROOT / "evals" / "gsm8k_math" / "holdouts" / "v1" / "cases.jsonl.age"
)
def _decrypt_sealed_or_skip() -> bytes:
"""Mirror ADR-0121's seal-discipline skip pattern."""
key_path_str = os.environ.get("CORE_HOLDOUT_KEY")
if not key_path_str:
pytest.skip("CORE_HOLDOUT_KEY not set; per ADR-0119.7 seal discipline")
try:
import pyrage
from pyrage.x25519 import Identity
except ImportError:
pytest.skip("pyrage not installed")
key_path = Path(key_path_str)
if not key_path.exists():
pytest.skip(f"CORE_HOLDOUT_KEY={key_path} does not exist")
identity = Identity.from_str(key_path.read_text(encoding="utf-8").strip())
return pyrage.decrypt(_SEALED_PATH.read_bytes(), [identity])
def _solve_and_realize(text: str) -> str:
"""Helper: full pipeline → realized prose string."""
from generate.math_parser import parse_problem
from generate.math_realizer import realize
from generate.math_solver import solve
g = parse_problem(text)
return realize(g.initial_state, solve(g)).as_prose()
# ---------------------------------------------------------------------------
# End-to-end canonical renderings (ADR-0123 invariants 1-5)
# ---------------------------------------------------------------------------
class TestCompareAdditiveMoreCanonical:
"""ADR-0123 invariant ``adr_0123_realize_compare_additive_more_canonical``."""
PROBLEM = (
"Alice has 5 apples. Bob has 3 more apples than Alice. "
"How many apples does Bob have?"
)
def test_prose_contains_comparison_and_state_clauses(self) -> None:
prose = _solve_and_realize(self.PROBLEM)
assert "3 more apples than Alice" in prose
assert "Bob a total of 8 apples" in prose
def test_answer_sentence_present(self) -> None:
prose = _solve_and_realize(self.PROBLEM)
# ADR-0118's _answer_sentence template
assert "Bob has 8 apples." in prose
def test_setup_sentence_present(self) -> None:
prose = _solve_and_realize(self.PROBLEM)
assert "Alice has 5 apples." in prose
class TestCompareAdditiveFewerCanonical:
"""ADR-0123 invariant ``adr_0123_realize_compare_additive_fewer_canonical``."""
PROBLEM = (
"Anna has 10 flowers. Mary has 5 fewer flowers than Anna. "
"How many flowers does Mary have?"
)
def test_prose_contains_fewer_clause(self) -> None:
prose = _solve_and_realize(self.PROBLEM)
assert "5 fewer flowers than Anna" in prose
def test_prose_contains_leaving_state(self) -> None:
# The 'fewer' branch uses 'leaving … with a total of …'
# to read naturally with subtraction semantics.
prose = _solve_and_realize(self.PROBLEM)
assert "leaving Mary with a total of 5 flowers" in prose
def test_answer_sentence_present(self) -> None:
prose = _solve_and_realize(self.PROBLEM)
assert "Mary has 5 flowers." in prose
class TestCompareMultiplicativeTwiceCanonical:
"""ADR-0123 invariant
``adr_0123_realize_compare_multiplicative_twice_canonical``."""
PROBLEM = (
"Carla has 7 marbles. Ben has twice as many marbles as Carla. "
"How many marbles does Ben have?"
)
def test_prose_normalizes_twice_to_n_times(self) -> None:
# The parser maps 'twice' to factor=2; the realizer renders
# the numeric form. This is deliberate — round-trip drift on
# 'twice' vs '2 times' is acceptable; the underlying graph is
# the source of truth.
prose = _solve_and_realize(self.PROBLEM)
assert "2 times as many marbles as Carla" in prose
def test_prose_contains_state_clause(self) -> None:
prose = _solve_and_realize(self.PROBLEM)
assert "Ben a total of 14 marbles" in prose
class TestCompareMultiplicativeNTimesCanonical:
"""ADR-0123 invariant
``adr_0123_realize_compare_multiplicative_n_times_canonical``."""
PROBLEM = (
"Tom has 3 cookies. Sara has 4 times as many cookies as Tom. "
"How many cookies does Sara have?"
)
def test_prose_renders_n_times(self) -> None:
prose = _solve_and_realize(self.PROBLEM)
assert "4 times as many cookies as Tom" in prose
assert "Sara a total of 12 cookies" in prose
class TestCompareFractionHalfCanonical:
"""ADR-0123 invariant ``adr_0123_realize_compare_fraction_half_canonical``."""
PROBLEM = (
"Tom has 8 cookies. Lisa has half as many cookies as Tom. "
"How many cookies does Lisa have?"
)
def test_prose_uses_literal_half_word(self) -> None:
# factor==0.5 + direction=='fraction' renders "half as many"
# rather than "0.5 as many" — the natural English form.
prose = _solve_and_realize(self.PROBLEM)
assert "half as many cookies as Tom" in prose
assert "0.5 as many" not in prose
def test_prose_contains_state_clause(self) -> None:
prose = _solve_and_realize(self.PROBLEM)
assert "Lisa a total of 4 cookies" in prose
# ---------------------------------------------------------------------------
# Byte-determinism + plurality independence (ADR-0123 invariants 6-7)
# ---------------------------------------------------------------------------
class TestRealizerByteDeterministic:
"""ADR-0123 invariant ``adr_0123_realize_byte_deterministic``."""
@pytest.mark.parametrize(
"problem",
[
"Alice has 5 apples. Bob has 3 more apples than Alice. How many apples does Bob have?",
"Anna has 10 flowers. Mary has 5 fewer flowers than Anna. How many flowers does Mary have?",
"Carla has 7 marbles. Ben has twice as many marbles as Carla. How many marbles does Ben have?",
"Tom has 8 cookies. Lisa has half as many cookies as Tom. How many cookies does Lisa have?",
],
)
def test_realize_twice_produces_byte_equal_output(
self, problem: str
) -> None:
from generate.math_parser import parse_problem
from generate.math_realizer import realize
from generate.math_solver import solve
g = parse_problem(problem)
t = solve(g)
r1 = realize(g.initial_state, t)
r2 = realize(g.initial_state, t)
assert r1.canonical_bytes() == r2.canonical_bytes()
assert r1.as_prose() == r2.as_prose()
class TestSingularPluralIndependence:
"""ADR-0123 invariant ``adr_0123_realize_singular_plural_independence``.
The comparison clause and the resolved-state clause pluralize
independently a delta of 1 takes the singular for the delta
surface, but the after_value drives the resolved-state surface
on its own count.
"""
def test_singular_delta_with_plural_after(self) -> None:
prose = _solve_and_realize(
"Tom has 4 apples. Sue has 1 more apple than Tom. "
"How many apples does Sue have?"
)
assert "1 more apple than Tom" in prose
assert "1 more apples than Tom" not in prose
# Resolved state pluralizes on its own count (5)
assert "5 apples" in prose
def test_plural_delta_with_singular_after(self) -> None:
# Fewer that leaves exactly 1 unit.
prose = _solve_and_realize(
"Anna has 4 flowers. Mary has 3 fewer flowers than Anna. "
"How many flowers does Mary have?"
)
assert "3 fewer flowers than Anna" in prose
# Resolved state singular on its own count (1)
assert "a total of 1 flower" in prose
# ---------------------------------------------------------------------------
# Refusal discipline (ADR-0123 invariants 8-11)
# ---------------------------------------------------------------------------
def _make_compare_step(
*,
actor: str = "Alice",
direction: str = "more",
delta=None,
factor=None,
reference: str = "Bob",
operand=None,
after: float = 8.0,
index: int = 0,
operation_kind: str = "compare_additive",
):
"""Build a SolutionStep with a Comparison operand (or override)."""
from generate.math_problem_graph import Comparison, Quantity
from generate.math_solver import SolutionStep
if operand is None:
operand = Comparison(
reference_actor=reference,
delta=delta,
factor=factor,
direction=direction, # type: ignore[arg-type]
)
return SolutionStep(
step_index=index,
operation_kind=operation_kind,
pack_lemma_id=f"en_arithmetic_v1:{operation_kind}",
actor=actor,
operand=operand,
target=None,
before_value=0.0,
after_value=after,
target_before=None,
target_after=None,
)
class TestRealizerRefusalDiscipline:
"""ADR-0123 invariants 8-11 — operand and direction shape refusals.
Several realizer defenses (missing-delta, missing-factor,
direction mismatch with operand shape) are *unreachable* via
ordinary dataclass construction because :class:`Comparison`'s
``__post_init__`` refuses those shapes first. The realizer's
code retains them as belt-and-suspenders for hand-bypassed
constructions (``object.__setattr__`` on the frozen instance);
we do not exercise them here because the substrate boundary is
the load-bearing guarantee. See
``test_pack_grounded_comparison.py`` and the substrate ADR's
own test suite for the ``Comparison.__post_init__`` refusals.
"""
def test_refuses_non_comparison_operand_on_additive(self) -> None:
from generate.math_problem_graph import Quantity
from generate.math_realizer import RealizerError, _compare_additive_sentence
step = _make_compare_step(operand=Quantity(value=3, unit="apples"))
with pytest.raises(RealizerError, match="requires a Comparison operand"):
_compare_additive_sentence(step)
def test_refuses_self_comparison_additive(self) -> None:
from generate.math_problem_graph import Quantity
from generate.math_realizer import RealizerError, _compare_additive_sentence
step = _make_compare_step(
actor="Alice",
reference="Alice",
delta=Quantity(value=3, unit="apples"),
)
with pytest.raises(RealizerError, match="self-comparison"):
_compare_additive_sentence(step)
def test_refuses_self_comparison_multiplicative(self) -> None:
from generate.math_realizer import (
RealizerError,
_compare_multiplicative_sentence,
)
step = _make_compare_step(
actor="Alice",
reference="Alice",
direction="times",
factor=2.0,
operation_kind="compare_multiplicative",
)
with pytest.raises(RealizerError, match="self-comparison"):
_compare_multiplicative_sentence(step, {"Alice": "apples"})
def test_refuses_missing_entity_units_on_multiplicative(self) -> None:
from generate.math_realizer import (
RealizerError,
_compare_multiplicative_sentence,
)
step = _make_compare_step(
direction="times",
factor=2.0,
reference="UnknownActor",
operation_kind="compare_multiplicative",
)
with pytest.raises(RealizerError, match="initial state"):
_compare_multiplicative_sentence(step, {"Alice": "apples"})
def test_step_sentence_requires_entity_units_for_multiplicative(self) -> None:
# If a caller bypasses realize() and invokes _step_sentence
# directly without providing entity_units, the multiplicative
# branch must refuse rather than silently render None as
# the unit.
from generate.math_realizer import RealizerError, _step_sentence
step = _make_compare_step(
direction="times",
factor=2.0,
operation_kind="compare_multiplicative",
)
with pytest.raises(RealizerError, match="entity_units"):
_step_sentence(step, None)
# ---------------------------------------------------------------------------
# Backwards-compatibility — ADR-0118 templates must re-render identically
# ---------------------------------------------------------------------------
class TestADR0118StepRealizerUnchanged:
"""ADR-0123 invariant ``adr_0123_adr_0118_stepped_realizer_unchanged``.
Every pre-comparison operation kind must re-render byte-
identically to its rendering on the substrate branch. The
realizer change is purely additive at the dispatch layer; it
must not alter the prior templates.
"""
def test_add_step_renders_unchanged(self) -> None:
# 'X has N. X buys M more.' → uses ADR-0118 add template.
prose = _solve_and_realize(
"Sarah has 3 apples. Sarah buys 4 more apples. "
"How many apples does Sarah have?"
)
# ADR-0118 pinned phrasing: "buys N more units, raising the
# total to M"
assert "buys 4 more apples" in prose
assert "raising the total to 7" in prose
def test_subtract_step_renders_unchanged(self) -> None:
prose = _solve_and_realize(
"Sarah has 10 apples. Sarah loses 3 apples. "
"How many apples does Sarah have?"
)
assert "loses 3 apples" in prose
assert "leaving 7" in prose
def test_setup_and_answer_sentences_unchanged(self) -> None:
prose = _solve_and_realize(
"Sarah has 3 apples. Sarah buys 4 more apples. "
"How many apples does Sarah have?"
)
# Setup
assert prose.startswith("Sarah has 3 apples.")
# Answer (ADR-0118 template)
assert prose.endswith("Sarah has 7 apples.")
# ---------------------------------------------------------------------------
# Sealed-holdout invariants — CORE_HOLDOUT_KEY required
# ---------------------------------------------------------------------------
class TestSealedHoldoutMeasurement:
"""ADR-0123 invariants
``adr_0123_sealed_correct_rate_zero_at_landing`` and
``adr_0123_sealed_wrong_zero_holds``.
Both are inherited from the substrate ADR: the realizer surface
cannot create matches the parser refuses, so the multi-
construction barrier holds at the surface layer too. The
realizer also cannot misparse it only renders successful
traces so ``wrong == 0`` holds by construction.
"""
def test_sealed_correct_rate_zero_at_landing(self) -> None:
from evals.gsm8k_math.runner import run_lane
plaintext = _decrypt_sealed_or_skip()
cases = [
json.loads(line)
for line in plaintext.decode("utf-8").splitlines()
if line.strip()
]
report = run_lane(cases)
rate = report.metrics["correct_rate"]
assert rate == 0.0, (
f"sealed-holdout correct_rate={rate}; ADR-0123 surface "
f"shipped with the lift gate explicitly deferred. A "
f"non-zero rate here means a future composition ADR has "
f"unlocked lifts — supersede ADR-0123 with a "
f"successful-lift ADR and update this test to the "
f"strict-lift form."
)
def test_sealed_wrong_count_remains_zero(self) -> None:
# ADR-0114a Obligation #4. The realizer surface cannot
# introduce a misparse because it only renders successful
# traces; for this to fail, the substrate's solver would
# have to confabulate, which the substrate's wrong-zero
# test already pins.
from evals.gsm8k_math.runner import run_lane
plaintext = _decrypt_sealed_or_skip()
cases = [
json.loads(line)
for line in plaintext.decode("utf-8").splitlines()
if line.strip()
]
report = run_lane(cases)
assert report.metrics["wrong"] == 0, (
f"sealed-holdout wrong count = {report.metrics['wrong']}; "
f"ADR-0114a Obligation #4 requires wrong==0."
)
class TestSealedSealIntegrity:
"""ADR-0123 — sealed seal byte-equal across this PR."""
def test_seal_unchanged(self) -> None:
if not _SEALED_PATH.exists():
pytest.skip(f"sealed holdout not present at {_SEALED_PATH}")
actual = hashlib.sha256(_SEALED_PATH.read_bytes()).hexdigest()
size = _SEALED_PATH.stat().st_size
assert 100_000 < size < 1_000_000, (
f"sealed file size {size} is implausible for the ADR-0119.7 "
f"GSM8K seal (~420kb expected)"
)
actual2 = hashlib.sha256(_SEALED_PATH.read_bytes()).hexdigest()
assert actual == actual2