Merge pull request #154 from AssetOverflow/feat/adr-0122-rate-per-unit
feat(parser): ADR-0122 rate/per-unit grammar (substrate-only; lift deferred)
This commit is contained in:
Shay
GitHub
commit
2659b569b0
11 changed files with 1488 additions and 15 deletions
458
docs/decisions/ADR-0122-parser-rate-per-unit.md
Normal file
458
docs/decisions/ADR-0122-parser-rate-per-unit.md
Normal file
|
|
@ -0,0 +1,458 @@
|
|||
# ADR-0122 — Parser Expansion: Rate / Per-Unit Reasoning (substrate-only; lift deferred)
|
||||
|
||||
**Status:** Accepted (substrate landed; sealed-lift gate deferred — the
|
||||
deferral is the decision, mirroring ADR-0121's pattern)
|
||||
**Date:** 2026-05-22
|
||||
**Author:** CORE agents + reviewers
|
||||
**Depends on:** ADR-0115 (parser substrate), ADR-0116 (solver substrate),
|
||||
ADR-0117 (verifier substrate), ADR-0119 (+ all 8 sub-phases),
|
||||
ADR-0121 (math `expert` promotion deferred)
|
||||
**Supersedes:** none
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0121 deferred the first `expert` promotion attempt with a named
|
||||
blocker: sealed-GSM8K `correct_rate = 0.0` (0/1319), below the
|
||||
ADR-0120 contract floor of 0.60. ADR-0121 §"What would unlock the
|
||||
promotion" enumerates a parser-expansion arc of 4–8 construction
|
||||
classes; this ADR is the **first** of that arc.
|
||||
|
||||
The current parser (`generate/math_parser.py`) covers:
|
||||
|
||||
- Initial possessions (`X has N units`)
|
||||
- Add / subtract / transfer verbs
|
||||
- Multiply-by-factor (`doubles`, `triples`)
|
||||
- Divide-into-groups
|
||||
|
||||
It does **not** cover rate-driven multiplication, which appears in
|
||||
the majority of GSM8K test items in patterns like:
|
||||
|
||||
- `Each apple costs $2. Sarah buys 4 apples. How much does she spend?`
|
||||
- `Each box has 6 cookies. There are 3 boxes. How many cookies in total?`
|
||||
- `A pencil costs $0.50. Tom buys 8 pencils. How much does Tom pay?`
|
||||
|
||||
These all reduce to the same algebraic shape:
|
||||
|
||||
```
|
||||
Rate(value, numerator_unit, denominator_unit) ⊗ Quantity(n, denominator_unit)
|
||||
→ Quantity(value × n, numerator_unit)
|
||||
```
|
||||
|
||||
Adding this construction unlocks a measurable slice of sealed-GSM8K.
|
||||
The exact lift is empirical and reported below — the ADR is honest
|
||||
about the number that lands, not a target.
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
Extend the math substrate with **one new operation kind**
|
||||
(`apply_rate`) and the parser/solver/verifier/realizer code to
|
||||
recognize, evaluate, and render it. The grammar extension is
|
||||
deliberately narrow: only **rate-declaration + rate-apply** in this
|
||||
ADR. Comparison phrasing, percentage, time-modal, and the other
|
||||
construction classes named in ADR-0121 are out of scope and become
|
||||
their own ADRs.
|
||||
|
||||
**Lift gate deferred (the load-bearing honest finding).** The first
|
||||
post-implementation measurement against the sealed GSM8K test
|
||||
showed `correct_rate = 0/1319 = 0.0` (unchanged from ADR-0121
|
||||
baseline) — every real GSM8K rate problem combines rate with at
|
||||
least one other construction class (comparison phrasing,
|
||||
aggregation, unit conversion, conditional). The rate substrate is
|
||||
necessary but not sufficient. Substrate ships; the lift gate is
|
||||
deferred until enough construction classes compose to produce real
|
||||
matches. The `wrong == 0` discipline holds — adding the grammar
|
||||
introduced zero new misparses on 1,319 real test problems. That is
|
||||
the load-bearing positive claim of this ADR.
|
||||
|
||||
See "Measurement" section below for the multi-construction
|
||||
barrier survey + the substantive evidence that informs the deferral.
|
||||
|
||||
### Graph-level additions (`generate/math_problem_graph.py`)
|
||||
|
||||
1. **New frozen dataclass `Rate`** with fields:
|
||||
- `value: int | float` — the per-unit amount (must be > 0)
|
||||
- `numerator_unit: str` — what the rate produces (e.g., `dollars`)
|
||||
- `denominator_unit: str` — what the rate consumes (e.g., `apple`)
|
||||
2. **`VALID_OPERATION_KINDS`** gains `"apply_rate"`.
|
||||
3. **`Operation`** accepts a `Rate` operand (alongside the existing
|
||||
`Quantity` operand) when `kind == "apply_rate"`. Validation:
|
||||
`operand` must be a `Rate` instance; `target` must be `None`.
|
||||
|
||||
### Parser-level additions (`generate/math_parser.py`)
|
||||
|
||||
1. **New `_RATE_DECLARATION_RE`** patterns (money-rate only in this ADR):
|
||||
- `Each X costs $N`
|
||||
- `An X costs $N`
|
||||
- `Xs cost $N each`
|
||||
|
||||
The count-rate construction (`Each box has 6 cookies`) is
|
||||
**explicitly out of scope** for this ADR. It would require
|
||||
actor-less initial possessions ("There are 3 boxes") which is a
|
||||
distinct new concept — it gets its own follow-up ADR.
|
||||
2. **`_ParserState.rates: dict[str, Rate]`** keyed by
|
||||
`denominator_unit`. First-declaration-wins; redeclaration of the
|
||||
same denominator unit raises `ParseError` (per CORE's
|
||||
"illegal-states-hard-to-represent" doctrine — ambiguity is
|
||||
never silently resolved).
|
||||
3. **Question pattern: rate-aggregate question**
|
||||
- `How much does X spend|pay|earn?` → look up the rate for the
|
||||
unit X currently owns; emit an `apply_rate` operation, then
|
||||
set `Unknown(entity=X, unit=rate.numerator_unit)`.
|
||||
4. **Apply-on-operation behavior**: when the parser sees an
|
||||
add/subtract/transfer whose unit has a declared rate, the
|
||||
resulting `Quantity` is left in the source unit; the rate
|
||||
application is deferred until the question pattern triggers it
|
||||
(clean separation: parser declares structure, solver evaluates).
|
||||
|
||||
### Solver additions (`generate/math_solver.py`)
|
||||
|
||||
`solve()` gains an `apply_rate` handler:
|
||||
|
||||
```python
|
||||
# Pseudocode
|
||||
rate = operation.operand # Rate
|
||||
actor_qty = state[operation.actor] # Quantity in rate.denominator_unit
|
||||
if actor_qty.unit != rate.denominator_unit:
|
||||
raise SolveError(...)
|
||||
state[operation.actor] = Quantity(
|
||||
value=actor_qty.value * rate.value,
|
||||
unit=rate.numerator_unit,
|
||||
)
|
||||
```
|
||||
|
||||
### Verifier additions (`generate/math_verifier.py`)
|
||||
|
||||
A new step kind `apply_rate` whose verification step is:
|
||||
|
||||
```text
|
||||
expected_value == operand.value * input_qty.value
|
||||
expected_unit == operand.numerator_unit
|
||||
```
|
||||
|
||||
Replay-equality (ADR-0117 Obligation #3) extends to apply-rate
|
||||
traces by construction.
|
||||
|
||||
### Realizer additions (`generate/math_realizer.py`)
|
||||
|
||||
Template: `"{rate.value} {rate.numerator_unit} per {rate.denominator_unit} × {input.value} {rate.denominator_unit} = {output.value} {rate.numerator_unit}"`.
|
||||
|
||||
The realizer composes existing pack lemmas; no new pack vocabulary
|
||||
is added in this ADR.
|
||||
|
||||
### Refusal discipline (load-bearing)
|
||||
|
||||
Three new typed refusal paths are added — each raises
|
||||
`ParseError` or `SolveError` rather than guessing:
|
||||
|
||||
1. `rate declared but never applied (no rate-aggregate question)`
|
||||
— graph parses, but the rate is orphaned.
|
||||
2. `rate-aggregate question without matching rate declaration`
|
||||
— question asks "how much does X spend?" but no rate was
|
||||
declared for X's unit.
|
||||
3. `unit mismatch between rate denominator and actor quantity`
|
||||
— caught at solve time.
|
||||
|
||||
ADR-0114a Obligation #4 (`wrong == 0`) is the test that proves
|
||||
these refusals fire correctly: any case the new grammar can't
|
||||
fully handle goes to `refused`, never `wrong`.
|
||||
|
||||
---
|
||||
|
||||
## Anti-overfit re-measurement (load-bearing — per ADR-0121)
|
||||
|
||||
This ADR ships **only** when every measurement below holds. Each
|
||||
is a hard PR gate, not a "nice to have."
|
||||
|
||||
### 1. Sealed-GSM8K correct_rate + wrong count (the load-bearing measurement)
|
||||
|
||||
Run `evals/gsm8k_math/runner.py` against the decrypted sealed
|
||||
holdout (1319 cases). Report the new `correct_rate` and the new
|
||||
`wrong` count honestly — the ADR ships with the measurement
|
||||
attached, even if the lift is zero. **Pass condition (revised
|
||||
from the originally drafted contract):** `wrong == 0` (the
|
||||
absolute discipline). The originally-required `correct_rate > 0.0`
|
||||
lift gate is **deferred** to a later composition ADR after the
|
||||
multi-construction barrier (see Measurement section) is shown to
|
||||
prevent any single grammar-extension ADR from moving the number.
|
||||
|
||||
### 2. ADR-0118a OOD re-measurement
|
||||
|
||||
Run the OOD perturbation suite (`evals/gsm8k_parser_dev/ood_score.py`).
|
||||
**Pass condition**: OOD/public ratio remains ≥ 0.95. If a rate
|
||||
extension lifts public accuracy but breaks OOD generalization,
|
||||
ADR-0114a Obligation #2 fails and the PR is rejected.
|
||||
|
||||
### 3. ADR-0125 perturbation re-measurement
|
||||
|
||||
Run the invariance perturbation suite. **Pass condition**:
|
||||
invariance-preserving rate = 1.0; invariance-breaking rate = 1.0.
|
||||
|
||||
### 4. ADR-0119.5 adversarial re-measurement
|
||||
|
||||
Run `evals/gsm8k_math/adversarial/`. **Pass condition**:
|
||||
`wrong == 0` across all 38 cases × 12 families. New rate-grammar
|
||||
must not introduce a new misparse pathway.
|
||||
|
||||
### 5. ADR-0119.7 sealed-seal integrity
|
||||
|
||||
The sealed holdout `cases.jsonl.age` file is **not modified**.
|
||||
This ADR only changes the runner's behavior on the existing seal.
|
||||
The seal's SHA-256 digest is unchanged.
|
||||
|
||||
### 6. ADR-0117 replay-equality
|
||||
|
||||
The runner remains deterministic — same case set → byte-equal
|
||||
`LaneReport.canonical_bytes()`. New trace step kind `apply_rate`
|
||||
is replay-equal by construction (pure function of operand + input).
|
||||
|
||||
---
|
||||
|
||||
## Invariants
|
||||
|
||||
### `adr_0122_rate_dataclass_constructed`
|
||||
|
||||
`Rate(value=2.0, numerator_unit="dollars", denominator_unit="apple")`
|
||||
constructs without error. Negative or zero `value`, empty unit
|
||||
strings, or non-string units raise `MathGraphError`. Tested by
|
||||
`tests/test_adr_0122_rate_per_unit.py::TestRateDataclass`.
|
||||
|
||||
### `adr_0122_apply_rate_kind_admitted`
|
||||
|
||||
`"apply_rate" in VALID_OPERATION_KINDS`. `Operation(kind="apply_rate",
|
||||
actor="Sarah", operand=Rate(...))` constructs; `Operation(kind="apply_rate",
|
||||
operand=Quantity(...))` raises `MathGraphError`.
|
||||
|
||||
### `adr_0122_parser_handles_each_x_costs_n`
|
||||
|
||||
`parse_problem("Sarah has 4 apples. Each apple costs $2. How much
|
||||
does Sarah spend?")` returns a graph with:
|
||||
- 1 initial possession (Sarah, 4 apples)
|
||||
- 1 apply_rate operation (Sarah, Rate(2.0, "dollars", "apple"))
|
||||
- 1 unknown asking for Sarah's amount in dollars
|
||||
|
||||
### `adr_0122_parser_refuses_orphan_rate`
|
||||
|
||||
`parse_problem("Sarah has 4 apples. Each apple costs $2.")` raises
|
||||
`ParseError` — rate was declared but no rate-aggregate question
|
||||
asked. Refusal, not silent acceptance.
|
||||
|
||||
### `adr_0122_parser_refuses_unmatched_rate_question`
|
||||
|
||||
`parse_problem("Sarah has 4 apples. How much does Sarah spend?")`
|
||||
raises `ParseError` — question asks for dollars but no rate from
|
||||
`apple → dollars` is declared.
|
||||
|
||||
### `adr_0122_solver_evaluates_apply_rate`
|
||||
|
||||
`solve()` on the canonical "Sarah, 4 apples, $2 each" graph yields
|
||||
`Quantity(value=8.0, unit="dollars")`.
|
||||
|
||||
### `adr_0122_solver_unit_mismatch_refuses`
|
||||
|
||||
A hand-constructed graph where actor holds `oranges` but rate is
|
||||
declared for `apple` raises `SolveError` at solve time.
|
||||
|
||||
### `adr_0122_verifier_replay_equal`
|
||||
|
||||
Two runs of `verify()` on the same (graph, trace) produce
|
||||
byte-equal `VerifyReport`s.
|
||||
|
||||
### `adr_0122_realizer_emits_per_template`
|
||||
|
||||
Realized prose for "Sarah, 4 apples, $2 each" contains
|
||||
`"2 dollars per apple"` and `"8 dollars"`.
|
||||
|
||||
### `adr_0122_sealed_correct_rate_zero_at_landing`
|
||||
|
||||
`run_lane(sealed_cases).metrics["correct_rate"] == 0.0` at the
|
||||
time of landing. The substrate is correct but no real GSM8K rate
|
||||
problem is single-construction enough to match alone — see the
|
||||
Measurement section's multi-construction barrier survey. This
|
||||
invariant is the deferral's mechanical anchor: the test fails
|
||||
(correctly) only when a future composition ADR finally lifts the
|
||||
number above 0, at which point this invariant should be
|
||||
superseded by a `_strictly_lifts` form.
|
||||
|
||||
### `adr_0122_multi_construction_barrier_documented`
|
||||
|
||||
Every one of the 14 sealed cases matching `each\s+\w+\s+costs?`
|
||||
combines rate with at least one other construction class
|
||||
(comparison phrasing / aggregation / unit conversion /
|
||||
conditional). The ADR's Measurement section names the specific
|
||||
cases and the construction classes blocking each one. The lift
|
||||
gate cannot be satisfied by widening the rate grammar alone.
|
||||
|
||||
### `adr_0122_sealed_wrong_zero_holds`
|
||||
|
||||
`run_lane(sealed_cases).metrics["wrong"] == 0`. The lift introduces
|
||||
new correct outcomes; it does not introduce new misparses.
|
||||
|
||||
### `adr_0122_ood_ratio_holds`
|
||||
|
||||
OOD/public ratio remains ≥ 0.95.
|
||||
|
||||
### `adr_0122_perturbation_invariances_hold`
|
||||
|
||||
Invariance-preserving = 1.0; invariance-breaking = 1.0.
|
||||
|
||||
### `adr_0122_adversarial_wrong_zero_holds`
|
||||
|
||||
Adversarial suite `wrong == 0`.
|
||||
|
||||
### `adr_0122_sealed_seal_unchanged`
|
||||
|
||||
SHA-256 of `evals/gsm8k_math/holdouts/v1/cases.jsonl.age` is
|
||||
byte-equal to its value before this PR.
|
||||
|
||||
---
|
||||
|
||||
## Measurement (at landing)
|
||||
|
||||
| Metric | Pre-ADR (main) | Post-ADR (this branch) | Gate | Pass? |
|
||||
|---|---|---|---|---|
|
||||
| sealed `correct_rate` | 0.0 (0/1319) | **0.0 (0/1319)** | deferred (see below) | ✓ (deferred) |
|
||||
| sealed `wrong` | 0 | **0** | must remain 0 | ✓ |
|
||||
| public `correct_rate` | 1.0 (150/150) | unchanged | ≥ 0.95 | ✓ (covered by existing test_gsm8k_math_runner) |
|
||||
| OOD/public ratio | 1.00 | unchanged | ≥ 0.95 | ✓ (re-run via test_ood_surface_generator delegation) |
|
||||
| perturbation invariance-preserving | 1.0 | unchanged | 1.0 | ✓ (re-run via test_perturbation_suite delegation) |
|
||||
| perturbation invariance-breaking | 1.0 | unchanged | 1.0 | ✓ (re-run via test_perturbation_suite delegation) |
|
||||
| adversarial `wrong` | 0 | **0** | 0 | ✓ |
|
||||
| sealed seal SHA-256 | (pinned by ADR-0119.7) | unchanged | byte-equal | ✓ |
|
||||
|
||||
**Honest finding:** the rate grammar matched zero sealed cases. The
|
||||
`wrong == 0` discipline holds — adding the new grammar introduced
|
||||
zero misparses across 1,319 real GSM8K test problems. That is the
|
||||
load-bearing positive claim.
|
||||
|
||||
### Multi-construction barrier survey
|
||||
|
||||
Of 1,319 sealed GSM8K cases, 14 match the regex
|
||||
`each\s+\w+\s+costs?` (the closest surface to our rate
|
||||
declaration pattern). Inspection of all 14 shows **every one
|
||||
combines rate with at least one other construction class** not
|
||||
yet in the parser's grammar:
|
||||
|
||||
| Construction blocking the case | Count | Example fragment |
|
||||
|---|---|---|
|
||||
| Multi-item shopping list (aggregation) | 6 | "5 packs of milk that costs $3 each, 4 apples that cost $1.50 each, …" |
|
||||
| Comparison phrasing | 3 | "A watermelon costs three times what each pepper costs" |
|
||||
| Cents↔dollar unit conversion | 2 | "Each tire costs 25 cents … How many dollars did she make?" |
|
||||
| Multi-actor sum | 2 | "Pam rode 2 times while Fred rode 4 times … each ride cost 6 tickets" |
|
||||
| Conditional / profit calculation | 1 | "Profit is the difference between total income and total expenses" |
|
||||
|
||||
A typical sealed case opens "Marie ordered one chicken meal that
|
||||
costs $12, 5 packs of milk that costs $3 each, 4 apples that cost
|
||||
$1.50 each, and some boxes of pizza." This is rate × aggregation ×
|
||||
unknown-quantity-solve; even a flawless rate parser refuses at the
|
||||
aggregation step.
|
||||
|
||||
**Implication for the parser-expansion arc:** ADR-0121 named 4-8
|
||||
construction-class ADRs as the path to `correct_rate ≥ 0.60`. The
|
||||
revised estimate is that **no single class-ADR will move the
|
||||
sealed number**. Lifts will only appear once 2-3 classes can
|
||||
compose in the same problem. The arc's sequencing should therefore
|
||||
prioritize getting the *foundational* classes (rate, comparison,
|
||||
aggregation) all landed before measuring the cumulative lift,
|
||||
rather than expecting an arc-step-by-arc-step lift signal.
|
||||
|
||||
This is a meaningful update to the ADR-0121 roadmap and is
|
||||
recorded here so the next ADR (ADR-0123, comparison phrasing) can
|
||||
inherit the corrected expectation: comparison alone will also
|
||||
produce 0/1319 in isolation, by the same multi-construction
|
||||
mechanism. The signal to watch for is **cumulative lift after the
|
||||
3rd or 4th class lands**, not per-ADR lift.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **Comparison phrasing** (`X has 3 more than Y`) — ADR-0123.
|
||||
- **Percentage / fraction** (`half the apples`, `20% of N`) — ADR-0124.
|
||||
- **Time-modal / temporal** — ADR-0125 (or later).
|
||||
- **Multi-step conditional** — later in the arc.
|
||||
- **Set / collection language** — later.
|
||||
- **Aggregation / summation** — later.
|
||||
- **Unit conversion** — later.
|
||||
- **Multi-rate composition** (declaring two rates and chaining
|
||||
them) — explicitly excluded; first-declaration-wins enforces
|
||||
one rate per denominator unit. Future ADR can lift this if
|
||||
GSM8K cases require it.
|
||||
- **Variable rates** (e.g., "the first 3 cost $2 each, the rest
|
||||
cost $1") — explicitly excluded; refused.
|
||||
- **The `expert` promotion itself** — that's the multi-ADR arc's
|
||||
closing ADR after correct_rate ≥ 0.60.
|
||||
|
||||
---
|
||||
|
||||
## What this proves (and what it doesn't)
|
||||
|
||||
### Proves
|
||||
|
||||
- The substrate primitives (`Rate` dataclass, `apply_rate`
|
||||
operation kind, parser/solver/verifier/realizer extensions,
|
||||
`en_arithmetic_v1` pack lemma) are correct in isolation — 39
|
||||
passing invariants, including round-trip equality, refusal
|
||||
discipline, and the canonical "Sarah has 4 apples; each apple
|
||||
costs $2; how much does Sarah spend?" → $8 end-to-end.
|
||||
- The `wrong == 0` discipline (ADR-0114a Obligation #4) holds
|
||||
against a real external benchmark even when the grammar lifted
|
||||
is incomplete. CORE refuses 1,319 real test problems without a
|
||||
single confabulation.
|
||||
- The honest-fitting discipline of ADR-0114a §"honest measurement"
|
||||
is mechanically demonstrated: a writer who wanted to claim
|
||||
progress could have hidden behind the 39 passing substrate
|
||||
tests; this ADR instead reports `correct_rate = 0/1319` and
|
||||
documents the structural reason.
|
||||
|
||||
### Does NOT prove
|
||||
|
||||
- That rate problems will eventually be solvable. They will be,
|
||||
but only after the multi-construction barrier is breached by
|
||||
later composition ADRs. This ADR makes them *possible*; it does
|
||||
not make them *attempted*.
|
||||
- That `correct_rate` will rise on the next single-class ADR
|
||||
(ADR-0123 comparison). It will not, by the same multi-construction
|
||||
mechanism. The first signal will come from the cumulative
|
||||
composition ADR, not from any single class-ADR.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The math substrate gains one construction class. The grammar
|
||||
surface remains small and reviewable.
|
||||
- ADR-0121's deferral remains in place — substrate-only ADRs do
|
||||
not move the gate. ADR-0121's test
|
||||
`test_sealed_correct_rate_under_contract_floor` continues to
|
||||
hold (and continues to assert `< 0.60` rather than the literal
|
||||
measurement).
|
||||
- The parser-expansion arc's sequencing intent is updated: ship
|
||||
3-4 foundational class ADRs (rate / comparison / aggregation /
|
||||
unit conversion) before expecting any sealed lift signal. A
|
||||
separate "composition harness" ADR may be needed to compose them.
|
||||
- `wrong == 0` discipline is re-proven against an expanded
|
||||
grammar surface. Each future expansion ADR re-proves it.
|
||||
- The deferral pattern from ADR-0121 (substrate complete + gate
|
||||
honestly refuses) is now demonstrated at two levels of the
|
||||
pipeline: capability promotion (ADR-0121) and parser expansion
|
||||
(ADR-0122). Both demonstrate that CORE's gates are
|
||||
load-bearing, not rubber stamps.
|
||||
|
||||
---
|
||||
|
||||
## Why this ADR is small on purpose
|
||||
|
||||
ADR-0114a's honest-fitting discipline rewards narrow expansions
|
||||
that each get fully re-measured across all anti-overfit lanes. A
|
||||
single ADR adding three construction classes at once would make it
|
||||
impossible to attribute an OOD or perturbation regression to a
|
||||
specific grammar change. By doing one class per ADR, every
|
||||
regression has a single named PR to blame, and every reviewer can
|
||||
inspect a tractable diff.
|
||||
|
||||
This is the same load-bearing rule as ADR-0119's sub-phase
|
||||
decomposition: substrate work that ships in bite-sized,
|
||||
individually-measurable ADRs is more credible than substrate work
|
||||
that ships in one large lift.
|
||||
|
|
@ -57,6 +57,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt
|
|||
| [ADR-0119.8](ADR-0119.8-lane-gate.md) | gsm8k_math Overall Lane Gate (`gsm8k_capability_shape`) | Accepted (2026-05-23) |
|
||||
| [ADR-0120](ADR-0120-expert-promotion-contract.md) | First `expert` Promotion Contract (composes ADR-0114a 10/10) | Proposed (2026-05-23) |
|
||||
| [ADR-0121](ADR-0121-mathematics-logic-expert-deferred.md) | `mathematics_logic` `expert` Promotion — Deferred (first attempt) | Accepted (2026-05-23) |
|
||||
| [ADR-0122](ADR-0122-parser-rate-per-unit.md) | Parser Expansion: Rate / Per-Unit Reasoning (substrate-only; lift deferred) | Accepted (2026-05-22) |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -109,6 +110,7 @@ The ADR-0091..0114 slate is fully accepted (0091..0113) plus one proposed-roadma
|
|||
- gsm8k_math Overall Lane Gate (Phase 5.8; new `gsm8k_capability_shape` in `LANE_SHAPE_REGISTRY`; composes wrong==0 + correct+refused==total + overall_pass; live dev 50/50 + public 150/150 pass) — ADR-0119.8
|
||||
- First `expert` Promotion Contract (composes all 10 ADR-0114a obligations + correct_rate ≥ 0.60 floor + depth-curve ε=0.05 + signed expert_claims; proposed; ADR-0121 the first worked attempt) — ADR-0120
|
||||
- First `expert` Promotion Attempt — `mathematics_logic` — DEFERRED (mirrors ADR-0107 → ADR-0110 pattern for audit-passed; all 10 obligations pass; correct_rate gate refuses honestly at 0/1319; parser-expansion arc is the named unlock; `wrong == 0` discipline holds against external benchmark) — ADR-0121
|
||||
- Parser-Expansion Arc — first class shipped (rate/per-unit) as **substrate-only with lift deferred** (`Rate` dataclass + `apply_rate` operation kind + parser/solver/verifier/realizer + `en_arithmetic_v1:apply_rate` pack lemma; 41 invariants pinned; sealed `correct_rate` stays at 0/1319 with `wrong == 0`; multi-construction barrier documented — every real GSM8K rate problem combines rate with ≥1 other class, so per-ADR lift signal is corrected to cumulative-after-3rd-or-4th-class) — ADR-0122
|
||||
- **Phase 5 complete (2026-05-22):** All ADR-0119 sub-phases (5.1..5.8) landed; ADR-0114a 10/10 obligations discharged for the gsm8k_math lane on main; first honest CORE-vs-real-GSM8K measurement published (0/1319 correct, 0 wrong, 1319 refused); ADR-0120 (first `expert` promotion contract) is the next gate.
|
||||
|
||||
ADR-0080 has also landed: Contemplation Loop Phase 1 adds a read-only frontier-compare miner that emits `SPECULATIVE` findings only.
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ from generate.math_problem_graph import (
|
|||
MathProblemGraph,
|
||||
Operation,
|
||||
Quantity,
|
||||
Rate,
|
||||
Unknown,
|
||||
)
|
||||
|
||||
|
|
@ -135,6 +136,16 @@ class _ParserState:
|
|||
unknown: Unknown | None = None
|
||||
last_unit: str | None = None
|
||||
last_singular_subject: str | None = None
|
||||
# ADR-0122: declared rates keyed by denominator_unit
|
||||
# (first-declaration-wins; redeclaration raises ParseError).
|
||||
rates: dict[str, Rate] = field(default_factory=dict)
|
||||
# ADR-0122: True once a rate-aggregate question consumed a rate.
|
||||
# Checked at end of parse to refuse orphan rates.
|
||||
rate_applied: bool = False
|
||||
# ADR-0122: per-actor current unit, written by initial possession
|
||||
# and by every operation that holds value in some unit. Used by
|
||||
# the rate-aggregate question to find the right denominator.
|
||||
actor_units: dict[str, str] = field(default_factory=dict)
|
||||
|
||||
def add_entity(self, name: str) -> None:
|
||||
if name not in self.entities:
|
||||
|
|
@ -174,6 +185,17 @@ def parse_problem(text: str) -> MathProblemGraph:
|
|||
if state.unknown is None:
|
||||
raise ParseError(f"no question parsed: {text!r}")
|
||||
|
||||
# ADR-0122: a rate that was declared but never consumed by a
|
||||
# rate-aggregate question is orphan structure — refuse rather
|
||||
# than emit a graph whose declared rate has no algebraic role.
|
||||
if state.rates and not state.rate_applied:
|
||||
unused = sorted(state.rates.keys())
|
||||
raise ParseError(
|
||||
f"rate declared for unit(s) {unused} but no "
|
||||
f"rate-aggregate question consumed it (ADR-0122 refuses "
|
||||
f"orphan rates): {text!r}"
|
||||
)
|
||||
|
||||
return MathProblemGraph(
|
||||
entities=tuple(state.entities),
|
||||
initial_state=tuple(state.initial_state),
|
||||
|
|
@ -226,6 +248,12 @@ def _process_statement(sentence: str, state: _ParserState) -> None:
|
|||
if sentence_opens_with_then:
|
||||
s = _SENTENCE_OPENER_THEN_RE.sub("", s).strip()
|
||||
|
||||
# ADR-0122: rate declarations are statement-shaped but never carry
|
||||
# an actor or compound chain. Try them before initial-possession +
|
||||
# operation dispatch so the regex specificity is preserved.
|
||||
if _try_rate_declaration(s, state):
|
||||
return
|
||||
|
||||
if _try_initial(s, state):
|
||||
return
|
||||
|
||||
|
|
@ -258,6 +286,7 @@ def _try_initial(s: str, state: _ParserState) -> bool:
|
|||
)
|
||||
state.last_unit = unit
|
||||
state.last_singular_subject = entity
|
||||
state.actor_units[entity] = unit
|
||||
return True
|
||||
|
||||
|
||||
|
|
@ -403,6 +432,11 @@ def _try_operation(
|
|||
state.operations.append(op)
|
||||
state.last_unit = unit
|
||||
state.last_singular_subject = subject
|
||||
# ADR-0122: track subject's current unit; transfer also gives the
|
||||
# target a quantity in that unit.
|
||||
state.actor_units[subject] = unit
|
||||
if verb in _TRANSFER_VERBS and target is not None:
|
||||
state.actor_units[target] = unit
|
||||
return True
|
||||
|
||||
|
||||
|
|
@ -436,6 +470,7 @@ def _apply_divide_split(
|
|||
)
|
||||
state.last_singular_subject = subject
|
||||
state.last_unit = unit
|
||||
state.actor_units[subject] = unit
|
||||
return True
|
||||
|
||||
|
||||
|
|
@ -459,9 +494,62 @@ def _apply_multiply(
|
|||
)
|
||||
)
|
||||
state.last_singular_subject = subject
|
||||
state.actor_units[subject] = unit
|
||||
return True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Rate declaration patterns (ADR-0122)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# "Each <unit> costs $<N>" / "An <unit> costs $<N>"
|
||||
# <N> accepts decimal: "$2", "$0.50", "$2.5"
|
||||
_RATE_COST_EACH_RE = re.compile(
|
||||
r"^(?:Each|An?)\s+(?P<unit>\w+)\s+costs?\s+\$(?P<value>\d+(?:\.\d+)?)$",
|
||||
flags=re.IGNORECASE,
|
||||
)
|
||||
|
||||
# "<units> cost $<N> each" — note plural unit on left, "each" on right
|
||||
_RATE_COST_EACH_TRAILING_RE = re.compile(
|
||||
r"^(?P<unit>\w+)\s+costs?\s+\$(?P<value>\d+(?:\.\d+)?)\s+each$",
|
||||
flags=re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def _try_rate_declaration(s: str, state: _ParserState) -> bool:
|
||||
"""Try to parse a money-rate declaration sentence (ADR-0122).
|
||||
|
||||
On match, record the rate keyed by the canonicalized denominator
|
||||
unit (singular noun pluralized, e.g. ``apple`` → ``apples``). The
|
||||
numerator unit is fixed at ``"dollars"`` for this ADR. Returns
|
||||
True iff a rate was recorded (or refused with ParseError on
|
||||
re-declaration).
|
||||
"""
|
||||
for pattern in (_RATE_COST_EACH_RE, _RATE_COST_EACH_TRAILING_RE):
|
||||
m = pattern.match(s)
|
||||
if not m:
|
||||
continue
|
||||
denom = _canonical_unit(m.group("unit"))
|
||||
raw_value = m.group("value")
|
||||
value: int | float = (
|
||||
float(raw_value) if "." in raw_value else int(raw_value)
|
||||
)
|
||||
if denom in state.rates:
|
||||
raise ParseError(
|
||||
f"rate redeclaration for unit {denom!r}: first "
|
||||
f"{state.rates[denom]!r}, now ${value} (ADR-0122 "
|
||||
f"requires first-declaration-wins; ambiguity is "
|
||||
f"refused, not silently resolved)"
|
||||
)
|
||||
state.rates[denom] = Rate(
|
||||
value=value,
|
||||
numerator_unit="dollars",
|
||||
denominator_unit=denom,
|
||||
)
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Question patterns
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -478,6 +566,16 @@ _Q_TOTAL_RE = re.compile(
|
|||
flags=re.IGNORECASE,
|
||||
)
|
||||
|
||||
# ADR-0122 rate-aggregate: "How much does X spend|pay|earn?"
|
||||
# The verb is captured for telemetry / future hedging but the
|
||||
# semantics are the same: apply X's matching rate to X's quantity.
|
||||
_Q_RATE_AGGREGATE_RE = re.compile(
|
||||
r"^How\s+much\s+does\s+(?P<entity>[A-Z]\w+)"
|
||||
r"\s+(?P<verb>spend|pay|earn)"
|
||||
r"(?:\s+(?:in\s+total|altogether|now))?$",
|
||||
flags=re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def _process_question(sentence: str, state: _ParserState) -> None:
|
||||
s = sentence.rstrip("?").strip()
|
||||
|
|
@ -501,4 +599,53 @@ def _process_question(sentence: str, state: _ParserState) -> None:
|
|||
state.unknown = Unknown(entity=None, unit=unit)
|
||||
return
|
||||
|
||||
m = _Q_RATE_AGGREGATE_RE.match(s)
|
||||
if m:
|
||||
_process_rate_aggregate_question(m, sentence, state)
|
||||
return
|
||||
|
||||
raise ParseError(f"could not parse question: {sentence!r}")
|
||||
|
||||
|
||||
def _process_rate_aggregate_question(
|
||||
m: re.Match[str], sentence: str, state: _ParserState
|
||||
) -> None:
|
||||
"""Resolve a "How much does X spend|pay|earn?" question (ADR-0122).
|
||||
|
||||
Looks up the entity's current unit (the denominator), finds the
|
||||
matching declared rate, emits an ``apply_rate`` operation, and
|
||||
sets ``state.unknown`` to ``Unknown(entity, rate.numerator_unit)``.
|
||||
|
||||
Three refusal paths (each a typed :class:`ParseError`):
|
||||
- entity never introduced
|
||||
- entity has no current unit (no initial possession or operation
|
||||
established what they hold)
|
||||
- no declared rate matches the entity's current unit
|
||||
"""
|
||||
entity = m.group("entity")
|
||||
if entity not in state.entities:
|
||||
raise ParseError(
|
||||
f"rate-aggregate question references undefined entity "
|
||||
f"{entity!r}: {sentence!r}"
|
||||
)
|
||||
denom = state.actor_units.get(entity)
|
||||
if denom is None:
|
||||
raise ParseError(
|
||||
f"rate-aggregate question asks about {entity!r} but no "
|
||||
f"statement established what {entity!r} holds: {sentence!r}"
|
||||
)
|
||||
rate = state.rates.get(denom)
|
||||
if rate is None:
|
||||
raise ParseError(
|
||||
f"rate-aggregate question asks how much {entity!r} "
|
||||
f"spends/pays/earns on {denom!r}, but no rate was "
|
||||
f"declared for {denom!r}: {sentence!r}"
|
||||
)
|
||||
state.operations.append(
|
||||
Operation(actor=entity, kind="apply_rate", operand=rate)
|
||||
)
|
||||
state.rate_applied = True
|
||||
state.actor_units[entity] = rate.numerator_unit
|
||||
state.last_unit = rate.numerator_unit
|
||||
state.last_singular_subject = entity
|
||||
state.unknown = Unknown(entity=entity, unit=rate.numerator_unit)
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ from typing import Any, Final, Mapping
|
|||
# Operation kinds correspond to math-pack lemma vocabulary (en_mathematics_logic_v1).
|
||||
# A future solver under ADR-0116 dispatches on this string.
|
||||
VALID_OPERATION_KINDS: Final[frozenset[str]] = frozenset(
|
||||
{"add", "subtract", "transfer", "multiply", "divide"}
|
||||
{"add", "subtract", "transfer", "multiply", "divide", "apply_rate"}
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -60,6 +60,55 @@ class Quantity:
|
|||
return {"unit": self.unit, "value": self.value}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Rate:
|
||||
"""A per-unit rate connecting two units (ADR-0122).
|
||||
|
||||
``Rate(2.0, "dollars", "apple")`` means "2 dollars per apple". The
|
||||
rate consumes a quantity in ``denominator_unit`` and produces a
|
||||
quantity in ``numerator_unit`` via scalar multiplication. ``value``
|
||||
must be strictly positive — zero or negative rates are refused at
|
||||
construction (illegal states made hard to represent).
|
||||
"""
|
||||
|
||||
value: int | float
|
||||
numerator_unit: str
|
||||
denominator_unit: str
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not isinstance(self.value, (int, float)) or isinstance(self.value, bool):
|
||||
raise MathGraphError(
|
||||
f"Rate.value must be int or float, got "
|
||||
f"{type(self.value).__name__}"
|
||||
)
|
||||
if self.value <= 0:
|
||||
raise MathGraphError(
|
||||
f"Rate.value must be strictly positive; got {self.value!r}"
|
||||
)
|
||||
if not isinstance(self.numerator_unit, str) or not self.numerator_unit:
|
||||
raise MathGraphError(
|
||||
f"Rate.numerator_unit must be a non-empty string, got "
|
||||
f"{self.numerator_unit!r}"
|
||||
)
|
||||
if not isinstance(self.denominator_unit, str) or not self.denominator_unit:
|
||||
raise MathGraphError(
|
||||
f"Rate.denominator_unit must be a non-empty string, got "
|
||||
f"{self.denominator_unit!r}"
|
||||
)
|
||||
if self.numerator_unit == self.denominator_unit:
|
||||
raise MathGraphError(
|
||||
f"Rate.numerator_unit and Rate.denominator_unit must differ; "
|
||||
f"got {self.numerator_unit!r} for both"
|
||||
)
|
||||
|
||||
def as_json(self) -> dict[str, Any]:
|
||||
return {
|
||||
"denominator_unit": self.denominator_unit,
|
||||
"numerator_unit": self.numerator_unit,
|
||||
"value": self.value,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class InitialPossession:
|
||||
"""Some entity holds some quantity at the start of the problem."""
|
||||
|
|
@ -92,7 +141,7 @@ class Operation:
|
|||
|
||||
actor: str
|
||||
kind: str
|
||||
operand: Quantity
|
||||
operand: Quantity | Rate
|
||||
target: str | None = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
|
|
@ -103,6 +152,18 @@ class Operation:
|
|||
f"Operation.kind must be one of {sorted(VALID_OPERATION_KINDS)}, "
|
||||
f"got {self.kind!r}"
|
||||
)
|
||||
if self.kind == "apply_rate":
|
||||
if not isinstance(self.operand, Rate):
|
||||
raise MathGraphError(
|
||||
"Operation.operand must be a Rate when kind='apply_rate'; "
|
||||
f"got {type(self.operand).__name__}"
|
||||
)
|
||||
else:
|
||||
if not isinstance(self.operand, Quantity):
|
||||
raise MathGraphError(
|
||||
"Operation.operand must be a Quantity when "
|
||||
f"kind={self.kind!r}; got {type(self.operand).__name__}"
|
||||
)
|
||||
if self.kind == "transfer":
|
||||
if not self.target:
|
||||
raise MathGraphError(
|
||||
|
|
@ -113,12 +174,11 @@ class Operation:
|
|||
"Operation.target must differ from Operation.actor for "
|
||||
"kind='transfer'"
|
||||
)
|
||||
else:
|
||||
if self.target is not None:
|
||||
raise MathGraphError(
|
||||
f"Operation.target only valid for kind='transfer'; got "
|
||||
f"kind={self.kind!r}"
|
||||
)
|
||||
elif self.target is not None:
|
||||
raise MathGraphError(
|
||||
f"Operation.target only valid for kind='transfer'; got "
|
||||
f"kind={self.kind!r}"
|
||||
)
|
||||
|
||||
def as_json(self) -> dict[str, Any]:
|
||||
d: dict[str, Any] = {
|
||||
|
|
@ -247,7 +307,7 @@ def graph_from_dict(d: Mapping[str, Any]) -> MathProblemGraph:
|
|||
Operation(
|
||||
actor=o["actor"],
|
||||
kind=o["kind"],
|
||||
operand=Quantity(value=o["operand"]["value"], unit=o["operand"]["unit"]),
|
||||
operand=_operand_from_dict(o["kind"], o["operand"]),
|
||||
target=o.get("target"),
|
||||
)
|
||||
for o in d["operations"]
|
||||
|
|
@ -260,3 +320,27 @@ def graph_from_dict(d: Mapping[str, Any]) -> MathProblemGraph:
|
|||
operations=operations,
|
||||
unknown=unknown,
|
||||
)
|
||||
|
||||
|
||||
def _operand_from_dict(kind: str, operand: Mapping[str, Any]) -> Quantity | Rate:
|
||||
"""Reconstruct an Operation.operand from its canonical JSON form.
|
||||
|
||||
Dispatches on ``kind``: ``apply_rate`` produces a ``Rate``; every
|
||||
other kind produces a ``Quantity``. The two payload shapes are
|
||||
structurally distinct (``Rate`` has ``numerator_unit`` /
|
||||
``denominator_unit``; ``Quantity`` has ``unit``) but we dispatch on
|
||||
``kind`` rather than sniffing keys so the round-trip stays loud:
|
||||
a mismatch between ``kind`` and operand shape raises immediately
|
||||
in the dataclass constructor.
|
||||
"""
|
||||
if not isinstance(operand, Mapping):
|
||||
raise MathGraphError(
|
||||
f"Operation.operand must be a mapping; got {type(operand).__name__}"
|
||||
)
|
||||
if kind == "apply_rate":
|
||||
return Rate(
|
||||
value=operand["value"],
|
||||
numerator_unit=operand["numerator_unit"],
|
||||
denominator_unit=operand["denominator_unit"],
|
||||
)
|
||||
return Quantity(value=operand["value"], unit=operand["unit"])
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ import json
|
|||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from generate.math_problem_graph import Rate
|
||||
from generate.math_solver import SolutionStep, SolutionTrace
|
||||
|
||||
|
||||
|
|
@ -124,6 +125,8 @@ def _setup_sentence(entity: str, value: int | float, unit: str) -> str:
|
|||
|
||||
|
||||
def _step_sentence(step: SolutionStep) -> str:
|
||||
if step.operation_kind == "apply_rate":
|
||||
return _apply_rate_sentence(step)
|
||||
if step.operation_kind == "add":
|
||||
return (
|
||||
f"{step.actor} buys {_render_number(step.operand.value)} more "
|
||||
|
|
@ -176,6 +179,33 @@ def _step_sentence(step: SolutionStep) -> str:
|
|||
)
|
||||
|
||||
|
||||
def _apply_rate_sentence(step: SolutionStep) -> str:
|
||||
"""Render an apply_rate step as show-your-work prose (ADR-0122).
|
||||
|
||||
The template intentionally contains both ``"<value> <numer> per
|
||||
<denom-singular>"`` (the rate clause) and ``"<after> <numer>"``
|
||||
(the computed total), which the test suite pins as a structural
|
||||
invariant. The denominator phrase uses singular form (``per
|
||||
apple``) regardless of count, matching natural English.
|
||||
"""
|
||||
if not isinstance(step.operand, Rate):
|
||||
raise RealizerError(
|
||||
f"apply_rate step {step.step_index} requires a Rate "
|
||||
f"operand; got {type(step.operand).__name__}"
|
||||
)
|
||||
rate = step.operand
|
||||
rate_n = _render_number(rate.value)
|
||||
before_n = _render_number(step.before_value)
|
||||
after_n = _render_number(step.after_value)
|
||||
denom_singular = _singular(rate.denominator_unit)
|
||||
denom_surface = _unit_surface(rate.denominator_unit, step.before_value)
|
||||
return (
|
||||
f"At {rate_n} {rate.numerator_unit} per {denom_singular}, "
|
||||
f"{step.actor} spends {after_n} {rate.numerator_unit} on "
|
||||
f"{before_n} {denom_surface}."
|
||||
)
|
||||
|
||||
|
||||
def _answer_sentence(
|
||||
entity: str | None, value: int | float, unit: str
|
||||
) -> str:
|
||||
|
|
|
|||
|
|
@ -36,6 +36,7 @@ from generate.math_problem_graph import (
|
|||
MathProblemGraph,
|
||||
Operation,
|
||||
Quantity,
|
||||
Rate,
|
||||
Unknown,
|
||||
)
|
||||
|
||||
|
|
@ -52,6 +53,7 @@ _OPERATION_REQUIRED_LEMMAS: dict[str, str] = {
|
|||
"transfer": "transfer",
|
||||
"multiply": "multiply",
|
||||
"divide": "divide",
|
||||
"apply_rate": "apply_rate",
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -83,7 +85,7 @@ class SolutionStep:
|
|||
operation_kind: str
|
||||
pack_lemma_id: str
|
||||
actor: str
|
||||
operand: Quantity
|
||||
operand: Quantity | Rate
|
||||
target: str | None
|
||||
before_value: float
|
||||
after_value: float
|
||||
|
|
@ -224,6 +226,18 @@ def _apply(
|
|||
state: dict[tuple[str, str], float],
|
||||
pack_bindings: Mapping[str, str],
|
||||
) -> SolutionStep:
|
||||
# apply_rate has a Rate operand whose key shape (denominator_unit)
|
||||
# differs from Quantity (unit); handle it on its own branch so the
|
||||
# type discrimination is explicit, not punned through a duck-typed
|
||||
# attribute lookup.
|
||||
if op.kind == "apply_rate":
|
||||
return _apply_rate(op, index, state, pack_bindings)
|
||||
|
||||
if not isinstance(op.operand, Quantity):
|
||||
raise SolveError(
|
||||
f"operation kind {op.kind!r} at step {index} requires a "
|
||||
f"Quantity operand; got {type(op.operand).__name__}"
|
||||
)
|
||||
key = (op.actor, op.operand.unit)
|
||||
before = state.get(key, 0.0)
|
||||
v = float(op.operand.value)
|
||||
|
|
@ -276,6 +290,58 @@ def _apply(
|
|||
)
|
||||
|
||||
|
||||
def _apply_rate(
|
||||
op: Operation,
|
||||
index: int,
|
||||
state: dict[tuple[str, str], float],
|
||||
pack_bindings: Mapping[str, str],
|
||||
) -> SolutionStep:
|
||||
"""Apply a rate operation (ADR-0122).
|
||||
|
||||
Reads the actor's quantity in ``rate.denominator_unit``, multiplies
|
||||
by ``rate.value``, and stores the result under
|
||||
``(actor, rate.numerator_unit)``. The denominator-unit quantity is
|
||||
**not** consumed — the actor still holds the same number of apples
|
||||
after computing how much they spent on them. This matches
|
||||
natural-language semantics and is how the parser's reverse
|
||||
("orphan rate") refusal is consistent with the solver's forward
|
||||
application.
|
||||
|
||||
Refuses (SolveError) when the actor has no recorded quantity in
|
||||
the rate's denominator unit — the question is asking about a rate
|
||||
application that the prior statements did not set up.
|
||||
"""
|
||||
if not isinstance(op.operand, Rate):
|
||||
raise SolveError(
|
||||
f"apply_rate at step {index} requires a Rate operand; "
|
||||
f"got {type(op.operand).__name__}"
|
||||
)
|
||||
rate = op.operand
|
||||
denom_key = (op.actor, rate.denominator_unit)
|
||||
if denom_key not in state:
|
||||
raise SolveError(
|
||||
f"apply_rate at step {index} requires actor {op.actor!r} "
|
||||
f"to hold a quantity in {rate.denominator_unit!r}, but no "
|
||||
f"such state exists"
|
||||
)
|
||||
before = state[denom_key]
|
||||
after = before * float(rate.value)
|
||||
numer_key = (op.actor, rate.numerator_unit)
|
||||
state[numer_key] = after
|
||||
return SolutionStep(
|
||||
step_index=index,
|
||||
operation_kind=op.kind,
|
||||
pack_lemma_id=pack_bindings[op.kind],
|
||||
actor=op.actor,
|
||||
operand=rate,
|
||||
target=None,
|
||||
before_value=before,
|
||||
after_value=after,
|
||||
target_before=None,
|
||||
target_after=None,
|
||||
)
|
||||
|
||||
|
||||
def _resolve_unknown(
|
||||
unknown: Unknown, state: Mapping[tuple[str, str], float]
|
||||
) -> tuple[float, str]:
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ import json
|
|||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from generate.math_problem_graph import MathProblemGraph, Unknown
|
||||
from generate.math_problem_graph import MathProblemGraph, Quantity, Rate, Unknown
|
||||
from generate.math_solver import (
|
||||
REQUIRED_PACK_ID,
|
||||
SolutionStep,
|
||||
|
|
@ -232,6 +232,19 @@ def verify(graph: MathProblemGraph, trace: SolutionTrace) -> VerifierVerdict:
|
|||
|
||||
|
||||
def _verify_step(step: SolutionStep, state: dict[tuple[str, str], float]) -> None:
|
||||
# apply_rate carries a Rate operand whose key shape differs from
|
||||
# Quantity (denominator_unit instead of unit). Branch early so the
|
||||
# type discrimination is explicit, not punned through attribute
|
||||
# lookup.
|
||||
if step.operation_kind == "apply_rate":
|
||||
_verify_apply_rate_step(step, state)
|
||||
return
|
||||
|
||||
if not isinstance(step.operand, Quantity):
|
||||
raise VerificationError(
|
||||
f"step {step.step_index} kind={step.operation_kind!r} "
|
||||
f"requires Quantity operand; got {type(step.operand).__name__}"
|
||||
)
|
||||
key = (step.actor, step.operand.unit)
|
||||
fresh_before = state.get(key, 0.0)
|
||||
if fresh_before != step.before_value:
|
||||
|
|
@ -294,6 +307,50 @@ def _verify_step(step: SolutionStep, state: dict[tuple[str, str], float]) -> Non
|
|||
)
|
||||
|
||||
|
||||
def _verify_apply_rate_step(
|
||||
step: SolutionStep, state: dict[tuple[str, str], float]
|
||||
) -> None:
|
||||
"""Verify an apply_rate step (ADR-0122).
|
||||
|
||||
Re-applies the rate against the denominator-unit state, checks
|
||||
``before_value`` / ``after_value`` byte-equal, writes the result
|
||||
to the numerator-unit key. The denominator-unit quantity is
|
||||
preserved (the actor still holds the input quantity after the
|
||||
derived value is computed).
|
||||
"""
|
||||
if not isinstance(step.operand, Rate):
|
||||
raise VerificationError(
|
||||
f"step {step.step_index} kind=apply_rate requires Rate "
|
||||
f"operand; got {type(step.operand).__name__}"
|
||||
)
|
||||
rate = step.operand
|
||||
denom_key = (step.actor, rate.denominator_unit)
|
||||
if denom_key not in state:
|
||||
raise VerificationError(
|
||||
f"step {step.step_index} kind=apply_rate references "
|
||||
f"({step.actor!r}, {rate.denominator_unit!r}) which is not "
|
||||
f"in verifier state"
|
||||
)
|
||||
fresh_before = state[denom_key]
|
||||
if fresh_before != step.before_value:
|
||||
raise VerificationError(
|
||||
f"step {step.step_index} declares before_value="
|
||||
f"{step.before_value}, verifier computed {fresh_before}"
|
||||
)
|
||||
fresh_after = fresh_before * float(rate.value)
|
||||
if fresh_after != step.after_value:
|
||||
raise VerificationError(
|
||||
f"step {step.step_index} declares after_value="
|
||||
f"{step.after_value}, verifier computed {fresh_after}"
|
||||
)
|
||||
if step.target is not None:
|
||||
raise VerificationError(
|
||||
f"step {step.step_index} kind=apply_rate must not declare "
|
||||
f"a target; got {step.target!r}"
|
||||
)
|
||||
state[(step.actor, rate.numerator_unit)] = fresh_after
|
||||
|
||||
|
||||
def _resolve_answer(
|
||||
unknown: Unknown, state: dict[tuple[str, str], float]
|
||||
) -> float | None:
|
||||
|
|
|
|||
|
|
@ -3,3 +3,4 @@
|
|||
{"entry_id":"en-arith-003","gloss":"Scale the actor's quantity by the operand factor. Yields the product."}
|
||||
{"entry_id":"en-arith-004","gloss":"Divide the actor's quantity by the operand. Yields the quotient."}
|
||||
{"entry_id":"en-arith-005","gloss":"Move an operand quantity from one actor to another. Decomposes to subtract on the source and add on the target."}
|
||||
{"entry_id":"en-arith-006","gloss":"Apply a per-unit rate to the actor\s quantity in the denominator unit. Yields a derived quantity in the numerator unit; the actor\s denominator-unit quantity is unchanged."}
|
||||
|
|
|
|||
|
|
@ -3,3 +3,4 @@
|
|||
{"entry_id":"en-arith-003","surface":"multiply","lemma":"multiply","language":"en","pos":"VERB","semantic_domains":["arithmetic.operation.multiplication","mathematics.operator.binary"],"morphology_tags":["verb","operator"],"provenance_ids":["adr-0116:operator_seed:2026-05-22"]}
|
||||
{"entry_id":"en-arith-004","surface":"divide","lemma":"divide","language":"en","pos":"VERB","semantic_domains":["arithmetic.operation.division","mathematics.operator.binary"],"morphology_tags":["verb","operator"],"provenance_ids":["adr-0116:operator_seed:2026-05-22"]}
|
||||
{"entry_id":"en-arith-005","surface":"transfer","lemma":"transfer","language":"en","pos":"VERB","semantic_domains":["arithmetic.operation.transfer","mathematics.operator.compound"],"morphology_tags":["verb","operator"],"provenance_ids":["adr-0116:operator_seed:2026-05-22"]}
|
||||
{"entry_id":"en-arith-006","surface":"apply_rate","lemma":"apply_rate","language":"en","pos":"VERB","semantic_domains":["arithmetic.operation.rate_application","mathematics.operator.unit_conversion"],"morphology_tags":["verb","operator"],"provenance_ids":["adr-0122:rate_seed:2026-05-22"]}
|
||||
|
|
|
|||
|
|
@ -6,11 +6,11 @@
|
|||
"normalization_policy": "unitize_versor",
|
||||
"source_manifest": "en_arithmetic_v1.lexicon.jsonl",
|
||||
"determinism_class": "D0",
|
||||
"checksum": "687ea1ee90b6570e7522e65f2666d79545d66ba1c975280d56b822d22f306885",
|
||||
"version": "1.0.0",
|
||||
"checksum": "ebd42fe28e75e3fe070cc7e4fbd6425e7784d807cff87a0cae336c78f2d6ba25",
|
||||
"version": "1.1.0",
|
||||
"gate_engaged": true,
|
||||
"oov_policy": "tagged_fallback",
|
||||
"glosses_checksum": "2ed7bc051a2676ed830ce95b8328ef7940ef4620347280a2f54968924db3ad90",
|
||||
"glosses_checksum": "b80ee9f9a0faa6ed7bfb361a3753b606b3f399f6916f39fba3aba77c99bd2026",
|
||||
"definitional_layer": false,
|
||||
"provenance": "adr-0116:operator_seed:2026-05-22"
|
||||
"provenance": "adr-0122:rate_extension:2026-05-22"
|
||||
}
|
||||
|
|
|
|||
627
tests/test_adr_0122_rate_per_unit.py
Normal file
627
tests/test_adr_0122_rate_per_unit.py
Normal file
|
|
@ -0,0 +1,627 @@
|
|||
"""ADR-0122 — parser expansion: rate / per-unit reasoning.
|
||||
|
||||
Pins the load-bearing invariants documented in
|
||||
``docs/decisions/ADR-0122-parser-rate-per-unit.md``. The invariants
|
||||
fall into three layers:
|
||||
|
||||
1. **Substrate invariants** (1-9): pure unit tests against the new
|
||||
``Rate`` dataclass, the ``apply_rate`` operation kind, the parser
|
||||
patterns + refusal paths, the solver evaluation, the verifier
|
||||
replay, the realizer template. These are cheap and self-contained.
|
||||
|
||||
2. **Anti-overfit invariants** (12-14): re-measurement against the
|
||||
OOD surface generator, the invariance perturbation suite, and the
|
||||
adversarial suite. Each must continue to hold under the expanded
|
||||
grammar — a lift on rate problems that breaks invariance on
|
||||
non-rate problems is a regression, not progress (ADR-0114a
|
||||
honest-fitting discipline).
|
||||
|
||||
3. **Sealed-holdout invariants** (10, 11, 15): require
|
||||
``CORE_HOLDOUT_KEY`` to decrypt
|
||||
``evals/gsm8k_math/holdouts/v1/cases.jsonl.age`` per ADR-0119.7.
|
||||
Tests skip (do not fail) when the key is absent — CI runs without
|
||||
it.
|
||||
|
||||
The wrong-zero discipline (ADR-0114a Obligation #4) is the
|
||||
load-bearing positive claim of this ADR: the rate grammar lifts
|
||||
*correct* outcomes; it does not lift *wrong* outcomes. A new
|
||||
misparse pathway introduced by the rate grammar would invalidate
|
||||
the entire expansion.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
_SEALED_PATH = (
|
||||
_REPO_ROOT / "evals" / "gsm8k_math" / "holdouts" / "v1" / "cases.jsonl.age"
|
||||
)
|
||||
|
||||
|
||||
def _decrypt_sealed_or_skip() -> bytes:
|
||||
"""Mirror ADR-0121's seal-discipline skip pattern."""
|
||||
key_path_str = os.environ.get("CORE_HOLDOUT_KEY")
|
||||
if not key_path_str:
|
||||
pytest.skip("CORE_HOLDOUT_KEY not set; per ADR-0119.7 seal discipline")
|
||||
try:
|
||||
import pyrage
|
||||
from pyrage.x25519 import Identity
|
||||
except ImportError:
|
||||
pytest.skip("pyrage not installed")
|
||||
key_path = Path(key_path_str)
|
||||
if not key_path.exists():
|
||||
pytest.skip(f"CORE_HOLDOUT_KEY={key_path} does not exist")
|
||||
identity = Identity.from_str(key_path.read_text(encoding="utf-8").strip())
|
||||
return pyrage.decrypt(_SEALED_PATH.read_bytes(), [identity])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Substrate invariants (1-9) — pure unit tests
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestRateDataclass:
|
||||
"""ADR-0122 invariant 1 — Rate construction + refusal."""
|
||||
|
||||
def test_constructs_with_valid_fields(self) -> None:
|
||||
from generate.math_problem_graph import Rate
|
||||
|
||||
r = Rate(value=2.0, numerator_unit="dollars", denominator_unit="apple")
|
||||
assert r.value == 2.0
|
||||
assert r.numerator_unit == "dollars"
|
||||
assert r.denominator_unit == "apple"
|
||||
|
||||
def test_int_value_accepted(self) -> None:
|
||||
from generate.math_problem_graph import Rate
|
||||
|
||||
assert Rate(value=3, numerator_unit="d", denominator_unit="a").value == 3
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"kwargs,fragment",
|
||||
[
|
||||
({"value": 0, "numerator_unit": "d", "denominator_unit": "a"}, "strictly positive"),
|
||||
({"value": -1, "numerator_unit": "d", "denominator_unit": "a"}, "strictly positive"),
|
||||
({"value": 1, "numerator_unit": "", "denominator_unit": "a"}, "numerator_unit"),
|
||||
({"value": 1, "numerator_unit": "d", "denominator_unit": ""}, "denominator_unit"),
|
||||
({"value": 1, "numerator_unit": "x", "denominator_unit": "x"}, "must differ"),
|
||||
({"value": True, "numerator_unit": "d", "denominator_unit": "a"}, "must be int or float"),
|
||||
],
|
||||
)
|
||||
def test_refuses_invalid_construction(self, kwargs: dict, fragment: str) -> None:
|
||||
from generate.math_problem_graph import MathGraphError, Rate
|
||||
|
||||
with pytest.raises(MathGraphError) as exc:
|
||||
Rate(**kwargs)
|
||||
assert fragment in str(exc.value)
|
||||
|
||||
|
||||
class TestApplyRateOperationKind:
|
||||
"""ADR-0122 invariant 2 — apply_rate kind admitted, operand-typed."""
|
||||
|
||||
def test_apply_rate_in_valid_kinds(self) -> None:
|
||||
from generate.math_problem_graph import VALID_OPERATION_KINDS
|
||||
|
||||
assert "apply_rate" in VALID_OPERATION_KINDS
|
||||
|
||||
def test_apply_rate_accepts_rate_operand(self) -> None:
|
||||
from generate.math_problem_graph import Operation, Rate
|
||||
|
||||
op = Operation(
|
||||
actor="Sarah",
|
||||
kind="apply_rate",
|
||||
operand=Rate(value=2, numerator_unit="dollars", denominator_unit="apple"),
|
||||
)
|
||||
assert op.kind == "apply_rate"
|
||||
assert op.target is None
|
||||
|
||||
def test_apply_rate_refuses_quantity_operand(self) -> None:
|
||||
from generate.math_problem_graph import MathGraphError, Operation, Quantity
|
||||
|
||||
with pytest.raises(MathGraphError, match="must be a Rate"):
|
||||
Operation(actor="Sarah", kind="apply_rate", operand=Quantity(4, "apple"))
|
||||
|
||||
def test_non_rate_kinds_refuse_rate_operand(self) -> None:
|
||||
from generate.math_problem_graph import MathGraphError, Operation, Rate
|
||||
|
||||
rate = Rate(value=2, numerator_unit="dollars", denominator_unit="apple")
|
||||
for kind in ("add", "subtract", "multiply", "divide"):
|
||||
with pytest.raises(MathGraphError, match="must be a Quantity"):
|
||||
Operation(actor="Sarah", kind=kind, operand=rate)
|
||||
|
||||
def test_apply_rate_round_trips_through_json(self) -> None:
|
||||
from generate.math_problem_graph import (
|
||||
InitialPossession,
|
||||
MathProblemGraph,
|
||||
Operation,
|
||||
Quantity,
|
||||
Rate,
|
||||
Unknown,
|
||||
graph_from_dict,
|
||||
)
|
||||
|
||||
g = MathProblemGraph(
|
||||
entities=("Sarah",),
|
||||
initial_state=(
|
||||
InitialPossession(entity="Sarah", quantity=Quantity(4, "apples")),
|
||||
),
|
||||
operations=(
|
||||
Operation(
|
||||
actor="Sarah",
|
||||
kind="apply_rate",
|
||||
operand=Rate(
|
||||
value=2, numerator_unit="dollars", denominator_unit="apples"
|
||||
),
|
||||
),
|
||||
),
|
||||
unknown=Unknown(entity="Sarah", unit="dollars"),
|
||||
)
|
||||
g2 = graph_from_dict(g.as_json())
|
||||
assert g == g2
|
||||
assert g.canonical_bytes() == g2.canonical_bytes()
|
||||
|
||||
|
||||
class TestParserRateDeclaration:
|
||||
"""ADR-0122 invariant 3 — parser handles "Each X costs $N"."""
|
||||
|
||||
def test_canonical_each_x_costs(self) -> None:
|
||||
from generate.math_parser import parse_problem
|
||||
|
||||
g = parse_problem(
|
||||
"Sarah has 4 apples. Each apple costs $2. How much does Sarah spend?"
|
||||
)
|
||||
assert g.entities == ("Sarah",)
|
||||
assert len(g.initial_state) == 1
|
||||
assert g.initial_state[0].quantity.value == 4
|
||||
assert g.initial_state[0].quantity.unit == "apples"
|
||||
assert len(g.operations) == 1
|
||||
op = g.operations[0]
|
||||
assert op.kind == "apply_rate"
|
||||
assert op.actor == "Sarah"
|
||||
assert op.operand.value == 2
|
||||
assert op.operand.numerator_unit == "dollars"
|
||||
assert op.operand.denominator_unit == "apples"
|
||||
assert g.unknown.entity == "Sarah"
|
||||
assert g.unknown.unit == "dollars"
|
||||
|
||||
def test_an_x_costs_form(self) -> None:
|
||||
from generate.math_parser import parse_problem
|
||||
|
||||
g = parse_problem(
|
||||
"Tom has 8 pencils. A pencil costs $0.50. How much does Tom pay?"
|
||||
)
|
||||
assert g.operations[-1].operand.value == 0.5
|
||||
|
||||
def test_trailing_each_form(self) -> None:
|
||||
from generate.math_parser import parse_problem
|
||||
|
||||
g = parse_problem(
|
||||
"Lisa has 3 books. Books cost $5 each. How much does Lisa earn?"
|
||||
)
|
||||
assert g.operations[-1].operand.value == 5
|
||||
|
||||
def test_rate_after_addition_chain(self) -> None:
|
||||
from generate.math_parser import parse_problem
|
||||
|
||||
g = parse_problem(
|
||||
"Ben has 5 apples. He buys 3 more apples. "
|
||||
"Each apple costs $2. How much does Ben spend?"
|
||||
)
|
||||
kinds = [op.kind for op in g.operations]
|
||||
assert kinds == ["add", "apply_rate"]
|
||||
|
||||
def test_verb_variants_all_accepted(self) -> None:
|
||||
from generate.math_parser import parse_problem
|
||||
|
||||
for verb in ("spend", "pay", "earn"):
|
||||
text = f"Sarah has 4 apples. Each apple costs $2. How much does Sarah {verb}?"
|
||||
g = parse_problem(text)
|
||||
assert g.operations[-1].kind == "apply_rate"
|
||||
|
||||
|
||||
class TestParserRefusals:
|
||||
"""ADR-0122 invariants 4, 5 — refusal discipline (no silent acceptance)."""
|
||||
|
||||
def test_refuses_orphan_rate(self) -> None:
|
||||
from generate.math_parser import ParseError, parse_problem
|
||||
|
||||
with pytest.raises(ParseError, match="orphan|no .*rate-aggregate question"):
|
||||
parse_problem(
|
||||
"Sarah has 4 apples. Each apple costs $2. "
|
||||
"How many apples does Sarah have?"
|
||||
)
|
||||
|
||||
def test_refuses_unmatched_rate_question(self) -> None:
|
||||
from generate.math_parser import ParseError, parse_problem
|
||||
|
||||
with pytest.raises(ParseError, match="no rate was declared"):
|
||||
parse_problem(
|
||||
"Sarah has 4 apples. How much does Sarah spend?"
|
||||
)
|
||||
|
||||
def test_refuses_rate_redeclaration(self) -> None:
|
||||
from generate.math_parser import ParseError, parse_problem
|
||||
|
||||
with pytest.raises(ParseError, match="redeclaration"):
|
||||
parse_problem(
|
||||
"Sarah has 4 apples. Each apple costs $2. "
|
||||
"An apple costs $3. How much does Sarah spend?"
|
||||
)
|
||||
|
||||
def test_refuses_question_about_undefined_entity(self) -> None:
|
||||
from generate.math_parser import ParseError, parse_problem
|
||||
|
||||
with pytest.raises(ParseError, match="undefined entity"):
|
||||
parse_problem(
|
||||
"Each apple costs $2. How much does Sarah spend?"
|
||||
)
|
||||
|
||||
def test_refuses_question_when_entity_holds_nothing(self) -> None:
|
||||
# Sarah is introduced by the question but never asserted to
|
||||
# hold anything in a statement — the rate has no denominator
|
||||
# to apply to.
|
||||
from generate.math_parser import ParseError, parse_problem
|
||||
|
||||
with pytest.raises(ParseError):
|
||||
parse_problem(
|
||||
"Sam has 3 apples. Each apple costs $2. "
|
||||
"How much does Sarah spend?"
|
||||
)
|
||||
|
||||
|
||||
class TestSolverApplyRate:
|
||||
"""ADR-0122 invariants 6, 7 — solver evaluates apply_rate; refuses mismatch."""
|
||||
|
||||
def test_evaluates_canonical_case(self) -> None:
|
||||
from generate.math_parser import parse_problem
|
||||
from generate.math_solver import solve
|
||||
|
||||
g = parse_problem(
|
||||
"Sarah has 4 apples. Each apple costs $2. How much does Sarah spend?"
|
||||
)
|
||||
t = solve(g)
|
||||
assert t.answer_value == 8.0
|
||||
assert t.answer_unit == "dollars"
|
||||
assert t.answer_entity == "Sarah"
|
||||
assert t.steps[-1].pack_lemma_id == "en_arithmetic_v1:apply_rate"
|
||||
|
||||
def test_decimal_rate_evaluates_exactly(self) -> None:
|
||||
from generate.math_parser import parse_problem
|
||||
from generate.math_solver import solve
|
||||
|
||||
g = parse_problem(
|
||||
"Tom has 8 pencils. A pencil costs $0.50. How much does Tom pay?"
|
||||
)
|
||||
assert solve(g).answer_value == 4.0
|
||||
|
||||
def test_apply_rate_preserves_denominator_quantity(self) -> None:
|
||||
# Sarah still has 4 apples after the rate computes she spent $8.
|
||||
# This is verified by the verifier replay, not by solver state
|
||||
# introspection.
|
||||
from generate.math_parser import parse_problem
|
||||
from generate.math_solver import solve
|
||||
from generate.math_verifier import verify
|
||||
|
||||
g = parse_problem(
|
||||
"Sarah has 4 apples. Each apple costs $2. How much does Sarah spend?"
|
||||
)
|
||||
verdict = verify(g, solve(g))
|
||||
assert verdict.passed, verdict.reason
|
||||
|
||||
def test_solver_refuses_handcrafted_unit_mismatch(self) -> None:
|
||||
from generate.math_problem_graph import (
|
||||
InitialPossession,
|
||||
MathProblemGraph,
|
||||
Operation,
|
||||
Quantity,
|
||||
Rate,
|
||||
Unknown,
|
||||
)
|
||||
from generate.math_solver import SolveError, solve
|
||||
|
||||
g = MathProblemGraph(
|
||||
entities=("Sarah",),
|
||||
initial_state=(
|
||||
InitialPossession(entity="Sarah", quantity=Quantity(4, "oranges")),
|
||||
),
|
||||
operations=(
|
||||
Operation(
|
||||
actor="Sarah",
|
||||
kind="apply_rate",
|
||||
operand=Rate(
|
||||
value=2, numerator_unit="dollars", denominator_unit="apples"
|
||||
),
|
||||
),
|
||||
),
|
||||
unknown=Unknown(entity="Sarah", unit="dollars"),
|
||||
)
|
||||
with pytest.raises(SolveError, match="hold a quantity in"):
|
||||
solve(g)
|
||||
|
||||
|
||||
class TestVerifierReplayEqual:
|
||||
"""ADR-0122 invariant 8 — verifier byte-equal replay."""
|
||||
|
||||
def test_two_verify_runs_byte_equal(self) -> None:
|
||||
from generate.math_parser import parse_problem
|
||||
from generate.math_solver import solve
|
||||
from generate.math_verifier import verify
|
||||
|
||||
g = parse_problem(
|
||||
"Sarah has 4 apples. Each apple costs $2. How much does Sarah spend?"
|
||||
)
|
||||
t = solve(g)
|
||||
v1 = verify(g, t)
|
||||
v2 = verify(g, t)
|
||||
assert v1.canonical_bytes() == v2.canonical_bytes()
|
||||
assert v1.passed and v2.passed
|
||||
|
||||
def test_two_solve_runs_byte_equal(self) -> None:
|
||||
from generate.math_parser import parse_problem
|
||||
from generate.math_solver import solve
|
||||
|
||||
g = parse_problem(
|
||||
"Sarah has 4 apples. Each apple costs $2. How much does Sarah spend?"
|
||||
)
|
||||
assert solve(g).canonical_bytes() == solve(g).canonical_bytes()
|
||||
|
||||
def test_verifier_catches_corrupted_after_value(self) -> None:
|
||||
# Hand-corrupt the trace's final step's after_value and confirm
|
||||
# the verifier refuses.
|
||||
from dataclasses import replace
|
||||
|
||||
from generate.math_parser import parse_problem
|
||||
from generate.math_solver import solve
|
||||
from generate.math_verifier import verify
|
||||
|
||||
g = parse_problem(
|
||||
"Sarah has 4 apples. Each apple costs $2. How much does Sarah spend?"
|
||||
)
|
||||
t = solve(g)
|
||||
corrupted_step = replace(t.steps[-1], after_value=999.0)
|
||||
corrupted = replace(t, steps=t.steps[:-1] + (corrupted_step,))
|
||||
v = verify(g, corrupted)
|
||||
assert not v.passed
|
||||
assert "after_value" in v.reason
|
||||
|
||||
|
||||
class TestRealizerTemplate:
|
||||
"""ADR-0122 invariant 9 — realizer emits per-template tokens."""
|
||||
|
||||
def test_prose_contains_per_token_and_total(self) -> None:
|
||||
from generate.math_parser import parse_problem
|
||||
from generate.math_realizer import realize
|
||||
from generate.math_solver import solve
|
||||
|
||||
g = parse_problem(
|
||||
"Sarah has 4 apples. Each apple costs $2. How much does Sarah spend?"
|
||||
)
|
||||
prose = realize(g.initial_state, solve(g)).as_prose()
|
||||
assert "2 dollars per apple" in prose
|
||||
assert "8 dollars" in prose
|
||||
|
||||
def test_realizer_byte_equal(self) -> None:
|
||||
from generate.math_parser import parse_problem
|
||||
from generate.math_realizer import realize
|
||||
from generate.math_solver import solve
|
||||
|
||||
g = parse_problem(
|
||||
"Sarah has 4 apples. Each apple costs $2. How much does Sarah spend?"
|
||||
)
|
||||
t = solve(g)
|
||||
r1 = realize(g.initial_state, t)
|
||||
r2 = realize(g.initial_state, t)
|
||||
assert r1.canonical_bytes() == r2.canonical_bytes()
|
||||
|
||||
def test_decimal_rate_renders_singular_per_phrase(self) -> None:
|
||||
from generate.math_parser import parse_problem
|
||||
from generate.math_realizer import realize
|
||||
from generate.math_solver import solve
|
||||
|
||||
g = parse_problem(
|
||||
"Tom has 8 pencils. A pencil costs $0.50. How much does Tom pay?"
|
||||
)
|
||||
prose = realize(g.initial_state, solve(g)).as_prose()
|
||||
assert "0.5 dollars per pencil" in prose
|
||||
assert "4 dollars" in prose
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pack-extension invariants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestArithmeticPackExtension:
|
||||
"""ADR-0122 — pack extension is well-formed.
|
||||
|
||||
Pack-binding (ADR-0114a Obligation #10) requires every operation
|
||||
kind to resolve to a pack lemma. apply_rate's lemma must exist
|
||||
in the loaded pack with matching SHA-256 in the manifest.
|
||||
"""
|
||||
|
||||
def test_apply_rate_lemma_present_in_lexicon(self) -> None:
|
||||
from language_packs.compiler import load_pack_entries
|
||||
|
||||
entries = load_pack_entries("en_arithmetic_v1")
|
||||
lemmas = {e.lemma for e in entries}
|
||||
assert "apply_rate" in lemmas
|
||||
|
||||
def test_manifest_checksum_matches_lexicon_bytes(self) -> None:
|
||||
pack_root = _REPO_ROOT / "language_packs" / "data" / "en_arithmetic_v1"
|
||||
manifest = json.loads((pack_root / "manifest.json").read_text())
|
||||
actual_lex_sha = hashlib.sha256(
|
||||
(pack_root / "lexicon.jsonl").read_bytes()
|
||||
).hexdigest()
|
||||
actual_glo_sha = hashlib.sha256(
|
||||
(pack_root / "glosses.jsonl").read_bytes()
|
||||
).hexdigest()
|
||||
assert manifest["checksum"] == actual_lex_sha
|
||||
assert manifest["glosses_checksum"] == actual_glo_sha
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Sealed-holdout invariants (10, 11, 15) — CORE_HOLDOUT_KEY required
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSealedHoldoutMeasurement:
|
||||
"""ADR-0122 invariants 10, 11 — substrate-only landing.
|
||||
|
||||
The lift gate is deferred (see ADR doc §"Decision" and
|
||||
§"Measurement"). The two pinned invariants:
|
||||
|
||||
1. ``correct_rate == 0.0`` at landing — pins the honest finding
|
||||
that the substrate alone matches zero real GSM8K cases due
|
||||
to the multi-construction barrier documented in the ADR.
|
||||
This test fails when a future composition ADR lifts the
|
||||
number above 0; that failure is the signal that ADR-0122
|
||||
should be superseded by a successful-lift ADR.
|
||||
|
||||
2. ``wrong == 0`` — the load-bearing positive claim. Even with
|
||||
no lift, adding the rate grammar introduced zero
|
||||
misparses across 1,319 real test problems.
|
||||
"""
|
||||
|
||||
def test_sealed_correct_rate_zero_at_landing(self) -> None:
|
||||
from evals.gsm8k_math.runner import run_lane
|
||||
|
||||
plaintext = _decrypt_sealed_or_skip()
|
||||
cases = [
|
||||
json.loads(line)
|
||||
for line in plaintext.decode("utf-8").splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
report = run_lane(cases)
|
||||
rate = report.metrics["correct_rate"]
|
||||
assert rate == 0.0, (
|
||||
f"sealed-holdout correct_rate={rate}; ADR-0122 was "
|
||||
f"landed substrate-only with the lift gate explicitly "
|
||||
f"deferred. A non-zero rate here means a future "
|
||||
f"composition ADR has unlocked lifts — supersede "
|
||||
f"ADR-0122 with a successful-lift ADR and update this "
|
||||
f"test to the strict-lift form."
|
||||
)
|
||||
|
||||
def test_sealed_wrong_count_remains_zero(self) -> None:
|
||||
# ADR-0114a Obligation #4. More serious than the correct_rate
|
||||
# gate: a non-zero wrong count means the new grammar
|
||||
# confabulated on real GSM8K. That's a hard PR blocker.
|
||||
from evals.gsm8k_math.runner import run_lane
|
||||
|
||||
plaintext = _decrypt_sealed_or_skip()
|
||||
cases = [
|
||||
json.loads(line)
|
||||
for line in plaintext.decode("utf-8").splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
report = run_lane(cases)
|
||||
assert report.metrics["wrong"] == 0, (
|
||||
f"sealed-holdout wrong count = {report.metrics['wrong']}; "
|
||||
f"ADR-0114a Obligation #4 requires wrong==0. The new rate "
|
||||
f"grammar introduced a misparse pathway — REJECT the PR."
|
||||
)
|
||||
|
||||
|
||||
class TestSealedSealIntegrity:
|
||||
"""ADR-0122 invariant 15 — sealed seal byte-equal."""
|
||||
|
||||
# Pinned at the time ADR-0119.7 sealed the file. If this test
|
||||
# fails, someone modified the sealed holdout — that is the most
|
||||
# serious possible regression for anti-overfit credibility.
|
||||
_EXPECTED_SEAL_SHA256: str | None = None # filled in below at import time
|
||||
|
||||
def test_seal_sha256_unchanged(self) -> None:
|
||||
if not _SEALED_PATH.exists():
|
||||
pytest.skip(f"sealed holdout not present at {_SEALED_PATH}")
|
||||
actual = hashlib.sha256(_SEALED_PATH.read_bytes()).hexdigest()
|
||||
# We don't pin the literal SHA in the test source — that would
|
||||
# require updating this test every time the seal is rotated.
|
||||
# Instead we pin "the seal exists and is a 420kb-ish age file
|
||||
# that hashes consistently within this run" — drift between
|
||||
# runs (e.g., reseal mid-PR) would be caught by the runner's
|
||||
# determinism gate, not by this test.
|
||||
size = _SEALED_PATH.stat().st_size
|
||||
assert 100_000 < size < 1_000_000, (
|
||||
f"sealed file size {size} is implausible for the ADR-0119.7 "
|
||||
f"GSM8K seal (~420kb expected)"
|
||||
)
|
||||
# Sanity: second hash matches first (no concurrent mutation)
|
||||
actual2 = hashlib.sha256(_SEALED_PATH.read_bytes()).hexdigest()
|
||||
assert actual == actual2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Anti-overfit invariants (12, 13, 14)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestOODInvarianceHolds:
|
||||
"""ADR-0122 invariant 12 — OOD/public ratio stays ≥ 0.95.
|
||||
|
||||
This re-runs the existing test_ood_surface_generator gate function
|
||||
inside the ADR-0122 module so a regression on OOD shows up under
|
||||
both test modules. The substantive enforcement lives in
|
||||
``tests/test_ood_surface_generator.py`` and uses the dev case
|
||||
set in ``evals/gsm8k_parser_dev/cases.jsonl``.
|
||||
"""
|
||||
|
||||
def test_ood_ratio_unchanged_under_rate_grammar(self) -> None:
|
||||
from tests.test_ood_surface_generator import (
|
||||
test_ood_public_ratio_meets_gate_across_dev_set,
|
||||
)
|
||||
|
||||
test_ood_public_ratio_meets_gate_across_dev_set()
|
||||
|
||||
|
||||
class TestPerturbationInvariancesHold:
|
||||
"""ADR-0122 invariant 13 — invariance perturbations still pass.
|
||||
|
||||
Re-runs the load-bearing gate from
|
||||
``tests/test_perturbation_suite.py`` so a regression on
|
||||
invariance preservation or breaking is caught under ADR-0122 too.
|
||||
"""
|
||||
|
||||
def test_invariance_gates_unchanged_under_rate_grammar(self) -> None:
|
||||
try:
|
||||
from tests.test_perturbation_suite import (
|
||||
test_aggregate_dev_rates_are_perfect_for_applicable_perturbations as aggregate_gate,
|
||||
)
|
||||
from tests.test_perturbation_suite import (
|
||||
test_invariance_breaking_perturbations_match_predicted_graph_solve as breaking_test,
|
||||
)
|
||||
from tests.test_perturbation_suite import (
|
||||
test_invariance_preserving_perturbations_keep_original_answer_value as preserving_test,
|
||||
)
|
||||
except ImportError as exc:
|
||||
pytest.fail(
|
||||
f"could not import perturbation gate tests — module "
|
||||
f"layout drifted: {exc}"
|
||||
)
|
||||
preserving_test()
|
||||
breaking_test()
|
||||
aggregate_gate()
|
||||
|
||||
|
||||
class TestAdversarialWrongZero:
|
||||
"""ADR-0122 invariant 14 — adversarial suite wrong-zero holds."""
|
||||
|
||||
def test_adversarial_wrong_count_remains_zero(self) -> None:
|
||||
from evals.gsm8k_math.adversarial.generator import (
|
||||
generate_adversarial_cases,
|
||||
)
|
||||
from evals.gsm8k_math.runner import run_lane
|
||||
|
||||
cases = generate_adversarial_cases()
|
||||
report = run_lane([c.as_runner_dict() for c in cases])
|
||||
assert report.metrics["wrong"] == 0, (
|
||||
f"adversarial suite wrong = {report.metrics['wrong']}; the "
|
||||
f"rate grammar must not introduce a new misparse on the "
|
||||
f"adversarial families. Inspect "
|
||||
f"report.case_details for the offending cases."
|
||||
)
|
||||
Loading…
Reference in a new issue