core/docs/decisions/ADR-0138-comparative-reference-layer.md
Shay 0685c87e54
docs(ADR-0138): comparative-reference layer — design only (#209)
A small typed linguistic layer (FractionOperand, ComparativeOperand,
QuantityReference, ReferenceTarget) replacing the closed-and-deferred
ADR-0137 with a shape-driven design. Justified by the rescan-v3 ledger
showing 7 cases under fraction_operand + compound_comparative that
share a common deep structure (value relative to literal or
prior-sentence anchor) with multiple surface shapes.

Key commitments:
- Three typed phrase structures handle the closed surface set
  (frac/percent of X, N times as much/greater than X, frac more/less
  than X), where X is either a sentence-internal literal or a
  QuantityReference to a prior grounded quantity.
- Binding pass is minimal: sentence-order lookup of QuantityReference
  against prior grounded candidates with unique-match gating. No
  generic DeferredCandidate apparatus.
- Pinned English convention: "N times greater" = N+1 multiplier
  (documented + tested). Other reading requires explicit revisit.
- Supersedes ADR-0137 in scope; the deferral note's reopen criterion
  is met by this lighter, more specific design.
- S.1/S.2 short-circuits stay as joins of grounded candidates;
  this ADR does not subsume them.

Forcing case for cross-sentence binding: gsm8k-0029 ("three times
greater than the cost of the mouse" + "the mouse cost $16"). Clean,
unambiguous, two-sentence binding.

Self-correction caught before push: gsm8k-0010 was initially framed
as the forcing case, but its expected dataset answer requires a
non-standard English parse of "1/4 more than X" (the natural reading
gives 17; the dataset says 9, requiring X/4 + 7). Per non-negotiable
#2 the binding pass MUST refuse on this; 0010 is reclassified from
"expected admission" to "negative probe." 0029 takes the forcing-case
slot.

Honest expected admission delta: +1 to +2 (0005, 0029 high
confidence; 0041, 0043 partial credit only).

In scope: in the same PR family, fix the issue gsm8k-0029 surfaces by
implementing the binding pass. Out of scope: inverse arithmetic
(0004), multi-term aggregate references (0036), pronoun anchoring,
self-referential cycles.
2026-05-23 22:35:29 -07:00

19 KiB
Raw Permalink Blame History

ADR-0138 — Comparative-Reference Layer

Status: Draft (design-only) Date: 2026-05-23 Parent: ADR-0136 Informed by: ADR-0136.S3-post-rescan, ADR-0136.S2-post-rescan (ADR-0137 deferral note) Supersedes (in scope): ADR-0137 — closed without merging; reopened here in a lighter, shape-driven form


Context

The S.x corridor is bounded. Each phase ships one closed regex shape and yields 01 admissions + ~1 barrier shift. After S.0/S.1/S.2/S.3 the GSM8K probe sits at 3/50 admitted, wrong=0, with the next-largest blockers fragmented across heterogeneous shapes:

compound_statement      5  (heterogeneous — see ADR-0136.S.3 deferred list)
novel_initial_form      5  (S.4 candidate)
fraction_operand        4  ← target of this ADR
novel_initial_verb      4
compound_comparative    3  ← target of this ADR
conditional_question    3
context_filler          3  (correctly refused)

The seven cases under fraction_operand + compound_comparative are not seven separate regex shapes. They share a common deep structure: a value-bearing phrase whose value is relative to either a numeric anchor in the same sentence or a referenced quantity from a previous sentence.

Per-case decomposition:

Case Shape Anchor source
0005 decrease to 3/4 of its temperature + current temperature is 84 sentence-internal numeric
0029 keyboard cost three times greater than the cost of the mouse + mouse cost $16 sentence-internal numeric
0041 75% of the 2nd pan (after 2 pans…16 pieces per pan) sentence-internal referent
0043 father gave twice as much as her mother (after mother gave $4) prior-sentence quantity
0010 Marion has 1/4 more than what Yun currently has, plus 7 prior-sentence derived quantity
0004 Half of the kids… + 1/4 of the kids going to soccer camp… + 750 kids…in the afternoon inverse: known sub-quantity, unknown total
0036 as much again as Wednesday, Thursday and Friday combined multi-term aggregate reference

0005 and 0041 admit sentence-internal grounding. 0029 is the forcing case for cross-sentence binding — its comparative ("three times greater than the cost of the mouse") in one sentence resolves against a literal anchor ("the mouse cost $16") in a later sentence, with no ambiguity. 0043 requires a similar chain (twice as much as her mother after mother gave $4). 0010 is genuinely ambiguous English — its expected dataset answer requires a non-standard parse of "1/4 more than X"; this ADR refuses on it (see §Scope). 0004 and 0036 require additional machinery (inverse arithmetic, multi-term aggregation) and are out of scope.

Why this ADR, not S.x. S.x ships one regex shape per phase. The comparative-reference cluster is one deep structure with multiple surface shapes. Treating each surface as a separate S.x phase wastes effort and produces a forest of overlapping regexes. A small typed layer that represents the deep structure once and consumes multiple surfaces is the right unit of work.

Why not ADR-0137 as drafted. The closed-and-deferred ADR-0137 proposed a generic DeferredCandidate / BindingProof apparatus for any kind with open slots. Its deferral reason stands: subsuming the S.1/S.2 short-circuits (which are joins of grounded candidates, not deferred bindings) wouldn't have used the machinery. Now that gsm8k-0010 has surfaced a real deferred-binding case, the right response is not to revive the generic apparatus. It is to ship a typed reference layer with one shape — QuantityReference — that specifically handles cross-sentence quantity lookup, and to keep the machinery as light as the actual problem.


Decision

1. Three typed phrase structures, no generic deferred-candidate apparatus

Introduce three frozen, slotted dataclasses representing the deep structure of comparative/fractional phrases:

@dataclass(frozen=True, slots=True)
class FractionOperand:
    """A fraction-shaped operand: 1/4, half, 75%, 3/4, etc."""
    numerator: int                  # e.g. 1
    denominator: int                # e.g. 4
    of_target: ReferenceTarget | None  # e.g. "the 2nd pan", "the kids"
    source_span: str

@dataclass(frozen=True, slots=True)
class ComparativeOperand:
    """A comparison: 'three times greater than X', '1/4 more than X',
    'twice as much as X', 'half as long as X'."""
    direction: Literal["more", "less", "times_greater", "times_as_much",
                       "fraction_more", "fraction_less"]
    modifier: FractionOperand | int | float  # the multiplier or addend
    anchor: ReferenceTarget                  # what's being compared to
    source_span: str

@dataclass(frozen=True, slots=True)
class ReferenceTarget:
    """An anchor: either a literal numeric value visible in the same
    sentence, or a typed reference to a quantity introduced earlier."""
    kind: Literal["literal", "quantity_reference"]
    literal_value: int | float | None          # set when kind == 'literal'
    quantity_reference: QuantityReference | None  # set when kind == 'quantity_reference'

@dataclass(frozen=True, slots=True)
class QuantityReference:
    """A typed cross-sentence reference: 'what Yun currently has',
    'the cost of the mouse', 'the 2nd pan', 'her mother [gave]'."""
    entity: str | None             # named entity (Yun), or None for "the X" descriptors
    descriptor: str | None         # 'the 2nd pan', 'the cost of the mouse'
    qualifier: str | None          # 'currently', 'initially'
    source_span: str

    @property
    def needs_binding(self) -> bool:
        return True

These types are descriptive only at the first pass. They carry no admissibility check on their own; they are slots that the binding pass (§3) closes.

2. Extractor responsibilities (first pass)

Three new extractor functions in generate/math_candidate_parser.py, each producing typed phrases rather than full candidates:

  • extract_fraction_operands(sentence)list[FractionOperand]
  • extract_comparative_operands(sentence)list[ComparativeOperand]
  • (QuantityReference is constructed inside the above two when their anchor is non-literal.)

These extractors emit phrases. They do not produce CandidateInitial or CandidateOperation directly. Promotion happens in the binding pass.

3. The binding pass (lighter than ADR-0137's draft)

A single new control-flow stage between first-pass extraction and the existing Cartesian product. Its only job: resolve QuantityReference slots against prior-sentence grounded candidates, then promote fully-grounded ComparativeOperand / FractionOperand phrases into CandidateInitial (when they describe an entity's quantity) or slot-fillers for question candidates (when they describe a value asked about).

The binding pass:

  1. Iterates over ComparativeOperand + FractionOperand phrases in sentence order.
  2. For each phrase with a non-literal anchor, looks up the QuantityReference against the cumulative grounded-candidate set from earlier sentences.
  3. Resolves only when the reference is unique — single matching (entity, descriptor) → single quantity. Ambiguous references refuse the resolution (drop, not best-effort).
  4. Computes the resolved value via the comparative/fractional rule:
    • times_greater(N, anchor): anchor * (N + 1) (interpretation pinned by ADR-0138; N times greater is ambiguous in English and we pick one closed reading — see §Non-negotiables).
    • times_as_much(N, anchor): anchor * N
    • fraction_more(F, anchor): anchor * (1 + F)
    • fraction_less(F, anchor): anchor * (1 - F)
    • more(N, anchor): anchor + N
    • less(N, anchor): anchor - N (refuses on negative result)
  5. Promotes the resolved value into the existing CandidateInitial path via the same round-trip filter.

No DeferredCandidate type. No BindingProof type. The pass is a deterministic lookup-and-promote operation. Replay determinism comes from the same JSON-sort-keys serialization the cognition eval already uses.

4. Sentence-internal grounding (no binding pass needed)

When ComparativeOperand.anchor.kind == "literal", the value is extracted in the first pass without involving the binding pass. This handles the 0005 / 0029 cases (sentence-internal numeric anchor) and 0041 (sentence-internal referent + same-sentence quantity).

5. ADR-0137 supersession

The closed ADR-0137 is superseded in scope by this ADR. The deferral note in ADR-0136.S2-post-rescan.md stands; the binding machinery this ADR proposes is the answer to its "reopen criterion." This ADR is lighter because it commits to one typed reference shape (QuantityReference) rather than a generic DeferredCandidate apparatus.

The S.1/S.2 short-circuits in parse_and_solve stay as joins of grounded candidates. They are not retrospective binding and this ADR does not subsume them. (The canonical-runner staleness issue is fixed separately by teaching the runner to score short-circuit admissions.)


Scope

In scope (this ADR + its implementation phases)

  • FractionOperand, ComparativeOperand, ReferenceTarget, QuantityReference type machinery
  • First-pass extractors for the closed surface set:
    • <N> times greater than <anchor>
    • <N> times as much/many/long as <anchor>
    • <fraction> more than <anchor> (where <fraction> ∈ {1/2, 1/4, 3/4, half, twice} or <N>%)
    • <fraction> less than <anchor>
    • <fraction> of <descriptor>
    • <N>% of <descriptor>
  • Binding pass: sentence-order lookup of QuantityReference against prior grounded candidates with unique-match gating
  • Probe lane RC1_comparative_reference with the three cleanly in-scope cases (0029, 0041, 0010) plus negative probes (0005 — see below — 0004, 0036, 0043)
  • ADR-supersession note added to ADR-0136.S2-post-rescan.md acknowledging that 0137's reopen criterion is met by 0138

Out of scope (deferred to follow-up ADRs)

  • Inverse arithmetic (case 0004): "750 kids are afternoon, 1/4 are morning → total = 1000" requires solving for the unknown total from a known sub-quantity. Different machinery (inverse fraction inference).
  • Multi-term aggregate references (case 0036): "as much again as Wednesday, Thursday and Friday combined" requires summing multiple prior quantities through a single reference. Probably ADR-0139.
  • Self-reference (case 0005): "decrease to 3/4 of its temperature" — note this case is in the in-scope set as a negative probe: the binding pass must refuse on self-references (its resolves to the same entity whose value is being computed, yielding a circular binding). Confirm in the lane that 0005 still refuses.

Wait — re-examining 0005: "Addison mountain's temperature will decrease to 3/4 of its temperature. If the current temperature is 84 degrees, what will the temperature decrease by?" The anchor is sentence-internal — "the current temperature is 84." The "its" refers to "Addison mountain's temperature" which is given as 84. So 0005 is actually in scope under sentence-internal grounding once the question-form is parsed. Reclassified to in-scope. Negative probes remain 0004 and 0036.

Cases this ADR is expected to admit

Case Expected outcome Mechanism
0005 admit (answer=21) sentence-internal fraction-of-anchor + question-resolution rule
0029 admit (answer=64) cross-sentence QuantityReference resolution: comparative in sentence 2 ("three times greater than the cost of the mouse") resolved against sentence 3's literal ("the mouse cost $16")
0041 partial — likely still refuses needs sentence-internal referent resolution; depends on whether _INITIAL_HAS_RE or its sibling grounds "2 pans" with a per-pan quantity
0043 partial — needs chain (mother + father + total + purchase) the comparative twice as much as her mother resolves cleanly; total-aggregation across the rest is multi-term, out of scope

Honest expected admission delta: +1 to +2 (0005, 0029 at high confidence; 0041 and 0043 partial credit only if the linguistic structure composes with existing extractors).

On gsm8k-0010 (the case that surfaced the forcing function). The forcing case for cross-sentence binding turned out to be 0029, not 0010. 0010's sentence 2 ("Marion has 1/4 more than what Yun currently has, plus 7") has a genuinely ambiguous English reading. The dataset's expected answer is 9, which requires parsing the phrase as (1/4) * Yun + 7 = (1/4)*8 + 7 = 9 — treating "1/4 more" as "1/4 [of what Yun has]" rather than "1/4 more than [Yun's quantity]." The natural English reading (Yun * (1 + 1/4) + 7 = 17) disagrees with the dataset.

Per non-negotiable #2 of this ADR, 0010 must refuse. Multiple plausible parses of the same surface yield different numeric values; the binding pass refuses rather than picks one. 0010 is reclassified from "expected admission" to "negative probe" — it stays refused, and the reason (ambiguity) becomes a regression assertion in the lane.

This is the right outcome under refusal-first discipline, and it proves that the cross-sentence binding case must be justified by an unambiguous example. 0029 is that example.


Non-negotiables

  1. wrong == 0 remains a hard CI gate. Across the binding eval lane and the GSM8K probe.
  2. Ambiguous reference → refuse the binding. If QuantityReference has multiple matching prior candidates with different values, the resolution refuses; the problem refuses; no best-effort.
  3. Pinned English-reading convention for "N times greater." English usage is ambiguous (some readers: anchor + anchor * N; others: anchor * N). This ADR pins the additive reading (anchor * (N + 1)) and documents the choice in the ADR + the lane test. If a downstream case appears that requires the other reading, refuses-then-revisit; do not silently pick the other.
  4. No new short-circuit path in parse_and_solve. Promoted candidates flow through the existing graph machinery.
  5. No solver/graph/verifier changes. Linguistic layer ends at candidate emission.
  6. Determinism. Same input → byte-equal BindingPassResults, byte-equal promoted-candidate set, byte-equal report.json.
  7. Closed surface vocabulary. All comparative + fractional surface markers are enumerated. No wildcards. New markers require extending the closed set (in the same PR as the test case that forces them).

Probe set

Required admissions

Case Source Why this exercises the layer
0005 GSM8K sentence-internal fraction-of-anchor + literal-value path
0029 GSM8K cross-sentence QuantityReference resolution; the forcing case for the binding pass

Required refusals (negative probes)

  • gsm8k-0010 — ambiguous English (1/4 more than X parses as either X * (1 + 1/4) or (1/4) * X); the binding pass must refuse per non-negotiable #2
  • gsm8k-0004 — inverse arithmetic (must continue to refuse pending ADR-0139)
  • gsm8k-0036 — multi-term aggregate reference (must continue to refuse pending ADR-0139)
  • A synthetic ambiguous-reference case where two prior sentences introduce the same (entity, unit) with different values — must refuse (uniqueness gate)

Axis cases (synthetic, ≥15)

Closed combinations of the surface vocabulary × anchor kinds:

  • <frac> of <literal> ≥3
  • <frac> of <reference> ≥3
  • <N> times as much as <literal> ≥3
  • <N> times greater than <reference> ≥3
  • <frac> more than <reference> ≥3 (covers 0010's shape)

Phased rollout

One branch per phase. The 2026-05-23 worktree-race lesson stands.

Phase Deliverable Branch
0138.D This ADR docs/adr-0138-comparative-reference-layer (this PR)
0138.A generate/comparative/{types,extractors}.py + tests for the type machinery and the closed surface vocabulary (no binding pass yet) feat/adr-0138-types-and-extractors
0138.B Binding pass + QuantityReference resolution + 0010 admission + 0005/0029 admissions feat/adr-0138-binding-pass
0138.C Eval lane RC1_comparative_reference + probe set feat/adr-0138-rc1-lane

Each phase is its own PR. Each must hold wrong == 0, preserve all existing axis lanes (S.1, S.2, S.3, S.4 if landed), and not regress the rescan-v3 invariants.

After 0138.B lands, run a rescan v4 to record barrier shifts and update the post-rescan ADR.


Open questions (deferred to implementation ADRs, not this one)

  • Promotion of fractional-of-set phrases. "Half of the kids" with no prior quantification of "the kids" produces a refusal in 0138.B (no resolvable reference). Case 0004's full solution requires inverse arithmetic on this same shape. Future ADR-0139 (inverse arithmetic) will compose with this layer.
  • Pronoun resolution depth. "Marion has 1/4 more than what Yun currently has" names Yun explicitly. Cases like "he has 1/4 more than what she has" need pronoun resolution against the prior-actor set. Out of scope; refuse on pronoun-anchored references until a future ADR.
  • Telemetry. Whether QuantityReference resolution events emit into the existing turn-event telemetry sink. Probably no for now; cognition runtime is not consuming math problems on the hot path.

Consequences

  • The forcing function for retrospective binding (denied at #204 closure, surfaced at #208 rescan v3) is met by a typed reference layer rather than a generic deferred-candidate apparatus. Smaller, more specific, easier to audit.
  • The S.x corridor continues for shape-specific extensions (novel_initial_form, novel_initial_verb, the remaining compound_statement shapes). 0138 is parallel to S.x, not its replacement.
  • The corpus of comparative + fraction-of-quantity problems is large in GSM8K's full set (not just the 50-case train sample). This layer's value compounds as the probe widens beyond the train sample.
  • The pinned "N times greater = N+1 multiplier" reading is a CORE convention. If we later encounter a case that requires the other reading, the lane test will flag it and the closed-set decision becomes revisitable rather than buried.

What this ADR is not

  • Not a code change. No generate/comparative/ module exists at merge time.
  • Not a generic deferred-candidate framework. It commits to one typed reference shape (QuantityReference) and one binding rule (unique-match cross-sentence lookup).
  • Not a replacement for the S.x corridor. The two are parallel.
  • Not a license to add semantic guessing. Every comparative, fractional, and reference shape is a closed-set surface pattern with a deterministic resolution rule.
  • Not in conflict with CLAUDE.md. The binding pass is exactly the shape of "comprehend → recall → think" — hold a typed reference, bind it deterministically when the prior quantity is grounded, refuse when ambiguous.