54e6bfc0d0
Implements the 4-phase documentation reorganization master plan. - Consolidation: Merged brief/, handoff/, planning/, and decisions/ into briefs/, handoffs/, plans/, and adr/ respectively (101 ADRs relocated) - Root Cleanup: Relocated HANDOFF-gpt55-*.md and key top-level docs (runtime_contracts.md, etc.) to canonical folders. Added superseded alerts. - Indices & Navigation: Created docs/README.md navigation document, docs/sessions/README.md index, docs/adr/README.md index - Note: Also includes prior commit adding ADR-0200+ corpus hygiene governance (ADR-0225, dependency map, backfilled cross-references)
89 lines
3.4 KiB
Markdown
89 lines
3.4 KiB
Markdown
# ADR-0131.G.5 — Aggregate Answer Composition
|
|
|
|
**Status:** Accepted
|
|
**Parent:** [ADR-0131.G — GSM8K Coverage Probe](ADR-0131.G-gsm8k-coverage-probe.md)
|
|
**Date:** 2026-05-23
|
|
|
|
## Context
|
|
|
|
The aggregate-answer path — questions like "How many apples do they have
|
|
altogether?" — was functionally complete before this ADR. The parser
|
|
(`_Q_TOTAL_RE` in `generate/math_candidate_parser.py`) already emitted
|
|
`Unknown(entity=None, unit=<unit>)` for aggregate cues, and the solver
|
|
(`generate/math_solver.py`) already summed all terminal state entries
|
|
matching the questioned unit when `entity is None`.
|
|
|
|
What was missing:
|
|
|
|
1. **Vocabulary gap (now closed):** `"combined"` and `"together"` were
|
|
absent from `_Q_TOTAL_RE`'s tail alternation, causing questions using
|
|
those cues to be refused even when the solver would have produced the
|
|
correct sum.
|
|
2. **No pinned lane:** no curated axis cases proved the 2-entity,
|
|
3-entity, and degenerate aggregate paths end-to-end through
|
|
`parse_and_solve`.
|
|
|
|
## Decision
|
|
|
|
### Closed aggregate-cue vocabulary
|
|
|
|
Exactly four cues are admitted:
|
|
|
|
| Cue | Example tail |
|
|
|-----|-------------|
|
|
| `in total` | "How many apples do they have in total?" |
|
|
| `altogether` | "How many apples do they have altogether?" |
|
|
| `combined` | "How many apples do they have combined?" |
|
|
| `together` | "How many apples do they have together?" |
|
|
|
|
All four map to `entity=None` semantics — the solver sums all state
|
|
entries whose unit matches the questioned unit, across all entities.
|
|
|
|
### Solver path (pre-existing)
|
|
|
|
The `entity is None` branch in `_resolve_unknown` was not changed. It
|
|
sums `v for (_, unit), v in state.items() if unit == unknown.unit`.
|
|
This ADR extends the cue vocabulary and pins the lane, not the solver.
|
|
|
|
### Axis lane
|
|
|
|
20 curated cases at `evals/math_capability_axes/G5_aggregate/v1/cases.jsonl`:
|
|
|
|
| Shape | Count | Purpose |
|
|
|-------|-------|---------|
|
|
| 2-entity sum, no operations | 4 | one case per cue |
|
|
| 3-entity sum, no operations | 4 | one case per cue |
|
|
| 2-entity sum with add/subtract op | 4 | mixed cues |
|
|
| Single-entity degenerate | 4 | regression guard |
|
|
| Refusal: outside closed cue | 4 | wrong==0 probe |
|
|
|
|
Refusal cases use question forms outside the closed `_Q_TOTAL_RE`
|
|
pattern (e.g., "How many apples does everyone have?", "What is the
|
|
total number of coins?") to verify the parser correctly refuses
|
|
paraphrases not in the closed cue set.
|
|
|
|
### Gate
|
|
|
|
`wrong == 0` on every axis case. GSM8K `admitted_wrong == 0` preserved
|
|
(no admission movement expected — all 50 sample cases still refuse at
|
|
statement parsing; question-layer work cannot lift that).
|
|
|
|
## Deferred
|
|
|
|
- **Implicit aggregation without a cue word:** "How many apples do Sam
|
|
and Tom have?" requires coreference resolution (named-entity →
|
|
pronoun-equivalent grouping). Out of scope for the closed-cue model.
|
|
- **Rate-based aggregation:** "How many dollars did they earn in total?"
|
|
where the unit derives from a rate operation. Requires rate-verb
|
|
support in the statement parser.
|
|
- **GSM8K admission lift:** all 50 sample cases fail at statement
|
|
parsing (rate verbs, compound sentences, implicit entities).
|
|
Question-layer cue extensions cannot move that number.
|
|
|
|
## Evidence
|
|
|
|
- Axis runner: `evals/math_capability_axes/G5_aggregate/v1/runner.py`
|
|
- Report: `evals/math_capability_axes/G5_aggregate/v1/report.json`
|
|
- Tests: `tests/test_adr_0131_G5_aggregate.py`
|
|
- B3 lane unchanged.
|
|
- GSM8K `admitted_wrong == 0` preserved.
|