core-labs/core
core-labs/core
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

docs(reader-arc): recalibration ADR-0251 + preserved knowledge + overfit inventory

Browse source 
Records the reader-arc recalibration: halt bespoke-per-case regex work,
verified clean-base reset to main@854ce023, and eradication of
feat/reader-inc2-caseband's (PR #80) debt (loosened _token_in, empty
proof file, bundled provenance) without merging that branch.

Preserves the genuinely-general knowledge before the branch is retired:
the has_numeric_token multiplier-blindness bug (root cause + minimal fix
idea, deliberately not landed) and case 0148's correct-for-the-right-reason
decomposition, both independently re-verified against source rather than
copied from the branch's own claims (which included a stale narrative
about a graph-dump proof file that was actually committed empty).

Adds a file:line-cited overfit inventory (31 bespoke/single-case surface
patterns) as inventory only -- nothing removed, the existing regex reader
keeps serving unmodified.

ADR-0251 proposes (not builds) a geometric surface->canonical
normalization spike using existing off-serving primitives
(conformal_procrustes, VocabManifold, the cognition pipeline's Phase C
anti-unification telemetry), wrong=0-gated against holdout_dev/v1, with
Smith-chart algebra and any Fibonacci "dimensional cascade" explicitly
disclaimed. Awaiting ruling -- no implementation started.
This commit is contained in:
Shay 2026-07-19 13:42:47 -07:00
parent 854ce023d9
commit f8bf960d8d
4 changed files with 627 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

199
docs/adr/ADR-0251-reader-arc-recalibration-geometric-normalization-spike-proposal.md Normal file
View file

@ -0,0 +1,199 @@
# ADR-0251: Reader-Arc Recalibration — Halt Bespoke-Per-Case Regex Work; Clean-Base Reset; Proposed Geometric Surface→Canonical Normalization Spike
**Status**: §§1–4 (the recalibration decision — halt, reset, debt eradication, knowledge preservation) are
**executed**, verified by command, 2026-07-19. §5 (the geometric-normalization spike) is **Proposed —
awaiting ruling**. This ADR does not authorize any build; §5 is a direction proposal only.
**Date**: 2026-07-19
**Deciders**: Joshua Shay (ruling authority) + Claude (recalibration executor, this pass)
**Depends on**: `#77`/`#78`/`#79` (compare-multiplicative compiler tier + `NEUTRAL_COUNT_VERBS`
positive-polarity allowlist — the foundations this recalibration preserves), ADR-0136 family
(statement-layer corridor), ADR-0175 (calibrated attempt-and-eliminate learning — serving `wrong=0` vs.
sealed practice), ADR-0191 (candidate-graph completeness guard), ADR-0249 (reader→Hamiltonian compiler
composition frontier), ADR-0250 (tier-2 multi-entity arithmetic)
**Related (for §5's building blocks)**: ADR-0239 (Conformal Procrustes / analogical versor search — the
general primitive this spike would apply), ADR-0245 (CGA unification, mechanical sympathy and semantic
rigor), ADR-0246 (§6 explicitly disclaims Smith-chart algebra — honored here, not re-litigated)
**Supersedes**: `feat/reader-inc2-caseband` (PR #80) — **abandoned, not merged**
**Companion docs**: `docs/research/reader-arc-recalibration-preserved-knowledge-2026-07-19.md`,
`docs/research/0148-first-conversion-proof.md`, `docs/research/reader-arc-overfit-inventory-2026-07-19.md`
---
## 1. Context — why bespoke-per-case regex work is being halted
The GSM8K math reader (`generate/math_candidate_parser.py`, `generate/recognizer_match.py`,
`generate/recognizer_anchor_inject.py`, `generate/math_roundtrip.py`, `generate/math_candidate_graph.py`)
has advanced case-by-case: each new held-out case that needed a new surface form got a new, narrow regex
tailored to that case's exact wording. This has produced real, `wrong=0`-safe progress (the `#77`/`#78`/
`#79` compiler tier and positive-polarity substrate are sound and stay on `main`), but the *shape* of the
growth is overfitting-shaped: one bespoke pattern per case does not generalize, and the inventory in
§4 shows the pattern accumulating.
`feat/reader-inc2-caseband` (PR #80) is the branch that surfaced this concretely. It pursued case 0148
(the first real held-out conversion of this arc) and, per the PR's own review thread, succeeded at the
narrow goal — `report.json` shows exactly one case flipped (`0148: refused → correct`), `correct 5→6`,
`wrong=0` held — but introduced debt in the process:
1. **A shared grounding primitive was loosened fail-open.** `_token_in()` in `generate/math_roundtrip.py`
— used by every candidate's roundtrip-admissibility check, not just mass-noun ones — was widened to
accept a multi-word phrase whenever all of its component words appear *anywhere* in the source's token
set (order- and position-independent), rather than requiring the phrase to be a contiguous substring.
That weakens a shared safety gate to admit one family. This is the same "don't loosen a shared gate to
admit one case" failure the `#77` split was already built to avoid.
2. **A committed "proof" artifact was left empty.** `docs/research/0148-first-conversion-proof.md` was
committed at 0 bytes while the branch's own session-break summary claimed a graph dump had been
recorded there. It had not been. See the preserved-knowledge doc §3 for the verified discrepancy.
3. **Commits bundled provenance** — prior uncommitted WIP and `report.json` landed under one message,
muddying the record of what was reviewed.
None of this reached `main` — the branch was never merged (verified in §2). But the pattern it exhibits
(one more bespoke regex, one more case) is the thing being halted, independent of this specific branch's
debt. Per the ruling recorded here: **no further bespoke-per-case regex work proceeds on this reader.**
The existing regex reader is not being torn out — it keeps serving as-is (§2) — but it is not being
extended case-by-case going forward. The forward direction is proposed in §5, gated on a human ruling.
## 2. Decision — halt + clean-base reset (executed and verified)
All of the following were verified by direct command against the repository, not asserted:
- **Local `main` == canonical tip.** `git rev-parse` on `refs/heads/main`, `origin/main` (GitHub mirror),
and `forgejo/main` (source of truth per this project's routing doctrine) all resolve to
`854ce023d9cd58663ce9c9af0d62f92684ffb277` — the merge commit for PR #79 (increment-1 BAND plan:
seed + comparison `#78` + question-arithmetic). Local `main` had drifted to this tip via an earlier
`git reset: moving to forgejo/main` (from a stale `920695bd`); `git merge-base --is-ancestor 920695bd
854ce023` confirms that reset was fast-forward-equivalent — `854ce023` is a strict descendant, no
history was rewritten or lost.
- **`feat/reader-inc2-caseband` / PR #80 is abandoned, not merged.** `merged: false`,
`merge_commit_sha: null` on the PR; the branch's tip (`e3da148d`) is not an ancestor of `main`. The PR
is closed (not merged) with a comment referencing this ADR (see the PR thread). The branch ref itself
is left intact on the remote as an archived reference — no force-delete, no history rewrite.
- **Main is free of the branch's specific debt.** `grep -rn "_COMPARE_MASSNOUN_RE"` and a grep for the
loosened multi-word `_token_in` signature (`source_span: str | None = None`) both return zero matches
under `generate/`. No empty `*proof*.md` exists anywhere in `docs/` (`0148-first-conversion-proof.md`
is the one populated fresh in this recalibration, §3 below — 4.6 KB, not empty).
- **The legitimate foundations are intact and unmodified.** `compare_multiplicative` (the `#77` compiler
tier) and `NEUTRAL_COUNT_VERBS` / the positive-polarity allowlist (`#78`) are both present across
`generate/math_candidate_parser.py`, `generate/math_roundtrip.py`, `generate/recognizer_match.py`, and
related modules — these are sound, reviewed, and load-bearing, and this recalibration does not touch
them.
- **Smoke and the holdout baseline are green on this exact state.** `uv run core test --suite smoke -q`
→ 176 passed, 0 failed. `uv run python -m evals.gsm8k_math.holdout_dev.v1.runner` →
`correct=5 wrong=0 refused=495 (n=500)` — matching the committed `evals/gsm8k_math/holdout_dev/v1/
report.json` baseline exactly (the pre-branch state, before 0148's one-case flip). `wrong=0` holds.
No scope beyond the abandoned branch's own additions was removed. This recalibration is a reset to the
already-correct `main`, not a rollback of anything reviewed and merged.
## 3. Preserved knowledge (not lost)
Two facts of genuine, general value were extracted from the abandoned branch before it was set aside, and
are now captured — independently re-verified, not copied from the branch's own claims — in
`docs/research/reader-arc-recalibration-preserved-knowledge-2026-07-19.md`:
1. **The `has_numeric_token` multiplier-blindness bug** (`generate/math_candidate_parser.py:2374`) — the
closed word-number set the reader uses to decide whether a statement sentence carries parseable
numeric state does not include multiplicative comparatives (`twice`, `thrice`, `double`, `triple`).
This is real and reproducible (verified against 23 real cases in `holdout_dev/v1` where it matters),
but — also verified — it has not been a `wrong=0` hazard, because the independent ADR-0191
completeness guard (`generate/math_completeness.py`) already resolves multipliers correctly via the
`en_numerics_v1` pack and refuses rather than confabulates. Net effect: missed conversions, not wrong
answers. A minimal, general fix idea is documented (extend `has_numeric_token` to recognize the closed
`_ANCHOR_TO_FACTOR` multiplier set) but is **explicitly not landed** — per this project's floor, a
shared-primitive change lands only with full-500 `wrong=0` proof and paired with whatever downstream
op consumes it, never in isolation.
2. **Case 0148 solves correct-for-the-right-reason.** The real (non-empty) graph dump —
`docs/research/0148-first-conversion-proof.md` — shows the conditional-seed binding the *named*
reference (second day = 500), the compare op reading forward (first day = 2×second day, not an
inverted guess), and the summation covering exactly both registers to reach 1500. This is a validation
target for a future increment, not code being landed now.
## 4. Overfit inventory (a list, not a deletion)
A read-only survey of `generate/*.py` and `evals/refusal_taxonomy/*.py` on `main` identified bespoke,
single-case-shaped surface patterns — regexes and extractors that encode one sentence's exact wording
rather than a general grammatical class. The full, file:line-cited list lives in
`docs/research/reader-arc-overfit-inventory-2026-07-19.md`. This is **inventory, not a removal list** —
every pattern on it keeps serving unmodified; the reader must keep working until a proven replacement
exists (§7 of the task guardrails). Its purpose is to give the geometric-normalization spike (§5), if
ruled to proceed, a concrete target list of what "one general operator replacing N bespoke regexes" would
need to subsume before it could claim a positive generalization ratio.
## 5. Proposed forward direction — geometric surface→canonical normalization spike (AWAITING RULING)
**This section is a proposal. Nothing in it is authorized to be built by this ADR.** It is recorded so
the design-first gate has a concrete direction to rule on, per the task that produced this recalibration.
### 5.1 Thesis
Instead of hand-writing one new regex per newly-encountered surface shape, normalize a statement's surface
text into a canonical geometric representation and let a single general operator align novel surface forms
to already-known canonical forms — replacing N bespoke regex families with one normalization step plus a
small, closed set of canonical forms the solver already understands.
### 5.2 Building blocks — already implemented, off-serving; to be *wired*, not invented
Verified present in the repo, all currently off-serving/telemetry-only for this purpose:
- **`conformal_procrustes`** (`core/physics/dynamic_manifold.py:634`) — Kabsch-conformal Procrustes /
field conjugacy: finds the best versor (or grade-1 sandwich adjoint) mapping a source point-set onto a
target point-set. Docstring is explicit: *"Off-serving geometry; not wired into chat/runtime."* This is
the general primitive ADR-0239 (Proposed) already scopes for analogical transfer; this spike would be
one concrete application of it, not a new mechanism.
- **`VocabManifold`** (`vocab/manifold.py:68`) — the geometric vocabulary: every word is a unit versor in
Cl(4,1); `nearest(F)` finds the closest word by CGA inner product (no cosine similarity, no ANN index,
per the module's own normalization doctrine).
- **The cognition pipeline's "conformal anti-unifier" intent** (`core/cognition/pipeline.py:472-478`,
"Phase C — Geometric Anti-Unification") — currently read-only telemetry that captures graph-structural
context around OOV/pending holes "so that a future exact-CGA sub-graph anti-unifier can operate on
conformal neighbors rather than lexical token match." Explicitly observational only today ("never
affects surface, hash (yet), or any durable mutation").
The spike, if ruled to proceed, would be the work of *assembling* these three already-accepted-or-proposed
primitives into a reader-normalization path — not inventing new geometric machinery.
### 5.3 Proposed measurement — generalization ratio, `wrong=0`-gated
Define **generalization ratio** = (real `holdout_dev/v1` cases newly converted refused→correct) ÷ (new
concepts/parameters added to the normalizer to achieve that). Contrast baseline: the bespoke-regex era's
own ratio is ~1 case per bespoke regex family (0148's mass-noun frame converted exactly one case for one
new regex + one actor-binding relaxation + one Gate-A1 widening). The spike's burden of proof is a ratio
durably **greater than 1** — one normalization concept lifting more than one case — before it is allowed
to replace any existing regex path. Every measurement runs on `holdout_dev/v1` (open, 500 cases, no key
needed) via `uv run python -m evals.gsm8k_math.holdout_dev.v1.runner`; the sealed 1,319-case test is never
touched by this spike's iteration loop, per this project's train/dev/test discipline (ADR-0175 and the
`holdout_dev/v1` README's own doctrine). `wrong=0` is non-negotiable at every measurement — any
correct→wrong regression kills the spike's claim regardless of lift elsewhere.
### 5.4 Explicit disclaimers
- **Smith-chart algebra is NOT proposed and stays disclaimed.** ADR-0246 §6 already rules this out by
name ("no Smith-chart algebra") alongside a list of other non-goals (no analytic cast reactance, no
VSWR/atlas work). This spike does not revisit that ruling. Where an "impedance-matching" intuition might
be reached for, the equivalent capability is `conformal_procrustes` (§5.2) — an already-accepted
primitive — not a new Smith-chart library.
- **No "Fibonacci dimensional cascade" is proposed.** The only Accepted Fibonacci work in this repo
(ADR-0242, deterministic evidence-gated operators for optimization/scheduling/multi-scale allocation) is
unrelated to reader surface normalization, and nothing here proposes extending it into a new cascade
mechanism for this purpose.
- **This spike does not touch the sealed test, does not modify the existing regex reader, and does not
start with this ADR.** It requires a separate, explicit ruling before any code is written.
### 5.5 Scope boundary if ruled to proceed
Off-serving spike only, measured against `holdout_dev/v1`, `wrong=0`-gated at every step, with the
existing regex reader continuing to serve unmodified for the duration. Promotion of any spike output to
serving code would itself require its own review and ruling — this ADR does not pre-authorize that step
even conditionally.
## 6. What's ready to build next
Nothing is authorized to start building yet. What exists, verified, at the close of this recalibration:
a clean `main` at the canonical tip with the `#77`/`#78`/`#79` foundations intact and unmodified; the
abandoned branch's debt provably absent; two preserved, re-verified validation targets (the
`has_numeric_token` bug and the 0148 decomposition) waiting for whichever future increment picks them up;
a descriptive (non-deletion) overfit inventory scoping what a normalizer would need to subsume; and a
proposed — not started — geometric surface→canonical normalization spike with its building blocks
verified present, its measurement contract (`wrong=0`-gated generalization ratio on `holdout_dev/v1`)
defined, and its non-goals (Smith-chart algebra, any Fibonacci "dimensional cascade") explicitly
disclaimed. The next action is a human ruling on §5 — whether to authorize the spike, in what scope, and
against what acceptance bar — not further implementation.

103
docs/research/0148-first-conversion-proof.md Normal file
View file

@ -0,0 +1,103 @@
# Case 0148 — correct-for-the-right-reason graph dump (preserved evidence)
**Status**: Reference artifact — not a build, not landed on `main`. Preserved per the recalibration
(ADR-0251) after abandoning `feat/reader-inc2-caseband` (PR #80). See
`docs/research/reader-arc-recalibration-preserved-knowledge-2026-07-19.md` §2 for full context.
The abandoned branch committed this file **empty** (0 bytes) while its own session summary claimed a
graph dump had been committed here. That claim was false. This file replaces the empty placeholder with
the real artifact, generated fresh and independently re-verified — not copied from the branch's claims.
## Case
- **id**: `gsm8k-holdout-dev-v1-0148`
- **problem**: "At a people counting station, the number of people counted on the first day was twice
the total number counted on the second day. If 500 people were counted on the second day, how many
people were counted on the two days?"
- **expected_answer**: `1500.0`
## Provenance
Generated by running `generate.math_candidate_graph.parse_and_solve(text)` against
`origin/feat/reader-inc2-caseband @ e3da148d` (the abandoned branch's tip — the only place this surface
form parses; `main` correctly refuses this case, see below), in a throwaway detached worktree that has
since been removed. Command:
```
uv run python -c "
from dataclasses import asdict
from generate.math_candidate_graph import parse_and_solve
import json
text = 'At a people counting station, the number of people counted on the first day was twice the total number counted on the second day. If 500 people were counted on the second day, how many people were counted on the two days?'
r = parse_and_solve(text)
print('answer:', r.answer)
print('is_admitted:', r.is_admitted)
if r.selected_graph: print(json.dumps(asdict(r.selected_graph), indent=2, default=str))
"
```
## Result (branch code)
```json
{
"answer": 1500.0,
"is_admitted": true,
"refusal_reason": null,
"branches_enumerated": 1,
"branches_admissible": 1,
"selected_graph": {
"entities": ["the first day", "the second day"],
"initial_state": [
{
"entity": "the second day",
"quantity": {"value": 500, "unit": "people"}
}
],
"operations": [
{
"actor": "the first day",
"kind": "compare_multiplicative",
"operand": {
"reference_actor": "the second day",
"delta": null,
"factor": 2.0,
"direction": "times"
},
"target": null
}
],
"unknown": {"entity": null, "unit": "people"}
}
}
```
## Reading the graph — the 3-part correct-for-the-right-reason proof
1. **The conditional-seed bound the *named* reference.** `initial_state` seeds `"the second day" = 500`
— the entity named in "If 500 people were counted **on the second day**" — not a nearest/other/
unnamed entity.
2. **The compare op reads forward.** `operations[0]` binds `actor = "the first day"`,
`reference_actor = "the second day"`, `factor = 2.0`, `direction = "times"` — i.e. *first day = 2 ×
second day*, matching the source's "the number... on the first day **was twice**... on the second
day" directly (not an inverted/guessed direction).
3. **The unknown resolves via summation over exactly these two registers.** `unknown.unit = "people"`
with no entity restriction → sums both days: `500 + (2 × 500) = 1500`. This matches
`expected_answer` because of the binding above, not by coincidence — swapping which day were seeded
would still total 1500 under plain summation, so the answer alone never proves correctness; the
graph's bindings (steps 1–2) are what's being verified here.
## Cross-check: `main` refuses this case (verified, current tip)
```
$ uv run python -c "
from generate.math_candidate_graph import parse_and_solve
text = 'At a people counting station, the number of people counted on the first day was twice the total number counted on the second day. If 500 people were counted on the second day, how many people were counted on the two days?'
r = parse_and_solve(text)
print(r.answer, r.is_admitted, r.refusal_reason)
"
None False recognizer matched but produced no injection for statement: 'At a people counting station, the number of people counted on the first day was twice the total number counted on the second day.' (category=descriptive_setup_no_quantity)
```
`main` has no mass-noun compare actor-binding, so it correctly refuses rather than guessing — consistent
with `wrong=0` discipline. This case remains an open, documented target for a future (non-bespoke)
increment; nothing here re-lands the branch's code.

102
docs/research/reader-arc-overfit-inventory-2026-07-19.md Normal file
View file

@ -0,0 +1,102 @@
# GSM8K reader — overfit inventory (2026-07-19)
**Status**: Inventory only — descriptive, not prescriptive. Nothing on this list is being deleted,
disabled, or modified by this document or the recalibration it accompanies (ADR-0251). Every pattern
below keeps serving unmodified; the reader must keep working until a proven replacement exists.
## Purpose
A read-only survey of `generate/*.py` and `evals/refusal_taxonomy/shape_categories.py` on `main` @
`854ce023`, identifying regex constants, extractor functions, and closed-vocabulary lists whose shape or
motivating comment reads as tailored to one specific GSM8K case (or a small named cluster of cases) rather
than a general grammatical class. This gives the geometric surface→canonical normalization spike proposed
in ADR-0251 §5 — if a human ruling authorizes it — a concrete target list of what "one general operator
replacing N bespoke regexes" would need to subsume before it could claim a positive generalization ratio.
Two entries were spot-checked directly against source during this recalibration and confirmed accurate
(see the underlying commits for full context); the rest were produced by a dedicated read-only survey
citing exact file:line locations.
## Group 1 — mechanisms built to admit one named case (case ID in comment)
| # | File:Line | Name | Why it's bespoke |
|---|---|---|---|
| 1 | `generate/recognizer_match.py:557-599` | `_CROSS_SENTENCE_COMPOSITION_RE` | Section header: "ME-2 — cross-sentence subject binding (**admits case 0019**)"; quotes the exact source sentence ("John adopts a dog from a shelter...requires 3 vet appointments, which cost $400 each") as the spec. |
| 2 | `generate/recognizer_match.py:602-660` | `try_extract_cross_sentence_composition_anchor` | Exists solely to thread the case-0019 discourse subject through to #1. |
| 3 | `generate/recognizer_match.py:2123-2191` (esp. 2136-2148) | `match()` dispatcher's ME-2 branch | Docstring explicitly names case 0019 as the shape being admitted. |
| 4 | `generate/math_candidate_graph.py:676-703`, `:1107-1123` | RAT-1 discourse-subject pre-pass | Comment cites case 0019 ("John adopts a dog") by name as the motivating example for the whole pre-pass. |
| 5 | `generate/recognizer_match.py:1377-1460` | `_MULT_AGG_EACH_WEIGHING_RE` / `_try_extract_each_weighing_anchor` | Header: "Targets the canonical...shape (**case 0047** in the train_sample audit)" — long verb alternation built to cover one problem's wording variants. |
| 6 | `generate/math_candidate_parser.py:2693-2797` | `_COND_OP_Q_RE` / `extract_conditional_op_question_candidates` (ADR-0136.S.2) | Comment: "**Target shape (gsm8k-0042)**" — a whole candidate type + short-circuit solve path built for one case ID. |
| 7 | `generate/recognizer_match.py:1125-1140` | `_COMPOUND_REFUSE_SUBSTRINGS` (ADR-0250 widening entries) | Widening tokens (`" twice the "`, `" thrice the "`, etc.) justified by one verbatim confabulation example quoted in the comment. |
## Group 2 — "Gate A2b/A2c/A2d" question extractors (each is one exact question phrasing)
| # | File:Line | Name | Why it's bespoke |
|---|---|---|---|
| 8 | `generate/math_candidate_parser.py:2872-2877`, `3177-3200` | `_Q_KEEP_ON_HAND_RE` / `_pattern_d_keep_on_hand_candidates` (Gate A2b) | Matches only the literal idiom "keep on hand" — no generalization beyond that exact wording. |
| 9 | `generate/math_candidate_parser.py:2896-2906`, `3149-3174` | `_YIELD_RATE_RE` / `_Q_YIELD_RE` (Gate A2c) | "used to make one X" / "be able to make" is a single idiom, not a grammar class. |
| 10 | `generate/math_candidate_parser.py:2909-2919`, `3121-3146` | `_PEER_PICK_CONDITIONAL_RE` / `_Q_PEER_PICK_RE` (Gate A2d) | Literal phrase match ("friends pick the same amount as her/him"..."pick in all") — reads as built for one specific problem. |
| 11 | `generate/math_candidate_parser.py:985-993`, `1040-1068` | `_FRACTION_GIVE_RE` / `_fraction_give_candidates` (Gate A2b) | Fixed verb ("give") + fixed closed referent set (`that/it/them`). |
| 12 | `generate/math_candidate_parser.py:995-1003`, `1071-1087` | `_HALF_REST_RE` / `_half_rest_candidates` (Gate A2b) | Matches the literal phrase "half of the rest" — a single-sentence idiom, not a generalizable partition rule. |
## Group 3 — entity/label-specific initial-possession shapes
| # | File:Line | Name | Why it's bespoke |
|---|---|---|---|
| 13 | `generate/math_candidate_parser.py:267-285` | `_INITIAL_HAS_LABELED_RE` (ADR-0194) | Comment names three exact source sentences ("Jar A has 28 marbles.", "Section G has 10 cars.", "District 2 has 19 voters.") as the spec. |
| 14 | `generate/math_candidate_parser.py:287-304` | `_INITIAL_THERE_ARE_PREFIX_RE` (ADR-0136.S.4 Shape B) | Built around one quoted example ("In a building, there are a hundred ladies on the first-floor studying.") — optional clauses shaped to that sentence. |
| 15 | `generate/math_candidate_parser.py:1957-1964` | `_CONJ_BAGS_OF_RE` (Gate A2c conjoined) | Hardcoded two-clause "N bags of X and M bags of Y" shape. |
## Group 4 — closed-vocabulary lists sourced from named individual cases
| # | File:Line | Name | Why it's bespoke |
|---|---|---|---|
| 16 | `generate/math_candidate_parser.py:2986-3012` | `_FEMALE_NAMES` / `_MALE_NAMES` | Comment: "Closed-set whitelists **sourced from GSM8K train_sample observation**. Adding a name requires evidence from a refused case." Pronoun resolution by literal name enumeration, not general NER/gazetteer. |
| 17 | `generate/math_candidate_parser.py:2854-2861` | `_PATTERN_C_VERBS` | Closed verb whitelist "observed in GSM8K train_sample," grown one refused case at a time. |
| 18 | `generate/math_roundtrip.py:96-117`, `generate/math_candidate_parser.py:1192-1205` | `NEUTRAL_COUNT_VERBS` / `_COMPARISON_ANCHOR_VERBS` (ADR-0250 increment 1 — **the #78 allowlist**) | Curated verb frozensets, each vetted against one refusal — legitimate and load-bearing (this recalibration keeps it), but still individually case-grown rather than derived from a POS/semantic class. |
## Group 5 — `evals/refusal_taxonomy/shape_categories.py`: category rules justified by 1-4 named cases
Each rule's docstring cites specific `case NNNN` numbers from the 50-case train sample. Spot-checked
directly against source: `_is_unit_partition` (row 19) is explicitly a single-case-justified category —
"the 50-case sample carries **one instance** (case 0002); the rule is retained as a reserve."
| # | File:Line | Rule | Cases cited in docstring |
|---|---|---|---|
| 19 | `:296-311` | `_is_unit_partition` | Only case 0002 — kept as "reserve" |
| 20 | `:264-277` | `_is_nested_question_target` | 0009, 0042 |
| 21 | `:279-293` | `_is_conditional_quantity` | 0009, 0042 (docstring admits it "may return `False` for the entire current corpus") |
| 22 | `:314-325` | `_is_rate_with_currency` | 0001, 0011, 0022 |
| 23 | `:328-352` | `_is_currency_amount` | 0019, 0026, 0028 |
| 24 | `:355-368` | `_is_comparative_with_unit` | 0015, 0016, 0033 |
| 25 | `:371-387` | `_is_fractional_rate_of_change` | 0005, 0012, 0041 (explicitly excludes 0044 by name) |
| 26 | `:390-412` | `_is_temporal_aggregation` | 0013, 0014, 0024, 0050 |
| 27 | `:415-428` | `_is_indefinite_quantity` | 0004, 0040, 0043 |
| 28 | `:431-458` | `_is_multiplicative_aggregation` | 0025, 0045, 0047 |
| 29 | `:461-488` | `_is_discrete_count_statement` | 0023, 0027, 0046 |
| 30 | `:491-500` | `_is_descriptive_setup_no_quantity` | 0003, 0008, 0019 |
These are the taxonomy layer's version of the same pattern: one predicate per small case cluster, with the
literal source sentences quoted as the specification, instead of one regex per case.
## Bonus — non-regex decision heuristic shaped by one scenario
| # | File:Line | Name | Why it's bespoke |
|---|---|---|---|
| 31 | `generate/math_candidate_graph.py:452-467` | `_slot_count` (per-sentence ambiguity tiebreaker) | Docstring: implements "**the give-with-target case**" by name — a general-sounding "most-grounded-slots-wins" rule whose actual justification is one named scenario shape. |
## Explicitly excluded — general/closed, not bespoke
Reviewed and judged to cover broad closed grammatical classes rather than one sentence's literal wording,
so not listed as candidates: `_DAY_ENUM_RE`/`_day_enumeration_candidates` (day-of-week enumeration,
`math_candidate_parser.py:1823-1882`); `_INITIAL_HAS_RE`/`_INITIAL_THERE_ARE_RE` (standard "A has N X" /
"There are N X" shapes, `:145-247`); the `_CAPACITY_RE` family (ADR-0136.S.1 rate/event, `:2397-2508`);
and the currency-grounding logic in `math_roundtrip.py:323-433` (covers multiple currencies/time units
generically). `generate/math_completeness.py` and `evals/refusal_taxonomy/runner.py` were reviewed in
full and contain no case-pinned patterns.
## What this inventory does not claim
This list does not claim every pattern on it is wrong to have, nor that the reader should be rewritten.
The `#77`/`#78` foundations (compare-multiplicative compiler tier, `NEUTRAL_COUNT_VERBS` allowlist) are
sound, reviewed, and stay exactly as they are. This is a map of where the reader's growth has been
case-shaped rather than grammar-shaped, for whoever picks up the geometric-normalization question next.

223
docs/research/reader-arc-recalibration-preserved-knowledge-2026-07-19.md Normal file
View file

@ -0,0 +1,223 @@
# Reader-arc recalibration — preserved knowledge from `feat/reader-inc2-caseband` (PR #80)
**Date**: 2026-07-19
**Status**: Reference notes — validation targets for a future increment, not a build
**Origin**: Extracted before abandoning `feat/reader-inc2-caseband` (PR #80) per the recalibration
decision (see ADR-0251). The branch is **not merged**; nothing below authorizes landing its code.
Everything in this note was re-verified directly against source on `main @ 854ce023` and against the
abandoned branch's tip (`origin/feat/reader-inc2-caseband @ e3da148d`), not copied from the branch's own
claims — see §3 for a place where the branch's own narrative overstated what it verified.
---
## 1. The `has_numeric_token` multiplier-blindness bug (real, general, verified)
### 1.1 Root cause
`classify_sentence()` in `generate/math_candidate_parser.py:2385` (wrapping `has_numeric_token()` at
`:2374`) decides whether a statement sentence is skippable "context" or must carry numeric state:
```python
def has_numeric_token(sentence: str) -> bool:
if re.search(r"\d", sentence):
return True
return bool(_WORD_NUMBER_RE.search(sentence))
```
`_WORD_NUMBER_RE` is built from `WORD_NUMBERS` (`generate/math_roundtrip.py:218-228`) — the closed
cardinal set `zero..ninety, hundred, thousand` plus three ordinal-factor words `half/third/quarter`.
**It does not include multiplicative comparatives**: `twice`, `thrice`, `double`, `triple`,
`quadruple`, etc. A statement sentence whose *only* quantity signal is one of these words — no digit,
no cardinal word — is misclassified `"context"` and therefore treated as safe to drop.
Verified directly on `main`:
```python
>>> from generate.math_candidate_parser import classify_sentence
>>> classify_sentence("Jerry has twice as many apples as Ivan.")
'context'
```
This is the *exact* sentence the existing ADR-0191 comment in `generate/math_candidate_graph.py:670-673`
already names as the risk case ("`Jerry has twice as many … as Ivan` has no digit but a multiplier the
reading must account for") — the comment anticipated the failure mode; `has_numeric_token` does not yet
satisfy it.
### 1.2 Where it bites (and where it doesn't)
`classify_sentence` feeds a filter at `generate/math_candidate_graph.py:704-707`:
```python
numeric_statement_sentences = [s for s in statement_sentences if classify_sentence(s) == "numeric_state"]
if numeric_statement_sentences or not statement_sentences:
statement_sentences = numeric_statement_sentences
```
The guard (`numeric_statement_sentences or not statement_sentences`) only *replaces* `statement_sentences`
when the numeric-only subset is non-empty. Consequence, verified both ways:
- **Single-statement cases (e.g. case 0148 itself) are NOT affected** — if the multiplier-only sentence
is the *only* statement sentence, the numeric-only subset is empty, the guard is false, and the
original sentence (including the multiplier-only one) survives unfiltered. Confirmed: 0148's refusal on
`main` is **not** caused by this filter (see §2 — it's an actor-binding/surface gap instead).
- **Multi-statement cases ARE affected** — if at least one *other* statement sentence has a literal
digit/cardinal, the multiplier-only sentence is silently dropped from `statement_sentences` before any
recognizer/extractor sees it. Verified with a synthetic two-sentence case reproducing the ADR-0191
comment's own example:
```
'Ivan has 5 apples.' -> numeric_state
'Jerry has twice as many apples as Ivan.' -> context (dropped)
```
- **Real-corpus incidence, measured against `evals/gsm8k_math/holdout_dev/v1/cases.jsonl` (500 cases)**:
23 cases have a statement sentence containing only `twice/thrice/double/triple/quadruple` (no digit, no
cardinal) co-occurring with another statement sentence that does have a literal number — the exact shape
that triggers the drop. Examples: `0032`, `0060`, `0096`, `0112`, `0161`, `0165`, `0207` (several of
these — e.g. `0112` "Tim bought twice as many clowns as Rick bought guppies" — are load-bearing compare
clauses, not incidental frequency phrases like `0012`/`0127`/`0196`'s "twice a day"). This list is
**not triaged** — some of the 23 may already refuse/resolve correctly for unrelated reasons, or may be
masked by the completeness guard (§1.3). It is a candidate list for whoever picks this up next, not a
claimed fix count.
### 1.3 Why this has NOT been a `wrong=0` hazard (verified, not assumed)
`uncovered_quantities()` in `generate/math_completeness.py:194` (the ADR-0191 completeness guard) is a
**separate, already-correct** mechanism: it resolves multiplier anchors via the `en_numerics_v1` pack's
`lookup_multiplier` (`math_completeness.py:36-68`), independent of `has_numeric_token`. So even when the
primary filter silently drops a multiplier-only statement, the completeness guard still knows a "twice"
was present in the source, sees the chosen graph never consumed it, and **refuses** ("incomplete reading")
rather than emitting a wrong answer. This is refusal-only by construction (`math_completeness.py:19-23`),
so the bug's practical effect is **missed conversions (false refusals), not wrong answers**.
### 1.4 Minimal fix idea (NOT landed — do not cherry-pick without full-500 validation)
The abandoned branch's own fix (`generate/math_candidate_parser.py`, diff against `main`) was:
```python
_COMPARE_MULT_ANCHOR_TOKENS_RE: Final[re.Pattern[str]] = re.compile(
r"\b(?:" + "|".join(re.escape(k) for k in _ANCHOR_TO_FACTOR.keys()) + r")\b",
re.IGNORECASE,
)
def has_numeric_token(sentence: str) -> bool:
if re.search(r"\d", sentence):
return True
if _COMPARE_MULT_ANCHOR_TOKENS_RE.search(sentence):
return True
return bool(_WORD_NUMBER_RE.search(sentence))
```
i.e. extend `has_numeric_token` to also recognize the closed `_ANCHOR_TO_FACTOR` multiplier-word set
(`twice, thrice, half, quarter, third, ...` — the same closed set the compare-multiplicative compiler
already trusts). This is a **general, shared-primitive fix** (unlike the mass-noun regex in §2, which is
case-bespoke) and is plausibly the right minimal change. **It is deliberately not being re-landed now**:
per the recalibration guardrails, a shared-grounding-primitive change must be proven coverage-neutral-or-
positive AND `wrong=0` on the full 500 before it lands, and it belongs together with whatever compare op
actually consumes the newly-admitted statements (landing the classifier fix alone, with nothing downstream
able to extract the now-visible sentence, only changes *which* refusal reason a case gets — not its
correct/wrong/refused bucket). Whoever revisits this should re-run the full 500 with only this isolated
change and confirm zero `correct → refused/wrong` regressions before considering it a candidate for main.
---
## 2. Case 0148 — solves correct-for-the-right-reason (verified target, not landed)
**Case**: `gsm8k-holdout-dev-v1-0148` — *"At a people counting station, the number of people counted on
the first day was twice the total number counted on the second day. If 500 people were counted on the
second day, how many people were counted on the two days?"* — expected answer **1500**.
**Why it refuses on `main` today** (verified): the statement sentence *does* survive the
`has_numeric_token` filter (§1.2 — single-statement case), and the recognizer does match its shape, but
produces no injection:
```
refusal_reason: "recognizer matched but produced no injection for statement: 'At a people counting
station, the number of people counted on the first day was twice the total number counted on the second
day.' (category=descriptive_setup_no_quantity)"
```
The real gap is that `main`'s compare-multiplicative extractor requires a `ProperName` actor
(`extract_proper_noun_subject`), and 0148's actors are mass-noun/temporal descriptions ("the first day" /
"the second day") — not proper nouns, and not covered by any pattern on `main`. This is a genuine surface
gap, not a solver gap.
**Decomposition (the correct-for-the-right-reason target)**:
- seed: **second day = 500** (bound from the conditional clause "If 500 people were counted on the second
day")
- compare: **first day = 2 × second day** (forward multiplicative compare)
- question: summation over both registers → **500 + 1000 = 1500**
**Verified by running the abandoned branch's code** (`origin/feat/reader-inc2-caseband @ e3da148d`, via a
throwaway detached worktree, `uv run python -c 'from generate.math_candidate_graph import
parse_and_solve; ...'`, worktree since removed) — this is the actual graph dump, not the empty file the
branch committed (see §3):
```json
{
"answer": 1500.0,
"is_admitted": true,
"refusal_reason": null,
"branches_enumerated": 1,
"branches_admissible": 1,
"selected_graph": {
"entities": ["the first day", "the second day"],
"initial_state": [
{"entity": "the second day", "quantity": {"value": 500, "unit": "people"}}
],
"operations": [
{
"actor": "the first day",
"kind": "compare_multiplicative",
"operand": {"reference_actor": "the second day", "delta": null, "factor": 2.0, "direction": "times"},
"target": null
}
],
"unknown": {"entity": null, "unit": "people"}
}
}
```
This confirms the 3-part correct-for-the-right-reason standard: the conditional-seed bound the *named*
reference (second day = 500, not a nearest/other entity), the compare op reads **forward** (first =
2×second, matching the source's "first day was twice the ... second day" — not an inverted reading), and
the unknown resolves via summation over exactly these two registers — 1500 from *this* graph, not a
coincidental path. On `main` (no mass-noun frame), the same case correctly refuses rather than guessing.
**What produced this conversion on the branch** (for context — not being landed): a bespoke
`_COMPARE_MASSNOUN_RE` regex family (`generate/math_candidate_parser.py`), a relaxation of the
proper-noun-only actor gate in `inject_comparative_multiplicative`
(`generate/recognizer_anchor_inject.py`) to allow mass-noun actors when that regex matches, and Gate A1
quantity-marker widening (`_COMPARATIVE_TOKENS` in `generate/recognizer_match.py` and
`evals/refusal_taxonomy/shape_categories.py`). All of this is bespoke to the 0148 surface shape and is the
kind of pattern the geometric-normalization direction (ADR-0251) is meant to supersede rather than
multiply — see the overfit inventory.
**Full-500 measurement on the branch** (`evals/gsm8k_math/holdout_dev/v1/report.json` diff): exactly one
case flipped, `0148: refused → correct`; counts `correct 5→6, refused 495→494, wrong=0→0` (held). Clean
single-case signature, no compensating regressions — this is genuine evidence, reproduced here for the
record since it lived only in a diff on an abandoned branch.
---
## 3. A correction to the branch's own narrative (why "verify, don't claim" matters here)
The branch's session-break summary (`docs/sessions/session-break-summary-2026-07-19-1310.md`, branch tip)
claimed: *"Graph dump for case 0148 committed to `docs/research/0148-first-conversion-proof.md` as proof
of correct-for-the-right-reason binding."* This is false — the committed file is **0 bytes**
(`git cat-file -s origin/feat/reader-inc2-caseband:docs/research/0148-first-conversion-proof.md` → `0`).
The actual graph dump was never captured to disk on the branch; only the narrative claimed it was. The
real artifact now lives in `docs/research/0148-first-conversion-proof.md` on this recalibration branch
(§2 above, generated fresh from the branch's code and independently re-verified against `main`'s refusal
behavior for the same case).
This is also why the human reviewer's second PR-review pass (PR #80 comment) flagged "commit evidence
before merge" as a blocker rather than accepting the summary's claim at face value — the discrepancy
between claimed and actual state is exactly the failure mode "verify, don't claim" exists to catch.
---
## 4. Disposition
Nothing in this note authorizes building on it. It exists so that a future, non-bespoke increment (the
geometric-normalization spike proposed in ADR-0251, or whatever a human ruling selects) has real,
re-verified targets — the 0148 decomposition and the `has_numeric_token` fix idea — instead of having to
rediscover them from an abandoned branch.