From beafeb5ad236ac62610e91f69766e24573f93eb8 Mon Sep 17 00:00:00 2001 From: Shay Date: Sat, 23 May 2026 20:21:32 -0700 Subject: [PATCH 1/3] docs(ADR-0136): statement-layer corridor + S.0 taxonomy + resolve math-file conflicts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add ADR-0136-statement-layer-corridor.md: corridor overview, S.0-S.4 phase table, taxonomy summary (23 context-filler, 4 rate-class, 5 compound, 17 long-tail), standing invariants (admitted_wrong==0, context-filler safety rail, honest delta). - Restore generate/math_*.py to HEAD (main) — stash-pop conflict markers were cosmetic comment rewording; upstream version is authoritative. - Stage docs/reviewers.yaml math_expert_claims addition (already authored). --- .../ADR-0136-statement-layer-corridor.md | 122 ++++++++++++++++++ docs/reviewers.yaml | 11 ++ 2 files changed, 133 insertions(+) create mode 100644 docs/decisions/ADR-0136-statement-layer-corridor.md diff --git a/docs/decisions/ADR-0136-statement-layer-corridor.md b/docs/decisions/ADR-0136-statement-layer-corridor.md new file mode 100644 index 00000000..1308c93c --- /dev/null +++ b/docs/decisions/ADR-0136-statement-layer-corridor.md @@ -0,0 +1,122 @@ +# ADR-0136 — Statement-Layer Corridor: Graduated GSM8K Admission via Parser Extension + +**Status:** Active +**Date:** 2026-05-23 +**Author:** CORE agents + reviewers +**Parent:** [ADR-0131.G](./ADR-0131.G-gsm8k-coverage-probe.md) +**Depends on:** +[ADR-0115](./ADR-0115-math-problem-parser-and-graph.md), +[ADR-0116](./ADR-0116-deterministic-solver.md), +[ADR-0117](./ADR-0117-solution-trace-verifier.md) + +--- + +## Context + +ADR-0131.G pinned the GSM8K coverage probe at **0/50 admission** and established +`admitted_wrong == 0` as a standing architectural invariant. The G.x capability +axes (G.1–G.5) have landed on main; as of ADR-0131.5 (probe retirement), the +GSM8K probe is no longer a per-iteration gate — it activates only when a new +iteration claims Δadmission_rate ≥ 0.02. + +A taxonomy pass over all 50 refused cases (S.0) produced the following breakdown: + +| Primary barrier | Count | Notes | +|---|---|---| +| `context_filler` | 23 | Narrative scene-setters; parser's refusal is correct | +| `compound_statement` | 5 | Two operations in one sentence | +| rate/capacity/price class | 4 | Direct targets of S.1 | +| `distributive_multiply` | 1 (+5 secondary) | N bags × M items each | +| diverse long-tail | 17 | Age anchors, goal statements, multi-step chains, etc. | + +The 23 context-filler cases will not be addressed by statement-layer work alone — +they need semantic classification of scene-setting sentences, which is a separate +architectural concern. **The safety rail stands: if a sentence cannot parse into +a numeric initial-state candidate, the problem is refused.** + +The G.x micro-extension approach has been retired. Each G.x axis targeted one +regex shape in isolation; the return per axis was tiny and the axes were not +composable into a coherent parser narrative. The Statement-Layer Corridor replaces +G.x with a phased taxonomy-driven program that: + +1. Classifies all refusals before writing any code. +2. Targets the highest-signal unlockable barriers first. +3. Ships each phase as a self-contained unit with a curated axis lane, `wrong == 0` + gate, and honest GSM8K delta. +4. Never attempts to bypass the context-filler safety rail. + +--- + +## Decision + +Organize parser extension work as a **corridor of phases** rooted at ADR-0136: + +| Phase | ADR | Scope | Primary barrier targeted | +|---|---|---|---| +| S.0 | (this doc) | Taxonomy — deterministic classification of 50 refused cases | — | +| S.1 | ADR-0136.S.1 | Rate/event statements — capacity-rate + earnings-rate shapes | rate class (≤4 cases) | +| S.2 | ADR-0136.S.2 | Temporal statements — time anchors, duration expressions | time/age long-tail | +| S.3 | ADR-0136.S.3 | Compound statements — two operations in one sentence | `compound_statement` (5 cases) | +| S.4 | ADR-0136.S.4 | Coreference — pronoun + ellipsis resolution across sentences | cross-sentence barriers | + +Phases S.2–S.4 are deferred; scope and sequencing will be revisited after S.1 lands. + +--- + +## Taxonomy (S.0) + +Stored at: `evals/gsm8k_math/train_sample/v1/refusal_taxonomy.json` + +Schema v1. Each record carries: +- `case_id` — GSM8K case identifier +- `primary_barrier` — the single barrier that causes refusal even if all others were resolved +- `secondary_barriers` — additional barriers present in the problem +- `notes` — free-text rationale + +**Key finding.** The case with the shallowest barrier is `gsm8k-0014`: + +``` +Bob can shuck 10 oysters in 5 minutes. +How many oysters can he shuck in 2 hours? +``` + +This is a single-statement capacity-rate problem with a pronoun question. It is the +proof case for S.1: it must admit with `answer == 240.0`. + +--- + +## Invariants + +These invariants are non-negotiable across all corridor phases: + +- **`admitted_wrong == 0`** — no GSM8K case is admitted with a wrong answer. +- **Context-filler safety rail** — sentences without parseable numeric initial state + are refused; no soft-fail or skip-and-continue. +- **Honest delta** — each phase's PR body states the exact pre/post GSM8K admission + count; no rounding, no "approximately". +- **No solver/graph/verifier changes** — rate-path short-circuits and new extractors + live in `math_candidate_parser.py` and `math_candidate_graph.py` only. + +--- + +## Consequences + +- The G.x capability axis namespace is closed. New axes use the S.x naming convention. +- Each S.x phase ships its own curated axis lane at + `evals/math_capability_axes/S_/v1/cases.jsonl`. +- The GSM8K probe re-activates when a phase claims ≥1 new admission + (Δadmission_rate ≥ 0.02); otherwise it stays retired per ADR-0131.5. +- S.1 is the only phase that can honestly claim unlocking the context-filler-gated + cases once sentence-semantic classification lands (out of S.x scope). + +--- + +## Deferred + +- Context-filler gated problems (23 cases) — requires semantic classification of + narrative scene-setting sentences; architecturally separate from statement parsing. +- Conditional branching (`if she works more than 8 hours`) — needs branching semantics + in the solver. +- Percentage/interest rates — needs decimal arithmetic extension. +- Multi-statement rate problems (duration in separate sentence from capacity) — needs + coreference, addressed in S.4. diff --git a/docs/reviewers.yaml b/docs/reviewers.yaml index eec4e090..b2d9c27f 100644 --- a/docs/reviewers.yaml +++ b/docs/reviewers.yaml @@ -51,3 +51,14 @@ audit_passed_claims: evidence_revision: "adr-0124:reviewed:2026-05-22" signed_by: shay-j claim_digest: "17e24436b6875b89f6d1a5c2992557413c7ef456250f549d463159f54438c407" +math_expert_claims: + - domain_id: mathematics_logic + evidence_lanes: + - math_symbolic_equivalence/v1 (public) + - math_symbolic_equivalence/v1 (sealed) + - math_teaching_corpus/v1 + - math_bounded_grammar/v1 + evidence_revision: "adr-0120-math:reviewed:2026-05-23" + signed_by: shay-j + claim_digest: "94149794e8c19896851e062cf1f921cfa9ba04770b674bc3b4c33023f7c7331b" + \ No newline at end of file From 421ae8278165383ba01fa2e695e1a393398746a6 Mon Sep 17 00:00:00 2001 From: Shay Date: Sat, 23 May 2026 20:22:22 -0700 Subject: [PATCH 2/3] =?UTF-8?q?docs(briefs):=20archive=20all=202026-05-23?= =?UTF-8?q?=20parallel-dispatch=20briefs=20(L1=E2=80=93L17=20+=20README)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../L1-gemini-B2-teaching-corpus-eval.md | 39 +++ .../L10-ADR-0131-G2-comparatives.md | 44 ++++ .../L11-ADR-0131-G3-numerics.md | 45 ++++ .../L12-ADR-0131-G4-multi-clause.md | 49 ++++ .../L13-ADR-0131-G31-numerics-extensions.md | 52 ++++ .../L14-ADR-0114a-5-perturbation-suite.md | 71 ++++++ .../L15-ADR-0114a-2-ood-ratio.md | 72 ++++++ ...DR-0131-G5-aggregate-answer-composition.md | 139 +++++++++++ .../L17-ADR-0131-G6-rate-capacity.md | 135 ++++++++++ .../L17-ADR-0136-S1-rate-event-statements.md | 236 ++++++++++++++++++ .../L2-opus2-binding-graph-phase1.md | 49 ++++ .../L3-sealed-holdout-B1.md | 42 ++++ .../L4-opus2-binding-graph-phase2-adapter.md | 48 ++++ .../L5-opus2-binding-graph-phase3-units.md | 57 +++++ .../L6-gemini-B2-dataset-enrichment.md | 47 ++++ ...s2-binding-graph-phase4-question-target.md | 61 +++++ .../L8-B3-bounded-grammar-word-problems.md | 61 +++++ ...-ADR-0131-G1-verb-classes-initial-state.md | 40 +++ docs/briefs/parallel-2026-05-23/README.md | 42 ++++ 19 files changed, 1329 insertions(+) create mode 100644 docs/briefs/parallel-2026-05-23/L1-gemini-B2-teaching-corpus-eval.md create mode 100644 docs/briefs/parallel-2026-05-23/L10-ADR-0131-G2-comparatives.md create mode 100644 docs/briefs/parallel-2026-05-23/L11-ADR-0131-G3-numerics.md create mode 100644 docs/briefs/parallel-2026-05-23/L12-ADR-0131-G4-multi-clause.md create mode 100644 docs/briefs/parallel-2026-05-23/L13-ADR-0131-G31-numerics-extensions.md create mode 100644 docs/briefs/parallel-2026-05-23/L14-ADR-0114a-5-perturbation-suite.md create mode 100644 docs/briefs/parallel-2026-05-23/L15-ADR-0114a-2-ood-ratio.md create mode 100644 docs/briefs/parallel-2026-05-23/L16-ADR-0131-G5-aggregate-answer-composition.md create mode 100644 docs/briefs/parallel-2026-05-23/L17-ADR-0131-G6-rate-capacity.md create mode 100644 docs/briefs/parallel-2026-05-23/L17-ADR-0136-S1-rate-event-statements.md create mode 100644 docs/briefs/parallel-2026-05-23/L2-opus2-binding-graph-phase1.md create mode 100644 docs/briefs/parallel-2026-05-23/L3-sealed-holdout-B1.md create mode 100644 docs/briefs/parallel-2026-05-23/L4-opus2-binding-graph-phase2-adapter.md create mode 100644 docs/briefs/parallel-2026-05-23/L5-opus2-binding-graph-phase3-units.md create mode 100644 docs/briefs/parallel-2026-05-23/L6-gemini-B2-dataset-enrichment.md create mode 100644 docs/briefs/parallel-2026-05-23/L7-opus2-binding-graph-phase4-question-target.md create mode 100644 docs/briefs/parallel-2026-05-23/L8-B3-bounded-grammar-word-problems.md create mode 100644 docs/briefs/parallel-2026-05-23/L9-ADR-0131-G1-verb-classes-initial-state.md create mode 100644 docs/briefs/parallel-2026-05-23/README.md diff --git a/docs/briefs/parallel-2026-05-23/L1-gemini-B2-teaching-corpus-eval.md b/docs/briefs/parallel-2026-05-23/L1-gemini-B2-teaching-corpus-eval.md new file mode 100644 index 00000000..64fc8d44 --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L1-gemini-B2-teaching-corpus-eval.md @@ -0,0 +1,39 @@ +# L1 brief — Gemini — ADR-0131.2 Benchmark 2 (teaching-corpus math eval) + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-adr-0131-2-teaching-eval -b feat/adr-0131-2-teaching-corpus-eval origin/main +cd ../core-adr-0131-2-teaching-eval +``` + +**Scope.** Ship the second of three benchmarks for the composite math-expert promotion gate from ADR-0131. This benchmark measures whether the *teaching/replay loop itself* can carry math content end-to-end — propose → ratify → replay-equivalent on a small math teaching corpus. It is **not** a parser-shape benchmark and **not** a symbolic-equivalence benchmark (#167 already covers that). + +**Reference docs (read these, only these):** +1. `docs/decisions/ADR-0131-math-expert-rebench.md` — the composite-gate framing. +2. `evals/math_symbolic_equivalence/v1/` (the B1 lane on main, post-#167) — copy its runner/report/test shape exactly. Mirror, don't innovate. + +**What to ship:** +- `teaching/math_corpora/math_teaching_v1.jsonl` — ~20–30 hand-curated math teaching chains over a single bounded domain (arithmetic identity / linear equation / algebraic rewrite — pick one and stay in it). Each chain is propose-shape, ratification-ready. +- `evals/math_teaching_corpus/v1/{runner.py,cases.jsonl,README.md}` — CLI runner with `report.json` writeback, exit 0/1 on gate pass/fail. Lane gate = replay-equivalence holds across every chain AND `wrong == 0`. Mirror the B1 README's section structure. +- `tests/test_adr_0131_2_teaching_corpus_lane.py` — 6–10 lane tests: dataset integrity, exit criterion, replay-byte-equality of report.json, no chain proposes outside the bounded domain. +- `docs/decisions/ADR-0131.2-teaching-corpus-eval.md` — short ADR; cite ADR-0131 parent and ADR-0064 (cross-pack teaching) as precedent. + +**Hard constraints:** +- **Bounded domain only.** Pick one math micro-domain; no GSM8K-style word problems. Word problems are B3's job (binding graph), not yours. +- **Replay-equivalence is the gate, not %.** A 100% lane on 20 chains beats a 60% lane on 200. Curate hard. +- **No runtime path changes.** This is a new eval lane + new corpus + new ADR. Touch nothing in `core/`, `chat/`, `generate/`, `algebra/`. +- **Field invariant untouched.** Don't run anywhere near `versor_condition`. +- **Determinism.** `report.json` byte-equal across runs (lane test asserts this). + +**Out of scope (do not touch):** +- B1 hardening (PR #169 — different agent's lane). +- Binding-graph implementation (L2 — different agent's lane). +- B1 sealed holdout (L3 — different agent's lane). +- Promotion-gate wiring (ADR-0131.4 — sequential, after all three benchmarks land). + +**Target branch.** PR against `main`. Title: `feat(ADR-0131.2): teaching-corpus math eval — lane PASSED /`. Body must include lane result + scope-discipline section matching #167's framing. + +**Exit criterion.** PR opens with CI green, lane runner exits 0, `wrong == 0`, `replay_equivalent_count == total_chains`, and the new ADR is included. + +**Do not stack on another agent's branch.** Target main directly. If you need anything from L2 or L3, the answer is no — design around the gap. diff --git a/docs/briefs/parallel-2026-05-23/L10-ADR-0131-G2-comparatives.md b/docs/briefs/parallel-2026-05-23/L10-ADR-0131-G2-comparatives.md new file mode 100644 index 00000000..f4b4a6c8 --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L10-ADR-0131-G2-comparatives.md @@ -0,0 +1,44 @@ +# L10 brief — ADR-0131.G.2 — Capability axis: comparatives (additive + multiplicative) + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-adr-0131-g2-comparatives -b feat/adr-0131-g2-comparatives origin/main +cd ../core-adr-0131-g2-comparatives +``` + +**Scope.** Capability-axis iteration: land **candidate emitters** for `compare_additive` and `compare_multiplicative`. The round-trip verb tables (`COMPARE_ADDITIVE_ANCHORS`, `COMPARE_MULTIPLICATIVE_ANCHORS`) and the `Comparison` operation shape (`generate/math_problem_graph.py:128`) already exist — what's missing is the candidate-parser side. The comment at `generate/math_candidate_parser.py:30` explicitly flags this as a deferred phase. + +Comparatives are an **operation kind** the binding graph already admits; the architectural extension is purely on the parse side. Target shapes (closed set): + +- Additive: ` has N more than `, ` has N fewer than `, ` has N additional `. +- Multiplicative: ` has twice as many as `, ` has N times as many as `, ` has half as many as `. +- Nested compositional (load-bearing — appears in baseline refusal `Jen has 10 more ducks than four times…`): ` has N more than M times 's `. Treat as a *composed* `compare_additive(EntityA, compare_multiplicative(EntityB, M))` structure. + +**Reference docs (read these, only these):** +1. `docs/decisions/ADR-0131.G-gsm8k-coverage-probe.md` — iteration discipline (single axis, own coverage cases, GSM8K admission strictly increases or a refused-reason family is deliberately reduced, `admitted_wrong == 0`). +2. `generate/math_problem_graph.py` (Comparison) + `generate/math_roundtrip.py` (anchor tables) — the contract the new emitter must satisfy. Do **not** redefine direction vocab; emit into the existing four `direction ∈ {more,fewer,times,fraction}` slots. + +**What to ship:** +- **Parser extension** in `generate/math_candidate_parser.py`: new `_compare_additive_candidates` / `_compare_multiplicative_candidates` extractors emitting `CandidateOperation` records whose `op.kind` is `compare_additive` / `compare_multiplicative`. Regex specificity order documented; nested composition produces **two** candidates and the round-trip layer / binding graph (Phase 3, ADR-0134) picks the admissible composition. +- **Curated coverage cases** at `evals/math_capability_axes/G2_comparatives/v1/cases.jsonl` (~25 cases): ≥4 per direction (`more`, `fewer`, `times`, `fraction`) + ≥3 nested-composition cases + ≥4 refusal cases for **paraphrases outside the closed set** (e.g. "as many … as", "compared to", "in comparison with"). The refusal cases pin the scope boundary. +- **Runner + report** at `evals/math_capability_axes/G2_comparatives/v1/`. +- **Tests** at `tests/test_adr_0131_G2_comparatives.py` (~12): per-direction at-least-one passing, nested-composition at-least-one passing, refusal cases all refuse with typed parser error, `wrong == 0`, replay byte-equality, **GSM8K probe re-run** with `admission_rate` strictly increases OR `refused_reasons_top` for comparative-shape clauses strictly decreases (whichever is honest — pick one in the ADR and gate on it). +- **ADR** `docs/decisions/ADR-0131.G.2-comparatives.md`. Document closed-set anchor alternation; explicitly call out which paraphrases are deferred and why (they are not in the round-trip table either — admitting them would breach `wrong == 0`). +- **Refresh** `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json`. + +**Hard constraints:** +- **Direction vocab is closed** to the four `Comparison.direction` literals. No new literals. +- **`wrong == 0`** on the new axis and the probe. Nested-composition admissibility is the round-trip layer's job — do not bypass it. +- **No solver / binding-graph changes.** If a comparative parses but does not solve, that's a downstream gap → file it as a follow-up ADR, do **not** stub the solver. +- **Specificity order documented.** Multiplicative anchors that overlap with additive (`twice` vs `two more`) must have deterministic precedence pinned in the ADR. +- **No new modules under `algebra/`, `chat/`, `core/`.** +- **Determinism.** Report byte-equal across runs. + +**Out of scope:** verb classes for initial state (L9/G.1), rate verbs (L11/G.3), multi-clause distributive subjects (L12/G.4), implicit-comparison shapes ("the same number of … as"). + +**Target branch.** PR against `main`. Title: `feat(ADR-0131.G.2): comparative operations (additive + multiplicative) — admission N/50 (Δ+N)`. Body: per-direction case counts, refusal-set documentation, admission delta or refused-reason-family delta, link to ADR. + +**Exit criterion.** CI green; comparatives runner exits 0 with `wrong == 0`; chosen GSM8K-probe gate (admission ↑ OR comparative-clause refusals ↓) satisfied; B3 + L9/G.1 axis lanes unchanged; refreshed coverage report committed. + +**Do not stack on another agent's branch.** Target main directly. diff --git a/docs/briefs/parallel-2026-05-23/L11-ADR-0131-G3-numerics.md b/docs/briefs/parallel-2026-05-23/L11-ADR-0131-G3-numerics.md new file mode 100644 index 00000000..4cb281b9 --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L11-ADR-0131-G3-numerics.md @@ -0,0 +1,45 @@ +# L11 brief — ADR-0131.G.3 — Capability axis: numeric literals (money, fractions, compound numbers) + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-adr-0131-g3-numerics -b feat/adr-0131-g3-numerics origin/main +cd ../core-adr-0131-g3-numerics +``` + +**Scope.** Capability-axis iteration: extend the parser's `` slot to recognize numeric literal shapes that the baseline refuses on, by **consuming the existing `en_numerics_v1` pack** (ADR-0128) — *not* by hard-coding new regexes inline. The baseline `refused_reasons_top` shows clauses like `Tina makes $18.00 an hour`, `Aaron and his brother Carson each saved up $40`, `In one hour, Addison mountain's temperature will decrease to 3/4 of its temp…`, `Mandy started reading books with only 8 pages` (compound), `Allison … uploads 10 one-hour videos` (hyphenated compound), `Francine has five full boxes of crayons and 5 loose crayons` (word-number + adjective + bag-quantifier). + +Target literal classes (closed set, drawn from `en_numerics_v1`): + +- **Money:** `$N`, `$N.NN`, `N dollars`, `N cents`. Currency symbol → unit lift via `en_units_v1`. +- **Fractions:** `N/M`, `N/M of `, plus word forms already in `en_numerics_v1` (`one-half`, `three-quarters`). +- **Word-number compositions:** `five full boxes` — number + adjective + unit. Adjective is part of the unit phrase (`full boxes` ≢ `boxes`) per ADR-0127 substance-qualifier precedent. +- **Hyphenated compound numerics:** `one-hour`, `10 one-hour` — adjectival numeric modifying a head noun. + +**Reference docs (read these, only these):** +1. `docs/decisions/ADR-0131.G-gsm8k-coverage-probe.md` — iteration discipline. +2. `docs/decisions/ADR-0127-units-pack-and-units-aware-parser.md` + `docs/decisions/ADR-0128-numerics-pack.md` — the packs that already exist; the work is **consuming them at the parser layer**, not extending them. + +**What to ship:** +- **Parser extension** in `generate/math_candidate_parser.py`: widen the value-slot matcher to consult `en_numerics_v1.lexicon` (money + fraction + compound forms) before falling back to the existing integer/word-number regex. Currency-symbol → unit promotion routes through the `en_units_v1` resolver. The pack lookup is **deterministic, ordered, and cached** at parser-construction time; no I/O on hot path. +- **Curated coverage cases** at `evals/math_capability_axes/G3_numerics/v1/cases.jsonl` (~25): ≥5 per literal class + ≥5 refusal probes (e.g. `$N.NNNN` with too many decimals, fractions like `N/0`, ambiguous hyphenations like `one-hour-old`). Refusal probes pin the scope. +- **Runner + report** at `evals/math_capability_axes/G3_numerics/v1/`. +- **Tests** at `tests/test_adr_0131_G3_numerics.py` (~12): per-literal-class at-least-one passing, refusal probes all refuse typed, `wrong == 0`, replay byte-equality, **GSM8K probe re-run with admission strictly increases OR money/fraction-shape refusals strictly decrease** (declare in ADR, gate on it). +- **ADR** `docs/decisions/ADR-0131.G.3-numerics.md`. Cite ADR-0127/0128 packs; document the closed literal classes; pin currency-symbol → unit mapping; list every deferred shape. +- **Refresh** `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json`. + +**Hard constraints:** +- **Pack-driven, not regex-spam.** Every recognized literal must trace to an entry in `en_numerics_v1` (or to the existing integer/word-number regex). No silent inline alternations. +- **`wrong == 0`** on the new axis and the probe. If a fraction `N/0` slips through and the solver divides — that's a parser bug, fix the parser, do not weaken the verifier. +- **Currency → unit composition** is deterministic. `$18.00` parses as `value=18.00, unit=USD` (or whatever `en_units_v1` canonical id is) — pin the exact unit id in the ADR and assert in tests. +- **Scope is literals only.** "Each saved up $40" is in scope for the literal `$40`; the distributive-subject "each" is **out of scope** and routes to L12/G.4. The case may still refuse on the distributive — that's correct, and it should refuse with a **non-literal** reason (the literal layer succeeded). +- **No new modules under `algebra/`, `chat/`, `core/`.** No changes to packs themselves (consumption, not extension). +- **Determinism.** Pack lookup ordered; report byte-equal across runs. + +**Out of scope:** verb classes (L9/G.1), comparatives (L10/G.2), multi-clause / distributive subjects (L12/G.4), percentages (`30%`), scientific notation, locale-specific separators. + +**Target branch.** PR against `main`. Title: `feat(ADR-0131.G.3): numeric literals (money + fractions + compounds) — admission N/50 (Δ+N)`. Body: per-literal-class case counts, currency-symbol unit-id mapping, admission or refusal-family delta, link to ADR. + +**Exit criterion.** CI green; numerics runner exits 0 with `wrong == 0`; chosen GSM8K-probe gate satisfied; B3 + L9/G.1 + L10/G.2 lanes unchanged; refreshed coverage report committed. + +**Do not stack on another agent's branch.** Target main directly. diff --git a/docs/briefs/parallel-2026-05-23/L12-ADR-0131-G4-multi-clause.md b/docs/briefs/parallel-2026-05-23/L12-ADR-0131-G4-multi-clause.md new file mode 100644 index 00000000..196b102a --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L12-ADR-0131-G4-multi-clause.md @@ -0,0 +1,49 @@ +# L12 brief — ADR-0131.G.4 — Capability axis: multi-clause composition (conjunctions, distributive subjects, embedded quantifier phrases) + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-adr-0131-g4-multi-clause -b feat/adr-0131-g4-multi-clause origin/main +cd ../core-adr-0131-g4-multi-clause +``` + +**Scope.** Capability-axis iteration: extend the **candidate parser + binding graph** to handle within-sentence composition that the per-statement parser currently refuses on. Baseline clusters: `Aaron and his brother Carson each saved up $40` (conjoined subject + distributive `each`), `Francine has five full boxes of crayons and 5 loose crayons` (conjoined object NPs sharing a verb), `Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in each bag` (embedded quantifier `with N in each ` + conjunction). + +Target shapes (closed set): + +- **Conjoined subjects with `each`:** ` and each .` → emits **two** `InitialPossession` candidates (one per actor), same `(N, unit)`. +- **Conjoined object NPs sharing a verb:** ` has and .` → emits **two** `InitialPossession` candidates for the same entity. +- **Embedded quantifier phrases:** ` has with in each .` → emits a derived `InitialPossession` with `value = N*M, unit = ` **only if** the round-trip filter admits the product. The multiplication is a candidate, not a guarantee; the binding graph picks admissible compositions. +- **Conjoined embedded quantifiers:** ` has with in each and with in each.` → emits two derived candidates and a sum candidate. + +This is the **highest-risk axis** of the four — multi-clause composition is where confabulation risk is highest. Refusal-first stays paramount; admission gains must be small and load-bearing, not maximum-rate-chasing. + +**Reference docs (read these, only these):** +1. `docs/decisions/ADR-0131.G-gsm8k-coverage-probe.md` — iteration discipline. The "smell test" (admission moves on GSM8K but new axis cases don't all pass → reject) bites hardest here. +2. `generate/math_candidate_graph.py` — the candidate-graph topology. New candidates must compose through the same graph; nothing in the binding/admissibility layer changes. + +**What to ship:** +- **Parser extension** in `generate/math_candidate_parser.py`: three new extractors (`_conj_subject_each_candidates`, `_conj_object_candidates`, `_embedded_quantifier_candidates`) emitting multiple candidates per match. Source-span provenance covers the full sentence for each candidate. +- **Optional graph-side note** in `generate/math_candidate_graph.py` (read-only audit; only edit if a composed candidate is unreachable through existing edges — if so, the edge addition is a one-line widening, *not* a new admissibility rule). Decision to edit or not is documented in the ADR. +- **Curated coverage cases** at `evals/math_capability_axes/G4_multi_clause/v1/cases.jsonl` (~30 cases): ≥6 per shape + ≥6 refusal probes for shapes that look multi-clause but are **not** in the closed set (e.g. cross-sentence coreference `Aaron has 5. He gives 2 to Bob.`, ambiguous `each` scope, three-way conjunctions). Refusal probes are load-bearing — they pin the scope boundary the architecture refuses to cross. +- **Runner + report** at `evals/math_capability_axes/G4_multi_clause/v1/`. +- **Tests** at `tests/test_adr_0131_G4_multi_clause.py` (~15): per-shape at-least-one passing, refusal probes refuse typed, **`wrong == 0` (especially load-bearing here)**, replay byte-equality, **GSM8K probe re-run with admission strictly increases OR multi-clause refusals strictly decrease** (declare in ADR, gate on it), B3 + G.1/G.2/G.3 lanes unchanged. +- **ADR** `docs/decisions/ADR-0131.G.4-multi-clause.md`. Cite ADR-0131.G parent and ADR-0126 (candidate graph). Pin the closed shape set; document the `each`-scope policy (always distributive, never collective — refuse collective readings); document why cross-sentence coreference stays deferred. +- **Refresh** `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json`. + +**Hard constraints:** +- **`wrong == 0`** is non-negotiable. Multi-candidate emission means the round-trip filter does more work — if any composed candidate slips a wrong answer, **remove the shape**, do not weaken the filter. +- **Closed shape set.** Every recognized multi-clause structure matches exactly one of the listed extractors. No paraphrase tolerance. +- **No cross-sentence state.** This axis is strictly within-sentence. Pronoun/coreference across sentences stays refused. +- **Distributive `each` only.** Collective readings (`Aaron and Carson saved $40 together`) must refuse — explicit adversarial probe required. +- **No solver changes.** If a multi-clause case parses but does not solve, file as follow-up ADR. Do not stub. +- **No new modules under `algebra/`, `chat/`, `core/`.** +- **Determinism.** Multi-candidate ordering pinned; report byte-equal across runs. + +**Out of scope:** verb classes (L9/G.1), comparatives (L10/G.2), numeric literals (L11/G.3), cross-sentence coreference, ellipsis (`Aaron has 5, Carson 3`), three-way+ conjunctions, collective readings. + +**Target branch.** PR against `main`. Title: `feat(ADR-0131.G.4): multi-clause composition — admission N/50 (Δ+N)`. Body: per-shape case counts, refusal-set documentation, admission or refusal-family delta, explicit acknowledgement that this is the highest-risk axis and the `wrong == 0` evidence to back it. + +**Exit criterion.** CI green; multi-clause runner exits 0 with `wrong == 0`; chosen GSM8K-probe gate satisfied; B3 + L9/G.1 + L10/G.2 + L11/G.3 lanes unchanged; refreshed coverage report committed. + +**Do not stack on another agent's branch.** Target main directly. diff --git a/docs/briefs/parallel-2026-05-23/L13-ADR-0131-G31-numerics-extensions.md b/docs/briefs/parallel-2026-05-23/L13-ADR-0131-G31-numerics-extensions.md new file mode 100644 index 00000000..91b3f85b --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L13-ADR-0131-G31-numerics-extensions.md @@ -0,0 +1,52 @@ +# L13 brief — ADR-0131.G.3.1 — Numerics extensions (fractions + multi-currency + multi-token cardinals + word-num-adjective) + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-adr-0131-g31-numerics-ext -b feat/adr-0131-g31-numerics-ext origin/main +cd ../core-adr-0131-g31-numerics-ext +``` + +**Scope.** Follow-up to ADR-0131.G.3 (PR #183) extending literal recognition to the four shapes explicitly deferred in that ADR's "Out of scope, explicitly deferred to G.3.1+" section. The parent ADR's canonical-unit architectural decisions (cent for money via `en_units_v1`, pack-driven not regex-spam, refusal-first) stay absolute — this iteration extends scope inside those constraints, not around them. + +**Closed-set scope (this iteration only):** + +1. **Fractions end-to-end.** `_resolve_value` in `generate/math_candidate_parser.py` already handles `N/M` token-level (returns float via `Fraction(N,M)`). Land axis cases that exercise fractions through the full pipeline — likely requires widening `_INITIAL_HAS_RE`'s `(?:of|in|for|with) ` substance-qualifier handling so `Bob has 3/4 of a cup.` parses cleanly (the fraction is the value; `of a cup` carries the unit). Closed-set: digit/digit literal with `M > 0`. +2. **Multi-currency.** Extend `_money_unit_normalization` (math_candidate_parser.py) and `_unit_grounds` (math_roundtrip.py, the `"$" in source_span` check) to recognize `¢ € £ ¥ ₱`. Each gets its own canonical unit per `en_units_v1`. **Verify `en_units_v1.lexicon.jsonl` actually contains `euro / pound / yen / peso` before wiring; if any is missing, defer that currency to G.3.2 rather than hardcoding a unit name the pack doesn't recognize.** +3. **Space-separated multi-word cardinals.** `one hundred`, `two thousand five hundred`. `parse_compound_cardinal` in `language_packs/numerics_loader.py` already supports these; the parser's `_VALUE` regex doesn't yet match space-separated sequences because they'd span the unit slot boundary. Either (a) add a separate extractor that pre-passes value-slot whitespace runs through `parse_compound_cardinal`, or (b) widen `_VALUE` to greedily match cardinal-word sequences. **Pick one approach, document the choice in the ADR.** +4. **Word-number-adjective compositions.** `five full boxes` per ADR-0127 substance-qualifier precedent. Adjective inserted between cardinal and unit head noun; treat as part of the unit phrase, not a separate value-slot widening. + +**Reference docs (read these, only these):** +1. `docs/decisions/ADR-0131.G.3-numerics.md` — the parent; what's already done, what's deferred, the canonical-unit decision you must respect. +2. `docs/decisions/ADR-0127-units-pack-and-units-aware-parser.md` — substance-qualifier precedent for axis 4. + +**What to ship:** +- **Parser extensions** in `generate/math_candidate_parser.py` (+ `generate/math_roundtrip.py` for new grounding cases). +- **New axis lane** at `evals/math_capability_axes/G3_numerics/v1.1/` — additive sibling to v1. **Do not modify v1** (`evals/math_capability_axes/G3_numerics/v1/`); it's frozen as the audit-trail artifact for #183. v1.1 carries fresh `cases.jsonl`, `runner.py`, `report.json`. +- **Curated cases:** ≥4 per axis + ≥4 refusal probes pinning what's still out of scope (percentages `50%`, scientific notation `1e3`, locale separators `1,000`, three-decimal money `$1.234`). +- **Tests:** `tests/test_adr_0131_G31_numerics_extensions.py` — per-axis at-least-one passing, refusal probes refuse typed, `wrong == 0`, replay byte-equality, parent v1 lane still passes (no regression). +- **ADR:** `docs/decisions/ADR-0131.G.3.1-numerics-extensions.md` — cite #183 parent + ADR-0127 substance-qualifier precedent. Document the axis-3 approach choice (extractor vs `_VALUE` widening). +- **Refresh** `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json` — fractions or compound cardinals may now unlock some GSM8K cases. + +**Hard constraints:** +- **`wrong == 0`** on the v1.1 axis lane AND the GSM8K probe. +- **Pack-driven, not regex-spam.** Every recognized literal traces to `en_numerics_v1` or `en_units_v1`. No inline alternations for things the pack should own. +- **`en_units_v1.canonical_unit` precedence stays absolute** (cent for money, etc.) — same architectural decision G.3 set. New currencies normalize to their pack-pinned canonical unit; do not invent surface units. +- **Closed set per axis.** Refusal probes prove the boundary. If a paraphrase isn't in the closed set, it must refuse, not coerce. +- **Determinism:** `v1.1/report.json` byte-equal across runs. +- **No solver / binding-graph changes.** If a fraction parses but doesn't solve through, file a follow-up ADR; don't stub the solver. +- **Don't modify v1.** It's the audit-trail artifact for #183 and stays frozen. +- **Field invariant untouched.** No changes to `algebra/`, `chat/`, `core/`. + +**Out of scope (do not touch):** +- Percentages, scientific notation, locale separators, three-decimal money (these become refusal-probe cases — closed-set boundary). +- ADR-0131.4 promotion-gate wiring (main agent's parallel work). +- Anything in `algebra/`, `chat/`, `core/`. + +**Target branch.** PR against `main`. Title: `feat(ADR-0131.G.3.1): numerics extensions (fractions + multi-currency + multi-token cardinals + word-num-adjective) — axis lane N/N`. Body: per-axis case counts, currencies actually wired (vs. deferred to G.3.2 because missing from pack), axis-3 approach choice + rationale, link to ADR, GSM8K probe delta if any (admission may or may not move depending on which cases the extensions unlock). + +**Exit criterion.** CI green; v1.1 axis runner exits 0 with `wrong == 0`; v1 lane unchanged; GSM8K probe `admitted_wrong == 0` preserved. + +**Only run tests that exercise files you change plus the axis lane + GSM8K probe + the parent v1 lane.** Do not run the full suite — that's the lead's job at integration. + +**Do not stack on another agent's branch.** Target main directly. diff --git a/docs/briefs/parallel-2026-05-23/L14-ADR-0114a-5-perturbation-suite.md b/docs/briefs/parallel-2026-05-23/L14-ADR-0114a-5-perturbation-suite.md new file mode 100644 index 00000000..dcf18518 --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L14-ADR-0114a-5-perturbation-suite.md @@ -0,0 +1,71 @@ +# L14 brief — ADR-0114a.5 — Reasoning-isolation perturbation suite (Obligation #5 for B3) + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-adr-0114a-5-perturbation -b feat/adr-0114a-5-perturbation origin/main +cd ../core-adr-0114a-5-perturbation +``` + +**Scope.** Wire **ADR-0114a Obligation #5** (reasoning-isolation perturbation suite) for the math composite gate's B3 lane (bounded grammar). Plays to your recent parser-layer expertise (G.2 / G.4 / G.3.1) — perturbation is parser-input-modification work. + +ADR-0114a #5 reads: + +> `perturbation_score.py`'s **invariance-preserving rate == 1.0** AND **invariance-breaking predictable-change rate == 1.0**. + +Two classes of perturbation, each must hold absolutely (per ADR-0120 §"Threshold rationale" — ε=0 here, not 0.05; reasoning isolation is binary). + +**Reference docs (read these, only these):** +1. `docs/decisions/ADR-0125-reasoning-isolation-perturbation-suite.md` — methodological blueprint. Mirror the perturbation taxonomy where it transfers cleanly. +2. `evals/gsm8k_parser_dev/perturbation_score.py` — existing GSM8K-context perturbation scorer. Read the architecture; **do not** import from it (different domain assumptions — B3's bounded grammar is its own contract). Use it as a pattern source. + +**What to ship:** + +- **`core/capability/perturbation_b3.py`** — perturbation generator + scorer for B3. + - **Invariance-preserving perturbations** (each must NOT change the answer): + - **Entity rename**: `Sam → Alex`, `Bob → Carol` (drawn from a closed substitution pool documented in the ADR). Multi-occurrence consistent within a problem. + - **Commutative reorder**: `"Sam has 5 apples. Sam has 3 oranges."` ↔ `"Sam has 3 oranges. Sam has 5 apples."` (single-entity multi-unit initial-state only; respect distributive `each` boundaries from G.4). + - **Unit-noun synonym substitution** (closed set, pack-aligned): `apples ↔ oranges`, `dollars ↔ cents` only when the question's unit is also substituted consistently. + - **Invariance-breaking perturbations** (each MUST change the answer by a *predictable* delta): + - **Value replacement**: `5 → 7` in a single sentence; predicted delta = +2. + - **Op-verb flip**: `buys` ↔ `loses`; predicted delta sign-flips the corresponding op's contribution. + - For each case in B3's expected-correct subset, generate ≥3 invariance-preserving variants and ≥3 invariance-breaking variants. Run the pipeline on each. Tally rates. + - Module exposes `validate_perturbation_suite(lane_id="B3_bounded_grammar", cases_path=...) -> PerturbationReport`. Same shape as the pack-provenance auditor's `validate_lane` (PR #189) — mirror that pattern for consistency. + +- **CLI** `core capability perturbation` (parallel to `core capability pack-provenance`). Writes `evals/obligation_5_perturbation/.json`. Exit 0 iff both rates == 1.0. + +- **Tests** `tests/test_adr_0114a_5_perturbation.py` (~15): + - Each perturbation generator function is pure (same input → same output set). + - Invariance-preserving variants of a known case all produce the same expected_answer. + - Invariance-breaking variants produce the predicted delta. + - Determinism: report byte-equal across runs. + - Snapshot test: current main B3 satisfies obligation #5 (or document precisely which case-perturbation pair fails — refuse to pass spuriously). + - Empty-lane refusal: missing cases file → typed refusal. + +- **ADR** `docs/decisions/ADR-0114a.5-perturbation-suite.md`. Cite ADR-0114a parent, ADR-0125 (methodology), PR #189 (auditor pattern), ADR-0131.3 (B3 substrate). Document the closed perturbation taxonomy. Pin the entity-substitution pool + unit-synonym mapping. + +- **Refresh** the obligation-#10 audit report (`evals/obligation_10_pack_provenance/B3_bounded_grammar.json`) if your perturbation generator affects parser internals (it shouldn't — perturbation operates on the *input string*, not the parser — but worth re-running the auditor to confirm). + +**Hard constraints:** + +- **`wrong == 0` preserved on the B3 axis lane.** Perturbation testing is auxiliary; never relax the load-bearing invariant. +- **Closed perturbation set.** Every perturbation rule documented in the ADR + asserted in tests. No paraphrase-style fuzz that wanders into G.1/G.2/G.3/G.4 axis territory; this is *reasoning isolation*, not grammar coverage. +- **Pack-aligned synonyms.** Unit substitution consults `en_arithmetic_v1` / `en_units_v1` — never invent surface units. (Same rule the G. work established.) +- **Both rates must equal 1.0** for obligation #5 to pass. Anything less is a refusal — file the failing perturbation as a follow-up scope-reduction, do NOT weaken the threshold. +- **Determinism**: same seed (if you use one) and same cases produce byte-equal report. +- **No solver / binding-graph changes.** If a perturbation reveals a solver bug, file as a follow-up ADR; don't patch the solver from inside this PR. +- **No new modules under `algebra/`, `chat/`, `core/cognition/`.** New module lives under `core/capability/`. + +**Out of scope:** +- B1 (symbolic equivalence) + B2 (teaching corpus) equivalents — separate sub-ADRs (mirror PR #189's "B3 only, others deferred" structure). +- Cross-sentence semantic perturbations (pronoun rewrites, paraphrasing) — those are coverage axes, not reasoning isolation. +- ADR-0114a obligations #2 (OOD), #6 (depth curve — main agent is on this in parallel), #8 (adversarial). Don't bundle. +- Composite-gate wiring (ADR-0131.4 already shipped + sufficient for the benchmark portion; obligation auditors stay orthogonal). + +**Target branch.** PR against `main`. Title: `feat(ADR-0114a.5): reasoning-isolation perturbation suite — Obligation #5 wired for B3, PASSING /`. Body: per-perturbation-class counts, invariance-preserving rate, invariance-breaking predictable-change rate, link to ADR. + +**Exit criterion.** CI green; perturbation runner exits 0 with both rates == 1.0; B3 axis lane unchanged; obligation #10 still passes; ADR-0114a.5 included. + +**Only run tests that exercise files you change plus the B3 lane, the perturbation suite, and the obligation #10 auditor.** Do not run the full suite — that's the lead's job at integration. + +**Do not stack on another agent's branch.** Target main directly. Note: PR #189 (obligation #10) and the main agent's parallel PR (obligation #6) are independent — no merge order needed. diff --git a/docs/briefs/parallel-2026-05-23/L15-ADR-0114a-2-ood-ratio.md b/docs/briefs/parallel-2026-05-23/L15-ADR-0114a-2-ood-ratio.md new file mode 100644 index 00000000..e531252d --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L15-ADR-0114a-2-ood-ratio.md @@ -0,0 +1,72 @@ +# L15 brief — ADR-0114a.2 — OOD surface variation ratio (Obligation #2 for B3) + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-adr-0114a-2-ood-ratio -b feat/adr-0114a-2-ood-ratio origin/main +cd ../core-adr-0114a-2-ood-ratio +``` + +**Scope.** Wire **ADR-0114a Obligation #2** (OOD surface variation ratio) for the math composite gate's B3 lane. Continues your obligation-auditor work after L14 (#5 perturbation). + +ADR-0114a #2 reads (via ADR-0120's table): + +> `ood_score.py`'s **OOD/public ratio for the domain's lanes ≥ 0.95**. + +The OOD lane is a separate case set whose surface forms vary along **non-semantic axes** the public split didn't exercise (different entity-name distributions, different unit-noun distributions, different sentence-ordering patterns) while staying strictly **within B3's bounded grammar** (the grammar IS the contract; outside the grammar is refusal-territory, which is obligation #8's domain, not #2's). + +The obligation's spirit: a pattern-matcher overfits to surface distributions of the training/public set and falls off when surface varies; a deterministic reasoner stays approximately flat. The 0.95 ratio means OOD accuracy must be at least 95% of public accuracy. + +**Reference docs (read these, only these):** +1. `docs/decisions/ADR-0118a-ood-surface-generator.md` — methodological blueprint. Mirror the OOD generation taxonomy where it transfers; B3's bounded grammar narrows the design space (no paraphrase variance — that's adversarial, not OOD). +2. `core/capability/pack_provenance.py` (PR #189 / merged ADR-0114a.10) + `core/capability/depth_curve.py` (PR #190 / pending ADR-0114a.6) — the auditor pattern this PR mirrors. Same module-shape, same CLI-shape, same `validate_lane`/`evaluate_*` signature, same `obligation_N_*` verdict fields. + +**What to ship:** + +- **`evals/obligation_2_ood_ratio/v1/cases.jsonl`** — OOD case set, **≥30 cases**, strictly in-grammar (B3's grammar contract from ADR-0131.3 `grammar.md`). Each case is the **surface-varied sibling** of an existing B3 public case: + - **Entity-name pool**: rotate `Sam → {Maya, Liam, Noah, Diana, Felix, ...}` from a closed substitution pool documented in the ADR. Same letter-case discipline as B3 (capitalized proper nouns). + - **Unit-noun pool**: rotate `apples → {oranges, berries, marbles, pencils, books, candies, stamps, ...}` — all from `en_units_v1` (verify each is pack-recognized; if not, defer the unit). + - **Operation-order variance**: where B3 puts initial first, OOD puts the question first or interleaves (within grammar). Example: B3 `Sam has 5 apples. Sam buys 3. How many apples?` ↔ OOD `Maya buys 7 oranges after she has 4 oranges. How many oranges does Maya have?` (when grammar admits both forms). + - Each case carries `shape_category` matching one of B3's documented shapes + a new `public_sibling_case_id` field pointing to its B3 counterpart for audit traceability. + +- **`evals/obligation_2_ood_ratio/v1/runner.py`** — runs the OOD set through the candidate-graph pipeline; emits `report.json` mirroring B3's runner shape. + +- **`core/capability/ood_ratio.py`** — auditor. Reads B3's public `report.json` (cases_correct / cases_total) AND the OOD `report.json`. Computes `ood_ratio = ood_accuracy / public_accuracy`. Emits `OodRatioReport` with `obligation_2_ratio_satisfied` flag (gate: ratio ≥ 0.95). Same module-shape as pack_provenance.py / depth_curve.py. + +- **CLI** `core capability ood-ratio`. Writes `evals/obligation_2_ood_ratio/.json`. Exit 0 iff ratio ≥ 0.95 AND OOD `wrong == 0` AND public-baseline-accuracy > 0. + +- **Tests** `tests/test_adr_0114a_2_ood_ratio.py` (~15): + - Dataset integrity: ≥30 cases; every case has `public_sibling_case_id` resolving to a real B3 case; every case in-grammar (parser admits). + - Entity-name substitution: every OOD case has a *different* entity name than its public sibling. + - Unit-noun substitution: every OOD case has a *different* unit (when the public sibling's unit appears in the substitution pool). + - Auditor: ratio computed correctly; gate at 0.95 pinned (changing requires new ADR); `wrong == 0` is a separate gate (catches OOD cases where parser misroutes to a wrong answer). + - Refusal on missing public report or missing OOD cases. + - Determinism: report byte-equal across runs. + - Snapshot test: current main satisfies obligation #2 on the OOD set you ship. + +- **ADR** `docs/decisions/ADR-0114a.2-ood-ratio-auditor.md`. Cite parent ADR-0114a, ADR-0118a (methodology), PR #189 + PR #190 (auditor pattern), ADR-0131.3 (B3 grammar contract). Pin the entity-name and unit-noun substitution pools. + +**Hard constraints:** + +- **Strictly in-grammar.** Every OOD case must be parseable by B3's bounded grammar. Out-of-grammar cases are adversarial-territory (obligation #8), not OOD-territory. If a case refuses, it's a dataset bug — fix the case or drop it; don't relax the OOD framing. +- **Pack-aligned substitutions.** Every unit noun consults `en_units_v1` — never invent surface units. Entity names follow B3's existing capitalized-proper-noun convention. +- **No semantic perturbations.** OOD ≠ perturbation. Don't change values, don't swap operations, don't break invariants. Surface only (entity / unit / surface-order). +- **Gate at 0.95 pinned.** Changing the threshold requires a new ADR. +- **`wrong == 0` on the OOD set** (separate gate from the ratio; both must hold). +- **Determinism**: same OOD cases produce byte-equal report. +- **No solver / parser / pack changes.** This is pure evaluation substrate. +- **No new modules under `algebra/`, `chat/`, `core/cognition/`.** New module lives under `core/capability/`. + +**Out of scope:** +- B1 (symbolic equivalence) + B2 (teaching corpus) OOD equivalents — separate sub-ADRs (mirror PR #189's structure). +- Cross-grammar fuzzing (paraphrases beyond the grammar) — that's adversarial obligation #8 (main agent is on this in parallel). +- ADR-0114a obligations #5 (perturbation, your L14), #6 (depth curve, PR #190), #8 (adversarial, main agent). Don't bundle. +- Composite-gate or promotion-gate wiring. + +**Target branch.** PR against `main`. Title: `feat(ADR-0114a.2): OOD-ratio auditor — Obligation #2 wired for B3, ratio=`. Body: case count, family breakdown (entity / unit / order variance), public accuracy + OOD accuracy + computed ratio, link to ADR, plus an honest scope-limit if the substitution pool is smaller than ideal (deferred items go to a follow-up). + +**Exit criterion.** CI green; OOD runner exits 0 with `wrong == 0`; auditor exits 0 with `ratio ≥ 0.95`; B3 public lane unchanged; obligations #10 + #6 + #5 still pass (no cross-PR breakage); ADR-0114a.2 included. + +**Only run tests that exercise files you change plus the B3 public lane, the OOD lane, and the obligation-#10 / #5 / #6 auditors.** Do not run the full suite — that's the lead's job at integration. + +**Do not stack on another agent's branch.** Target main directly. Note: PR #190 (obligation #6) and the main agent's #8 PR will land in parallel — no merge order needed unless `core/cli.py` conflicts arise, in which case the same "keep both sibling commands" resolution from PR #189 applies. diff --git a/docs/briefs/parallel-2026-05-23/L16-ADR-0131-G5-aggregate-answer-composition.md b/docs/briefs/parallel-2026-05-23/L16-ADR-0131-G5-aggregate-answer-composition.md new file mode 100644 index 00000000..54f7b77c --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L16-ADR-0131-G5-aggregate-answer-composition.md @@ -0,0 +1,139 @@ +# L16 brief — ADR-0131.G.5 — Capability axis: aggregate answer composition + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-adr-0131-g5-aggregate -b feat/adr-0131-g5-aggregate origin/main +cd ../core-adr-0131-g5-aggregate +``` + +**Scope.** Close the aggregate-answer composition loop opened by G.4's +multi-entity candidate emission. The parser already emits +`Unknown(entity=None, unit=)` for questions containing +`in total` or `altogether` (lines 688–735 of +`generate/math_candidate_parser.py`). The solver already sums terminal +state across all entities when `entity is None` (lines 525–532 of +`generate/math_solver.py`). The gap is two-fold: + +1. **Vocabulary gap**: `"combined"` and `"together"` are not in `_Q_TOTAL_RE`'s + tail alternation — "How many apples do they have combined?" is refused at + the question layer even though the two-entity sum would be correct. +2. **No pinned lane**: there are no curated axis cases proving the + two-entity and three-entity aggregate paths work end-to-end through + `parse_and_solve`. The capability exists but is untested as a named lane. + +**Pre-flight check (run before coding):** + +```python +from generate.math_candidate_graph import parse_and_solve +r = parse_and_solve( + "Sam has 5 apples. Tom has 3 apples. How many apples do they have altogether?" +) +assert r.answer == 8.0 and r.refusal_reason is None # already passes +r2 = parse_and_solve( + "Sam has 5 apples. Tom has 3 apples. How many apples do they have combined?" +) +assert r2.refusal_reason is not None # currently fails — this is the gap +``` + +**Reference docs (read these, only these):** + +1. `docs/decisions/ADR-0131.G-gsm8k-coverage-probe.md` — iteration + discipline; axis cases are independent of GSM8K; `admitted_wrong == 0` + is non-negotiable. +2. `generate/math_candidate_parser.py` lines 685–750 — `_Q_TOTAL_RE` and + `_Q_ENTITY_RE`; the change is a one-line alternation extension in + `_Q_TOTAL_RE`'s tail group. + +**What to ship:** + +- **Parser fix** in `generate/math_candidate_parser.py`: extend + `_Q_TOTAL_RE`'s tail alternation from + `(?:in\s+total|altogether|left|now)` to include `combined` and + `together` as aggregate cues. Map both to the same `entity=None` + semantics. Update the docstring's closed-cue list. + +- **Curated axis lane** at + `evals/math_capability_axes/G5_aggregate/v1/cases.jsonl` — **≥20 + cases** exercised end-to-end through `parse_and_solve` (not just the + parser): + + | Shape | Count | Cue coverage | + |---|---|---| + | 2-entity sum, no operations | ≥4 | `altogether`, `in total`, `combined`, `together` (≥1 each) | + | 3-entity sum, no operations | ≥4 | all four cues | + | 2-entity sum with one add/subtract op | ≥4 | mixed cues | + | Single-entity total cue (degenerate aggregate) | ≥4 | regression guard | + | Refusal: aggregate cue with mismatched units | ≥4 | wrong==0 probe | + + Each case carries `expected_answer` (numeric). The runner verifies + `r.answer == expected_answer` (exact float equality is fine — these are + integer initial states). Refusal cases verify `r.answer is None`. + +- **Runner + report** at `evals/math_capability_axes/G5_aggregate/v1/`. + Same shape as G2 runner: deterministic `report.json`, `wrong==0` gate, + byte-equal across runs. + +- **Tests** at `tests/test_adr_0131_G5_aggregate.py` (**≥15**): + - `"combined"` and `"together"` cues parse to `entity=None` (unit + confirmed via `extract_question_candidates`). + - All four cue words are in the closed-set docstring list. + - 2-entity axis cases produce the correct sum. + - 3-entity axis cases produce the correct sum. + - Single-entity degenerate case still works. + - Mismatched-unit cases refuse (no wrong answer emitted). + - `wrong == 0` on the full axis lane. + - Runner report is byte-equal across two back-to-back runs. + - B3 lane unchanged (import and run one B3 test as a regression guard). + - **GSM8K probe safety rail**: `admitted_wrong == 0` preserved on the + legacy probe (`train_sample_coverage_report.json`). No admission + movement is expected (all 50 sample cases still refuse at statement + parsing); gate on `admitted_wrong == 0`, not on admission rate. + +- **ADR** `docs/decisions/ADR-0131.G.5-aggregate-answer-composition.md`. + Cite ADR-0131.G parent. Document the closed aggregate-cue vocabulary + (`in total`, `altogether`, `combined`, `together`). State explicitly + that the solver path (`entity is None` → sum) was pre-existing — this + ADR extends the cue vocabulary and pins the lane, not the solver. Call + out what's deferred: implicit aggregation without a cue word ("How many + do Sam and Tom have?" with no aggregate cue — that requires coreference + and is out of scope), rate-based aggregation ("how many dollars did they + earn in total?" where the unit is derived from a rate operation). + +**Hard constraints:** + +- **`wrong == 0`** on every axis case and the GSM8K safety rail. The + solver sum is exact (integer initial states); any wrong answer means the + candidate graph selected a mismatched graph — fix the graph selection, + do not touch the solver. +- **Closed cue vocabulary.** Exactly four cues: `in total`, `altogether`, + `combined`, `together`. No synonym expansion, no paraphrase tolerance. + Document the deferred forms. +- **No solver changes.** The `entity is None` sum is already correct. + This ADR is parser-vocabulary + lane pinning only. +- **No new modules under `algebra/`, `chat/`, `core/`.** Parser change is + a one-line regex extension. New files live under `evals/` and `tests/`. +- **Determinism.** `report.json` byte-equal across runs. + +**Out of scope:** +- Implicit aggregation without a cue word (coreference). +- Rate-unit aggregation ("total earnings" from rate operations). +- GSM8K admission movement (statement parsing is the current bottleneck + for every sample case; question-layer work cannot lift that). +- Any changes to `math_solver.py`, `math_problem_graph.py`, or the + binding-graph layer. + +**Target branch.** PR against `main`. Title: +`feat(ADR-0131.G.5): aggregate answer composition — combined/together cues wired, axis lane N/N, wrong==0`. +Body: cue vocabulary, per-shape case counts, pre/post pre-flight check +output, link to ADR. Honest note that no GSM8K admission movement is +expected (statement parsing is the probe bottleneck). + +**Exit criterion.** CI green; axis runner exits 0 with `wrong == 0`; +B3 lane unchanged; GSM8K `admitted_wrong == 0` preserved; ADR committed. + +**Only run tests that exercise files you change plus the G5 axis lane, +the B3 lane, and the GSM8K safety rail.** Do not run the full suite — +that is the lead's job at integration. + +**Do not stack on another agent's branch.** Target main directly. diff --git a/docs/briefs/parallel-2026-05-23/L17-ADR-0131-G6-rate-capacity.md b/docs/briefs/parallel-2026-05-23/L17-ADR-0131-G6-rate-capacity.md new file mode 100644 index 00000000..149f0d79 --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L17-ADR-0131-G6-rate-capacity.md @@ -0,0 +1,135 @@ +# L17 brief — ADR-0131.G.6 — Capability axis: capacity-rate verbs + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-adr-0131-g6-rate-capacity -b feat/adr-0131-g6-rate-capacity origin/main +cd ../core-adr-0131-g6-rate-capacity +``` + +**Scope.** Admit capacity-rate problems — "Bob can shuck 10 oysters in 5 +minutes. How many oysters can he shuck in 2 hours?" is the shortest +unambiguous GSM8K sample case (81 chars, case `gsm8k-0014`). No part of +the current parser or candidate graph handles `" can N + in M ."` or `"How many can in T +?"`. Both are missing. + +**Pre-flight check (run before coding):** + +```python +from generate.math_candidate_graph import parse_and_solve +r = parse_and_solve( + "Bob can shuck 10 oysters in 5 minutes. How many oysters can he shuck in 2 hours?" +) +assert r.refusal_reason is not None # currently refuses — this is the gap +``` + +**Architecture constraint.** `math_solver.py` and `math_problem_graph.py` +are **off-limits** — do not modify them. The capacity-rate path must be +self-contained in the parser + candidate graph. Implement as a +**short-circuit path in `parse_and_solve`**: when both a capacity-rate +statement candidate and a capacity-rate question candidate are found, +compute `(N / M) × T` (with time-unit conversion) and return the answer +directly, bypassing `_build_graph` and `solve`. This is structurally +identical to the aggregate path (`entity=None` → solver sums), which also +short-circuits at the question layer. + +**Reference docs (read these, only these):** + +1. `generate/math_candidate_parser.py` lines 424–763 — the initial-state + extractor and question extractor patterns; model new extractors on + these shapes (`@dataclass(frozen=True, slots=True)`, return + `list[Candidate...]`, compile regex as module-level `Final` constants). +2. `generate/math_candidate_graph.py` lines 277–400 — `parse_and_solve` + structure; see how the aggregate path branches before `_build_graph`. + +**What to ship:** + +- **Parser additions** in `generate/math_candidate_parser.py`: + + - `CandidateCapacityRate` frozen dataclass: `actor`, `count` + (float), `unit`, `per_count` (float), `per_unit`, `source_span`. + Computed rate `= count / per_count`. + - `_CAPACITY_RE`: matches `" can N in M + ."` — closed `` set: `shuck`, `pick`, `read`, + `write`, `run`, `make`, `produce`, `pack`, `paint`, `type`; closed + `` set: `second`, `minute`, `hour`, `day`. + - `extract_capacity_candidates(sentence) -> list[CandidateCapacityRate]`. + - `CandidateCapacityQuestion` frozen dataclass: `actor` (str | None), + `unit`, `per_count` (float), `per_unit`, `source_span`. + - `_CAPACITY_Q_RE`: matches `"How many can in T + ?"` — same closed verb + time-unit sets; `` may be + a pronoun (`he`, `she`, `they`, `it`). + - `extract_capacity_question_candidates(sentence) -> list[CandidateCapacityQuestion]`. + - Time-unit conversion helper: `_to_seconds(count, unit) -> float` — + second=1, minute=60, hour=3600, day=86400. Division gives + rate-per-second; multiplication gives total count. + +- **Candidate graph branch** in `generate/math_candidate_graph.py`: + Add a short-circuit check in `parse_and_solve` before the Cartesian + product: if `len(statement_sentences) == 1` and the statement yields + exactly one `CandidateCapacityRate` and the question yields exactly one + `CandidateCapacityQuestion`, compute `rate_per_sec × T_seconds` and + return. Actor pronoun resolution: if `question.actor` is a pronoun, + accept any capacity candidate; if it is a named entity, require it to + match `capacity.actor` (case-insensitive). `wrong == 0` must hold; if + the actor doesn't match or the rate is zero, return refusal. + +- **Curated axis lane** at + `evals/math_capability_axes/G6_rate_capacity/v1/cases.jsonl` — + **≥16 cases**: + + | Shape | Count | Notes | + |---|---|---| + | Same time-unit in statement and question | ≥4 | minutes → minutes | + | Cross time-unit (statement minutes, question hours) | ≥4 | requires conversion | + | Pronoun actor in question | ≥4 | "how many can he …" | + | Refusal: mismatched unit (oysters vs books) | ≥4 | wrong==0 probe | + + Include `gsm8k-0014` verbatim as one case (source-labeled, not a + special branch — the general path must admit it). + +- **Runner + report** at `evals/math_capability_axes/G6_rate_capacity/v1/`. + Same shape as G5 runner. `wrong == 0` gate. + +- **Tests** at `tests/test_adr_0131_G6_rate_capacity.py` (**≥15**): + - `_CAPACITY_RE` matches canonical forms. + - `_CAPACITY_Q_RE` matches pronoun and named-actor forms. + - Same-unit and cross-unit round-trips produce correct counts. + - `gsm8k-0014` verbatim admits with answer `240.0`. + - Mismatched-unit refusals hold. + - `wrong == 0` on full axis lane. + - Byte-equal report across two runs. + - B3 lane regression guard. + - GSM8K safety rail: `admitted_wrong == 0` preserved. + +- **ADR** `docs/decisions/ADR-0131.G.6-rate-capacity.md`. Parent: + ADR-0131.G. Document closed verb set, closed time-unit set, conversion + table, short-circuit path rationale. State explicitly that + `math_solver.py` and `math_problem_graph.py` are unchanged. Call out + deferred: multi-statement rate (duration asserted separately from + capacity), money/earnings rates (`makes $18/hour`), rate operations + on pre-existing state. + +**Hard constraints:** + +- `wrong == 0` on every axis case and the GSM8K safety rail. +- No changes to `math_solver.py`, `math_problem_graph.py`, or + `math_verifier.py`. +- Closed verb set and closed time-unit set — no wildcard matching. +- `gsm8k-0014` must admit verbatim. +- `report.json` byte-equal across runs. + +**Out of scope:** + +- Multi-statement rate (duration in a separate sentence from capacity). +- Money/earnings rates ("makes $18.00 an hour" — needs money parsing). +- Rate applied to pre-existing denominator state (existing `apply_rate`). +- Any GSM8K admission beyond gsm8k-0014 and gsm8k-0018 (Xavier goals). + +**Target branch.** PR against `main`. Title: +`feat(ADR-0131.G.6): capacity-rate verbs — can-verb-N-in-M shape, axis lane N/N, wrong==0`. + +**Exit criterion.** CI green; axis runner exits 0 with `wrong == 0`; +`gsm8k-0014` admits with answer `240.0`; B3 unchanged; GSM8K +`admitted_wrong == 0` preserved; ADR committed. diff --git a/docs/briefs/parallel-2026-05-23/L17-ADR-0136-S1-rate-event-statements.md b/docs/briefs/parallel-2026-05-23/L17-ADR-0136-S1-rate-event-statements.md new file mode 100644 index 00000000..1c581832 --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L17-ADR-0136-S1-rate-event-statements.md @@ -0,0 +1,236 @@ +# L17 brief — ADR-0136.S.1 — Statement corridor: rate/event statement parsing + +**Worktree (use the one already created):** + +```bash +cd ../core-adr-0131-g6-rate-capacity # branch: feat/adr-0131-g6-rate-capacity +``` + +> Rename this ADR to ADR-0136.S.1 in all doc headers; the worktree and branch stay. + +--- + +## Taxonomy ground truth (read before coding) + +`evals/gsm8k_math/train_sample/v1/refusal_taxonomy.json` (just written, on disk): + +| Primary barrier | Cases | Notes | +|---|---|---| +| `context_filler` | **23** | Narrative scene-setters — legitimately refuse; not parser gaps | +| `compound_statement` | 5 | Two ops in one sentence | +| rate/capacity/price class | **4** | Direct S.1 targets | +| `distributive_multiply` | 1 (+5 secondary) | N bags × M items each | +| diverse long-tail | 17 | Age anchors, goal statements, multi-step chains, etc. | + +**Key constraint on expectations:** S.1 can honestly claim `0/50 → ≤4/50` admission +lift, not a dramatic jump. The 23 context-filler cases legitimately have no +parseable numeric state in their opening sentence — the parser's refusal there +is correct. Do not attempt to skip or soft-fail context sentences. If a +sentence cannot be parsed, the problem is refused. This is the safety rail. + +The one case with **capacity_rate as its only barrier** is `gsm8k-0014`: + +``` +Bob can shuck 10 oysters in 5 minutes. How many oysters can he shuck in 2 hours? +``` + +This is the proof case. It must admit with answer `240.0`. + +--- + +## Scope + +Extend the parser and candidate graph to handle two closed statement shapes +that appear as the **primary** (and often sole) barrier in 4 of the 50 cases: + +**Shape A — capacity-rate** (cases 0014, 0018-secondary, 0034-secondary): +``` + can N in M . +``` +→ rate = N/M (units per time-unit). Question: "How many X can Actor verb in T time?" +→ answer = rate × T (with time-unit conversion). + +**Shape B — earnings/price rate** (case 0001 primary, but conditional blocks it; +cases 0011, 0022, 0044 are also rate-class): +``` + makes/earns/charges/receives N per/an/a . +``` +→ `EarningsRate(actor, amount, money_unit, per_unit)`. +Question: "How much money does Actor make in T time?" → `amount × T` (with +conversion when needed). Start with the simplest closed set; see constraints below. + +**Do not** attempt to unlock context-filler-gated cases. The 23 context-filler +sentences refuse correctly; the rate parsing behind them is irrelevant until those +sentences parse. + +--- + +## Pre-flight checks (run before coding) + +```python +from generate.math_candidate_graph import parse_and_solve + +# Shape A — must currently refuse +r = parse_and_solve("Bob can shuck 10 oysters in 5 minutes. How many oysters can he shuck in 2 hours?") +assert r.refusal_reason is not None # gap + +# Shape B — must currently refuse +r = parse_and_solve("Tina makes $18.00 an hour. Tina works 5 hours. How much money does Tina make?") +assert r.refusal_reason is not None # gap (note: simplified, no conditional) +``` + +--- + +## Reference docs (these only) + +1. `generate/math_candidate_parser.py` lines 424–763 — initial-state extractor + and question extractor patterns; model new extractors on these shapes. +2. `generate/math_candidate_graph.py` lines 277–400 — `parse_and_solve` + structure; the capacity short-circuit path goes here (before `_build_graph`). + +--- + +## What to ship + +### Parser additions (`generate/math_candidate_parser.py`) + +**Shape A — capacity-rate:** + +- `CandidateCapacity` frozen dataclass: `actor`, `count` (float), `unit`, + `per_count` (float), `per_unit`, `source_span`. +- `_CAPACITY_RE` — closed verb set: `shuck`, `pick`, `pack`, `make`, `produce`, + `type`, `read`, `write`, `paint`, `run`, `score`, `answer`, `complete`; + closed time-unit set: `second`, `minute`, `hour`, `day`. + Pattern: `" can N in M ."` (case-insensitive). +- `extract_capacity_candidates(sentence)`. +- `CandidateCapacityQuestion` frozen dataclass: `actor` (str|None), `unit`, + `per_count` (float), `per_unit`, `source_span`. `actor` is None for pronoun + forms (`he/she/they/it`). +- `_CAPACITY_Q_RE` — same closed verb + time-unit sets. + Pattern: `"How many can in T ?"`. +- `extract_capacity_question_candidates(sentence)`. +- Time-unit helper `_to_seconds(count, unit) -> float`: second=1, minute=60, + hour=3600, day=86400. + +**Shape B — earnings rate:** + +- `CandidateEarningsRate` frozen dataclass: `actor`, `amount` (float), `unit` + (canonicalize `dollar`/`$`), `per_unit`, `source_span`. +- `_EARNINGS_RE` — closed verb set: `make`, `earn`, `receive`, `get`, `charge`; + closed currency tokens: `$`, `dollar`, `dollars`, `cent`, `cents`; + closed per-unit alternation: `per `, `a `, `an `, + `for each `, `every `. + Pattern: `" <$|N> ."`. +- `extract_earnings_candidates(sentence)`. +- `CandidateEarningsQuestion` frozen dataclass: `actor`, `unit`, `time_count` + (float), `time_unit`, `source_span`. +- `_EARNINGS_Q_RE`: `"How much money does make/earn in T ?"`. +- `extract_earnings_question_candidates(sentence)`. + +### Candidate graph short-circuit (`generate/math_candidate_graph.py`) + +Add before the Cartesian product in `parse_and_solve`: + +**Capacity path:** if `len(statement_sentences) == 1` and that sentence yields +exactly one `CandidateCapacity` and the question yields one +`CandidateCapacityQuestion`: +- Actor match: if `question.actor` is not None, require + `capacity.actor.lower() == question.actor.lower()`; else accept any. +- Compute: `rate_per_sec = capacity.count / capacity.per_count / _to_seconds(1, capacity.per_unit)`; + `answer = rate_per_sec * _to_seconds(question.per_count, question.per_unit)`. +- Return `CandidateGraphResult(answer=answer, ...)` if `answer > 0`. + +**Earnings path:** if `len(statement_sentences) == 1` (or 2 with one being a +named-entity duration statement like `"Tina works 5 hours."`) and the +statement yields one `CandidateEarningsRate` and the question yields one +`CandidateEarningsQuestion`: +- Actor match: require names match (case-insensitive). +- Compute: convert `earnings.per_unit` and `question.time_unit` to seconds; + derive `rate_per_sec`; `answer = rate_per_sec * question.time_count_seconds`. +- Return answer. + +Both paths must gate: `answer > 0` and actor resolved; else fall through to +refusal (do not emit `wrong` answers — if uncertain, refuse). + +### Curated axis lane (`evals/math_capability_axes/S1_rate_events/v1/cases.jsonl`) + +**≥20 cases** — independent of GSM8K phrasing: + +| Shape | Count | Requirements | +|---|---|---| +| Capacity same time-unit (min→min) | ≥4 | base case | +| Capacity cross time-unit (min→hr, sec→min) | ≥4 | conversion required | +| Capacity pronoun question | ≥4 | `he`/`she` question form | +| Earnings same time-unit (hr→hr) | ≥4 | base earnings case | +| Refusal: closed-verb miss (outside verb set) | ≥4 | wrong==0 probe | + +Include `gsm8k-0014` verbatim as one of the capacity cases. Include a +simplified Tina-style earnings case (no conditional) as one earnings case. + +### Runner + report + +`evals/math_capability_axes/S1_rate_events/v1/runner.py` — same shape as G5 +runner. `wrong == 0` gate. Byte-equal `report.json`. + +### Tests (`tests/test_adr_0136_S1_rate_events.py`, ≥15) + +- `_CAPACITY_RE` matches canonical forms; refuses closed-verb misses. +- `_EARNINGS_RE` matches `makes $X an hour`, `earns $X per hour`, `receives $X for each unit`. +- Time conversion: minutes→hours, seconds→minutes. +- `gsm8k-0014` verbatim admits with answer `240.0`. +- Simplified Tina ("makes $18/hr, works 5 hours → $90") admits. +- Pronoun actor resolves. +- Mismatched-actor refuses (not wrong — refuses). +- `wrong == 0` on full axis lane. +- Byte-equal report. +- B3 regression guard. +- GSM8K `admitted_wrong == 0` + post-S.1 `admission_rate` honestly reported. + +### ADR (`docs/decisions/ADR-0136.S.1-rate-event-statements.md`) + +Parent: `ADR-0136-statement-layer-corridor.md` (write as sibling doc, one +paragraph stub, if parent doesn't exist yet). Document: +- Taxonomy finding: 23/50 context-filler (correctly refused), 4/50 rate-class + direct targets. +- Closed verb sets and why (no wildcard matching). +- Short-circuit path rationale. +- Honest GSM8K claim: `0/50 → ≤4/50`; exact delta stated in PR body. +- Deferred: context-filler gated problems (need context-sentence semantic + classification, out of scope), conditional branching (overtime), percentage + rates (interest), multi-statement earnings (duration in separate sentence). + +--- + +## Hard constraints + +- `wrong == 0` on every axis case and GSM8K safety rail. +- No changes to `math_solver.py`, `math_problem_graph.py`, `math_verifier.py`. +- Closed verb sets — no regex wildcards for verbs. +- `gsm8k-0014` must admit verbatim with `answer == 240.0`. +- Do not attempt context-filler cases. If a sentence can't parse, refuse. +- Honest admission delta in PR body: count exactly which GSM8K cases newly admit. + +--- + +## Out of scope + +- Context-filler gated rate cases (23 cases — needs semantic classification). +- Conditional branching (`if she works more than 8 hours`). +- Percentage/interest rates (10% simple interest). +- Multi-statement earnings (duration asserted in a separate sentence from the + rate — needs general duration-statement parser first). +- Any GSM8K admission beyond the 4 rate-class primary cases. + +--- + +## Target branch + +PR against `main` from `feat/adr-0131-g6-rate-capacity`. Title: +`feat(ADR-0136.S.1): rate/event statement parsing — capacity + earnings shapes, axis lane N/N, wrong==0, gsm8k-0014 admits`. + +Body: taxonomy summary, per-shape case counts, pre/post GSM8K probe delta +(honest count), link to taxonomy JSON and ADR. + +**Exit criterion.** CI green; axis runner exits 0 with `wrong == 0`; +`gsm8k-0014` admits with answer `240.0`; GSM8K `admitted_wrong == 0` and +honest admission delta stated; B3 unchanged; ADR committed. diff --git a/docs/briefs/parallel-2026-05-23/L2-opus2-binding-graph-phase1.md b/docs/briefs/parallel-2026-05-23/L2-opus2-binding-graph-phase1.md new file mode 100644 index 00000000..c31da2cb --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L2-opus2-binding-graph-phase1.md @@ -0,0 +1,49 @@ +# L2 brief — Opus#2 — Semantic-Symbolic Binding Graph, Phase 1 (data model only) + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-binding-graph-p1 -b feat/binding-graph-phase1 origin/main +cd ../core-binding-graph-p1 +``` + +**Scope.** Ship Phase 1 *only* of the binding graph layer proposed in `docs/implementation/semantic-symbolic-binding-graph-proposal.md` (PR #170, now on main). Phase 1 is **data-model-only**: frozen dataclasses + invariants + tests. **No runtime wiring, no adapter, no equation binding, no parser change.** Phases 2–5 are deferred to follow-up PRs. + +**Reference docs (read these, only these):** +1. `docs/implementation/semantic-symbolic-binding-graph-proposal.md` — your spec. The dataclass list under "Adds" is authoritative. +2. `generate/math_symbolic_normalizer.py` (post-#167) — only to see the existing `Polynomial` dataclass style (frozen, terms-dict, variables-tuple). Match its idiom; do **not** import from it. + +**What to ship:** +- `generate/binding_graph/__init__.py` — public API surface. +- `generate/binding_graph/model.py` — frozen dataclasses per the proposal: + - `SemanticSymbolicBindingGraph` + - `SymbolBinding` + - `BoundFact` + - `BoundEquation` + - `BoundUnknown` + - `BoundConstraint` + - `SourceSpanLink` + Use `@dataclass(frozen=True, slots=True)`. All collections immutable (`tuple`, `frozenset`, `MappingProxyType` if needed). No mutation methods. +- `generate/binding_graph/allocation.py` — deterministic symbol allocator. Given a sorted iterable of NL noun-phrases, returns a stable `tuple[SymbolBinding, ...]`. Pure function. Deterministic across runs. +- `tests/test_binding_graph_model.py` — 30–50 tests covering: frozen invariants, equality, hashability where applicable, allocation determinism, refusal on bad input (empty span, duplicate binding id, etc.), `SourceSpanLink` round-trip. +- `docs/decisions/ADR-0132-binding-graph-data-model.md` — short ADR ratifying Phase 1; cite #170's proposal doc; explicit "Phase 2+ deferred" section. + +**Hard constraints:** +- **Pure data layer.** No I/O, no parser calls, no algebra calls, no `numpy`, no runtime field touch. +- **Immutability.** Every field is immutable; every collection is `tuple`/`frozenset`. The "create new objects, never mutate" rule from coding-style.md is load-bearing here. +- **Deterministic allocation.** Same input order → same `SymbolBinding` ids → byte-equal serialization. Lane test asserts this. +- **Refusal-first.** Invalid construction raises a typed error (`BindingGraphError`, sibling of `SymbolicError`); never silently coerces. +- **No coupling to the symbolic normalizer.** The binding graph references symbolic expressions by *string canonical form*, not by importing `Polynomial`. Decoupling is the whole point of the layer. + +**Out of scope (do not touch — these are Phase 2+):** +- Adapter from existing `MathProblemGraph` (Phase 2). +- Unit-aware equation binding (Phase 3). +- Question-target binding (Phase 4). +- Bounded grammar integration (Phase 5 / B3). +- Any change to `chat/`, `generate/intent.py`, `generate/realizer.py`. + +**Target branch.** PR against `main`. Title: `feat(binding-graph): Phase 1 data model (ADR-0132)`. Body must reference #170 as parent and list Phase 2–5 as deferred follow-ups. + +**Exit criterion.** PR opens with CI green, all new tests pass, `pyright` clean on new files, ADR-0132 included. Runtime behavior byte-identical to main (no integration yet, by design). + +**Do not stack on another agent's branch.** Target main directly. diff --git a/docs/briefs/parallel-2026-05-23/L3-sealed-holdout-B1.md b/docs/briefs/parallel-2026-05-23/L3-sealed-holdout-B1.md new file mode 100644 index 00000000..fa67e06a --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L3-sealed-holdout-B1.md @@ -0,0 +1,42 @@ +# L3 brief — (Opus#2 or Gemini, small lane) — ADR-0131.1 Sealed Holdout for B1 + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-adr-0131-1-sealed-holdout -b feat/adr-0131-1-sealed-holdout origin/main +cd ../core-adr-0131-1-sealed-holdout +``` + +**Scope.** Add a pyrage-X25519 sealed-holdout split to Benchmark 1 (symbolic equivalence v1, merged in #167). Mirror the methodology already proven on ADR-0119.7 for GSM8K — same crypto, same exit-criterion shape, different content. This makes the B1 lane's claim externally credible (operator cannot have peeked at the holdout cases). + +**Reference docs (read these, only these):** +1. `docs/decisions/ADR-0119.7-sealed-gsm8k-test.md` — your methodological blueprint. Copy the sealed-holdout structure exactly. +2. `evals/math_symbolic_equivalence/v1/` (on main, post-#167) — the lane you're hardening. Do not modify `cases.jsonl`; you are *adding* a sealed split, not replacing. + +**What to ship:** +- `evals/math_symbolic_equivalence/v1/sealed_holdout.age` — pyrage-X25519-encrypted JSONL of ~10–15 hand-curated cases. **Categories must match the curated 30 cases** (no novel categories — the sealed set is a held-out *sample* of the same distribution, not a different benchmark). +- `evals/math_symbolic_equivalence/v1/sealed_holdout.pubkey` — X25519 public key (private key stays off-repo per ADR-0119.7). +- `evals/math_symbolic_equivalence/v1/sealed_runner.py` — CLI that decrypts (env var `CORE_SEALED_KEY`) and runs the lane; refuses cleanly with typed error when key absent. Writes `sealed_report.json`. +- `tests/test_adr_0131_1_sealed_holdout.py` — 4–6 tests: encrypted file exists, decryption refuses without key, decrypted contents parse as valid case JSON, sealed-runner exit criterion matches public-runner (gate: `wrong == 0`, `correct_rate ≥ 0.95`). +- `docs/decisions/ADR-0131.1.S-sealed-holdout.md` — short ADR; cite ADR-0131 parent, ADR-0119.7 as methodology source, and #167 as the lane being hardened. + +**Hard constraints:** +- **No peek.** The cases.jsonl on main is the public corpus; the sealed corpus must not duplicate any case_id or expression-pair from it. Lane test asserts disjointness post-decryption. +- **Same engine, no scope expansion.** The sealed runner calls the same `check_equivalence` API as the public runner. **Do not depend on PR #169** (the v1.B refactor) — that branch is in draft and may not land first. Stay within v1's univariate-integer-polynomial scope. +- **Refusal-first holds.** Out-of-scope sealed cases must refuse, not coerce. +- **Determinism.** `sealed_report.json` byte-equal across runs when `CORE_SEALED_KEY` set. +- **Trust boundary.** Decryption is the only filesystem-write you do; no logging of decrypted contents, no echo of raw cases to stdout. Cite the trust-boundary section of CLAUDE.md in the ADR. + +**Out of scope (do not touch):** +- B2 / B3 lanes. +- Binding-graph implementation. +- #169 v1.B hardening — work strictly within v1's contract. +- Promotion-gate wiring. + +**Coordinate with #169.** If #169 reconciles and merges *before* you open, rebase your branch onto the resulting main; the sealed cases must still pass under the v1.B engine (which is a strict superset of v1, so they should). If they don't, refuse to widen scope — flag it and stop. + +**Target branch.** PR against `main`. Title: `feat(ADR-0131.1.S): sealed holdout for symbolic equivalence v1`. Body must include lane result on sealed split + scope-discipline section. + +**Exit criterion.** PR opens with CI green, sealed runner exits 0 (`wrong == 0`, gate passed) under a CI-provided key, ADR-0131.1.S included. + +**Do not stack on another agent's branch.** Target main directly. diff --git a/docs/briefs/parallel-2026-05-23/L4-opus2-binding-graph-phase2-adapter.md b/docs/briefs/parallel-2026-05-23/L4-opus2-binding-graph-phase2-adapter.md new file mode 100644 index 00000000..446afa7b --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L4-opus2-binding-graph-phase2-adapter.md @@ -0,0 +1,48 @@ +# L4 brief — Opus#2 — Binding Graph Phase 2 (adapter from `MathProblemGraph`) + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-binding-graph-p2 -b feat/binding-graph-phase2-adapter origin/main +cd ../core-binding-graph-p2 +``` + +**Scope.** Phase 2 of the binding-graph layer (ADR-0132 ratified Phase 1; #171 merged). Add a **pure-function adapter** that converts an existing `generate.math_problem_graph.MathProblemGraph` into a `SemanticSymbolicBindingGraph`. Still no runtime wiring — Phase 2 is structural translation only. Phases 3 (unit-aware equation binding), 4 (question-target binding), and 5 (bounded-grammar integration / B3) remain deferred to follow-up PRs. + +**Reference docs (read these, only these):** +1. `docs/decisions/ADR-0132-binding-graph-data-model.md` — your Phase 1 ADR; the data model is the codomain of the adapter. +2. `generate/math_problem_graph.py` — the input type. Note its determinism guarantees (frozen+slots, order-of-introduction tuples, `canonical_bytes`). The adapter must preserve those. + +**What to ship:** +- `generate/binding_graph/adapter.py` — `bind_math_problem_graph(g: MathProblemGraph) -> SemanticSymbolicBindingGraph`. Pure function. Deterministic: byte-equal input graph → byte-equal output binding graph (assert via `canonical_bytes`-style round-trip in tests). +- Mapping discipline (lock these as constants at the top of `adapter.py`): + - Each entity → one `SymbolBinding` with `semantic_role="entity"`, `symbol_id = entity_`. + - Each `InitialPossession` → one `SymbolBinding` with `semantic_role="quantity"` and a `BoundFact` (`q___t0 = value [unit]`). + - Each `Operation` → at minimum one `BoundEquation` capturing the operand semantics, with `dependencies` referencing the source-state symbols. `operation_kind` mirrors the `MathProblemGraph` op kind verbatim (string passthrough — the closed vocab is shared by design). + - `Unknown` → one `BoundUnknown` referencing the bound symbol whose value the solver must determine. +- Public surface: add `bind_math_problem_graph` to `generate/binding_graph/__init__.py`. +- `tests/test_binding_graph_adapter.py` — 30–40 tests covering: every operation kind round-trips, entity/quantity mapping is deterministic across orderings (introduction-order preserved), unknown-binding is single-target, `source_span` propagation when the `MathProblemGraph` carries span info (skip span fields cleanly when absent), refusal-first on malformed input (`AdapterError` typed exception, sibling of `BindingGraphError`), and a hash-stability test (`bind(g) == bind(g)` byte-equal across runs). +- `docs/decisions/ADR-0133-binding-graph-adapter.md` — short ADR ratifying Phase 2; cite ADR-0132 parent and ADR-0115 (`MathProblemGraph` origin). Explicit "Phase 3+ deferred" section. + +**Hard constraints:** +- **Pure function, pure data.** No I/O, no parser calls, no solver calls, no `numpy`. The adapter is a deterministic translation. +- **No mutation.** Output is a new `SemanticSymbolicBindingGraph`; input is not touched (it can't be — frozen — but assert the invariant in tests anyway). +- **Operation-kind passthrough.** Do not invent a new closed vocab. The `MathProblemGraph.VALID_OPERATION_KINDS` set is authoritative; `BoundEquation.operation_kind` accepts the same eight strings. If you find yourself wanting to translate or rename, stop and refuse. +- **Determinism is load-bearing.** The tuples in the output BindingGraph must reflect introduction order from the input. Two `MathProblemGraph` instances that hash-equal must produce two `SemanticSymbolicBindingGraph` instances that hash-equal. +- **Unit-aware *equation binding* is Phase 3, not Phase 2.** In Phase 2 it is acceptable for `BoundEquation` to carry `unit_proof = None` or a placeholder; do not attempt dimensional analysis. (Confirm in the ADR that this gap is by design and the next ADR closes it.) +- **Field invariant untouched.** Don't go near `versor_condition`, `algebra/`, or anything in the runtime hot path. + +**Out of scope (do not touch — Phase 3+):** +- Unit-aware equation admissibility checks (Phase 3 / ADR-0134). +- Question-target binding refinement (Phase 4). +- Bounded-grammar / B3 integration (Phase 5). +- Any change to `chat/`, `core/`, `generate/intent.py`, `generate/realizer.py`, `generate/math_problem_graph.py` itself. +- B1.B (PR #169 — separate lane). +- B2 teaching-corpus eval (L1 — Gemini's lane). +- B1 sealed holdout (L3 — undispatched). + +**Target branch.** PR against `main`. Title: `feat(binding-graph): Phase 2 adapter from MathProblemGraph (ADR-0133)`. Body must reference #170 (proposal), #171 (Phase 1), and explicitly list Phase 3–5 as deferred. + +**Exit criterion.** PR opens with CI green, all new tests pass, `pyright` clean on new files, ADR-0133 included. Runtime behavior byte-identical to main (no integration yet, by design). + +**Do not stack on another agent's branch.** Target main directly. If main does not yet contain ADR-0132 + Phase 1, stop and flag — Phase 2 requires that foundation. diff --git a/docs/briefs/parallel-2026-05-23/L5-opus2-binding-graph-phase3-units.md b/docs/briefs/parallel-2026-05-23/L5-opus2-binding-graph-phase3-units.md new file mode 100644 index 00000000..8b9e7537 --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L5-opus2-binding-graph-phase3-units.md @@ -0,0 +1,57 @@ +# L5 brief — Opus#2 — Binding Graph Phase 3 (unit-aware equation admissibility) + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-binding-graph-p3 -b feat/binding-graph-phase3-units origin/main +cd ../core-binding-graph-p3 +``` + +**Scope.** Phase 3 of the binding-graph layer. Phase 1 (#171, ADR-0132) shipped the data model with `BoundEquation.unit_proof` as a placeholder. Phase 2 (#174, ADR-0133) shipped the `MathProblemGraph` adapter and explicitly left `unit_proof = None` as a Phase-3 gap. Close it: add **deterministic, refusal-first unit-aware admissibility checking** that populates `unit_proof` and stamps `admissibility_status` on every `BoundEquation` produced by the adapter (or constructed directly). + +Still no runtime wiring outside `generate/binding_graph/` and no solver invocation — Phase 3 is structural and dimensional analysis only. Phase 4 (question-target binding refinement) and Phase 5 (B3 / bounded-grammar integration) remain deferred. + +**Reference docs (read these, only these):** +1. `docs/decisions/ADR-0132-binding-graph-data-model.md` + `docs/decisions/ADR-0133-binding-graph-adapter.md` — your prior phases. The Phase-1 dataclass shape (`BoundEquation.unit_proof: UnitProof | None`, `admissibility_status: Literal[...]`) is the contract you're filling in. +2. `packs/en_units_v1/` (loaded into the runtime via ADR-0127) — the **canonical unit vocabulary**. Your unit algebra must be expressed in those units only; do not invent new ones. Read the pack's lexicon + relation files to identify the closed set of units and their dimensional families (length, time, mass, currency, count, rate-of-X-per-Y, etc.). + +**What to ship:** +- `generate/binding_graph/units.py` — pure unit algebra. Functions: + - `parse_unit(canonical_id: str) -> UnitVector` — convert a closed-vocab unit id into a frozen `UnitVector` (dimension exponents over the base dimensions in the units pack). + - `unit_product(a: UnitVector, b: UnitVector) -> UnitVector`, `unit_quotient`, `unit_inverse` — algebra primitives. + - `units_equal(a, b)` — strict equality on the dimension-exponent vector. + - All pure, frozen, deterministic. No I/O at call time. +- `generate/binding_graph/admissibility.py` — `check_admissibility(equation: BoundEquation, *, symbols: Mapping[str, SymbolBinding]) -> AdmissibilityResult`. Returns a `UnitProof` on success; a typed `AdmissibilityError` reason on refusal. **Dispatch is on `operation_kind`** — the closed eight-string vocab is your full case table: + - `add` / `subtract` / `compare_additive`: lhs unit ≡ rhs unit ≡ each operand unit; refuse if any disagree. + - `multiply` / `divide`: lhs unit = product / quotient of operand units; no equality requirement among operands. + - `transfer`: actor and target must share the same unit; lhs unit = that unit. + - `apply_rate`: lhs unit = rate-unit × duration-unit (or equivalent); the rate must have form `X/Y` where Y is the operand's unit. + - `compare_multiplicative`: lhs is dimensionless (ratio); operand units must be equal so they cancel. + - Any operand whose unit is `None` (unbound) → refuse with `unit_unbound`. + - Any equation whose `symbols` lookup is missing → refuse with `unknown_symbol`. +- `generate/binding_graph/adapter.py` (modify, surgically): wire `check_admissibility` into the adapter so every `BoundEquation` it produces carries either `admissibility_status="admissible"` + populated `unit_proof`, or `admissibility_status="refused"` + populated `refusal_reason`. **Do not change the adapter's input/output types or signature.** Existing 41 Phase-2 tests must still pass. +- Public surface: add `check_admissibility`, `UnitVector`, `UnitProof`, `AdmissibilityError`, `parse_unit` (and minimal algebra primitives) to `generate/binding_graph/__init__.py`. +- `tests/test_binding_graph_units.py` — 30–40 tests covering the algebra primitives (commutativity, inverse-of-inverse, refusal on unknown unit-id). +- `tests/test_binding_graph_admissibility.py` — 50–70 tests covering each operation kind's admissibility rules with both positive and negative cases, plus the typed-refusal contract. +- `tests/test_binding_graph_adapter_units.py` — 20–30 integration tests covering the adapter's Phase-3 behavior: every Phase-2 case still round-trips, but now `unit_proof` is populated (or refusal is typed); a few intentionally unit-mismatched `MathProblemGraph` inputs produce `admissibility_status="refused"` with the correct reason. +- `docs/decisions/ADR-0134-binding-graph-admissibility.md` — short ADR; cite ADR-0132 + ADR-0133 parents and ADR-0127 (units pack). Explicit "Phase 4–5 deferred" section. + +**Hard constraints:** +- **Closed unit vocabulary.** Every unit id used in algebra must already exist in `en_units_v1`. If you find yourself wanting to invent or coerce, stop and refuse with `unknown_unit`. +- **Refusal-first.** Unit mismatches never silently coerce. The `BoundEquation` exits the adapter with `admissibility_status="refused"` and a typed `refusal_reason`. Never raise from the adapter on a unit mismatch — the data-model has a slot for it; use the slot. +- **Determinism.** `unit_product(a, b) == unit_product(b, a)` byte-equal; `bind_math_problem_graph(g)` byte-equal across runs (Phase-2 invariant must not regress). +- **Pure, no I/O.** Unit-pack data must be loaded at module import (or lazily memoized) and frozen; no per-call filesystem reads. +- **No solver coupling.** You're not solving equations. You're verifying the equation, if solved, would be dimensionally consistent. +- **Field invariant untouched.** Stay out of `algebra/`, `chat/`, `core/`, `generate/intent.py`, `generate/realizer.py`, runtime hot path. + +**Out of scope (do not touch — Phase 4+):** +- Question-target binding refinement (Phase 4). +- Bounded-grammar / B3 integration (Phase 5). +- `MathProblemGraph` itself (input type — read-only here). +- Symbolic equivalence engine (#167, #169 — separate lane). + +**Target branch.** PR against `main`. Title: `feat(binding-graph): Phase 3 unit-aware admissibility (ADR-0134)`. Body must reference ADR-0132, ADR-0133, ADR-0127 (units pack); list Phase 4–5 deferred; report test count (expect ~210 total: 110 from Phase 1+2 still passing, +100ish new). + +**Exit criterion.** PR opens with CI green, all new + existing binding-graph tests pass, `pyright` clean on new files, ADR-0134 included. Runtime behavior byte-identical to main (no integration yet). + +**Do not stack on another agent's branch.** Target main directly. diff --git a/docs/briefs/parallel-2026-05-23/L6-gemini-B2-dataset-enrichment.md b/docs/briefs/parallel-2026-05-23/L6-gemini-B2-dataset-enrichment.md new file mode 100644 index 00000000..2d9235fc --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L6-gemini-B2-dataset-enrichment.md @@ -0,0 +1,47 @@ +# L6 brief — Gemini — ADR-0131.2.B B2 dataset enrichment (load-bearing curation) + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-adr-0131-2b-enrichment -b feat/adr-0131-2b-b2-enrichment origin/main +cd ../core-adr-0131-2b-enrichment +``` + +**Scope.** Strengthen the B2 teaching-corpus math eval (shipped in #172, ADR-0131.2) so its lane gate is *load-bearing*, not trivially satisfied. The v1 dataset you shipped passes the gate as specified — but every chain has `expected="replay_equivalent"` and every chain cites the same cognition-pack evidence ref (`cause_truth_grounds_knowledge`). That makes `wrong == 0` mechanically true regardless of engine behavior. v1.B closes that gap. + +**Reference docs (read these, only these):** +1. `docs/decisions/ADR-0131.2-teaching-corpus-eval.md` (your prior ADR; the v1 contract you're extending — not breaking). +2. `evals/math_symbolic_equivalence/v1/cases.jsonl` + `generated_cases.py` (post-#169 if it has merged, otherwise post-#167) — pattern reference for **positive + negative + refused** case mixing in a single lane file. Mirror that shape. + +**What to ship:** +- `teaching/math_corpora/math_teaching_v1.jsonl` (modify): replace the placeholder evidence ref `cause_truth_grounds_knowledge` with **honest grounding refs**. Each chain must cite at least one ref from a **math-relevant** source — either: + - a chain id internal to this same corpus that the new chain builds on, OR + - a lemma id from `en_mathematics_logic_v1` / `en_arithmetic_v1` / `en_units_v1` packs that the chain references. + Do not invent refs that don't exist. Mass placeholder reuse is the v1 sin — fix it. +- `evals/math_teaching_corpus/v1/cases.jsonl` (modify + extend): keep the 30 positive `expected="replay_equivalent"` cases (renumbered if needed), and **add ~10 negative cases** of two flavors: + - `expected="not_equivalent"` — chains whose propose-shape produces a candidate that the existing teaching loop will *correctly reject* (e.g., subject lemma not in the math pack; redundant chain already in corpus; missing required evidence; cycle in chain graph). The point is to exercise the refusal/rejection paths so `wrong == 0` becomes a real assertion. + - `expected="refused"` — chains whose propose call raises `ProposalError` deterministically (e.g., malformed `proposed_chain`, empty subject). Mirrors the symbolic-equivalence refusal-first contract. +- `evals/math_teaching_corpus/v1/runner.py` (light modify): nothing structural — the existing `_score_one` already handles `rejected`/`pending`/`ProposalError` arms; the new cases just exercise the previously-untaken branches. +- `evals/math_teaching_corpus/v1/README.md` (modify): document the v1.B mix (positive + negative + refused), update gate language to "wrong == 0 across all three case classes". +- `tests/test_adr_0131_2_teaching_corpus_lane.py` (modify): add a test that asserts the dataset contains **at least one case of each expected class** (`replay_equivalent`, `not_equivalent`, `refused`). This is what makes the gate non-trivial — without diversity, the assertion is empty. +- `docs/decisions/ADR-0131.2.B-teaching-corpus-enrichment.md` (new): short follow-up ADR; cite ADR-0131.2 parent; explicitly call out that v1's gate was trivially satisfied and v1.B is the load-bearing version. Be honest in the framing. + +**Hard constraints:** +- **Stay in the single bounded math micro-domain you chose for v1** (`en_mathematics_logic_v1`). Do not widen the domain — the bounded-domain rule from the original L1 brief still holds. +- **No real corpus mutation.** The runner's `tempfile.TemporaryDirectory()` isolation must continue to hold. Verify nothing you change leaks into live `teaching/proposals.jsonl` or the cognition corpus. +- **Refusal-first.** Refused cases must produce `ProposalError` from the live `propose_from_candidate`, not synthetic exceptions in the runner. If you cannot deterministically trigger a refusal in `propose_from_candidate`, drop that case category and document why; do **not** fake it. +- **Determinism.** `report.json` byte-equal across runs (existing invariant; new cases must not introduce nondeterminism). +- **No runtime path changes.** Same constraint as v1. +- **Honest evidence.** Every evidence ref you cite must resolve to a real corpus chain or pack lemma. Lane test should fail-loud if any ref is dangling. + +**Out of scope (do not touch):** +- B1.B (#169) — separate lane, may or may not have merged when you start. +- Binding-graph PRs (#171, #174, ongoing L5) — different domain. +- Sealed holdout for B1 (#173) — already merged; mirror its honesty in your enrichment, but don't change it. +- Promotion-gate wiring (ADR-0131.4) — sequential. + +**Target branch.** PR against `main`. Title: `feat(ADR-0131.2.B): B2 teaching-corpus enrichment — load-bearing gate`. Body must include: counts by `expected` class, dangling-ref check result, and an honest acknowledgement that v1's gate was trivially satisfied and v1.B fixes that. + +**Exit criterion.** PR opens with CI green, lane runner exits 0 with the new mix, `wrong == 0` across **all three case classes** (not just replay_equivalent), and ADR-0131.2.B included. Existing lane tests still pass; new diversity test passes. + +**Do not stack on another agent's branch.** Target main directly. diff --git a/docs/briefs/parallel-2026-05-23/L7-opus2-binding-graph-phase4-question-target.md b/docs/briefs/parallel-2026-05-23/L7-opus2-binding-graph-phase4-question-target.md new file mode 100644 index 00000000..8c0c6138 --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L7-opus2-binding-graph-phase4-question-target.md @@ -0,0 +1,61 @@ +# L7 brief — Opus#2 — Binding Graph Phase 4 (question-target binding refinement) + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-binding-graph-p4 -b feat/binding-graph-phase4-question-target origin/main +cd ../core-binding-graph-p4 +``` + +**Scope.** Phase 4 of the binding-graph layer. Phase 1 (#171, ADR-0132) introduced `BoundUnknown` as the question target. Phase 2 (#174, ADR-0133) mapped `MathProblemGraph.Unknown` to a `BoundUnknown` referencing the entity/unit pair. Phase 3 (#176, ADR-0134) added unit-aware admissibility on equations. **Phase 4 refines `BoundUnknown` from "the symbol whose value the solver determines" to "the symbol at a specific temporal/state index with a specific question-form."** + +Concretely: the question "How many apples does Tina have?" doesn't currently bind to a unique symbol — Tina-apples at `t=0`, after operation 3, and at the terminal state are three different symbols. Phase 4 deterministically resolves *which* of those the unknown refers to, and stamps the question form (count / rate / total / difference / ratio / identity) so downstream consumers know what verdict shape to produce. + +Still no runtime wiring, still no solver invocation. Phase 5 (bounded-grammar / B3 integration) remains deferred. + +**Reference docs (read these, only these):** +1. `docs/decisions/ADR-0132-binding-graph-data-model.md`, `ADR-0133-binding-graph-adapter.md`, `ADR-0134-binding-graph-admissibility.md` — your prior phases. `BoundUnknown` and the adapter's `Unknown → BoundUnknown` mapping are the contract you're refining. +2. `generate/math_problem_graph.py` — input shape; the operation sequence drives the state-index resolution. + +**What to ship:** +- `generate/binding_graph/model.py` (modify, surgically): extend `BoundUnknown` with two new mandatory fields: + - `state_index: StateIndex` — a closed-vocab tagged union: `Literal["initial"]`, `Literal["terminal"]`, or `Operation(operation_index=int)` (a frozen dataclass — not a string — so the operation index is type-checked). + - `question_form: Literal["count", "rate", "total", "difference", "ratio", "identity"]`. + Both fields **required** — no `None` defaults. Update `__post_init__` to validate (`operation_index` ≥ 0 and < graph operation count when used inside a `SemanticSymbolicBindingGraph`; cross-collection check belongs to the graph constructor). + Phase 1–3 tests that construct `BoundUnknown` directly will need the new fields supplied; expect ~5–10 test fixture updates inside the binding_graph test files (Phase 1+2+3 by you). Mark those edits explicitly in the PR diff so the reconciliation is auditable. +- `generate/binding_graph/question_target.py` (new) — pure-function resolver: + - `resolve_state_index(g: MathProblemGraph) -> StateIndex` — walks the operation sequence, returns `terminal` when operations exist, `initial` when not. + - `infer_question_form(g: MathProblemGraph) -> Literal[...]` — closed-vocab dispatch on the operation kinds touching the unknown's entity/unit; e.g. all-`add`/`subtract`/`transfer` → `count`; presence of `apply_rate` → `total` or `rate` depending on which symbol the unknown binds; `compare_additive` → `difference`; `compare_multiplicative` → `ratio`; no operations → `identity`. + - `bound_unknown_from_math_problem_graph(g: MathProblemGraph) -> BoundUnknown` — the new public entry point that the adapter uses; deterministic, refusal-first via a typed `QuestionTargetError` (sibling of `AdapterError`/`AdmissibilityError`). +- `generate/binding_graph/adapter.py` (modify, surgically): replace the existing `Unknown → BoundUnknown` mapping with a call to `bound_unknown_from_math_problem_graph(g)`. Do not change `bind_math_problem_graph`'s signature; do not change the equation/fact/symbol mapping. +- Public surface: add `StateIndex`, `Operation` (state-index variant), `resolve_state_index`, `infer_question_form`, `bound_unknown_from_math_problem_graph`, `QuestionTargetError` to `generate/binding_graph/__init__.py`. +- `tests/test_binding_graph_question_target.py` — 50–70 tests covering: each operation-kind family producing the expected `question_form`; `state_index` resolution at boundary cases (0 operations, 1 operation, many operations, unknown referencing an entity that no operation touches); refusal-first on the typed `QuestionTargetError` paths (unknown entity not in `entities`, ambiguous form when multiple operation kinds touch the same symbol with no clear precedence — pick one closed rule and document it). +- `tests/test_binding_graph_adapter_question_target.py` — 30–40 integration tests covering: every Phase-2 adapter test case still round-trips with the refined `BoundUnknown`; new cases exercising `state_index` and `question_form` end-to-end; hash-stability across runs (Phase 2 invariant must not regress); a few intentionally-ambiguous `MathProblemGraph` inputs produce `QuestionTargetError`. +- `docs/decisions/ADR-0135-binding-graph-question-target.md` — short ADR. Cite ADR-0132, ADR-0133, ADR-0134 parents. Explicit "Phase 5 deferred" section. Document the closed `question_form` vocabulary and the precedence rule when multiple operation kinds touch the same symbol. + +**Hard constraints:** +- **Closed `question_form` vocabulary.** Exactly the six values above; do not invent more. If a `MathProblemGraph` shape doesn't fit any, refuse with `QuestionTargetError(reason="unmappable_question_form")`. Never silently coerce to "count" as a default. +- **Closed `state_index` vocabulary.** `initial`, `terminal`, or `Operation(operation_index=N)`. No "midway", no negative indices, no strings outside that set. +- **Refusal-first.** Ambiguity refuses with a typed reason; never picks arbitrarily. The precedence rule (when multiple op-kinds touch the unknown's symbol) must be deterministic and documented in the ADR. +- **Pure, no I/O.** No filesystem reads at call time. +- **Determinism.** `bind_math_problem_graph(g)` byte-equal across runs after the Phase 4 wiring (Phase 2 invariant). The `BoundUnknown`'s new fields are stable functions of the input. +- **No solver coupling.** You're determining *which symbol* the question targets and *what form* the answer takes — not solving anything. +- **No mutation of `MathProblemGraph`.** Input is read-only (frozen anyway, but assert in tests). +- **Field invariant untouched.** Stay out of `algebra/`, `chat/`, `core/`, runtime hot path. + +**Compatibility / fixture migration:** +- The two new required fields on `BoundUnknown` will break ~5–10 Phase 1+2+3 test fixtures that construct `BoundUnknown` directly. Update them in the same PR. Do not add default values to keep them backward-compat — required fields are load-bearing. +- The Phase 2 adapter's behavior changes (`Unknown → BoundUnknown` now populates the new fields). The byte-equal hash-stability invariant must continue to hold on a fresh-run basis (hash will differ from Phase 3 main, by design — that's not a regression). + +**Out of scope (do not touch — Phase 5):** +- Bounded-grammar / B3 integration. +- Any change to `chat/`, `core/`, `generate/intent.py`, `generate/realizer.py`, `generate/math_problem_graph.py`. +- B1.B and B1.S follow-ups (separate lanes). +- B2 v1.B enrichment (L6, Gemini's lane). +- Promotion-gate wiring (ADR-0131.4). + +**Target branch.** PR against `main`. Title: `feat(binding-graph): Phase 4 question-target binding (ADR-0135)`. Body must reference ADR-0132 / -0133 / -0134; list Phase 5 deferred; report test count (expect ~300+ total: 226 from Phase 1+2+3 still passing, +80ish new). + +**Exit criterion.** PR opens with CI green, all new + existing binding-graph tests pass, `pyright` clean on new files, ADR-0135 included. Runtime behavior byte-identical to main (no integration yet). + +**Do not stack on another agent's branch.** Target main directly. If main does not yet contain ADR-0132 + -0133 + -0134, stop and flag — Phase 4 requires all three. diff --git a/docs/briefs/parallel-2026-05-23/L8-B3-bounded-grammar-word-problems.md b/docs/briefs/parallel-2026-05-23/L8-B3-bounded-grammar-word-problems.md new file mode 100644 index 00000000..d6a98e51 --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L8-B3-bounded-grammar-word-problems.md @@ -0,0 +1,61 @@ +# L8 brief — (agent assignment TBD by lead) — ADR-0131.3 Benchmark 3: bounded-grammar word problems + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-adr-0131-3-bounded-grammar -b feat/adr-0131-3-bounded-grammar origin/main +cd ../core-adr-0131-3-bounded-grammar +``` + +**Scope.** Third and final benchmark of the composite math-expert promotion gate from ADR-0131. B3 demonstrates the architecture's ability to handle **word problems** — but only those expressible in a **bounded grammar**, the deterministic subset the system can recognize. Outside the bounded grammar: refuse, never guess. Inside: solve through the full typed pipeline (parser → `MathProblemGraph` → binding graph → solver → verifier), with `wrong == 0`. + +This is the benchmark that closes the GSM8K-arc lesson. Instead of paraphrase-chasing a benchmark designed to reward flexibility, B3 measures the architecture-aligned claim: *within a bounded grammar, deterministic end-to-end correctness*. The grammar's boundedness is the contract; coverage is a separate, honest, measurable property — not a promise. + +**Reference docs (read these, only these):** +1. `docs/decisions/ADR-0131-math-expert-rebench.md` — the composite-gate framing; B1 and B2 are siblings. +2. `docs/decisions/ADR-0115-math-problem-parser-and-graph.md` + `ADR-0116-deterministic-solver.md` + `ADR-0117-solution-trace-verifier.md` — the existing pipeline B3 exercises. Read what they already do; do not reinvent. + +**What to ship:** +- `evals/math_bounded_grammar/v1/grammar.md` — the **bounded grammar specification**. A small, explicit set of sentence templates the parser recognizes. Examples (illustrative, not exhaustive — pick a coherent set and pin it): + - ` has .` + - ` gives to .` + - ` has times as many as .` + - `How many does have?` + - `Each costs . buys . How much does spend?` + Each template documented with its `MathProblemGraph` mapping. The grammar must be **closed** — every sentence either matches a template or is refused. +- `evals/math_bounded_grammar/v1/cases.jsonl` — ~40–60 hand-curated word problems split across **three classes** (mirror the B2 v1.B shape): + - `expected="solved_correct"` (~30 cases): grammar-conformant problems with a single correct numeric answer. Span the full set of templates and operation kinds (`add`, `subtract`, `transfer`, `multiply`, `divide`, `apply_rate`, `compare_additive`, `compare_multiplicative`). + - `expected="solved_wrong"` (~5 cases): grammar-conformant problems whose expected answer is **deliberately wrong** — exercises that the verifier catches solver errors, not just parser errors. (Use these sparingly; their purpose is to assert `wrong == 0` is load-bearing, not trivially satisfied.) + - `expected="refused"` (~10–15 cases): problems outside the bounded grammar that must refuse cleanly through the parser's typed-refusal contract. Categories: paraphrase outside the templates, unit not in `en_units_v1`, ambiguous entity reference, multi-step problems beyond the grammar's depth, etc. +- `evals/math_bounded_grammar/v1/runner.py` — CLI runner. For each case: invoke the pipeline end-to-end (parser → `MathProblemGraph` → `bind_math_problem_graph` → admissibility → solver → verifier). Compare verifier output against `expected`. Emit `report.json` (deterministic, byte-equal across runs). +- `evals/math_bounded_grammar/v1/README.md` — methodology, scope, dataset categorization, exit criterion, link to the grammar spec. +- `tests/test_adr_0131_3_bounded_grammar_lane.py` — 10–15 lane tests: + - Dataset integrity (case_ids unique, every case_id matches one of the documented categories). + - **Grammar closure**: every `solved_correct` and `solved_wrong` case parses; every `refused` case raises the typed refusal — assert at the parser level, not just at end-of-pipeline. + - Exit criterion (`wrong == 0` across all three classes; `correct_rate ≥ 0.95`). + - Replay byte-equality of `report.json` across runs. + - **Class diversity**: at least one case of each `expected` class (same load-bearing-gate logic as B2 v1.B). + - Operation-kind coverage: each of the 8 `VALID_OPERATION_KINDS` exercised by at least one `solved_correct` case. +- `docs/decisions/ADR-0131.3-bounded-grammar.md` — the ADR. Cite ADR-0131 parent, ADR-0115/-0116/-0117 (pipeline), ADR-0132 through ADR-0135 (binding graph) as foundation. Document the bounded grammar as a **scope statement, not a coverage claim** — pin language like "the architecture solves problems in this grammar; problems outside it refuse." Defer wider grammar expansion to v1.B / future ADRs. + +**Hard constraints:** +- **Closed grammar.** Every recognized sentence matches exactly one template. Ambiguity → refuse. No paraphrase tolerance; the grammar is the contract. +- **No new parser logic if avoidable.** If `generate/math_parser.py` already accepts the bounded grammar, route through it. If it doesn't, **prefer extending it** (in this PR) over building a parallel parser — but extensions must stay within the templates documented in `grammar.md`. +- **End-to-end pipeline.** The runner must invoke the **real solver + real verifier** for `solved_correct` and `solved_wrong` cases. No shortcuts; no mock solver. If the existing solver doesn't handle a template, drop that template from v1 and document it as deferred — do not stub. +- **Refusal-first.** Out-of-grammar cases refuse at the **parser** layer or, if the parser admits but downstream refuses, at the **binding-graph admissibility** layer (Phase 3, ADR-0134). Either is acceptable as long as the refusal is typed and deterministic. Never silently coerce. +- **Determinism.** `report.json` byte-equal across runs. Two consecutive runs produce identical hashes. +- **Field invariant untouched.** No changes to `algebra/`, `chat/`, `core/`. The pipeline runs in its existing shape. +- **Honest framing.** The PR body and README must make clear that B3's lane gate measures *correctness within the bounded grammar*, not coverage of natural-language math. The composite gate (B1+B2+B3) is the structural claim, not a free-form NL claim. + +**Out of scope (do not touch):** +- L7 binding-graph Phase 4 (Opus#2's lane) — depend on Phase 1+2+3 (already on main) only. If Phase 4 lands before this PR opens, rebase and adopt the refined `BoundUnknown` signature; do not block on it. +- Promotion-gate wiring (ADR-0131.4 — sequential, after all three benchmarks land). +- GSM8K parser work, GSM8K dataset, GSM8K removal amendment (ADR-0131.5 — separate, docs-only follow-up). +- B1.B and B1.S follow-ups. +- B2 v1.B enrichment (L6 lane). + +**Target branch.** PR against `main`. Title: `feat(ADR-0131.3): bounded-grammar word-problem benchmark — lane PASSED N/N`. Body must include: counts by `expected` class, operation-kind coverage table, the grammar's template list (or link to grammar.md), and an honest acknowledgement of the scope statement (bounded, not free-form). + +**Exit criterion.** PR opens with CI green, lane runner exits 0 with `wrong == 0` across all three classes, `correct_rate ≥ 0.95`, every operation kind exercised, byte-equal report across runs, ADR-0131.3 included. + +**Do not stack on another agent's branch.** Target main directly. diff --git a/docs/briefs/parallel-2026-05-23/L9-ADR-0131-G1-verb-classes-initial-state.md b/docs/briefs/parallel-2026-05-23/L9-ADR-0131-G1-verb-classes-initial-state.md new file mode 100644 index 00000000..e1b3b030 --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/L9-ADR-0131-G1-verb-classes-initial-state.md @@ -0,0 +1,40 @@ +# L9 brief — ADR-0131.G.1 — Capability axis: state-introducing verb classes + +**Worktree setup (do this first, non-negotiable):** + +```bash +git worktree add ../core-adr-0131-g1-verb-classes -b feat/adr-0131-g1-verb-classes origin/main +cd ../core-adr-0131-g1-verb-classes +``` + +**Scope.** First capability-axis iteration on top of ADR-0131.G's coverage probe. Extends the **initial-possession candidate emitter** to recognize a closed set of *state-introducing* verbs beyond the current `has/have/is/are/was/were`. Target verbs (drawn directly from the baseline `refused_reasons_top` clusters in `train_sample_coverage_report.json`): `bought / buys`, `sells`, `collected`, `saved (up)`, `started (with)`, `had`, `makes` *(rate-introducing — only the initial-state reading, not the rate reading; that's L11/G.3's axis)*. + +The architectural claim: each of these verbs introduces a quantity into the same `InitialPossession` shape ` ` — semantically equivalent to `has`, syntactically distinct. The candidate emitter widens; the binding graph and solver are untouched. + +**Reference docs (read these, only these):** +1. `docs/decisions/ADR-0131.G-gsm8k-coverage-probe.md` — the iteration discipline; every G. must do all 4 items in its "Every subsequent ADR-0131.G. must" section. +2. `generate/math_candidate_parser.py` — extend `_INITIAL_HAS_RE` / `CandidateInitial` (do not branch into a new module; this is the same shape with a wider anchor alternation). + +**What to ship:** +- **Parser extension** in `generate/math_candidate_parser.py`: widen `CandidateInitial.matched_anchor` validation set and `_INITIAL_HAS_RE` anchor alternation to include the listed verbs. Past-tense forms (`bought`, `collected`, `saved`, `had`, `started`) are first-class. Round-trip filter stays the firewall — wider grammar, same wrong-rate-zero. +- **Curated coverage cases** at `evals/math_capability_axes/G1_verb_classes/v1/cases.jsonl` (~20 cases, split by verb): each is ` .` followed by a question the existing pipeline can answer. These exercise the new capability **independently of GSM8K**. +- **Runner** at `evals/math_capability_axes/G1_verb_classes/v1/runner.py` (pure adapter over `evals.gsm8k_math.runner` infrastructure or `chat/runtime` end-to-end; deterministic `report.json`). +- **Tests** at `tests/test_adr_0131_G1_verb_classes.py` (~10): safety rail (`wrong == 0`) on the new axis; per-verb at-least-one passing case; closed outcome vocab; replay byte-equality; **GSM8K probe re-runs and `admission_rate` strictly increases** vs the baseline `0.0`; B3 lane unchanged. +- **ADR** `docs/decisions/ADR-0131.G.1-verb-classes-initial-state.md`. Cite ADR-0131.G parent; document the closed verb set as a scope statement; declare what is *not* added (no rate verbs, no comparatives, no acquisition-with-cost semantics — those are sibling axes). +- **Refresh** `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json` (the diff-able number). PR title must include the new admission_rate. + +**Hard constraints:** +- **Closed verb set.** Every added anchor is enumerated in the ADR and in code. No paraphrase tolerance, no synonym expansion. +- **`wrong == 0`** preserved on both the new axis lane and the GSM8K probe. If round-trip would let a wrong answer through on any new verb, **shrink the verb set**, do not weaken the filter. +- **Initial-state reading only.** `makes` in "Tina makes $18.00 an hour" is rate-introducing — that case must continue to refuse (or route to L11). Verify with an explicit adversarial probe case. +- **No new modules under `algebra/`, `chat/`, `core/`.** No changes to the binding graph, solver, or verifier. +- **Determinism.** Report byte-equal across runs. +- **Smell test (ADR-0131.G):** if admission moves on GSM8K but the new axis cases don't all pass, the change is a template in disguise — reject. + +**Out of scope:** rate verbs (L11/G.3), comparatives (L10/G.2), conjoined subjects (L12/G.4), commerce-with-price semantics, multi-statement coreference. + +**Target branch.** PR against `main`. Title: `feat(ADR-0131.G.1): state-introducing verb classes — admission N/50 (Δ+N from baseline)`. Body: new admission_rate, refused_reason delta vs baseline, per-verb case counts, link to ADR. + +**Exit criterion.** CI green; new axis runner exits 0 with `wrong == 0`; GSM8K probe `admission_rate` strictly increases; B3 lane unchanged; refreshed coverage report committed. + +**Do not stack on another agent's branch.** Target main directly. diff --git a/docs/briefs/parallel-2026-05-23/README.md b/docs/briefs/parallel-2026-05-23/README.md new file mode 100644 index 00000000..783423b3 --- /dev/null +++ b/docs/briefs/parallel-2026-05-23/README.md @@ -0,0 +1,42 @@ +# Parallel dispatch — 2026-05-23 + +Three lanes, all targeting `main`, no cross-stacking. Reconciliation seat: lead (Shay + main agent). + +| Lane | Brief | Agent | Worktree | Branch | +|---|---|---|---|---| +| L1 | [B2 — teaching-corpus math eval](./L1-gemini-B2-teaching-corpus-eval.md) | Gemini | `../core-adr-0131-2-teaching-eval` | `feat/adr-0131-2-teaching-corpus-eval` | +| L2 | [Binding-graph Phase 1 (data model)](./L2-opus2-binding-graph-phase1.md) | Opus#2 | `../core-binding-graph-p1` | `feat/binding-graph-phase1` | +| L3 | [B1 sealed holdout](./L3-sealed-holdout-B1.md) | Opus#2 or Gemini (small) | `../core-adr-0131-1-sealed-holdout` | `feat/adr-0131-1-sealed-holdout` | + +Sequential follow-ups (do not dispatch until prerequisites land): +- **B3 — bounded-grammar word-problems (ADR-0131.3)** — waits on L2 reaching Phase 2–3. +- **ADR-0131.4 promotion wiring** — waits on B1.B (#169) + B2 + B3 all landed. +- **ADR-0131.5 GSM8K-removal amendment** — docs-only, lands last. + +## ADR-0131.G capability-axis iterations (post-#181 baseline = 0/50) + +PR #181 (`ADR-0131.G`) pinned the GSM8K coverage probe as a diff-able admission number with capability-first iteration discipline. Each iteration extends **one** capability axis, ships its own curated coverage cases (axis lane), and re-runs both the axis lane and the GSM8K probe — admission rises (or a refused-reason family deliberately shrinks) as a *side effect*, not the goal. `admitted_wrong == 0` is the non-negotiable gate. + +These four are **independent axes** — they can dispatch in parallel, each targeting `main`, each in its own worktree. + +| Lane | Brief | Axis | Worktree | Branch | +|---|---|---|---|---| +| L9 | [G.1 verb classes for initial-state](./L9-ADR-0131-G1-verb-classes-initial-state.md) | state-introducing verbs beyond `has/is` | `../core-adr-0131-g1-verb-classes` | `feat/adr-0131-g1-verb-classes` | +| L10 | [G.2 comparatives](./L10-ADR-0131-G2-comparatives.md) | `compare_additive` + `compare_multiplicative` candidate emitters | `../core-adr-0131-g2-comparatives` | `feat/adr-0131-g2-comparatives` | +| L11 | [G.3 numeric literals](./L11-ADR-0131-G3-numerics.md) | money + fractions + compound numbers (consume `en_numerics_v1`) | `../core-adr-0131-g3-numerics` | `feat/adr-0131-g3-numerics` | +| L12 | [G.4 multi-clause composition](./L12-ADR-0131-G4-multi-clause.md) | conjoined subjects + distributive `each` + embedded quantifier phrases | `../core-adr-0131-g4-multi-clause` | `feat/adr-0131-g4-multi-clause` | + +| L16 | [G.5 aggregate answer composition](./L16-ADR-0131-G5-aggregate-answer-composition.md) | `combined`/`together` cue vocab + 2/3-entity sum lane | `../core-adr-0131-g5-aggregate` | `feat/adr-0131-g5-aggregate` | +| L17 | [G.6 capacity-rate verbs](./L17-ADR-0131-G6-rate-capacity.md) | `can N in M