From 94fca8ea5de07a17d1345f9de0be1bf6aec29046 Mon Sep 17 00:00:00 2001 From: Shay Date: Wed, 27 May 2026 07:21:01 -0700 Subject: [PATCH] docs: Brief 11 EOD status footer + 2026-05-27 session narrative (#356) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two docs-only updates capturing the day's work: 1. Appended a "Status update — 2026-05-27 EOD" footer to the Brief 11 handoff doc with the completion table (11A/11B-step-1/11B-step-2 docs+lexicon/11D merged; 11C absorbed into W3-A; 11D candidate E ADR merged) and the current post-#348 baseline taxonomy. 2. New session doc SESSION-2026-05-27-adr-0167-parallel-dispatch.md alongside the existing SESSION-2026-05-26-comprehension-reader.md. Captures the architectural pivot (audit-as-teaching-evidence vs the rejected refusal-class dispatch table), the parallel-dispatch experiment (5 operators / 3 waves / 6 PRs), what worked, what surfaced as load-bearing (case 0050 hazard), and what's deferred. No code changes. No runtime effect. --- ...N-2026-05-27-adr-0167-parallel-dispatch.md | 140 ++++++++++++++++++ ...-reader-closure-and-capability-snapshot.md | 37 +++++ 2 files changed, 177 insertions(+) create mode 100644 docs/decisions/SESSION-2026-05-27-adr-0167-parallel-dispatch.md diff --git a/docs/decisions/SESSION-2026-05-27-adr-0167-parallel-dispatch.md b/docs/decisions/SESSION-2026-05-27-adr-0167-parallel-dispatch.md new file mode 100644 index 00000000..d3e0e056 --- /dev/null +++ b/docs/decisions/SESSION-2026-05-27-adr-0167-parallel-dispatch.md @@ -0,0 +1,140 @@ +# SESSION-2026-05-27 — ADR-0167 parallel-dispatch (audit-as-teaching-evidence) + +**Date:** 2026-05-27 +**Author:** Shay +**Companion ADR:** [ADR-0167](./ADR-0167-audit-as-teaching-evidence.md) +**Parent session:** [SESSION-2026-05-26 (Brief 11 comprehension reader)](./SESSION-2026-05-26-comprehension-reader.md) +**Anchor:** [[thesis-decoding-not-generating]] + +--- + +## What happened + +Brief 11's closure pass landed across the day (11A→11B→11D merged; 11C +absorbed into ADR-0167 W3-A). Mid-day, while reviewing the Brief 11B +audit taxonomy, a question surfaced: + +> See, if it has this kind of information, can't it also be used to help +> it try and (re)solve problems? + +That observation — that the refusal taxonomy is itself a queue of +*teachable moments*, not just a diagnostic dump — became ADR-0167. By +end of day, the ADR + the LexicalClaim-first implementation slice were +mostly landed (W1-A + W2-A/B/C/D merged; W3-A in flight with Opus 4.7). + +## The architectural pivot + +The temptation was a refusal-class dispatch table: +`missing_operator → specialised handler`. Reader hits +`multi_quantity_composition`? Route to a frame-splitter. Hits +`fraction_percentage_literal`? Route to a fraction subroutine. That +direction was rejected explicitly in the ADR (§"Why this is not a +refusal-class dispatch table") because it is library-of-handlers — the +same anti-pattern regex sentence templates represented, which +[[adr-0164-comprehension-reader]] retired. + +The right direction came from the thesis: the engine doesn't store +another found thing; it *surfaces what it failed to find* in a shape the +operator can teach against. The audit taxonomy is the queue; the +existing contemplation/HITL teaching corridor is the resolution path. No +new mechanism — wire the math reader's evidence into the same loop +cognition already uses. + +## The dispatch experiment + +ADR-0167's implementation was a clean target for the parallel-agent +pattern: 6 PRs across 3 waves, 5 model operators matched to brief shape. + +| Wave | PR | Brief | Operator | Why this model | +|---|---|---|---|---| +| W1 | #350 | MathReaderRefusalEvidence schema | Opus 4.6/4.7 | Foundation; must be right first time | +| W2 | #352 | Audit→evidence adapter | GPT-5.3-Codex | Mechanical type-A→type-B + tests | +| W2 | #353 | Lexical claim signature + dedup | Sonnet 4.6 | Pure-Python tight-scope normalisation | +| W2 | #351 | Domain discriminator + audit | Gemini | Long-context survey of every call site | +| W2 | #354 | LexicalClaim ratification handler | GPT-5.5 | Highest-stakes; GitHub-connector review coordination | +| W3 | (open) | E2E determinism + cognition regression | Opus 4.7 | Integration verification; deep reasoning | + +Briefs lived in `tmp/wave2.md` and `tmp/wave3.md`. Each operator pointed +to their section header. Shared constraints (worktree isolation, +wrong=0, ADR-0166, uv, explicit staging) at the top of each wave doc. + +## What worked + +**Brief-shape matching.** Earlier sessions mixed agents poorly — feedback +captured in [[feedback-shay-workstyle]] and the new +[[feedback-parallel-dispatch-pattern]] memory entry. This time the +mapping was deliberate: Opus to load-bearing schema, Codex to mechanical +wiring, Gemini to long-context surveys, Sonnet to tight-scope normalisation, +GPT-5.5 to high-stakes pack-mutation. Result: 5/5 briefs produced +usable work, 4 needed no handler intervention, 1 (Codex) tapped out at +rate limits but had finished implementation — handler finished the +commit+push+PR mechanically. + +**Single monitor on terminal CI state.** Earlier polling-loop monitors +spammed every 60 seconds with `pending`. The fix: `until` loop in the +Monitor command, emit only when state is no longer pending. One event +per PR completion, no noise. + +**Wave-N dispatches AFTER wave-(N-1) merges to main.** Briefly tested +the alternative (Gemini accidentally merged W1-A's branch into its W2-C +branch because W1-A wasn't yet on main); harmless this time but the +discipline is to wait. Memory entry pins this. + +## What surfaced as load-bearing + +**Case `gsm8k-train-sample-v1-0050` is the canary.** During Brief 11B I +tried the naive `pre_frame_filler_sentence` fix (drain +`statement_terminator` at pre-frame). It lifted 2 cases to admitted, but +case 0050 silently produced a partial graph that would project to the +wrong answer. The fix was rejected per Brief 11 §"correct-count greed," +and the hazard is now pinned across the test suite — see +[[feedback-wrong-zero-hazard-case-0050]]. W2-D's `SAFE_CATEGORIES = +{"drain_token"}` allowlist extends the same defence: frame-opener +categories cannot be ratified through LexicalClaim because reclassifying +a verb like `does` to `accumulation_verb` would re-introduce the hazard. + +**The thesis answer to "can it use the taxonomy to resolve?"** The +right wire is *into the teaching corridor*, not *into the runtime*. The +engine learns from its own refusal data through reviewed correction — +the loop the thesis demands. ADR-0167 is what makes that loop concrete +for the math domain. + +## What's deferred + +- **Four non-lexical sub-types** (FrameClaim / CompositionClaim / + ReferenceClaim / SlotClaim) — separate ADRs, follow-up scope. +- **Workbench v1 rendering of math candidates** — ADR-0167 §Q4. +- **Two partition risks Gemini flagged** in the W2-C audit: + contemplation pack indexing (`teaching/contemplation.py` uses + hardcoded cognition pack/corpus indexes), and replay gate default in + `teaching/proposals.py`. Both need follow-up but don't block the + LexicalClaim slice. + +## What closes the day + +When W3-A merges: + +- The LexicalClaim slice is operational end-to-end (refusal → evidence + → signature → ratification → row movement, with cognition regression + holding green and case 0050 hazard pinned) +- 11C (the capability snapshot) closes as a side effect of W3-A's + Deliverable 2 +- The thesis claim — *the engine teaches itself in the math domain + through reviewed correction* — becomes a green test (`tests/ + test_math_evidence_e2e.py::test_lexical_ratification_advances_unknown_word_row`) + +Brief 11 closes alongside W3-A. The next session can pick up either a +non-lexical sub-type ADR, the workbench wiring, the cognition pack +indexing partition, or continued GSM8K operator closure — Brief 11D +already named those candidates. + +--- + +## Cross-references + +- [ADR-0167](./ADR-0167-audit-as-teaching-evidence.md) — the scoping ADR +- `docs/handoff/ADR-0167-PARALLEL-WORK-PLAN.md` — the 3-wave / 6-PR dispatch plan +- `docs/handoff/BRIEF-11-phase-2-reader-closure-and-capability-snapshot.md` — Brief 11 with EOD status footer +- `docs/handoff/ADR-0167-W2C-cross-domain-audit.md` — Gemini's audit (5 construction / 8 consumption sites) +- `evals/gsm8k_math/train_sample/v1/audit_brief_11.md` — the bottleneck table the wire is built around +- `tmp/wave2.md` / `tmp/wave3.md` — the dispatch briefs (kept in-repo as the parallel-agent playbook for future waves) diff --git a/docs/handoff/BRIEF-11-phase-2-reader-closure-and-capability-snapshot.md b/docs/handoff/BRIEF-11-phase-2-reader-closure-and-capability-snapshot.md index 1bdab672..1e73e43e 100644 --- a/docs/handoff/BRIEF-11-phase-2-reader-closure-and-capability-snapshot.md +++ b/docs/handoff/BRIEF-11-phase-2-reader-closure-and-capability-snapshot.md @@ -261,3 +261,40 @@ Suggested table: Brief 11 is done when the repo has a reproducible, reviewed capability snapshot after Phase 2 reader closure and a precise backlog of remaining missing operators. It is not done merely because a document exists. This document is the operator handoff; the engineering sequence above is the work. + +--- + +## Status update — 2026-05-27 EOD + +The Brief 11 sequence landed across the day. Completion path: + +| Sub-brief | Status | PR(s) | Notes | +|---|---|---|---| +| 11A — reader closure audit infrastructure | ✅ Merged | #343 | `generate/comprehension/audit.py` + 18 audit tests | +| 11B-step-1 — per-case audit artifact + extended taxonomy | ✅ Merged | #345 | 3 new missing-operator labels close the None-operator gap | +| 11B-step-2 — verb-vocabulary analysis (docs-only) | ✅ Merged | #347 | All "unknown verbs" found to be lexicon-present; pre_frame_filler is structural | +| 11B-step-2 — lexicon-entry closure | ✅ Merged | #348 | 12 drain_token lemmas + 1 alias; `unknown_word` row 11→5; wrong=0 preserved | +| 11C — capability snapshot rerun | ✅ Absorbed | (in W3-A) | Folded into ADR-0167 W3-A's Deliverable 2 (post-W2 baseline section in `audit_brief_11.md`) | +| 11D — next-capability proposal | ✅ Merged | #346 | Candidate A (continued GSM8K closure) recommended | +| 11D — Candidate E (audit-as-evidence) | ✅ Merged | #349 | ADR-0167 scoping ADR + parallel work plan | + +The bottleneck table after 11B-step-2 lexicon closure (current post-Brief-11 +baseline; W3-A will produce the post-ADR-0167-LexicalClaim baseline): + +| refusal_reason | count | Δ vs 11B-step-1 | +|---|---:|---:| +| incomplete_operation | 20 | +2 | +| unexpected_category | 17 | +3 | +| unknown_word | 5 | **−6** | +| unattached_quantity | 4 | +1 | +| unresolved_pronoun | 3 | 0 | +| no_question_target | 1 | 0 | + +`wrong == 0` held throughout. Case `gsm8k-train-sample-v1-0050` remains +refused at sentence_index=0 (pinned by multiple test suites). + +**Next:** ADR-0167 W3-A closes the LexicalClaim slice with an e2e test +that walks the full loop (refusal → evidence → ratification → row-moves), +plus a cognition-corridor regression pass. After W3-A, Brief 11 closes +and the engine can ratify math-domain lexical claims from its own +refusal evidence through the existing HITL teaching corridor.