docs: Brief 11 EOD status footer + 2026-05-27 session narrative (#356)
Two docs-only updates capturing the day's work: 1. Appended a "Status update — 2026-05-27 EOD" footer to the Brief 11 handoff doc with the completion table (11A/11B-step-1/11B-step-2 docs+lexicon/11D merged; 11C absorbed into W3-A; 11D candidate E ADR merged) and the current post-#348 baseline taxonomy. 2. New session doc SESSION-2026-05-27-adr-0167-parallel-dispatch.md alongside the existing SESSION-2026-05-26-comprehension-reader.md. Captures the architectural pivot (audit-as-teaching-evidence vs the rejected refusal-class dispatch table), the parallel-dispatch experiment (5 operators / 3 waves / 6 PRs), what worked, what surfaced as load-bearing (case 0050 hazard), and what's deferred. No code changes. No runtime effect.
This commit is contained in:
Shay
GitHub
parent
7e6c4e81f7
commit
94fca8ea5d
2 changed files with 177 additions and 0 deletions
140
docs/decisions/SESSION-2026-05-27-adr-0167-parallel-dispatch.md
Normal file
140
docs/decisions/SESSION-2026-05-27-adr-0167-parallel-dispatch.md
Normal file
|
|
@ -0,0 +1,140 @@
|
|||
# SESSION-2026-05-27 — ADR-0167 parallel-dispatch (audit-as-teaching-evidence)
|
||||
|
||||
**Date:** 2026-05-27
|
||||
**Author:** Shay
|
||||
**Companion ADR:** [ADR-0167](./ADR-0167-audit-as-teaching-evidence.md)
|
||||
**Parent session:** [SESSION-2026-05-26 (Brief 11 comprehension reader)](./SESSION-2026-05-26-comprehension-reader.md)
|
||||
**Anchor:** [[thesis-decoding-not-generating]]
|
||||
|
||||
---
|
||||
|
||||
## What happened
|
||||
|
||||
Brief 11's closure pass landed across the day (11A→11B→11D merged; 11C
|
||||
absorbed into ADR-0167 W3-A). Mid-day, while reviewing the Brief 11B
|
||||
audit taxonomy, a question surfaced:
|
||||
|
||||
> See, if it has this kind of information, can't it also be used to help
|
||||
> it try and (re)solve problems?
|
||||
|
||||
That observation — that the refusal taxonomy is itself a queue of
|
||||
*teachable moments*, not just a diagnostic dump — became ADR-0167. By
|
||||
end of day, the ADR + the LexicalClaim-first implementation slice were
|
||||
mostly landed (W1-A + W2-A/B/C/D merged; W3-A in flight with Opus 4.7).
|
||||
|
||||
## The architectural pivot
|
||||
|
||||
The temptation was a refusal-class dispatch table:
|
||||
`missing_operator → specialised handler`. Reader hits
|
||||
`multi_quantity_composition`? Route to a frame-splitter. Hits
|
||||
`fraction_percentage_literal`? Route to a fraction subroutine. That
|
||||
direction was rejected explicitly in the ADR (§"Why this is not a
|
||||
refusal-class dispatch table") because it is library-of-handlers — the
|
||||
same anti-pattern regex sentence templates represented, which
|
||||
[[adr-0164-comprehension-reader]] retired.
|
||||
|
||||
The right direction came from the thesis: the engine doesn't store
|
||||
another found thing; it *surfaces what it failed to find* in a shape the
|
||||
operator can teach against. The audit taxonomy is the queue; the
|
||||
existing contemplation/HITL teaching corridor is the resolution path. No
|
||||
new mechanism — wire the math reader's evidence into the same loop
|
||||
cognition already uses.
|
||||
|
||||
## The dispatch experiment
|
||||
|
||||
ADR-0167's implementation was a clean target for the parallel-agent
|
||||
pattern: 6 PRs across 3 waves, 5 model operators matched to brief shape.
|
||||
|
||||
| Wave | PR | Brief | Operator | Why this model |
|
||||
|---|---|---|---|---|
|
||||
| W1 | #350 | MathReaderRefusalEvidence schema | Opus 4.6/4.7 | Foundation; must be right first time |
|
||||
| W2 | #352 | Audit→evidence adapter | GPT-5.3-Codex | Mechanical type-A→type-B + tests |
|
||||
| W2 | #353 | Lexical claim signature + dedup | Sonnet 4.6 | Pure-Python tight-scope normalisation |
|
||||
| W2 | #351 | Domain discriminator + audit | Gemini | Long-context survey of every call site |
|
||||
| W2 | #354 | LexicalClaim ratification handler | GPT-5.5 | Highest-stakes; GitHub-connector review coordination |
|
||||
| W3 | (open) | E2E determinism + cognition regression | Opus 4.7 | Integration verification; deep reasoning |
|
||||
|
||||
Briefs lived in `tmp/wave2.md` and `tmp/wave3.md`. Each operator pointed
|
||||
to their section header. Shared constraints (worktree isolation,
|
||||
wrong=0, ADR-0166, uv, explicit staging) at the top of each wave doc.
|
||||
|
||||
## What worked
|
||||
|
||||
**Brief-shape matching.** Earlier sessions mixed agents poorly — feedback
|
||||
captured in [[feedback-shay-workstyle]] and the new
|
||||
[[feedback-parallel-dispatch-pattern]] memory entry. This time the
|
||||
mapping was deliberate: Opus to load-bearing schema, Codex to mechanical
|
||||
wiring, Gemini to long-context surveys, Sonnet to tight-scope normalisation,
|
||||
GPT-5.5 to high-stakes pack-mutation. Result: 5/5 briefs produced
|
||||
usable work, 4 needed no handler intervention, 1 (Codex) tapped out at
|
||||
rate limits but had finished implementation — handler finished the
|
||||
commit+push+PR mechanically.
|
||||
|
||||
**Single monitor on terminal CI state.** Earlier polling-loop monitors
|
||||
spammed every 60 seconds with `pending`. The fix: `until` loop in the
|
||||
Monitor command, emit only when state is no longer pending. One event
|
||||
per PR completion, no noise.
|
||||
|
||||
**Wave-N dispatches AFTER wave-(N-1) merges to main.** Briefly tested
|
||||
the alternative (Gemini accidentally merged W1-A's branch into its W2-C
|
||||
branch because W1-A wasn't yet on main); harmless this time but the
|
||||
discipline is to wait. Memory entry pins this.
|
||||
|
||||
## What surfaced as load-bearing
|
||||
|
||||
**Case `gsm8k-train-sample-v1-0050` is the canary.** During Brief 11B I
|
||||
tried the naive `pre_frame_filler_sentence` fix (drain
|
||||
`statement_terminator` at pre-frame). It lifted 2 cases to admitted, but
|
||||
case 0050 silently produced a partial graph that would project to the
|
||||
wrong answer. The fix was rejected per Brief 11 §"correct-count greed,"
|
||||
and the hazard is now pinned across the test suite — see
|
||||
[[feedback-wrong-zero-hazard-case-0050]]. W2-D's `SAFE_CATEGORIES =
|
||||
{"drain_token"}` allowlist extends the same defence: frame-opener
|
||||
categories cannot be ratified through LexicalClaim because reclassifying
|
||||
a verb like `does` to `accumulation_verb` would re-introduce the hazard.
|
||||
|
||||
**The thesis answer to "can it use the taxonomy to resolve?"** The
|
||||
right wire is *into the teaching corridor*, not *into the runtime*. The
|
||||
engine learns from its own refusal data through reviewed correction —
|
||||
the loop the thesis demands. ADR-0167 is what makes that loop concrete
|
||||
for the math domain.
|
||||
|
||||
## What's deferred
|
||||
|
||||
- **Four non-lexical sub-types** (FrameClaim / CompositionClaim /
|
||||
ReferenceClaim / SlotClaim) — separate ADRs, follow-up scope.
|
||||
- **Workbench v1 rendering of math candidates** — ADR-0167 §Q4.
|
||||
- **Two partition risks Gemini flagged** in the W2-C audit:
|
||||
contemplation pack indexing (`teaching/contemplation.py` uses
|
||||
hardcoded cognition pack/corpus indexes), and replay gate default in
|
||||
`teaching/proposals.py`. Both need follow-up but don't block the
|
||||
LexicalClaim slice.
|
||||
|
||||
## What closes the day
|
||||
|
||||
When W3-A merges:
|
||||
|
||||
- The LexicalClaim slice is operational end-to-end (refusal → evidence
|
||||
→ signature → ratification → row movement, with cognition regression
|
||||
holding green and case 0050 hazard pinned)
|
||||
- 11C (the capability snapshot) closes as a side effect of W3-A's
|
||||
Deliverable 2
|
||||
- The thesis claim — *the engine teaches itself in the math domain
|
||||
through reviewed correction* — becomes a green test (`tests/
|
||||
test_math_evidence_e2e.py::test_lexical_ratification_advances_unknown_word_row`)
|
||||
|
||||
Brief 11 closes alongside W3-A. The next session can pick up either a
|
||||
non-lexical sub-type ADR, the workbench wiring, the cognition pack
|
||||
indexing partition, or continued GSM8K operator closure — Brief 11D
|
||||
already named those candidates.
|
||||
|
||||
---
|
||||
|
||||
## Cross-references
|
||||
|
||||
- [ADR-0167](./ADR-0167-audit-as-teaching-evidence.md) — the scoping ADR
|
||||
- `docs/handoff/ADR-0167-PARALLEL-WORK-PLAN.md` — the 3-wave / 6-PR dispatch plan
|
||||
- `docs/handoff/BRIEF-11-phase-2-reader-closure-and-capability-snapshot.md` — Brief 11 with EOD status footer
|
||||
- `docs/handoff/ADR-0167-W2C-cross-domain-audit.md` — Gemini's audit (5 construction / 8 consumption sites)
|
||||
- `evals/gsm8k_math/train_sample/v1/audit_brief_11.md` — the bottleneck table the wire is built around
|
||||
- `tmp/wave2.md` / `tmp/wave3.md` — the dispatch briefs (kept in-repo as the parallel-agent playbook for future waves)
|
||||
|
|
@ -261,3 +261,40 @@ Suggested table:
|
|||
Brief 11 is done when the repo has a reproducible, reviewed capability snapshot after Phase 2 reader closure and a precise backlog of remaining missing operators.
|
||||
|
||||
It is not done merely because a document exists. This document is the operator handoff; the engineering sequence above is the work.
|
||||
|
||||
---
|
||||
|
||||
## Status update — 2026-05-27 EOD
|
||||
|
||||
The Brief 11 sequence landed across the day. Completion path:
|
||||
|
||||
| Sub-brief | Status | PR(s) | Notes |
|
||||
|---|---|---|---|
|
||||
| 11A — reader closure audit infrastructure | ✅ Merged | #343 | `generate/comprehension/audit.py` + 18 audit tests |
|
||||
| 11B-step-1 — per-case audit artifact + extended taxonomy | ✅ Merged | #345 | 3 new missing-operator labels close the None-operator gap |
|
||||
| 11B-step-2 — verb-vocabulary analysis (docs-only) | ✅ Merged | #347 | All "unknown verbs" found to be lexicon-present; pre_frame_filler is structural |
|
||||
| 11B-step-2 — lexicon-entry closure | ✅ Merged | #348 | 12 drain_token lemmas + 1 alias; `unknown_word` row 11→5; wrong=0 preserved |
|
||||
| 11C — capability snapshot rerun | ✅ Absorbed | (in W3-A) | Folded into ADR-0167 W3-A's Deliverable 2 (post-W2 baseline section in `audit_brief_11.md`) |
|
||||
| 11D — next-capability proposal | ✅ Merged | #346 | Candidate A (continued GSM8K closure) recommended |
|
||||
| 11D — Candidate E (audit-as-evidence) | ✅ Merged | #349 | ADR-0167 scoping ADR + parallel work plan |
|
||||
|
||||
The bottleneck table after 11B-step-2 lexicon closure (current post-Brief-11
|
||||
baseline; W3-A will produce the post-ADR-0167-LexicalClaim baseline):
|
||||
|
||||
| refusal_reason | count | Δ vs 11B-step-1 |
|
||||
|---|---:|---:|
|
||||
| incomplete_operation | 20 | +2 |
|
||||
| unexpected_category | 17 | +3 |
|
||||
| unknown_word | 5 | **−6** |
|
||||
| unattached_quantity | 4 | +1 |
|
||||
| unresolved_pronoun | 3 | 0 |
|
||||
| no_question_target | 1 | 0 |
|
||||
|
||||
`wrong == 0` held throughout. Case `gsm8k-train-sample-v1-0050` remains
|
||||
refused at sentence_index=0 (pinned by multiple test suites).
|
||||
|
||||
**Next:** ADR-0167 W3-A closes the LexicalClaim slice with an e2e test
|
||||
that walks the full loop (refusal → evidence → ratification → row-moves),
|
||||
plus a cognition-corridor regression pass. After W3-A, Brief 11 closes
|
||||
and the engine can ratify math-domain lexical claims from its own
|
||||
refusal evidence through the existing HITL teaching corridor.
|
||||
|
|
|
|||
Loading…
Reference in a new issue