core/docs/handoff/ADR-0167-PARALLEL-WORK-PLAN.md
Shay 4f0815ef9a
docs(ADR-0167): audit-as-teaching-evidence (math reader → contemplation wire) (#349)
* docs(ADR-0167): audit-as-teaching-evidence (math reader → contemplation wire)

Scoping ADR for Brief 11D Candidate E. Routes math-reader refusal audit
rows into the existing contemplation/HITL teaching corridor as a new
candidate source (`MathReaderRefusalEvidence`).

Key decisions:
- Evidence-only — never directly admits a math fact; only ratification
  through HITL queue can change runtime behaviour
- Five sub-types proposed (Lexical / Frame / Composition / Reference /
  Slot claims) mapping to the audit taxonomy
- Scope first to LexicalClaim — lowest-risk, highest-count
- Six open questions called out for the implementation ADR

ADR-0166 three-question test passes; implementation passes only when
the six open questions are answered with LexicalClaim-first scope.

No code in this PR.

* docs(ADR-0167): parallel work plan — 6-PR/3-wave dispatch across 5 model operators
2026-05-27 06:21:43 -07:00

11 KiB

ADR-0167 — Parallel Work Plan

Date: 2026-05-27 Parent ADR: ADR-0167 Goal: Land the LexicalClaim-first slice of the math reader → contemplation wire across four cooperating operators in two waves, with strict worktree isolation and shared invariants.


Shared constraints (every brief)

  • Open dedicated git worktree add per the parallel-agent worktree rule
  • wrong == 0 non-negotiable; verify against case gsm8k-train-sample-v1-0050 whenever runtime is touched
  • No new canonical eval lanes (ADR-0166)
  • No teaching-store / pack mutation as direct side effect of the wire — pack writes happen only through ratified handlers
  • uv venv / uv pip install / uv run — never pip --break-system-packages, never /tmp scratch venvs
  • Stage explicit files; never git add -A; NEVER commit engine_state/
  • Cognition teaching-corridor tests must remain green at every layer

Wave 1 — Foundation (single blocking brief)

Until Wave 1 lands, Wave 2 cannot start. Wave 1 ships one PR.

W1-A — Schema + canonical-bytes for MathReaderRefusalEvidence

Recommended operator: Opus 4.6/4.7 Why this model: Deepest reasoning. The output is an architectural schema that has to be right the first time (it's the type every Wave 2 brief depends on). One file of types + one file of round-trip tests.

Deliverables:

  • teaching/math_evidence.py (new) — frozen dataclass MathReaderRefusalEvidence with:
    • case_id: str
    • sentence_index: int
    • token_index: int
    • refusal_reason: str
    • missing_operator: str | None
    • claim_signature: str (normalised dedup key — see W2-B)
    • evidence_hash: str (canonical-bytes sha256)
    • audit_row: AuditRow (existing type from generate/comprehension/audit.py)
    • sub_type: Literal["lexical", "frame", "composition", "reference", "slot"]
  • teaching/math_evidence.py includes to_canonical_bytes() mirroring state.to_canonical_bytes() patterns (sort keys, omit None, decimal canonicalisation if needed)
  • tests/test_math_evidence_schema.py (new):
    • Round-trip canonical bytes determinism (same input → byte-identical hash)
    • Frozen-dataclass immutability
    • claim_signature is stable across two refusals with the same surface semantics (placeholder; W2-B finalises the normalisation rules)
    • Cross-sub-type hash distinctness (lexical claim for crayons ≠ frame claim for crayons)

Out of scope for W1-A:

  • Audit-to-evidence adapter (that's W2-A)
  • Dedup policy implementation (W2-B specifies; W1-A only places the field)
  • Ratification handlers (W2-D)

Exit: PR merged to main; the type is importable; tests are green; no runtime change outside teaching/.


Wave 2 — Parallel build (four briefs, dispatched in one message)

All four branch off main (post-W1-A merge). Each in its own worktree. Each opens its own PR.

W2-A — Audit → candidate adapter

Recommended operator: GPT-5.3-Codex (or Sonnet 4.6 as second choice) Why this model: Mechanical wiring with a defined contract. Codex excels at "take type A, produce type B, write tests." Short cycle time.

Deliverables:

  • teaching/math_contemplation.py (new) — function audit_to_evidence(audit_rows: list[AuditRow]) -> list[MathReaderRefusalEvidence]
  • Maps missing_operatorsub_type per the table in ADR-0167
  • Computes evidence_hash from MathReaderRefusalEvidence.to_canonical_bytes()
  • Leaves claim_signature as the empty string for non-lexical sub-types (W2-B fills it for lexical)
  • tests/test_math_contemplation_adapter.py — 8+ tests:
    • Round-trip from audit_brief_11.json produces N evidence records (one per refused case)
    • Determinism: same audit input → byte-identical evidence list
    • Mapping table is exhaustive (no missing_operator falls through to None sub_type)
    • Empty audit → empty evidence list

Dependencies: Wave 1 (MathReaderRefusalEvidence type) Exit: Adapter callable from a test; cognition tests untouched.

W2-B — Dedup policy + claim signature normalisation (LexicalClaim only)

Recommended operator: Sonnet 4.6 Why this model: Pure-Python text-normalisation work with clear invariants. Sonnet is fast and reliable on this shape. Output is tightly scoped and testable.

Deliverables:

  • teaching/math_claim_signature.py (new) — function lexical_claim_signature(surface: str, refusal_detail: str) -> str
  • Normalisation rules (deterministic; documented in module docstring):
    • Lowercase the surface
    • Strip leading/trailing punctuation
    • Encode the unknown-token from refusal_detail literally
    • Hash with sha256, return hex
  • Update teaching/math_contemplation.py (W2-A's file) so that lexical sub_type evidence carries the computed signature; non-lexical pass empty string (deferred to follow-up ADR)
  • tests/test_math_claim_signature.py — 10+ tests:
    • Identical surface → identical signature
    • Different surface → different signature
    • Punctuation strip leaves the same signature
    • Two GSM8K cases both refusing on crayons produce one signature
    • Real-data sanity: run over audit_brief_11.json, assert no false collisions among the actual lexicon_entry cases

Dependencies: Wave 1; coordinates with W2-A on which file owns the signature call. Exit: Lexical evidence rows carry a stable signature; dedup test proves identical claims collapse.

W2-C — Cross-domain partition audit + discriminator

Recommended operator: Gemini (long-context, mechanical audit) Why this model: This is a scan-many-files survey: find every contemplation/teaching code path that touches DiscoveryCandidate, identify where domain discrimination must be added, list every test that touches the candidate type, propose minimal surgical patches. Gemini's long-context window suits the scan; the architecture call remains the operator's.

Deliverables (docs + minimal-impl PR):

  • docs/handoff/ADR-0167-W2C-cross-domain-audit.md (new) — survey of every code path that constructs or consumes DiscoveryCandidate, with explicit yes/no on whether each path needs to read a domain field
  • Minimal domain: Literal["cognition", "math"] field added to DiscoveryCandidate (default "cognition" to keep existing cognition tests passing without changes)
  • tests/test_candidate_domain_partition.py — assert:
    • Existing cognition candidates default to domain="cognition"
    • A math candidate can be constructed with domain="math"
    • Round-trip serialisation preserves the field

Hard constraint: all existing cognition teaching-corridor tests must remain green with zero modification. Dependencies: Wave 1 (so the audit can reference the math evidence type accurately). Exit: Domain field present; cognition tests green; survey doc identifies any remaining partition risk for Wave 3.

W2-D — LexicalClaim ratification handler

Recommended operator: GPT-5.5 / 5.4 (highest-stakes implementation; needs GitHub connector access for cross-PR coordination) Why this model: Touches the highest-risk surface: pack files. Needs the most careful handling of wrong=0, manifest checksum, and ratification provenance. GPT-5.5's longer-step coding plus GitHub connector keeps it coordinated with #348's lexicon work.

Deliverables:

  • teaching/math_lexical_ratification.py (new) — function apply_lexical_claim(claim: MathReaderRefusalEvidence, category: str, reviewer: str) -> RatificationReceipt
  • Writes to language_packs/data/en_core_math_v1/lexicon/<category>.jsonl with the rules established by #348 (alphabetical sort, provenance tag, alias-vs-lemma decision)
  • Provenance tag: phase_2_reader_ratified_<reviewer>_<YYYY-MM-DD>
  • Manifest checksum recompute decision: source-file edits do NOT regenerate lexicon.jsonl (matches #348's pattern); document this in the function's docstring
  • RatificationReceipt includes: target_file, lemma, category, provenance, file_sha256_before, file_sha256_after, evidence_hash
  • tests/test_math_lexical_ratification.py — 10+ tests, including:
    • Round-trip: write a lemma, verify it loads through load_lexicon
    • Idempotency: applying the same claim twice raises a deterministic AlreadyRatified error (no silent dup)
    • Manifest checksum invariant: source-file write does not change manifest.json's declared checksum
    • Hazard pin: ratifying does as accumulation_verb (mis-category) raises WrongZeroViolationCandidate (because case 0050's does is currently modal_aux and reclassifying would risk wrong>0)
  • Workbench integration is out of scope (ADR-0167 §"Open Questions Q4"); the function returns a receipt, the workbench wiring is a follow-up PR.

Dependencies: Wave 1; coordinates with #348's pack patterns. Exit: Operator can call apply_lexical_claim() from a Python session to ratify a single lexical evidence row; all tests green.


Wave 3 — Integration + regression (after Wave 2 fully lands)

Single brief; sequential after all Wave 2 PRs merge.

W3-A — End-to-end determinism + cognition regression

Recommended operator: Opus 4.6/4.7 (or Sonnet 4.6 if Opus is busy) Why this model: Verification work; the test suite is the contract. Deep reasoning helps spot subtle invariant breaks across the wire.

Deliverables:

  • tests/test_math_evidence_e2e.py — end-to-end test:
    • Load audit_brief_11.json
    • Adapter produces evidence list
    • Two reruns produce byte-identical evidence list (replay equivalence)
    • Ratify one lexical claim
    • Re-run audit; the previously-refused case now passes through that lemma (advances unknown_word row by one)
    • Cognition teaching-corridor regression: existing evals/identity_divergence/ lanes still green
  • Update evals/gsm8k_math/train_sample/v1/audit_brief_11.md with a "post-W2 baseline" row in the taxonomy table

Hard constraint: if any cognition test breaks, the wire is not ready to merge. Exit: Full LexicalClaim slice operational; ready for first operator-driven math ratification.


Dispatch protocol

When ready to launch Wave 2:

Single message → three Agent tool calls in parallel:
  1. subagent_type=general-purpose → W2-A brief (Codex-style ops)
  2. subagent_type=general-purpose → W2-B brief (Sonnet-style ops)
  3. subagent_type=general-purpose → W2-C brief (Gemini-style ops)
+ separate dispatch to GPT-5.5 via GitHub connector → W2-D brief

W2-D goes to GPT-5.5 separately because the ratification handler touches the highest-risk surface and benefits from human-paced review coordination via the connector.

Wave 3 is single-operator, dispatched after Wave 2 fully merges.


Operator workload (rough estimate)

Wave Brief Operator Effort
1 W1-A Opus small
2 W2-A Codex small
2 W2-B Sonnet small
2 W2-C Gemini medium
2 W2-D GPT-5.5 medium
3 W3-A Opus small

Six PRs total. Two waves of true parallelism. One serial foundation, one serial integration. Every wave gate is wrong == 0 + cognition tests green.


What this plan does NOT do

  • Does not add new eval lanes (ADR-0166)
  • Does not wire workbench v1 (ADR-0167 §Q4 — out of scope)
  • Does not ship the four non-lexical sub-types (deferred to ADR-0168+)
  • Does not mutate cognition packs (math wire only)
  • Does not auto-ratify anything (HITL always)