* docs(ADR-0167): audit-as-teaching-evidence (math reader → contemplation wire) Scoping ADR for Brief 11D Candidate E. Routes math-reader refusal audit rows into the existing contemplation/HITL teaching corridor as a new candidate source (`MathReaderRefusalEvidence`). Key decisions: - Evidence-only — never directly admits a math fact; only ratification through HITL queue can change runtime behaviour - Five sub-types proposed (Lexical / Frame / Composition / Reference / Slot claims) mapping to the audit taxonomy - Scope first to LexicalClaim — lowest-risk, highest-count - Six open questions called out for the implementation ADR ADR-0166 three-question test passes; implementation passes only when the six open questions are answered with LexicalClaim-first scope. No code in this PR. * docs(ADR-0167): parallel work plan — 6-PR/3-wave dispatch across 5 model operators
11 KiB
ADR-0167 — Parallel Work Plan
Date: 2026-05-27 Parent ADR: ADR-0167 Goal: Land the LexicalClaim-first slice of the math reader → contemplation wire across four cooperating operators in two waves, with strict worktree isolation and shared invariants.
Shared constraints (every brief)
- Open dedicated
git worktree addper the parallel-agent worktree rule wrong == 0non-negotiable; verify against casegsm8k-train-sample-v1-0050whenever runtime is touched- No new canonical eval lanes (ADR-0166)
- No teaching-store / pack mutation as direct side effect of the wire — pack writes happen only through ratified handlers
uv venv/uv pip install/uv run— neverpip --break-system-packages, never/tmpscratch venvs- Stage explicit files; never
git add -A; NEVER commitengine_state/ - Cognition teaching-corridor tests must remain green at every layer
Wave 1 — Foundation (single blocking brief)
Until Wave 1 lands, Wave 2 cannot start. Wave 1 ships one PR.
W1-A — Schema + canonical-bytes for MathReaderRefusalEvidence
Recommended operator: Opus 4.6/4.7 Why this model: Deepest reasoning. The output is an architectural schema that has to be right the first time (it's the type every Wave 2 brief depends on). One file of types + one file of round-trip tests.
Deliverables:
teaching/math_evidence.py(new) — frozen dataclassMathReaderRefusalEvidencewith:case_id: strsentence_index: inttoken_index: intrefusal_reason: strmissing_operator: str | Noneclaim_signature: str(normalised dedup key — see W2-B)evidence_hash: str(canonical-bytes sha256)audit_row: AuditRow(existing type fromgenerate/comprehension/audit.py)sub_type: Literal["lexical", "frame", "composition", "reference", "slot"]
teaching/math_evidence.pyincludesto_canonical_bytes()mirroringstate.to_canonical_bytes()patterns (sort keys, omit None, decimal canonicalisation if needed)tests/test_math_evidence_schema.py(new):- Round-trip canonical bytes determinism (same input → byte-identical hash)
- Frozen-dataclass immutability
claim_signatureis stable across two refusals with the same surface semantics (placeholder; W2-B finalises the normalisation rules)- Cross-sub-type hash distinctness (lexical claim for
crayons≠ frame claim forcrayons)
Out of scope for W1-A:
- Audit-to-evidence adapter (that's W2-A)
- Dedup policy implementation (W2-B specifies; W1-A only places the field)
- Ratification handlers (W2-D)
Exit: PR merged to main; the type is importable; tests are green;
no runtime change outside teaching/.
Wave 2 — Parallel build (four briefs, dispatched in one message)
All four branch off main (post-W1-A merge). Each in its own worktree.
Each opens its own PR.
W2-A — Audit → candidate adapter
Recommended operator: GPT-5.3-Codex (or Sonnet 4.6 as second choice) Why this model: Mechanical wiring with a defined contract. Codex excels at "take type A, produce type B, write tests." Short cycle time.
Deliverables:
teaching/math_contemplation.py(new) — functionaudit_to_evidence(audit_rows: list[AuditRow]) -> list[MathReaderRefusalEvidence]- Maps
missing_operator→sub_typeper the table in ADR-0167 - Computes
evidence_hashfromMathReaderRefusalEvidence.to_canonical_bytes() - Leaves
claim_signatureas the empty string for non-lexical sub-types (W2-B fills it for lexical) tests/test_math_contemplation_adapter.py— 8+ tests:- Round-trip from
audit_brief_11.jsonproduces N evidence records (one per refused case) - Determinism: same audit input → byte-identical evidence list
- Mapping table is exhaustive (no
missing_operatorfalls through toNonesub_type) - Empty audit → empty evidence list
- Round-trip from
Dependencies: Wave 1 (MathReaderRefusalEvidence type)
Exit: Adapter callable from a test; cognition tests untouched.
W2-B — Dedup policy + claim signature normalisation (LexicalClaim only)
Recommended operator: Sonnet 4.6 Why this model: Pure-Python text-normalisation work with clear invariants. Sonnet is fast and reliable on this shape. Output is tightly scoped and testable.
Deliverables:
teaching/math_claim_signature.py(new) — functionlexical_claim_signature(surface: str, refusal_detail: str) -> str- Normalisation rules (deterministic; documented in module docstring):
- Lowercase the surface
- Strip leading/trailing punctuation
- Encode the unknown-token from
refusal_detailliterally - Hash with sha256, return hex
- Update
teaching/math_contemplation.py(W2-A's file) so that lexical sub_type evidence carries the computed signature; non-lexical pass empty string (deferred to follow-up ADR) tests/test_math_claim_signature.py— 10+ tests:- Identical surface → identical signature
- Different surface → different signature
- Punctuation strip leaves the same signature
- Two GSM8K cases both refusing on
crayonsproduce one signature - Real-data sanity: run over
audit_brief_11.json, assert no false collisions among the actuallexicon_entrycases
Dependencies: Wave 1; coordinates with W2-A on which file owns the signature call. Exit: Lexical evidence rows carry a stable signature; dedup test proves identical claims collapse.
W2-C — Cross-domain partition audit + discriminator
Recommended operator: Gemini (long-context, mechanical audit)
Why this model: This is a scan-many-files survey: find every
contemplation/teaching code path that touches DiscoveryCandidate,
identify where domain discrimination must be added, list every test
that touches the candidate type, propose minimal surgical patches.
Gemini's long-context window suits the scan; the architecture call
remains the operator's.
Deliverables (docs + minimal-impl PR):
docs/handoff/ADR-0167-W2C-cross-domain-audit.md(new) — survey of every code path that constructs or consumesDiscoveryCandidate, with explicit yes/no on whether each path needs to read adomainfield- Minimal
domain: Literal["cognition", "math"]field added toDiscoveryCandidate(default"cognition"to keep existing cognition tests passing without changes) tests/test_candidate_domain_partition.py— assert:- Existing cognition candidates default to
domain="cognition" - A math candidate can be constructed with
domain="math" - Round-trip serialisation preserves the field
- Existing cognition candidates default to
Hard constraint: all existing cognition teaching-corridor tests must remain green with zero modification. Dependencies: Wave 1 (so the audit can reference the math evidence type accurately). Exit: Domain field present; cognition tests green; survey doc identifies any remaining partition risk for Wave 3.
W2-D — LexicalClaim ratification handler
Recommended operator: GPT-5.5 / 5.4 (highest-stakes implementation; needs GitHub connector access for cross-PR coordination) Why this model: Touches the highest-risk surface: pack files. Needs the most careful handling of wrong=0, manifest checksum, and ratification provenance. GPT-5.5's longer-step coding plus GitHub connector keeps it coordinated with #348's lexicon work.
Deliverables:
teaching/math_lexical_ratification.py(new) — functionapply_lexical_claim(claim: MathReaderRefusalEvidence, category: str, reviewer: str) -> RatificationReceipt- Writes to
language_packs/data/en_core_math_v1/lexicon/<category>.jsonlwith the rules established by #348 (alphabetical sort, provenance tag, alias-vs-lemma decision) - Provenance tag:
phase_2_reader_ratified_<reviewer>_<YYYY-MM-DD> - Manifest checksum recompute decision: source-file edits do NOT
regenerate
lexicon.jsonl(matches #348's pattern); document this in the function's docstring RatificationReceiptincludes: target_file, lemma, category, provenance, file_sha256_before, file_sha256_after, evidence_hashtests/test_math_lexical_ratification.py— 10+ tests, including:- Round-trip: write a lemma, verify it loads through
load_lexicon - Idempotency: applying the same claim twice raises a deterministic
AlreadyRatifiederror (no silent dup) - Manifest checksum invariant: source-file write does not change
manifest.json's declared checksum - Hazard pin: ratifying
doesasaccumulation_verb(mis-category) raisesWrongZeroViolationCandidate(because case 0050'sdoesis currentlymodal_auxand reclassifying would risk wrong>0)
- Round-trip: write a lemma, verify it loads through
- Workbench integration is out of scope (ADR-0167 §"Open Questions Q4"); the function returns a receipt, the workbench wiring is a follow-up PR.
Dependencies: Wave 1; coordinates with #348's pack patterns.
Exit: Operator can call apply_lexical_claim() from a Python
session to ratify a single lexical evidence row; all tests green.
Wave 3 — Integration + regression (after Wave 2 fully lands)
Single brief; sequential after all Wave 2 PRs merge.
W3-A — End-to-end determinism + cognition regression
Recommended operator: Opus 4.6/4.7 (or Sonnet 4.6 if Opus is busy) Why this model: Verification work; the test suite is the contract. Deep reasoning helps spot subtle invariant breaks across the wire.
Deliverables:
tests/test_math_evidence_e2e.py— end-to-end test:- Load audit_brief_11.json
- Adapter produces evidence list
- Two reruns produce byte-identical evidence list (replay equivalence)
- Ratify one lexical claim
- Re-run audit; the previously-refused case now passes through that
lemma (advances
unknown_wordrow by one) - Cognition teaching-corridor regression: existing
evals/identity_divergence/lanes still green
- Update
evals/gsm8k_math/train_sample/v1/audit_brief_11.mdwith a "post-W2 baseline" row in the taxonomy table
Hard constraint: if any cognition test breaks, the wire is not ready to merge. Exit: Full LexicalClaim slice operational; ready for first operator-driven math ratification.
Dispatch protocol
When ready to launch Wave 2:
Single message → three Agent tool calls in parallel:
1. subagent_type=general-purpose → W2-A brief (Codex-style ops)
2. subagent_type=general-purpose → W2-B brief (Sonnet-style ops)
3. subagent_type=general-purpose → W2-C brief (Gemini-style ops)
+ separate dispatch to GPT-5.5 via GitHub connector → W2-D brief
W2-D goes to GPT-5.5 separately because the ratification handler touches the highest-risk surface and benefits from human-paced review coordination via the connector.
Wave 3 is single-operator, dispatched after Wave 2 fully merges.
Operator workload (rough estimate)
| Wave | Brief | Operator | Effort |
|---|---|---|---|
| 1 | W1-A | Opus | small |
| 2 | W2-A | Codex | small |
| 2 | W2-B | Sonnet | small |
| 2 | W2-C | Gemini | medium |
| 2 | W2-D | GPT-5.5 | medium |
| 3 | W3-A | Opus | small |
Six PRs total. Two waves of true parallelism. One serial foundation,
one serial integration. Every wave gate is wrong == 0 + cognition
tests green.
What this plan does NOT do
- Does not add new eval lanes (ADR-0166)
- Does not wire workbench v1 (ADR-0167 §Q4 — out of scope)
- Does not ship the four non-lexical sub-types (deferred to ADR-0168+)
- Does not mutate cognition packs (math wire only)
- Does not auto-ratify anything (HITL always)