docs(handoff): Brief 11 — Phase 2 reader closure + capability snapshot sequencing (#342)
* docs(handoff): Brief 11 reader closure and snapshot plan * docs(handoff): Brief 11 — Phase 2 reader closure + capability snapshot sequencing
This commit is contained in:
Shay
GitHub
parent
834d1ae577
commit
33c2b10fbf
2 changed files with 555 additions and 0 deletions
|
|
@ -0,0 +1,263 @@
|
|||
# Brief 11 — Phase 2 Reader Closure and Capability Snapshot
|
||||
|
||||
**Status:** Active handoff
|
||||
**Date:** 2026-05-27
|
||||
**Branch:** `docs/brief-11-handoff`
|
||||
**Scope:** Private repo planning and execution handoff only. No public-safe disclosure. No runtime claim is made by this document.
|
||||
|
||||
---
|
||||
|
||||
## Thesis
|
||||
|
||||
Brief 11 should close the current comprehension-reader corridor before widening measurement or domain scope.
|
||||
|
||||
The latest project state establishes three facts that must govern sequencing:
|
||||
|
||||
1. `gsm8k_math` is not blocked by a lack of eval lanes. It is blocked by incomplete statement understanding.
|
||||
2. The ADR-0164 reader is the correct replacement path for the brittle ADR-0163 recognizer/injector surface.
|
||||
3. ADR-0166 now forbids authoring new canonical eval lanes ahead of the capability they measure.
|
||||
|
||||
Therefore Brief 11 is not "make more benchmarks." Brief 11 is the bounded closure pass that turns the Phase 2 reader line into a reliable capability snapshot, then tells the next operator exactly what can safely happen after it.
|
||||
|
||||
---
|
||||
|
||||
## Current operating facts
|
||||
|
||||
Recent PR descriptions and session logs show the corridor clearly:
|
||||
|
||||
- Phase 1 reader coexistence is additive and flag-gated. With the flag off, behavior should remain byte-identical.
|
||||
- Phase 1 handles question sentences; statement sentences remained on the regex parser.
|
||||
- Phase 2 statement-frame work is the dominant path because most refused cases are caused by recognized-but-skipped statements / incomplete solver graphs.
|
||||
- Wrong-answer protection is currently more important than raising correct count by unsafe partial admission. The wrong=0 guard must remain non-negotiable.
|
||||
- Main has ADR-0166, which states that measurement follows capability, not the other way around.
|
||||
|
||||
Brief 11 must preserve those facts instead of drifting back into lane proliferation or superficial UI/demo work.
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Produce a mechanically useful handoff for the next engineering pass after Brief 10 / Phase 2 reader work:
|
||||
|
||||
1. Define the closure gates for the Phase 2 statement reader.
|
||||
2. Define the capability snapshot that should be run immediately after closure.
|
||||
3. Define the no-go boundaries that prevent unsafe widening.
|
||||
4. Define the next branch/PR sequence so a lead engineer can continue without re-litigating the strategy.
|
||||
|
||||
---
|
||||
|
||||
## Non-goals
|
||||
|
||||
Brief 11 does not:
|
||||
|
||||
- add new OOD eval lanes;
|
||||
- claim expert-level GSM8K capability;
|
||||
- relax wrong=0;
|
||||
- bypass the reader with broader regex patches;
|
||||
- add public-facing disclosure material;
|
||||
- promote any audit/expert/demo tier;
|
||||
- mutate seed packs or teaching stores as a side effect of chat/runtime smoke tests.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance gates
|
||||
|
||||
### Gate 1 — Reader completeness delta
|
||||
|
||||
Run the canonical GSM8K train-sample measurement with the reader path enabled and disabled.
|
||||
|
||||
Required outputs:
|
||||
|
||||
- baseline regex-only result;
|
||||
- reader-enabled result;
|
||||
- refused/correct/wrong counts;
|
||||
- per-case refusal reason taxonomy;
|
||||
- explicit list of cases fixed by Phase 2 statement frames;
|
||||
- explicit list of cases still refused and why.
|
||||
|
||||
Hard invariant:
|
||||
|
||||
- `wrong == 0` must hold.
|
||||
|
||||
If `correct` rises while `wrong > 0`, the PR is not a success. It is a regression with useful diagnostics.
|
||||
|
||||
### Gate 2 — Recognized-skipped statement audit
|
||||
|
||||
For every still-refused case where the reader recognized tokens but did not inject a complete graph, emit a compact audit row:
|
||||
|
||||
```text
|
||||
case_id | sentence_index | recognized_terms | skipped_frame | missing_operator | refusal_reason
|
||||
```
|
||||
|
||||
This is the main Brief 11 artifact because it turns the remaining GSM8K bottleneck into queued capability work instead of guesswork.
|
||||
|
||||
### Gate 3 — Graph completeness proof
|
||||
|
||||
For every admitted case, prove the solver graph is complete before candidate projection:
|
||||
|
||||
- all quantities have owners or intentionally anonymous scope;
|
||||
- all units are canonicalized;
|
||||
- operations are typed;
|
||||
- question target slot resolves to an entity/unit/kind;
|
||||
- no recognized-but-uncommitted frame is present.
|
||||
|
||||
This gate is the successor to the wrong=0 guard. The guard catches unsafe admission; graph completeness prevents unsafe admission from being attempted.
|
||||
|
||||
### Gate 4 — Determinism and trace stability
|
||||
|
||||
Run the reader-enabled measurement twice with identical input and config.
|
||||
|
||||
Required:
|
||||
|
||||
- identical case outcomes;
|
||||
- identical trace hashes where applicable;
|
||||
- stable refusal taxonomy ordering;
|
||||
- no nondeterministic iteration over entity registries, frame sets, or lexicon categories.
|
||||
|
||||
### Gate 5 — Feature-flag hygiene
|
||||
|
||||
The reader path must remain explicitly controlled until the capability snapshot justifies default-on behavior.
|
||||
|
||||
Required:
|
||||
|
||||
- flag-off path remains byte-compatible with current main expectations;
|
||||
- flag-on path produces trace evidence that reader code actually executed;
|
||||
- fallback behavior is explicit: reader refusal may fall through only when doing so cannot hide a recognized-skipped statement.
|
||||
|
||||
### Gate 6 — Capability snapshot, not promotion
|
||||
|
||||
After the above gates pass, run the existing relevant lanes / tables to populate the current capability snapshot.
|
||||
|
||||
Allowed:
|
||||
|
||||
- rerun existing GSM8K / capability-axis lanes;
|
||||
- update current metric artifacts;
|
||||
- populate existing TBD rows only where the lane already exists and produces signal.
|
||||
|
||||
Forbidden:
|
||||
|
||||
- adding new canonical lanes just to make the table look complete;
|
||||
- treating a snapshot as expert promotion;
|
||||
- merging a measurement PR that conceals capability absence behind aggregate numbers.
|
||||
|
||||
---
|
||||
|
||||
## Recommended branch / PR sequence
|
||||
|
||||
### PR 11A — Reader Phase 2 closure audit
|
||||
|
||||
**Branch:** `feat/brief-11-reader-closure-audit`
|
||||
|
||||
Deliverables:
|
||||
|
||||
- audit artifact for recognized-skipped statement cases;
|
||||
- per-case refusal taxonomy;
|
||||
- graph-completeness assertion helpers;
|
||||
- tests around incomplete graph refusal.
|
||||
|
||||
Exit condition:
|
||||
|
||||
- measurement identifies the exact remaining missing operators without raising `wrong`.
|
||||
|
||||
### PR 11B — Reader closure fixes
|
||||
|
||||
**Branch:** `feat/brief-11-reader-closure-fixes`
|
||||
|
||||
Deliverables:
|
||||
|
||||
- minimal statement-frame fixes for the highest-leverage missing operators;
|
||||
- no broad regex expansion unless proven temporary and fenced;
|
||||
- deterministic tests for each newly admitted case;
|
||||
- updated measurement artifact.
|
||||
|
||||
Exit condition:
|
||||
|
||||
- correct count improves or refusal taxonomy shrinks, while `wrong == 0`.
|
||||
|
||||
### PR 11C — Existing-lane capability snapshot
|
||||
|
||||
**Branch:** `eval/brief-11-capability-snapshot`
|
||||
|
||||
Deliverables:
|
||||
|
||||
- rerun existing relevant lanes only;
|
||||
- update metric ledgers / result JSONs;
|
||||
- summarize what changed after the reader closure;
|
||||
- explicitly list deferred lanes that remain blocked by missing capability.
|
||||
|
||||
Exit condition:
|
||||
|
||||
- snapshot numbers are reproducible and do not claim promotion.
|
||||
|
||||
### PR 11D — Next capability proposal
|
||||
|
||||
**Branch:** `docs/brief-11-next-capability-proposal`
|
||||
|
||||
Deliverables:
|
||||
|
||||
- a concise proposal selecting the next capability after GSM8K reader closure;
|
||||
- must pass ADR-0166's three-question test before any new lane is authored;
|
||||
- should compare: continued GSM8K operator closure, cross-domain reader generalization, tool-use trace integration, and Workbench demo hardening.
|
||||
|
||||
Exit condition:
|
||||
|
||||
- lead engineer has a yes/no decision artifact, not another sprawling roadmap.
|
||||
|
||||
---
|
||||
|
||||
## Failure modes to avoid
|
||||
|
||||
### 1. Correct-count greed
|
||||
|
||||
Raising correct count by allowing incomplete graph admission violates the project. Any such result should be treated as a failed experiment, not progress.
|
||||
|
||||
### 2. Regex relapse
|
||||
|
||||
ADR-0165 exists because regex can be useful at boundaries but dangerous as a semantic substitute. Brief 11 should not broaden regex to impersonate reader comprehension.
|
||||
|
||||
### 3. Eval surface inflation
|
||||
|
||||
New lanes before capability will generate uniform refusals and strategic noise. ADR-0166 forbids this.
|
||||
|
||||
### 4. Snapshot-as-promotion
|
||||
|
||||
A capability snapshot is a diagnostic state report. It is not an expert promotion, investor claim, or public disclosure.
|
||||
|
||||
### 5. Hidden side effects
|
||||
|
||||
Reader measurement must not mutate teaching stores, seed packs, runtime state, or public-facing artifacts unless the PR explicitly declares and tests those mutations.
|
||||
|
||||
---
|
||||
|
||||
## Minimum evidence package for the final Brief 11 PR
|
||||
|
||||
The final PR in this sequence should include:
|
||||
|
||||
```text
|
||||
- command(s) run
|
||||
- environment/config flags
|
||||
- baseline result path
|
||||
- reader-enabled result path
|
||||
- before/after correct/refused/wrong table
|
||||
- still-refused taxonomy
|
||||
- fixed-case taxonomy
|
||||
- graph completeness assertion summary
|
||||
- determinism rerun summary
|
||||
- explicit deferred-work list
|
||||
```
|
||||
|
||||
Suggested table:
|
||||
|
||||
| Mode | Correct | Refused | Wrong | Notes |
|
||||
|---|---:|---:|---:|---|
|
||||
| regex-only baseline | TBD | TBD | TBD | current main behavior |
|
||||
| reader Phase 1 | TBD | TBD | TBD | question-frame reader only |
|
||||
| reader Phase 2 closure | TBD | TBD | TBD | statement-frame reader path |
|
||||
|
||||
---
|
||||
|
||||
## Definition of done
|
||||
|
||||
Brief 11 is done when the repo has a reproducible, reviewed capability snapshot after Phase 2 reader closure and a precise backlog of remaining missing operators.
|
||||
|
||||
It is not done merely because a document exists. This document is the operator handoff; the engineering sequence above is the work.
|
||||
|
|
@ -0,0 +1,292 @@
|
|||
# Brief 11 — Phase 2 Reader Closure and Capability Snapshot
|
||||
|
||||
**Status:** Proposed handoff
|
||||
**Date:** 2026-05-27
|
||||
**Branch:** `docs/brief-11-handoff`
|
||||
**Scope:** Private repo planning and execution handoff only. No public-safe disclosure. No runtime claim is made by this document.
|
||||
|
||||
---
|
||||
|
||||
## Thesis
|
||||
|
||||
Brief 11 should close the current comprehension-reader corridor before widening measurement or domain scope.
|
||||
|
||||
The latest project state establishes three facts that must govern sequencing:
|
||||
|
||||
1. `gsm8k_math` is not blocked by a lack of eval lanes. It is blocked by incomplete statement understanding.
|
||||
2. The ADR-0164 reader is the correct replacement path for the brittle ADR-0163 recognizer/injector surface.
|
||||
3. ADR-0166 now forbids authoring new canonical eval lanes ahead of the capability they measure.
|
||||
|
||||
Therefore Brief 11 is not "make more benchmarks." Brief 11 is the bounded closure pass that turns the Phase 2 reader line into a reliable capability snapshot, then tells the next operator exactly what can safely happen after it.
|
||||
|
||||
---
|
||||
|
||||
## Current operating facts
|
||||
|
||||
Recent PR descriptions and session logs show the corridor clearly:
|
||||
|
||||
- Phase 1 reader coexistence is additive and flag-gated. With the flag off, behavior should remain byte-identical.
|
||||
- Phase 1 handles question sentences; statement sentences remained on the regex parser.
|
||||
- Phase 2 statement-frame work is the dominant path because most refused cases are caused by recognized-but-skipped statements / incomplete solver graphs.
|
||||
- Wrong-answer protection is currently more important than raising correct count by unsafe partial admission. The wrong=0 guard must remain non-negotiable.
|
||||
- Main has ADR-0166, which states that measurement follows capability, not the other way around.
|
||||
|
||||
Brief 11 must preserve those facts instead of drifting back into lane proliferation or superficial UI/demo work.
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Produce a mechanically useful handoff for the next engineering pass after Brief 10 / Phase 2 reader work:
|
||||
|
||||
1. Define the closure gates for the Phase 2 statement reader.
|
||||
2. Define the capability snapshot that should be run immediately after closure.
|
||||
3. Define the no-go boundaries that prevent unsafe widening.
|
||||
4. Define the next branch/PR sequence so a lead engineer can continue without re-litigating the strategy.
|
||||
|
||||
---
|
||||
|
||||
## Non-goals
|
||||
|
||||
Brief 11 does not:
|
||||
|
||||
- add new OOD eval lanes;
|
||||
- claim expert-level GSM8K capability;
|
||||
- relax wrong=0;
|
||||
- bypass the reader with broader regex patches;
|
||||
- add public-facing disclosure material;
|
||||
- promote any audit/expert/demo tier;
|
||||
- mutate seed packs or teaching stores as a side effect of chat/runtime smoke tests.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance gates
|
||||
|
||||
### Gate 1 — Reader completeness delta
|
||||
|
||||
Run the canonical GSM8K train-sample measurement with the reader path enabled and disabled.
|
||||
|
||||
Required outputs:
|
||||
|
||||
- baseline regex-only result;
|
||||
- reader-enabled result;
|
||||
- refused/correct/wrong counts;
|
||||
- per-case refusal reason taxonomy;
|
||||
- explicit list of cases fixed by Phase 2 statement frames;
|
||||
- explicit list of cases still refused and why.
|
||||
|
||||
Hard invariant:
|
||||
|
||||
- `wrong == 0` must hold.
|
||||
|
||||
If `correct` rises while `wrong > 0`, the PR is not a success. It is a regression with useful diagnostics.
|
||||
|
||||
### Gate 2 — Recognized-skipped statement audit
|
||||
|
||||
For every still-refused case where the reader recognized tokens but did not inject a complete graph, emit a compact audit row:
|
||||
|
||||
```text
|
||||
case_id | sentence_index | recognized_terms | skipped_frame | missing_operator | refusal_reason
|
||||
```
|
||||
|
||||
This is the main Brief 11 artifact because it turns the remaining GSM8K bottleneck into queued capability work instead of guesswork.
|
||||
|
||||
### Gate 3 — Graph completeness proof
|
||||
|
||||
For every admitted case, prove the solver graph is complete before candidate projection:
|
||||
|
||||
- all quantities have owners or intentionally anonymous scope;
|
||||
- all units are canonicalized;
|
||||
- operations are typed;
|
||||
- question target slot resolves to an entity/unit/kind;
|
||||
- no recognized-but-uncommitted frame is present.
|
||||
|
||||
This gate is the successor to the wrong=0 guard. The guard catches unsafe admission; graph completeness prevents unsafe admission from being attempted.
|
||||
|
||||
### Gate 4 — Determinism and trace stability
|
||||
|
||||
Run the reader-enabled measurement twice with identical input and config.
|
||||
|
||||
Required:
|
||||
|
||||
- identical case outcomes;
|
||||
- identical trace hashes where applicable;
|
||||
- stable refusal taxonomy ordering;
|
||||
- no nondeterministic iteration over entity registries, frame sets, or lexicon categories.
|
||||
|
||||
### Gate 5 — Feature-flag hygiene
|
||||
|
||||
The reader path must remain explicitly controlled until the capability snapshot justifies default-on behavior.
|
||||
|
||||
Required:
|
||||
|
||||
- flag-off path remains byte-compatible with current main expectations;
|
||||
- flag-on path produces trace evidence that reader code actually executed;
|
||||
- fallback behavior is explicit: reader refusal may fall through only when doing so cannot hide a recognized-skipped statement.
|
||||
|
||||
### Gate 6 — Capability snapshot, not promotion
|
||||
|
||||
After the above gates pass, run the existing relevant lanes / tables to populate the current capability snapshot.
|
||||
|
||||
Allowed:
|
||||
|
||||
- rerun existing GSM8K / capability-axis lanes;
|
||||
- update current metric artifacts;
|
||||
- populate existing TBD rows only where the lane already exists and produces signal.
|
||||
|
||||
Forbidden:
|
||||
|
||||
- adding new canonical lanes just to make the table look complete;
|
||||
- treating a snapshot as expert promotion;
|
||||
- merging a measurement PR that conceals capability absence behind aggregate numbers.
|
||||
|
||||
---
|
||||
|
||||
## Recommended branch / PR sequence
|
||||
|
||||
### PR 11A — Reader Phase 2 closure audit
|
||||
|
||||
**Branch:** `feat/brief-11-reader-closure-audit`
|
||||
|
||||
Deliverables:
|
||||
|
||||
- audit artifact for recognized-skipped statement cases;
|
||||
- per-case refusal taxonomy;
|
||||
- graph-completeness assertion helpers;
|
||||
- tests around incomplete graph refusal.
|
||||
|
||||
Exit condition:
|
||||
|
||||
- measurement identifies the exact remaining missing operators without raising `wrong`.
|
||||
|
||||
### PR 11B — Reader closure fixes
|
||||
|
||||
**Branch:** `feat/brief-11-reader-closure-fixes`
|
||||
|
||||
Deliverables:
|
||||
|
||||
- minimal statement-frame fixes for the highest-leverage missing operators;
|
||||
- no broad regex expansion unless proven temporary and fenced;
|
||||
- deterministic tests for each newly admitted case;
|
||||
- updated measurement artifact.
|
||||
|
||||
Exit condition:
|
||||
|
||||
- correct count improves or refusal taxonomy shrinks, while `wrong == 0`.
|
||||
|
||||
### PR 11C — Existing-lane capability snapshot
|
||||
|
||||
**Branch:** `eval/brief-11-capability-snapshot`
|
||||
|
||||
Deliverables:
|
||||
|
||||
- rerun existing relevant lanes only;
|
||||
- update metric ledgers / result JSONs;
|
||||
- summarize what changed after the reader closure;
|
||||
- explicitly list deferred lanes that remain blocked by missing capability.
|
||||
|
||||
Exit condition:
|
||||
|
||||
- snapshot numbers are reproducible and do not claim promotion.
|
||||
|
||||
### PR 11D — Next capability proposal
|
||||
|
||||
**Branch:** `docs/brief-11-next-capability-proposal`
|
||||
|
||||
Deliverables:
|
||||
|
||||
- a concise proposal selecting the next capability after GSM8K reader closure;
|
||||
- must pass ADR-0166's three-question test before any new lane is authored;
|
||||
- should compare: continued GSM8K operator closure, cross-domain reader generalization, tool-use trace integration, and Workbench demo hardening.
|
||||
|
||||
Exit condition:
|
||||
|
||||
- lead engineer has a yes/no decision artifact, not another sprawling roadmap.
|
||||
|
||||
---
|
||||
|
||||
## Failure modes to avoid
|
||||
|
||||
### 1. Correct-count greed
|
||||
|
||||
Raising correct count by allowing incomplete graph admission violates the project. Any such result should be treated as a failed experiment, not progress.
|
||||
|
||||
### 2. Regex relapse
|
||||
|
||||
ADR-0165 exists because regex can be useful at boundaries but dangerous as a semantic substitute. Brief 11 should not broaden regex to impersonate reader comprehension.
|
||||
|
||||
### 3. Eval surface inflation
|
||||
|
||||
New lanes before capability will generate uniform refusals and strategic noise. ADR-0166 forbids this.
|
||||
|
||||
### 4. Snapshot-as-promotion
|
||||
|
||||
A capability snapshot is a diagnostic state report. It is not an expert promotion, investor claim, or public disclosure.
|
||||
|
||||
### 5. Hidden side effects
|
||||
|
||||
Reader measurement must not mutate teaching stores, seed packs, runtime state, or public-facing artifacts unless the PR explicitly declares and tests those mutations.
|
||||
|
||||
---
|
||||
|
||||
## Minimum evidence package for the final Brief 11 PR
|
||||
|
||||
The final PR in this sequence should include:
|
||||
|
||||
```text
|
||||
- command(s) run
|
||||
- environment/config flags
|
||||
- baseline result path
|
||||
- reader-enabled result path
|
||||
- before/after correct/refused/wrong table
|
||||
- still-refused taxonomy
|
||||
- fixed-case taxonomy
|
||||
- graph completeness assertion summary
|
||||
- determinism rerun summary
|
||||
- explicit deferred-work list
|
||||
```
|
||||
|
||||
Suggested table:
|
||||
|
||||
| Mode | Correct | Refused | Wrong | Notes |
|
||||
|---|---:|---:|---:|---|
|
||||
| regex-only baseline | TBD | TBD | TBD | current main behavior |
|
||||
| reader Phase 1 | TBD | TBD | TBD | question-frame reader only |
|
||||
| reader Phase 2 closure | TBD | TBD | TBD | statement-frame reader path |
|
||||
|
||||
---
|
||||
|
||||
## Definition of done
|
||||
|
||||
Brief 11 is done when the repo has a reproducible, reviewed capability snapshot after Phase 2 reader closure and a precise backlog of remaining missing operators.
|
||||
|
||||
It is not done merely because a document exists. This document is the operator handoff; the engineering sequence above is the work.
|
||||
|
||||
---
|
||||
|
||||
## Suggested PR body template
|
||||
|
||||
```markdown
|
||||
## Summary
|
||||
|
||||
Brief 11 handoff: closes the ADR-0164 Phase 2 reader corridor before widening measurement scope. Defines gates for reader completeness, recognized-skipped statement audit, graph completeness, determinism, feature-flag hygiene, and existing-lane snapshot.
|
||||
|
||||
## Why
|
||||
|
||||
Recent PRs show the GSM8K bottleneck is incomplete statement understanding, not missing eval lanes. ADR-0166 requires capability before measurement expansion.
|
||||
|
||||
## Deliverables
|
||||
|
||||
- docs/handoffs/BRIEF-11-phase-2-reader-closure-and-capability-snapshot.md
|
||||
|
||||
## Test plan
|
||||
|
||||
Docs-only. No runtime change.
|
||||
|
||||
## Follow-up sequence
|
||||
|
||||
- 11A: reader closure audit
|
||||
- 11B: closure fixes
|
||||
- 11C: existing-lane capability snapshot
|
||||
- 11D: next capability proposal
|
||||
```
|
||||
Loading…
Reference in a new issue