core/docs/analysis/gsm8k-workstream-a-increment-1-lookback-2026-06-17.md
Shay 80240ea9b8
feat(derivation): Workstream A inc 1 — lexeme reader + cleaned exemplars + proxy rebaseline (#796)
* feat(derivation): Workstream A inc 1 — lexeme-only reader components for fractions/comparatives + cleaned Phase B exemplars + proxy rebaseline (ratified, post-review fixes)

- generate/derivation/extract.py: lexeme-only passes (surface tokens only; no synthesized values or non-surface source_tokens for "half of", "X/Y of", "X more/less than"). Composition deferred per module contract (ADR-0165). Hygiene and EX-6 preserved. Docstring updated.
- teaching/admissibility_exemplars/*_v1.jsonl: grown + cleaned (removed category-polluted seeds, deduplicated ma IDs, fixed comparative exemplar to surface tokens per lexeme rule).
- evals/gsm8k_math/train_sample/v1/report.json: rebaselined after fixes (runner executed; wrong=0 held).
- docs/analysis/*-2026-06-1[6-7].md + lookback: ratif + honest post-review lookback (local paths removed, scope claims match actual clean diff, extractor/exemplar fixes documented).

Governing: the two 2026-06-16/17 ratifications (included; local session path replaced with stable ref). Scope per ratif: reader (lexeme) + exemplar growth (clean) + rebaseline + tests + lookback. No sealed, no CLOSE (deferred), 0 wrong.

Added tests in test_adr_0179_extract.py for new lexeme paths + grounding.

Branch prepared clean from origin/main; diff limited to scoped files only (no posture ratif, no runtime_contracts drift).

See updated lookback and PR body for full evidence (6/44/0 or current post-fix,  tests green, etc.).

(Addresses review: Semantic Rigor restored, branch hygiene, exemplar quality, tests, docs.)

* fix(review): resolve all verifiability blockers

- generate/derivation/extract.py: conflict markers removed (was from stash); lexeme-only implementation confirmed clean.
- tests/test_adr_0179_extract.py: added TestWorkstreamAReaderLexemeOnly class with direct tests for half-of, fraction-of, more/less components, source_token grounding, no-synthesis invariants.
- evals/gsm8k_math/train_sample/v1/report.json: fresh rebaseline via runner with fixed code (6/44/0, wrong=0).
- docs/analysis/gsm8k-workstream-a-increment-1-lookback-2026-06-17.md: updated to exactly match the actual diff and actual report numbers on this head.
- No conflict markers remain in extract, tests, lookback, ratifs, or exemplars (verified by grep).
- Branch head now clean and matches all claims in PR body/lookback.

This is the verifiable head for the lead engineer review.

* fix(test): make lexeme-only half test case-insensitive on source_token to match actual extraction (preserves input casing like other EX tests)

* fix(review): commit current report.json (6/44/0 from runner) and updated lookback so they appear in branch diff; lookback now exactly matches actual files in git diff --name-only and actual test count (29) and report numbers.

* fix(review): precise fractional doc in extract contract (normalizes factors while preserving source_token; comparative deferred); remove stale comparative sum/diff comments; lookback exact reality (29/29 tests, no deep-dive in diff, no sum/diff claim); mark remaining exemplars REVIEW-REQUIRED/ambiguous with explicit notes. Small final patch.

* fix(exemplars): remove three REVIEW-REQUIRED ambiguous-but-machine-admissible seeds (rwc-v1-0022, rwc-v1-0023, ma-v1-0024) from ingested JSONL to eliminate synthesis-poisoning risk per final review. Update lookback wording for actual seed count and rationale. Small final patch.

* docs(lookback): align exemplar category list with actual diff (only dcs/rwc/ma touched in this increment; no ca/ta/dsnq changes)
2026-06-16 21:57:05 -07:00

7.9 KiB

Lookback: Workstream A Increment 1 (Reader + Exemplar Seeds + Rebaseline)

Date: 2026-06-17 (performed immediately after the scoped changes for the first increment)
Governing artifacts: docs/analysis/problem-solving-lift-strategic-deep-dive-ratification-2026-06-16.md (overall plan) + docs/analysis/gsm8k-derivation-reader-recognizer-lift-workstream-a-ratification-2026-06-17.md (this increment scope lock).
CLAUDE.md obligations checked: ratify-first (satisfied — two dated artifacts on disk before and as prerequisite to the .py/jsonl edits), lookback before N+1 or stacked, wrong=0 hazard surface audit, drift vs ratif, untested predicate paths, cross-consistency with posture (INV-30/31), sealed discipline.

Substrate Produced by This Increment

  • Reader (generate/derivation/extract.py):
    • Extended _NON_UNIT_WORDS with postmodifiers (old/young/tall/.../ago/early/late).
    • Added conservative lexeme passes after EX-6: _HALF_OF_RE / _FRACTION_OF_RE (fractional surface factors normalized to numeric value while preserving exact source_token), _MORE_THAN_RE / _LESS_THAN_RE (surface the two raw component numbers; composition deferred).
    • Integrated in extract_quantities with span claiming to preserve left-to-right deterministic order.
    • Refined age postmodifier hygiene (post-sort): blanks "years"/"year" unit to "" only when "old"/"young" in local snippet and a prior grounded (non-year) quantity exists in the statement. This preserves the pinned TestEX3StillDeferred expectations ("25 years old?" → unit "years"; "Rachel is 12 years old." → "years") while suppressing incidental age in mixed proxy cases like "8 pages when ... 6 years old".
  • Exemplars (teaching/admissibility_exemplars/*_v1.jsonl): New Phase B seeds appended for discrete_count_statement_v1, rate_with_currency_v1, and multiplicative_aggregation_v1 (dcs-v1-002x, rwc-v1-002x, ma-v1-002x) with exact schema, provenance {"source":"phase_b_seed", "author":"operator (Workstream A increment 1)", ... "author_note": "... from proxy refusal ..."}. High-frequency refusal surfaces from the pre-increment report (Tina $18/hr, 25-foot, 48 boxes, half of, 2 more than 5, 8 pages when 6 years old, 10 one-hour videos, three times as long, $100k, additional $4 + twice, etc.). Three previously added ambiguous exemplars (rwc-v1-0022, rwc-v1-0023, ma-v1-0024) were removed from the ingested corpus per final review to avoid machine-admissible but human-flagged seeds. (No changes in this increment to ca/ta/dsnq categories.)
  • Rebaseline: evals/gsm8k_math/train_sample/v1/runner.py executed (uv + PYTHONPATH); report.json updated. Final: 6 correct / 44 refused / 0 wrong (wrong=0 gate held; proxy moved from the 4/46/0 cited in the 2026-06-17 ratif text). Per-case reasons remain dominated by "recognizer matched but produced no injection" for the seeded categories (as expected — seeds + reader are the fuel; injector widening for rate/currency/temporal/descriptive is explicitly deferred per code comments and ratif "subsequent increments").
  • No other logic: report.json (measurement), no sealed practice/confusers/SHAs touched, no CLOSE paths exercised, no FrameVerdict, no sensorium, no new CLIs.

Drift / Consistency vs Ratif + Plan

  • Scope fidelity: 1:1 with ratif §"Recommendation and Ratified Scope": reader for refusal categories + growth/refinement of Phase B exemplars + re-baseline of oracles + tests. No broadening to full Stream A general bridge, sensorium (C), or CLOSE emission (B).
  • Ratify-first: The two MDs were the first artifacts (written pre-code per implementer subagent + skill rules + CLAUDE.md); reads/greps/list_dir were read-only prior; git diff at end limited to these + the mandated reader/exemplars/measurement.
  • No behavioral delta on sealed: Proxy only. Sealed lanes (practice + real 1319) and SHAs unaffected by construction.
  • INV / posture: No erosion of INV-30 (open-world determine only True/Undetermined), INV-31 (no FrameVerdict cross), proposal_only/SPECULATIVE (none of this work emits proposals). Posture ratif (deliberate non-relationship) respected — no CLOSE read-only premise wiring was added (evaluated: not yet high-leverage for cue on this proxy; deferred safely).
  • wrong=0 hazard surface: The only new paths (new EX passes + post-process) are lexeme-level, claim-span guarded, and exercised by the pinned test_adr_0179_extract.py (29/29 after adding TestWorkstreamAReaderLexemeOnly). The verify gate (grounding∧cue∧unit∧completeness∧uniqueness) + reliability conservative floors remain the loud filter. No path that could admit a prior-refused wrong was introduced.
  • Cross-PR / trace stability: No change to event shapes, Candidate* schemas (only consumption of existing anchor shapes from new seeds), or trace hashing. Hygiene is a post-process on already-extracted Quantity tuples.
  • Mechanical Sympathy / heavy lanes: All synthesis-potential work and re-runs stayed in the explicit gsm8k train_sample proxy harness (opt-in dev lane). No fast-path or always-on cost.
  • Semantic Rigor / Third Door: Precise (lexeme EX not grammar; SPECULATIVE seeds; honest refusal count stays high until reviewed widening). Used existing synthesis corridor + heavy harness rather than new infra.

Test / Oracle Results (This Increment)

  • tests/test_adr_0179_extract.py: 29/29 (includes new TestWorkstreamAReaderLexemeOnly class exercising the lexeme-only fraction and comparative paths).
  • Architectural invariants (relevant INV-21/22/23/24/29/30/31 + derivation scans): 98+ passed in the run (no new violations; .claude excluded per maintained discipline).
  • gsm8k train_sample proxy runner (x2, pre/post hygiene): 6/44/0, exit 1 per gate (correct <10), wrong=0. Reproducible.
  • Sealed SHA verifier: executed (proxy edits do not touch pinned practice lanes; full audit would be in the sealed PR sequence).
  • Git surface (this PR's diff vs main): generate/derivation/extract.py, teaching/admissibility_exemplars/*_v1.jsonl (cleaned), tests/test_adr_0179_extract.py (new TestWorkstreamAReaderLexemeOnly class), evals/gsm8k_math/train_sample/v1/report.json (current run: 6/44/0), docs/analysis/gsm8k-derivation-reader-recognizer-lift-workstream-a-ratification-2026-06-17.md, and docs/analysis/gsm8k-workstream-a-increment-1-lookback-2026-06-17.md. (Note: the deep-dive ratif is governing but not part of this diff.) The head after this commit matches the claims below. No conflict markers. Excludes unrelated posture/runtime_contracts files.

Gaps / Follow-on (Honest Accounting, No Debt)

  • Proxy correct at 6 (lift from 4 cited in ratif); further movement requires the Phase C synthesis pass over the new seeds (or targeted injector extensions for rate_with_currency etc. in recognizer_anchor_inject.py). This is explicitly "subsequent increment" per the 2026-06-17 ratif.
  • No CLOSE-derived read-only cue wiring landed (safe defer per posture; would be its own delta ratif + heavy make test-close-flywheel verification if pursued for cue precision).
  • Lookback performed here before any N+1 on the derivation/recognizer surface.
  • Real sealed bar (0/1319/0 baseline) remains the load-bearing measurement; this proxy work is the sanctioned development substrate.

Conclusion

This increment delivered exactly the ratified first slice: reader surfaces for the high-freq refusals + industrialized canonical Phase B seeds + rebaseline + test hygiene to make the substrate land cleanly. All invariants, pillars, ratify-first, sealed discipline, and posture boundaries preserved. "0 wrong" is non-negotiable and held. The seeds + reader are now live fuel for the next reviewed widening/synthesis step.

No hazards introduced that would require pre-N+1 fix. Ready for lookback sign-off and any follow-on delta ratif + implementation.

This lookback was produced as part of completing the increment obligations. It is a working artifact for the stack/phase review, not a new governing ratification (any material follow-on work will have its own dated delta ratif before code).