docs: semantic-state transition blueprint and ADR-0184 scope (#489)
* docs: add semantic state transition blueprint * docs: add ADR-0184 semantic state transitions * docs: add RB-GSM solver design input notes
This commit is contained in:
parent
171b1cd7a5
commit
390b6f376d
3 changed files with 2146 additions and 0 deletions
551
docs/decisions/ADR-0184-scoped-semantic-state-transitions.md
Normal file
551
docs/decisions/ADR-0184-scoped-semantic-state-transitions.md
Normal file
|
|
@ -0,0 +1,551 @@
|
|||
# ADR-0184 — Scoped Semantic State Transitions for English Multi-Step Reasoning
|
||||
|
||||
**Status:** Proposed
|
||||
**Date:** 2026-05-29
|
||||
**Author:** ChatGPT / Josh direction
|
||||
**Blueprint:** [`docs/handoff/SEMANTIC-STATE-TRANSITION-BLUEPRINT.md`](../handoff/SEMANTIC-STATE-TRANSITION-BLUEPRINT.md)
|
||||
**Anchor:** [[thesis-decoding-not-generating]]
|
||||
**Builds on:** ADR-0165, ADR-0174, ADR-0176, ADR-0177, ADR-0178, ADR-0179, ADR-0182
|
||||
**Supersedes:** no runtime path yet; this is a scope-setting ADR for the next implementation sequence.
|
||||
|
||||
---
|
||||
|
||||
## 1. Decision
|
||||
|
||||
CORE will introduce a sealed derivation-lane semantic-state substrate for English multi-step math reasoning.
|
||||
|
||||
The substrate reads word problems into scoped, entity-owned state transitions before arithmetic derivation:
|
||||
|
||||
```text
|
||||
text
|
||||
-> lexeme extraction + clause segmentation
|
||||
-> semantic frames
|
||||
-> entity-bound state transitions
|
||||
-> semantic ledger / world candidates
|
||||
-> GroundedDerivation replay
|
||||
-> existing self-verification / classification
|
||||
-> cross-composer pooling
|
||||
-> answer or refusal
|
||||
```
|
||||
|
||||
This ADR does **not** authorize a serving-path change. The first implementation must remain sealed under `generate/derivation/**` and must emit ordinary `GroundedDerivation` candidates for the existing verifier/pool to judge.
|
||||
|
||||
The immediate implementation target is to promote the proven accumulation logic in `generate/derivation/accumulate.py` into a reusable semantic-state package before adding transfer, comparison, rate/container, temporal replay, or DAG logic.
|
||||
|
||||
---
|
||||
|
||||
## 2. Why this is needed
|
||||
|
||||
CORE's arithmetic verification layer is stronger than its semantic reading layer.
|
||||
|
||||
The current derivation lane can verify candidate arithmetic traces using grounding, cue evidence, unit consistency, completeness, uniqueness, and commit eligibility. That is the right final judge. But hard English word problems fail earlier: the system often lacks a scoped representation of the situation the text describes.
|
||||
|
||||
The missing layer is not more arithmetic. It is a typed world-state reading:
|
||||
|
||||
```text
|
||||
Sam has 14 apples.
|
||||
He buys 9 more.
|
||||
How many apples does Sam have now?
|
||||
```
|
||||
|
||||
should first become:
|
||||
|
||||
```text
|
||||
SET_STATE(Sam.apples, 14)
|
||||
GAIN(Sam.apples, 9)
|
||||
TARGET(final Sam.apples)
|
||||
```
|
||||
|
||||
and only then replay to:
|
||||
|
||||
```text
|
||||
14 + 9 = 23
|
||||
```
|
||||
|
||||
Without this layer, future work will keep reimplementing the same semantic logic inside local composers:
|
||||
|
||||
- subject / referent binding;
|
||||
- gain/loss polarity;
|
||||
- question target scope;
|
||||
- temporal scope;
|
||||
- foreign distractor classification;
|
||||
- transfer handling;
|
||||
- comparison direction;
|
||||
- rate/container binding;
|
||||
- branch/DAG quantity reuse.
|
||||
|
||||
That is the path to composer spaghetti and wrong=0 hazards.
|
||||
|
||||
---
|
||||
|
||||
## 3. What already exists
|
||||
|
||||
The new substrate should use the existing system rather than bypass it.
|
||||
|
||||
### Existing pieces to preserve
|
||||
|
||||
- `generate/derivation/extract.py` — lexeme-level quantity/unit/source-token extraction.
|
||||
- `generate/derivation/clauses.py` — deterministic clause segmentation and local clause results.
|
||||
- `generate/derivation/accumulate.py` — first working state-transition composer: single-referent gain/loss accumulation.
|
||||
- `generate/derivation/target.py` — target extraction and prior-state question guard.
|
||||
- `generate/derivation/verify.py` — final proof gate: grounding, cue, unit, completeness, classification, uniqueness.
|
||||
- `generate/derivation/pool.py` — cross-composer candidate pooling, disagreement refusal, commit eligibility.
|
||||
- `generate/comprehension/state.py` — immutable state, entity/quantity/question/refusal/hypothesis concepts and canonical hashing; useful as design precedent, not automatically the active scoring path.
|
||||
|
||||
### Existing pieces to avoid misusing
|
||||
|
||||
- Do **not** resurrect the old broad comprehension lifecycle as the active scoring reader unless a separate audit proves it is now load-bearing.
|
||||
- Do **not** revive gender-blind most-recent antecedent pronoun resolution.
|
||||
- Do **not** make semantic worlds commit directly.
|
||||
- Do **not** replace `verify.py`.
|
||||
- Do **not** let any composer pick first non-`None` when multiple competing readings exist; use pooling.
|
||||
|
||||
---
|
||||
|
||||
## 4. Where this fits
|
||||
|
||||
Add a new sealed derivation-lane package:
|
||||
|
||||
```text
|
||||
generate/derivation/state/
|
||||
__init__.py
|
||||
model.py
|
||||
bind.py
|
||||
change.py
|
||||
ledger.py
|
||||
target.py
|
||||
replay.py
|
||||
refusals.py
|
||||
```
|
||||
|
||||
Later phases may add:
|
||||
|
||||
```text
|
||||
generate/derivation/state/transfer.py
|
||||
generate/derivation/state/compare.py
|
||||
generate/derivation/state/rate.py
|
||||
generate/derivation/state/time.py
|
||||
generate/derivation/state/world.py
|
||||
generate/derivation/state/dag.py
|
||||
```
|
||||
|
||||
The initial integration path is deliberately conservative:
|
||||
|
||||
```text
|
||||
accumulate.py public functions
|
||||
-> semantic-state helper modules
|
||||
-> semantic ledger
|
||||
-> replay to GroundedDerivation
|
||||
-> existing verify/select/classify
|
||||
```
|
||||
|
||||
`pool.py` continues to arbitrate final candidates.
|
||||
|
||||
---
|
||||
|
||||
## 5. Initial object language
|
||||
|
||||
The first version should model only what accumulation already needs.
|
||||
|
||||
Suggested minimal types:
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class EntityMention:
|
||||
surface: str
|
||||
canonical: str
|
||||
clause_index: int
|
||||
token_index: int
|
||||
kind: str # proper | pronoun | implicit | unknown
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class QuantityMention:
|
||||
value: float
|
||||
unit: str
|
||||
source_token: str
|
||||
clause_index: int
|
||||
role: str # state | delta | scalar | rate | target | unknown
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class CueMention:
|
||||
surface: str
|
||||
cue_kind: str # gain | loss | aggregate | multiplicative | temporal | ...
|
||||
clause_index: int
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class StateKey:
|
||||
entity: str
|
||||
unit: str
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class StateTransition:
|
||||
key: StateKey
|
||||
op: str # set | gain | loss
|
||||
quantity: QuantityMention
|
||||
cue: CueMention
|
||||
clause_index: int
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class SemanticLedger:
|
||||
transitions: tuple[StateTransition, ...]
|
||||
```
|
||||
|
||||
The first replay target is a standard `GroundedDerivation`.
|
||||
|
||||
This ADR intentionally does not require all final types up front. It requires that types be frozen, deterministic, and covered by tests before they are used for candidate generation.
|
||||
|
||||
---
|
||||
|
||||
## 6. Wrong=0 obligations
|
||||
|
||||
Every implementation phase must preserve the following.
|
||||
|
||||
### 6.1 Semantic worlds do not commit directly
|
||||
|
||||
A semantic world may only produce a candidate. It cannot answer by itself.
|
||||
|
||||
Commit path remains:
|
||||
|
||||
```text
|
||||
SemanticWorld / SemanticLedger
|
||||
-> GroundedDerivation
|
||||
-> self_verifies / classify_derivation
|
||||
-> resolve_pooled
|
||||
```
|
||||
|
||||
A test must fail if a semantic-world-only answer bypasses `verify.py` or `pool.py`.
|
||||
|
||||
### 6.2 Existing complete-commit rules remain intact
|
||||
|
||||
`complete` remains the only commit-eligible classification. Exempt/distractor readings may enter the pool only to force disagreement/refusal, never as sole committable answers.
|
||||
|
||||
### 6.3 Ambiguous referents refuse
|
||||
|
||||
If a state transition needs an entity and the entity cannot be safely bound, the semantic world refuses or is not emitted.
|
||||
|
||||
Required tests:
|
||||
|
||||
```text
|
||||
Sam has 14 apples. He buys 9 more. -> same referent, candidate allowed
|
||||
Sam has 14 apples. Tom buys 9 more. -> new actor, candidate refused
|
||||
Alice has 5. Bob has 3. She buys 2. -> ambiguous, refused unless explicitly resolved by a future safe rule
|
||||
```
|
||||
|
||||
### 6.4 Ambiguous polarity refuses
|
||||
|
||||
A gain/loss event may be emitted only when polarity is unambiguous and cue-grounded.
|
||||
|
||||
Required tests:
|
||||
|
||||
```text
|
||||
gets 4 more -> gain
|
||||
loses 4 -> loss
|
||||
owns 4 -> no change cue, refused
|
||||
mixed gain/loss cue in same change frame -> refused
|
||||
```
|
||||
|
||||
### 6.5 Unsupported temporal target refuses
|
||||
|
||||
Until temporal replay exists, prior-state questions remain refused.
|
||||
|
||||
Required tests:
|
||||
|
||||
```text
|
||||
How much did Lisa have before lunch? -> refuse until time replay exists
|
||||
How much does Lisa have left? -> forward/net target allowed
|
||||
before in body narrative does not trip prior-state refusal
|
||||
used to make does not trip prior-state refusal
|
||||
```
|
||||
|
||||
### 6.6 Determinism
|
||||
|
||||
All emitted candidates must be deterministic:
|
||||
|
||||
- no random ordering;
|
||||
- no unordered set/dict iteration in outputs;
|
||||
- stable candidate ordering;
|
||||
- frozen dataclasses or canonical bytes for replay-critical state.
|
||||
|
||||
---
|
||||
|
||||
## 7. Implementation sequence
|
||||
|
||||
### S1 — Extract proven accumulation helpers
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
generate/derivation/state/bind.py
|
||||
generate/derivation/state/change.py
|
||||
```
|
||||
|
||||
Move behavior-equivalent helpers from `accumulate.py`:
|
||||
|
||||
```text
|
||||
_subject_token -> leading_subject_token
|
||||
_same_referent -> continues_anchor_referent
|
||||
_polarity -> classify_change_polarity
|
||||
_cue -> select_change_cue
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- accumulation tests unchanged and green;
|
||||
- pool tests unchanged and green;
|
||||
- new helper tests prove non-vacuous referent and polarity guards;
|
||||
- no serving imports;
|
||||
- no runner changes;
|
||||
- no behavior change.
|
||||
|
||||
### S2 — Add accumulation semantic ledger
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
generate/derivation/state/model.py
|
||||
generate/derivation/state/ledger.py
|
||||
generate/derivation/state/replay.py
|
||||
```
|
||||
|
||||
Represent:
|
||||
|
||||
```text
|
||||
SET_STATE(entity, unit, value)
|
||||
GAIN(entity, unit, value)
|
||||
LOSS(entity, unit, value)
|
||||
```
|
||||
|
||||
Then replay the ledger into `GroundedDerivation`.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- `compose_accumulation()` output unchanged;
|
||||
- `accumulation_candidates()` output unchanged or explicitly equivalent by candidate answer/key tests;
|
||||
- direct manual construction of accumulation chains in `accumulate.py` is minimized or removed;
|
||||
- replay tests prove ledger -> `GroundedDerivation` correctness.
|
||||
|
||||
### S3 — Add semantic target wrapper
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
generate/derivation/state/target.py
|
||||
```
|
||||
|
||||
Wrap current target extraction with semantic fields:
|
||||
|
||||
```text
|
||||
entity: optional
|
||||
unit: optional
|
||||
time_scope: final | prior | unknown
|
||||
relation: count | difference | aggregate | unknown
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- current prior-state guard behavior preserved;
|
||||
- forward/net questions still resolve;
|
||||
- target wrapper is conservative and refuses unsupported scope.
|
||||
|
||||
### S4 — Add semantic candidate source to pool
|
||||
|
||||
Create public surface:
|
||||
|
||||
```python
|
||||
def semantic_state_candidates(problem_text: str) -> tuple[GroundedDerivation, ...]: ...
|
||||
```
|
||||
|
||||
Initially it delegates to accumulation-backed worlds.
|
||||
|
||||
Then update `pool.py` from:
|
||||
|
||||
```text
|
||||
accumulation_candidates, multiplicative_candidates, candidate_chains
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```text
|
||||
semantic_state_candidates, multiplicative_candidates, candidate_chains
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- all ADR-0182 pool tests remain green;
|
||||
- clean accumulation still commits;
|
||||
- distractor cases still refuse;
|
||||
- exempt-only still never commits;
|
||||
- direct `pool.py` import of `accumulation_candidates` can be removed.
|
||||
|
||||
### S5 — Transfer events
|
||||
|
||||
Add source/target entity transitions:
|
||||
|
||||
```text
|
||||
Sam gives Tom 3 apples
|
||||
-> Sam.apples -= 3
|
||||
-> Tom.apples += 3
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- source-target binding tests;
|
||||
- ambiguous target refused;
|
||||
- no initial state refused;
|
||||
- target-specific question required for commit.
|
||||
|
||||
### S6 — Comparison / difference frames
|
||||
|
||||
Add difference questions:
|
||||
|
||||
```text
|
||||
Sam has 10 apples. Tom has 7 apples. How many more apples does Sam have than Tom?
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- direction explicit -> commit;
|
||||
- direction ambiguous -> refuse;
|
||||
- same-unit aggregate does not become difference by accident.
|
||||
|
||||
### S7 — Rate / container frames
|
||||
|
||||
Add structurally bound products:
|
||||
|
||||
```text
|
||||
24 boxes with 12 erasers each
|
||||
60 miles per hour for 2 hours
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- legitimate products commit;
|
||||
- distractor duration with unrelated target refuses;
|
||||
- protected existing product positives do not regress;
|
||||
- product-of-all fallback remains until semantic rate frames cover positives.
|
||||
|
||||
### S8 — Temporal replay
|
||||
|
||||
Use ledger history to answer:
|
||||
|
||||
```text
|
||||
before
|
||||
after
|
||||
initially
|
||||
finally
|
||||
left
|
||||
now
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- prior-state questions answer only when event boundary is unambiguous;
|
||||
- ambiguous temporal scope refuses;
|
||||
- current prior-state refusal tests update only when the new capability is proven.
|
||||
|
||||
### S9 — Held worlds and DAGs
|
||||
|
||||
Represent multiple possible semantic worlds and quantity reuse / branch structures.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- linear cases unchanged;
|
||||
- unresolved multi-world ambiguity refuses;
|
||||
- no DAG candidate commits without proof obligations equivalent to current grounding/completeness/target requirements.
|
||||
|
||||
---
|
||||
|
||||
## 8. Immediate next PR authorized by this ADR
|
||||
|
||||
Branch:
|
||||
|
||||
```text
|
||||
feat/adr-0184-s1-semantic-state-helpers
|
||||
```
|
||||
|
||||
Scope:
|
||||
|
||||
1. Add `generate/derivation/state/__init__.py`.
|
||||
2. Add `generate/derivation/state/bind.py`.
|
||||
3. Add `generate/derivation/state/change.py`.
|
||||
4. Move proven helper logic from `accumulate.py` into those modules.
|
||||
5. Keep `compose_accumulation()` unchanged externally.
|
||||
6. Keep `accumulation_candidates()` unchanged externally.
|
||||
7. Add tests for extracted helpers.
|
||||
8. Do not touch serving.
|
||||
9. Do not change runners.
|
||||
10. Do not delete old reader files.
|
||||
|
||||
This is intentionally a behavior-equivalent refactor. It creates the seam for multi-step English reasoning without expanding capability in the same PR.
|
||||
|
||||
---
|
||||
|
||||
## 9. Explicit non-actions
|
||||
|
||||
Do **not** do any of the following in S1:
|
||||
|
||||
- no serving import;
|
||||
- no `chat/**` edit;
|
||||
- no `generate/math_roundtrip.py` edit;
|
||||
- no runner behavior change;
|
||||
- no deletion of `generate/comprehension/lifecycle.py`;
|
||||
- no deletion of `generate/math_parser.py`;
|
||||
- no product-of-all fallback removal;
|
||||
- no transfer/comparison/rate/temporal/DAG implementation;
|
||||
- no broad grammar parser;
|
||||
- no pronoun resolver beyond current conservative same-referent guard.
|
||||
|
||||
These are future phases or separate ADRs.
|
||||
|
||||
---
|
||||
|
||||
## 10. Dead-weight policy
|
||||
|
||||
This ADR does not delete code immediately. It establishes deletion criteria.
|
||||
|
||||
### Composer-local helper deletion
|
||||
|
||||
After S1/S2, helper logic for subject extraction, referent continuity, polarity, cue selection, and accumulation replay should not remain private to `accumulate.py` except as thin compatibility wrappers.
|
||||
|
||||
### Pool prior-state guard migration
|
||||
|
||||
After S3, direct prior-state logic in `pool.py` should migrate into semantic target handling or become a compatibility call.
|
||||
|
||||
### Product-of-all fallback demotion
|
||||
|
||||
Only after S7 proves rate/container semantic frames protect current positives and refuse known distractors may product-of-all fallback be demoted or deleted.
|
||||
|
||||
### Old comprehension lifecycle
|
||||
|
||||
Do not delete casually. Demote or delete only after an audit proves it is inert relative to active scoring and no tests depend on it as a runtime path.
|
||||
|
||||
---
|
||||
|
||||
## 11. Acceptance criteria for this ADR moving from Proposed to Accepted
|
||||
|
||||
This ADR may move to Accepted when:
|
||||
|
||||
1. S1 lands with no behavior change and tests prove helper extraction is non-vacuous.
|
||||
2. S2 lands with semantic ledger replay for accumulation and accumulation behavior remains equivalent.
|
||||
3. `resolve_pooled` can source accumulation through `semantic_state_candidates()` with ADR-0182 behavior preserved.
|
||||
4. Serving remains unchanged.
|
||||
5. Existing wrong=0 lanes remain wrong=0.
|
||||
6. Documentation identifies all old code paths that are retained, demoted, or scheduled for later deletion.
|
||||
|
||||
---
|
||||
|
||||
## 12. Final doctrine
|
||||
|
||||
CORE should complete English multi-step reasoning by reading into scoped semantic state, not by accumulating more arithmetic shapes.
|
||||
|
||||
The durable loop is:
|
||||
|
||||
```text
|
||||
read the world stated by the text
|
||||
-> mutate scoped state under cue-licensed rules
|
||||
-> bind the question to a state/relation/time
|
||||
-> replay into arithmetic proof objects
|
||||
-> let existing verifier and pool accept or refuse
|
||||
```
|
||||
|
||||
The first safe step is not more coverage. It is extracting the already-proven accumulation semantics into a reusable substrate so the next capability layers can compose without becoming local patches.
|
||||
399
docs/handoff/RB-GSM-SOLVER-DESIGN-INPUT.md
Normal file
399
docs/handoff/RB-GSM-SOLVER-DESIGN-INPUT.md
Normal file
|
|
@ -0,0 +1,399 @@
|
|||
# RB-GSM Solver Design Input Notes
|
||||
|
||||
**Status:** design input / cautionary mapping
|
||||
**Branch:** `docs/semantic-state-transition-blueprint`
|
||||
**Related PR:** #489
|
||||
**Related docs:**
|
||||
|
||||
- `docs/handoff/SEMANTIC-STATE-TRANSITION-BLUEPRINT.md`
|
||||
- `docs/decisions/ADR-0184-scoped-semantic-state-transitions.md`
|
||||
|
||||
---
|
||||
|
||||
## 1. Why this note exists
|
||||
|
||||
Josh provided an external-style sketch of a pure rule-based GSM8K solver: loose regex extraction, keyword dictionaries, if-then rules, a state builder, an operation queue, and deterministic arithmetic execution.
|
||||
|
||||
The sketch is directionally useful because it reinforces the same high-level move captured in ADR-0184:
|
||||
|
||||
```text
|
||||
English story
|
||||
-> signal extraction
|
||||
-> state machine
|
||||
-> operation / transition list
|
||||
-> exact arithmetic replay
|
||||
```
|
||||
|
||||
However, the sketch also contains claims and shortcuts CORE must **not** import as doctrine:
|
||||
|
||||
- “100% coverage with 30–50 rules” is an empirical claim, not an architectural fact.
|
||||
- “Last actor works 99%” is not acceptable as a commit rule under `wrong = 0`.
|
||||
- Regex and keyword hits may propose readings, but they must not decide answers.
|
||||
- A first matching rule must not bypass pooling, completeness, target binding, or disagreement refusal.
|
||||
|
||||
This document records what to use, what to reject, and how to fold the useful pieces into CORE's semantic-state transition plan.
|
||||
|
||||
---
|
||||
|
||||
## 2. Useful principles to keep
|
||||
|
||||
### 2.1 Loose extraction, strict interpretation
|
||||
|
||||
The RB-GSM sketch correctly separates rough signal collection from final logic:
|
||||
|
||||
```text
|
||||
loose regex = candidate signal collector
|
||||
keyword / cue table = deterministic interpretation proposal
|
||||
state machine + verifier = final decision path
|
||||
```
|
||||
|
||||
CORE already follows a related discipline in `generate/derivation/extract.py`: extraction should identify lexeme-level quantities and surface evidence, not decide sentence meaning.
|
||||
|
||||
ADR-0184 should preserve this:
|
||||
|
||||
- regex may collect numbers, word-numbers, units, cue candidates, names, and punctuation boundaries;
|
||||
- regex must not encode whole problem templates;
|
||||
- semantic rules decide whether a signal becomes a state transition;
|
||||
- verifier/pool decides whether the candidate can commit.
|
||||
|
||||
### 2.2 State machine before arithmetic
|
||||
|
||||
The RB-GSM sketch's strongest idea is the same as ADR-0184:
|
||||
|
||||
```text
|
||||
text -> state builder -> operation queue -> executor
|
||||
```
|
||||
|
||||
CORE should translate this as:
|
||||
|
||||
```text
|
||||
text -> semantic ledger -> GroundedDerivation replay -> verifier / pool
|
||||
```
|
||||
|
||||
The difference is that CORE's operation queue should be a typed, scoped semantic ledger, not a loosely ordered list of calculator commands.
|
||||
|
||||
### 2.3 Cue tables as proposal tables
|
||||
|
||||
The sketch's keyword dictionary is useful, but in CORE these should be cue tables that propose frame candidates:
|
||||
|
||||
```text
|
||||
"has" / "had" -> possible SET_STATE
|
||||
"buys" / "gets" -> possible GAIN
|
||||
"loses" / "spends" -> possible LOSS
|
||||
"more than" -> possible COMPARISON / DIFFERENCE
|
||||
"altogether" -> possible AGGREGATE target
|
||||
"left" -> possible final/net target or loss result cue
|
||||
"each" / "per" -> possible RATE / CONTAINER binding
|
||||
"half" / "twice" -> possible SCALAR / COMPARATIVE
|
||||
```
|
||||
|
||||
They should not directly commit arithmetic.
|
||||
|
||||
### 2.4 Executor remains simple
|
||||
|
||||
Once the semantic ledger is correct, arithmetic execution should stay boring:
|
||||
|
||||
```text
|
||||
SET_STATE -> assign
|
||||
GAIN -> add
|
||||
LOSS -> subtract
|
||||
RATE -> multiply when structurally bound
|
||||
TRANSFER -> subtract from source, add to target
|
||||
DIFFERENCE -> subtract two bound states
|
||||
```
|
||||
|
||||
This matches CORE's preference: make the hard part the reading/proof, not the arithmetic.
|
||||
|
||||
---
|
||||
|
||||
## 3. Claims to reject or downgrade
|
||||
|
||||
### 3.1 Reject unproven 100% dataset coverage claims
|
||||
|
||||
The provided sketch claims that 30–50 rules can solve every GSM8K problem after small tuning. CORE should not encode this as truth.
|
||||
|
||||
Correct CORE posture:
|
||||
|
||||
```text
|
||||
Rule coverage is measured by lanes, not asserted.
|
||||
```
|
||||
|
||||
Any coverage claim must be backed by:
|
||||
|
||||
- train_sample report;
|
||||
- practice report;
|
||||
- confuser probe;
|
||||
- sealed holdout where available;
|
||||
- wrong=0 preservation;
|
||||
- perturbation / paraphrase checks where available.
|
||||
|
||||
### 3.2 Reject last-actor pronoun as a commit rule
|
||||
|
||||
The sketch says a simple last-actor rule works for ambiguous pronouns because GSM8K stories are linear.
|
||||
|
||||
CORE must not use that as a commit rule.
|
||||
|
||||
Allowed:
|
||||
|
||||
```text
|
||||
single active actor + pronoun continuation -> candidate may emit
|
||||
```
|
||||
|
||||
Forbidden:
|
||||
|
||||
```text
|
||||
multiple prior actors + pronoun -> pick most recent and commit
|
||||
```
|
||||
|
||||
Required behavior:
|
||||
|
||||
```text
|
||||
multiple possible antecedents -> refuse / hold until a future safe resolver exists
|
||||
```
|
||||
|
||||
### 3.3 Reject first-matching operation execution
|
||||
|
||||
The sketch's operation queue implies sequential execution of matched operations. CORE should avoid any first-match priority that can hide competing readings.
|
||||
|
||||
Required behavior:
|
||||
|
||||
```text
|
||||
multiple plausible readings -> pooled candidates -> disagreement refuses
|
||||
```
|
||||
|
||||
This preserves ADR-0182's core lesson.
|
||||
|
||||
### 3.4 Reject regex as final logic
|
||||
|
||||
Regex should not decide:
|
||||
|
||||
- final operation;
|
||||
- actor binding;
|
||||
- question target;
|
||||
- temporal target;
|
||||
- rate binding;
|
||||
- comparison direction;
|
||||
- transfer source/target.
|
||||
|
||||
Regex may propose candidates only.
|
||||
|
||||
---
|
||||
|
||||
## 4. Mapping RB-GSM components into CORE
|
||||
|
||||
| RB-GSM sketch | CORE equivalent | Notes |
|
||||
|---|---|---|
|
||||
| `number_pattern` | `generate/derivation/extract.py` | Keep lexeme-level only. |
|
||||
| `name_pattern` | `state/bind.py` entity mention collector | Capitalized names are signals, not proof. |
|
||||
| `keyword_table` | `state/change.py`, `state/rate.py`, `state/compare.py` cue tables | Cue tables emit semantic frame candidates. |
|
||||
| `variables` dict | `SemanticLedger` / entity-owned `StateKey` | Must include entity + unit/scope. |
|
||||
| `op_queue` | ordered `StateTransition` tuple | Transitions remain typed and replayable. |
|
||||
| `execute_operations` | `state/replay.py` -> `GroundedDerivation` | Existing verifier still judges. |
|
||||
| `answer_question` | semantic target binding + pool | Target must bind before commit. |
|
||||
| `readable_steps` | derivation trace / replay explanation | Deterministic, auditable. |
|
||||
|
||||
---
|
||||
|
||||
## 5. Recommended adjustments to ADR-0184 implementation
|
||||
|
||||
### 5.1 Add a cue-table policy
|
||||
|
||||
ADR-0184 S1/S2 should explicitly define cue tables as **proposal tables**, not answer rules.
|
||||
|
||||
Policy:
|
||||
|
||||
```text
|
||||
A cue table entry may create a candidate semantic frame.
|
||||
A cue table entry may not directly commit an arithmetic operation.
|
||||
Every emitted candidate must still pass replay, verification, classification, and pooling.
|
||||
```
|
||||
|
||||
### 5.2 Add a loose-signal collector concept
|
||||
|
||||
S1 can stay helper extraction only, but S2 should prepare the seam for:
|
||||
|
||||
```text
|
||||
SignalBundle:
|
||||
names
|
||||
quantities
|
||||
cue_hits
|
||||
question_markers
|
||||
temporal_markers
|
||||
```
|
||||
|
||||
This does not need to be implemented immediately, but the model should not prevent it.
|
||||
|
||||
### 5.3 Keep accumulation as the first worked example
|
||||
|
||||
Use the Natalia-style pattern from the sketch as a future fixture class, but implement in CORE terms:
|
||||
|
||||
```text
|
||||
Natalia sold 48 clips in April.
|
||||
She sold half as many in May.
|
||||
How many altogether?
|
||||
```
|
||||
|
||||
Expected semantic reading:
|
||||
|
||||
```text
|
||||
SET_STATE(Natalia.clips.April, 48)
|
||||
SCALE_COPY(Natalia.clips.May, Natalia.clips.April, 0.5)
|
||||
TARGET(aggregate Natalia.clips over April+May)
|
||||
```
|
||||
|
||||
Important: this is **not** S1 or S2. It requires scalar-copy and aggregate target support, likely after basic accumulation ledger and semantic target wrapper exist.
|
||||
|
||||
### 5.4 Treat rate examples as S7, not S1
|
||||
|
||||
The Weng babysitting example:
|
||||
|
||||
```text
|
||||
Weng earns $12 an hour. Yesterday she babysat 50 minutes. How much did she earn?
|
||||
```
|
||||
|
||||
requires rate binding and unit conversion. It belongs under ADR-0184 S7 or a later rate/unit sub-ADR, not under S1 helper extraction.
|
||||
|
||||
### 5.5 Add anti-overfit language
|
||||
|
||||
Every semantic-state ADR/sub-ADR should include:
|
||||
|
||||
```text
|
||||
Regex collects signals; it does not license final meaning.
|
||||
Cue dictionaries emit candidates; they do not bypass verifier/pool.
|
||||
New keyword entries require tests showing both positive use and at least one refusal guard.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Revised immediate S1 non-scope
|
||||
|
||||
Even with the RB-GSM input, S1 should remain behavior-equivalent.
|
||||
|
||||
Do not add yet:
|
||||
|
||||
- broad keyword table;
|
||||
- operation queue executor;
|
||||
- scalar-copy / half-as-many logic;
|
||||
- rate conversions;
|
||||
- cross-month aggregation;
|
||||
- all-problem solver loop;
|
||||
- last-actor pronoun resolver;
|
||||
- confidence string or any claim of 100% coverage.
|
||||
|
||||
S1 remains:
|
||||
|
||||
```text
|
||||
extract proven helper logic from accumulate.py
|
||||
-> state/bind.py
|
||||
-> state/change.py
|
||||
-> helper tests
|
||||
-> no behavior change
|
||||
```
|
||||
|
||||
This matters because if S1 expands capability, it becomes impossible to tell whether the new semantic-state seam is safe.
|
||||
|
||||
---
|
||||
|
||||
## 7. Future capability ordering informed by RB-GSM
|
||||
|
||||
The RB-GSM sketch is most useful as a reminder of common GSM8K frame families. The ordering should be:
|
||||
|
||||
1. **Existing accumulation extraction** — already proven; refactor only.
|
||||
2. **Semantic ledger replay** — `SET_STATE`, `GAIN`, `LOSS`.
|
||||
3. **Semantic question target** — final/prior/count/aggregate skeleton.
|
||||
4. **Scalar copy / comparative amount** — half/twice/as-many; Natalia-style cases.
|
||||
5. **Aggregate target over scoped states** — altogether / total / in all across compatible states.
|
||||
6. **Transfer** — source and target entity mutations.
|
||||
7. **Difference questions** — how many more/fewer than.
|
||||
8. **Rate/container binding** — each/per/hour/minutes.
|
||||
9. **Temporal replay** — before/after/originally/finally.
|
||||
10. **Held worlds / DAGs** — quantity reuse and branching.
|
||||
|
||||
This ordering differs slightly from the first ADR-0184 sequence by inserting scalar-copy and scoped aggregation before transfer. That may be higher payoff for GSM8K because many problems ask for totals over derived period/entity states.
|
||||
|
||||
This should be reviewed before coding S3+.
|
||||
|
||||
---
|
||||
|
||||
## 8. New guardrail tests suggested by this input
|
||||
|
||||
Add these once their phase begins:
|
||||
|
||||
### Loose regex must not over-decide
|
||||
|
||||
```text
|
||||
"Alice has 5 apples. Bob has 3 apples. How many apples does Alice have?"
|
||||
```
|
||||
|
||||
A loose name/number extractor may see `Alice`, `Bob`, `5`, `3`, and `apples`, but no rule may sum 5+3 for Alice.
|
||||
|
||||
### Cue hit must not commit alone
|
||||
|
||||
```text
|
||||
"Kate studies for 3 hours and buys 5 pencils. How many pencils?"
|
||||
```
|
||||
|
||||
A `for` cue may be collected, but it must not force multiplication into the pencil target.
|
||||
|
||||
### Last actor is not proof
|
||||
|
||||
```text
|
||||
"Alice has 5 apples. Bob has 3 apples. She buys 2 more apples. How many apples does Alice have?"
|
||||
```
|
||||
|
||||
If gender/antecedent resolution is not proven, refuse.
|
||||
|
||||
### Half-as-many requires source binding
|
||||
|
||||
```text
|
||||
"Natalia sold 48 clips in April. She sold half as many in May. How many altogether?"
|
||||
```
|
||||
|
||||
`half` is not merely `current_result *= 0.5`; it creates a state for May by copying from a bound prior April state, then aggregate target sums April+May.
|
||||
|
||||
### Rate conversion requires dimensional binding
|
||||
|
||||
```text
|
||||
"Weng earns $12 an hour. She babysat 50 minutes. How much did she earn?"
|
||||
```
|
||||
|
||||
`minutes` and `hour` require unit conversion only because the rate denominator is hour and the worked duration is minutes. No global minutes/hour rule should fire without a rate frame.
|
||||
|
||||
---
|
||||
|
||||
## 9. Bottom line
|
||||
|
||||
The RB-GSM sketch strengthens the ADR-0184 direction but does not replace CORE's safety discipline.
|
||||
|
||||
Use:
|
||||
|
||||
```text
|
||||
loose signal collection
|
||||
cue dictionaries
|
||||
state machine
|
||||
operation/transition replay
|
||||
simple arithmetic execution
|
||||
```
|
||||
|
||||
Do not use:
|
||||
|
||||
```text
|
||||
unproven 100% claims
|
||||
last-actor commits
|
||||
first-rule-wins execution
|
||||
regex as final meaning
|
||||
keyword hits that bypass verification
|
||||
```
|
||||
|
||||
CORE's version should be:
|
||||
|
||||
```text
|
||||
loose lexeme signals
|
||||
-> cue-table semantic frame proposals
|
||||
-> scoped state ledger
|
||||
-> GroundedDerivation replay
|
||||
-> existing verifier/classifier
|
||||
-> pooled disagreement/commit eligibility
|
||||
```
|
||||
|
||||
That keeps the useful classic rule-based insight while preserving the project doctrine: deterministic, auditable, refusal-first, and wrong=0-safe.
|
||||
1196
docs/handoff/SEMANTIC-STATE-TRANSITION-BLUEPRINT.md
Normal file
1196
docs/handoff/SEMANTIC-STATE-TRANSITION-BLUEPRINT.md
Normal file
File diff suppressed because it is too large
Load diff
Loading…
Reference in a new issue