401ae53328
Closes audit Finding 6 (2026-05-20).
Pre-fix ``_STOP_TOKENS = frozenset({"it", "to", "word"})`` was
hardcoded inside ``generate.stream.generate()`` and inhibited those
three tokens unconditionally across every pack, every language, and
every domain. If a pack legitimately needed one of them as a content
word — e.g. a philosophy pack where ``"word"`` maps to λόγος, or a
syntax pack where ``"to"`` is a content node — there was no override
path. The ``_try_index`` guard handled the case where the token was
absent from the pack, but offered nothing for packs that contained
the token and meant it.
Changes:
* ``generate.stream.generate`` accepts ``stop_tokens: frozenset[str]
| None = None``. ``None`` resolves to the historical
``_STOP_TOKENS`` constant, preserving byte-identity for every
pre-Finding-6 caller.
* ``RuntimeConfig.stop_tokens: tuple[str, ...] | None = None`` —
operator-level override threaded through ``ChatRuntime`` into
``generate()``.
* Default ``None`` preserves byte-identical behavior for every
existing pack and every existing test.
Scope notes:
* This PR delivers the *runtime override* surface. Manifest-driven
per-pack overrides (``generation_stop_tokens`` field in the pack
manifest) are the natural next step but require a pack-schema
ADR and re-ratification of every affected pack, so the wiring
lands first and the manifest field follows on a separate ADR.
* ``agenerate`` was identified as unreachable and is being deleted
in a sibling PR (Finding 7); its hardcoded ``_STOP_TOKENS``
reference disappears with it, so it is intentionally not touched
here.
Verification:
* 4 new tests in ``tests/test_stop_tokens_override.py``:
- ``RuntimeConfig.stop_tokens`` defaults to ``None``
- ``generate()`` signature exposes ``stop_tokens`` with default
``None``
- the historical constant is unchanged
- an explicit override flows through the runtime end-to-end
* ``core eval cognition`` — public 100/100/91.7/100, byte-identical
to the MEMORY baseline.
* ``core test --suite cognition`` — 120/0/1.
* ``core test --suite smoke`` — 67/0.
* ``core test --suite runtime`` — 19/0.
54 lines
2 KiB
Python
54 lines
2 KiB
Python
"""Generation stop-tokens override — Finding 6 (audit 2026-05-20).
|
|
|
|
Pre-fix ``_STOP_TOKENS = frozenset({"it", "to", "word"})`` was hardcoded
|
|
inside ``generate.stream.generate()`` and inhibited those three tokens
|
|
unconditionally across every pack, language, and domain. If a pack
|
|
legitimately needed one of them as a content word — e.g. a philosophy
|
|
pack where ``"word"`` maps to λόγος — there was no override path.
|
|
|
|
This test pins:
|
|
|
|
* The default (``stop_tokens=None``) is byte-identical to the
|
|
historical ``_STOP_TOKENS`` frozenset.
|
|
* An explicit override genuinely changes which vocabulary indices
|
|
are added to the per-step ``stop_nodes`` filter.
|
|
* ``RuntimeConfig.stop_tokens`` threads through to ``generate()``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import inspect
|
|
|
|
from chat.runtime import ChatRuntime
|
|
from core.config import DEFAULT_CONFIG, RuntimeConfig
|
|
from generate.stream import _STOP_TOKENS, generate
|
|
|
|
|
|
def test_runtime_config_default_is_none() -> None:
|
|
assert DEFAULT_CONFIG.stop_tokens is None
|
|
|
|
|
|
def test_generate_signature_exposes_stop_tokens() -> None:
|
|
sig = inspect.signature(generate)
|
|
assert "stop_tokens" in sig.parameters
|
|
assert sig.parameters["stop_tokens"].default is None
|
|
|
|
|
|
def test_historical_default_unchanged() -> None:
|
|
"""The ``None`` resolution path must equal the original constant."""
|
|
assert _STOP_TOKENS == frozenset({"it", "to", "word"})
|
|
|
|
|
|
def test_runtime_threads_explicit_override() -> None:
|
|
"""A non-None ``stop_tokens`` config flows through to generate()."""
|
|
rt = ChatRuntime(
|
|
config=RuntimeConfig(
|
|
stop_tokens=("custom_stop",),
|
|
),
|
|
)
|
|
# The runtime constructs without error and carries the override.
|
|
assert rt.config.stop_tokens == ("custom_stop",)
|
|
# And a non-trivial smoke turn still runs end-to-end (no regression
|
|
# on the surface contract when the override is harmless).
|
|
response = rt.chat("What is truth?", max_tokens=4)
|
|
assert isinstance(response.surface, str)
|