core/docs/research/vocab-trigger-instrument-2026-07-24.md

Vocab-Trigger Instrument — Baseline Measurement

Tier S3 (generalization-arc-2026-07-24 §6). Implements the measurable test named in docs/handoffs/COMPREHENSION-READER-AUDIT.md (§Q5) — a refusal histogram split mechanism-vs-coverage plus an admissions-per-batch counter — generalized from the math reader (the audit's original subject) to the two composers this arc actually ships: chat/deduction_surface.py and chat/curriculum_surface.py. Implementation: evals/vocab_trigger_instrument.py; tests: tests/test_vocab_trigger_instrument.py (11, registered in the deductive suite).

This document does not propose any pack/curriculum expansion. It reports where the two composers stand today and hands over a rerunnable instrument, per the standing doctrine (project-base-knowledge-pack-expansion-trigger): pack/base-knowledge expansion becomes load-bearing exactly when refusals shift from mechanism scope to vocabulary scope — surface that shift, don't act on it here.

1. The rule (closed, not invented)

A decline is one of three classes:

• mechanism — the reader/gate cannot parse the SHAPE (a closed-grammar gap: sentence_shape_out_of_band, quantifier_out_of_band, ...).
• coverage — a term or relation the ratified content never taught (untaught_vocabulary, out_of_curriculum — the curriculum composer's own naming, taken verbatim, not reinterpreted).
• engine_refused — a fully-read, well-formed argument the ROBDD engine itself declines (inconsistent_premises, out_of_regime_or_malformed from generate/proof_chain/entail.py). Neither a shape gap nor a vocabulary gap; no lexicon or curriculum batch moves this bucket.

This third bucket was not in the brief's two-way framing and was found while building the instrument: an early version misattributed engine-declined arguments (read successfully, but self-contradictory) to whichever reader band happened to be tried last in the fallback cascade — a real bug, not a design choice, caught by test_deduction_refusal_reason_recovers_engine_level_decline pinning "p. Not p. Therefore q." to (band="v1", reason="inconsistent_premises") rather than a v6-EX shape reason it never actually hit.

2. Baseline measurement (2026-07-24, committed lane corpora)

Run: uv run python -m evals.vocab_trigger_instrument. Deterministic — pure over evals/deduction_serve/*/cases.jsonl, evals/curriculum_serve/*/cases.jsonl, and the same production decision paths the lanes score against (test_build_report_is_deterministic pins byte-identical output across repeat runs).

composer	n	admitted	declined	mechanism	coverage	engine_refused
deduction	166	126	40	35	0	5
curriculum	32	26	6	2	4	0
combined	198	152	46	37	4	5

2.1 Deduction: zero coverage-class refusals, and why that's correct

Every one of the 40 declined deduction cases is mechanism (35, closed-shape gates — the largest single bucket is v6_exist:mixed_structure_out_of_band at 11) or engine_refused (5, all inconsistent_premises — 4 pinned Band-v1 contradiction probes plus one natural-English contradiction admitted by Band v2-EN). Zero coverage refusals is the honest count, not a gap in this instrument: none of the seven deduction bands read open vocabulary — they read a small closed set of connective/quantifier words (generate/proof_chain/{english,member,cond_member,verb,exist}.py), so there is no vocabulary axis for them to refuse on. A future band that reads open English prose (rather than a closed relation grammar) would be the first one this instrument could ever show a nonzero coverage count for.

2.2 Curriculum: the coverage axis is real and already exercised

4 of the physics split's 6 declines are coverage (3 untaught_vocabulary, 1 out_of_curriculum); the other 2 are mechanism (question_shape_out_of_band). This is the axis project-curriculum-grounded-serving and S6's volume quantification already describe from the ratified-chain-count side; this instrument confirms it from the refusal side using the SAME production decision path, independently.

3. How to use this for a future batch

uv run python -m evals.vocab_trigger_instrument --report before.json
# ... ratify new curriculum content or widen a band's grammar ...
uv run python -m evals.vocab_trigger_instrument --report after.json
uv run python -m evals.vocab_trigger_instrument --compare before.json --report after.json

The last command prints the admitted-count delta per composer — the falsifiable measurement Option A of the audit's Q5 calls for, generalized: a real batch should move admitted up and the matching declined-class count down. No claim is made here about what that delta should be for any specific batch; that is content strategy, not this instrument's job.

4. Non-goals

• Does not run against live chat traffic — only the committed, sealed lane corpora (adding a live-traffic feed would be a different, larger instrument and is not requested here).
• Does not add a new SHA-pinned CI lane; the report is deterministic and could be pinned later if a gate on this split is wanted, but nothing requested that for Tier S.
• Does not reclassify ambiguous_reading / empty_curriculum as coverage even though they are vocabulary-adjacent — only the two reasons the brief named explicitly are coverage-class (§1).

Relates to project-base-knowledge-pack-expansion-trigger, project-curriculum-grounded-serving, project-generalization-arc.