core/evals
Shay 66ef4ad07c
feat(brief-11/11B-step-2): lexicon closure — unknown_word 11→5, wrong=0 preserved (#348)
## Summary

Lexicon-entry closure track per Brief 11D recommendation (Candidate A,
sub-PR 1). Adds 12 drain_token lemmas + 1 alias to `en_core_math_v1`.

`unknown_word` row strictly decreases: **11 → 5** (-6 cases moved past
the first-pass vocabulary gap). `wrong == 0` preserved. `correct` does
not move because admitted=0 (the unblocked cases now refuse at
downstream frames — real new work becoming visible, not regression, per
Brief 11 §Gate 1).

## Additions (all category=drain_token)

| Lemma     | Surfaced from              |
|-----------|----------------------------|
| along     | case 0049 (3rd-wave)       |
| animals   | case 0040 (3rd-wave)       |
| decrease  | case 0005                  |
| jacks     | case 0024 (jumping jacks)  |
| length    | case 0006 (3rd-wave)       |
| previous  | case 0006                  |
| reach     | case 0015                  |
| stray     | case 0040                  |
| too       | case 0039                  |
| uphill    | case 0049                  |
| which     | case 0001                  |
| your      | case 0001 (3rd-wave)       |
| weight → weights (alias) | case 0021     |

All classified as `drain_token` (the only category that cannot open a
frame and therefore cannot create wrong admissions per Brief 11
§"correct-count greed" doctrine). Reclassifying any as
accumulation/depletion/transfer verbs would risk wrong>0 by opening a
malformed operation_frame.

## wrong=0 verification

- `assert audit_problem(case_0050)` returns `ReaderRefusal` at
  sentence_index 0 (pinned by `test_hazard_case_0050_remains_refused_pre_frame`)
- 50-case audit: `admitted=0, refused=50` (pinned by
  `test_no_case_admits_after_lexicon_closure`)
- No reader runtime changes; pack-only mutation in a single
  per-category source file
- Manifest checksum unchanged: source-file edit doesn't regenerate the
  compiled `lexicon.jsonl`; loader reads per-category sources for
  alias-aware entries (see `generate/comprehension/lexicon.py:127`)

## Test plan

- 11 new tests in `tests/test_brief_11b_step2_lexicon.py`:
  - 4 pack-additions pinning (categories, provenance, aliases, sort order)
  - 4 reader-effect / hazard tests (admitted=0, case 0050 refused,
    unknown_word row strictly decreased, manifest checksum unchanged)
  - 2 loader-integrity tests (new lemmas + aliases resolve through
    `load_lexicon` → `lookup`)
- 12 existing tests in `tests/test_brief_11b_audit_artifact.py` pass
  (taxonomy counts updated to post-step-2 values)
- 23 existing tests in `tests/test_brief_11_audit.py` pass

## Hard invariants preserved

- `wrong == 0` — no admissions, no frame-opener miscategorisation
- ADR-0166 — no new canonical eval lanes; existing
  `gsm8k_math/train_sample/v1/` artifact updated in-place
- No teaching-store mutation; pack mutation is explicit, single-file,
  reviewed
- Manifest checksum unchanged (compiled lexicon.jsonl byte-identical)

## Follow-up

- 3 lexicon_entry refusals remain (case 0001 '+', case 0040 'sees',
  case 0049 'path'). Not addressed in this PR: '+' is an arithmetic
  literal (would change semantics of drain), 'sees' and 'path' have
  many other downstream barriers. Address with next-bottleneck PR.
- The 6 cases now refusing at later frames feed directly into Brief
  11D Candidate A sub-PR 2 (which bottleneck class to attack next).
2026-05-27 06:06:41 -07:00
..
adversarial_identity
anchor_lens_tour
anti_regression feat(epistemic): Phase 3 state tagging spine (#220) 2026-05-24 11:26:06 -07:00
articulation
articulation_of_status
audit_tour
calibration
classical_literature_ood
cognition
cold_start_grounding fix(quarantine): drain all 60 quarantined tests — QUARANTINE=∅ (#267) 2026-05-25 11:22:12 -07:00
compositionality
compound_intent_decomposition
contemplation_quality feat(W-025): contemplation quality eval lane (ADR-0159) (#286) 2026-05-25 20:38:52 -07:00
contradiction_detection
conversation
conversational_thread_coherence
cross_domain_transfer
curriculum_loop_closure
demo_composition fix(W-015): replace _slerp_toward with rotor-geodesic anchor pull (#255) 2026-05-24 20:05:25 -07:00
deterministic_fluency
discourse_paragraph
domain_contract_validation chore(ci): re-pin drifted lane SHAs + refresh canonical reports (#229) 2026-05-24 14:25:11 -07:00
elementary_mathematics_ood
english_fluency_ood
fabrication_control chore(ci): re-pin drifted lane SHAs + refresh canonical reports (#229) 2026-05-24 14:25:11 -07:00
forward_semantic_control
foundational_biology_ood
foundational_physics_ood
frontier_compare
grammatical_coverage
gsm8k_math feat(brief-11/11B-step-2): lexicon closure — unknown_word 11→5, wrong=0 preserved (#348) 2026-05-27 06:06:41 -07:00
gsm8k_parser_dev feat: add ADR-0125 perturbation suite 2026-05-22 17:12:33 -07:00
hebrew_fluency
identity_divergence
industry_demos
inference_closure feat: ADR-0122 systems_software audit-passed deferred (lane-shape mismatch) 2026-05-22 16:31:59 -07:00
introspection
koine_greek_fluency
lab
learning_arc fix(W-025): polish contemplation-quality eval lane follow-ups (#290) 2026-05-26 09:39:18 -07:00
learning_loop
long_context_cost
math_bounded_grammar/v1 feat(ADR-0131.3): bounded-grammar word-problem benchmark — lane PASSED 50/50 (#180) 2026-05-23 11:27:04 -07:00
math_capability_axes feat(ADR-0163.D.4): question grammar extension — mass nouns, comparatives, pronoun-entity resolution (#310) 2026-05-26 16:19:37 -07:00
math_expert_claims/v1 fix(quarantine): drain all 60 quarantined tests — QUARANTINE=∅ (#267) 2026-05-25 11:22:12 -07:00
math_symbolic_equivalence feat(ADR-0131.1.F): frontier-baseline comparison harness for B1 (#178) 2026-05-23 12:14:06 -07:00
math_teaching_corpus/v1 feat(ADR-0131.2.B): B2 teaching-corpus enrichment — load-bearing gate (#177) 2026-05-23 11:29:48 -07:00
miner_loop_closure
monotonic_learning
multi_agent_composition
multi_sentence_response
multi_step_reasoning
obligation_2_ood_ratio feat(ADR-0114a.2): OOD-ratio auditor — Obligation #2 wired for B3, ratio=1.00 (#193) 2026-05-23 16:25:28 -07:00
obligation_5_perturbation feat(ADR-0114a.5): reasoning-isolation perturbation suite — Obligation #5 wired for B3, PASSING 130/130 preserving, 68/68 breaking (#191) 2026-05-23 16:07:59 -07:00
obligation_6_depth_curve feat(ADR-0114a.6): depth-curve auditor — Obligation #6 wired for B3 (assertion holds, coverage gap named) (#190) 2026-05-23 16:19:58 -07:00
obligation_8_adversarial feat(ADR-0114a.8): adversarial auditor — Obligation #8 wired, PASSING; surfaces 2 known parser-layer gaps (#192) 2026-05-23 16:11:37 -07:00
obligation_10_pack_provenance feat(ADR-0114a.10): pack-provenance auditor — Obligation #10 wired for B3, PASSING 2026-05-23 15:44:53 -07:00
orthogonality_tour
prompt_diversity
provenance
public_demo chore(ratify): accept four Phase C round-2 recognizers (round 2) (#309) 2026-05-26 16:03:01 -07:00
realizer_guard
refusal_calibration
refusal_taxonomy feat(ADR-0163.B.2): four new exemplar corpora — discrete_count_statement, multiplicative_aggregation, currency_amount, plus temporal_aggregation v2 widening (#306) 2026-05-26 14:36:59 -07:00
register_diagnostics
register_tour
reports
results
reviewer_registry
sample_efficiency
self_consistency_over_time
symbolic_logic feat: ADR-0122 systems_software audit-passed deferred (lane-shape mismatch) 2026-05-22 16:31:59 -07:00
teaching_injection_resistance
walkthrough_chain
warmed_session_consistency
zero_code_domain_acquisition
__init__.py
_parallel.py
baseline_runner.py
CLAIMS.md
cognition_cases.jsonl
framework.py
holdout_runner.py
metrics.py
parallel.py
run_cognition_eval.py feat(W-008): L10 Shape B hybrid engine-state persistence (#271) 2026-05-25 11:45:54 -07:00