core/evals
Shay 7d6760ae5b feat(evals): the fluency lanes report the writer that actually ships (Phase 4)
Phase 4 of the grammar-unification arc, resolved by option (b) of the plan.

The problem: 149 green fluency cases scored `realize_target`, and
`core/cognition/pipeline.py` never calls it -- it calls `realize_semantic`.
`english_fluency_ood` reported 117/117 + 39/39 for a function that does not
speak, for the whole life of the lane.

The plan framed this as "which realizer is better". Reading the source says it
is not a quality question. `render_semantic`'s signature is
`(intent, subject, predicate, obj, secondary, language, root)` -- no `negated`,
no `quantifier`, no `tense`, no `aspect` -- and `realize_semantic` never reads
them off the step. So the serving writer cannot express content the
ArticulationStep is carrying:

    negated=False -> 'Knowledge is defined as opinion.'
    negated=True  -> 'Knowledge is defined as opinion.'

It serves the AFFIRMATIVE of a negated proposition. That is the ADR-0261 §5.1
family, not a fluency defect. Pinned here as a defect; NOT fixed, because
fixing it changes served output and belongs to Shay.

Measured -- identical contract, all seven scored corpora
--------------------------------------------------------
    bucket                                n     realize_target   realize_semantic
    feature-bearing, single node        214          207               49
    no features, multi-node             100          100                3
    no features, single node (CONTROL)   33           33               33
    total                               347          340               85

The control is what makes the rest mean anything. Every corpus hardcodes
IntentTag.UNKNOWN, so "the serving writer scores badly" could have been an
artifact of never giving it a real intent. On the 33 cases carrying nothing it
cannot express, the two are IDENTICAL -- so the gap on the other 314 is the
dropped features and the clause joining, not the intent.

Delivered
---------
- `grammatical_coverage/runner.py` reports `serving_accuracy` beside
  `accuracy`; `english_fluency_ood` delegates to that run_lane and gains it
  for free. The realizer is a parameter now instead of a hardcoded import.
- `tests/test_phase4_realizer_resolution.py` -- the control, the decomposition,
  the negation defect pin, and the §6 evidence.
- The claim is restated at both places it was made: the lane docstring and
  `grammar_roundtrip/contract.md`.

Zero served bytes change. No serving authorization needed.

§6 -- the plan's pre-commitment was wrong, and is corrected rather than
quietly edited
------------------------------------------------------------------------
§6 forks on read_rate: risen => Phase 5 diversity; near-zero => §1.8 graph-model
mismatch => ADR. It pre-committed to the second. Neither is what the
measurement says.

g_read_rate went to 1/293. The unblocked case was blocked by a one-line WRITER
defect (predicate-nominal object agreement, #135), not by §1.8. The other 292:

    no_template_match     289   reader has no SUBJ-VERB-OBJ template at all
    unknown_morphology      2   prepositional objects (reserved_word_in_np)
    unsupported_negation    1   reader has no negated-categorical template

Every one is the reader declining a CONSTRUCTION, not a projection disagreeing
about a graph it parsed. Where a construction is in both inventories the round
trip closes exactly. So the barrier is the OVERLAP of the two construction
inventories, currently one construction wide -- Phase 5's item 1, tractable,
not an ADR-scale model decision.

§1.6 read "uniform no_template_match" as evidence FOR the type mismatch. It is
not: no_template_match is a coverage fact, and the corpus was 289/293 bare
transitives, a construction the reader never claimed to read. The measurement
was mostly reporting the corpus's composition.

And a finding in my own stack
------------------------------
`tests/test_realizer_quantifier_agreement.py` lived ONLY in the `cognition`
suite, which is not on the AGENTS.md pre-push gate. So every pin Phases 3 and 4
added to it -- including the invariant covering all twelve inflection branches
-- ran in NO gate. That is why smoke stayed at 621 across two PRs that added 13
tests between them, and it is the same silent-red shape the smoke list already
calls out for test_adr_index.py.

Registered into `deductive`, which now runs 504 instead of 406.

Mutation
--------
    baseline                                             12 pass
    serving metric computed with realize_target           4 FAIL
    render_semantic GAINS a `negated` parameter           2 FAIL
    realize_semantic delegated to realize_target          6 FAIL

Row 2 matters most: if someone FIXES the negation defect, the pin forces a
deliberate revision instead of passing silently.

Still open, still Shay's: option (a), promoting realize_target to the serving
path. It now has a price tag -- 340/347 over 85/347 and the ability to say
"not", against a move in every surface hash and whatever the Shadow Coherence
Gate ruling in core/cognition/surface_resolution.py was protecting.

[Verification]: in-worktree on CPython 3.12.13 with `uv sync --locked` --
smoke 621, deductive 504 (was 406; +12 new Phase 4 pins, +86 previously
ungated), lane pins 11/11 unchanged, no pin edited.
2026-07-27 11:57:48 -07:00
..
adr_0242_v2_energy_compare feat: close ADR-0241/0242 post-Accept backlog (local-first mastery) 2026-07-16 15:25:43 -07:00
adr_0243_cognitive_lifecycle feat(adr-0243): Phase 4 falsifiability benchmark — metrics eval + CLI dispatcher 2026-07-17 12:59:24 -07:00
adr_0244_gamma_calibration fix(tests): clear the last reds — a machine-specific path and a stale drift pin 2026-07-25 16:31:06 +00:00
adr_0244_identity_gate fix(cognition): pre-stage land repairs after L2/PASSTHROUGH excision 2026-07-20 13:34:59 -07:00
adr_0244_qtop_vacuity feat(adr-0244): prove Q_top vacuity — D4 hollow-gate evidence + annotation upgrade 2026-07-17 14:39:52 -07:00
adr_0246_discrimination feat(adr-0246): Opus audit + §3.7 admit surface + serve wiring + §6.3 discrimination 2026-07-17 22:51:46 -07:00
adr_0246_geometric_suite draft(adr-0246): slice-1 scaffold — §6.1/§6.2 eval suite + malformed-F guard 2026-07-17 22:09:32 -07:00
adr_0246_grounding_feasibility feat(adr-0246): completion — §4.1 records, path serve integration, §11 feasibility (honest NULL), ADR body Proposed 2026-07-17 23:36:41 -07:00
adr_0246_mismatch_diagnostic feat(adr-0246): §3 induced-action primitives (pure, off-serving) 2026-07-17 21:36:57 -07:00
adversarial_identity
analogical_transfer feat(adr-0240/0243): seam S2 — chiral-composed, harness-driven biography write-path 2026-07-18 10:15:38 -07:00
anchor_lens_tour parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
anti_regression feat: strengthen visibility and measurement of CLOSE flywheel proposal review/ratification side (#794) 2026-06-16 19:33:25 -07:00
articulation feat(demo): core demo articulation — discourse-planner spine, end-to-end 2026-05-19 13:41:24 -07:00
articulation_of_status
audio_sensorium feat(adr-0181-p4): audio compiler eval gate lane (sensorium/audio) (#470) 2026-05-29 11:20:31 -07:00
audit_tour chore: Refactor CLI and Governance Anchors (#926) 2026-07-03 12:34:56 -07:00
calibration chore: Refactor CLI and Governance Anchors (#926) 2026-07-03 12:34:56 -07:00
capability_index chore(generalization): Tier S housekeeping — smoke promotion, capability-index entries, promotion sweep (S5) 2026-07-24 17:13:19 -07:00
classical_literature_ood
close_derived_climb chore: Refactor CLI and Governance Anchors (#926) 2026-07-03 12:34:56 -07:00
cognition fix(cognition): expose hash_surface; point the register-matrix pin at it 2026-07-25 14:06:36 +00:00
cold_start_grounding fix(quarantine): drain all 60 quarantined tests — QUARANTINE=∅ (#267) 2026-05-25 11:22:12 -07:00
combined_rate_oracle docs(cmb): lookback review + fix single-agent-attribution hygiene hazard (H3) 2026-06-08 13:27:17 -07:00
compositionality
compound_intent_decomposition feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
comprehension feat: one-hop sound relational entailment (inverse/symmetric) + capability-index lane (#775) 2026-06-15 11:39:41 -07:00
constraint_oracle feat(verified): P1-C — split bound_slots_digest into separate obligation 2026-06-08 17:17:31 -07:00
contemplation_quality feat(W-025): contemplation quality eval lane (ADR-0159) (#286) 2026-05-25 20:38:52 -07:00
contradiction_detection
conversation feat(demo): humanise teaching-grounded surface for layperson display 2026-05-19 14:14:02 -07:00
conversational_thread_coherence feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
cross_domain_transfer
curriculum_loop_closure feat(deduction-serve): Band v6-EX — existential witnesses, decided (ADR-0261) 2026-07-24 14:14:37 -07:00
curriculum_serve feat(curriculum): taught negatives — polarity is read (ADR-0264 R1-R4, R8) 2026-07-26 12:19:19 -07:00
deduction_serve fix(evals): the deduction lane hashes the prose it serves 2026-07-26 19:45:48 -07:00
deductive_logic test(l10): add independent-gold adversarial logic fixtures 2026-06-05 09:08:23 -07:00
demo_composition fix(ci): path-stable env deltas so lane SHA pins stop thrashing 2026-07-14 17:41:04 -07:00
determination_closure feat(determine): idle deductive consolidation — the loop learns from determined facts (Step D) 2026-06-06 12:28:09 -07:00
determination_estimation Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
deterministic_fluency feat(evals): deterministic_fluency lane — six structural predicates 2026-05-19 07:16:44 -07:00
dimensional feat: dimensional-reasoning lane — 3rd diversity-panel domain 2026-06-04 16:38:56 -07:00
discourse_paragraph
domain_contract_validation chore(ci): re-pin drifted lane SHAs + refresh canonical reports (#229) 2026-05-24 14:25:11 -07:00
edge_budget chore: Refactor CLI and Governance Anchors (#926) 2026-07-03 12:34:56 -07:00
elementary_mathematics_ood feat(capability): ADR-0110 promote mathematics_logic to expert_demo (#118) 2026-05-22 12:59:23 -07:00
english_fluency_ood
environment_falsification Add offline witness log importer 2026-06-06 12:37:57 -07:00
event_vision_sensorium Add event vision sensorium lane 2026-06-06 12:37:57 -07:00
fabrication_control chore(ci): re-pin drifted lane SHAs + refresh canonical reports (#229) 2026-05-24 14:25:11 -07:00
field_incidence work done towards furthering comprehension, contemplation, learning, and also began working on fixing some failing tests 2026-06-16 12:27:59 -07:00
flywheel_demo Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
forward_semantic_control Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
foundational_biology_ood
foundational_physics_ood feat: ADR-0111 physics expert-demo promotion (second successful) 2026-05-22 14:37:36 -07:00
frame_verdict_text_cwa feat(frame-verdict): closed-world FrameVerdict substrate — PR-1..4 + hardening (ADR-0222 B4) (#787) 2026-06-16 06:23:03 -07:00
frontier_compare feat(contemplation): land ADR-0080 phase 1 (#119) 2026-05-22 13:10:03 -07:00
generalization feat(evals): implement safe metadata and fail-closed evaluator check for GSM1K 2026-06-23 07:52:32 -07:00
grammar_roundtrip feat(evals): the fluency lanes report the writer that actually ships (Phase 4) 2026-07-27 11:57:48 -07:00
grammatical_coverage feat(evals): the fluency lanes report the writer that actually ships (Phase 4) 2026-07-27 11:57:48 -07:00
gsm8k_math test(reader-arc): pin deterministic tune/measure split BEFORE grammar work 2026-07-18 16:39:04 -07:00
gsm8k_parser_dev feat: add ADR-0125 perturbation suite 2026-05-22 17:12:33 -07:00
hebrew_fluency feat(packs): ADR-0103 — attach hebrew_fluency + koine_greek_fluency lanes to ADR-0102 (#106) 2026-05-22 09:43:46 -07:00
identity_divergence
industry_demos Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
inference_closure feat: ADR-0122 systems_software audit-passed deferred (lane-shape mismatch) 2026-05-22 16:31:59 -07:00
introspection
koine_greek_fluency feat(packs): ADR-0103 — attach hebrew_fluency + koine_greek_fluency lanes to ADR-0102 (#106) 2026-05-22 09:43:46 -07:00
l10_always_on feat(identity): split engine identity from build provenance (ADR-0220 PR C) (#774) 2026-06-15 11:38:04 -07:00
l10_continuity test(l10): W2-R arbitrary-interruption recovery harness (ADR-0219) 2026-06-15 02:45:29 -07:00
lab research(evals): phi separation probe for ADR-0081 follow-up (#57) 2026-05-20 12:34:59 -07:00
learning_arc feat(workbench): add UI catch-up evidence scaffolding (#891) 2026-06-23 10:09:45 -07:00
learning_loop fix(ci): re-pin public_demo lane SHA after showcase content drift (#807) 2026-06-17 15:42:40 -07:00
long_context_cost chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
math_bounded_grammar/v1 feat(ADR-0131.3): bounded-grammar word-problem benchmark — lane PASSED 50/50 (#180) 2026-05-23 11:27:04 -07:00
math_capability_axes chore(evals): refresh stale committed reports 2026-06-03 01:01:50 -07:00
math_expert_claims/v1 docs(claims): ADR-0200 reconciliation — expert claim to audit-passed truth 2026-06-02 10:06:16 -07:00
math_symbolic_equivalence feat(ADR-0131.1.F): frontier-baseline comparison harness for B1 (#178) 2026-05-23 12:14:06 -07:00
math_teaching_corpus/v1 feat(ADR-0131.2.B): B2 teaching-corpus enrichment — load-bearing gate (#177) 2026-05-23 11:29:48 -07:00
miner_loop_closure feat(teaching): implement ADR-0095 — Miner-Sourced Teaching Proposals 2026-05-21 18:18:51 -07:00
monotonic_learning
multi_agent_composition
multi_sentence_response feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
multi_step_reasoning
obligation_2_ood_ratio feat(ADR-0114a.2): OOD-ratio auditor — Obligation #2 wired for B3, ratio=1.00 (#193) 2026-05-23 16:25:28 -07:00
obligation_5_perturbation feat(ADR-0114a.5): reasoning-isolation perturbation suite — Obligation #5 wired for B3, PASSING 130/130 preserving, 68/68 breaking (#191) 2026-05-23 16:07:59 -07:00
obligation_6_depth_curve feat(ADR-0114a.6): depth-curve auditor — Obligation #6 wired for B3 (assertion holds, coverage gap named) (#190) 2026-05-23 16:19:58 -07:00
obligation_8_adversarial feat(ADR-0114a.8): adversarial auditor — Obligation #8 wired, PASSING; surfaces 2 known parser-layer gaps (#192) 2026-05-23 16:11:37 -07:00
obligation_10_pack_provenance feat(ADR-0114a.10): pack-provenance auditor — Obligation #10 wired for B3, PASSING 2026-05-23 15:44:53 -07:00
orthogonality_tour parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
prompt_diversity feat(adr-0085): gloss-aware CAUSE composer — explanation frame from glosses (#70) 2026-05-20 15:55:08 -07:00
proof_carrying_promotion_demo feat(demos): Implement ADR-0218 PR D proof-carrying promotion demo (#927) 2026-07-03 13:01:49 -07:00
proofwriter_owa feat(eval): ProofWriter-OWA refusal-floor lane (B1) — independent oracle, measure-only (#779) 2026-06-15 12:28:20 -07:00
propositional_logic feat(comprehend): propositional-logic comprehension (4th domain, flagship oracle) 2026-06-05 23:24:54 -07:00
provenance
public_demo fix(ci): re-pin public_demo lane — real content drift, not the env flake (S2) 2026-07-24 16:58:05 -07:00
rate_oracle test(rate-oracle): port _canonical_outcome non-vacuous validation to R3 (R3-vac) 2026-06-08 07:38:20 -07:00
realizer_guard chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
refusal_calibration feat(identity): L11 identity continuity — same identity across reboot, not just same bytes 2026-06-05 13:52:57 -07:00
refusal_taxonomy fix: green test-fast suite, consolidate ADR graph under docs/adr, and complete governance cohesion anchors 2026-06-30 17:56:12 -07:00
register_diagnostics feat(evals): per-intent register-firing diagnostic + CI gate + tests (#103) 2026-05-21 07:05:23 -07:00
register_tour fix(ci): re-pin public_demo lane SHA after showcase content drift (#807) 2026-06-17 15:42:40 -07:00
relational feat(reader): add overlaps_event finite-verb reader surface (B3) (#783) 2026-06-15 14:56:09 -07:00
relational_inference/v1 feat(reader): add overlaps_event finite-verb reader surface (B3) (#783) 2026-06-15 14:56:09 -07:00
relational_metric feat(oracle): narrow reverse-solve contract — base of one more/fewer_than (PR-7a) 2026-06-07 06:22:36 -07:00
relational_operator_ablation feat(eval): sealed relational operator ablation lane v1 2026-07-19 21:00:15 -07:00
relational_transitive feat(determine): add transitive strict-order relational inference (#781) 2026-06-15 14:20:30 -07:00
reports Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
results feat: ADR-0086 + ADR-0087 + 100-register catalog — cognition lane closure 2026-05-21 00:08:12 -07:00
reviewer_registry feat(capability): implement ADR-0092 — Reviewer Registry v1 2026-05-21 18:01:24 -07:00
sample_efficiency
self_consistency_over_time feat(runtime+evals): warm-path pack grounding + three long-span lanes 2026-05-19 08:26:38 -07:00
sensorimotor_sensorium Add sensorium eval and governance runway 2026-06-03 20:53:05 -07:00
sensorium Add event vision sensorium lane 2026-06-06 12:37:57 -07:00
set_membership evals: add staged independent gold lanes 2026-06-05 16:03:26 -07:00
setup_oracle feat(comprehension): inverse reader frame — base of a more/fewer-than (PR-7b / R2 C0) 2026-06-07 07:06:26 -07:00
structure_mapping feat(trackb): S1–S4 symbolic SME, selector, pure-S1 coverage gain 2026-07-19 19:49:07 -07:00
syllogism feat(comprehend): multi-word NP chunking under a canonicalization contract 2026-06-05 22:47:34 -07:00
symbolic_logic feat: ADR-0122 systems_software audit-passed deferred (lane-shape mismatch) 2026-05-22 16:31:59 -07:00
teaching_injection_resistance
total_ordering feat(comprehend): multi-word NP chunking under a canonicalization contract 2026-06-05 22:47:34 -07:00
vision_sensorium Add vision evidence and sensorimotor contracts 2026-06-03 20:27:46 -07:00
walkthrough_chain feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
warmed_session_consistency feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
zero_code_domain_acquisition
__init__.py
_parallel.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
baseline_runner.py
CLAIMS.md Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
cognition_cases.jsonl
framework.py feat(evals): ADR-0105 — sealed holdout encryption via age (#108) 2026-05-22 10:09:43 -07:00
generalized_lift_instrument.py chore(adr-0250): ratify Accepted + fold in PR #74 review findings 2026-07-18 15:07:29 -07:00
holdout_runner.py feat(evals): ADR-0105 — sealed holdout encryption via age (#108) 2026-05-22 10:09:43 -07:00
lift_evidence_handoff.py feat(evals): seams S4+S5 — generalized-lift instrument + ports/handoff evidence 2026-07-18 10:34:26 -07:00
logic_cnf_compiler.py feat(adr-0249): P3 structural formula→CNF converter (deduction leg) 2026-07-18 12:40:31 -07:00
metrics.py fix(evals): carry hash_surface on the CaseResult the register matrix actually uses 2026-07-25 14:25:31 +00:00
multi_register_program.py feat(reader-arc): compare compiler-side guards for inverse + chained shapes 2026-07-18 17:14:16 -07:00
parallel.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
run_cognition_eval.py fix(evals): carry hash_surface on the CaseResult the register matrix actually uses 2026-07-25 14:25:31 +00:00
turn_program.py feat(adr-0249): P4 turn-program compiler + chained-relaxation executor 2026-07-18 12:55:31 -07:00
vocab_trigger_instrument.py feat(generalization): vocab-trigger instrument — mechanism-vs-coverage refusal histogram (S3) 2026-07-24 16:54:13 -07:00