core/evals/prompt_diversity/contract.md

Prompt-Diversity Eval Lane — Contract

Lane: prompt_diversity Version: v1 (proposed) Created: 2026-05-20 Companion to: ADR-0084 (Definitional Layer for Lexicon Packs)

---

What this lane measures

Every other eval lane in this repo measures one of two things:

• a single architectural property (versor closure, anchor-lens engagement, register variation), against a small fixture, or
• end-to-end cognition correctness against ~13 prompts rooted on the same handful of subjects (light, truth, knowledge, memory, wisdom).

We've been overfitting to that fixture without admitting it. Every ADR since 0048 has used the same demo prompt — "Why does light exist?" — to claim surface lift. The system's response to that prompt has gotten visibly longer; we do not actually know what it does on prompts of different shape, sophistication, or domain.

This lane measures how surface quality and grounding generalize across question types, not just on the canonical chain-walk fixture.

Axes the suite varies

Axis	Levels
Question shape	definition, cause/why, comparison, narrative ("tell me about X"), procedure ("how do I X"), recall ("do you remember X"), correction ("no, X is Y"), verification ("does X require Y"), unknown-intent fallback
Sophistication	bare lemma ("light?"); simple full question ("what is light?"); multi-clause question ("what does light have to do with knowledge?"); embedded clause ("when you say X reveals Y, do you mean Z?")
Domain	cognition, kinship (en_core_relations_v2), cross-pack composition (cognition × kinship), OOV (deliberately unknown lemma), adversarial (ambiguous subject, contradiction, identity probe)
Surface expectation	propositional answer; explanation; sequence; comparison-contrast; honest refusal; honest "I don't know"; honest "no session evidence yet"

The cross-product is intentionally not enumerated. The suite picks ~50 cases that cover the matrix at one or two cases per cell.

Scoring rubric

Two metrics carried over from the cognition lane:

Metric	Definition
intent_accuracy	Fraction of cases with correct intent classification.
versor_closure_rate	Fraction with versor_condition < 1e-6.

Three new metrics specific to this lane:

Metric	Definition
response_shape_fit	Fraction of cases where the surface's structural shape matches the question shape. Definition → noun-phrase or copular sentence. Cause/why → explanation (cause-marker or "because"-style or definition-driven). Procedure → ordered sequence. Comparison → two-subject contrast. Refusal/unknown → honest disclosure. Measured by a per-case expected_shape field in the JSONL plus a small per-shape classifier in the runner.
audit_in_surface_rate	Fraction of cases whose surface string contains audit metadata that should belong in telemetry (e.g. teaching-grounded (, No session evidence yet., cognition.X; logos.Y). Lower is better. This metric exists explicitly to quantify the leak that ADR-0085 / a future "surface vs envelope" ADR will close. Today this rate is essentially 100% — the suite establishes a baseline so progress is measurable.
gloss_quote_rate	Fraction of cases whose surface visibly draws from a pack gloss (post-ADR-0084) rather than only from semantic_domains tags. v1: 0% (composer is unchanged in ADR-0084). The metric is in place so ADR-0085's lift is quantifiable.

Pass criteria

v1 has no pass thresholds beyond versor_closure_rate == 1.00. The lane's job at v1 is to establish a baseline distribution across the matrix. Pass thresholds get set in v2 after one full pass through ADR-0084 → 0085 → 0086 has run and we know which axes are actually moveable.

This is deliberate: setting a threshold against today's baseline would just freeze a fixture. The point is the distribution, not the score.

What this lane does NOT measure

• Naturalness or fluency of language (no LLM judge — we have no ground truth for "natural-sounding" and refuse to mint one).
• Factual correctness of the corpus (that's the ratification pipeline's job).
• Performance / latency (that's the bench's job).
• Cross-provider comparison (that's frontier_compare/'s job).

Categories tested (v1 case file)

definition_simple        — "What is X?" for X in {definition, evidence, recall, kinship, parent}
definition_multi_clause  — "When you say X is Y, do you mean Z?"
cause_simple             — "Why does X exist?" for X in {light, knowledge, family}
cause_multi_clause       — "Why does X imply Y if Y is itself the result of Z?"
comparison_simple        — "Compare X and Y"
comparison_cross_pack    — "How does kinship relate to knowledge?"
narrative_simple         — "Tell me about X"
narrative_multi_subject  — "Tell me how X, Y, and Z relate"
procedure_simple         — "How do I verify X?"
procedure_unknown_subject— "How do I verify zorblax?"
recall_session_empty     — "What was the last thing I said about X?"
correction_simple        — "No, X is actually Y"
correction_indirect      — "I think you misunderstood — let me try again"
verification_simple      — "Does X require Y?"
verification_double_neg  — "Doesn't X not require Y?"
unknown_intent_fallback  — "Mhm." / "..." / "Hmm interesting"
oov_single_word          — A nonsense lemma
oov_in_real_question     — "What is the role of {nonsense} in epistemology?"
adversarial_ambiguous    — "Set" / "Bank" / "Lead" (homographs)
adversarial_identity     — "You're not actually intelligent, are you?"
adversarial_contradiction— "I know X reveals truth, but you said X hides truth"

Target: ~50 cases. Each case carries id, category, question_shape, sophistication, domain, prompt, expected_intent, expected_shape, expected_terms (optional), requires_versor_closure.

Runner

runner.py reuses the existing evals._parallel.run_cases_parallel worker pool and evals.framework plumbing. Reports to evals/prompt_diversity/results/v1_{public,dev}_{timestamp}.json.

How to run

core eval prompt-diversity
# or
python -m evals.prompt_diversity.runner

How to read the output

JSON report with the five metrics above plus per-case breakdowns grouped by (question_shape, sophistication, domain). The audit_in_surface_rate and gloss_quote_rate distributions tell you where surface quality lives today vs. where ADR-0085 will move it.

When it has failed and why

This lane is new. The expected v1 state:

• intent_accuracy: ~70–85% (current intent classifier was trained on cognition-shaped prompts; cross-pack and adversarial cases will dip).
• versor_closure_rate: 1.00 (the algebra invariant should hold for every case the pipeline accepts).
• response_shape_fit: low — most cases will fall back to the chain-walk shape regardless of question shape.
• audit_in_surface_rate: ~100% — every surface today carries trust-boundary text in-band.
• gloss_quote_rate: 0% — composer doesn't know about glosses yet.

The v1 numbers ARE the baseline. The lane fails its purpose only if the distribution looks identical to the cognition lane (i.e. the suite isn't actually diverse).

Cross-references

• ADR-0084 — definitional layer this suite is calibrated to.
• Future ADR-0085 — gloss-aware composer; expected to move gloss_quote_rate and response_shape_fit upward.
• Future "surface-vs-envelope" ADR — expected to drop audit_in_surface_rate toward zero by routing trust-boundary text to telemetry.
• evals/cognition/contract.md — the lane this one extends across axes.