f6f8ee603f
Replaces the per-pack-aggregate diagnostic landed at 58ac780 with a
per-intent matrix decomposition authored by Codex on a parallel
worktree. Codex's design directly answers the original motivating
question — "which packs' marker pools don't fire on which intent
shapes" — that the aggregate version flattened.
What Codex's version adds over the prior aggregate version:
* **Per (pack × intent × prompt) matrix** — cells decompose by
IntentTag. The C_stance / DEFINITION collapse pattern surfaced
in the widened tour is now directly visible as
matrix[register]["DEFINITION"][*].opening_fired == False.
* **Replayed-variant verification** — every cell records
decorate_surface()'s opening/closing AND asserts the resulting
variant_id matches the runtime's emitted register_variant_id
byte-for-byte. Catches future drift between the replayed
selection and live selection in a single field
(variant_id_matches_runtime / all_replayed_variants_match_runtime).
* **Representative-prompt classification gate** — the companion
test confirms every prompt in REPRESENTATIVE_PROMPTS actually
classifies to its declared IntentTag. If intent classification
drifts, the corpus is invalidated immediately rather than
silently producing meaningless diagnostic output.
* **--fail-on-gap CI mode** — exits 1 when any non-empty marker
bucket never fires across its representative-prompt slice.
Convertible into a CI gate once the deliberate-silent vs
accidental-silent distinction is curated.
* **--register / --intent filters** + **--output PATH** — operator
ergonomics for targeted debugging and report archival.
* **3 pytest cases** — corpus integrity, subset-report shape,
full main()/--output round-trip.
Path: Codex authored at scripts/diagnose_register_firing.py.
Relocated to evals/register_diagnostics/run_firing_diagnostic.py to
match the convention used by evals/register_tour/, anchor_lens_tour/,
orthogonality_tour/, learning_loop/ — measurement artifacts live
under evals/, not scripts/. Test import path adjusted accordingly.
The sys.path bootstrap _REPO_ROOT computation was updated from
.parent.parent to .parents[2] to account for the new path depth.
Verified:
PYTHONPATH=. pytest tests/test_register_firing_diagnostic.py -v
→ 3 passed in 5.39s
PYTHONPATH=. python -m evals.register_diagnostics.run_firing_diagnostic \
--register convivial_v1 --intent DEFINITION --intent CAUSE
→ emits per-cell matrix with variant_id_matches_runtime=True
PYTHONPATH=. python -m evals.register_diagnostics.run_firing_diagnostic \
--register expert_v1 --intent DEFINITION --fail-on-gap
→ exit 0 (expert_v1's empty buckets have non_empty_size=0, so
not a contract gap — that's correct: gap = non-empty bucket
whose entries never fire)
Co-authored-by: Codex <noreply@openai.com>
63 lines
2 KiB
Python
63 lines
2 KiB
Python
"""Register marker-firing diagnostic script."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
|
|
from generate.intent import IntentTag, classify_intent
|
|
from evals.register_diagnostics.run_firing_diagnostic import (
|
|
REPRESENTATIVE_PROMPTS,
|
|
build_report,
|
|
main,
|
|
)
|
|
|
|
|
|
def test_representative_prompts_classify_to_declared_intents():
|
|
for intent, prompts in REPRESENTATIVE_PROMPTS.items():
|
|
assert len(prompts) == 3
|
|
for prompt in prompts:
|
|
assert classify_intent(prompt).tag is intent
|
|
|
|
|
|
def test_build_report_records_marker_engagement_for_register_subset():
|
|
report = build_report(
|
|
register_ids=("convivial_v1",),
|
|
intents=(IntentTag.DEFINITION, IntentTag.CAUSE),
|
|
)
|
|
|
|
assert report["diagnostic"] == "register_marker_firing"
|
|
assert report["registers"] == ["convivial_v1"]
|
|
assert report["intents"] == ["DEFINITION", "CAUSE"]
|
|
assert report["all_replayed_variants_match_runtime"] is True
|
|
|
|
cells = report["matrix"]["convivial_v1"]["DEFINITION"]
|
|
assert len(cells) == 3
|
|
assert all("opening_fired" in cell for cell in cells)
|
|
assert all("closing_fired" in cell for cell in cells)
|
|
assert any(cell["opening_fired"] for cell in cells)
|
|
|
|
summaries = {
|
|
summary["intent"]: summary
|
|
for summary in report["summaries"]
|
|
if summary["register_id"] == "convivial_v1"
|
|
}
|
|
assert summaries["DEFINITION"]["openings"]["non_empty_size"] > 0
|
|
assert summaries["DEFINITION"]["openings"]["fire_count"] > 0
|
|
|
|
|
|
def test_main_can_write_json_report(tmp_path):
|
|
output = tmp_path / "register_firing.json"
|
|
rc = main([
|
|
"--register",
|
|
"default_neutral_v1",
|
|
"--intent",
|
|
"DEFINITION",
|
|
"--output",
|
|
str(output),
|
|
])
|
|
|
|
assert rc == 0
|
|
payload = json.loads(output.read_text(encoding="utf-8"))
|
|
assert payload["registers"] == ["default_neutral_v1"]
|
|
assert payload["intents"] == ["DEFINITION"]
|
|
assert payload["matrix"]["default_neutral_v1"]["DEFINITION"]
|