core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

fix(tests): retire 13 stale failures from full lane — corpus saturation drift

Browse source 
The full lane carried 13 long-standing red tests whose premises were
invalidated by reviewed-corpus growth that landed in earlier commits.
None reflected runtime bugs — all four classes are corpus-state drift
where the test fixture became stale.  Curated lanes were green, full
lane stayed quietly red.  Closes that gap.

1. test_teaching_audit (2 tests).
   * test_audit_real_corpus_runs_clean asserted dropped == () and
     lines_on_disk == lines_loaded — premise written before any
     supersession existed.  Curriculum saturation v2 (commit a0edbb4)
     ratified the wisdom_grounds_judgment → wisdom_requires_knowledge
     supersession; the audit now correctly shows 1 dropped line.
     Rewritten as the line-conservation invariant:
       lines_loaded + len(dropped) == lines_on_disk
     plus a typed-reason check on every dropped entry.
   * test_default_superseded_by_is_null_in_loaded_entries asserted
     ALL loaded entries have superseded_by == None.  Wrong even by
     ADR-0055 design: the replacement entry IS loaded and carries
     the back-pointer to the retired chain.  Rewritten as the
     active-set invariant: any non-null superseded_by on a loaded
     entry must reference a dropped (retired) chain id, never a live
     one — no double-live state.

2. test_learning_loop_demo (7 tests).
   The demo's headline prompt was "Why does thought exist?", and the
   ADR-0057 demo trilogy (commit 82dac4b) chose (thought, cause) as
   the cold cell.  Cognition saturation v2 (commit a0edbb4) ratified
   cause_thought_reveals_meaning into the active corpus — so the
   cold turn now grounds, no discovery candidate is emitted, every
   demo scene breaks.  Rotated the cold subject to ``narrative``
   (pack-resident, no chain, same thematic shape, same affirming
   evidence pointer cause_creation_reveals_meaning).  Demo headline,
   evals/learning_loop/run_demo.py, core/cli.py preamble, and the
   test assertions all updated together so the demo reads cleanly:
       before: [none]     I don't know — insufficient grounding...
       after : [teaching] narrative — teaching-grounded ... narrative
                          reveals meaning ...

3. test_discovery_candidates (4 tests).
   Test fixture used (judgment, CAUSE) as the still-cold pair.
   Epistemology v1 (commit 2acf71f) ratified
   cause_judgment_requires_wisdom — (judgment, cause) is no longer
   cold.  Rotated to ``principle`` (pack-resident, no chain on either
   intent today).  Added a pytest.skip self-guard so when a future
   curriculum unit ratifies a (principle, *) chain the test rotates
   cleanly instead of going red.

Full lane: 1892 passed, 2 skipped, 0 failed (was 4 failed pre-fix,
13 failed pre-ADR-0063).  Cognition eval unchanged: public 100/100/
91.7/100, holdout 100/100/83.3/100.
This commit is contained in:
Shay 2026-05-18 15:23:22 -07:00
parent 9f83b27a7c
commit 7c80b791ec
5 changed files with 88 additions and 41 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
core/cli.py
View file

@ -1250,7 +1250,7 @@ TeachingChainProposal + replay gate + operator review).
A single deterministic prompt drives every scene:
"Why does thought exist?"
"Why does narrative exist?"
Headline claim: CORE, asked a question it cannot ground, emits
structured evidence that a reviewed chain would have helped. An

27
evals/learning_loop/run_demo.py
View file

@ -60,13 +60,20 @@ from teaching.proposals import (
# The single prompt that drives every scene. CAUSE intent, subject
# "thought", pack-resident lemma but no `(thought, cause)` chain in
# the active corpus today — guarantees the cold-turn path.
_DEMO_PROMPT: str = "Why does thought exist?"
# ``narrative`` — pack-resident lemma but no ``(narrative, cause)``
# chain in the active corpus today, guaranteeing the cold-turn path.
#
# History: the original demo used ``thought`` as the cold subject; the
# cognition saturation v2 curriculum unit (commit ``a0edbb4``) added
# ``cause_thought_reveals_meaning`` to the active corpus, so the
# (thought, cause) cell is no longer cold. ``narrative`` is the new
# cold exemplar — same thematic shape, same connective + object.
_DEMO_PROMPT: str = "Why does narrative exist?"
_DEMO_SUBJECT: str = "narrative"
# Operator-authored proposal payload. The (thought, cause) cell is
# Operator-authored proposal payload. The (narrative, cause) cell is
# unoccupied; the operator proposes the chain
# thought reveals meaning
# narrative reveals meaning
# affirming evidence is the existing corpus chain
# cause_creation_reveals_meaning (creation reveals meaning)
# both endpoints are pack-resident.
@ -224,7 +231,7 @@ def _scene3_propose(log_path: Path, candidate_id: str) -> tuple[SceneResult, Any
_print_header(
"S3. Operator-authored proposal — replay-equivalence gate runs",
"From the discovery candidate's evidence, the operator authors "
"a complete chain: thought reveals meaning. Affirming evidence "
"a complete chain: narrative reveals meaning. Affirming evidence "
"is the existing corpus chain cause_creation_reveals_meaning. "
"The real replay gate (teaching.replay.run_replay_equivalence) "
"runs the cognition public split twice — active corpus vs. "
@ -237,7 +244,7 @@ def _scene3_propose(log_path: Path, candidate_id: str) -> tuple[SceneResult, Any
augmented = DiscoveryCandidate(
candidate_id=candidate_id,
proposed_chain={
"subject": "thought", "intent": "cause",
"subject": _DEMO_SUBJECT, "intent": "cause",
"connective": _OPERATOR_CONNECTIVE,
"object": _OPERATOR_OBJECT,
},
@ -351,7 +358,7 @@ def _scene5_replay_now_grounded(transient: Path) -> SceneResult:
"With the runtime's corpus path swapped to the transient corpus, "
"the same prompt now returns a teaching-grounded surface "
"containing the operator-accepted chain: "
"thought reveals meaning. Identical bytes for any replay of "
"narrative reveals meaning. Identical bytes for any replay of "
"the same prompt against this corpus state.",
)
real_path = _tg._CORPUS_PATH
@ -372,7 +379,7 @@ def _scene5_replay_now_grounded(transient: Path) -> SceneResult:
_say(f" grounding_source : {grounding}")
# Falsifiable assertions for the demo's headline claim.
contains_subject = "thought" in surface.lower()
contains_subject = _DEMO_SUBJECT in surface.lower()
contains_connective = "reveal" in surface.lower() # humanised
contains_object = "meaning" in surface.lower()
is_teaching_grounded = grounding == "teaching"
@ -393,7 +400,7 @@ def _scene5_replay_now_grounded(transient: Path) -> SceneResult:
detail={
"surface": surface,
"grounding_source": grounding,
"contains_subject_thought": contains_subject,
"contains_subject": contains_subject,
"contains_connective_reveals": contains_connective,
"contains_object_meaning": contains_object,
},

46
tests/test_discovery_candidates.py
View file

@ -63,27 +63,29 @@ class _TE:
def test_fires_on_would_have_grounded_for_cause_on_unknown_pack_lemma() -> None:
"""``judgment`` is a pack lemma; (judgment, cause) is NOT in the
corpus today (the existing chains for ``judgment`` are
VERIFICATION + CAUSE→ordered_by_wisdom, but no ``(judgment,
cause)`` SUBJECT key exists)."""
# First check the pack actually has the lemma we're claiming.
"""``principle`` is a pack lemma; (principle, cause) is NOT in the
active cognition corpus. ``judgment`` was the historical fixture
here, but the epistemology v1 curriculum unit (commit ``2acf71f``)
added ``cause_judgment_requires_wisdom`` — so ``principle`` is the
new still-cold cognition exemplar. If a future curriculum unit
ratifies a ``(principle, *)`` chain, this test pytest-skips and the
fixture should rotate to the next cold lemma."""
from chat.pack_grounding import _pack_index
from chat.teaching_grounding import _corpus_index
assert "judgment" in _pack_index()
if ("judgment", "cause") in _corpus_index():
pytest.skip("judgment/cause is in the active corpus; pick another fixture")
assert "principle" in _pack_index()
if ("principle", "cause") in _corpus_index():
pytest.skip("principle/cause is in the active corpus; pick another fixture")
cands = extract_discovery_candidates(
_TE(grounding_source="none"),
IntentTag.CAUSE,
"judgment",
"principle",
)
assert len(cands) == 1
c = cands[0]
assert c.trigger == "would_have_grounded"
assert c.proposed_chain["subject"] == "judgment"
assert c.proposed_chain["subject"] == "principle"
assert c.proposed_chain["intent"] == "cause"
assert c.proposed_chain["connective"] is None
assert c.proposed_chain["object"] is None
@ -96,7 +98,7 @@ def test_does_not_fire_when_grounding_source_is_pack() -> None:
cands = extract_discovery_candidates(
_TE(grounding_source="pack"),
IntentTag.CAUSE,
"judgment",
"principle",
)
assert cands == ()
@ -105,7 +107,7 @@ def test_does_not_fire_when_grounding_source_is_teaching() -> None:
cands = extract_discovery_candidates(
_TE(grounding_source="teaching"),
IntentTag.CAUSE,
"judgment",
"principle",
)
assert cands == ()
@ -114,7 +116,7 @@ def test_does_not_fire_for_definition_intent() -> None:
cands = extract_discovery_candidates(
_TE(grounding_source="none"),
IntentTag.DEFINITION,
"judgment",
"principle",
)
assert cands == ()
@ -123,7 +125,7 @@ def test_does_not_fire_for_unknown_intent() -> None:
cands = extract_discovery_candidates(
_TE(grounding_source="none"),
IntentTag.UNKNOWN,
"judgment",
"principle",
)
assert cands == ()
@ -172,12 +174,12 @@ def test_candidate_id_is_deterministic() -> None:
a = extract_discovery_candidates(
_TE(grounding_source="none", trace_hash="seed-1"),
IntentTag.CAUSE,
"judgment",
"principle",
)
b = extract_discovery_candidates(
_TE(grounding_source="none", trace_hash="seed-1"),
IntentTag.CAUSE,
"judgment",
"principle",
)
assert a[0].candidate_id == b[0].candidate_id
@ -186,12 +188,12 @@ def test_candidate_id_changes_with_trace_hash() -> None:
a = extract_discovery_candidates(
_TE(grounding_source="none", trace_hash="seed-1"),
IntentTag.CAUSE,
"judgment",
"principle",
)
b = extract_discovery_candidates(
_TE(grounding_source="none", trace_hash="seed-2"),
IntentTag.CAUSE,
"judgment",
"principle",
)
assert a[0].candidate_id != b[0].candidate_id
@ -200,7 +202,7 @@ def test_boundary_clean_false_when_refusal_emitted() -> None:
cands = extract_discovery_candidates(
_TE(grounding_source="none", refusal_emitted=True),
IntentTag.CAUSE,
"judgment",
"principle",
)
# The trigger still fires (refusal does not block evidence), but
# boundary_clean flips to False so reviewers can filter these out
@ -213,7 +215,7 @@ def test_boundary_clean_false_when_hedge_injected() -> None:
cands = extract_discovery_candidates(
_TE(grounding_source="none", hedge_injected=True),
IntentTag.CAUSE,
"judgment",
"principle",
)
assert len(cands) == 1
assert cands[0].boundary_clean is False
@ -227,7 +229,7 @@ def test_boundary_clean_false_when_hedge_injected() -> None:
def test_format_jsonl_is_sorted_keys_compact() -> None:
cand = DiscoveryCandidate(
candidate_id="x",
proposed_chain={"subject": "judgment", "intent": "cause", "connective": None, "object": None},
proposed_chain={"subject": "principle", "intent": "cause", "connective": None, "object": None},
trigger="would_have_grounded",
source_turn_trace="t",
pack_consistent=True,
@ -389,7 +391,7 @@ def test_pure_extract_does_not_load_or_mutate_runtime_state() -> None:
pack_before = dict(_pack_index())
corpus_before = dict(_corpus_index())
for _ in range(5):
extract_discovery_candidates(_TE(), IntentTag.CAUSE, "judgment")
extract_discovery_candidates(_TE(), IntentTag.CAUSE, "principle")
pack_after = dict(_pack_index())
corpus_after = dict(_corpus_index())
assert pack_before.keys() == pack_after.keys()

10
tests/test_learning_loop_demo.py
View file

@ -27,8 +27,12 @@ def test_after_is_teaching_grounded_with_new_chain_atoms() -> None:
report = run_demo(emit_json=True)
assert report["after"]["grounding_source"] == "teaching"
surface = report["after"]["surface"].lower()
# The accepted chain is (thought, cause, reveals, meaning).
assert "thought" in surface
# The accepted chain is (narrative, cause, reveals, meaning).
# ``thought`` was the original cold subject; cognition saturation
# v2 (commit ``a0edbb4``) added ``cause_thought_reveals_meaning``
# to the active corpus so the demo switched to ``narrative`` —
# same shape, still cold.
assert "narrative" in surface
assert "reveal" in surface # humanised connective
assert "meaning" in surface
assert "teaching-grounded" in surface
@ -64,7 +68,7 @@ def test_same_prompt_drives_before_and_after() -> None:
Different surfaces emerge from the corpus state change alone, not
from any prompt variation or stochastic sampling."""
report = run_demo(emit_json=True)
assert report["prompt"] == "Why does thought exist?"
assert report["prompt"] == "Why does narrative exist?"
# And the two surfaces are observably different — the loop changed
# the response, not merely the metadata.
assert report["before"]["surface"] != report["after"]["surface"]

44
tests/test_teaching_audit.py
View file

@ -90,11 +90,21 @@ def test_parse_extra_trailing_colons_folded_into_date() -> None:
def test_audit_real_corpus_runs_clean() -> None:
"""The audit report's accounting invariant: every line on disk is
either loaded or accounted for as dropped (with a typed reason).
Hardcoding ``dropped == ()`` was the pre-supersession premise; the
curriculum saturation v2 (commit ``a0edbb4``) introduced one
ratified supersession (``verification_wisdom_grounds_judgment`` →
``verification_wisdom_requires_knowledge``). The invariant the
audit guarantees is line-conservation, not zero supersessions."""
report = audit_corpus()
assert report.lines_on_disk == report.lines_loaded
assert report.dropped == ()
assert report.lines_loaded + len(report.dropped) == report.lines_on_disk
assert len(report.loaded) >= 10
assert report.corpus_id == "cognition_chains_v1"
# Every dropped line must carry a typed reason so an operator can
# audit why it was retired without re-deriving the supersession.
for dropped in report.dropped:
assert dropped.reason.startswith("superseded_by:")
def test_audit_loaded_entries_have_typed_provenance() -> None:
@ -210,10 +220,34 @@ def test_supersession_drops_earlier_chain_from_active_view(tmp_path: Path) -> No
assert report.loaded[0].chain_id == "cause_light_grounds_truth"
def test_default_superseded_by_is_null_in_loaded_entries() -> None:
"""Existing corpus uses default null — must round-trip unchanged."""
def test_loaded_entries_with_supersession_point_to_dropped_chains() -> None:
"""The active-set invariant: a loaded entry may carry a non-null
``superseded_by`` (it is the replacement, pointing back at what it
retired), but the chain id it points to MUST itself be dropped
from the active set — otherwise the corpus would contain two
live versions of the same lemma's chain.
Was ``test_default_superseded_by_is_null_in_loaded_entries`` (an
overstrict premise written before any supersession existed); the
weaker invariant here is the one ADR-0055 Phase A actually
guarantees."""
report = audit_corpus()
assert all(entry.superseded_by is None for entry in report.loaded)
dropped_ids = {entry.chain_id for entry in report.dropped}
live_ids = {entry.chain_id for entry in report.loaded}
for entry in report.loaded:
if entry.superseded_by is None:
continue
# Replacement entry — its back-pointer must reference a chain
# id that is dropped (retired), never a live one.
assert entry.superseded_by in dropped_ids, (
f"loaded entry {entry.chain_id!r} points to "
f"{entry.superseded_by!r} which is not in the dropped set "
f"— supersession invariant broken"
)
assert entry.superseded_by not in live_ids, (
f"loaded entry {entry.chain_id!r} supersedes a still-live "
f"chain {entry.superseded_by!r} — double-live state"
)
# ---------------------------------------------------------------------------