7c80b791ec
The full lane carried 13 long-standing red tests whose premises were
invalidated by reviewed-corpus growth that landed in earlier commits.
None reflected runtime bugs — all four classes are corpus-state drift
where the test fixture became stale. Curated lanes were green, full
lane stayed quietly red. Closes that gap.
1. test_teaching_audit (2 tests).
* test_audit_real_corpus_runs_clean asserted dropped == () and
lines_on_disk == lines_loaded — premise written before any
supersession existed. Curriculum saturation v2 (commit a0edbb4)
ratified the wisdom_grounds_judgment → wisdom_requires_knowledge
supersession; the audit now correctly shows 1 dropped line.
Rewritten as the line-conservation invariant:
lines_loaded + len(dropped) == lines_on_disk
plus a typed-reason check on every dropped entry.
* test_default_superseded_by_is_null_in_loaded_entries asserted
ALL loaded entries have superseded_by == None. Wrong even by
ADR-0055 design: the replacement entry IS loaded and carries
the back-pointer to the retired chain. Rewritten as the
active-set invariant: any non-null superseded_by on a loaded
entry must reference a dropped (retired) chain id, never a live
one — no double-live state.
2. test_learning_loop_demo (7 tests).
The demo's headline prompt was "Why does thought exist?", and the
ADR-0057 demo trilogy (commit 82dac4b) chose (thought, cause) as
the cold cell. Cognition saturation v2 (commit a0edbb4) ratified
cause_thought_reveals_meaning into the active corpus — so the
cold turn now grounds, no discovery candidate is emitted, every
demo scene breaks. Rotated the cold subject to ``narrative``
(pack-resident, no chain, same thematic shape, same affirming
evidence pointer cause_creation_reveals_meaning). Demo headline,
evals/learning_loop/run_demo.py, core/cli.py preamble, and the
test assertions all updated together so the demo reads cleanly:
before: [none] I don't know — insufficient grounding...
after : [teaching] narrative — teaching-grounded ... narrative
reveals meaning ...
3. test_discovery_candidates (4 tests).
Test fixture used (judgment, CAUSE) as the still-cold pair.
Epistemology v1 (commit 2acf71f) ratified
cause_judgment_requires_wisdom — (judgment, cause) is no longer
cold. Rotated to ``principle`` (pack-resident, no chain on either
intent today). Added a pytest.skip self-guard so when a future
curriculum unit ratifies a (principle, *) chain the test rotates
cleanly instead of going red.
Full lane: 1892 passed, 2 skipped, 0 failed (was 4 failed pre-fix,
13 failed pre-ADR-0063). Cognition eval unchanged: public 100/100/
91.7/100, holdout 100/100/83.3/100.
74 lines
2.9 KiB
Python
74 lines
2.9 KiB
Python
"""Learning-loop demo — pins the load-bearing before/after claim.
|
|
|
|
If any assertion fails, the headline claim ("CORE learned a new chain
|
|
from a cold turn and the same prompt is now teaching-grounded with
|
|
provenance") no longer holds.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from evals.learning_loop.run_demo import run_demo
|
|
|
|
|
|
def test_demo_closes_the_full_loop() -> None:
|
|
report = run_demo(emit_json=True)
|
|
assert report["learning_loop_closed"] is True
|
|
assert report["active_corpus_byte_identical"] is True
|
|
assert len(report["scenes"]) == 5
|
|
|
|
|
|
def test_before_is_ungrounded_disclosure() -> None:
|
|
report = run_demo(emit_json=True)
|
|
assert report["before"]["grounding_source"] == "none"
|
|
assert "insufficient grounding" in report["before"]["surface"].lower()
|
|
|
|
|
|
def test_after_is_teaching_grounded_with_new_chain_atoms() -> None:
|
|
report = run_demo(emit_json=True)
|
|
assert report["after"]["grounding_source"] == "teaching"
|
|
surface = report["after"]["surface"].lower()
|
|
# The accepted chain is (narrative, cause, reveals, meaning).
|
|
# ``thought`` was the original cold subject; cognition saturation
|
|
# v2 (commit ``a0edbb4``) added ``cause_thought_reveals_meaning``
|
|
# to the active corpus so the demo switched to ``narrative`` —
|
|
# same shape, still cold.
|
|
assert "narrative" in surface
|
|
assert "reveal" in surface # humanised connective
|
|
assert "meaning" in surface
|
|
assert "teaching-grounded" in surface
|
|
|
|
|
|
def test_s1_emits_one_discovery_candidate() -> None:
|
|
report = run_demo(emit_json=True)
|
|
s1 = report["scenes"][0]
|
|
assert s1["scene"] == "S1_cold_turn"
|
|
assert s1["detail"]["discovery_candidates_emitted"] >= 1
|
|
|
|
|
|
def test_s3_replay_gate_reports_no_regression() -> None:
|
|
report = run_demo(emit_json=True)
|
|
s3 = report["scenes"][2]
|
|
assert s3["scene"] == "S3_propose_replay_pass"
|
|
ev = s3["detail"]["replay_evidence"]
|
|
assert ev["replay_equivalent"] is True
|
|
assert ev["regressed_metrics"] == []
|
|
assert s3["detail"]["state"] == "pending"
|
|
|
|
|
|
def test_s4_active_corpus_byte_identical_after_accept() -> None:
|
|
report = run_demo(emit_json=True)
|
|
s4 = report["scenes"][3]
|
|
assert s4["scene"] == "S4_accept_against_transient"
|
|
assert s4["detail"]["active_corpus_byte_identical"] is True
|
|
assert s4["detail"]["transient_lines_after"] == s4["detail"]["transient_lines_before"] + 1
|
|
|
|
|
|
def test_same_prompt_drives_before_and_after() -> None:
|
|
"""The same input string drives both sides of the before/after pair.
|
|
Different surfaces emerge from the corpus state change alone, not
|
|
from any prompt variation or stochastic sampling."""
|
|
report = run_demo(emit_json=True)
|
|
assert report["prompt"] == "Why does narrative exist?"
|
|
# And the two surfaces are observably different — the loop changed
|
|
# the response, not merely the metadata.
|
|
assert report["before"]["surface"] != report["after"]["surface"]
|