core/evals
Claude 9b19085674
feat(evals,tests): PR-11 — the soak becomes evidence, and its cadence becomes a pin (R-9 + R-2)
Rank 4 on the docket. The L10 always-on lane has passed a 5000-beat soak since
2026-07-19. That result lived as PROSE IN A CONTRACT FILE — no machine-readable
artifact, no run SHA, no rerun path, and a deterministic_digest that report.py
computed and nothing pinned. The contract's own closing line asked for the pin
("Pin it once the lane is trusted so a regression flips it") and it was never
added. A recorded prose result with no re-derivable artifact is testimony, not
evidence, and that file is exactly where the distinction was meant to be
enforced.

R-2 first, before the run — as the plan required, so the soak measured a
decided question rather than deciding one by accident.

R-2 was ruled B (beats govern cognition; wall-clock permitted in telemetry
only). Measured: this lane carries NO WALL-CLOCK AT ALL — runner.py,
predicates.py and report.py contain no time, datetime, or timestamp of any
kind. That is stronger than B requires, and it is KEPT rather than "completed",
for three reasons now recorded in the contract:

  1. B's permissive half is a permission, not an obligation. Adding a field so
     the code matches a ruling's letter, then adding a pin to guard the field,
     is a mechanism whose only purpose is to be guarded.
  2. "When was this last run?" is already the artifact's COMMIT DATE —
     immutable, verifiable, stated once. A generated_at inside the report would
     be a second copy of one fact, drifting the moment a report is regenerated
     and not committed. That is G-23's defect class, registered this same week.
  3. It keeps the digest honestly deterministic. A timestamp would have to be
     excluded from it — a value in the artifact that the artifact's own
     integrity check ignores.

The contract also records what WOULD change this: a predicate whose claim is
about duration rather than sequence (H5's resource-cost leg is the live
candidate). B is already ruled, so that needs no new decision — only the
predicate that requires it.

The re-run: 5000 beats, reboot at 2500, under the POST-R-3 profile. All four
predicates pass. Committed as
evals/l10_always_on/results/report-5000-6c67e88d.json, digest f814fd97….

The most useful result was not the pass. Every DETERMINISTIC fact reproduced
exactly — vault bounded at 6, convergence at beat 1 with a 4999-beat tail,
clean reboot with derived learning intact — while the machine-variant float
moved from the prose's 1.389e-07 to a worst of 6.177e-08. That is the first
real evidence that excluding versor_condition from the digest was correct
rather than convenient: a digest covering it would have flipped on a machine
change and taught every reader to ignore it. The prose figure is superseded,
not contradicted — it was never a reproducible quantity.

tests/test_l10_soak_evidence.py, on the gate, guarding two DIFFERENT failure
modes that are easy to conflate:

  REGRESSION — the digest is pinned, plus the horizon (a 24-beat report pinned
  as though it were the 5000-beat one would pass everything else), the four
  predicates by name (all_gates_pass alone is satisfied by an empty predicate
  list), and a vacuity guard on the attested set.

  STALENESS — the mode nothing guarded, and the one that had ALREADY FIRED.
  R-9 ruled the cadence change-triggered, not clock-triggered: local-first
  doctrine makes "nightly" a ruling rather than a cron line, and a nightly that
  does not run is green for the wrong reason (H-9). So instead of "when did we
  last run it?", the pin asks the question that matters — DOES THIS EVIDENCE
  STILL DESCRIBE THE CODE THAT SHIPS? — by holding the SHA-256 of every source
  the soak attests: chat/always_on.py, chat/always_on_daemon.py, and the three
  harness modules.

That is not hypothetical. It had already fired silently for six weeks: the
2026-07-19 soak ran with accrue_realized_knowledge absent from
CONTINUOUS_LIFE_CONFIG_FLAGS, so from R-3 (PR-5, this same day) onward the
recorded result attested a process that no longer existed — and nothing in the
repository could notice. HAD THIS PIN EXISTED, PR-5 WOULD HAVE FAILED IT. That
is precisely the intended behaviour, and it is verified by sabotage: touching
always_on_daemon.py turns it red.

The failure message says RE-RUN THE SOAK — DO NOT EDIT THE HASHES, because
updating a hash to match changed source without re-running converts the pin
into a record asserting evidence that does not exist. That is the failure this
whole arc is about, committed inside its own remedy.

Three sabotages observed red: a touched daemon (the staleness mode), a
regressed verdict in the artifact, and a shorter horizon pinned as if it were
5000 beats.

H1-H4 promoted onto the gate, and the cheap option was deliberately refused.
Measured 20.95s for the four real-soak "holds" and 0.46s for the eight mutation
"bites" — 98% of the cost is the holds. Shipping only the bites, or parking the
file in a curated-but-unreachable suite, would have satisfied R-9's letter and
bought nothing; a suite nobody runs is the non-guarantee this arc spent PR-4
closing. Those four holds are the ONLY tests on any gate that run
chat/always_on's run_continuous end-to-end, and that loop just shipped a
six-week silent divergence. 21s per push to actually run the process that broke
is the most defensible time on this gate. Removed from full_only_baseline.txt
in the same edit; the baseline shrank 746 -> 745.

Closes G-5. Proof-of-life has moved from prose to committed, pinned,
change-guarded evidence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wcw2pnMBwyvmNyQg4uPEt4
2026-07-28 07:13:24 +00:00
..
adr_0242_v2_energy_compare feat: close ADR-0241/0242 post-Accept backlog (local-first mastery) 2026-07-16 15:25:43 -07:00
adr_0243_cognitive_lifecycle feat(adr-0243): Phase 4 falsifiability benchmark — metrics eval + CLI dispatcher 2026-07-17 12:59:24 -07:00
adr_0244_gamma_calibration fix(tests): clear the last reds — a machine-specific path and a stale drift pin 2026-07-25 16:31:06 +00:00
adr_0244_identity_gate fix(cognition): pre-stage land repairs after L2/PASSTHROUGH excision 2026-07-20 13:34:59 -07:00
adr_0244_qtop_vacuity feat(adr-0244): prove Q_top vacuity — D4 hollow-gate evidence + annotation upgrade 2026-07-17 14:39:52 -07:00
adr_0246_discrimination feat(adr-0246): Opus audit + §3.7 admit surface + serve wiring + §6.3 discrimination 2026-07-17 22:51:46 -07:00
adr_0246_geometric_suite draft(adr-0246): slice-1 scaffold — §6.1/§6.2 eval suite + malformed-F guard 2026-07-17 22:09:32 -07:00
adr_0246_grounding_feasibility feat(adr-0246): completion — §4.1 records, path serve integration, §11 feasibility (honest NULL), ADR body Proposed 2026-07-17 23:36:41 -07:00
adr_0246_mismatch_diagnostic feat(adr-0246): §3 induced-action primitives (pure, off-serving) 2026-07-17 21:36:57 -07:00
adversarial_identity
analogical_transfer feat(adr-0240/0243): seam S2 — chiral-composed, harness-driven biography write-path 2026-07-18 10:15:38 -07:00
anchor_lens_tour
anti_regression feat: strengthen visibility and measurement of CLOSE flywheel proposal review/ratification side (#794) 2026-06-16 19:33:25 -07:00
articulation
articulation_of_status
audio_sensorium
audit_tour chore: Refactor CLI and Governance Anchors (#926) 2026-07-03 12:34:56 -07:00
calibration chore: Refactor CLI and Governance Anchors (#926) 2026-07-03 12:34:56 -07:00
capability_index chore(generalization): Tier S housekeeping — smoke promotion, capability-index entries, promotion sweep (S5) 2026-07-24 17:13:19 -07:00
classical_literature_ood
close_derived_climb chore: Refactor CLI and Governance Anchors (#926) 2026-07-03 12:34:56 -07:00
cognition fix(cognition): expose hash_surface; point the register-matrix pin at it 2026-07-25 14:06:36 +00:00
cold_start_grounding
combined_rate_oracle docs(cmb): lookback review + fix single-agent-attribution hygiene hazard (H3) 2026-06-08 13:27:17 -07:00
compositionality
compound_intent_decomposition
comprehension feat: one-hop sound relational entailment (inverse/symmetric) + capability-index lane (#775) 2026-06-15 11:39:41 -07:00
constraint_oracle feat(verified): P1-C — split bound_slots_digest into separate obligation 2026-06-08 17:17:31 -07:00
construction_inventory feat(evals): measure the reader/writer construction inventories against each other 2026-07-27 14:11:07 -07:00
contemplation_quality
contradiction_detection
conversation
conversational_thread_coherence
cross_domain_transfer
curriculum_loop_closure feat(deduction-serve): Band v6-EX — existential witnesses, decided (ADR-0261) 2026-07-24 14:14:37 -07:00
curriculum_serve feat(curriculum): taught negatives — polarity is read (ADR-0264 R1-R4, R8) 2026-07-26 12:19:19 -07:00
deduction_serve fix(evals): the deduction lane hashes the prose it serves 2026-07-26 19:45:48 -07:00
deductive_logic test(l10): add independent-gold adversarial logic fixtures 2026-06-05 09:08:23 -07:00
demo_composition fix(ci): path-stable env deltas so lane SHA pins stop thrashing 2026-07-14 17:41:04 -07:00
determination_closure feat(determine): idle deductive consolidation — the loop learns from determined facts (Step D) 2026-06-06 12:28:09 -07:00
determination_estimation Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
deterministic_fluency
dimensional feat: dimensional-reasoning lane — 3rd diversity-panel domain 2026-06-04 16:38:56 -07:00
discourse_paragraph
domain_contract_validation
edge_budget chore: Refactor CLI and Governance Anchors (#926) 2026-07-03 12:34:56 -07:00
elementary_mathematics_ood
english_fluency_ood
environment_falsification Add offline witness log importer 2026-06-06 12:37:57 -07:00
event_vision_sensorium Add event vision sensorium lane 2026-06-06 12:37:57 -07:00
fabrication_control
field_incidence work done towards furthering comprehension, contemplation, learning, and also began working on fixing some failing tests 2026-06-16 12:27:59 -07:00
flywheel_demo Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
forward_semantic_control Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
foundational_biology_ood
foundational_physics_ood
frame_verdict_text_cwa feat(frame-verdict): closed-world FrameVerdict substrate — PR-1..4 + hardening (ADR-0222 B4) (#787) 2026-06-16 06:23:03 -07:00
frontier_compare
generalization feat(evals): implement safe metadata and fail-closed evaluator check for GSM1K 2026-06-23 07:52:32 -07:00
grammar_roundtrip feat(evals): measure the reader/writer construction inventories against each other 2026-07-27 14:11:07 -07:00
grammatical_coverage feat(evals): the fluency lanes report the writer that actually ships (Phase 4) 2026-07-27 11:57:48 -07:00
gsm8k_math test(reader-arc): pin deterministic tune/measure split BEFORE grammar work 2026-07-18 16:39:04 -07:00
gsm8k_parser_dev
hebrew_fluency
identity_divergence
industry_demos Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
inference_closure
introspection
koine_greek_fluency
l10_always_on feat(evals,tests): PR-11 — the soak becomes evidence, and its cadence becomes a pin (R-9 + R-2) 2026-07-28 07:13:24 +00:00
l10_continuity test(l10): W2-R arbitrary-interruption recovery harness (ADR-0219) 2026-06-15 02:45:29 -07:00
lab
learning_arc feat(workbench): add UI catch-up evidence scaffolding (#891) 2026-06-23 10:09:45 -07:00
learning_loop fix(ci): re-pin public_demo lane SHA after showcase content drift (#807) 2026-06-17 15:42:40 -07:00
long_context_cost
math_bounded_grammar/v1
math_capability_axes chore(evals): refresh stale committed reports 2026-06-03 01:01:50 -07:00
math_expert_claims/v1 docs(claims): ADR-0200 reconciliation — expert claim to audit-passed truth 2026-06-02 10:06:16 -07:00
math_symbolic_equivalence
math_teaching_corpus/v1
miner_loop_closure
monotonic_learning
multi_agent_composition
multi_sentence_response
multi_step_reasoning
obligation_2_ood_ratio
obligation_5_perturbation
obligation_6_depth_curve
obligation_8_adversarial
obligation_10_pack_provenance
orthogonality_tour
prompt_diversity
proof_carrying_promotion_demo feat(demos): Implement ADR-0218 PR D proof-carrying promotion demo (#927) 2026-07-03 13:01:49 -07:00
proofwriter_owa feat(eval): ProofWriter-OWA refusal-floor lane (B1) — independent oracle, measure-only (#779) 2026-06-15 12:28:20 -07:00
propositional_logic feat(comprehend): propositional-logic comprehension (4th domain, flagship oracle) 2026-06-05 23:24:54 -07:00
provenance
public_demo fix(ci): re-pin public_demo lane — real content drift, not the env flake (S2) 2026-07-24 16:58:05 -07:00
rate_oracle test(rate-oracle): port _canonical_outcome non-vacuous validation to R3 (R3-vac) 2026-06-08 07:38:20 -07:00
realizer_guard
refusal_calibration feat(identity): L11 identity continuity — same identity across reboot, not just same bytes 2026-06-05 13:52:57 -07:00
refusal_taxonomy fix: green test-fast suite, consolidate ADR graph under docs/adr, and complete governance cohesion anchors 2026-06-30 17:56:12 -07:00
register_diagnostics
register_tour fix(ci): re-pin public_demo lane SHA after showcase content drift (#807) 2026-06-17 15:42:40 -07:00
relational feat(reader): add overlaps_event finite-verb reader surface (B3) (#783) 2026-06-15 14:56:09 -07:00
relational_inference/v1 feat(reader): add overlaps_event finite-verb reader surface (B3) (#783) 2026-06-15 14:56:09 -07:00
relational_metric feat(oracle): narrow reverse-solve contract — base of one more/fewer_than (PR-7a) 2026-06-07 06:22:36 -07:00
relational_operator_ablation feat(eval): sealed relational operator ablation lane v1 2026-07-19 21:00:15 -07:00
relational_transitive feat(determine): add transitive strict-order relational inference (#781) 2026-06-15 14:20:30 -07:00
reports Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
results
reviewer_registry
sample_efficiency
self_consistency_over_time
sensorimotor_sensorium Add sensorium eval and governance runway 2026-06-03 20:53:05 -07:00
sensorium Add event vision sensorium lane 2026-06-06 12:37:57 -07:00
set_membership evals: add staged independent gold lanes 2026-06-05 16:03:26 -07:00
setup_oracle feat(comprehension): inverse reader frame — base of a more/fewer-than (PR-7b / R2 C0) 2026-06-07 07:06:26 -07:00
structure_mapping feat(rnd): ADR-0252 §5 run to a verdict — NO-GO, and the two prior GOs are void 2026-07-28 02:55:31 +00:00
syllogism feat(comprehend): multi-word NP chunking under a canonicalization contract 2026-06-05 22:47:34 -07:00
symbolic_logic
teaching_injection_resistance
total_ordering feat(comprehend): multi-word NP chunking under a canonicalization contract 2026-06-05 22:47:34 -07:00
vision_sensorium Add vision evidence and sensorimotor contracts 2026-06-03 20:27:46 -07:00
walkthrough_chain
warmed_session_consistency
zero_code_domain_acquisition
__init__.py
_parallel.py
baseline_runner.py
CLAIMS.md Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
cognition_cases.jsonl
framework.py
generalized_lift_instrument.py chore(adr-0250): ratify Accepted + fold in PR #74 review findings 2026-07-18 15:07:29 -07:00
holdout_runner.py
lift_evidence_handoff.py feat(evals): seams S4+S5 — generalized-lift instrument + ports/handoff evidence 2026-07-18 10:34:26 -07:00
logic_cnf_compiler.py feat(adr-0249): P3 structural formula→CNF converter (deduction leg) 2026-07-18 12:40:31 -07:00
metrics.py fix(evals): carry hash_surface on the CaseResult the register matrix actually uses 2026-07-25 14:25:31 +00:00
multi_register_program.py feat(reader-arc): compare compiler-side guards for inverse + chained shapes 2026-07-18 17:14:16 -07:00
parallel.py
run_cognition_eval.py fix(evals): carry hash_surface on the CaseResult the register matrix actually uses 2026-07-25 14:25:31 +00:00
turn_program.py feat(adr-0249): P4 turn-program compiler + chained-relaxation executor 2026-07-18 12:55:31 -07:00
vocab_trigger_instrument.py feat(generalization): vocab-trigger instrument — mechanism-vs-coverage refusal histogram (S3) 2026-07-24 16:54:13 -07:00