Define the long-term architectural posture for CLOSE-derived facts and FrameVerdict as deliberate non-interaction. Adds the ratification artifact and updates runtime contracts to preserve the open/proposal/review lane separately from the closed-world FrameVerdict lane, with INV-30/INV-31 and pillar references. PR: #795
823 lines
44 KiB
Markdown
823 lines
44 KiB
Markdown
# Runtime Contracts
|
||
|
||
This document freezes the runtime contracts used by chat, telemetry, memory,
|
||
and future teaching work. It exists to prevent contract drift between tests,
|
||
runtime code, and future cognitive pipeline work.
|
||
|
||
## Field invariant
|
||
|
||
CORE state is a versor field. Runtime code must preserve the core closure
|
||
contract:
|
||
|
||
```text
|
||
versor_condition(F) < 1e-6
|
||
```
|
||
|
||
If a propagation path violates this invariant, fix the operator path or the
|
||
explicit closure boundary that owns the transition. Do not hide violations by
|
||
changing tests or silently downgrading the invariant.
|
||
|
||
## ChatResponse contract
|
||
|
||
`ChatResponse.surface`
|
||
: The selected user-facing response. This is the exact string returned by
|
||
`ChatRuntime.respond()` and should match what the user receives.
|
||
|
||
`ChatResponse.walk_surface`
|
||
: The manifold/token-walk evidence surface. It is trace evidence for what the
|
||
field traversal produced. It is not necessarily the user-facing response.
|
||
|
||
`ChatResponse.articulation_surface`
|
||
: The proposition/realizer surface. This is the structured linguistic
|
||
realization of the current proposition or proposition graph.
|
||
|
||
Current selection policy:
|
||
|
||
```text
|
||
surface = determination_surface (when accrue_realized_knowledge AND the turn
|
||
DETERMINED an answer over realized knowledge)
|
||
surface = [approximate] estimate (Step E — when estimation_enabled AND the turn was a
|
||
REFUSED converse query whose predicate-class holds a
|
||
genuine SERVE license; DISCLOSED, never asserted)
|
||
surface = _UNKNOWN_DOMAIN_SURFACE (when the unknown-domain gate fired)
|
||
surface = articulation_surface (otherwise — the default)
|
||
walk_surface = retained telemetry/evidence (always)
|
||
articulation_surface = retained always (neither determination nor estimate replaces it)
|
||
```
|
||
|
||
### Unknown-domain gate honour
|
||
|
||
When `vault/decompose.py::UnknownDomainGate` fires, ChatRuntime returns
|
||
the safety stub `_UNKNOWN_DOMAIN_SURFACE` ("I don't have field
|
||
coordinates for that yet.") and `vault_hits == 0`.
|
||
`CognitiveTurnPipeline` honours that stub: the user-facing `surface`
|
||
remains the gate's response and is *not* overridden by the realizer's
|
||
fallback articulation. The realizer's surface always survives in
|
||
`walk_surface` as evidence — only the user-facing selection is
|
||
gated. This closes `evals/calibration/gaps.md` Finding 2.
|
||
|
||
Future realizer work may change the selection policy, but must update this
|
||
document and the contract tests in the same PR.
|
||
|
||
## Workbench cognitive pipeline record
|
||
|
||
`CognitiveTurnResult` is the complete in-memory turn record. Workbench
|
||
journaling persists a narrower `CognitivePipelineRecord` alongside each new
|
||
`ChatTurnResult` / `TurnJournalEntry`. The canonical read projection is
|
||
`GET /trace/{turn_id}/pipeline`; `/trace/{turn_id}` may embed the same record
|
||
for envelope inspection, but the visualizer reads the projection.
|
||
|
||
Persisted stages are:
|
||
|
||
```text
|
||
input -> intent -> PropositionGraph -> ArticulationTarget -> realizer
|
||
-> walk_telemetry -> trace_hash
|
||
```
|
||
|
||
Contract:
|
||
|
||
- The record is curated stage evidence, not raw runtime state. It persists
|
||
structured stage summaries/details, `trace_hash`, and scalar
|
||
`versor_condition`; it must not persist `field_state_before`,
|
||
`field_state_after`, `FieldState.F`, holonomy, or proposition versor arrays.
|
||
- A live `/chat/turn` with a trace hash but no `status="recorded"`
|
||
`CognitivePipelineRecord` fails before journal append. A future required
|
||
stage added to the record contract must fail loudly until the producer is
|
||
widened.
|
||
- Pre-widening journal rows are lawful evidence gaps: `/trace/{turn_id}` must
|
||
show `missing_evidence`, not a synthesized green pipeline.
|
||
- The Trace visualizer may derive a stage rail, deterministic DAG, propagation
|
||
list, and selected-stage inspector from the persisted record. It must not
|
||
infer unrecorded stage internals by replaying the turn in the browser.
|
||
- Replay treats `pipeline_record` as critical evidence. Divergence in the
|
||
persisted pipeline record is evidence against equivalence, not wall-clock
|
||
noise.
|
||
|
||
### Determination surface (Step B-2)
|
||
|
||
When `accrue_realized_knowledge` is enabled and a question turn is **Determined**
|
||
over realized knowledge (`generate.determine.determine` returns a `Determined`),
|
||
the user-facing `surface` is the rendered determination. The hardened
|
||
CLOSE derived-climb yardstick (`evals/close_derived_climb`) now exercises
|
||
this path explicitly via `determine()` asserts with `rule='direct'` on
|
||
materialized derived facts (see "semantic_positives_determined_direct" in
|
||
its contract and the Dedicated CLOSE Flywheel Regression Surface in
|
||
docs/testing-lanes.md + docs/analysis/close-flywheel-dedicated-regression-surface-ratification-2026-06-16.md).
|
||
(`generate.determine.render_determination`) — the engine answers *from what it
|
||
accrued in the conversation*. The basis is rendered **honestly**: SPECULATIVE
|
||
grounds (today's only case) read as "as I was told", never "verified"; D0 only
|
||
asserts `answer=True`, so the surface is an affirmation, never a fabricated or
|
||
asserted-False string.
|
||
|
||
The realizer's `articulation_surface` is **retained unchanged** as evidence — the
|
||
determination is a user-facing *selection*, exactly like the unknown-domain gate,
|
||
not a rewrite of the realizer output. An **Undetermined** turn (refused,
|
||
open-world) keeps the default articulation surface — the honest "I don't know".
|
||
With the flag off, this path is never taken and the surface is unchanged. This is
|
||
selection only: it adds no field op, no normalization, and proposes no learning
|
||
(`accrue_in_turn` writes SESSION memory through the INV-21 vault writer; the HITL
|
||
teaching path is untouched).
|
||
|
||
### Idle consolidation (Step D — CLOSE)
|
||
|
||
When `consolidate_determinations` is enabled, `ChatRuntime.idle_tick` runs a
|
||
**consolidation pass** (`generate.determine.consolidate_once`): one semi-naive layer
|
||
of the member/subset deductive closure over the held self. For every sound one-hop
|
||
inference — `member(s,b) ∧ subset(b,t) → member(s,t)` and `subset(a,b) ∧ subset(b,t)
|
||
→ subset(a,t)`; **never `member ∘ member`** (instance-of is not transitive) — whose
|
||
conclusion is not yet realized, the hop is **verified by the sound+complete
|
||
proof_chain ROBDD** (reusing DETERMINE's single verifier) and written back as a
|
||
realized record via `generate.realize.realize_derived`, so the next `determine`
|
||
reaches it directly. Across idle ticks the directly-answerable set climbs
|
||
monotonically to the deductive-closure fixed point; a saturated tick consolidates
|
||
nothing (`IdleTickResult.facts_consolidated == 0`).
|
||
|
||
**Relational transitive extension (PR-1, 2026-06-16):** The same pass now also
|
||
consolidates direct 2-hop (and, over subsequent ticks, multi-hop) conclusions for the
|
||
four *declared* strict-order predicates in `TRANSITIVE_PREDICATES`
|
||
(`less_than`, `greater_than`, `before_event`, `after_event`) using the identical
|
||
mechanism: one-hop candidate collection (same-predicate only), mandatory verification
|
||
via the Phase-C `_relational_transitive` searcher + proof_chain ROBDD, and
|
||
`realize_derived(..., rule="transitive")`. Non-transitive predicates (parent_of,
|
||
sibling_of, left_of, etc.), inverse mixing, symmetric "promotion", cross-predicate
|
||
composition, and reflexive conclusions remain refused. The member/subset is-a rules
|
||
and the `member ∘ member` fallacy bite are completely unchanged.
|
||
|
||
Contract (applies uniformly to both the is-a and the declared relational transitive cases):
|
||
|
||
- **SPECULATIVE / as-told.** A fact derived from SPECULATIVE premises stays
|
||
SPECULATIVE — a sound inference never upgrades the *standing* of its premises;
|
||
COHERENT is never minted here.
|
||
- **SESSION memory, not reviewed learning.** Consolidation is an extension of the
|
||
`generate.realize` session path — **not** corpus mutation and **not** coupled to
|
||
proposals. The teaching/review HITL path is untouched (no parallel learning path).
|
||
- **wrong=0 by proof-gating.** Only proof_chain-`ENTAILED` conclusions are written;
|
||
the `member ∘ member` fallacy is structurally unreachable; relational candidates
|
||
are gated by the same verifier that DETERMINE already uses for the four predicates.
|
||
- **Replayable provenance.** Each derived record carries a `Derivation` (premise
|
||
`structure_key`s + rule + the `entailed` verdict), so a replay re-derives and
|
||
re-verifies — the soundness claim can meaningfully fail.
|
||
- **No new normalization, no closure/repair.** Writes reuse the INV-21 vault writer;
|
||
`algebra/versor.py` keeps closure. Off by default; the falsification lane is
|
||
`evals.determination_closure`.
|
||
|
||
See `docs/analysis/close-relational-transitive-pr1-2026-06-16.md` for the exact
|
||
positives/negatives, multi-hop climb evidence, branch discipline, and the 3-PR
|
||
sequence plan that governs the rest of this capability slice.
|
||
|
||
### Idle proposal review (read-only)
|
||
|
||
When `review_pending_proposals` is enabled, `ChatRuntime.idle_tick` runs a **read-only**
|
||
sub-pass (after the consolidation pass) that scans the comprehension-failure proposal sink
|
||
(`teaching/proposals/comprehension_failures/`, the contemplation pass's proposal-only
|
||
artifacts) and surfaces a summary in `IdleTickResult.proposal_review` (`safe`, `total`,
|
||
`review_needed`, `malformed`, `by_family`, `errors`).
|
||
|
||
Contract:
|
||
|
||
- **Read-only, no mutation.** The sub-pass scans, reports, and dry-checks; it writes, moves,
|
||
and deletes nothing. It does **not** set `did_work`, so it never triggers a checkpoint, and
|
||
it never advances learning, ratifies, mounts, or modifies a reader.
|
||
- **Failure-isolated.** A reporter exception is **captured, not propagated** — surfaced as
|
||
`proposal_review.safe == False` with `errors == ("proposal_review_failed:<type>",)` — so a
|
||
malformed sink or filesystem error can never corrupt the idle tick's state or return.
|
||
- **Default off, additive.** Disabled, `proposal_review` is `None` and `IdleTickResult` is
|
||
unchanged for existing callers. This is **not** a second idle loop and **not** the L10
|
||
always-on heartbeat — it only surfaces existing review obligations.
|
||
|
||
### Derived CLOSE proposal bridge (PR-2)
|
||
|
||
When `review_derived_close_proposals` is enabled, `ChatRuntime.idle_tick` (after the
|
||
consolidation pass) runs a deterministic scan over realized derived facts. A fact is
|
||
eligible only if:
|
||
|
||
- `derived is True`
|
||
- `derivation is not None` and `derivation.verdict == "entailed"`
|
||
- `epistemic_status == "speculative"`
|
||
- `relation_predicate` is `"member"`, `"subset"`, or one of the four
|
||
`TRANSITIVE_PREDICATES` (`less_than`, `greater_than`, `before_event`, `after_event`)
|
||
|
||
It emits reviewable proposal-only artifacts (source=`"derived_close_fact"`) carrying
|
||
predicate, arguments, derivation (rule/verdict/premise keys), epistemic status, and
|
||
structure/dedupe keys. Artifacts are written to `teaching/proposals/derived_close_facts/`
|
||
and deduplicated by a stable key (predicate + arguments + derivation + structure_key).
|
||
|
||
Contract:
|
||
|
||
- **Proposal-only, review-gated.** The artifacts are for human/HITL review. No corpus
|
||
mutation, no ratification, no COHERENT minted, no serving change, no determine change.
|
||
- **Skips safely.** Non-derived facts, malformed records, unsupported predicates, and
|
||
non-entailed derivations are counted and skipped (never crash, never emit).
|
||
- **Deterministic.** Emission order and dedupe are independent of recall order; repeated
|
||
ticks on the same state emit zero new proposals.
|
||
- **Default off.** Additive to the existing comprehension-failure proposal review. When
|
||
off, no derived-close proposals are emitted and `IdleTickResult` is unchanged.
|
||
|
||
The Dedicated CLOSE Flywheel Regression Surface (Claim-B Level) plus the review-visibility strengthening now provide structured `proposal_review_posture` (in the climb) and `proposal_review_summary` (in the anti-regression demo) that make the review-gated posture of these proposals and the transition events in exercised teaching proposal paths first-class, observable signals in the heavy verification lane. See `docs/analysis/close-flywheel-proposal-review-visibility-ratification-2026-06-16.md` and `docs/testing-lanes.md`. No change to the emission contract or review boundaries.
|
||
|
||
See `docs/analysis/close-derived-proposal-bridge-2026-06-16.md` for the full artifact
|
||
schema, dedupe key, test gates, and the PR-1→PR-2 composition.
|
||
|
||
### Architectural Posture: CLOSE-Derived Facts and FrameVerdict (deliberate long-term separation)
|
||
|
||
CLOSE-derived facts and the proposals that carry them for review (source=`"derived_close_fact"`, born with `status="proposal_only"`, `epistemic_status="speculative"`, `requires_review=true`) operate entirely within the open-world derivation / determine / reviewed-teaching promotion path. They are accretive (sound inference over realized content under the open-world assumption), replayably checksummed in the Claim-B yardstick, and subject to the full proposal-only / SPECULATIVE / review-gated / provisional-standing discipline (INV-21–24, INV-29). Promotion to durable standing (corpus append or COHERENT) occurs only through reviewed teaching gates or the certificate-carrying `apply_certified_promotion` path — never autonomously and never by crossing into closed-world machinery.
|
||
|
||
`FrameVerdict` (and its `ClosedFrame` context) is the B4 closed-world substrate (ADR-0222). It requires an explicit declared-complete snapshot (`world_assumption` CLOSED or BOUNDED_CLOSED + `closure_declared=true`) and produces a distinct sealed type carrying `FrameVerdictKind` (ENTAILED_TRUE / ENTAILED_FALSE / UNDETERMINED / CONTRADICTION / SCOPE_BOUNDARY) plus positive-refutation proofs. Construction is funneled through a single allowlisted site; the path is default-dark and off-serving. It is type-distinct from `Determined` and never carries an `answer` bool.
|
||
|
||
**There is no data flow, no type sharing, no import path, and no semantic composition in either direction.** A CLOSE-derived fact (even after review, corpus append, or COHERENT promotion) is never packaged as a `ClosedFrame`, never passed to `evaluate_frame_verdict` or the perception adapter, and never influences a `FrameVerdict`. A `FrameVerdict` (or `ClosedFrame`) is never supplied to `determine()`, never becomes a premise in CLOSE derivation or consolidation, never appears in `IdleTickResult.derived_close_proposals_emitted`, never participates in `ProposalLog` / `ReviewState` transitions, and never affects epistemic status or vault promotion.
|
||
|
||
This deliberate non-interaction is mechanically enforced and non-vacuously verified by two invariants (see `tests/test_architectural_invariants.py`):
|
||
|
||
- **INV-30 (open-world DETERMINE never asserts False):** `determine()` produces only `Determined(answer=True, ...)` at three visible sites or `Undetermined` refusals. Absence never refutes. Closed-world negation is a distinct, lane-scoped type.
|
||
- **INV-31 (closed-world FrameVerdict cannot reach the open-world runtime):** Two-part firewall — (A) transitive import containment of the open-world spine (`chat.runtime`, `session.context`, `vault.store`) from `generate.frame_verdict` / `core.response_governance.frame_verdict` plus an exact construction allowlist (only `generate/frame_verdict/_construct.py`); (B) typed refusal (`determine` rejects `ClosedFrame` at eligibility; forged objects are type-rejected at the sole disposition adapter).
|
||
|
||
The increased richness and observability of the CLOSE proposal review/ratification side (#791–#794) — `proposal_review_posture` (all_proposal_only / all_speculative / all_requires_review / none_accepted_or_promoted guarantees), `proposal_review_summary` (scenes by review_state + log_transitions from hermetic teaching gates + close_derived merge), integration into the anti-regression demo and the `make test-close-flywheel` lane — was achieved entirely *inside* the open/proposal/teaching surfaces and the dedicated heavy verification lane. No boundary was crossed and no FrameVerdict or closed-world reasoning was involved or required.
|
||
|
||
**Long-term architectural posture (ratified):** Strict separation is the commitment. CLOSE-derived facts (and their full review/ratification lifecycle) address open-world accumulation, sound derivation, proposal-gated growth, and human/HITL ratification of what the autonomous half emits. FrameVerdict / closed-world addresses lane-scoped, declared-complete snapshot entailment and positive refutation under explicit CWA or perception assumptions. The two are orthogonal; the increased visibility of one does not create a requirement or justification to mix them.
|
||
|
||
Any future controlled interaction (however narrow, read-only, or "for targeted CWA checks over a ratified CLOSE subset") is a material architectural change. It requires a new or amended ADR, explicit updates to INV-30 and/or INV-31 with new non-vacuous anchors, re-verification that `wrong_total` remains 0, versor closure, epistemic rules, and determination contracts, plus fresh ratification and updates to this document and the invariants test suite. Until such a ratified change, the current strong separation (SPECULATIVE + proposal-gated + INV-31 firewall + INV-30 open-world True-only) must be preserved by construction.
|
||
|
||
See `docs/analysis/close-derived-facts-frameverdict-posture-ratification-2026-06-16.md` (the governing ratification artifact for this posture) for the full analysis of risks/benefits/implications, the explicit pillar alignment (Mechanical Sympathy, Semantic Rigor, Third Door), and the verification obligations. Cross-references: B4 lookback (`docs/analysis/b4-frameverdict-implementation-lookback-2026-06-15.md`), the Derived CLOSE proposal bridge and visibility ratifications (2026-06-16 family), `docs/Whitepaper.md` §IV, and the determination / epistemic / provisional-standing sections of this document.
|
||
|
||
### Estimation surface (Step E — ESTIMATION)
|
||
|
||
When `estimation_enabled` and a turn is a **converse query** DETERMINE refused (told
|
||
`p(a,b)`, asked `p(b,a)`), the engine offers a **calibrated, disclosed** estimate
|
||
instead of always refusing — but only through the ADR-0206 reach bridge:
|
||
|
||
- The blind converse-guesser (`generate.determine.estimate`) proposes `p(b,a)` holds.
|
||
- `govern_response` widens to `APPROXIMATE` **iff** the predicate-class holds a genuine
|
||
`Action.SERVE` `LicenseDecision` on the **ratified, committed** reliability ledger
|
||
(`generate/determine/data/estimation_ledger.json`, θ_SERVE=0.99, ADR-0175). An
|
||
unlicensed class stays `STRICT` — the honest refusal is unchanged.
|
||
- `shape_surface` **discloses** the estimate as `[approximate] …` (a converse guess is
|
||
`UNVERIFIED_POSSIBLE`, never in APPROXIMATE's fully-grounded admissible set).
|
||
|
||
Contract:
|
||
|
||
- **wrong=0 by construction.** An estimate is *always* disclosed (`[approximate]`),
|
||
never asserted as fact — a wrong estimate is a disclosed-wrong, not a silent one.
|
||
And it is offered only for a class whose committed track record clears the Wilson
|
||
floor (earned by volume: ≥657 perfect commits for SERVE).
|
||
- **Never a designed-in default; never self-authored.** Absent a cleared license →
|
||
refuse. Ceilings stay at safe defaults (the engine never raises its own bar); the
|
||
ledger is sealed-practice output, hash-verified on load (a hand-edited ledger is
|
||
rejected).
|
||
- **Session/serving only.** No corpus mutation, no proposal — the HITL teaching path
|
||
is untouched. Off by default; the falsification lane is `evals.determination_estimation`.
|
||
|
||
### Refusal contract (ADR-0024 Phase 2)
|
||
|
||
When the inner-loop admissibility check leaves no admissible destination
|
||
for the next step, the generation walk in `generate/stream.py` raises
|
||
`generate.exhaustion.InnerLoopExhaustion`, a typed subclass of
|
||
`ValueError` carrying:
|
||
|
||
```text
|
||
reason : RefusalReason (machine-readable taxonomy)
|
||
region_label : str (which AdmissibilityRegion blocked)
|
||
step_index : int (-1 = pre-walk empty intersection;
|
||
>=0 = in-walk per-step exhaustion)
|
||
rejected_attempts : tuple[(int, str, float), ...] (per-step evidence)
|
||
```
|
||
|
||
Reason codes are minimal in Phase 2: a single `INNER_LOOP_EXHAUSTION`
|
||
covers both raise sites. Phase 4 (rotor-frame admissibility, ADR-0025)
|
||
is expected to add a second reason for rotor exhaustion.
|
||
|
||
`CognitiveTurnResult.refusal_reason` carries the stable string value of
|
||
the `RefusalReason` when a turn refuses, and the empty string otherwise.
|
||
`compute_trace_hash` folds `refusal_reason` into the payload only when
|
||
non-empty, preserving byte-identical hashes for non-refused turns
|
||
relative to pre-Phase-2 (determinism invariant). When the field is
|
||
non-empty, it becomes load-bearing in replay equality.
|
||
|
||
Backward compatibility: `InnerLoopExhaustion` is a `ValueError`, so
|
||
every pre-Phase-2 `except ValueError` handler in `chat/runtime.py`,
|
||
eval lanes, and tests continues to catch it without modification.
|
||
|
||
Residual silent path (out of scope for Phase 2, future ADR):
|
||
`ChatRuntime.respond()` and `arespond()` still convert any `ValueError`
|
||
to the empty string for their public `str` return contract, so a real
|
||
turn that refuses today produces `surface == ""` with
|
||
`refusal_reason == ""` — the typed evidence is unread between the
|
||
raise site and the result. The plumbing on `CognitiveTurnResult`,
|
||
`compute_trace_hash`, and `CognitiveTurnPipeline` is in place so a
|
||
future ADR can wire materialisation (e.g. propagate the typed
|
||
exception to `ChatResponse.refusal_reason` or catch at the pipeline
|
||
seam) without re-deriving the contract.
|
||
|
||
### Ranked-with-margin contract (ADR-0026 / Phase 3)
|
||
|
||
The static `admissibility_threshold` documented above (ADR-0024 Phase 2)
|
||
is supplemented by a scale-invariant margin gate (ADR-0026 Phase 3).
|
||
The runtime selects mode via `RuntimeConfig.admissibility_mode`:
|
||
|
||
```text
|
||
RuntimeConfig.admissibility_mode : "threshold" | "margin" (default: "threshold")
|
||
RuntimeConfig.admissibility_margin : float (default: 0.4)
|
||
```
|
||
|
||
In **threshold mode** (back-compat, ADR-0024):
|
||
|
||
```text
|
||
admit iff cga_inner(versor(candidate), relation_blade) > admissibility_threshold
|
||
```
|
||
|
||
In **margin mode** (ADR-0026):
|
||
|
||
```text
|
||
rank candidates by cga_inner(versor(candidate), relation_blade), descending,
|
||
stable tie-break by candidate index
|
||
admit iff (single candidate)
|
||
or (score(top) > 0 AND score(top) - score(second) >= admissibility_margin)
|
||
```
|
||
|
||
`generate.admissibility.rank_candidates_by_blade` returns the ranked
|
||
list with deterministic tie-break, and `generate.admissibility.check_margin`
|
||
returns a typed `MarginVerdict` (`admitted`, `top`, `second`, `gap`,
|
||
`reason`). The selection invariant is that the *score difference* is
|
||
the gate, not the absolute score — making the gate robust to per-blade
|
||
norm variation that defeated static threshold tuning on the
|
||
Phase 4 characterization corpus (see
|
||
`docs/evals/phase5_stratified_findings.md`).
|
||
|
||
Refusal in margin mode is materialised through the same
|
||
`InnerLoopExhaustion` mechanism as threshold mode, with
|
||
`RefusalReason.INNER_LOOP_EXHAUSTION` carrying the full ranked
|
||
candidate list as evidence so the failure mode is "no candidate has
|
||
margin over its successor" rather than "no candidate exceeded
|
||
threshold T."
|
||
|
||
The default δ = 0.4 was selected from the minimum observed margin in
|
||
the Phase 3 v2 corpus (0.456) and is *falsifiable*: any case
|
||
surfacing a blade-gap below δ where margin-mode refusal is the wrong
|
||
behavior must be reported as an ADR-0026 falsification rather than
|
||
silently patched per case. Phase 5's 20-case stratified corpus does
|
||
not falsify δ = 0.4.
|
||
|
||
### Rotor admissibility contract (ADR-0025 / Phase 4)
|
||
|
||
The destination-side admissibility documented above (token-side blade
|
||
alignment, ADR-0024 / Phase 3) is complemented by a rotor-side check:
|
||
when a region carries a non-null `frame_versor`, the inner loop
|
||
additionally verifies that the rotor's effect on the current field
|
||
stays within the frame's admissible cone:
|
||
|
||
```text
|
||
F' = versor_apply(V, F_current)
|
||
score = cga_inner(F', frame_versor)
|
||
admit iff score > 0
|
||
```
|
||
|
||
`generate.rotor_admissibility.check_rotor_admissibility` performs
|
||
this pure semantic check. It lives at the same generation/propagation
|
||
seam as the inner loop — in `generate/rotor_admissibility.py`, a
|
||
sibling-but-separate module to `generate/admissibility.py` — **not**
|
||
in `algebra/versor.py` (admissibility is a pack-semantic test, not a
|
||
closure invariant) and **not** in `field/propagate.py` (forbidden
|
||
normalization/repair site). The placement is the load-bearing
|
||
architectural decision in ADR-0025.
|
||
|
||
Refusal is materialised through the same `InnerLoopExhaustion`
|
||
mechanism as destination-side refusal, but with
|
||
`RefusalReason.ROTOR_REJECTION` instead of `INNER_LOOP_EXHAUSTION`,
|
||
so the trace names the axis that ran out. In threshold mode, a step
|
||
that exhausts after *any* rotor rejection is reported under
|
||
`ROTOR_REJECTION`; pure destination exhaustion stays
|
||
`INNER_LOOP_EXHAUSTION`. In margin mode, the rotor check runs on the
|
||
top-ranked admissible candidate after destination margin admits; on
|
||
rotor refusal the typed exception carries the full destination
|
||
ranking plus the rejected rotor's score as evidence.
|
||
|
||
The `versor_condition(F) < 1e-6` invariant remains the algebra
|
||
layer's responsibility on actual propagation. `check_rotor_admissibility`
|
||
does not mutate field state and does not enforce closure — it only
|
||
asks whether applying `V` to `F` would leave the field in the
|
||
frame's half-space.
|
||
|
||
## TurnEvent contract
|
||
|
||
`TurnEvent.surface`
|
||
: Exact emitted user-facing response for the turn.
|
||
|
||
`TurnEvent.walk_surface`
|
||
: Exact manifold/token-walk evidence surface for the turn.
|
||
|
||
`TurnEvent.articulation_surface`
|
||
: Exact proposition/realizer surface for the turn.
|
||
|
||
`TurnEvent.vault_hits`
|
||
: Actual count of recall hits applied during generation. Never hardcode this.
|
||
|
||
`TurnEvent.flagged`
|
||
: Mirrors `IdentityScore.flagged` for filtering and trace inspection.
|
||
|
||
## Identity contract
|
||
|
||
Identity checks are telemetry/gating signals. A flagged identity score must not
|
||
silently erase useful generation unless an explicit hard-block policy is
|
||
configured and tested.
|
||
|
||
Canonical call style:
|
||
|
||
```python
|
||
IdentityCheck().check(trajectory, manifold)
|
||
```
|
||
|
||
Legacy constructor injection:
|
||
|
||
```python
|
||
IdentityCheck(manifold=manifold).check(trajectory)
|
||
```
|
||
|
||
is supported temporarily and emits `DeprecationWarning`. New code must not use
|
||
it.
|
||
|
||
## Memory and teaching contract
|
||
|
||
Session memory can be immediate and local to the running context. This includes
|
||
idle consolidation of soundly-derived (proof_chain-verified) facts back into the
|
||
held self (Step D — see *Idle consolidation* above): it is the immediate session
|
||
tier, not reviewed memory, and proposes nothing.
|
||
|
||
Reviewed memory must be explicit: user corrections or teaching examples become
|
||
reviewed memory only through the reviewed teaching loop.
|
||
|
||
Pack mutation is proposal-only until reviewed. Runtime correction capture must
|
||
not directly rewrite language packs, frames, identity axes, or operator code.
|
||
|
||
Identity manifold mutation by user prompt or correction is forbidden.
|
||
|
||
### Provisional vs durable standing (the typed learning boundary)
|
||
|
||
The boundary is between **durable** and **provisional** standing, not between
|
||
"reviewed" and "everything else". Provisional state — session memory, idle
|
||
SPECULATIVE consolidation, sealed practice ledgers, reliability counts, emitted
|
||
proposals, disclosed licensed estimates — may update autonomously *iff* it is
|
||
typed (carries its standing), isolated (same `VaultStore.store` path, no
|
||
parallel memory), replayable (no clock, no LLM, no sampling), and cannot
|
||
masquerade as ratified truth (written SPECULATIVE; rendered `as_told` /
|
||
`[approximate]` / "proposal"). Durable standing — corpus/pack/policy/identity
|
||
mutation, and any promotion to COHERENT/verified — stays reviewed
|
||
(`teaching/*`, proposal-only) or proof-carrying (ADR-0218
|
||
`apply_certified_promotion`, re-verified from a curator-certified coherent
|
||
base).
|
||
|
||
This is enforced by failing-when-violated invariants, not convention: INV-21
|
||
(vault-writer allowlist), INV-22/INV-23 (default SPECULATIVE), INV-24 (recall
|
||
categorization; user-facing evidence is COHERENT-only), INV-29 (only
|
||
`vault/store.py` transitions `epistemic_status`; the certificate gate is the
|
||
only default-reachable COHERENT producer), and INV-30 (the open-world
|
||
`determine()` gear asserts only `answer=True` or refuses — never `False`;
|
||
closed-world entailed-negation must be a distinct, lane-scoped type).
|
||
|
||
## Environmental falsification contract (ADR-0211)
|
||
|
||
`sensorium.environment.falsification` compares expected afferent evidence with
|
||
actual `ObservationFrame` evidence. It is a deterministic replay surface, not a
|
||
fusion layer, not reviewed memory, and not a mutable world model.
|
||
|
||
The v1 verdict set is closed:
|
||
|
||
```text
|
||
SUPPORTED | FALSIFIED
|
||
```
|
||
|
||
`SUPPORTED` means every expected slot matched by merge key and no unexpected
|
||
slot appeared. `FALSIFIED` means at least one expected slot was missing,
|
||
changed, or accompanied by unexpected evidence. Neither verdict promotes a
|
||
claim to reviewed memory or mutates packs, Vault state, identity axes, operator
|
||
code, or runtime policy.
|
||
|
||
Forbidden in the falsification bench:
|
||
|
||
- raw pixels, PCM, event streams, byte payloads, actuator traces, or decoded
|
||
action payloads in traces;
|
||
- motor/efferent units in `ExpectedObservationFrame` or `FalsificationRun`;
|
||
- learned latents as substrate;
|
||
- probabilistic confidence, hardware-noise envelopes, or tolerance thresholds in
|
||
v1 verdicts;
|
||
- `generate/*` dependencies, Vault mutation, or `ModalityRegistry.decode`.
|
||
|
||
## Testing policy
|
||
|
||
Tests should protect load-bearing behavior:
|
||
|
||
- versor closure
|
||
- deterministic replay
|
||
- runtime response/telemetry contracts
|
||
- memory correctness
|
||
- identity protection
|
||
- teaching/correction safety
|
||
- articulation contract
|
||
|
||
Avoid tests that preserve stale constructors, private helper shapes, or exact
|
||
formatting that is not part of a documented contract.
|
||
|
||
## Epistemic surface (ADR-0021)
|
||
|
||
CORE exposes a typed `epistemic_status` on the teaching and lexicon
|
||
surfaces. The status is a **position in the revision graph**, not a
|
||
source-trust tier:
|
||
|
||
| Status | Meaning |
|
||
|---------------|------------------------------------------------------------------------------------------|
|
||
| `COHERENT` | Fits current field geometry; no incoherence with reviewed claims detected at admission. |
|
||
| `CONTESTED` | Incoherent with at least one reviewed claim; review pending; not load-bearing. |
|
||
| `SPECULATIVE` | Proposed; not yet reviewed for coherence; admissible only as a candidate. |
|
||
| `FALSIFIED` | Incoherent under accumulated evidence; eligible for Stage-3 inversion; retained. |
|
||
|
||
### Non-hardening invariant
|
||
|
||
No reviewed claim or proposition-graph edge ever becomes unrevisable.
|
||
No `final`, `frozen`, `axiom`, or `permanent` flag exists or may be
|
||
added on the runtime data model. The closest such property in the
|
||
architecture is the *mathematical* closure check
|
||
`versor_condition(F) < 1e-6` — never an epistemic seal on a claim.
|
||
The invariant is enforced by `tests/test_epistemic_invariants.py`.
|
||
|
||
### Curator review rule
|
||
|
||
`epistemic_status` transitions are computed from coherence with the
|
||
existing reviewed field — not asserted by source authority. At v1 the
|
||
judgment is curator-mediated, with one rule:
|
||
|
||
> The curator's only admissible reasoning is *geometric*: does the
|
||
> claim cohere with already-reviewed claims, or does it produce
|
||
> incoherence? Source credentials, popularity, or institutional
|
||
> position must not be invoked as justification.
|
||
|
||
### Schema surfaces
|
||
|
||
| Surface | Field | Default at creation |
|
||
|-----------------------------------------|----------------------------------------|-----------------------|
|
||
| `teaching.PackMutationProposal` | `epistemic_status: EpistemicStatus` | `SPECULATIVE` |
|
||
| `teaching.ReviewedTeachingExample` | `epistemic_status: EpistemicStatus` | `SPECULATIVE` |
|
||
| `language_packs.schema.LexicalEntry` | `epistemic_status: str` | `"coherent"` (seed) |
|
||
| `core.cognition.trace.compute_trace_hash` | `teaching_epistemic_status: str` | `""` if no proposal |
|
||
|
||
Promotion of a proposal's status uses the immutable updater
|
||
`PackMutationProposal.with_status(...)` — original is never mutated.
|
||
|
||
The status of the load-bearing proposal in a turn is folded into
|
||
`trace_hash` so replay detects when a downstream surface was produced
|
||
under a different epistemic frame than at the time of recall.
|
||
|
||
## Test organization target
|
||
|
||
Future test moves should follow this taxonomy:
|
||
|
||
| Area | Destination |
|
||
|---|---|
|
||
| versor closure, holonomy, motors, null cone, energy physics | `tests/algebra/` or `tests/physics/` |
|
||
| chat runtime, config, async runtime, identity gate telemetry | `tests/runtime/` |
|
||
| articulation, proposition, surface assembly, future pipeline | `tests/cognition/` |
|
||
| correction capture, reviewed memory, consolidation | `tests/teaching/` |
|
||
| language pack loading and seed pack invariants | `tests/packs/` |
|
||
|
||
Do not reorganize tests as a standalone churn PR unless it directly reduces
|
||
contract ambiguity or unlocks a cognitive subsystem.
|
||
|
||
## Formation trust boundaries
|
||
|
||
The Formation Pipeline (see `docs/formation_pipeline_plan.md`) introduces six
|
||
trust boundaries between the world and the manifold. Every boundary has a
|
||
content-addressed input and output; every rejection produces an audit record.
|
||
No silent failures.
|
||
|
||
| # | Boundary | Input | Output | Trust contract |
|
||
|---|---|---|---|---|
|
||
| 1 | Mining → Smelting | URLs / files | `OreBundle` | Untrusted text in; untrusted entries out. No code execution from sources; no dynamic imports; source URLs sandboxed; SHA-256 captured per entry. |
|
||
| 2 | Smelting → Forge | `OreBundle` + extracted candidates | `Candidate*` lists | Untrusted candidates in. The Forge is the *only* validator. Identity-override patterns and path-traversal in source SHAs are rejected at the Forge, not here. |
|
||
| 3 | Forge → Compose | `Candidate*` lists | `ValidatedTripleSet` | Every candidate runs through `teaching.relation_parse.parse_triple`, identity-axis screening, source allow-list, pack collision check, and the cross-reference rule. Output entries carry `EpistemicStatus.SPECULATIVE`. No pack mutation. |
|
||
| 4 | Compose → Compile/Run | `ValidatedTripleSet` | `CourseYAML` → `FormationPlan` | Deterministic, byte-stable composition. No mutation of the language pack manifest. |
|
||
| 5 | Run → Ratify | `FormationPlan` | `list[CognitiveTurnResult]` | The runner is a thin shim over `CognitiveTurnPipeline.run()`. It cannot invent operators; it can only invoke existing ones. Hard-halts on `versor_condition(F) >= 1e-6`. No identity-manifold mutation, ever. |
|
||
| 6 | Ratify → Promote | `MasteryReport` (self-sealed) | reviewed teaching apply | Promotion requires a self-sealed `MasteryReport` whose SHA verifies, whose prerequisites are present in the `MasteredCoursesIndex`, and whose triples are submitted through `teaching/review.py` — the existing reviewed apply path. ADR-0021's "one mutation path" invariant is preserved. |
|
||
|
||
Content-addressing rules (binding across the whole pipeline):
|
||
|
||
- All hashed payloads are canonical JSON (sorted keys, tight separators,
|
||
UTF-8, no NaN/Infinity). Floats are forbidden in hashed payloads; encode
|
||
numerics as strings or integers.
|
||
- `MasteryReport.report_sha256` is self-sealing: SHA over the payload with
|
||
`report_sha256` blanked, then written back into the field. Verifiers
|
||
reverse the process.
|
||
- No pickle. Pickle defeats replay determinism and is a code-execution
|
||
surface.
|
||
|
||
See `formation/hashing.py`, `formation/cache.py`, and `formation/forge.py`
|
||
for the implementation of each rule.
|
||
|
||
---
|
||
|
||
## Expert-Demo Promotion Contract (ADR-0106 + ADR-0109)
|
||
|
||
Adds a domain-aware, reviewer-signed promotion gate to the capability
|
||
ledger surface (`ledger_report()`). Distinct from the runtime/turn
|
||
contracts above: this contract governs *what the ledger is allowed to
|
||
claim about a domain*, not what the runtime does on any single turn.
|
||
|
||
Per ADR-0108, the contract has been demonstrated end-to-end —
|
||
refused once (ADR-0107), amended once (ADR-0109), succeeded against
|
||
`mathematics_logic` (ADR-0110), and succeeded against `physics`
|
||
without further contract change (ADR-0111).
|
||
|
||
### Surface
|
||
|
||
`ledger_report()` returns a `domains` list. Each row carries:
|
||
|
||
```text
|
||
status ∈ {blocked, seeded, grounded, reasoning-capable, audit-passed}
|
||
predicates.audit_passed bool
|
||
audit_passed_reason str (one-line legibility for operators)
|
||
```
|
||
|
||
A row carries `audit_passed=True` iff **all** of:
|
||
|
||
1. `reasoning_capable == True` (the ADR-0091 nine-predicate gate).
|
||
2. A signed `ExpertDemoClaim` exists in `docs/reviewers.yaml` for the
|
||
domain.
|
||
3. The signer named in `claim.signed_by` has `eval` scope for the
|
||
domain per `ReviewerRegistry.can_review` (ADR-0092).
|
||
4. Every lane in `claim.evidence_lanes` is attached to at least one of
|
||
the domain's ratified packs (no cross-domain bleed).
|
||
5. Every named lane's public + holdout metrics meet the threshold for
|
||
that lane's registered shape (ADR-0109; see §"Lane-shape registry"
|
||
below).
|
||
6. The canonical evidence-bundle SHA-256 reproduces `claim.claim_digest`
|
||
byte-for-byte.
|
||
|
||
Any failure leaves the row at `reasoning-capable` with
|
||
`audit_passed_reason` populated.
|
||
|
||
### Schema
|
||
|
||
`docs/reviewers.yaml` additively gains an `audit_passed_claims` block:
|
||
|
||
```yaml
|
||
audit_passed_claims:
|
||
- domain_id: mathematics_logic
|
||
evidence_lanes:
|
||
- elementary_mathematics_ood
|
||
- inference_closure
|
||
- fabrication_control
|
||
evidence_revision: "adr-0110:reviewed:2026-05-22"
|
||
signed_by: shay-j
|
||
claim_digest: "94d74781..."
|
||
```
|
||
|
||
The block is optional; absence yields zero promotions. Schema
|
||
validation is loud: unknown signer ids, malformed digests, and
|
||
duplicate `domain_id` values are all rejected at load time
|
||
(`core.capability.reviewers.load_reviewer_registry`).
|
||
|
||
### Lane-shape registry (ADR-0109)
|
||
|
||
Threshold rules dispatch by lane shape, not uniformly. Registry in
|
||
`core/capability/expert_demo.py`:
|
||
|
||
| Lane id | Shape | Threshold |
|
||
|---|---|---|
|
||
| `cognition` | `cognition_shape` | surface_groundedness ≥ 0.95, term_capture_rate ≥ 0.85, intent_accuracy ≥ 0.95, versor_closure_rate == 1.0 |
|
||
| `elementary_mathematics_ood` | `accuracy_shape` | accuracy ≥ 0.95 (passed/total fallback) |
|
||
| `foundational_physics_ood` | `accuracy_shape` | same |
|
||
| `symbolic_logic` | `symbolic_logic_shape` | same as `accuracy_shape` |
|
||
| `hebrew_fluency` | `accuracy_shape` | same |
|
||
| `koine_greek_fluency` | `accuracy_shape` | same |
|
||
| `inference_closure` | `inference_shape` | all_pass_rate ≥ 0.95, replay_determinism == 1.0, overall_pass == True |
|
||
| `fabrication_control` | `refusal_shape` | every `by_class` bucket: refused == n, fabricated == 0 |
|
||
|
||
**Unknown lane ids fail closed** with reason
|
||
`lane <id> has no registered shape — introduce via ADR amendment`.
|
||
Adding a lane to the audit-passed surface requires an explicit registry
|
||
entry, which requires an ADR amendment.
|
||
|
||
### Replay invariant
|
||
|
||
`core.capability.audit_passed.derive_evidence_digest` is deterministic
|
||
in field order (sorted keys, compact separators). Re-running it
|
||
against the on-disk lane results at the same `evidence_revision`
|
||
reproduces `claim_digest` byte-for-byte. Drift in any lane result
|
||
demotes the row back to `reasoning-capable` until re-signed.
|
||
|
||
### Fail-closed registry
|
||
|
||
If `docs/reviewers.yaml` fails to parse,
|
||
`reporting._load_registry_for_expert_demo` returns an empty registry
|
||
(zero reviewers, zero claims) rather than raising. Every domain row
|
||
falls back to `audit_passed=false`. A broken registry must never
|
||
silently grant `audit_passed=true`.
|
||
|
||
### Trust boundary
|
||
|
||
- Pack mutation remains proposal-only (ADR-0029/0064 discipline).
|
||
- A reviewer signature in `audit_passed_claims` does not authorize
|
||
pack mutation — only a ledger-row promotion. The two paths remain
|
||
separate.
|
||
- `evidence_revision` may be a labeled string (e.g.
|
||
`adr-0110:reviewed:2026-05-22`) or a raw git sha. The load-bearing
|
||
invariant is replay byte-equality, not git-sha format.
|
||
|
||
---
|
||
|
||
## GSM8K Math Capability Lane (Phase 5 substrate — ADR-0119 + sub-phases)
|
||
|
||
Established by ADR-0119 through ADR-0119.8. The lane sits on the
|
||
parser → solver → verifier → realizer pipeline (ADR-0115 through
|
||
ADR-0118) and adds a scoring runner and a formal lane shape.
|
||
|
||
### Lane shape: `gsm8k_capability_shape`
|
||
|
||
Registered in `LANE_SHAPE_REGISTRY` as `LANE_SHAPE_REGISTRY["gsm8k_math"] =
|
||
"gsm8k_capability_shape"` (ADR-0119.8; `core/capability/expert_demo.py`).
|
||
|
||
The checker (`_check_gsm8k_capability_shape`) is distinct from the existing
|
||
five shapes because the metric keys and composition rule differ.
|
||
|
||
**Required metrics:**
|
||
|
||
| Key | Type | Meaning |
|
||
|---|---|---|
|
||
| `cases_total` | int | Total cases scored; must be > 0 |
|
||
| `correct` | int | Answer matches ground truth AND verifier passes |
|
||
| `wrong` | int | Verifier passed but answer mismatched, OR realizer failed |
|
||
| `refused` | int | ParseError or SolveError (typed refusal) |
|
||
| `overall_pass` | bool | `wrong == 0 AND correct + refused == cases_total` |
|
||
|
||
**Gate conditions (all must hold):**
|
||
|
||
| Condition | Reason |
|
||
|---|---|
|
||
| `cases_total > 0` | Non-empty input |
|
||
| `wrong == 0` | ADR-0114a Obligation #4: zero confabulation |
|
||
| `correct + refused == cases_total` | Outcome accounting completeness |
|
||
| `overall_pass` is true (when present) | Runner self-consistency |
|
||
|
||
The gate passes an all-refused result (0 correct, 0 wrong, N refused)
|
||
intentionally. Whether that qualifies for `expert` promotion is ADR-0120's
|
||
job (it sets the minimum correct_rate); this shape layer verifies only
|
||
runner self-consistency.
|
||
|
||
**Current measurements on main (2026-05-23):**
|
||
|
||
| Split | cases_total | correct | wrong | refused | gate |
|
||
|---|---|---|---|---|---|
|
||
| dev (CORE-original) | 50 | 50 | 0 | 0 | ✓ |
|
||
| public (CORE-original) | 150 | 150 | 0 | 0 | ✓ |
|
||
| holdout (real GSM8K test, sealed) | 1,319 | 0 | 0 | 1,319 | ✓ |
|
||
| adversarial suite | 38 | 5 | 0 | 33 | ✓ |
|
||
|
||
### Empty-expected_unit sentinel rule
|
||
|
||
GSM8K answers are pure numbers with no unit. The runner (ADR-0119.3,
|
||
extended in ADR-0119.7) implements:
|
||
|
||
> An empty `expected_unit` means the case carries no unit-level
|
||
> expectation; the runner skips the unit comparison and grades on
|
||
> answer value alone.
|
||
|
||
This is minimal extension to the outcome rules; the `wrong == 0`
|
||
discipline is preserved.
|
||
|
||
### Seal discipline (ADR-0119.1 + ADR-0119.7)
|
||
|
||
The real GSM8K test set is age-encrypted at
|
||
`evals/gsm8k_math/holdouts/v1/cases.jsonl.age` (420,486 bytes; 1,319
|
||
cases). Three invariants govern seal discipline:
|
||
|
||
1. **Plaintext never on disk.** `scripts/seal_gsm8k_test.py` holds the
|
||
plaintext in memory only during encryption; only ciphertext is written.
|
||
2. **CORE_HOLDOUT_KEY required to decrypt.** The runner raises a typed
|
||
error when the env var is absent. Tests that require decryption skip
|
||
(do not fail) without the key. No CI workflow sets the key.
|
||
3. **Development team operates blind.** The recipient identity lives
|
||
outside the repo at `~/.config/core/holdout_keys/repo_holdout.txt`
|
||
(per ADR-0119.1). A release event signed by a reviewer is required
|
||
to open the lane.
|
||
|
||
Re-encryption and re-seal events go through their own ADR amendment.
|
||
|
||
### Adversarial suite contract
|
||
|
||
`evals/gsm8k_math/adversarial/generator.py` generates 38 cases across 12
|
||
families designed to exploit weak grammar / solver coverage. The gate
|
||
from ADR-0119.5 and ADR-0114a Obligation #8:
|
||
|
||
> **Misparse rate must be zero on the adversarial suite.** A misparse
|
||
> is defined as `outcome == "wrong"` on the adversarial runner output —
|
||
> CORE ran to completion and emitted a wrong answer (confabulation).
|
||
> Refused rate may be arbitrarily high; that is the safe failure mode.
|
||
|
||
The `subtle_in_grammar` family (4 cases, all `correct`) proves the gate
|
||
is not trivially satisfied by refusing everything.
|
||
|
||
### Evidence location
|
||
|
||
- Lane runner: `evals/gsm8k_math/runner.py`
|
||
- Lane shape checker: `core/capability/expert_demo.py`
|
||
- Adversarial: `evals/gsm8k_math/adversarial/`
|
||
- Depth-curve harness: `evals/gsm8k_math/scoring/depth_curve.py`
|
||
- Frontier baseline: `evals/gsm8k_math/baselines/`
|
||
- Sealed holdout: `evals/gsm8k_math/holdouts/v1/cases.jsonl.age`
|
||
- ADR chain: ADR-0119, ADR-0119.1 through ADR-0119.8
|