Commit graph

2 commits

Author SHA1 Message Date
Shay
c1d08ff9d5 feat(curriculum): sealed-practice producer for curriculum serving (Phase C)
Builds the producer `chat/curriculum_serve_license.py` already named as the
only writer of the ledger it reads — `evals.curriculum_serve.practice.runner
.seal_ledger` — which did not exist. Until now, authoring curriculum could
not earn a license at any volume.

A band's case space is ENUMERATED, not authored: every exam question that
routes to that subject (resolve_domain) whose relation is in that family, one
committed case per distinct query atom. So `committed == distinct` is not a
property the generator maintains, it is the same fact twice — the structural
form of ADR-0264 R9, and why this producer cannot develop the deduction
producer's exposure (CASES_PER_BAND=720 filled from spaces as small as 28).
CASES_PER_BAND=660 follows `evals/determination_estimation` per
DIVISION-OF-WORK §4: a ceiling on committed volume, not a quota.

11 bands, wrong=0, inflation 1.0 everywhere.

THE LEDGER IS DELIBERATELY NOT COMMITTED. The plan's Phase-1 exit criterion
asked for "a real (still-unearned) ledger"; that state is unreachable.
Reliability is commitment precision and a correct UNKNOWN is a commitment, so
conservative_floor(660,660)=0.990046 clears theta_SERVE on non-commitments
alone. Sealing today licenses FOUR bands on evidence that is 99.0%-99.98%
correct non-commitment — which ADR-0262 §5.1 rules unacceptable and which
ClassTally structurally cannot see (no verdict axis). Writing the artifact is
therefore a ratification and belongs to Phase D's explicit human-run verb.
The open outcome-mix ruling is now blocking, not academic.

Two corrections to my own earlier records, both measured:

- ADR-0264 §4.2 sized bands from per-TERM exclusivity; the router's predicate
  is per-PAIR, which is looser. Every ceiling was understated, and
  systems_software_causal is 720 not 630 — flipping it from "cannot reach
  657" to "can". 4 of 11 bands can reach the floor, not 3. Now pinned as
  measured values in BAND_ATOM_SPACE.
- `then` and `therefore` are taught philosophy_theology lemmas AND control
  words of the argument reader (`therefore` is the conclusion marker). 164
  routable atoms refuse compiled_premises_unreadable — coverage misses, never
  wrongs — in the band Phase F retargets to. The vocabulary boundary is not
  screened against the reader's grammar; a pack teaching and/or/not/if would
  open a much larger hole the same way. Recorded, not fixed (§6.5), pinned as
  a two-lemma bound.

Registers curriculum_serve in AUDIT_SOURCES, retiring the declared `None`
that was Phase C's forcing function, and adds the producer to `deductive`.

[Verification]: in-worktree on canonical CPython 3.12.13 with `uv sync
--locked`: smoke 571 (569 + 1 test + 1 parametrization), deductive 310
(291 + 19). Mutation-checked: injecting the deduction padding pattern into
band_cases turns 5 tests red including the R9 audit and the seal guard.
curriculum_serve lane unchanged at n=32 correct=32 wrong=0 anti_recall=5;
all 11 lane SHA pins match; no ledger written, so no serving change.
2026-07-26 09:52:54 -07:00
Shay
13a4b05d62 test(reliability): volume-honesty invariant + distinct-evidence audit
Phase B of the curriculum-license-loop arc. Stacked on the ADR-0264 branch.

The invariant: conservative_floor is a one-sided Wilson lower bound and Wilson
assumes INDEPENDENT trials. CORE's pipeline is deterministic, so replaying an
identical case is not a second trial -- it is the same trial with a guaranteed
outcome. A ledger's `committed` count is an upper bound on its evidence; the
defensible figure is its distinct-decision count.

Phase B was scoped as a precaution for a curriculum producer that does not
exist yet. Running the instrument against the producers that DO exist found it
already violated:

  21 of the 25 ratified deduction_serve bands do not clear theta_SERVE=0.99 on
  distinct evidence. Three inflate 28 distinct cases into 720 committed
  (honest floor 0.8084 vs claimed 0.9909). deduction_serving_enabled was
  ratified ON 2026-07-24 and is live.

The estimation producer (ADR-0175) is clean -- 660 committed, 660 distinct,
zero repeats, evidently chosen just above the 657 a perfect record needs. So
this is a regression from a standard already established in the repo, not an
architectural gap. `claimed` is identical for all 25 deduction bands because
every band commits 720 with a perfect record: the gate sees 25
indistinguishable passes while honest floors span 0.8084..0.9909.

NOT REPAIRED. The ledger is SHA-sealed, ratified, and gating a live flag, so
re-sealing it is Shay's ratification, not a test's. wrong=0 still holds -- no
band answered anything incorrectly. What is not established is reliability at
the volume claimed. Measured, pinned in both directions, written down.

Added:
- core/reliability_gate/evidence.py -- pure measurement, imported by no serving
  path. volume_for_theta() DERIVES 657 from conservative_floor rather than
  restating it, so a WILSON_Z change cannot leave a stale literal behind.
- tests/test_volume_honesty.py -- 13 tests. AUDIT_SOURCES is checked against
  CAPABILITY_LEDGERS, so a new licensed capability cannot ship unaudited; the
  curriculum entry is a declared None that fails the moment its ledger exists.
  That is Phase C's forcing function against repeating the pattern.
- docs/research/distinct-evidence-audit-2026-07-25.md -- full audit + the open
  outcome-mix ruling for Shay.
- ADR-0264 R9 amended with the measured exposure.

Decision key is case TEXT deliberately: a tighter key (normalized atom) would
collapse spelling variants and report MORE inflation, so text-identity
under-reports and every number is a floor on the real gap.

Outcome mix is deliberately NOT pinned. ClassTally has no verdict axis, the
deduction producer already balances 240/240/240 by construction, and imposing
a per-verdict-class floor would retroactively fail all 25 bands on a criterion
no ADR has ratified (smallest per-band class count is 120). Recorded for a
ruling with numbers instead.

[Verification]: uv sync --locked on canonical CPython 3.12.13; in-worktree
smoke 569 passed (556 baseline + 13), deductive 285 passed. Registered in the
`smoke` curated suite -- not left to `full`, which gates nothing. Mutation-
checked: doubling CASES_PER_BAND -> 2 failed; reporting committed as distinct
-> 4 failed; gating on `claimed` instead of `honest` -> 3 failed; tree restores
to 13 passed.
2026-07-25 16:03:10 -07:00