core-labs/core
core-labs/core
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Volume-honesty invariant + distinct-evidence audit — 21 of 25 ratified deduction bands are short (Phase B) #117

Merged
core-labs merged 1 commit from test/volume-honesty-invariant into main 2026-07-25 23:28:44 +00:00
Conversation 0 Commits 1 Files changed 6 +787 -10
core-labs commented 2026-07-25 23:12:45 +00:00
Owner
Copy link

Phase B of the curriculum-license-loop arc. Stacked on #116 — that PR merges first, then this one. Merge-commit, not squash.

The invariant

conservative_floor is a one-sided Wilson lower bound, and Wilson assumes independent trials. CORE's pipeline is deterministic — that is the founding property — so replaying an identical case is not a second trial; it is the same trial with a guaranteed outcome. Two identical cases in a practice ledger are perfectly correlated and contribute one trial's worth of evidence between them.

So a ledger's committed count is an upper bound on its evidence, and the defensible figure is its distinct-decision count.

Phase B was a precaution. It found the exposure already live.

This unit was scoped to gate a curriculum producer that does not exist yet. Running the instrument against the producers that do exist:

21 of the 25 ratified deduction_serve bands do not clear θ_SERVE = 0.99 on distinct evidence. Three of them inflate 28 distinct cases into 720 committed — honest floor 0.8084 against a claimed 0.9909. deduction_serving_enabled was ratified ON on 2026-07-24 and is live.

band committed distinct × claimed honest clears
en_condmem_disjunctive 720 28 25.7 0.9909 0.8084 NO
en_exist_chain 720 28 25.7 0.9909 0.8084 NO
en_exist_negative 720 28 25.7 0.9909 0.8084 NO
en_exist_witness 720 35 20.6 0.9909 0.8406 NO
atomic 720 98 7.3 0.9909 0.9366 NO
… 16 more 0.9909 0.9779–0.9862 NO
en_verb_fact 720 660 1.1 0.9909 0.9909 yes
en_conditional_chain 720 720 1.0 0.9909 0.9909 yes

Note the claimed column: identical for all 25, because every band commits 720 with a perfect record. The gate sees 25 indistinguishable passes while honest floors span 0.8084–0.9909.

The mechanism is a single constant — CASES_PER_BAND = 720 applied uniformly to bands whose template × vocabulary space is as small as 28 distinct instances. The producer fills the quota by cycling. That it is one constant is part of why it went unnoticed.

This is a regression from an established standard, not an architectural gap

The estimation producer (ADR-0175) is clean: 660 committed, 660 distinct, zero repeats — CASES_PER_CLASS = 660 evidently chosen just above the 657 a perfect record needs. The GSM8K practice corpus is 150/150. So the standard was already set and met before ADR-0256's producer. That is asserted as a test (test_estimation_producer_has_no_inflation), because it is the existence proof that the invariant is satisfiable.

NOT repaired — deliberately

The ledger is SHA-sealed, ratified, and gating a live flag. Re-sealing it is Shay's ratification, not a test's. This PR measures it, pins it in both directions, and writes it down.

wrong = 0 still holds — no band answered anything incorrectly, and no pinned lane was wrong about an answer it gave. What is not established is reliability at the volume claimed: a band that decided 28 distinct questions correctly has earned the floor for 28 (0.8084), not for 720 (0.9909). Nor does this establish that the bands cannot reach 657 distinct — most plainly can, by widening template vocabulary as en_conditional_chain already does. The remedy looks mechanical; the authority to re-seal a ratified ledger is what is not.

What landed

  • core/reliability_gate/evidence.py — pure measurement, imported by no serving path. volume_for_theta() derives 657 from conservative_floor rather than restating it, so a WILSON_Z change cannot leave a stale literal behind in a docstring or test.
  • tests/test_volume_honesty.py — 13 tests. AUDIT_SOURCES is checked against CAPABILITY_LEDGERS, so a licensed capability cannot ship unaudited; the curriculum_serve entry is a declared None that fails the moment its ledger exists. That is Phase C's forcing function against repeating this pattern.
  • docs/research/distinct-evidence-audit-2026-07-25.md — full audit plus the open ruling below.
  • ADR-0264 R9 amended with the measured exposure.

The decision key is case text, deliberately: a tighter key (the normalized atom) would collapse spelling variants and report more inflation, so text-identity under-reports and every number above is a floor on the real gap.

Open ruling for you — outcome mix

Deliberately not pinned here. ClassTally has no verdict axis (correct/wrong only), so a correct UNKNOWN is indistinguishable from a correct ENTAILED and mix can only be enforced at the producer. The deduction producer already balances 240/240/240 by construction, and gold.py says why.

The open question is whether each verdict class needs its own volume. A SERVE license covers entailed, refuted and unknown alike, yet the smallest per-band verdict-class count is 120 (en_atomic) — every class in every band is below 657. Imposing a per-class floor would retroactively fail all 25 bands on a criterion no ADR has ratified, so this audit does not do it. Recorded with numbers instead; probably wants answering once alongside the ADR-0199 per-class-vs-Wilson-floor question.

[Verification]: uv sync --locked on canonical CPython 3.12.13; in-worktree smoke 569 passed (556 baseline + 13), deductive 285 passed. Registered in the smoke curated suite — not left to full, which gates nothing. Mutation-checked: doubling CASES_PER_BAND → 2 failed; reporting committed as distinct → 4 failed; gating on claimed instead of honest → 3 failed; tree restores to 13 passed.

Phase B of the curriculum-license-loop arc. **Stacked on #116** — that PR merges first, then this one. Merge-commit, not squash. ## The invariant `conservative_floor` is a one-sided Wilson lower bound, and **Wilson assumes independent trials.** CORE's pipeline is deterministic — that is the founding property — so replaying an identical case is not a second trial; it is the same trial with a guaranteed outcome. Two identical cases in a practice ledger are perfectly correlated and contribute one trial's worth of evidence between them. So a ledger's `committed` count is an **upper bound** on its evidence, and the defensible figure is its distinct-decision count. ## Phase B was a precaution. It found the exposure already live. This unit was scoped to gate a curriculum producer that does not exist yet. Running the instrument against the producers that *do* exist: > **21 of the 25 ratified `deduction_serve` bands do not clear θ_SERVE = 0.99 on distinct evidence.** Three of them inflate **28 distinct cases into 720 committed** — honest floor **0.8084** against a claimed **0.9909**. `deduction_serving_enabled` was ratified ON on 2026-07-24 and is live. | band | committed | distinct | × | claimed | honest | clears | |---|---|---|---|---|---|---| | en_condmem_disjunctive | 720 | 28 | 25.7 | 0.9909 | 0.8084 | **NO** | | en_exist_chain | 720 | 28 | 25.7 | 0.9909 | 0.8084 | **NO** | | en_exist_negative | 720 | 28 | 25.7 | 0.9909 | 0.8084 | **NO** | | en_exist_witness | 720 | 35 | 20.6 | 0.9909 | 0.8406 | **NO** | | atomic | 720 | 98 | 7.3 | 0.9909 | 0.9366 | **NO** | | … 16 more | | | | 0.9909 | 0.9779–0.9862 | **NO** | | en_verb_fact | 720 | 660 | 1.1 | 0.9909 | 0.9909 | yes | | en_conditional_chain | 720 | 720 | 1.0 | 0.9909 | 0.9909 | yes | Note the `claimed` column: **identical for all 25**, because every band commits 720 with a perfect record. The gate sees 25 indistinguishable passes while honest floors span 0.8084–0.9909. The mechanism is a single constant — `CASES_PER_BAND = 720` applied uniformly to bands whose template × vocabulary space is as small as 28 distinct instances. The producer fills the quota by cycling. That it is one constant is part of why it went unnoticed. ## This is a regression from an established standard, not an architectural gap The **`estimation`** producer (ADR-0175) is clean: 660 committed, 660 distinct, zero repeats — `CASES_PER_CLASS = 660` evidently chosen just above the 657 a perfect record needs. The GSM8K practice corpus is 150/150. So the standard was already set and met before ADR-0256's producer. That is asserted as a test (`test_estimation_producer_has_no_inflation`), because it is the existence proof that the invariant is satisfiable. ## NOT repaired — deliberately The ledger is SHA-sealed, ratified, and gating a live flag. Re-sealing it is Shay's ratification, not a test's. This PR measures it, pins it **in both directions**, and writes it down. **`wrong = 0` still holds** — no band answered anything incorrectly, and no pinned lane was wrong about an answer it gave. What is *not* established is reliability at the volume claimed: a band that decided 28 distinct questions correctly has earned the floor for 28 (0.8084), not for 720 (0.9909). Nor does this establish that the bands *cannot* reach 657 distinct — most plainly can, by widening template vocabulary as `en_conditional_chain` already does. The remedy looks mechanical; the authority to re-seal a ratified ledger is what is not. ## What landed - **`core/reliability_gate/evidence.py`** — pure measurement, imported by no serving path. `volume_for_theta()` **derives** 657 from `conservative_floor` rather than restating it, so a `WILSON_Z` change cannot leave a stale literal behind in a docstring or test. - **`tests/test_volume_honesty.py`** — 13 tests. `AUDIT_SOURCES` is checked against `CAPABILITY_LEDGERS`, so a licensed capability cannot ship unaudited; the `curriculum_serve` entry is a declared `None` that **fails the moment its ledger exists**. That is Phase C's forcing function against repeating this pattern. - **`docs/research/distinct-evidence-audit-2026-07-25.md`** — full audit plus the open ruling below. - **ADR-0264 R9** amended with the measured exposure. The decision key is case **text**, deliberately: a tighter key (the normalized atom) would collapse spelling variants and report *more* inflation, so text-identity under-reports and every number above is a **floor** on the real gap. ## Open ruling for you — outcome mix Deliberately **not** pinned here. `ClassTally` has no verdict axis (`correct`/`wrong` only), so a correct UNKNOWN is indistinguishable from a correct ENTAILED and mix can only be enforced at the producer. The deduction producer already balances **240/240/240** by construction, and `gold.py` says why. The open question is whether each verdict class needs its **own** volume. A SERVE license covers entailed, refuted and unknown alike, yet the smallest per-band verdict-class count is **120** (`en_atomic`) — every class in every band is below 657. Imposing a per-class floor would retroactively fail all 25 bands on a criterion no ADR has ratified, so this audit does not do it. Recorded with numbers instead; probably wants answering once alongside the ADR-0199 per-class-vs-Wilson-floor question. [Verification]: `uv sync --locked` on canonical CPython 3.12.13; in-worktree smoke **569 passed** (556 baseline + 13), deductive **285 passed**. Registered in the **`smoke`** curated suite — not left to `full`, which gates nothing. Mutation-checked: doubling `CASES_PER_BAND` → 2 failed; reporting `committed` as `distinct` → 4 failed; gating on `claimed` instead of `honest` → 3 failed; tree restores to 13 passed.
core-labs added 1 commit 2026-07-25 23:12:46 +00:00 
Phase B of the curriculum-license-loop arc. Stacked on the ADR-0264 branch.

The invariant: conservative_floor is a one-sided Wilson lower bound and Wilson
assumes INDEPENDENT trials. CORE's pipeline is deterministic, so replaying an
identical case is not a second trial -- it is the same trial with a guaranteed
outcome. A ledger's `committed` count is an upper bound on its evidence; the
defensible figure is its distinct-decision count.

Phase B was scoped as a precaution for a curriculum producer that does not
exist yet. Running the instrument against the producers that DO exist found it
already violated:

  21 of the 25 ratified deduction_serve bands do not clear theta_SERVE=0.99 on
  distinct evidence. Three inflate 28 distinct cases into 720 committed
  (honest floor 0.8084 vs claimed 0.9909). deduction_serving_enabled was
  ratified ON 2026-07-24 and is live.

The estimation producer (ADR-0175) is clean -- 660 committed, 660 distinct,
zero repeats, evidently chosen just above the 657 a perfect record needs. So
this is a regression from a standard already established in the repo, not an
architectural gap. `claimed` is identical for all 25 deduction bands because
every band commits 720 with a perfect record: the gate sees 25
indistinguishable passes while honest floors span 0.8084..0.9909.

NOT REPAIRED. The ledger is SHA-sealed, ratified, and gating a live flag, so
re-sealing it is Shay's ratification, not a test's. wrong=0 still holds -- no
band answered anything incorrectly. What is not established is reliability at
the volume claimed. Measured, pinned in both directions, written down.

Added:
- core/reliability_gate/evidence.py -- pure measurement, imported by no serving
  path. volume_for_theta() DERIVES 657 from conservative_floor rather than
  restating it, so a WILSON_Z change cannot leave a stale literal behind.
- tests/test_volume_honesty.py -- 13 tests. AUDIT_SOURCES is checked against
  CAPABILITY_LEDGERS, so a new licensed capability cannot ship unaudited; the
  curriculum entry is a declared None that fails the moment its ledger exists.
  That is Phase C's forcing function against repeating the pattern.
- docs/research/distinct-evidence-audit-2026-07-25.md -- full audit + the open
  outcome-mix ruling for Shay.
- ADR-0264 R9 amended with the measured exposure.

Decision key is case TEXT deliberately: a tighter key (normalized atom) would
collapse spelling variants and report MORE inflation, so text-identity
under-reports and every number is a floor on the real gap.

Outcome mix is deliberately NOT pinned. ClassTally has no verdict axis, the
deduction producer already balances 240/240/240 by construction, and imposing
a per-verdict-class floor would retroactively fail all 25 bands on a criterion
no ADR has ratified (smallest per-band class count is 120). Recorded for a
ruling with numbers instead.

[Verification]: uv sync --locked on canonical CPython 3.12.13; in-worktree
smoke 569 passed (556 baseline + 13), deductive 285 passed. Registered in the
`smoke` curated suite -- not left to `full`, which gates nothing. Mutation-
checked: doubling CASES_PER_BAND -> 2 failed; reporting committed as distinct
-> 4 failed; gating on `claimed` instead of `honest` -> 3 failed; tree restores
to 13 passed.
core-labs changed target branch from docs/adr-negative-curriculum to main 2026-07-25 23:28:28 +00:00
core-labs merged commit dc11d44d29 into main 2026-07-25 23:28:44 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: core-labs/core#117
No description provided.
Delete branch "test/volume-honesty-invariant"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?