From e6b59cc585026b2e15b870fa3b256e872ac805f0 Mon Sep 17 00:00:00 2001 From: Shay Date: Fri, 24 Jul 2026 16:44:41 -0700 Subject: [PATCH] =?UTF-8?q?docs(generalization):=20Tier=20S=20evidence=20p?= =?UTF-8?q?acket=20=E2=80=94=20ratification=20record=20+=20curriculum-volu?= =?UTF-8?q?me=20quantification=20(S1,=20S6)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...iculum-volume-quantification-2026-07-24.md | 106 ++++++++++ .../ratification-packet-2026-07-24.md | 196 ++++++++++++++++++ 2 files changed, 302 insertions(+) create mode 100644 docs/research/curriculum-volume-quantification-2026-07-24.md create mode 100644 docs/research/ratification-packet-2026-07-24.md diff --git a/docs/research/curriculum-volume-quantification-2026-07-24.md b/docs/research/curriculum-volume-quantification-2026-07-24.md new file mode 100644 index 00000000..c425bf1d --- /dev/null +++ b/docs/research/curriculum-volume-quantification-2026-07-24.md @@ -0,0 +1,106 @@ +# Curriculum-Volume Quantification — ADR-0262 §5.1 + +**Tier S6** (generalization-arc-2026-07-24 §6). Evidence only. This +document does not propose ratifying curriculum content or flipping +`curriculum_serving_enabled` — both are Shay's decisions +([[project-curriculum-serving-ratified]] doctrine: a flag flips on earned +evidence, not proximity to it). It answers one question precisely: *how +short of a SERVE license is each served `(subject × relation family)` band +today, measured against the actual ratified corpora, not an estimate.* + +Method: `teaching.curriculum_premises.load_curriculum(domain)` — the same +loader the curriculum composer uses in serving — run against each of the +four subjects `chat.curriculum_surface.SERVED_DOMAINS` currently routes to. +Counts below are chain counts (ratified, vocabulary-resident, `review_status +== "reviewed"`), i.e. exactly the population `compile_premises` would draw +premises from; nothing here touches unmounted or unratified content. + +## 1. The floor, restated precisely (§1.2 of the ratification packet) + +`Action.SERVE` needs a one-sided Wilson lower bound ≥ θ=0.99 +(`core/reliability_gate/ceilings.py`, `core/reliability_gate/floor.py`). For +a perfect record this reduces to `committed / (committed + z²)` with +`z = 2.576`, which crosses 0.99 at **committed ≥ 657**. + +657 committed cases alone is not sufficient — the plan (§6) and the +ratification packet both require a **genuine outcome mix**, not 657 +restatements of the same fact padded with paraphrase (repetition-padding +was already considered and rejected as dishonest earlier in this arc: +`conservative_floor` cannot distinguish "657 independent facts" from "16 +facts asked 41 times each," but a ledger that gamed the floor that way +would not have *earned* anything). Treating "genuine mix" as roughly a +three-way split across `entailed` / `refuted` / `unknown` (the plan's own +working assumption, not a formally ratified policy — flagged here as an +assumption, not fact) puts the **entailed bucket's floor at ≈219 distinct +taught relations** — `⌈657 / 3⌉`. That is where the plan's "≈219 per +subject × family" figure comes from; this document supplies the measured +`n` for every currently-served band to compare against it. + +## 2. Measured ratified-chain counts, per subject × family (2026-07-24) + +| domain | family | ratified chains (n) | 657÷n | 219÷n | +|---|---|---:|---:|---:| +| physics | causal | 7 | 94× | 31× | +| physics | modal | 9 | 73× | 24× | +| mathematics_logic | modal | 4 | 164× | 55× | +| mathematics_logic | sequence | 4 | 164× | 55× | +| mathematics_logic | contrast | 8 | 82× | 27× | +| mathematics_logic | evidential | 8 | 82× | 27× | +| systems_software | causal | 7 | 94× | 31× | +| systems_software | modal | 6 | 110× | 37× | +| systems_software | sequence | 3 | 219× | 73× | +| philosophy_theology | modal | 8 | 82× | 27× | +| philosophy_theology | contrast | 8 | 82× | 27× | + +11 bands are live (a subject × family pair with ≥1 ratified chain routes +questions today, per `chat.curriculum_surface.band_for`); every one is +between 24× and 73× short of the entailed-bucket floor alone, before even +authoring the matching refuted/unknown volume. **No served subject has a +family close enough to flag as "next."** Systems-software `sequence` (n=3) +is the thinnest; mathematics_logic `modal`/`sequence` (n=4 each) are tied +for second-thinnest; physics `modal` (n=9) is the closest of the eleven, +still 24× short. + +Total ratified chains per subject (all families, informational context — +not a band-level floor since bands are scored per family, not per subject): +physics 16, mathematics_logic 24, systems_software 16, +philosophy_theology 16. + +## 3. Vocabulary is not the constraint + +`Curriculum.vocabulary` (every lemma the subject's mounted packs teach) is +16 lemmas for physics/mathematics_logic/systems_software and **151** for +philosophy_theology — an order of magnitude more vocabulary than chains in +every subject. The gate `untaught_vocabulary` (§4.6) is not what is holding +any band back; the terms are taught, the *relations between them* are not +authored in ratified volume. This confirms ADR-0262 §5.1's framing: the +blocker is curriculum content authorship, not reader/vocabulary coverage — +see also S3's mechanism-vs-coverage instrument +(`docs/research/vocab-trigger-instrument-2026-07-24.md`), which finds the +curriculum path's own refusals are overwhelmingly coverage-class for a +different reason (relation, not term, absent) while carrying zero +`untaught_vocabulary` refusals in the committed lane corpus. + +## 4. What one subject fully reaching the floor would look like + +Taking physics `modal` (the closest band, n=9) as an illustration only: it +would need roughly 210 more ratified `requires`/`enables` relations, +authored and reviewed through the existing curriculum ratification +pipeline, before an entailed bucket alone could support a 657-committed +band — and that is before authoring the matching refuted/unknown volume +this document is not attempting to size (refuted questions need explicit +negative curriculum content the corpora do not currently carry at all; +unknown questions are comparatively cheap to author from +in-vocabulary-but-untaught pairings, so they are not expected to be the +binding constraint). + +## 5. Non-goals of this document + +- Does not recommend which subject or family to grow first — that is + content strategy, Shay's call. +- Does not estimate authoring effort/time (no clock-time estimates per + standing convention). +- Does not touch `curriculum_serving_enabled` or any corpus file. + +Relates to [[project-curriculum-grounded-serving]], +[[project-generalization-arc]], `docs/adr/ADR-0262-curriculum-grounded-serving.md` §5.1. diff --git a/docs/research/ratification-packet-2026-07-24.md b/docs/research/ratification-packet-2026-07-24.md new file mode 100644 index 00000000..a46bd931 --- /dev/null +++ b/docs/research/ratification-packet-2026-07-24.md @@ -0,0 +1,196 @@ +# Ratification Packet — `deduction_serving_enabled` and `curriculum_serving_enabled` + +**Tier S1** (generalization-arc-2026-07-24 §6). Evidence collation only — the +flag flip is Shay's decision alone, never inferred from this document or from +approval of adjacent work. This packet has two independent sections: one +recording a decision already made, one recording why a decision has not been +made. + +--- + +## 1. `deduction_serving_enabled` — RATIFIED ON (2026-07-24, main @ `5fd9a861`) + +This section is a **record**, not an input to the decision. Shay ratified the +flag flip on 2026-07-24; the evidence below is what stood behind that +ratification, preserved for audit and for anyone rolling back or re-deriving +the decision later. + +### 1.1 Sealed-practice ledger — 25 bands, 720/720 wrong=0 each + +`chat/data/deduction_serve_ledger.json` (schema `deduction_serve_ledger_v1`, +provenance `evals.deduction_serve.practice.runner.seal_ledger`, content +`a056f899…`). The engine only *reads* this artifact — it is committed, +SHA-sealed, and never written by serving. + +| # | classes | committed | correct | wrong | refused | +|---|---|---|---|---|---| +| 25 | see below | 720 each | 720 each | **0** | 0 | + +Classes (25): `atomic`, `categorical`, `conditional_chain`, +`conditional_single`, `disjunctive` (Band v1/v1b) · `en_atomic`, +`en_conditional_chain`, `en_conditional_single`, `en_disjunctive` (Band +v2-EN, ADR-0257) · `en_member_atomic`, `en_member_chain`, +`en_member_negative`, `en_member_single` (Band v3-MEM, ADR-0258) · +`en_condmem_chain`, `en_condmem_conditional`, `en_condmem_disjunctive`, +`en_condmem_fused` (Band v4-CM, ADR-0259) · `en_verb_chain`, `en_verb_fact`, +`en_verb_negative`, `en_verb_universal` (Band v5-VP, ADR-0260) · +`en_exist_chain`, `en_exist_negative`, `en_exist_universal`, +`en_exist_witness` (Band v6-EX, ADR-0261). + +### 1.2 Earned-license gate ceilings (ADR-0175 §4a, `core/reliability_gate/`) + +`Action.SERVE` ceiling is θ=0.99 (`core/reliability_gate/ceilings.py`). The +floor is a pinned one-sided Wilson lower bound +(`core/reliability_gate/floor.py::conservative_floor`), deterministic and +replay-stable (no learned weights, no sampling). For a perfect record +(`successes == committed`) the bound reduces to +`committed / (committed + z²)` with `z = WILSON_Z = 2.576`; solving +`committed / (committed + z²) ≥ 0.99` gives `committed ≥ 99·z² ≈ 656.9`, +i.e. **n ≥ 657** — where the standing "n≥657 committed" figure comes from, +not a chosen round number. All 25 classes commit 720 ≥ 657 with a perfect +record, clearing the floor with margin. + +### 1.3 Serve-lane results — production pipeline, independent gold + +`evals/deduction_serve/report.json` runs the *exact* pipeline +`chat/deduction_surface.py` calls in serving (shape-gate → reader → +projector → ROBDD engine), scored against independently-authored gold — +**166/166 correct, 0 wrong, 0 declined**, across six splits: + +| split | n | band | +|---|---|---| +| v1 | 28 | propositional / categorical | +| v2_en | 26 | English opaque-clause arguments | +| v2_member | 26 | singular-membership arguments | +| v2_condmem | 26 | conditional-membership fusion | +| v2_verb | 28 | verb-predicate arguments | +| v2_exist | 32 | existential arguments | + +`wrong` staying 0 is the lane's non-negotiable gate (a decline is a coverage +miss, tracked but never conflated with a wrong answer); `all_correct=true` +confirms every committed case, not just the wrong-count, matches gold. + +### 1.4 Commit gate — narrow by construction + +`chat/deduction_surface.py::looks_like_deductive_argument` is a single +regex: a sentence-initial `"therefore"` conclusion clause +(`_ARGUMENT_CONCLUSION_RE`). This is a genuine signal of intent — text +shaped that way IS an argument — so the composer can commit on shape alone +without risking hijacking ordinary turns. (Contrast with curriculum's gate, +§2.4 below, which cannot use shape alone.) Once committed, every branch is +fail-closed (INV-34): a decided verdict, a typed reader refusal, or the +generic out-of-band surface — never a silent fall-through to a different +composer. + +### 1.5 Blast radius (verified before the flip, not assumed) + +- Deductive suite (`core test --suite deductive`) — 268 passed with the flag + on. +- `chat/runtime.py::_maybe_pack_grounded_surface` — the deduction branch is + checked **before** the empty-vault gate (unlike pack/teaching/partial/oov): + a pure function of input text, independent of vault/field warm-or-cold + state, so it cannot perturb any other dispatch path's preconditions. A + claimed turn sets `grounding_source="deduction"`; every other path is + reached exactly as before when the composer declines + (`DispatchAttempt(source="deduction", outcome="skipped", ...)`). +- Lane pins (`scripts/verify_lane_shas.py`) stayed 10/11 across the flip — + unchanged, i.e. the flip touched nothing any other pinned lane measures. + +### 1.6 Rollback + +One line: `deduction_serving_enabled: bool = True` → `False` in +`core/config.py`. Flag-off is byte-identical to pre-arc dispatch — pinned by +`tests/test_deduction_surface.py::test_flag_off_preserves_pack_token_gloss_byte_identity`. +No residue: the composer is a pure function gated entirely by the one flag +read at `chat/runtime.py:1748`; nothing it touches persists state. + +### 1.7 Unearned shapes stay disclosed, not withheld + +Every band above is licensed. A future band, or a deployment whose ledger is +stripped, still gets the sound ROBDD answer — served DISCLOSED via the +`_UNVERIFIED_SHAPE_DISCLOSURE` hedge (`chat/deduction_surface.py`), never +presented as verified. This is what makes serving an *earned* capability +rather than a merely flagged one, and it is why disclosure is not itself +evidence against ratifying a *licensed* band. + +--- + +## 2. `curriculum_serving_enabled` — stays OFF (ADR-0262) + +This section exists to make the non-decision as auditable as the decision +above: **every curriculum band is unearned**, so flipping this flag would +enable a capability that answers *disclosed*, never authoritatively — sound, +but with no demonstrated track record on the pipeline serving it. + +### 2.1 Same shape of evidence, opposite conclusion + +`evals/curriculum_serve/report.json`: physics lane **32/32 correct, 0 wrong, +0 declined**, `all_correct=true`, 5 anti-recall probes (≥ the §4.7 floor of +3). This is a clean lane pass — the mechanism is proven — but it says +nothing about *volume*, which is the actual gate. + +### 2.2 Why no band can earn SERVE from present data (ADR-0262 §5.1) + +The reliability floor (§1.2 above) needs **n ≥ 657 committed** per band, and +a band here is `(subject × relation family)` — e.g. +`curriculum_physics_causal`. Physics teaches 7 causal + 9 modal relations +total (§6 below has the full count), so **at most 16 questions in the +entire subject can ever come back `entailed`** — roughly 25× short of a +single band's floor, before even asking for a *mix* of entailed/refuted/ +unknown outcomes (a real reliability claim needs more than one outcome +class repeated). This is a volume ceiling on the ratified curriculum, not a +defect in the reader, the bands, or the engine — all of which are the same +verified components §1 already licensed. + +### 2.3 The ledger says it plainly + +Every one of the curriculum bands present in a ledger would show `committed` +in the low tens, nowhere near 657. (No curriculum-specific sealed ledger +exists yet, precisely because sealing one at this volume would just +document the shortfall — see [[project-curriculum-grounded-serving]] and +S6, `docs/research/curriculum-volume-quantification-2026-07-24.md`.) + +### 2.4 Commit gate — routability, not shape (ADR-0262 §5.3, already fixed) + +Unlike deduction's `"therefore"` gate, curriculum's `"Does …?"` shape is not +itself a signal of intent — it is one of the most common ways to open *any* +English question. `chat/curriculum_surface.py::is_curriculum_question` +commits only when the question parses to three tokens AND both terms are +vocabulary some served subject actually teaches (`resolve_domain`); an +`untaught_vocabulary` refusal is explicitly **not** a claim (the composer +declines the turn entirely, byte-identical to before this composer +existed). This was a real defect found and fixed this arc (commit-gate +firing on any `Does …?` text, e.g. "Does the build pass?"); the fix is +already on `main` and is not part of what remains gated by this flag. + +### 2.5 What flipping ON today would mean + +Every served answer would carry the `_UNVERIFIED_SHAPE_DISCLOSURE` hedge — +"(answered from my curriculum, but I haven't yet earned a verified track +record on questions of this shape)" — because `_license_gate` +(`chat/curriculum_surface.py`) finds no `Action.SERVE` license for any +`curriculum__` band. The capability would be real and +sound (same argument bands, same ROBDD engine as §1) but never +authoritative. This is not unsafe to ship; it is simply not yet *earned*, +and the standing rule (Shay, 2026-07-24) is that a flag flips on earned +evidence, not on proximity to it — see +[[project-deduction-serving-ratified]]. + +### 2.6 Rollback / blast radius, for parity with §1.6 + +Same shape as deduction: one line in `core/config.py` +(`curriculum_serving_enabled: bool = False`, already the default — nothing +to roll back), checked immediately after the deduction branch in +`chat/runtime.py:1764`, pure function of text + ratified curriculum, no +vault/field dependency. Flag-off is byte-identical to pre-arc dispatch by +construction (the branch is dead code when the flag is False). + +--- + +## 3. What would close §2 (evidence only — not a proposal to act) + +Nothing in Tier S closes this gap; it closes only when ratified curriculum +content grows roughly 25× per served `(subject × family)` — a content +question for Shay, not an engineering one. See S6's quantification, +`docs/research/curriculum-volume-quantification-2026-07-24.md`, for the +per-subject arithmetic.