From ec9102a8d6cb9ea9996e9e92055a003f43012cfb Mon Sep 17 00:00:00 2001 From: Shay Date: Tue, 9 Jun 2026 09:41:34 -0700 Subject: [PATCH] docs(verified): scope serving-time verification gate --- ...ified-serving-wiring-scoping-2026-06-09.md | 240 ++++++++++++++++++ 1 file changed, 240 insertions(+) create mode 100644 docs/analysis/verified-serving-wiring-scoping-2026-06-09.md diff --git a/docs/analysis/verified-serving-wiring-scoping-2026-06-09.md b/docs/analysis/verified-serving-wiring-scoping-2026-06-09.md new file mode 100644 index 00000000..145256d1 --- /dev/null +++ b/docs/analysis/verified-serving-wiring-scoping-2026-06-09.md @@ -0,0 +1,240 @@ +# VERIFIED Serving Wiring Scoping — verified_serving_enabled and Gold-Free Independence + +## 1. Current State + +Currently, the VERIFIED verification pipeline exists off-serving only. No served [verified] surface exists, no serving-time config or gate exists, and no `verify.py` serving wiring is implemented. + +The existing off-serving spine consists of: +- **P1-A**: Defines the contract in `core/epistemic_disclosure/verified_contract.py`. This contract includes: + - `VerificationProof` (the data structure containing digests and lineages) + - `VerificationObligation` (the strict checklist of obligations) + - `evaluate_verification` (the pure-logic evaluation function enforcing the contract) + - `disclosure_for_verification` (the single sanctioned route to transition a result to the verified state) + - Defines the meaning of **VERIFIED**: two independent reads must converge on a single canonical structure. This requires: + - Independent reads (`primary_reader_lineage` must not equal `independent_reader_lineage`) + - Convergent canonical read digests (`primary_read_digest` must equal `independent_read_digest`) + - `derivation_digest` present + - `bound_slots_digest` present + - `back_substitution_digest` present + - `boundary_clear` is `True` + - `contradiction_clear` is `True` + - `limitation` is `None` +- **P1-B**: Adds an off-serving R2 verification producer in `evals/constraint_oracle/verified_producer.py`: + - Extracts the primary read from the problem text via `read_constraint_problem(text)`. + - Obtains the independent read from a hand-authored gold setup (`gold_setup`). + - The gold answer itself never enters the verification process (the gold structure setup signature is matched, not the answer). + - The producer is strictly for evaluation and runs off-serving only. +- **P1-C**: Introduces `bound_slots_digest` as a distinct, separable proof obligation to ensure the answer binds only to the stated slots (asked unknowns) and not phantom slots. +- **Serving Isolation**: + - No served VERIFIED surface exists. + - No `verified_serving_enabled` gate exists. + - No `verify.py` serving integration exists. + +## 2. Hard Non-Claim + +The gold-setup-backed producer implemented in `evals/constraint_oracle/verified_producer.py` **cannot** be used for serving. + +What P1-B proves: +- The VERIFIED contract logic is sound and functional. +- The verification producer can successfully assemble valid `VerificationProof` objects. +- Wrong reader structures diverge and correctly fail the independent verification check. +- The actual gold answer is not required to verify the canonical structural solve. + +What P1-B does **NOT** prove: +- Serving-time independence (since gold setups do not exist for runtime user requests). +- Gold-free verification. +- User-visible served `[verified]` status. +- Benchmark or `CLAIMS.md` movement. +- AGI capabilities. + +Any served `[verified]` status requires a gold-free independent read source at serving time. + +## 3. Proposed Served Gate: verified_serving_enabled + +To control the rollout of serving-time verification, a future configuration gate must be defined: + +- The configuration field `verified_serving_enabled` does not exist yet. +- This document does not add or implement this configuration gate. +- The future gate must be default-false and fail-closed: if the config field is missing or malformed, it must evaluate to `False`. +- The gate must only control the served/user-visible `[verified]` surface. +- The gate must never affect off-serving evaluation proof generation or off-serving validation runs. +- The gate must strictly prohibit eval-gold-backed verification pipelines from touching serving code. +- Enabling the gate is necessary but not sufficient: even if `verified_serving_enabled` is `True`, a served VERIFIED verdict is only allowed if all contract proof obligations successfully pass. + +| Config State | verified_serving_enabled | Served VERIFIED Allowed? | +| --- | --- | --- | +| missing field | False | No | +| default config | False | No | +| explicit false | False | No | +| explicit true | True | Only if gold-free proof obligations pass | + +## 4. Gold-Free Independent Read Requirement + +Before any served `[verified]` surface can exist, a serving proof must use a gold-free independent read source. + +A served proof must adhere to the following: +- Primary reader lineage is recorded. +- Independent reader lineage is recorded. +- The lineages must use distinct lineage identifiers. +- The primary read digest and independent read digest must converge. +- No gold setup is used. +- No gold answer is used. +- No benchmark fixture is used. +- No eval-lane imports are performed. +- Same reader twice (invoking the exact same reader lineage twice on the same text) is strictly rejected. +- Second solver over one read (running two solvers over the same read signature) is strictly rejected. + +### Comparison + +* **Invalid (Pseudo-Independence):** + ```text + primary read ──> solver A + └───> solver B ──> same answer ──> VERIFIED (INVALID) + ``` +* **Valid (True Independence):** + ```text + primary read lineage ───────────> canonical digest C ──┐ + ├──> convergent digests ──> VERIFIED (VALID) + gold-free independent lineage ──> canonical digest C ──┘ + ``` + +The valid workflow requires: +- Primary read lineage converges to canonical digest `C`. +- Gold-free independent read lineage converges to canonical digest `C`. +- Derivation is computed from stated quantities. +- Bound slots are present. +- Back-substitution succeeds. +- No boundary has fired. +- No contradiction is present. +- No unresolved limitation exists. + +The primary unsolved technical milestone is designing and implementing a gold-free independent reader/verifier source. + +## 5. Serving Eligibility Criteria + +For any response to be eligible for served `[verified]` status, all of the following conditions must be met: + +- `source_problem_digest` is present. +- `primary_reader_lineage` is present. +- `independent_reader_lineage` is present. +- `primary_reader_lineage` and `independent_reader_lineage` are distinct. +- `primary_read_digest` is present. +- `independent_read_digest` is present. +- `primary_read_digest` and `independent_read_digest` are identical (converge). +- `derivation_digest` is present. +- `bound_slots_digest` is present. +- `back_substitution_digest` is present. +- `boundary_clear` is `True`. +- `contradiction_clear` is `True`. +- `limitation` is `None`. +- `evaluate_verification(...)` evaluates to `VerificationVerdict.VERIFIED`. +- `disclosure_for_verification(...)` is the only functional pathway used to transition the result to `(EpistemicState.VERIFIED, DisclosureClaim.VERIFIED)`. +- `verified_serving_enabled` is explicitly configured to `True`. +- The proof producer is gold-free and serving-eligible. + +## 6. Holdout and Kill-Switch Gates + +Before any served `[verified]` behavior can be wired to production, the serving-time verifier must clear a set of holdout and kill-switch gates: +- **Sealed Holdout Lane**: A dedicated evaluation run on a sealed, off-distribution holdout dataset. +- **Wrong=0 Requirement**: Zero incorrect answers are permitted for any result designated as VERIFIED. +- **Poison Fixtures**: Comprehensive verification tests using poison test cases to prove that: + - Wrong-read structures diverge and fail. + - Same-reader-twice reads fail verification. + - Answer matches gold, but missing proof fails verification. + - Absence of refusal alone without proof fails verification. + - Missing `bound_slots_digest` fails verification. + - Triggered boundaries fail verification. + - Contradictions present fail verification. + - Unresolved limitations fail verification. +- **Deterministic Replay Digest**: The verification trace must be replayable, yielding identical digests. +- **Kill-Switch**: Default off / fail-closed behavior must be validated. +- **Gate Disabled Test**: A test must verify that disabling `verified_serving_enabled` suppresses the served `[verified]` label even when a fully valid proof is present. +- **Gate Enabled Test**: A test must verify that enabling `verified_serving_enabled` still requires all proof obligations to pass before serving. + +## 7. verify.py Boundary + +The transition of the verification verdict to serving must enforce strict boundaries at the `verify.py` layer: +- `verify.py` may eventually consume a verified verdict from the contract. +- `verify.py` must not define or redefine the semantic meaning of VERIFIED. +- The formal meaning of VERIFIED remains exclusively defined in `core/epistemic_disclosure/verified_contract.py`. +- `verify.py` must never attempt to repair failed proofs. +- `verify.py` must not treat raw answer correctness (matching gold) as verification. +- `verify.py` must not import or use eval-lane gold datasets. +- No hot-path repair or silent corrections may occur in `verify.py`. + +## 8. Served Behavior Matrix + +| Proof / Gate State | Served Behavior | +| --- | --- | +| Valid off-serving eval-gold proof | Never served | +| Valid gold-free proof + gate disabled | No served `[verified]` | +| Valid gold-free proof + gate enabled | Served `[verified]` may be allowed | +| Wrong read divergence | No served `[verified]` | +| Same reader twice | No served `[verified]` | +| Answer matches gold but proof missing | No served `[verified]` | +| No refusal but no proof | No served `[verified]` | +| Boundary fired | No served `[verified]` | +| Contradiction present | No served `[verified]` | +| Unresolved limitation | No served `[verified]` | + +## 9. Required Future Tests Before Wiring + +The following test suite must be implemented and pass before any serving-time code can land: +- `verified_serving_enabled` defaults to `False`. +- A missing config field for `verified_serving_enabled` resolves to `False`. +- Explicit `True` configuration is required to serve `[verified]`. +- A disabled gate suppresses served `[verified]` even when a valid proof exists. +- An enabled gate still enforces and requires all proof obligations to pass. +- Eval-gold verification producers cannot be imported or accessed by serving modules (AST / dependency check). +- Eval-gold-backed proofs are rejected from being served. +- Same reader twice fails verification. +- Running a second solver over one reading fails verification. +- Answer-gold match without verification proof does not serve `[verified]`. +- Absence of refusal alone does not serve `[verified]`. +- Direct construction of `EpistemicState.VERIFIED` is blocked outside of `disclosure_for_verification`. +- Poison wrong-read inputs do not serve `[verified]`. +- Missing `bound_slots_digest` does not serve `[verified]`. +- Triggered boundary does not serve `[verified]`. +- Contradiction present does not serve `[verified]`. +- Unresolved limitation does not serve `[verified]`. +- `verify.py` consumes the contract outcome but does not define its rules. + +## 10. Non-Claims + +This scoping document establishes architectural boundaries only. + +- This document does not implement served VERIFIED. +- This document does not add the `verified_serving_enabled` config gate. +- This document does not wire `verify.py` to serving. +- This document does not move the evaluation producer (`verified_producer.py`) into the core serving layer. +- This document does not make gold-backed verification serving-safe. +- This document does not change benchmark metrics. +- This document does not update `CLAIMS.md`. +- This document does not solve the design of the gold-free independent reader. +- This document does not claim AGI progress. + +## 11. Recommended Next Slices + +The implementation of serving-time verification should proceed in isolated, review-gated slices: + +### Slice 1: Configuration Gate +- Add default-dark `verified_serving_enabled` helper only. +- No runtime wiring, no `verify.py` wiring. +- Strict import checks preventing eval producer imports. +- Tests demonstrating default-dark behavior and config defaults. + +### Slice 2: Gold-Free Independent Reader +- Design and prototype a gold-free independent reader/verifier source. +- Maintain off-serving isolation. +- Generate independent reader lineage and canonical digests. +- No integration with the served path. + +### Slice 3: Verification Harness & Poison Fixtures +- Implement a holdout-gated verification harness. +- Integrate poison fixtures and test cases. +- Validate deterministic replay digests. +- Do not expose any served surface. + +### Slice 4: verify.py Consumption +- Only after all prior slices are approved and pass tests, scope the consumption of the verified verdict within `verify.py`. +- No implementation without separate architectural review.