diff --git a/docs/research/independent-comprehension-agreement-gate.md b/docs/research/independent-comprehension-agreement-gate.md new file mode 100644 index 00000000..fa6e0f86 --- /dev/null +++ b/docs/research/independent-comprehension-agreement-gate.md @@ -0,0 +1,173 @@ +# Independent Comprehension Agreement Gate + +**Status:** research-control document +**Date:** 2026-06-04 +**Scope:** non-serving; defines the gate that must exist before any new field-backed or GSM8K-positive capability promotion. + +## 1. Problem + +The recent GSM8K post-mortem established the central failure mode: a committing reader can appear safe on a built-against slice while being unsound on held-out or sealed data. The repair was not another recognizer branch; it was the independent-gold discipline (INV-25) plus the universal-structure plan. + +PRs #554-#558 now give CORE the foundation: + +- independent-gold invariant and SHA-pinned deductive lane; +- binding-graph interlingua neutrality (INV-26); +- finite-entity grounding into propositional entailment; +- dimensional reasoning with an independent dimensional oracle; +- a 3-domain anti-overfit panel. + +The remaining unsolved problem is not solving after a structure is already extracted. It is **independent comprehension**: can two genuinely independent readers construct the same canonical problem structure from the same task, and only then solve or verify it? + +## 2. Doctrine + +```text +A second solver over the same extracted structure is not a second derivation. +A second derivation begins with an independently constructed structure. +``` + +Agreement must therefore be tested at the structure boundary, not only at the final answer boundary. + +Bad independence: + +```text +shared parser -> symbolic solver +shared parser -> field/geometric solver +``` + +Good independence: + +```text +symbolic reader -> canonical structure +field/geometric reader -> canonical structure +structure-equivalence gate -> oracle/proof solver -> commitment eligibility +``` + +## 3. Gate definition + +A candidate capability may become promotion-eligible only if it passes all of the following gates. + +| Gate | Requirement | Failure result | +|---|---|---| +| Reader independence | The two readers do not share parser, candidate generator, gold, or structure-building code. | Refuse / research failure | +| Canonical structure | Both readers emit a canonical, hashable structure with target, facts, relations, dimensions, and provenance. | Refuse | +| Structure equivalence | Structures are equal under a domain-specific equivalence relation stricter than same-answer. | Refuse | +| Independent gold | A proof checker, oracle, sealed set, or other non-SUT arbiter agrees. | Wrong/fail | +| Replay | Structure hashes and verdict traces are deterministic across replay. | Fail | +| Promotion wall | No serving path changes until held-out/sealed wrong=0 holds. | Block promotion | + +## 4. Canonical structure minimum + +Every participating structure must carry at least: + +```text +structure_id +source_case_id +entities +quantities +units_or_types +relations +question_target +provenance_spans_or_hashes +unsupported_features +trace_sha256 +``` + +The target is load-bearing. Two readers that agree on facts but bind different targets have not agreed. + +## 5. Equivalence semantics + +The first equivalence primitive should be intentionally narrow. + +Equivalent: + +- same target; +- same required entities; +- same quantity values after canonical numeric normalization; +- same dimensions/types; +- same relation kind and operands; +- same proof obligations; +- same unused-evidence classification. + +Not equivalent: + +- same numeric answer but different relation; +- same relation but different target; +- one structure ignores a required source quantity; +- one structure treats a distractor as relevant and the other exempts it; +- units/dimensions differ; +- either structure depends on an unsupported inferred fact. + +## 6. Disagreement taxonomy + +Closed disagreement reasons for the first implementation: + +```text +missing_target +relation_mismatch +quantity_mismatch +dimension_mismatch +entity_binding_mismatch +unused_evidence_mismatch +unsupported_feature_mismatch +same_answer_different_structure +oracle_disagreement +reader_not_independent +``` + +These reasons should feed future contemplation/practice reports directly. The learning signal should identify the failed primitive, not merely the domain class. + +## 7. Required proof of reader independence + +A reader pair must provide a `DecoderIndependenceProof`-style record: + +```text +reader_a_module +reader_b_module +shared_import_allowlist +forbidden_shared_modules +oracle_module +uses_same_parser: false +uses_same_candidate_generator: false +uses_same_gold: false +``` + +The test should include at least one AST/import scan like INV-25/INV-26, plus at least one behavioral non-vacuity test proving the guard can fail. + +## 8. First implementation target + +The first implementation should not be GSM8K and should not be broad natural language. + +Recommended target: + +```text +structured quantitative-relational lane + -> symbolic structured compiler + -> field/geometric structured compiler + -> canonical structure equivalence + -> independent quantitative oracle +``` + +This gives the field side a domain where geometry/metric relations are native, while avoiding the prior GSM8K failure mode. + +## 9. Stop conditions + +Stop and document instead of patching around the result if: + +1. the field reader imports or calls the symbolic reader; +2. the two readers only agree on final answers, not structures; +3. equivalence admits same-answer/different-reason cases; +4. the oracle shares code with either reader; +5. a proposed gate passes only on built-against examples; +6. any route to serving is proposed before held-out/sealed wrong=0. + +## 10. PR sequence + +1. **Structure-equivalence primitive** — `core/reasoning/structure_equivalence.py` plus tests. +2. **Quantitative-relational structured lane** — independent oracle and structured cases. +3. **Field quantitative decoder prototype** — no symbolic reader imports. +4. **Controlled NL compiler** — only after structured symbolic/field agreement exists. +5. **GSM8K structure diagnostic** — non-serving, coverage-only. + +## 11. Success definition + +The first milestone succeeds when a supported case can be read by two independent readers into equivalent canonical structures, solved against independent gold, replayed deterministically, and refused on disagreement without touching serving.