diff --git a/docs/research/field-reasoner-wedge-selection.md b/docs/research/field-reasoner-wedge-selection.md new file mode 100644 index 00000000..4505ca91 --- /dev/null +++ b/docs/research/field-reasoner-wedge-selection.md @@ -0,0 +1,252 @@ +# Field-Reasoner Wedge Selection + +**Status:** research-control document +**Date:** 2026-06-04 +**Scope:** identifies the first honest field-as-reasoner target after the deductive pivot and 3-domain anti-overfit panel. + +## 1. Context + +The universal-structure arc established three facts: + +1. Deductive propositional logic is a checkable substrate, not the best field wedge. +2. A clean algebraic/geometric propositional decoder still behaved as `O(2^n)` enumeration, so logic is combinatorial terrain. +3. Field independence must live in **reading/comprehension**, not merely in solving the same extracted structure. + +Therefore the next field-reasoner wedge must be a domain where geometric/metric structure is natural, where independent gold is available, and where success helps return to quantitative comprehension without reopening the unsafe GSM8K serving bridge. + +## 2. Candidate domains + +| Candidate | Benefit | Risk | Decision | +|---|---|---|---| +| Pure propositional logic | Exact gold already exists | Combinatorial; field route degenerates to enumeration | Reject as first wedge | +| Finite-entity logic | Good grounding compiler | Still Boolean after lowering | Keep as substrate, not field wedge | +| Dimensional reasoning | Already independently golded; unit algebra is metric/type-like | Too narrow if only product/quotient | Use as supporting axis | +| Ratio/proportion | Metric-relational; direct bridge to GSM8K | Needs target binding and structure equivalence | **Choose** | +| Part-whole residuals | Natural quantitative structure | Prior GSM8K traps if promoted early | Include after ratio | +| Euclidean geometry | True geometric territory | Bigger encoding surface | Later wedge | +| Systems/software execution | Excellent independent gold | Not geometric; sandboxing surface | Later fourth panel domain | + +## 3. Selected wedge + +The first honest field-as-reasoner wedge is: + +```text +structured quantitative relations: + ratios + proportions + rates + part-whole residuals + dimensional consistency +``` + +This domain is selected because it is: + +- close enough to GSM8K to matter; +- structured enough to avoid uncontrolled natural-language ambiguity; +- naturally metric/geometric rather than purely Boolean; +- independently checkable with a small oracle; +- compatible with the binding-graph interlingua and dimensional lane. + +## 4. Minimal structured schema + +### Ratio/proportion + +```json +{ + "id": "qr-v1-0001", + "family": "proportion", + "given": [ + {"name": "flour", "value": 2, "unit": "cup"}, + {"name": "servings", "value": 8, "unit": "count"} + ], + "target": {"known": {"name": "servings", "value": 20, "unit": "count"}, "unknown": "flour"}, + "gold": {"value": 5, "unit": "cup"} +} +``` + +Canonical relation: + +```text +flour / servings = 2 / 8 +x_flour / 20_servings = 2 / 8 +x_flour = 5 +``` + +### Rate + +```json +{ + "id": "qr-v1-0002", + "family": "rate", + "given": [ + {"name": "distance", "value": 60, "unit": "mile"}, + {"name": "time", "value": 2, "unit": "hour"} + ], + "target": {"unknown": "speed"}, + "gold": {"value": 30, "unit": "mile_per_hour"} +} +``` + +### Part-whole residual + +```json +{ + "id": "qr-v1-0003", + "family": "part_whole", + "whole": {"name": "apples", "value": 30, "unit": "count"}, + "parts": [{"name": "sold", "value": 12, "unit": "count"}], + "target": {"unknown": "remaining"}, + "gold": {"value": 18, "unit": "count"} +} +``` + +## 5. Independent oracle + +The quantitative-relations oracle must not import the symbolic compiler, field decoder, or binding-graph implementation. + +Oracle responsibilities: + +```text +validate schema +reject unsupported families +compute exact rational result +compute/canonicalize result unit +return typed refusal for unsupported or malformed cases +``` + +Use exact rational arithmetic in the oracle. Floating tolerance should not be part of v1. + +Closed oracle outcomes: + +```text +correct +wrong +refused_malformed +refused_unsupported_family +refused_dimension_conflict +refused_divide_by_zero +``` + +## 6. Symbolic compiler + +The symbolic compiler is allowed to be direct and typed. It reads the structured JSON and emits canonical quantitative structure. + +It should not solve by side effect. It emits: + +```text +entities +quantities +relation_edges +target +unit_constraints +provenance_hash +``` + +The solver/oracle consumes that structure separately. + +## 7. Field/geometric compiler + +The field compiler must be independent of the symbolic compiler. + +Permitted: + +- import field/algebra primitives; +- import neutral binding-graph dataclasses, once the structure boundary exists; +- emit the same canonical structure type; +- use dimensional unit primitives only through an allowlisted neutral boundary, if needed. + +Forbidden: + +- importing the symbolic quantitative compiler; +- importing its parser/case lowering helpers; +- consuming the symbolic compiler's output; +- matching only final numeric answers. + +The first field encoding may be modest. Acceptable candidates: + +| Encoding | Why acceptable in v1 | +|---|---| +| dimensioned vector relation | Minimal metric structure | +| affine scalar constraint | Good for rates and residuals | +| proportional line relation | Natural geometric ratio representation | +| CGA point/line meet | Stronger, later variant if v1 succeeds | + +## 8. Required tests + +Minimum test stack: + +1. oracle handles all v1 families with exact rational results; +2. oracle refuses malformed/unsupported/divide-by-zero cases; +3. symbolic compiler emits canonical structure without solving; +4. field compiler emits canonical structure without importing symbolic compiler; +5. structure equivalence catches same-answer/different-relation; +6. `symbolic_structure == field_structure` on supported cases; +7. `answer == oracle_gold` on supported cases; +8. disagreement produces refusal, not fallback to one reader; +9. replay hash stability; +10. AST independence tests for oracle and field compiler. + +## 9. Initial fixture shape + +Start with 24-32 cases. + +| Family | Positive | Negative/refusal | +|---|---:|---:| +| proportion | 6 | 2 | +| rate | 4 | 2 | +| part-whole residual | 4 | 2 | +| dimensional consistency | 4 | 2 | +| mixed distractor/exempt | 2 | 2 | + +Do not author hundreds of cases. This is not a corpus-tuning lane. It is a mechanism proof lane. + +## 10. GSM8K re-entry rule + +GSM8K remains out of serving scope until this wedge proves itself. + +Allowed after the structured lane: + +```text +GSM8K structure diagnostic only +``` + +Forbidden until held-out/sealed gates pass: + +```text +serving bridge +train_sample-only promotion +regex positive branch +field solver fallback on symbolic parse +``` + +## 11. Research outputs + +The wedge is successful only if it produces all of: + +1. a structured quantitative-relations lane with independent gold; +2. a symbolic compiler that emits canonical structure; +3. a field/geometric compiler that independently emits equivalent structure; +4. an equivalence gate that refuses on structural disagreement; +5. deterministic evidence traces; +6. no serving changes. + +## 12. Next PR stack + +1. `core/reasoning/structure_equivalence.py` — canonical structure equivalence. +2. `evals/quantitative_relations/oracle.py` — exact independent oracle. +3. `evals/quantitative_relations/v1/cases.jsonl` — small mixed fixture. +4. `generate/field_reasoning/quantitative_decoder.py` — field reader prototype. +5. `evals/gsm8k_math/structure_diagnostic/` — later, non-serving. + +## 13. Stop conditions + +Stop if: + +- field structure construction cannot be made independent; +- the field route is only a second solver; +- ratios/proportions require broad NL parsing before structured schema works; +- oracle and SUT disagree and the cause is not understood; +- same-answer/different-structure slips through equivalence; +- a serving path becomes tempting before sealed evidence exists. + +The correct outcome of a failed wedge is a falsification record, not a workaround. diff --git a/docs/research/independent-comprehension-agreement-gate.md b/docs/research/independent-comprehension-agreement-gate.md new file mode 100644 index 00000000..fa6e0f86 --- /dev/null +++ b/docs/research/independent-comprehension-agreement-gate.md @@ -0,0 +1,173 @@ +# Independent Comprehension Agreement Gate + +**Status:** research-control document +**Date:** 2026-06-04 +**Scope:** non-serving; defines the gate that must exist before any new field-backed or GSM8K-positive capability promotion. + +## 1. Problem + +The recent GSM8K post-mortem established the central failure mode: a committing reader can appear safe on a built-against slice while being unsound on held-out or sealed data. The repair was not another recognizer branch; it was the independent-gold discipline (INV-25) plus the universal-structure plan. + +PRs #554-#558 now give CORE the foundation: + +- independent-gold invariant and SHA-pinned deductive lane; +- binding-graph interlingua neutrality (INV-26); +- finite-entity grounding into propositional entailment; +- dimensional reasoning with an independent dimensional oracle; +- a 3-domain anti-overfit panel. + +The remaining unsolved problem is not solving after a structure is already extracted. It is **independent comprehension**: can two genuinely independent readers construct the same canonical problem structure from the same task, and only then solve or verify it? + +## 2. Doctrine + +```text +A second solver over the same extracted structure is not a second derivation. +A second derivation begins with an independently constructed structure. +``` + +Agreement must therefore be tested at the structure boundary, not only at the final answer boundary. + +Bad independence: + +```text +shared parser -> symbolic solver +shared parser -> field/geometric solver +``` + +Good independence: + +```text +symbolic reader -> canonical structure +field/geometric reader -> canonical structure +structure-equivalence gate -> oracle/proof solver -> commitment eligibility +``` + +## 3. Gate definition + +A candidate capability may become promotion-eligible only if it passes all of the following gates. + +| Gate | Requirement | Failure result | +|---|---|---| +| Reader independence | The two readers do not share parser, candidate generator, gold, or structure-building code. | Refuse / research failure | +| Canonical structure | Both readers emit a canonical, hashable structure with target, facts, relations, dimensions, and provenance. | Refuse | +| Structure equivalence | Structures are equal under a domain-specific equivalence relation stricter than same-answer. | Refuse | +| Independent gold | A proof checker, oracle, sealed set, or other non-SUT arbiter agrees. | Wrong/fail | +| Replay | Structure hashes and verdict traces are deterministic across replay. | Fail | +| Promotion wall | No serving path changes until held-out/sealed wrong=0 holds. | Block promotion | + +## 4. Canonical structure minimum + +Every participating structure must carry at least: + +```text +structure_id +source_case_id +entities +quantities +units_or_types +relations +question_target +provenance_spans_or_hashes +unsupported_features +trace_sha256 +``` + +The target is load-bearing. Two readers that agree on facts but bind different targets have not agreed. + +## 5. Equivalence semantics + +The first equivalence primitive should be intentionally narrow. + +Equivalent: + +- same target; +- same required entities; +- same quantity values after canonical numeric normalization; +- same dimensions/types; +- same relation kind and operands; +- same proof obligations; +- same unused-evidence classification. + +Not equivalent: + +- same numeric answer but different relation; +- same relation but different target; +- one structure ignores a required source quantity; +- one structure treats a distractor as relevant and the other exempts it; +- units/dimensions differ; +- either structure depends on an unsupported inferred fact. + +## 6. Disagreement taxonomy + +Closed disagreement reasons for the first implementation: + +```text +missing_target +relation_mismatch +quantity_mismatch +dimension_mismatch +entity_binding_mismatch +unused_evidence_mismatch +unsupported_feature_mismatch +same_answer_different_structure +oracle_disagreement +reader_not_independent +``` + +These reasons should feed future contemplation/practice reports directly. The learning signal should identify the failed primitive, not merely the domain class. + +## 7. Required proof of reader independence + +A reader pair must provide a `DecoderIndependenceProof`-style record: + +```text +reader_a_module +reader_b_module +shared_import_allowlist +forbidden_shared_modules +oracle_module +uses_same_parser: false +uses_same_candidate_generator: false +uses_same_gold: false +``` + +The test should include at least one AST/import scan like INV-25/INV-26, plus at least one behavioral non-vacuity test proving the guard can fail. + +## 8. First implementation target + +The first implementation should not be GSM8K and should not be broad natural language. + +Recommended target: + +```text +structured quantitative-relational lane + -> symbolic structured compiler + -> field/geometric structured compiler + -> canonical structure equivalence + -> independent quantitative oracle +``` + +This gives the field side a domain where geometry/metric relations are native, while avoiding the prior GSM8K failure mode. + +## 9. Stop conditions + +Stop and document instead of patching around the result if: + +1. the field reader imports or calls the symbolic reader; +2. the two readers only agree on final answers, not structures; +3. equivalence admits same-answer/different-reason cases; +4. the oracle shares code with either reader; +5. a proposed gate passes only on built-against examples; +6. any route to serving is proposed before held-out/sealed wrong=0. + +## 10. PR sequence + +1. **Structure-equivalence primitive** — `core/reasoning/structure_equivalence.py` plus tests. +2. **Quantitative-relational structured lane** — independent oracle and structured cases. +3. **Field quantitative decoder prototype** — no symbolic reader imports. +4. **Controlled NL compiler** — only after structured symbolic/field agreement exists. +5. **GSM8K structure diagnostic** — non-serving, coverage-only. + +## 11. Success definition + +The first milestone succeeds when a supported case can be read by two independent readers into equivalent canonical structures, solved against independent gold, replayed deterministically, and refused on disagreement without touching serving.