From 5c77c9eeceaefa015015aa0d3375cc080cb84bac Mon Sep 17 00:00:00 2001 From: Shay Date: Thu, 4 Jun 2026 19:44:22 -0700 Subject: [PATCH] =?UTF-8?q?feat(field-wedge):=20ablation=20verdict=20?= =?UTF-8?q?=E2=80=94=20field=20is=20decoration=20on=20additive=20(C3)=20(P?= =?UTF-8?q?hase=20W.2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The falsifiable experiment's measurements #2 (ablation) and #3 (diversity). Builds the competent, code-disjoint SYMBOLIC reader (the control arm AND the C3 capability path) and the ablation instrument that runs both readers through the real verify_tier2_agreement gate. VERDICT: C3 — the field is decoration on this domain (a sanctioned, honest negative): - field_wrong_commits = [] (wrong=0 holds; the per-step drift guard refuses bad ints) - field_caught_symbolic_errors = [] (the field caught ZERO symbolic errors) - per-class diversity = 0 everywhere (both readers agree and are both correct) - the only admitted-set change is the field LOSING coverage at the precision ceiling. Insight: on forward-substitutable relations, geometric translation IS arithmetic addition, so there is no metric over-determination for the field to exploit — field and symbol are common-mode (Knight-Leveson), not a genuine second derivation. This is the deductive finding's twin: logic was combinatorial (field can't earn it), additive is arithmetically trivial (field adds nothing). The field needs metric-nontrivial AND arithmetically-hard structure to earn a reasoning role — dedicated research, not near-term. Field-as-reasoner is NOT earned; no field vote enters any serving path; the field stays a servant. Capability path = symbolic (C3), not shipped here. - generate/relational_symbolic_reader.py: competent independent reader (pure int). - evals/relational_metric/ablation.py: the reusable decoration instrument. - docs/analysis/field-wedge-ablation-result-2026-06-04.md: the recorded verdict. All prior artifacts STAY (field reader = real wrong=0 read demo + 3rd panel domain). Green: full wedge suite 104; 53 architectural invariants. --- .../field-wedge-ablation-result-2026-06-04.md | 83 +++++++++ evals/relational_metric/ablation.py | 162 ++++++++++++++++++ generate/relational_symbolic_reader.py | 127 ++++++++++++++ tests/test_relational_ablation.py | 52 ++++++ tests/test_relational_symbolic_reader.py | 59 +++++++ 5 files changed, 483 insertions(+) create mode 100644 docs/analysis/field-wedge-ablation-result-2026-06-04.md create mode 100644 evals/relational_metric/ablation.py create mode 100644 generate/relational_symbolic_reader.py create mode 100644 tests/test_relational_ablation.py create mode 100644 tests/test_relational_symbolic_reader.py diff --git a/docs/analysis/field-wedge-ablation-result-2026-06-04.md b/docs/analysis/field-wedge-ablation-result-2026-06-04.md new file mode 100644 index 00000000..5147af4b --- /dev/null +++ b/docs/analysis/field-wedge-ablation-result-2026-06-04.md @@ -0,0 +1,83 @@ + + +# The field-reasoner wedge ablation — result: C3 (decoration on this domain) + +This records the outcome of the falsifiable experiment designed in +[`field-reasoner-wedge-design-and-falsification-2026-06-04.md`](./field-reasoner-wedge-design-and-falsification-2026-06-04.md). +The verdict is the dossier's sanctioned negative: **the field reads correctly but is +decoration here.** That is a success — it answers the field-as-reasoner question with +evidence instead of deferring it. + +## What was measured + +Two readings of the same forward-substitutable quantitative-relational micro-domain +(additive / part-whole), scored against an independent arithmetic oracle: + +- **Field reader** (`generate/relational_field_reader.py`) — reads TEXT into conformal + points on the e1 number line; additive relations are conformal translator versors; + the answer reads back by projective dehomogenization. (The geometric SUT.) +- **Symbolic reader** (`generate/relational_symbolic_reader.py`) — a competent, + code-disjoint reading by plain integer arithmetic; no `algebra` (INV-27 disjoint). + +The ablation (`evals/relational_metric/ablation.py`) runs both through the real +`verify_tier2_agreement` gate and asks the only questions that matter. + +## Result + +| Measurement | Value | Meaning | +|---|---|---| +| **field_wrong_commits** | `[]` | wrong=0 holds — the field never commits a bad integer (the per-step drift guard refuses instead). | +| **field_caught_symbolic_errors** | `[]` | The field caught **zero** comprehension errors the symbolic reader made. (Measurement #2 = fail.) | +| **field_lost_coverage** | `[rm-v1-0015]` | The only admitted-set change is the field **refusing a correct answer** at the precision ceiling — a liability, not signal. | +| **per-class diversity** | `0` everywhere | On every committable class the readers AGREE and are BOTH CORRECT: `disagree=0`, `double_fault=0`. (Measurement #3 = no diversity.) | +| **verdict** | **`C3`** | The field is decoration / a coverage liability on this domain. | + +## Why — the load-bearing insight + +**On forward-substitutable relations, geometric translation *is* arithmetic addition.** +`versor_apply(T_δ, embed[x]) = embed[x+δ]` is exactly `x + δ`. There is no metric +over-determination — no betweenness, incidence, or continuous-constraint structure — +for the field to exploit that a competent symbolic reader lacks. So the two readings are +**redundant** (the common-mode the dossier predicted via Knight–Leveson), and agreement +adds no independent soundness signal. The field's *one* genuine marginal check +(over-determination coherence) is matched by the symbolic reader's `over_determined_conflict` +refusal, so even there it is not unique. And the field's precision ceiling makes it a +strict coverage liability versus pure integer arithmetic. + +This is the deductive-logic finding's twin: there, logic was *combinatorial* not metric, +so the field could not earn it; here, the metric domain is *arithmetically trivial*, so +the geometry adds nothing. **The field earns a reasoning role only where geometry +genuinely exceeds the symbolic alternative — not yet demonstrated on any built domain.** + +## What this changes + +- **Field-as-reasoner is NOT earned.** No field vote enters any serving or capability + path. The field stays a servant (its existing cognition-turn role). No "the field + reasons" claim is made. +- **The capability path is symbolic (C3).** The symbolic reader is correct (15/15 here, + including the case the field refuses). Shipping quantitative capability safely is the + dossier's Phase 3+ on the symbolic path — **two** code-disjoint symbolic readings + agreeing + independent/sealed gold (one reader alone is the resolve_pooled risk). Not + built here; this slice answered the experiment, it did not ship serving. +- **The `t2_precision` lever stays dormant** for this domain — there is no genuine second + derivation to feed it (field⟂symbol is common-mode here). + +## What stays (all load-bearing, none reverted) + +- The **f64 conformal foundation** (Phase 0A) — net infrastructure hardening. +- The **reader-lineage firewall + INV-27** (Phase 0B/0C) — backs any future agreement gate. +- The **field reader + relational-metric lane** — a real wrong=0 demonstration that the + field *can* read (Phase W.1), and a valid 3rd independently-golded panel domain. +- The **symbolic reader + ablation instrument** — the C3 reader and a reusable decoration + test for the next domain. + +## The honest next bet (dedicated research, not the near-term sequence) + +Field-as-reasoner remains an open question for a domain where geometry genuinely beats +arithmetic: **continuous / over-determined / multi-constraint** structure (relative +position, betweenness, incidence among many points, ratio as cross-ratio on a +non-degenerate frame) — where a symbolic step-arithmetic reader and a field constraint +reading could genuinely diverge, and agreement would be a real second derivation. Until +such a domain is built and the ablation shows `field_caught_symbolic_errors > 0`, the +field stays a servant. Two negative domains (logic: combinatorial; additive: trivial) +now bound the search: the field needs *metric-nontrivial, arithmetically-hard* structure. diff --git a/evals/relational_metric/ablation.py b/evals/relational_metric/ablation.py new file mode 100644 index 00000000..1c17b896 --- /dev/null +++ b/evals/relational_metric/ablation.py @@ -0,0 +1,162 @@ +"""The decoration instrument — does the geometric FIELD reader add independent signal? + +Measurements #2 (ablation) and #3 (per-class diversity) of the field-reasoner wedge +falsifiable experiment. For each case it runs BOTH readers and the real +``verify_tier2_agreement`` gate, then asks the only questions that matter: + +- **field_caught_symbolic_errors** — cases where the SYMBOLIC reader commits a WRONG + answer (≠ gold) but the agreement gate REFUSES (the field disagreed or refused). + This is the dossier's measurement-#2 PASS signal: the field refusing a comprehension + error the symbol path alone admits. +- **field_lost_coverage** — cases where the symbolic reader commits the CORRECT answer + but the gate refuses *because of the field* (field refused or disagreed). The field + subtracting a correct answer is a liability, not signal. +- **admitted_set_changed** — does the gate's admitted set differ from symbolic-alone? +- per-class double-fault / agreement diversity. + +VERDICT: +- ``PASS`` iff ``field_caught_symbolic_errors > 0`` (the field adds error-catching + signal) and never commits a wrong answer. +- ``C3`` (field is decoration / a coverage liability on this domain) otherwise — a + sanctioned, honest outcome: the symbolic reading carries the capability. + +Run: PYTHONPATH=. .venv/bin/python -m evals.relational_metric.ablation +""" + +from __future__ import annotations + +import json +import sys +from collections import defaultdict +from pathlib import Path +from typing import Any + +from core.reasoning import OperatorEvidence, verify_tier2_agreement +from evals.relational_metric.oracle import OracleError, oracle_answer +from generate.relational_field_reader import READER_LINEAGE as FIELD_LINEAGE +from generate.relational_field_reader import read_relational as field_read +from generate.relational_symbolic_reader import READER_LINEAGE as SYM_LINEAGE +from generate.relational_symbolic_reader import read_relational as sym_read + +_CASES = Path(__file__).resolve().parent / "v1" / "cases.jsonl" + + +def _commit_key(answer: int, unit: str | None) -> str: + return f"{answer}:{unit}" + + +def _gate_admits(field_r: Any, sym_r: Any) -> int | None: + """Run the real Tier-2 gate; return the agreed answer or None (refuse).""" + if field_r.refused or sym_r.refused: + return None + ev_field = OperatorEvidence( + domain="relational_metric", operator="field_read", outcome="committed", + reason="field_commit", input_keys=(), check_keys=(), + commitment_key=_commit_key(field_r.answer, field_r.answer_unit), + structural_signature="field.geometric.number_line", + reader_lineage=FIELD_LINEAGE, + ) + ev_sym = OperatorEvidence( + domain="relational_metric", operator="symbolic_read", outcome="committed", + reason="symbolic_commit", input_keys=(), check_keys=(), + commitment_key=_commit_key(sym_r.answer, sym_r.answer_unit), + structural_signature="symbolic.arithmetic.schema", + reader_lineage=SYM_LINEAGE, + ) + verdict = verify_tier2_agreement((ev_field, ev_sym)) + return field_r.answer if verdict.verified else None + + +def run() -> dict[str, Any]: + cases = [json.loads(l) for l in _CASES.read_text().splitlines() if l.strip()] + + rows: list[dict[str, Any]] = [] + per_class: dict[str, dict[str, int]] = defaultdict( + lambda: {"n": 0, "agree_commit": 0, "disagree": 0, + "both_correct": 0, "double_fault": 0} + ) + field_caught: list[str] = [] + field_lost_coverage: list[str] = [] + field_wrong: list[dict[str, Any]] = [] + + for case in cases: + cid, cls = case["id"], case["class"] + try: + gold = oracle_answer(case["relations"], case["query"]) + except OracleError: + continue + f = field_read(case["text"]) + s = sym_read(case["text"]) + gate = _gate_admits(f, s) + + f_ans = None if f.refused else f.answer + s_ans = None if s.refused else s.answer + sym_alone = s_ans # symbolic-alone admits its committed answer + + pc = per_class[cls] + pc["n"] += 1 + if f_ans is not None and s_ans is not None: + if f_ans == s_ans: + pc["agree_commit"] += 1 + if f_ans == gold: + pc["both_correct"] += 1 + else: + pc["disagree"] += 1 + if f_ans != gold and s_ans != gold: + pc["double_fault"] += 1 + + # The field's own wrong=0 (it must never commit a wrong answer). + if f_ans is not None and f_ans != gold: + field_wrong.append({"id": cid, "field": f_ans, "gold": gold}) + + # measurement #2 signals + if s_ans is not None and s_ans != gold and gate is None: + field_caught.append(cid) # symbolic committed WRONG; gate refused + if s_ans is not None and s_ans == gold and gate is None: + field_lost_coverage.append(cid) # symbolic correct; gate refused + + rows.append({ + "id": cid, "class": cls, "gold": gold, + "field": f_ans, "field_refused": f.refused and f.refusal_reason, + "symbolic": s_ans, "symbolic_refused": s.refused and s.refusal_reason, + "gate_admits": gate, "symbolic_alone": sym_alone, + }) + + gate_admitted = {r["id"]: r["gate_admits"] for r in rows if r["gate_admits"] is not None} + symbolic_admitted = {r["id"]: r["symbolic_alone"] for r in rows if r["symbolic_alone"] is not None} + admitted_changed = gate_admitted != symbolic_admitted + + verdict = ( + "PASS" if (field_caught and not field_wrong) + else "C3_field_is_decoration_or_liability" + ) + + return { + "total": len(rows), + "verdict": verdict, + "field_caught_symbolic_errors": field_caught, + "field_lost_coverage": field_lost_coverage, + "field_wrong_commits": field_wrong, + "admitted_set_changed": admitted_changed, + "gate_admitted_count": len(gate_admitted), + "symbolic_alone_admitted_count": len(symbolic_admitted), + "per_class": {k: dict(v) for k, v in per_class.items()}, + "rows": rows, + } + + +def main() -> int: + report = run() + summary = {k: v for k, v in report.items() if k != "rows"} + print(json.dumps(summary, indent=2)) + # The instrument is non-failing: a C3 verdict is a sanctioned, honest outcome. + # It only hard-fails if the field ever commits a WRONG answer (wrong=0 breach). + if report["field_wrong_commits"]: + print("FIELD COMMITTED A WRONG ANSWER (wrong!=0 breach):", + report["field_wrong_commits"], file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/generate/relational_symbolic_reader.py b/generate/relational_symbolic_reader.py new file mode 100644 index 00000000..dc725461 --- /dev/null +++ b/generate/relational_symbolic_reader.py @@ -0,0 +1,127 @@ +"""The SYMBOLIC reader for forward-substitutable quantitative relations. + +The second, code-disjoint reading of the same micro-domain as +``generate.relational_field_reader`` — but it resolves by plain integer arithmetic +over a parsed schema, importing **no** ``algebra`` / ``field`` (proven import-disjoint +from the field reader by INV-27). It is: + +- the control arm of the ablation experiment (does the geometric FIELD reader add any + signal over a competent symbolic reader, or is it decoration?), and +- the C3 capability path if the field does not earn its role (a second independent + reading whose agreement with another reader is the wrong=0 gate). + +It is a *competent* reader, not a strawman: it reads the same grammar and applies the +same fences (multiplicative/ratio, forward-substitutability, negatives), and it +detects over-determination (a quantity stated two ways that conflict) and refuses — +so the ablation is a fair test. It has NO float/precision limit (pure int), so unlike +the field it commits arbitrarily large quantities. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass + +# Stable identity for the Tier-2 gate (registered in INV-27). +READER_LINEAGE: str = "symbolic.relational_schema" + +_FENCED_CUE = re.compile( + r"\b(times|twice|double|triple|thrice|half|quarter|ratio|per|each)\b" +) + +# Independent regex set (own spelling/order) — same grammar, different implementation. +_REL_MORE = re.compile(r"(\w+) (?:has|have) (\d+) more \w+ than (\w+)") +_REL_FEWER = re.compile(r"(\w+) (?:has|have) (\d+) (?:fewer|less) \w+ than (\w+)") +_REL_FACT = re.compile(r"(\w+) (?:has|have) (\d+) (\w+)") +_Q_SINGLE = re.compile(r"how many (\w+) does (\w+) have") +_Q_SUM = re.compile(r"how many (\w+) do (\w+) and (\w+)(?: and (\w+))? have") + + +class SymbolicReaderError(ValueError): + """Out-of-grammar case → typed refusal.""" + + +@dataclass(frozen=True, slots=True) +class SymbolicReading: + refused: bool + answer: int | None = None + answer_unit: str | None = None + refusal_reason: str | None = None + reader_lineage: str = READER_LINEAGE + + +def _refuse(reason: str) -> SymbolicReading: + return SymbolicReading(refused=True, refusal_reason=reason) + + +def read_relational(text: str) -> SymbolicReading: + if not isinstance(text, str) or not text.strip(): + return _refuse("empty_input") + if _FENCED_CUE.search(text.lower()): + return _refuse("fenced_multiplicative") + try: + return _read(text.lower()) + except SymbolicReaderError as exc: + return _refuse(str(exc)) + + +def _set(values: dict[str, int], entity: str, value: int) -> None: + """Assign, detecting over-determination (a conflicting re-statement).""" + if entity in values and values[entity] != value: + raise SymbolicReaderError("over_determined_conflict") + values[entity] = value + + +def _read(text: str) -> SymbolicReading: + parts = [s.strip() for s in re.split(r"[.?!]", text) if s.strip()] + + unit: str | None = None + q_single: str | None = None + q_sum: list[str] | None = None + for s in parts: + if "how many" not in s: + continue + if (m := _Q_SUM.search(s)) is not None: + unit = m.group(1) + q_sum = [g for g in m.groups()[1:] if g] + elif (m := _Q_SINGLE.search(s)) is not None: + unit, q_single = m.group(1), m.group(2) + else: + raise SymbolicReaderError("unparsed_query") + break + if q_single is None and q_sum is None: + raise SymbolicReaderError("no_query") + + values: dict[str, int] = {} + for s in parts: + if "how many" in s: + continue + if (m := _REL_MORE.search(s)) is not None: + ent, n, ref = m.group(1), int(m.group(2)), m.group(3) + if ref not in values: + raise SymbolicReaderError("non_forward_substitutable") + _set(values, ent, values[ref] + n) + elif (m := _REL_FEWER.search(s)) is not None: + ent, n, ref = m.group(1), int(m.group(2)), m.group(3) + if ref not in values: + raise SymbolicReaderError("non_forward_substitutable") + _set(values, ent, values[ref] - n) + elif (m := _REL_FACT.search(s)) is not None: + ent, n = m.group(1), int(m.group(2)) + _set(values, ent, n) + else: + raise SymbolicReaderError("unparsed_sentence") + + if q_single is not None: + if q_single not in values: + raise SymbolicReaderError("query_entity_unresolved") + answer = values[q_single] + else: + assert q_sum is not None + if any(p not in values for p in q_sum): + raise SymbolicReaderError("query_entity_unresolved") + answer = sum(values[p] for p in q_sum) + + if answer < 0: + raise SymbolicReaderError("negative_quantity") + return SymbolicReading(refused=False, answer=answer, answer_unit=unit) diff --git a/tests/test_relational_ablation.py b/tests/test_relational_ablation.py new file mode 100644 index 00000000..6cc2834c --- /dev/null +++ b/tests/test_relational_ablation.py @@ -0,0 +1,52 @@ +"""The ablation verdict — pinned as the honest experimental result. + +The field-reasoner wedge's measurement #2/#3 on the simple additive/part-whole +micro-domain: the geometric field reader is **decoration** — it catches no +comprehension error the symbolic reader makes, and the only place it changes the +admitted set is by *losing* coverage (refusing a correct answer at the precision +ceiling). This test pins that finding; if a future change makes the field genuinely +catch a symbolic error, ``field_caught_symbolic_errors`` becomes non-empty and the +PASS assertion here must be revisited (a deliberately meaningful pin). + +The one INVARIANT that must always hold regardless of the verdict: the field never +commits a wrong answer (wrong=0 is structural). +""" + +from __future__ import annotations + +from evals.relational_metric.ablation import run + + +def test_field_never_commits_wrong(): + """wrong=0 — the non-negotiable. The field refuses rather than commit a bad int.""" + report = run() + assert report["field_wrong_commits"] == [], report["field_wrong_commits"] + + +def test_verdict_is_honest_c3_on_this_domain(): + """On forward-substitutable additive relations the field adds no independent signal: + geometric translation == arithmetic addition, so it catches zero symbolic errors.""" + report = run() + assert report["field_caught_symbolic_errors"] == [] + assert report["verdict"].startswith("C3") + + +def test_field_only_changes_admitted_set_by_losing_coverage(): + """The admitted set differs from symbolic-alone only because the field REFUSES a + correct answer (the precision ceiling) — a liability, not error-catching signal.""" + report = run() + assert report["admitted_set_changed"] is True + assert report["gate_admitted_count"] < report["symbolic_alone_admitted_count"] + assert report["field_lost_coverage"] # non-empty: the ceiling case + + +def test_readers_agree_and_are_both_correct_on_committable_classes(): + """Per-class diversity is ZERO on every committable class — the readers are + redundant here (the dossier's predicted common-mode on a metric-trivial domain).""" + report = run() + for cls, stats in report["per_class"].items(): + if cls.startswith("coverage_"): + continue + assert stats["disagree"] == 0 + assert stats["double_fault"] == 0 + assert stats["both_correct"] == stats["n"] diff --git a/tests/test_relational_symbolic_reader.py b/tests/test_relational_symbolic_reader.py new file mode 100644 index 00000000..749a8ec4 --- /dev/null +++ b/tests/test_relational_symbolic_reader.py @@ -0,0 +1,59 @@ +"""The symbolic reader — the ablation control arm and the C3 capability path. + +It must be a *competent* reader (correct on the grammar, fences the same out-of-domain +cases, detects over-determination), so the ablation is a fair test of whether the field +adds anything over it. Unlike the field it has no precision limit (pure int). +""" + +from __future__ import annotations + +from generate.relational_symbolic_reader import READER_LINEAGE, read_relational + + +def test_commits_additive_more_than(): + r = read_relational( + "Tom has 3 marbles. Jane has 5 more marbles than Tom. " + "How many marbles does Jane have?" + ) + assert not r.refused and r.answer == 8 + assert r.reader_lineage == READER_LINEAGE + + +def test_commits_part_whole_sum(): + r = read_relational( + "Tom has 3 marbles. Jane has 5 more marbles than Tom. " + "How many marbles do Tom and Jane have?" + ) + assert not r.refused and r.answer == 11 + + +def test_commits_beyond_field_ceiling(): + """The symbolic reader has no f64 precision limit — it commits where the field + refuses (over_ceiling). This is the field's coverage liability, not a symbolic bug.""" + r = read_relational("Gus has 9000000 apples. How many apples does Gus have?") + assert not r.refused and r.answer == 9000000 + + +def test_fences_multiplicative(): + r = read_relational( + "Tom has 3 marbles. Jane has twice as many marbles as Tom. " + "How many marbles does Jane have?" + ) + assert r.refused and r.refusal_reason == "fenced_multiplicative" + + +def test_detects_over_determination(): + """A competent reader refuses a conflicting re-statement (so the field's geometric + coherence check is not a unique advantage).""" + r = read_relational( + "Tom has 3 marbles. Tom has 5 marbles. How many marbles does Tom have?" + ) + assert r.refused and r.refusal_reason == "over_determined_conflict" + + +def test_refuses_forward_reference(): + r = read_relational( + "Jane has 5 more marbles than Tom. Tom has 3 marbles. " + "How many marbles does Jane have?" + ) + assert r.refused and r.refusal_reason == "non_forward_substitutable"