diff --git a/docs/analysis/hebrew-koine-greek-logos-pack-capability-roadmap-2026-07-19.md b/docs/analysis/hebrew-koine-greek-logos-pack-capability-roadmap-2026-07-19.md new file mode 100644 index 00000000..0e7229a0 --- /dev/null +++ b/docs/analysis/hebrew-koine-greek-logos-pack-capability-roadmap-2026-07-19.md @@ -0,0 +1,220 @@ +# Hebrew and Koine Greek Logos Pack Capability Roadmap + +**Date:** 2026-07-19 +**Type:** No-implementation audit / planning dossier +**Authority tree:** `feat/deterministic-relational-operator-ablation` +**Companion docs:** + +- `docs/analysis/relational-operator-ablation-cartography-2026-07-19.md` — Phase 1 cartography + go/no-go +- `docs/analysis/relational-operator-ablation-dossier-2026-07-19.md` — English `fraction_decrease` ablation (IMPLEMENTED AND MEASURED) +- This file — separate HE/GRC Logos capability audit (planning only) + +**Boundary:** The relational-operator ablation remains closed. English-only `fraction_decrease` depth was intentionally inert (no observed-morphology basis). That result must not be generalized to “Hebrew/Greek never help,” nor used to authorize English→pseudo-depth. This roadmap does **not** authorize a giant pack build. + +--- + +## 1. Executive verdict + +**READY ONLY FOR PREREQUISITE WIRING OR MIGRATION** + +| Label | Why | +|-------|-----| +| Not ready for broad Logos implementation | Dual pack systems; sparse morph tags; holonomy crown proof not robust; no executable morph→canonical-relation mapping with provenance | +| Not architecturally blocked | Canonical language-independent seams exist (`BoundRelation`, teaching operators, pack checksum/OOV, `resolve_entry` / `node_depths`) | +| Broader “Logos improves all domains” thesis | Scientifically under-specified until a narrow **observed-HE/GRC** constraint ablation exists | + +### Ablation boundary (preserve) + +English-only `fraction_decrease` sealed n=8: depth **intentionally inert**. Do not reopen ablation fixtures/results except for demonstrated defects. Do not claim HE/GRC help arithmetic without observed morph input. + +--- + +## 2. Hebrew capability matrix + +Status: **LIVE** | **EVALUABLE** | **DIAGNOSTIC** | **TEST-ONLY** | **DORMANT** | **SCAFFOLD** | **SPECIFIED** | **ABSENT** | **CONTRADICTED** + +| Capability | Exact evidence | Compiler | Runtime | Proof/tests | Gap | Overclaim risk | Priority | +|------------|----------------|----------|---------|-------------|-----|----------------|----------| +| Structured morphology schema | `packs/schema.py` `MorphologyEntry`; `packs/data/he_*` | LIVE `_apply_morphology` | Manifold via `load_pack` | `test_morphology_registry`, holonomy morph | No paradigm generator | Medium if “full morph” | P0 (tags live) | +| Legacy triliteral / romanized root | `packs/compiler.py` `_is_hebrew_root`, `_triliteral_root` | LIVE feature | Manifold | holonomy same-root | Dual encodings | Low | P1 document/migrate | +| Root identity on resolve | `chat/pack_resolver.py` `resolve_entry`, `resolve_token_depths` | Lookup | LIVE PropGraph / recognition | `test_3lang_depth_capability`, `test_oov_pipeline` | Exact surface match only | Medium | P0 observed-HE | +| Root-sense disambiguation | Multi-surface rare; no sense graph | ABSENT | First-match | — | Homographs silent | **Critical** | **P0 prerequisite** | +| Binyan as licensed semantics | Sparse `binyan` tags (qal/piel) | LIVE as infl rotor | Manifold only | sparse data | No root-specific licensing | **Critical** if universalized | P2 after sense model | +| Verbal aspect | Sparse `aspect: perfect` | Tags only | Not clause-level | source DRAFT notes | Not temporal operators | High | P2–P3 | +| Voice / reflexive / passive | Essentially ABSENT in compiled HE | — | — | — | Niphal/Hithpael not systematic | High | P3 | +| Construct state | 1× construct tag; source `he:construct-chain` frame | Tag only | Frames DISCONNECTED | — | Not executable possession | High if genitive-universalized | P1 candidate | +| Nominal gender/number | LIVE tags | LIVE rotors | Tags | registry | No agreement checker | Medium | P2 | +| Pronominal suffixes | Sparse `suffix_chain` | LIVE | Ingest OOV decomp | ingest gate | Not full paradigm | Medium | P3 | +| Lexical valency / frames | `packs/he/frames.jsonl` (4 frames) | **DISCONNECTED** from `packs/data` | Not in `load_pack` | CLI validate only | No LIVE frame→BoundRelation | High | **P1 migration** | +| Semantic domains | lexicon `semantic_domains` | LIVE | Manifold | domain contracts | Not morph-derived | Low | Keep | +| Alignment edges | `alignment.jsonl` | Load nudge | Geometry | alignment tests | Data ≠ reasoning | Medium | Honest limits | +| Readback HE | `_assemble_he`, ADR-0030 hedges; no `readback_rules.py` | — | Limited assembly | identity depth tests | Fluency C01 only | High if “full readback” | See fluency honesty | +| Correction / contradiction | Teaching chains ADR-0102 | N/A | Teaching/eval | domain tests | Not morph-triggered | Medium | Tie morph to existing ops | +| OOV fail-closed | `gate_engaged=true`, `fail_closed` | Enforced | LIVE | pack runtime tests | OK | Low | Preserve | +| Fluency eval | `evals/hebrew_fluency` | — | Articulation | contract.md C01 | Not morph-constraint eval | High if misused | Separate lane | +| Executable morph→canonical relation | — | ABSENT | ABSENT | ABSENT | Core Logos gap | Critical | Next vertical slice | + +--- + +## 3. Greek capability matrix + +| Capability | Exact evidence | Compiler | Runtime | Proof/tests | Gap | Overclaim risk | Priority | +|------------|----------------|----------|---------|-------------|-----|----------------|----------| +| Lemma inventory | `packs/data/grc_*` (11 / 38) | LIVE | `resolve_entry` | registry, 3lang | Micro seed | Medium | P0 honesty | +| Sense disambiguation | No compiled `senses.jsonl` on data packs | ABSENT | First-match | — | Colwell needs sense+syntax | **Critical** | **P0 prerequisite** | +| Case tags | Nearly all **nominative** only | LIVE rotors | Tags only | registry order | No productive case paradigm | **Critical** if case→role | P1 observed forms only | +| Number/gender/agreement | Tags present | LIVE | No agreement engine | — | No S-V check | High | P2 | +| Prep + case | Source `el:relational-pros` | DISCONNECTED | ABSENT | — | Not enforced | High | After case diversity | +| Valency / frames | `packs/grc\|el/frames.jsonl` | DISCONNECTED | ABSENT | — | Dual-system gap | High | **P1 migration** | +| Aspect / tense-form | Present indicative dominant | Tags | Not operators | — | Aorist/perfect system ABSENT | High | P2–P3 licensed tables | +| Voice / mood | Sparse active/middle; indicative | Tags | — | — | No full mood system | High | P3 | +| Participles / infinitives | ABSENT as productive class | — | — | — | Out of seed scope | Medium | Defer | +| Semantic domains | `logos.core`, etc. | LIVE | Manifold | ADR-0102 | OK | Low | Keep | +| Alignment | grc↔he↔en edges | Load nudge | Geometry | alignment tests | Not robust holonomy | Medium | Honest limits | +| Colwell / anarthrous | Source frame + notes | DISCONNECTED | ABSENT | — | Not runtime | High if claimed LIVE | P2 fail-closed | +| Readback GRC | `_assemble_grc` | — | Limited | depth tests | Fluency-class | Medium | `evals/koine_greek_fluency` | +| Geometric signature for GRC | `resolve_geometric_signature` needs `senses.jsonl` | ABSENT for depth packs | EN math when present | fraction EN | HE/GRC cannot supply scale signatures | Medium | Domain concern | +| Executable morph→canonical relation | ABSENT | ABSENT | ABSENT | ABSENT | Core gap | Critical | Observed-GRC later | + +--- + +## 4. Shared Logos substrate matrix + +| Capability | Evidence | Compiler | Runtime | Proof/tests | Gap | Overclaim risk | Priority | +|------------|----------|----------|---------|-------------|-----|----------------|----------| +| Roles DEPTH_ROOT / DEPTH_RELATION | `LanguageRole`; manifests | Schema LIVE | Load | ADR-0015 | Source `el` vs runtime `grc` | Medium | Prefer `grc` | +| Checksum + OOV | `load_pack` | LIVE | ChatRuntime | pack tests | Source ADR-0005 gates never true | Low | Keep | +| Ordered morph composition | `_apply_morphology` | LIVE | Manifold | holonomy morph | Not semantic licensing | Medium | Keep | +| Alignment graph | `alignment/graph.py` | Load nudge | Geometry | alignment tests | Not clause reasoning | High if “reasoning” | Data + local geometry | +| Holonomy encode | `algebra/holonomy.py` | LIVE encode | Ingest path | **Honest-fail crown** | Crown **not robust** | **Critical** | Do not claim trilingual proof | +| PropGraph depth spine | `GraphNode` language/root/morphology_id | N/A | LIVE he/grc | 3lang tests | No BoundRelation emission | Medium | Bridge for constraints | +| Depth decorate vs decide | `enrich_assessments_with_depth` vs anti_unifier | — | Match LIVE; math decorate | 3lang, oov | EN math correctly inert | Low if labeled | Preserve | +| Canonical roles | `BoundRelation`, `RoleObligation` | N/A | Math LIVE (fraction) | problem_frame tests | Language-independent | Low | **Reuse only** | +| Teaching causal/contradiction | `hebrew_greek_textual_reasoning_chains_v1.jsonl` | N/A | Teaching/domain | ADR-0102 | Lexeme-level, not morph | Medium | Constraint consumer | +| GrammarAttractor | schema | Thin | DORMANT | — | Not hot path | Medium | Defer | +| `packs.evidence` | holonomy helpers | — | TEST-ONLY / dormant | holonomy tests | Not production | Medium | Don’t build runtime on it | +| Sensorium HE/GRC | `gate_engaged=False` default | Scaffold | Divergent from data True | — | Dual truth | Medium | Align or drop fiction | +| Domain contract | ADR-0091/0102 four packs | Manifest | Ledger reasoning-capable | ratification tests | Fluency ≠ morph-constraint | Medium | Keep honesty | + +**Trilingual alignment:** curated alignment data + local manifold geometry; **clause holonomy crown proof is CONTRADICTED as robust** (`docs/analysis/holonomy-resonance-proof-not-robust-2026-06-14.md`, honest-fail tests). + +--- + +## 5. Pack taxonomy (strict) + +| Tier | Contents | Must not smuggle | +|------|----------|------------------| +| Universal canonical Logos primitives | `BoundRelation`, RoleObligations, epistemic status, VersorBinding/CGA, teaching operator families, shared `semantic_domains` | HE/GRC-only hidden reasoners | +| Hebrew surface + mapping rules | Observed surfaces, roots, tags, **authored** mapping rules + provenance | English→HE inference; universal binyan→semantics | +| Greek surface + mapping rules | Observed lemmas/forms, attested case tags, **authored** prep+case rules | Nominative=agent; English→case | +| English articulation | Operational base, realizer, fluency | Treating EN as depth proof | +| Domain packs | math/physics/etc. organs | Domain-private morph ontology | +| Anchor / lens / register | `packs/anchor_lens/he_*`, `grc_*` | Lens as morphology engine | +| Safety / identity | identity/safety/ethics; ADR-0030 hedges | Identity as role solver | +| Evaluation / mastery | fluency, fabrication_control, inference_closure, ablations | Fluency pass as morph-constraint pass | + +--- + +## 6. Phased vertical-slice roadmap + +### Phase 0 — Prerequisites (no bulk pack growth) + +1. Freeze dual-system map: LIVE = `packs/data/*`; source `packs/{he,grc,el}` = draft/disconnected. +2. Sense/ambiguity policy: first-match is fail-open for meaning → multi-candidate or refuse on collision. +3. Entrypoint already LIVE: HE `resolve_entry` / `define אמת` path (or GRC lemma path). +4. No lexicon expansion until Phase 1 ablation. + +### Phase 1 — Smallest observed-morphology constraint (separate implementation PR) + +| Condition | Behavior | +|-----------|----------| +| Canonical only | Teaching operator/chain without morph constraint | +| Executable morph constraint | Authored rule with provenance → constraint or refuse | +| Metadata-only | `node_depths` / root notes; **must not** change outcome | +| Invalid / ambiguous | Homograph without sense rule → **refuse** | + +**Consumer:** existing teaching/contradiction/abstention seams — **not** GSM8K, not new ontology. +**Inputs:** observed HE/GRC only. +**Success:** wrong=0; metadata inert; outcome changes only with provenance. + +### Phase 2 — One morph phenomenon + +- HE: root identity + ambiguity refuse → construct-state candidate → binyan only with license tables +- GRC: lemma+attested case as evidence only → prep+case after non-nominative data → aspect later + +### Phase 3 — Schema/compiler extensions (only if Phase 1 shows value) + +Optional `mapping_rules.jsonl`; sense inventory; compile-time rule validation; constraint readback. + +### Phase 4 — Broader domains + +Only after Phase 1–2 pass; never by inventing morph for English word problems. + +--- + +## 7. No-go list + +- Decorative packs not consumed by runtime/eval +- Unlicensed morph→semantics (nominative=agent, Hiphil=always cause, aorist=punctual, …) +- Parallel HE/GRC reasoner outside shared canonical operators +- English→Greek case / Hebrew binyan pseudo-evidence +- Coverage via guessing / relaxing refuse +- Claiming holonomy crown proof is LIVE +- Fluency C01 or ADR-0102 reasoning-capable as morph-constraint proof +- Broad lexicon growth before observed-morph ablation +- Filling all binyanim/case tables as “progress” without rules+eval +- Building production Logos on dormant `packs.evidence` +- Reopening English fraction_decrease ablation to “add depth value” without observed morph + +--- + +## 8. Minimum contract for future domain packs + +1. Shared language-independent constraint catalog IDs +2. Mapping rules: rule_id, language, features, preconditions, counterexamples, validity, fail-closed +3. Provenance: pack_id, version, morphology_id/entry_id, spans +4. Deterministic compile → immutable ordered candidates +5. Readback to surface features + canonical constraint +6. Contradiction with COHERENT evidence → abstain, never override +7. Eval: observed-language inputs + metadata-only + sealed holdout +8. No pack-private geometric meaning space (ADR-0005) + +--- + +## 9. Future ablation/proof design (not authorized here) + +| Arm | HE | GRC | Cross-lang | Canonical | +|-----|----|-----|------------|-----------| +| Input | Observed HE | Observed GRC | Authored paired clauses | Language-agnostic operator | +| Executable | Morph rule → constraint | Morph rule → constraint | Data-checked alignment (not crown holonomy until metric fixed) | Existing operators | +| Metadata-only | Roots on, rule off | Case tags on, rule off | Alignment labels off | N/A | +| Fail | Ambiguous root/sense | Ambiguous lemma/case | Conflicting alignments | Insufficient evidence | + +--- + +## 10. Recommended next implementation unit + +| Field | Value | +|-------|--------| +| **Branch** | `feat/observed-he-morph-constraint-v0` | +| **Worktree** | `../core-observed-he-morph-constraint` | +| **Why HE first** | LIVE root resolution + denser depth tests; GRC case inventory nominative-skewed | +| **PR-sized scope** | Ambiguity refuse; one authored mapping-rule type + provenance; consumer = teaching/contradiction/abstention; sealed observed-HE fixture; metadata-only control; no bulk lexicon; no English math; no holonomy crown “fix” | +| **Out of scope** | Pack expansion, GRC case tables, binyan universals, fluency C02–C13, dual ontology | + +### Reviewer acceptance criteria (future PR) + +- [ ] Observed Hebrew inputs (or explicit OOV refuse) +- [ ] Mapping rules with rule_id, preconditions, counterexamples, provenance +- [ ] Metadata-only outcome-identical to rule-off +- [ ] Ambiguous → refuse; wrong=0 +- [ ] No English→HE morph invention +- [ ] No holonomy crown proof claim +- [ ] Does not modify relational-operator ablation fixtures/results +- [ ] Smoke + targeted tests green + +--- + +## 11. One-line summary + +Hebrew/Greek Logos packs are **LIVE as sparse compiled manifolds + depth recognition + domain teaching chains**, but **not** as executable, falsifiable morph→canonical-constraint systems. Holonomy crown proof is **not robust**. Next work is **prerequisite wiring + one observed-HE constraint ablation**, not a giant pack build. diff --git a/docs/analysis/relational-operator-ablation-cartography-2026-07-19.md b/docs/analysis/relational-operator-ablation-cartography-2026-07-19.md new file mode 100644 index 00000000..89b63416 --- /dev/null +++ b/docs/analysis/relational-operator-ablation-cartography-2026-07-19.md @@ -0,0 +1,152 @@ +# Relational Operator Ablation — Cartography Ledger & Go/No-Go + +**Date:** 2026-07-19 +**Branch:** `feat/deterministic-relational-operator-ablation` +**Worktree:** `../core-relational-operator-ablation` +**Authority commit:** `d8d62b8e` (main tip at branch creation) +**Status:** Evidence ledger for Phase 1–2; implementation of the narrow GO slice is companion code under `generate/relational_operator_ablation.py` and `evals/relational_operator_ablation/`. + +**Related:** HE/GRC broader capability is **not** authorized by this ledger. See `docs/analysis/hebrew-koine-greek-logos-pack-capability-roadmap-2026-07-19.md` (verdict: READY ONLY FOR PREREQUISITE WIRING OR MIGRATION). + +--- + +## 1. Executive decision (closed) + +**Verdict for this work product:** **IMPLEMENTED AND MEASURED** (narrow slice only). + +| Scope | Decision | +|-------|----------| +| Full hypothesis: ancient morphology automatically improves GSM8K arithmetic | **NO-GO** (scientific precondition: no valid English→depth mapping; anti-circularity) | +| Narrow fraction_decrease relational-operator ablation | **GO** — family already LIVE with ProblemFrame + VersorBinding + A2k | +| Depth as *executable* math contribution on English word problems | **NO-GO for implementation of new mapping** — metadata-only control proves inert | + +**Relation family chosen from live evidence:** `proportional_change.decrease_to_fraction` / organ `fraction_decrease` +(`generate/construction_affordances.py` `_SERVING_AUTHORIZED_FAMILIES`; `generate/derivation/fraction_decrease.py`; Gate A2k in `generate/math_candidate_graph.py`). + +### Decision criteria checklist + +| Criterion | Pass/Fail | Evidence | +|-----------|-----------|----------| +| Canonical relation seam exists | **Pass** | `BoundRelation(relation_type="decrease_to_fraction")`, roles via `problem_frame_bound_relations` | +| Deterministic reconstructable operator | **Pass** | `VersorBinding` + CGA dilation; same text → same answer | +| Correction/readback or native integrity | **Pass (native)** | Self-verify + span provenance + `versor_condition < 1e-6`; no fabricated dual-adjoint | +| Testable eval lane | **Pass** | Sealed `evals/relational_operator_ablation/v1` + existing train_sample organ tests | +| Baseline vs operator vs depth vs metadata-only | **Pass** | Ablation conditions in harness | +| Fail-closed | **Pass** | Missing/ambiguous/hazard → refuse | +| Narrow non-claim scope | **Pass** | No full GSM8K claim; no EN→HE/GRC translation | + +--- + +## 2. Live-path maps + +### A. Text/chat — LIVE (not math serving) + +```text +text → CognitiveTurnPipeline.run (core/cognition/pipeline.py) + → tokenize / resolve_token_depths (chat/pack_resolver.py) + → recognize (recognition/anti_unifier.py; root-canonical for he/grc) + → ChatRuntime.chat (chat/runtime.py) + → ingest.gate → FieldState + → PropositionGraph (generate/graph_planner.py) + → realizer / gates / refusal + → CognitiveTurnResult.node_depths +``` + +`contract_assessment=None` at `resolve_surface` — ProblemFrame **DISCONNECTED** from chat spine. + +### B. Language packs — dual layer + +| Layer | Path | Status | +|-------|------|--------| +| Source ADR-0005 | `packs/{en,he,grc,el}/` status=draft, gates false | **DRAFT / DISCONNECTED** from compiler | +| Compiled | `packs/data/*` → `packs.compiler.load_pack` → `_apply_morphology` | **LIVE** | +| Depth IDs | `DEPTH_PACK_IDS` = he_core_cognition_v1, he_logos_micro_v1, grc_logos_* | **LIVE** | + +### C. Problem-frame / math + +```text +text → build_problem_frame (generate/problem_frame_builder.py) [DIAGNOSTIC IR + A2k input] + → proposals / BoundRelation / BoundQuestionTarget + → assess_fraction_decrease / assess_geometric_proposals + → VersorBinding dilation + → resolve_promotable_fraction_decrease (generate/derivation/fraction_decrease.py) + → answer [LIVE Gate A2k] + +parse_and_solve (generate/math_candidate_graph.py) [LIVE GSM8K serving orchestrator] +``` + +| Family | serving_allowed | Status | +|--------|-----------------|--------| +| fraction_decrease | **True** | **LIVE** A2k | +| percent_partition | False catalog; live prose A2l | **LIVE but DISCONNECTED** from frame | +| quantity_entity | False | **DIAGNOSTIC** | +| unary_delta | False | **DIAGNOSTIC** (no derivation organ) | + +### D. Evaluation + +| Split | N | correct / wrong / refused | Scorer | +|-------|---|---------------------------|--------| +| Sealed GSM8K test (recorded) | 1319 | **0 / 0 / 1319** | operator-only holdout | +| train_sample report.json | 50 | **30 / 0 / 20** | `_score_one_candidate_graph` | +| holdout_dev report.json | 500 | **5 / 0 / 495** | same | +| CORE public synthetic | 150 | **150 / 0 / 0** | classic `_score_one` | + +**Note:** Order is always **correct / wrong / refused**. Prior prose that wrote `30/20/0` swapped wrong/refused. + +--- + +## 3. Design truth matrix + +| Claim | Spec | Implementation | Runtime | Tests | Status | +|-------|------|----------------|---------|-------|--------| +| Hebrew root morphology | ADR-0015 | `MorphologyEntry.root`; `_apply_morphology` | LIVE manifold + resolver | `test_morphology_registry`, holonomy | **LIVE (sparse)** | +| Hebrew binyan | ADR-0005 | Inflection tag only | geometric tag | sparse data | **Partial** | +| Greek case/aspect/voice | ADR-0015 | Compiled inflection tags | LIVE tags | morphology_registry | **Partial** | +| Anchor lens | pack inventory | `packs/anchor_lens/` | LIVE overlay | inventory | **LIVE** | +| PropGraph / depth | 3lang plan | `depth_canonical`, pipeline | LIVE he/grc input | `test_3lang_depth_capability` | **LIVE spine** | +| Depth → math answer | — | `enrich_assessments_with_depth` explanation only | decoration | same | **Inert on math** | +| Canonical math roles | kernel substrate | BoundRelation roles | A2k + diagnostic | problem_frame tests | **LIVE fraction** | +| VersorBinding | CGA construction boundary | dilation payload | LIVE A2k | fraction + 3lang tests | **LIVE** | +| Readback / integrity | organ self-verify | spans + verified derivation | LIVE | fraction_decrease | **Partial (native)** | +| GSM8K eval | ADR-0119+ | train_sample/holdout/sealed | LIVE open | runners | **LIVE** | +| serving_allowed all False | prior claim | one True family | — | construction_affordances | **Contradicted** | +| FGL | prior claim | — | — | — | **ABSENT** | +| ADR-0248 as Logos | prior claim | integrity handoffs | observe-only | ring3 | **Miscited** | +| ADR-0196 ancient packs | prior claim | Python/Rust/Zig | — | ADR text | **False** | +| EN GSM8K → GRC case / HE binyan | prior claim | — | — | — | **ABSENT** | + +--- + +## 4. Claim corrections (prior synthesis) + +1. **EN/HE/GRC roles** — Partially confirmed (ADRs + LIVE compiled packs; source packs draft). +2. **ADR-0005/0015** — Confirmed as architecture; implementation is `packs/data` compiler path. +3. **packs/he, packs/grc draft + compiler** — Confirmed split: draft source vs LIVE compiled. +4. **Ordered morph composition** — Confirmed LIVE+tested in `packs/compiler._apply_morphology`. +5. **depth_canonical / enrich** — Confirmed; enrich is **decoration**; recognize is **matching**. +6. **ProblemFrame symbols** — Confirmed under `generate/`. +7. **All serving_allowed=False** — **False**; fraction_decrease is True. +8. **No EN→case/binyan math path** — Confirmed absent. +9. **GSM8K all refused** — Sealed test **0/0/1319** recorded; train_sample **30/0/20**; holdout_dev **5/0/495**. claims_ledger C **7/43/0** is **stale**. +10. **FGL / ADR-0248 Logos** — FGL absent; ADR-0248 = integrity handoffs. +11. **ADR-0196** — Python/Rust/Zig, not ancient packs. + +--- + +## 5. Implementation plan (narrow GO) + +1. Pure condition runners: baseline scalar from frame roles; operator geometric; metadata-only depth; invalid refuse. +2. No new depth→role mapping from English. +3. Sealed fixture v1 (fraction_decrease + confusers). +4. Measurement report + tests. +5. Smoke gate. + +--- + +## 6. Remaining gaps (not in this slice) + +- Engineering: wire ProblemFrame into CognitiveTurnPipeline (optional future). +- Engineering: unify percent_partition prose organ with frame contracts. +- Scientific: whether *observed* he/grc morphology can constrain math on bilingual inputs (unproven; not this PR). +- Rejected: inventing Greek cases for English GSM8K. +- Deferred: expanding draft `packs/he|grc` gates. diff --git a/docs/analysis/relational-operator-ablation-dossier-2026-07-19.md b/docs/analysis/relational-operator-ablation-dossier-2026-07-19.md new file mode 100644 index 00000000..3971157c --- /dev/null +++ b/docs/analysis/relational-operator-ablation-dossier-2026-07-19.md @@ -0,0 +1,154 @@ +# Deterministic Relational Operator Ablation — Final Engineering Dossier + +**Branch:** `feat/deterministic-relational-operator-ablation` +**Worktree:** `/Users/kaizenpro/Projects/core-relational-operator-ablation` +**Date:** 2026-07-19 + +**Document set on this branch:** + +| Doc | Role | +|-----|------| +| `docs/analysis/relational-operator-ablation-cartography-2026-07-19.md` | Phase 1 cartography, design-truth matrix, go/no-go | +| `docs/analysis/relational-operator-ablation-dossier-2026-07-19.md` | This file — ablation implementation + measurement | +| `docs/analysis/hebrew-koine-greek-logos-pack-capability-roadmap-2026-07-19.md` | Separate HE/GRC Logos audit (planning only; no pack build) | + +--- + +## Executive verdict + +**IMPLEMENTED AND MEASURED** + +Narrow vertical slice only: family `proportional_change.decrease_to_fraction` / organ `fraction_decrease`. +Ancient-language → GSM8K arithmetic is **not** claimed and is **not** implemented (scientific NO-GO under anti-circularity). + +**Depth on English-only cases was intentionally inert** (no observed-morphology basis). That does not test observed Hebrew/Greek input or the broader Logos-pack thesis — see the HE/GRC roadmap for the audited next path (`feat/observed-he-morph-constraint-v0`). + +--- + +## Ground-truth corrections + +| Prior claim | Classification | Evidence | +|-------------|----------------|----------| +| EN articulation / HE depth-root / GRC depth-relation | Partially confirmed | ADR-0005/0015; LIVE `packs/data/he_*`, `grc_*`; draft `packs/he\|grc` | +| ADR-0005/0015 architecture | Confirmed | Accepted ADRs; compiler path `packs.compiler._apply_morphology` | +| Source packs draft + morph compiler | Confirmed (split) | Source DISCONNECTED; compiled LIVE | +| Ordered morph composition | Confirmed LIVE | `MorphologyEntry`; tests holonomy/registry | +| depth_canonical / enrich | Confirmed; enrich=decoration | `recognition/depth_canonical.py`; math inert | +| ProblemFrame symbols | Confirmed | `generate/problem_frame*.py` | +| All `serving_allowed=False` | **False** | fraction_decrease **True** in `construction_affordances.py` | +| EN→Greek case/Hebrew binyan math path | **Absent** | No imports in math organs | +| GSM8K all refused | Split-dependent | Sealed **0/0/1319**; train_sample **30/0/20**; holdout_dev **5/0/495** (correct/wrong/refused) | +| FGL / ADR-0248 as Logos | **Absent / miscited** | ADR-0248 = integrity handoffs | +| ADR-0196 ancient packs | **False** | Python/Rust/Zig doctrine | + +--- + +## Architecture map + +See `docs/analysis/relational-operator-ablation-cartography-2026-07-19.md`. + +Summary: + +- **Chat path LIVE**; ProblemFrame **DISCONNECTED** (`contract_assessment=None`). +- **Math serving LIVE** via `parse_and_solve`; Gate A2k uses ProblemFrame + VersorBinding. +- **Depth LIVE** for he/grc recognition; **inert** for English math answers. + +--- + +## Implemented mechanism + +| Layer | Mechanism | +|-------|-----------| +| Canonical relation | `BoundRelation(relation_type="decrease_to_fraction")` with roles base_quantity, scale, state_entity, transition, unit | +| Baseline operator | Pure rational `base * (1 - scale)` from grounded frame roles + shared organ hazard refuse gates | +| Geometric operator | Live `resolve_promotable_fraction_decrease` → CGA dilation `VersorBinding` | +| Depth-derived mapping | **None executable** on English; recorded `inactive_no_mapping` with rule id | +| Metadata-only | `assess_contracts(..., depth=)` appends `[root:…]`; must not change answer | +| Provenance | Spans, scale surface, roles, mapping_rule_id, pack ids (empty when inactive) | +| Readback | `RelationReadback` reconstructs roles, scale, missing bindings, versor_error | +| Fail-closed | Missing/hazard/adversarial → refuse; wrong=0 on sealed fixture | + +Axioms: geometry-first (CGA dilation), field/structured IR (ProblemFrame), propagation via versor_apply, native integrity (not invented dual-adjoint), reconstruction via readback, compilation-last (no new ontology), reality-over-inheritance (depth inert proven). + +--- + +## Files changed + +| Path | Purpose | Key symbols | +|------|---------|-------------| +| `docs/analysis/relational-operator-ablation-cartography-2026-07-19.md` | Evidence ledger + go/no-go | — | +| `docs/analysis/relational-operator-ablation-dossier-2026-07-19.md` | This dossier | — | +| `generate/relational_operator_ablation.py` | Condition runners | `run_baseline`, `run_operator`, `run_depth`, `run_metadata_only`, `run_invalid` | +| `evals/relational_operator_ablation/` | Sealed lane | `v1/cases.jsonl`, `v1/runner.py`, `v1/report.json` | +| `tests/test_relational_operator_ablation.py` | Unit/integration/adversarial proofs | 20 tests | + +--- + +## Proof and test matrix + +| Command | Result | +|---------|--------| +| `PYTHONPATH=. python3 -m evals.relational_operator_ablation.v1.runner` | passed; report_sha256 `53df6d17…` | +| `PYTHONPATH=. python3 -m pytest -q tests/test_relational_operator_ablation.py` | **20 passed** | +| `uv run core test --suite smoke -q` | **176 passed** (~132s) | + +Key tests: + +- baseline rational / operator geometric / metadata inert / depth inactive +- sealed gold correct; adversarial refuse not wrong +- preexisting scale>1 A2k gap isolated (not masked) +- determinism, no input mutation, readback provenance + +--- + +## Ablation results (n=8 sealed) + +Order: **correct / wrong / refused** + +| Condition | correct | wrong | refused | coverage | Notes | +|-----------|---------|-------|---------|----------|-------| +| baseline | 2 | 0 | 6 | 2/8 | Frame roles + rational math | +| operator | 2 | 0 | 6 | 2/8 | Live VersorBinding path | +| depth | 2 | 0 | 6 | 2/8 | = operator; mapping inactive | +| metadata_only | 2 | 0 | 6 | 2/8 | Root notes present; answers match operator | +| invalid (adversarial subset n=6) | 0 | 0 | 6 | 0/6 | Prefer refuse over wrong | + +**Findings:** metadata_only inert ✓; depth inactive equals operator ✓; wrong=0 ✓; deterministic ✓. + +**Limitations:** n=8; not GSM8K-wide; depth executable unproven for English; chat spine still disconnected from ProblemFrame; pre-existing A2k scale>1 gap documented separately. + +**Scientific conclusion:** Canonical relation operators (baseline scalar vs geometric) agree when both commit; depth **metadata does not improve arithmetic** on this English slice; executable depth mapping was correctly refused as inactive rather than fabricated. + +--- + +## Remaining gaps + +| Kind | Item | +|------|------| +| Engineering | Wire ProblemFrame into CognitiveTurnPipeline backpressure | +| Engineering | Unify percent_partition prose organ with frame contracts | +| Engineering | Fix A2k scale>1 admit (pre-existing; isolated test) | +| Scientific | Whether observed he/grc morphology can constrain bilingual math | +| Deferred | Expand draft source packs / holonomy crown proof | +| Rejected | Invent Greek case / Hebrew binyan from English GSM8K | + +--- + +## PR-ready summary + +**Branch:** `feat/deterministic-relational-operator-ablation` +**Title:** `feat(eval): deterministic relational operator ablation (fraction_decrease)` + +**Description:** +Evidence-led cartography of language packs, depth spine, and ProblemFrame/math paths. Implements a narrow sealed ablation for `fraction_decrease` comparing math-frame baseline, geometric operator, inactive depth contribution, metadata-only control, and adversarial refuse. Proves depth metadata is inert on English arithmetic; does not invent ancient-language role mapping. + +`[Verification]: Smoke suite passed locally (176 passed, ~132s); ablation tests 20 passed; ablation runner passed (n=8, wrong=0).` + +**Risks:** Imports private hazard helpers from fraction_decrease for shared refuse gates; scale>1 organ gap remains. +**Rollback:** Revert branch commits; no serving flag changes. +**Reviewer checklist:** +- [ ] Cartography matrix rows cite live paths +- [ ] No English→case/binyan mapping introduced +- [ ] Metadata-only inert invariant holds +- [ ] wrong=0 on sealed fixture +- [ ] Preexisting A2k gap not masked diff --git a/evals/relational_operator_ablation/__init__.py b/evals/relational_operator_ablation/__init__.py new file mode 100644 index 00000000..4a6a5613 --- /dev/null +++ b/evals/relational_operator_ablation/__init__.py @@ -0,0 +1 @@ +"""Deterministic Relational Operator Ablation eval lane (fraction_decrease).""" diff --git a/evals/relational_operator_ablation/v1/__init__.py b/evals/relational_operator_ablation/v1/__init__.py new file mode 100644 index 00000000..03fe0361 --- /dev/null +++ b/evals/relational_operator_ablation/v1/__init__.py @@ -0,0 +1 @@ +"""Sealed v1 fixture + runner for relational operator ablation.""" diff --git a/evals/relational_operator_ablation/v1/cases.jsonl b/evals/relational_operator_ablation/v1/cases.jsonl new file mode 100644 index 00000000..5e5b56ad --- /dev/null +++ b/evals/relational_operator_ablation/v1/cases.jsonl @@ -0,0 +1,8 @@ +{"case_id": "roa-v1-0001", "problem": "In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. If the current temperature of the mountain is 84 degrees, what will the temperature decrease by?", "expected_answer": 21.0, "expected_unit": "degrees", "expected_outcome": "correct", "tags": ["fraction_decrease", "gold", "train_sample_0005"], "notes": "Canonical decrease-to-fraction delta; gold=21."} +{"case_id": "roa-v1-0002", "problem": "In two hours, Cedar peak's temperature will decrease to 2/3 of its temperature. If the current temperature of the peak is 60 degrees, what will the temperature decrease by?", "expected_answer": 20.0, "expected_unit": "degrees", "expected_outcome": "correct", "tags": ["fraction_decrease", "gold", "sibling"], "notes": "Sibling shape: 60*(1-2/3)=20."} +{"case_id": "roa-v1-0003", "problem": "In one hour, the lake's temperature will decrease to 3/4 of its temperature. If the current temperature of the lake is 80 degrees, what will the temperature be?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "final_value", "invalid"], "notes": "Final-value question must refuse delta organ (not guess 60)."} +{"case_id": "roa-v1-0004", "problem": "Yun had 20 paperclips initially, but then lost 12. Marion has 1/4 more than what Yun currently has, plus 7. How many paperclips does Marion have?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "affine_confuser", "invalid"], "notes": "Percent/fraction affine confuser; fraction_decrease must refuse."} +{"case_id": "roa-v1-0005", "problem": "A school has 100 students. Half of the students are girls, the other half are boys. 20% of the girls have dogs at home and 10% of the boys have dogs at home. How many students own dogs?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "percent_partition", "invalid"], "notes": "Percent partition is out of family scope for this ablation; must refuse on fraction_decrease conditions."} +{"case_id": "roa-v1-0006", "problem": "In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. If the current temperature of the mountain is 84 degrees and the cabin is 40 degrees, what will the temperature decrease by?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "distractor_quantity", "invalid"], "notes": "Multiple nearby temperatures; organ must refuse rather than pick one."} +{"case_id": "roa-v1-0007", "problem": "The mountain will decrease by 3/4 of its temperature. Current temperature is 84 degrees. What will the temperature decrease by?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "percent_change_vs_of", "invalid"], "notes": "decrease by fraction language (not decrease to); refuse."} +{"case_id": "roa-v1-0008", "problem": "In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature and then to 1/2 of its temperature. If the current temperature of the mountain is 84 degrees, what will the temperature decrease by?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "multi_fraction_hazard", "invalid"], "notes": "Multiple slash-fractions hazard; organ must refuse rather than pick one."} diff --git a/evals/relational_operator_ablation/v1/report.json b/evals/relational_operator_ablation/v1/report.json new file mode 100644 index 00000000..8c0c6d95 --- /dev/null +++ b/evals/relational_operator_ablation/v1/report.json @@ -0,0 +1,1727 @@ +{ + "conditions": [ + "baseline", + "operator", + "depth", + "metadata_only", + "invalid" + ], + "counts": { + "baseline": { + "correct": 2, + "correct_rate": 0.25, + "coverage": 2, + "coverage_rate": 0.25, + "n": 8, + "refused": 6, + "refused_rate": 0.75, + "wrong": 0, + "wrong_rate": 0.0 + }, + "depth": { + "correct": 2, + "correct_rate": 0.25, + "coverage": 2, + "coverage_rate": 0.25, + "n": 8, + "refused": 6, + "refused_rate": 0.75, + "wrong": 0, + "wrong_rate": 0.0 + }, + "invalid": { + "correct": 0, + "correct_rate": 0.0, + "coverage": 0, + "coverage_rate": 0.0, + "n": 6, + "refused": 6, + "refused_rate": 1.0, + "wrong": 0, + "wrong_rate": 0.0 + }, + "metadata_only": { + "correct": 2, + "correct_rate": 0.25, + "coverage": 2, + "coverage_rate": 0.25, + "n": 8, + "refused": 6, + "refused_rate": 0.75, + "wrong": 0, + "wrong_rate": 0.0 + }, + "operator": { + "correct": 2, + "correct_rate": 0.25, + "coverage": 2, + "coverage_rate": 0.25, + "n": 8, + "refused": 6, + "refused_rate": 0.75, + "wrong": 0, + "wrong_rate": 0.0 + } + }, + "determinism": [ + { + "case_id": "roa-v1-0001", + "operator_repeat_identical": true + }, + { + "case_id": "roa-v1-0002", + "operator_repeat_identical": true + }, + { + "case_id": "roa-v1-0003", + "operator_repeat_identical": true + }, + { + "case_id": "roa-v1-0004", + "operator_repeat_identical": true + }, + { + "case_id": "roa-v1-0005", + "operator_repeat_identical": true + }, + { + "case_id": "roa-v1-0006", + "operator_repeat_identical": true + }, + { + "case_id": "roa-v1-0007", + "operator_repeat_identical": true + }, + { + "case_id": "roa-v1-0008", + "operator_repeat_identical": true + } + ], + "family": "proportional_change.decrease_to_fraction", + "findings": { + "baseline_wrong_zero": true, + "depth_executable_inactive_equals_operator": true, + "metadata_only_inert": true, + "operator_deterministic": true, + "operator_solves_gold_fraction_cases": true, + "operator_wrong_zero": true, + "scientific_note": "Depth executable mapping is intentionally inactive on English-only inputs (anti-circularity). Metadata-only he-root labels must not change answers. Operator geometric path and baseline scalar path should agree when both commit." + }, + "inert_checks": [ + { + "baseline_eq_operator_when_both_commit": true, + "case_id": "roa-v1-0001", + "metadata_has_root_note_when_runnable": true, + "operator_eq_depth": true, + "operator_eq_metadata": true + }, + { + "baseline_eq_operator_when_both_commit": true, + "case_id": "roa-v1-0002", + "metadata_has_root_note_when_runnable": true, + "operator_eq_depth": true, + "operator_eq_metadata": true + }, + { + "baseline_eq_operator_when_both_commit": null, + "case_id": "roa-v1-0003", + "metadata_has_root_note_when_runnable": null, + "operator_eq_depth": true, + "operator_eq_metadata": true + }, + { + "baseline_eq_operator_when_both_commit": null, + "case_id": "roa-v1-0004", + "metadata_has_root_note_when_runnable": null, + "operator_eq_depth": true, + "operator_eq_metadata": true + }, + { + "baseline_eq_operator_when_both_commit": null, + "case_id": "roa-v1-0005", + "metadata_has_root_note_when_runnable": null, + "operator_eq_depth": true, + "operator_eq_metadata": true + }, + { + "baseline_eq_operator_when_both_commit": null, + "case_id": "roa-v1-0006", + "metadata_has_root_note_when_runnable": null, + "operator_eq_depth": true, + "operator_eq_metadata": true + }, + { + "baseline_eq_operator_when_both_commit": null, + "case_id": "roa-v1-0007", + "metadata_has_root_note_when_runnable": null, + "operator_eq_depth": true, + "operator_eq_metadata": true + }, + { + "baseline_eq_operator_when_both_commit": null, + "case_id": "roa-v1-0008", + "metadata_has_root_note_when_runnable": true, + "operator_eq_depth": true, + "operator_eq_metadata": true + } + ], + "lane": "relational_operator_ablation/v1", + "limitations": [ + "n=8 sealed cases; not a full GSM8K claim.", + "Depth-as-executable is unproven for English arithmetic; recorded inactive.", + "percent_partition dual-track is out of family scope for this slice.", + "Chat CognitiveTurnPipeline still passes contract_assessment=None." + ], + "organ": "fraction_decrease", + "per_case": [ + { + "baseline": { + "answer": 21.0, + "answer_unit": "degrees", + "case_id": "roa-v1-0001", + "condition": "baseline", + "depth": null, + "expected_answer": 21.0, + "explanation_has_root_note": false, + "frame_runnable": true, + "operator_bindings": 0, + "outcome": "correct", + "readback": { + "base_surface": "mention-0003", + "explanation": "all fraction-decrease roles and delta target obligations are grounded", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0003" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:49:60" + ], + [ + "unit", + "mention-0005" + ] + ], + "scale_surface": "3/4", + "versor_error": 2.220446049250313e-16 + }, + "refusal_reason": "" + }, + "case_id": "roa-v1-0001", + "depth": { + "answer": 21.000000000007148, + "answer_unit": "degrees", + "case_id": "roa-v1-0001", + "condition": "depth", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_executable.fraction_decrease.v0_absent", + "reason": "No authored executable depth\u2192relation mapping for English surface; anti-circularity forbids inventing Greek case / Hebrew binyan from English.", + "source_pack_ids": [], + "source_record_ids": [], + "validity": "inactive_no_mapping" + }, + "expected_answer": 21.0, + "explanation_has_root_note": false, + "frame_runnable": true, + "operator_bindings": 1, + "outcome": "correct", + "readback": { + "base_surface": "mention-0003", + "explanation": "all fraction-decrease roles and delta target obligations are grounded", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0003" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:49:60" + ], + [ + "unit", + "mention-0005" + ] + ], + "scale_surface": "3/4", + "versor_error": 2.220446049250313e-16 + }, + "refusal_reason": "" + }, + "invalid": { + "case_id": "roa-v1-0001", + "condition": "invalid", + "note": "invalid condition reserved for refuse-gold/adversarial tags", + "outcome": "skipped_not_adversarial" + }, + "metadata_only": { + "answer": 21.000000000007148, + "answer_unit": "degrees", + "case_id": "roa-v1-0001", + "condition": "metadata_only", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_metadata.enrich_assessments_with_depth.v1", + "reason": "Depth labels decorate ContractAssessment.explanation only; not executable.", + "source_pack_ids": [ + "synthetic_metadata_only" + ], + "source_record_ids": [ + "p0" + ], + "validity": "metadata_only" + }, + "expected_answer": 21.0, + "explanation_has_root_note": true, + "frame_runnable": true, + "operator_bindings": 1, + "outcome": "correct", + "readback": { + "base_surface": "mention-0003", + "explanation": "all fraction-decrease roles and delta target obligations are grounded [root:\u05d0-\u05de-\u05e0]", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0003" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:49:60" + ], + [ + "unit", + "mention-0005" + ] + ], + "scale_surface": "3/4", + "versor_error": 2.220446049250313e-16 + }, + "refusal_reason": "" + }, + "operator": { + "answer": 21.000000000007148, + "answer_unit": "degrees", + "case_id": "roa-v1-0001", + "condition": "operator", + "depth": null, + "expected_answer": 21.0, + "explanation_has_root_note": false, + "frame_runnable": true, + "operator_bindings": 1, + "outcome": "correct", + "readback": { + "base_surface": "mention-0003", + "explanation": "all fraction-decrease roles and delta target obligations are grounded", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0003" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:49:60" + ], + [ + "unit", + "mention-0005" + ] + ], + "scale_surface": "3/4", + "versor_error": 2.220446049250313e-16 + }, + "refusal_reason": "" + }, + "tags": [ + "fraction_decrease", + "gold", + "train_sample_0005" + ] + }, + { + "baseline": { + "answer": 20.0, + "answer_unit": "degrees", + "case_id": "roa-v1-0002", + "condition": "baseline", + "depth": null, + "expected_answer": 20.0, + "explanation_has_root_note": false, + "frame_runnable": true, + "operator_bindings": 0, + "outcome": "correct", + "readback": { + "base_surface": "mention-0003", + "explanation": "all fraction-decrease roles and delta target obligations are grounded", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0003" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:44:55" + ], + [ + "unit", + "mention-0005" + ] + ], + "scale_surface": "2/3", + "versor_error": 0.0 + }, + "refusal_reason": "" + }, + "case_id": "roa-v1-0002", + "depth": { + "answer": 20.00000000000606, + "answer_unit": "degrees", + "case_id": "roa-v1-0002", + "condition": "depth", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_executable.fraction_decrease.v0_absent", + "reason": "No authored executable depth\u2192relation mapping for English surface; anti-circularity forbids inventing Greek case / Hebrew binyan from English.", + "source_pack_ids": [], + "source_record_ids": [], + "validity": "inactive_no_mapping" + }, + "expected_answer": 20.0, + "explanation_has_root_note": false, + "frame_runnable": true, + "operator_bindings": 1, + "outcome": "correct", + "readback": { + "base_surface": "mention-0003", + "explanation": "all fraction-decrease roles and delta target obligations are grounded", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0003" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:44:55" + ], + [ + "unit", + "mention-0005" + ] + ], + "scale_surface": "2/3", + "versor_error": 0.0 + }, + "refusal_reason": "" + }, + "invalid": { + "case_id": "roa-v1-0002", + "condition": "invalid", + "note": "invalid condition reserved for refuse-gold/adversarial tags", + "outcome": "skipped_not_adversarial" + }, + "metadata_only": { + "answer": 20.00000000000606, + "answer_unit": "degrees", + "case_id": "roa-v1-0002", + "condition": "metadata_only", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_metadata.enrich_assessments_with_depth.v1", + "reason": "Depth labels decorate ContractAssessment.explanation only; not executable.", + "source_pack_ids": [ + "synthetic_metadata_only" + ], + "source_record_ids": [ + "p0" + ], + "validity": "metadata_only" + }, + "expected_answer": 20.0, + "explanation_has_root_note": true, + "frame_runnable": true, + "operator_bindings": 1, + "outcome": "correct", + "readback": { + "base_surface": "mention-0003", + "explanation": "all fraction-decrease roles and delta target obligations are grounded [root:\u05d0-\u05de-\u05e0]", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0003" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:44:55" + ], + [ + "unit", + "mention-0005" + ] + ], + "scale_surface": "2/3", + "versor_error": 0.0 + }, + "refusal_reason": "" + }, + "operator": { + "answer": 20.00000000000606, + "answer_unit": "degrees", + "case_id": "roa-v1-0002", + "condition": "operator", + "depth": null, + "expected_answer": 20.0, + "explanation_has_root_note": false, + "frame_runnable": true, + "operator_bindings": 1, + "outcome": "correct", + "readback": { + "base_surface": "mention-0003", + "explanation": "all fraction-decrease roles and delta target obligations are grounded", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0003" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:44:55" + ], + [ + "unit", + "mention-0005" + ] + ], + "scale_surface": "2/3", + "versor_error": 0.0 + }, + "refusal_reason": "" + }, + "tags": [ + "fraction_decrease", + "gold", + "sibling" + ] + }, + { + "baseline": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0003", + "condition": "baseline", + "depth": null, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": "mention-0003", + "explanation": "diagnostic candidate is not runnable: delta_decrease_target_unbound", + "has_versor_binding": false, + "missing_bindings": [ + "delta_decrease_target_unbound" + ], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0003" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:41:52" + ], + [ + "unit", + "mention-0005" + ] + ], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: delta_decrease_target_unbound" + }, + "case_id": "roa-v1-0003", + "depth": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0003", + "condition": "depth", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_executable.fraction_decrease.v0_absent", + "reason": "No authored executable depth\u2192relation mapping for English surface; anti-circularity forbids inventing Greek case / Hebrew binyan from English.", + "source_pack_ids": [], + "source_record_ids": [], + "validity": "inactive_no_mapping" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 1, + "outcome": "refused", + "readback": { + "base_surface": "mention-0003", + "explanation": "Assessed via geometric algebra from frame-grounded scale", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0003" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:41:52" + ], + [ + "unit", + "mention-0005" + ] + ], + "scale_surface": "3/4", + "versor_error": 2.220446049250313e-16 + }, + "refusal_reason": "diagnostic candidate is not runnable: delta_decrease_target_unbound" + }, + "invalid": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0003", + "condition": "invalid", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "invalid.control", + "reason": "Adversarial/invalid control case; refuse preferred over wrong.", + "source_pack_ids": [], + "source_record_ids": [], + "validity": "invalid" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 1, + "outcome": "refused", + "readback": { + "base_surface": "mention-0003", + "explanation": "Assessed via geometric algebra from frame-grounded scale", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0003" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:41:52" + ], + [ + "unit", + "mention-0005" + ] + ], + "scale_surface": "3/4", + "versor_error": 2.220446049250313e-16 + }, + "refusal_reason": "diagnostic candidate is not runnable: delta_decrease_target_unbound" + }, + "metadata_only": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0003", + "condition": "metadata_only", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_metadata.enrich_assessments_with_depth.v1", + "reason": "Depth labels decorate ContractAssessment.explanation only; not executable.", + "source_pack_ids": [ + "synthetic_metadata_only" + ], + "source_record_ids": [ + "p0" + ], + "validity": "metadata_only" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": "mention-0003", + "explanation": "diagnostic candidate is not runnable: delta_decrease_target_unbound", + "has_versor_binding": false, + "missing_bindings": [ + "delta_decrease_target_unbound" + ], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0003" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:41:52" + ], + [ + "unit", + "mention-0005" + ] + ], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: delta_decrease_target_unbound" + }, + "operator": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0003", + "condition": "operator", + "depth": null, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 1, + "outcome": "refused", + "readback": { + "base_surface": "mention-0003", + "explanation": "Assessed via geometric algebra from frame-grounded scale", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0003" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:41:52" + ], + [ + "unit", + "mention-0005" + ] + ], + "scale_surface": "3/4", + "versor_error": 2.220446049250313e-16 + }, + "refusal_reason": "diagnostic candidate is not runnable: delta_decrease_target_unbound" + }, + "tags": [ + "adversarial", + "final_value", + "invalid" + ] + }, + { + "baseline": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0004", + "condition": "baseline", + "depth": null, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, delta_decrease_target_required, unbound_base_quantity", + "has_versor_binding": false, + "missing_bindings": [ + "decrease_relation_ambiguous", + "base_quantity_unbound", + "scale_unbound", + "state_entity_unbound", + "delta_decrease_target_required" + ], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, delta_decrease_target_required, unbound_base_quantity" + }, + "case_id": "roa-v1-0004", + "depth": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0004", + "condition": "depth", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_executable.fraction_decrease.v0_absent", + "reason": "No authored executable depth\u2192relation mapping for English surface; anti-circularity forbids inventing Greek case / Hebrew binyan from English.", + "source_pack_ids": [], + "source_record_ids": [], + "validity": "inactive_no_mapping" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "no_fraction_decrease_assessment", + "has_versor_binding": false, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, delta_decrease_target_required, unbound_base_quantity" + }, + "invalid": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0004", + "condition": "invalid", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "invalid.control", + "reason": "Adversarial/invalid control case; refuse preferred over wrong.", + "source_pack_ids": [], + "source_record_ids": [], + "validity": "invalid" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "no_fraction_decrease_assessment", + "has_versor_binding": false, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, delta_decrease_target_required, unbound_base_quantity" + }, + "metadata_only": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0004", + "condition": "metadata_only", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_metadata.enrich_assessments_with_depth.v1", + "reason": "Depth labels decorate ContractAssessment.explanation only; not executable.", + "source_pack_ids": [ + "synthetic_metadata_only" + ], + "source_record_ids": [ + "p0" + ], + "validity": "metadata_only" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "no_fraction_decrease_in_assess_contracts", + "has_versor_binding": false, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "operator_refused" + }, + "operator": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0004", + "condition": "operator", + "depth": null, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "no_fraction_decrease_assessment", + "has_versor_binding": false, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, delta_decrease_target_required, unbound_base_quantity" + }, + "tags": [ + "adversarial", + "affine_confuser", + "invalid" + ] + }, + { + "baseline": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0005", + "condition": "baseline", + "depth": null, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, delta_decrease_target_required, unbound_base_quantity", + "has_versor_binding": false, + "missing_bindings": [ + "decrease_relation_ambiguous", + "base_quantity_unbound", + "scale_unbound", + "state_entity_unbound", + "delta_decrease_target_required" + ], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, delta_decrease_target_required, unbound_base_quantity" + }, + "case_id": "roa-v1-0005", + "depth": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0005", + "condition": "depth", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_executable.fraction_decrease.v0_absent", + "reason": "No authored executable depth\u2192relation mapping for English surface; anti-circularity forbids inventing Greek case / Hebrew binyan from English.", + "source_pack_ids": [], + "source_record_ids": [], + "validity": "inactive_no_mapping" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "no_fraction_decrease_assessment", + "has_versor_binding": false, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, delta_decrease_target_required, unbound_base_quantity" + }, + "invalid": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0005", + "condition": "invalid", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "invalid.control", + "reason": "Adversarial/invalid control case; refuse preferred over wrong.", + "source_pack_ids": [], + "source_record_ids": [], + "validity": "invalid" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "no_fraction_decrease_assessment", + "has_versor_binding": false, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, delta_decrease_target_required, unbound_base_quantity" + }, + "metadata_only": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0005", + "condition": "metadata_only", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_metadata.enrich_assessments_with_depth.v1", + "reason": "Depth labels decorate ContractAssessment.explanation only; not executable.", + "source_pack_ids": [ + "synthetic_metadata_only" + ], + "source_record_ids": [ + "p0" + ], + "validity": "metadata_only" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "no_fraction_decrease_in_assess_contracts", + "has_versor_binding": false, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "operator_refused" + }, + "operator": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0005", + "condition": "operator", + "depth": null, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "no_fraction_decrease_assessment", + "has_versor_binding": false, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, delta_decrease_target_required, unbound_base_quantity" + }, + "tags": [ + "adversarial", + "percent_partition", + "invalid" + ] + }, + { + "baseline": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0006", + "condition": "baseline", + "depth": null, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, unbound_base_quantity", + "has_versor_binding": false, + "missing_bindings": [ + "decrease_relation_ambiguous", + "base_quantity_unbound", + "scale_unbound", + "state_entity_unbound" + ], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, unbound_base_quantity" + }, + "case_id": "roa-v1-0006", + "depth": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0006", + "condition": "depth", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_executable.fraction_decrease.v0_absent", + "reason": "No authored executable depth\u2192relation mapping for English surface; anti-circularity forbids inventing Greek case / Hebrew binyan from English.", + "source_pack_ids": [], + "source_record_ids": [], + "validity": "inactive_no_mapping" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "Assessed via geometric algebra from frame-grounded scale", + "has_versor_binding": false, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, unbound_base_quantity" + }, + "invalid": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0006", + "condition": "invalid", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "invalid.control", + "reason": "Adversarial/invalid control case; refuse preferred over wrong.", + "source_pack_ids": [], + "source_record_ids": [], + "validity": "invalid" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "Assessed via geometric algebra from frame-grounded scale", + "has_versor_binding": false, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, unbound_base_quantity" + }, + "metadata_only": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0006", + "condition": "metadata_only", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_metadata.enrich_assessments_with_depth.v1", + "reason": "Depth labels decorate ContractAssessment.explanation only; not executable.", + "source_pack_ids": [ + "synthetic_metadata_only" + ], + "source_record_ids": [ + "p0" + ], + "validity": "metadata_only" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, unbound_base_quantity", + "has_versor_binding": false, + "missing_bindings": [ + "decrease_relation_ambiguous", + "base_quantity_unbound", + "scale_unbound", + "state_entity_unbound" + ], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, unbound_base_quantity" + }, + "operator": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0006", + "condition": "operator", + "depth": null, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "Assessed via geometric algebra from frame-grounded scale", + "has_versor_binding": false, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, unbound_base_quantity" + }, + "tags": [ + "adversarial", + "distractor_quantity", + "invalid" + ] + }, + { + "baseline": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0007", + "condition": "baseline", + "depth": null, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, delta_decrease_target_unbound, unbound_base_quantity", + "has_versor_binding": false, + "missing_bindings": [ + "decrease_relation_ambiguous", + "base_quantity_unbound", + "scale_unbound", + "state_entity_unbound", + "delta_decrease_target_unbound" + ], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, delta_decrease_target_unbound, unbound_base_quantity" + }, + "case_id": "roa-v1-0007", + "depth": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0007", + "condition": "depth", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_executable.fraction_decrease.v0_absent", + "reason": "No authored executable depth\u2192relation mapping for English surface; anti-circularity forbids inventing Greek case / Hebrew binyan from English.", + "source_pack_ids": [], + "source_record_ids": [], + "validity": "inactive_no_mapping" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "no_fraction_decrease_assessment", + "has_versor_binding": false, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, delta_decrease_target_unbound, unbound_base_quantity" + }, + "invalid": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0007", + "condition": "invalid", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "invalid.control", + "reason": "Adversarial/invalid control case; refuse preferred over wrong.", + "source_pack_ids": [], + "source_record_ids": [], + "validity": "invalid" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "no_fraction_decrease_assessment", + "has_versor_binding": false, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, delta_decrease_target_unbound, unbound_base_quantity" + }, + "metadata_only": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0007", + "condition": "metadata_only", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_metadata.enrich_assessments_with_depth.v1", + "reason": "Depth labels decorate ContractAssessment.explanation only; not executable.", + "source_pack_ids": [ + "synthetic_metadata_only" + ], + "source_record_ids": [ + "p0" + ], + "validity": "metadata_only" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "no_fraction_decrease_in_assess_contracts", + "has_versor_binding": false, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "operator_refused" + }, + "operator": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0007", + "condition": "operator", + "depth": null, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": false, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": null, + "explanation": "no_fraction_decrease_assessment", + "has_versor_binding": false, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [], + "scale_surface": null, + "versor_error": null + }, + "refusal_reason": "diagnostic candidate is not runnable: decrease_relation_ambiguous, base_quantity_unbound, scale_unbound, state_entity_unbound, delta_decrease_target_unbound, unbound_base_quantity" + }, + "tags": [ + "adversarial", + "percent_change_vs_of", + "invalid" + ] + }, + { + "baseline": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0008", + "condition": "baseline", + "depth": null, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": true, + "operator_bindings": 0, + "outcome": "refused", + "readback": { + "base_surface": "mention-0005", + "explanation": "all fraction-decrease roles and delta target obligations are grounded", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0005" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:49:60" + ], + [ + "unit", + "mention-0007" + ] + ], + "scale_surface": "3/4", + "versor_error": 2.220446049250313e-16 + }, + "refusal_reason": "baseline_hazard_surface" + }, + "case_id": "roa-v1-0008", + "depth": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0008", + "condition": "depth", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_executable.fraction_decrease.v0_absent", + "reason": "No authored executable depth\u2192relation mapping for English surface; anti-circularity forbids inventing Greek case / Hebrew binyan from English.", + "source_pack_ids": [], + "source_record_ids": [], + "validity": "inactive_no_mapping" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": true, + "operator_bindings": 1, + "outcome": "refused", + "readback": { + "base_surface": "mention-0005", + "explanation": "all fraction-decrease roles and delta target obligations are grounded", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0005" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:49:60" + ], + [ + "unit", + "mention-0007" + ] + ], + "scale_surface": "3/4", + "versor_error": 2.220446049250313e-16 + }, + "refusal_reason": "operator_refused" + }, + "invalid": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0008", + "condition": "invalid", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "invalid.control", + "reason": "Adversarial/invalid control case; refuse preferred over wrong.", + "source_pack_ids": [], + "source_record_ids": [], + "validity": "invalid" + }, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": true, + "operator_bindings": 1, + "outcome": "refused", + "readback": { + "base_surface": "mention-0005", + "explanation": "all fraction-decrease roles and delta target obligations are grounded", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0005" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:49:60" + ], + [ + "unit", + "mention-0007" + ] + ], + "scale_surface": "3/4", + "versor_error": 2.220446049250313e-16 + }, + "refusal_reason": "operator_refused" + }, + "metadata_only": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0008", + "condition": "metadata_only", + "depth": { + "evidence_spans": [], + "mapping_rule_id": "depth_metadata.enrich_assessments_with_depth.v1", + "reason": "Depth labels decorate ContractAssessment.explanation only; not executable.", + "source_pack_ids": [ + "synthetic_metadata_only" + ], + "source_record_ids": [ + "p0" + ], + "validity": "metadata_only" + }, + "expected_answer": null, + "explanation_has_root_note": true, + "frame_runnable": true, + "operator_bindings": 1, + "outcome": "refused", + "readback": { + "base_surface": "mention-0005", + "explanation": "all fraction-decrease roles and delta target obligations are grounded [root:\u05d0-\u05de-\u05e0]", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0005" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:49:60" + ], + [ + "unit", + "mention-0007" + ] + ], + "scale_surface": "3/4", + "versor_error": 2.220446049250313e-16 + }, + "refusal_reason": "all fraction-decrease roles and delta target obligations are grounded [root:\u05d0-\u05de-\u05e0]" + }, + "operator": { + "answer": null, + "answer_unit": null, + "case_id": "roa-v1-0008", + "condition": "operator", + "depth": null, + "expected_answer": null, + "explanation_has_root_note": false, + "frame_runnable": true, + "operator_bindings": 1, + "outcome": "refused", + "readback": { + "base_surface": "mention-0005", + "explanation": "all fraction-decrease roles and delta target obligations are grounded", + "has_versor_binding": true, + "missing_bindings": [], + "relation_type": "decrease_to_fraction", + "roles": [ + [ + "base_quantity", + "mention-0005" + ], + [ + "scale", + "mention-0001" + ], + [ + "state_entity", + "mention-0000" + ], + [ + "transition", + "span:49:60" + ], + [ + "unit", + "mention-0007" + ] + ], + "scale_surface": "3/4", + "versor_error": 2.220446049250313e-16 + }, + "refusal_reason": "operator_refused" + }, + "tags": [ + "adversarial", + "multi_fraction_hazard", + "invalid" + ] + } + ], + "report_sha256": "53df6d17df934f0c44f5d1063b16259f6d9db0b59834b7ca20f895eb5e5719c7", + "sample_size": 8, + "schema_version": 1 +} diff --git a/evals/relational_operator_ablation/v1/runner.py b/evals/relational_operator_ablation/v1/runner.py new file mode 100644 index 00000000..1ef26120 --- /dev/null +++ b/evals/relational_operator_ablation/v1/runner.py @@ -0,0 +1,251 @@ +"""Sealed measurement runner for Deterministic Relational Operator Ablation v1. + +Conditions: baseline | operator | depth | metadata_only | invalid (adversarial cases). + +Run: + PYTHONPATH=. python -m evals.relational_operator_ablation.v1.runner + +Writes ``report.json`` beside this module (override via CORE_ROA_REPORT_PATH). +""" + +from __future__ import annotations + +import hashlib +import json +import os +import sys +from collections import Counter +from pathlib import Path +from typing import Any + +from generate.relational_operator_ablation import ( + AblationCase, + ConditionName, + answers_match, + run_all_conditions, + run_condition, +) + +_HERE = Path(__file__).resolve().parent +_CASES = _HERE / "cases.jsonl" +_REPORT = Path( + os.environ.get("CORE_ROA_REPORT_PATH", str(_HERE / "report.json")) +) +_SCHEMA_VERSION = 1 +_LANE = "relational_operator_ablation/v1" +_CONDITIONS: tuple[ConditionName, ...] = ( + "baseline", + "operator", + "depth", + "metadata_only", + "invalid", +) + + +def _load_cases(path: Path = _CASES) -> list[AblationCase]: + cases: list[AblationCase] = [] + for line in path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if not line: + continue + raw = json.loads(line) + cases.append( + AblationCase( + case_id=raw["case_id"], + problem=raw["problem"], + expected_answer=raw.get("expected_answer"), + expected_unit=raw.get("expected_unit") or "", + expected_outcome=raw["expected_outcome"], + tags=tuple(raw.get("tags") or ()), + notes=raw.get("notes") or "", + ) + ) + return cases + + +def _empty_counts() -> dict[str, int]: + return {"correct": 0, "wrong": 0, "refused": 0, "n": 0, "coverage": 0} + + +def build_report(cases: list[AblationCase]) -> dict[str, Any]: + per_condition: dict[str, dict[str, int]] = { + c: _empty_counts() for c in _CONDITIONS + } + per_case: list[dict[str, Any]] = [] + inert_checks: list[dict[str, Any]] = [] + determinism: list[dict[str, Any]] = [] + + for case in cases: + row: dict[str, Any] = {"case_id": case.case_id, "tags": list(case.tags)} + results = run_all_conditions(case) + + # Always score primary four conditions. + for name in ("baseline", "operator", "depth", "metadata_only"): + result = results[name] + bucket = per_condition[name] + bucket["n"] += 1 + bucket[result.outcome] += 1 + if result.answer is not None: + bucket["coverage"] += 1 + row[name] = result.as_json() + + # Invalid condition only for adversarial/refuse-gold cases. + if "invalid" in results: + result = results["invalid"] + bucket = per_condition["invalid"] + bucket["n"] += 1 + bucket[result.outcome] += 1 + if result.answer is not None: + bucket["coverage"] += 1 + row["invalid"] = result.as_json() + else: + row["invalid"] = { + "condition": "invalid", + "case_id": case.case_id, + "outcome": "skipped_not_adversarial", + "note": "invalid condition reserved for refuse-gold/adversarial tags", + } + + # Metadata-only must match operator answers (inert depth labels). + op = results["operator"] + meta = results["metadata_only"] + depth = results["depth"] + base = results["baseline"] + inert_checks.append( + { + "case_id": case.case_id, + "operator_eq_metadata": answers_match(op, meta), + "operator_eq_depth": answers_match(op, depth), + "baseline_eq_operator_when_both_commit": ( + answers_match(base, op) + if base.answer is not None and op.answer is not None + else None + ), + "metadata_has_root_note_when_runnable": meta.explanation_has_root_note + if meta.frame_runnable + else None, + } + ) + + # Determinism: re-run operator twice. + again = run_condition("operator", case) + determinism.append( + { + "case_id": case.case_id, + "operator_repeat_identical": answers_match(op, again) + and op.refusal_reason == again.refusal_reason, + } + ) + per_case.append(row) + + # Coverage fraction + for name, bucket in per_condition.items(): + n = bucket["n"] or 1 + bucket["coverage_rate"] = bucket["coverage"] / n + bucket["correct_rate"] = bucket["correct"] / n + bucket["wrong_rate"] = bucket["wrong"] / n + bucket["refused_rate"] = bucket["refused"] / n + + inert_ok = all(c["operator_eq_metadata"] for c in inert_checks) + depth_inactive_ok = all(c["operator_eq_depth"] for c in inert_checks) + det_ok = all(d["operator_repeat_identical"] for d in determinism) + no_wrong_on_operator = per_condition["operator"]["wrong"] == 0 + no_wrong_on_baseline = per_condition["baseline"]["wrong"] == 0 + + findings = { + "metadata_only_inert": inert_ok, + "depth_executable_inactive_equals_operator": depth_inactive_ok, + "operator_deterministic": det_ok, + "operator_wrong_zero": no_wrong_on_operator, + "baseline_wrong_zero": no_wrong_on_baseline, + "operator_solves_gold_fraction_cases": per_condition["operator"]["correct"] + >= 2, + "scientific_note": ( + "Depth executable mapping is intentionally inactive on English-only " + "inputs (anti-circularity). Metadata-only he-root labels must not " + "change answers. Operator geometric path and baseline scalar path " + "should agree when both commit." + ), + } + + body = { + "schema_version": _SCHEMA_VERSION, + "lane": _LANE, + "family": "proportional_change.decrease_to_fraction", + "organ": "fraction_decrease", + "sample_size": len(cases), + "conditions": list(_CONDITIONS), + "counts": per_condition, + "findings": findings, + "inert_checks": inert_checks, + "determinism": determinism, + "per_case": per_case, + "limitations": [ + "n=8 sealed cases; not a full GSM8K claim.", + "Depth-as-executable is unproven for English arithmetic; recorded inactive.", + "percent_partition dual-track is out of family scope for this slice.", + "Chat CognitiveTurnPipeline still passes contract_assessment=None.", + ], + } + # Content-addressed digest of counts+findings+case outcomes (stable). + digest_src = json.dumps( + { + "counts": per_condition, + "findings": findings, + "outcomes": [ + { + "id": row["case_id"], + "baseline": row["baseline"]["outcome"], + "operator": row["operator"]["outcome"], + "depth": row["depth"]["outcome"], + "metadata_only": row["metadata_only"]["outcome"], + } + for row in per_case + ], + }, + sort_keys=True, + separators=(",", ":"), + ).encode("utf-8") + body["report_sha256"] = hashlib.sha256(digest_src).hexdigest() + return body + + +def write_report(report: dict[str, Any], path: Path = _REPORT) -> None: + path.write_text( + json.dumps(report, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + +def main() -> int: + cases = _load_cases() + report = build_report(cases) + write_report(report) + findings = report["findings"] + ok = ( + findings["metadata_only_inert"] + and findings["depth_executable_inactive_equals_operator"] + and findings["operator_deterministic"] + and findings["operator_wrong_zero"] + and findings["baseline_wrong_zero"] + and findings["operator_solves_gold_fraction_cases"] + ) + print( + json.dumps( + { + "lane": _LANE, + "sample_size": report["sample_size"], + "counts": report["counts"], + "findings": findings, + "report_sha256": report["report_sha256"], + "passed": ok, + }, + indent=2, + sort_keys=True, + ) + ) + return 0 if ok else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/generate/relational_operator_ablation.py b/generate/relational_operator_ablation.py new file mode 100644 index 00000000..2f5c1015 --- /dev/null +++ b/generate/relational_operator_ablation.py @@ -0,0 +1,582 @@ +"""Deterministic Relational Operator Ablation — fraction_decrease vertical slice. + +Scientific framing +------------------ +Hypothesis under test (narrow): explicit canonical relation operators compiled +into the existing geometric (VersorBinding / CGA) layer are measurable against +a math-frame-only baseline, and depth *metadata* is inert unless an explicit +executable mapping is activated. + +This module does **not**: +- translate English problems into Hebrew or Greek, +- invent case/binyan roles from English, +- relax refusal gates, +- or claim full-GSM8K competence. + +Conditions +---------- +- ``baseline``: ProblemFrame roles + pure rational arithmetic (no VersorBinding). +- ``operator``: live Gate-A2k geometric path (``resolve_promotable_fraction_decrease``). +- ``depth``: same as operator with an explicit depth-contribution record. + On English-only inputs there is **no** valid executable depth mapping + (anti-circularity); contribution is recorded as inactive. +- ``metadata_only``: depth labels passed through ``assess_contracts`` (decorate + explanations) then operator path — must not change the answer. +- ``invalid`` / adversarial: typed refusal, never a guessed answer. + +Integrity model (project-native, not invented dual-adjoint jargon) +------------------------------------------------------------------- +- VersorBinding enforces ``versor_condition < 1e-6`` at construction. +- Organ self-verification requires grounded fraction token + base quantity. +- Readback reconstructs roles, scale surface, spans, and condition lineage. +""" + +from __future__ import annotations + +from dataclasses import asdict, dataclass, field +from fractions import Fraction +from typing import Any, Literal, Mapping + +from generate.derivation.fraction_decrease import ( + _asks_decrease_delta, + _has_hazard_surface, + compose_fraction_decrease, + resolve_promotable_fraction_decrease, +) +from generate.derivation.target import _question_clause +from generate.problem_frame_builder import build_problem_frame +from generate.problem_frame_contracts import ( + ContractAssessment, + assess_contracts, + assess_fraction_decrease, + assess_geometric_proposals, +) + +ConditionName = Literal[ + "baseline", + "operator", + "depth", + "metadata_only", + "invalid", +] + +OutcomeName = Literal["correct", "wrong", "refused"] + +# Stable mapping-rule id for the (currently empty) English executable depth path. +DEPTH_EXECUTABLE_MAPPING_RULE: str = "depth_executable.fraction_decrease.v0_absent" +DEPTH_METADATA_RULE: str = "depth_metadata.enrich_assessments_with_depth.v1" + + +@dataclass(frozen=True, slots=True) +class DepthContribution: + """Provenance for any depth-derived constraint (executable or inactive).""" + + mapping_rule_id: str + source_pack_ids: tuple[str, ...] + source_record_ids: tuple[str, ...] + evidence_spans: tuple[tuple[int, int, str], ...] + validity: Literal["inactive_no_mapping", "metadata_only", "executable", "invalid"] + reason: str + + def as_json(self) -> dict[str, Any]: + return asdict(self) + + +@dataclass(frozen=True, slots=True) +class RelationReadback: + """Human-auditable reconstruction of the grounded relation.""" + + relation_type: str + roles: tuple[tuple[str, str], ...] # (role, target_id_or_surface) + scale_surface: str | None + base_surface: str | None + missing_bindings: tuple[str, ...] + explanation: str + has_versor_binding: bool + versor_error: float | None + + def as_json(self) -> dict[str, Any]: + return asdict(self) + + +@dataclass(frozen=True, slots=True) +class AblationResult: + condition: ConditionName + case_id: str + outcome: OutcomeName + answer: float | None + answer_unit: str | None + expected_answer: float | None + refusal_reason: str + readback: RelationReadback | None + depth: DepthContribution | None + frame_runnable: bool + operator_bindings: int + explanation_has_root_note: bool = False + + def as_json(self) -> dict[str, Any]: + payload = { + "condition": self.condition, + "case_id": self.case_id, + "outcome": self.outcome, + "answer": self.answer, + "answer_unit": self.answer_unit, + "expected_answer": self.expected_answer, + "refusal_reason": self.refusal_reason, + "frame_runnable": self.frame_runnable, + "operator_bindings": self.operator_bindings, + "explanation_has_root_note": self.explanation_has_root_note, + "readback": None if self.readback is None else self.readback.as_json(), + "depth": None if self.depth is None else self.depth.as_json(), + } + return payload + + +@dataclass(frozen=True, slots=True) +class AblationCase: + case_id: str + problem: str + expected_answer: float | None + expected_unit: str + expected_outcome: Literal["correct", "refused"] + tags: tuple[str, ...] = () + depth_override: Mapping[str, Mapping[str, str]] | None = None + notes: str = "" + + +def _score( + *, + answer: float | None, + expected: float | None, + expected_outcome: str, +) -> OutcomeName: + if answer is None: + return "refused" + if expected is None: + # Gold is abstention; any committed answer is wrong. + return "wrong" + if abs(float(answer) - float(expected)) < 1e-6: + return "correct" + return "wrong" + + +def _readback_from_assessment( + assessment: ContractAssessment, + frame_roles: tuple[tuple[str, str], ...], +) -> RelationReadback: + scale_surface = None + versor_error = None + has_binding = bool(assessment.bindings) + if assessment.bindings: + scale_surface = assessment.bindings[0].semantic_identity + versor_error = float(assessment.bindings[0].versor_error) + return RelationReadback( + relation_type="decrease_to_fraction", + roles=frame_roles, + scale_surface=scale_surface, + base_surface=next((t for r, t in frame_roles if r == "base_quantity"), None), + missing_bindings=tuple(assessment.missing_bindings), + explanation=assessment.explanation, + has_versor_binding=has_binding, + versor_error=versor_error, + ) + + +def _frame_roles(frame) -> tuple[tuple[str, str], ...]: + roles: list[tuple[str, str]] = [] + for relation in frame.bound_relations: + if relation.relation_type != "decrease_to_fraction": + continue + for role in relation.roles: + roles.append((role.role, role.target_id)) + return tuple(roles) + + +def _extract_base_and_scale(frame) -> tuple[Fraction, Fraction, str, str] | None: + """Return (base, scale, base_surface, scale_surface) from grounded frame roles.""" + quantities = {q.fact_id: q for q in frame.quantities} + mentions = {m.mention_id: m for m in frame.mentions} + # Map mention_id → quantity value via fact_id when available. + mention_value: dict[str, Fraction] = {} + for mention in frame.mentions: + if mention.kind != "quantity": + continue + if mention.fact_id and mention.fact_id in quantities: + value = quantities[mention.fact_id].value + mention_value[mention.mention_id] = ( + value if isinstance(value, Fraction) else Fraction(str(value)) + ) + # Prefer role targets from the unique decrease_to_fraction relation. + relations = [ + r for r in frame.bound_relations if r.relation_type == "decrease_to_fraction" + ] + if len(relations) != 1: + return None + relation = relations[0] + base_id = next((r.target_id for r in relation.roles if r.role == "base_quantity"), None) + scale_id = next((r.target_id for r in relation.roles if r.role == "scale"), None) + if base_id is None or scale_id is None: + return None + if base_id not in mention_value or scale_id not in mention_value: + # Fall back: match by surface token against scalars. + return None + base = mention_value[base_id] + scale = mention_value[scale_id] + base_surface = mentions[base_id].surface if base_id in mentions else str(base) + scale_surface = mentions[scale_id].surface if scale_id in mentions else str(scale) + if not (0 < float(scale) < 1): + return None + return base, scale, base_surface, scale_surface + + +def run_baseline(case: AblationCase) -> AblationResult: + """Canonical math-frame path: rational arithmetic from grounded roles only. + + Shares organ *refusal* discipline (question shape + hazard surfaces) with + the live fraction_decrease organ so the ablation compares scalar vs geometric + *execution*, not divergent gate policy. Does **not** apply VersorBinding/CGA. + """ + frame = build_problem_frame(case.problem) + assessment = assess_fraction_decrease(frame) + roles = _frame_roles(frame) + readback = _readback_from_assessment(assessment, roles) + question = _question_clause(case.problem) + + if not assessment.runnable: + return AblationResult( + condition="baseline", + case_id=case.case_id, + outcome=_score(answer=None, expected=case.expected_answer, expected_outcome=case.expected_outcome), + answer=None, + answer_unit=None, + expected_answer=case.expected_answer, + refusal_reason=assessment.explanation or "frame_not_runnable", + readback=readback, + depth=None, + frame_runnable=False, + operator_bindings=0, + ) + + # Shared refuse gates with generate.derivation.fraction_decrease (no geometry). + if not _asks_decrease_delta(question): + return AblationResult( + condition="baseline", + case_id=case.case_id, + outcome="refused", + answer=None, + answer_unit=None, + expected_answer=case.expected_answer, + refusal_reason="baseline_question_not_decrease_delta", + readback=readback, + depth=None, + frame_runnable=True, + operator_bindings=0, + ) + if _has_hazard_surface(case.problem, question): + return AblationResult( + condition="baseline", + case_id=case.case_id, + outcome="refused", + answer=None, + answer_unit=None, + expected_answer=case.expected_answer, + refusal_reason="baseline_hazard_surface", + readback=readback, + depth=None, + frame_runnable=True, + operator_bindings=0, + ) + + extracted = _extract_base_and_scale(frame) + if extracted is None: + return AblationResult( + condition="baseline", + case_id=case.case_id, + outcome="refused", + answer=None, + answer_unit=None, + expected_answer=case.expected_answer, + refusal_reason="baseline_scale_or_base_unresolved", + readback=readback, + depth=None, + frame_runnable=True, + operator_bindings=0, + ) + + base, scale, _base_s, _scale_s = extracted + # decrease_by = base * (1 - scale) — pure rational, no geometric payload. + answer = float(base * (Fraction(1) - scale)) + unit = "" + for mention in frame.mentions: + if mention.kind == "unit": + unit = mention.surface + break + + return AblationResult( + condition="baseline", + case_id=case.case_id, + outcome=_score( + answer=answer, + expected=case.expected_answer, + expected_outcome=case.expected_outcome, + ), + answer=answer, + answer_unit=unit or None, + expected_answer=case.expected_answer, + refusal_reason="", + readback=readback, + depth=None, + frame_runnable=True, + operator_bindings=0, + ) + + +def run_operator(case: AblationCase) -> AblationResult: + """Executable geometric operator path (live A2k organ).""" + frame = build_problem_frame(case.problem) + assessment = assess_fraction_decrease(frame) + geom = assess_geometric_proposals(frame) + geom_fd = next( + (g for g in geom if g.candidate_organ == "fraction_decrease"), + None, + ) + roles = _frame_roles(frame) + # Prefer obligation-complete assessment for readback; fall back to geometric. + read_src = assessment if assessment.runnable else geom_fd + readback = ( + _readback_from_assessment(read_src, roles) + if read_src is not None + else RelationReadback( + relation_type="decrease_to_fraction", + roles=roles, + scale_surface=None, + base_surface=None, + missing_bindings=(), + explanation="no_fraction_decrease_assessment", + has_versor_binding=False, + versor_error=None, + ) + ) + + resolution = resolve_promotable_fraction_decrease(case.problem) + if resolution is None: + reason = "operator_refused" + if not assessment.runnable: + reason = assessment.explanation or reason + elif geom_fd is not None and not geom_fd.runnable: + reason = geom_fd.explanation or reason + return AblationResult( + condition="operator", + case_id=case.case_id, + outcome=_score(answer=None, expected=case.expected_answer, expected_outcome=case.expected_outcome), + answer=None, + answer_unit=None, + expected_answer=case.expected_answer, + refusal_reason=reason, + readback=readback, + depth=None, + frame_runnable=assessment.runnable, + operator_bindings=len(assessment.bindings) if assessment.bindings else ( + len(geom_fd.bindings) if geom_fd else 0 + ), + ) + + return AblationResult( + condition="operator", + case_id=case.case_id, + outcome=_score( + answer=float(resolution.answer), + expected=case.expected_answer, + expected_outcome=case.expected_outcome, + ), + answer=float(resolution.answer), + answer_unit=resolution.answer_unit or None, + expected_answer=case.expected_answer, + refusal_reason="", + readback=readback, + depth=None, + frame_runnable=assessment.runnable, + operator_bindings=len(assessment.bindings) or ( + len(geom_fd.bindings) if geom_fd else 0 + ), + ) + + +def run_depth(case: AblationCase) -> AblationResult: + """Operator path + explicit depth-contribution provenance. + + For English-only problems there is no authored executable depth mapping. + The contribution is recorded as ``inactive_no_mapping`` so the experiment + cannot claim depth value it did not earn. Answer equals the operator path. + """ + op = run_operator(case) + depth = DepthContribution( + mapping_rule_id=DEPTH_EXECUTABLE_MAPPING_RULE, + source_pack_ids=(), + source_record_ids=(), + evidence_spans=(), + validity="inactive_no_mapping", + reason=( + "No authored executable depth→relation mapping for English surface; " + "anti-circularity forbids inventing Greek case / Hebrew binyan from English." + ), + ) + return AblationResult( + condition="depth", + case_id=op.case_id, + outcome=op.outcome, + answer=op.answer, + answer_unit=op.answer_unit, + expected_answer=op.expected_answer, + refusal_reason=op.refusal_reason, + readback=op.readback, + depth=depth, + frame_runnable=op.frame_runnable, + operator_bindings=op.operator_bindings, + explanation_has_root_note=op.explanation_has_root_note, + ) + + +def run_metadata_only( + case: AblationCase, + depth_labels: Mapping[str, Mapping[str, str]] | None = None, +) -> AblationResult: + """Depth labels decorate assessments; operator answer must stay unchanged. + + Labels default to a synthetic he root note that must not affect arithmetic. + """ + labels: dict[str, dict[str, str]] = { + k: dict(v) + for k, v in ( + depth_labels + or case.depth_override + or {"p0": {"language": "he", "root": "א-מ-נ"}} + ).items() + } + frame = build_problem_frame(case.problem) + assessments = assess_contracts(frame, depth=labels) + fd = next((a for a in assessments if a.candidate_organ == "fraction_decrease"), None) + has_root_note = bool( + fd is not None and "[root:" in (fd.explanation or "") + ) + + # Operator path is independent of assess_contracts decoration. + resolution = resolve_promotable_fraction_decrease(case.problem) + roles = _frame_roles(frame) + readback = ( + _readback_from_assessment(fd, roles) + if fd is not None + else RelationReadback( + relation_type="decrease_to_fraction", + roles=roles, + scale_surface=None, + base_surface=None, + missing_bindings=(), + explanation="no_fraction_decrease_in_assess_contracts", + has_versor_binding=False, + versor_error=None, + ) + ) + depth = DepthContribution( + mapping_rule_id=DEPTH_METADATA_RULE, + source_pack_ids=("synthetic_metadata_only",), + source_record_ids=tuple(sorted(labels.keys())), + evidence_spans=(), + validity="metadata_only", + reason="Depth labels decorate ContractAssessment.explanation only; not executable.", + ) + + if resolution is None: + return AblationResult( + condition="metadata_only", + case_id=case.case_id, + outcome=_score(answer=None, expected=case.expected_answer, expected_outcome=case.expected_outcome), + answer=None, + answer_unit=None, + expected_answer=case.expected_answer, + refusal_reason=(fd.explanation if fd is not None else "operator_refused"), + readback=readback, + depth=depth, + frame_runnable=bool(fd and fd.runnable), + operator_bindings=len(fd.bindings) if fd and fd.bindings else 0, + explanation_has_root_note=has_root_note, + ) + + return AblationResult( + condition="metadata_only", + case_id=case.case_id, + outcome=_score( + answer=float(resolution.answer), + expected=case.expected_answer, + expected_outcome=case.expected_outcome, + ), + answer=float(resolution.answer), + answer_unit=resolution.answer_unit or None, + expected_answer=case.expected_answer, + refusal_reason="", + readback=readback, + depth=depth, + frame_runnable=bool(fd and fd.runnable), + operator_bindings=len(fd.bindings) if fd and fd.bindings else 0, + explanation_has_root_note=has_root_note, + ) + + +def run_invalid(case: AblationCase) -> AblationResult: + """Force the invalid/adversarial condition: commit only on clean organ admit.""" + # Same as operator scoring: confusers must refuse, not guess. + result = run_operator(case) + return AblationResult( + condition="invalid", + case_id=result.case_id, + outcome=result.outcome, + answer=result.answer, + answer_unit=result.answer_unit, + expected_answer=result.expected_answer, + refusal_reason=result.refusal_reason or ( + "committed" if result.answer is not None else "refused" + ), + readback=result.readback, + depth=DepthContribution( + mapping_rule_id="invalid.control", + source_pack_ids=(), + source_record_ids=(), + evidence_spans=(), + validity="invalid", + reason="Adversarial/invalid control case; refuse preferred over wrong.", + ), + frame_runnable=result.frame_runnable, + operator_bindings=result.operator_bindings, + ) + + +_CONDITION_RUNNERS = { + "baseline": run_baseline, + "operator": run_operator, + "depth": run_depth, + "metadata_only": run_metadata_only, + "invalid": run_invalid, +} + + +def run_condition(condition: ConditionName, case: AblationCase) -> AblationResult: + return _CONDITION_RUNNERS[condition](case) + + +def run_all_conditions(case: AblationCase) -> dict[ConditionName, AblationResult]: + # For gold-solve cases, skip invalid runner (same as operator); for refuse gold, use invalid. + results: dict[ConditionName, AblationResult] = {} + for name in ("baseline", "operator", "depth", "metadata_only"): + results[name] = run_condition(name, case) # type: ignore[arg-type] + if case.expected_outcome == "refused" or "adversarial" in case.tags or "invalid" in case.tags: + results["invalid"] = run_invalid(case) + return results + + +def answers_match(a: AblationResult, b: AblationResult, *, tol: float = 1e-6) -> bool: + """True when both refuse, or both commit equal answers.""" + if a.answer is None and b.answer is None: + return True + if a.answer is None or b.answer is None: + return False + return abs(float(a.answer) - float(b.answer)) < tol diff --git a/tests/test_relational_operator_ablation.py b/tests/test_relational_operator_ablation.py new file mode 100644 index 00000000..ff620f84 --- /dev/null +++ b/tests/test_relational_operator_ablation.py @@ -0,0 +1,318 @@ +"""Unit, integration, and adversarial proofs for relational operator ablation.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from generate.relational_operator_ablation import ( + AblationCase, + DEPTH_EXECUTABLE_MAPPING_RULE, + DEPTH_METADATA_RULE, + answers_match, + run_all_conditions, + run_baseline, + run_condition, + run_depth, + run_metadata_only, + run_operator, +) +from algebra.versor import versor_condition + +_CASES = ( + Path(__file__).resolve().parents[1] + / "evals" + / "relational_operator_ablation" + / "v1" + / "cases.jsonl" +) + +GOLD_0001 = AblationCase( + case_id="unit-gold-0001", + problem=( + "In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. " + "If the current temperature of the mountain is 84 degrees, what will the temperature " + "decrease by?" + ), + expected_answer=21.0, + expected_unit="degrees", + expected_outcome="correct", + tags=("fraction_decrease", "gold"), +) + +FINAL_VALUE = AblationCase( + case_id="unit-final-value", + problem=( + "In one hour, the lake's temperature will decrease to 3/4 of its temperature. " + "If the current temperature of the lake is 80 degrees, what will the temperature be?" + ), + expected_answer=None, + expected_unit="", + expected_outcome="refused", + tags=("adversarial", "invalid"), +) + +MULTI_FRACTION = AblationCase( + case_id="unit-multi-fraction", + problem=( + "In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature " + "and then to 1/2 of its temperature. If the current temperature of the mountain is 84 " + "degrees, what will the temperature decrease by?" + ), + expected_answer=None, + expected_unit="", + expected_outcome="refused", + tags=("adversarial", "invalid"), +) + +SCALE_OOR_PREEXISTING = AblationCase( + case_id="unit-scale-oor-preexisting", + problem=( + "In one hour, the river will decrease to 5/4 of its level. " + "If the current level is 40 feet, what will the level decrease by?" + ), + expected_answer=None, + expected_unit="", + expected_outcome="refused", + tags=("preexisting_gap",), +) + + +def _load_sealed() -> list[AblationCase]: + cases: list[AblationCase] = [] + for line in _CASES.read_text(encoding="utf-8").splitlines(): + if not line.strip(): + continue + raw = json.loads(line) + cases.append( + AblationCase( + case_id=raw["case_id"], + problem=raw["problem"], + expected_answer=raw.get("expected_answer"), + expected_unit=raw.get("expected_unit") or "", + expected_outcome=raw["expected_outcome"], + tags=tuple(raw.get("tags") or ()), + notes=raw.get("notes") or "", + ) + ) + return cases + + +# --------------------------------------------------------------------------- +# Unit +# --------------------------------------------------------------------------- + + +def test_baseline_pure_rational_no_versor_binding() -> None: + result = run_baseline(GOLD_0001) + assert result.outcome == "correct" + assert result.answer == pytest.approx(21.0) + assert result.operator_bindings == 0 + assert result.readback is not None + assert result.readback.has_versor_binding is True or result.frame_runnable + # Baseline path itself does not *require* bindings to compute; frame may still + # produce them in assess_fraction_decrease for readback. + assert result.condition == "baseline" + + +def test_operator_commits_geometric_binding_with_versor_integrity() -> None: + result = run_operator(GOLD_0001) + assert result.outcome == "correct" + assert result.answer == pytest.approx(21.0, abs=1e-6) + assert result.operator_bindings >= 1 + assert result.readback is not None + assert result.readback.scale_surface == "3/4" + assert result.readback.has_versor_binding is True + assert result.readback.versor_error is not None + assert result.readback.versor_error < 1e-6 + + +def test_metadata_only_decorates_but_does_not_change_answer() -> None: + op = run_operator(GOLD_0001) + meta = run_metadata_only(GOLD_0001) + assert answers_match(op, meta) + assert meta.explanation_has_root_note is True + assert meta.depth is not None + assert meta.depth.validity == "metadata_only" + assert meta.depth.mapping_rule_id == DEPTH_METADATA_RULE + + +def test_depth_condition_records_inactive_mapping_and_matches_operator() -> None: + op = run_operator(GOLD_0001) + depth = run_depth(GOLD_0001) + assert answers_match(op, depth) + assert depth.depth is not None + assert depth.depth.validity == "inactive_no_mapping" + assert depth.depth.mapping_rule_id == DEPTH_EXECUTABLE_MAPPING_RULE + assert depth.depth.source_pack_ids == () + + +def test_determinism_two_runs_identical() -> None: + a = run_operator(GOLD_0001) + b = run_operator(GOLD_0001) + assert answers_match(a, b) + assert a.refusal_reason == b.refusal_reason + assert a.operator_bindings == b.operator_bindings + + +def test_baseline_and_operator_agree_when_both_commit() -> None: + base = run_baseline(GOLD_0001) + op = run_operator(GOLD_0001) + assert base.answer is not None and op.answer is not None + assert answers_match(base, op) + + +# --------------------------------------------------------------------------- +# Integration — sealed fixture end-to-end +# --------------------------------------------------------------------------- + + +def test_sealed_fixture_has_expected_shape() -> None: + cases = _load_sealed() + assert len(cases) == 8 + gold = [c for c in cases if c.expected_outcome == "correct"] + refuse = [c for c in cases if c.expected_outcome == "refused"] + assert len(gold) == 2 + assert len(refuse) == 6 + + +def test_sealed_gold_cases_operator_and_baseline_correct() -> None: + for case in _load_sealed(): + if case.expected_outcome != "correct": + continue + base = run_baseline(case) + op = run_operator(case) + assert base.outcome == "correct", case.case_id + assert op.outcome == "correct", case.case_id + assert answers_match(base, op) + + +def test_sealed_adversarial_cases_refuse_not_wrong() -> None: + for case in _load_sealed(): + if case.expected_outcome != "refused": + continue + op = run_operator(case) + base = run_baseline(case) + assert op.outcome == "refused", (case.case_id, op.outcome, op.answer) + assert base.outcome == "refused", (case.case_id, base.outcome, base.answer) + assert op.outcome != "wrong" + assert base.outcome != "wrong" + + +def test_runner_report_invariants() -> None: + from evals.relational_operator_ablation.v1.runner import build_report, _load_cases + + report = build_report(_load_cases()) + assert report["sample_size"] == 8 + assert report["findings"]["metadata_only_inert"] is True + assert report["findings"]["depth_executable_inactive_equals_operator"] is True + assert report["findings"]["operator_deterministic"] is True + assert report["findings"]["operator_wrong_zero"] is True + assert report["findings"]["baseline_wrong_zero"] is True + assert report["counts"]["operator"]["correct"] >= 2 + assert report["counts"]["operator"]["wrong"] == 0 + assert report["report_sha256"] + # Deterministic digest across two builds. + report2 = build_report(_load_cases()) + assert report["report_sha256"] == report2["report_sha256"] + + +# --------------------------------------------------------------------------- +# Adversarial +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "case", + [FINAL_VALUE, MULTI_FRACTION], + ids=["final_value", "multi_fraction"], +) +def test_adversarial_refuse_closed(case: AblationCase) -> None: + results = run_all_conditions(case) + for name, result in results.items(): + assert result.outcome == "refused", (name, result.outcome, result.answer) + assert result.answer is None + + +def test_preexisting_scale_out_of_range_gap_isolated() -> None: + """Document pre-existing Gate A2k gap: scale>1 may still admit geometrically. + + Frame assess_fraction_decrease marks scale_out_of_range (baseline refuses). + Geometric A2k path may still compute a negative delta — not introduced by + this ablation. Do not mask; pin the split so a future organ fix flips this. + """ + base = run_baseline(SCALE_OOR_PREEXISTING) + op = run_operator(SCALE_OOR_PREEXISTING) + assert base.outcome == "refused" + assert base.answer is None + # Pre-existing: operator may wrong-commit. If a future PR fixes A2k, this + # assertion should be updated to require refuse. + assert op.outcome in {"refused", "wrong"} + if op.outcome == "wrong": + assert op.answer is not None + + +def test_malformed_depth_labels_cannot_elevate_runnable() -> None: + """Garbage depth must not create a runnable assessment that wasn't runnable.""" + # Use a refuse-gold problem. + broken_depth = { + "p0": {"language": "he", "root": "NOT_A_REAL_ROOT_!!!@@@"}, + "p1": {"language": "grc", "root": ""}, + } + refuse_case = FINAL_VALUE + clean = run_operator(refuse_case) + polluted = run_metadata_only(refuse_case, depth_labels=broken_depth) + assert clean.outcome == "refused" + assert polluted.outcome == "refused" + assert polluted.answer is None + # Must not flip to correct via depth. + assert polluted.outcome != "correct" + + +def test_valid_depth_metadata_on_gold_does_not_change_answer() -> None: + depths = { + "p0": {"language": "he", "root": "א-מ-נ"}, + "p1": {"language": "grc", "root": "λογ"}, + } + op = run_operator(GOLD_0001) + meta = run_metadata_only(GOLD_0001, depth_labels=depths) + assert answers_match(op, meta) + assert meta.explanation_has_root_note is True + + +def test_distractor_quantity_case_refuses() -> None: + cases = {c.case_id: c for c in _load_sealed()} + case = cases["roa-v1-0006"] + assert run_operator(case).outcome == "refused" + assert run_baseline(case).outcome == "refused" + + +def test_percent_partition_out_of_family_refuses_on_fraction_organ() -> None: + cases = {c.case_id: c for c in _load_sealed()} + case = cases["roa-v1-0005"] + assert run_operator(case).outcome == "refused" + + +def test_affine_confuser_refuses() -> None: + cases = {c.case_id: c for c in _load_sealed()} + case = cases["roa-v1-0004"] + assert run_operator(case).outcome == "refused" + + +def test_no_mutation_of_input_case() -> None: + problem = GOLD_0001.problem + run_operator(GOLD_0001) + run_baseline(GOLD_0001) + run_metadata_only(GOLD_0001) + assert GOLD_0001.problem == problem + + +def test_readback_preserves_role_provenance_on_gold() -> None: + result = run_operator(GOLD_0001) + assert result.readback is not None + role_names = {r for r, _ in result.readback.roles} + assert "base_quantity" in role_names + assert "scale" in role_names + assert result.readback.scale_surface == "3/4"