core-labs/core
core-labs/core
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Merge pull request 'feat(eval): deterministic relational operator ablation + HE/GRC Logos roadmap' (#87) from feat/deterministic-relational-operator-ablation into main

Browse source 
Reviewed-on: #87
---
Delivered

┌─────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────┐
│ Item                        │ Location                                                                         │
├─────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤
│ Cartography ledger          │ docs/analysis/relational-operator-ablation-cartography-2026-07-19.md             │
├─────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤
│ Ablation dossier            │ docs/analysis/relational-operator-ablation-dossier-2026-07-19.md                 │
├─────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤
│ HE/GRC Logos roadmap        │ docs/analysis/hebrew-koine-greek-logos-pack-capability-roadmap-2026-07-19.md     │
├─────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤
│ Ablation code + sealed eval │ generate/relational_operator_ablation.py, evals/relational_operator_ablation/v1/ │
├─────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤
│ Tests                       │ tests/test_relational_operator_ablation.py                                       │
└─────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────┘

Git

Branch: feat/deterministic-relational-operator-ablation
Remote: Forgejo core-labs/core (5 commits)

1. df029a84 — cartography ledger
2. 1d4d51bc — ablation runners
3. 87b01868 — sealed eval lane v1
4. 84f7a6f7 — tests + ablation dossier
5. 08881ab6 — HE/GRC Logos roadmap + cross-links

Pull request

#87

• Title: feat(eval): deterministic relational operator ablation + HE/GRC Logos roadmap
• Mergeable: yes
• [Verification]: smoke 176 + ablation 20 + runner n=8 wrong=0 documented on the PR

Fresh start for next PR

Roadmap recommends:

┌──────────┬──────────────────────────────────────────────────────────────────────────────┐
│ Field    │ Value                                                                        │
├──────────┼──────────────────────────────────────────────────────────────────────────────┤
│ Branch   │ feat/observed-he-morph-constraint-v0                                         │
├──────────┼──────────────────────────────────────────────────────────────────────────────┤
│ Worktree │ ../core-observed-he-morph-constraint                                         │
├──────────┼──────────────────────────────────────────────────────────────────────────────┤
│ Scope    │ Observed-Hebrew morph constraint only (not pack bulk-load, not English math) │
└──────────┴──────────────────────────────────────────────────────────────────────────────┘
This commit is contained in:
Joshua Matthew-Catudio Shay 2026-07-20 04:16:08 +00:00
commit 5fae5a67f2
10 changed files with 3414 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

220
docs/analysis/hebrew-koine-greek-logos-pack-capability-roadmap-2026-07-19.md Normal file
View file

@ -0,0 +1,220 @@
# Hebrew and Koine Greek Logos Pack Capability Roadmap
**Date:** 2026-07-19
**Type:** No-implementation audit / planning dossier
**Authority tree:** `feat/deterministic-relational-operator-ablation`
**Companion docs:**
- `docs/analysis/relational-operator-ablation-cartography-2026-07-19.md` — Phase 1 cartography + go/no-go
- `docs/analysis/relational-operator-ablation-dossier-2026-07-19.md` — English `fraction_decrease` ablation (IMPLEMENTED AND MEASURED)
- This file — separate HE/GRC Logos capability audit (planning only)
**Boundary:** The relational-operator ablation remains closed. English-only `fraction_decrease` depth was intentionally inert (no observed-morphology basis). That result must not be generalized to “Hebrew/Greek never help,” nor used to authorize English→pseudo-depth. This roadmap does **not** authorize a giant pack build.
---
## 1. Executive verdict
**READY ONLY FOR PREREQUISITE WIRING OR MIGRATION**
| Label | Why |
|-------|-----|
| Not ready for broad Logos implementation | Dual pack systems; sparse morph tags; holonomy crown proof not robust; no executable morph→canonical-relation mapping with provenance |
| Not architecturally blocked | Canonical language-independent seams exist (`BoundRelation`, teaching operators, pack checksum/OOV, `resolve_entry` / `node_depths`) |
| Broader “Logos improves all domains” thesis | Scientifically under-specified until a narrow **observed-HE/GRC** constraint ablation exists |
### Ablation boundary (preserve)
English-only `fraction_decrease` sealed n=8: depth **intentionally inert**. Do not reopen ablation fixtures/results except for demonstrated defects. Do not claim HE/GRC help arithmetic without observed morph input.
---
## 2. Hebrew capability matrix
Status: **LIVE** | **EVALUABLE** | **DIAGNOSTIC** | **TEST-ONLY** | **DORMANT** | **SCAFFOLD** | **SPECIFIED** | **ABSENT** | **CONTRADICTED**
| Capability | Exact evidence | Compiler | Runtime | Proof/tests | Gap | Overclaim risk | Priority |
|------------|----------------|----------|---------|-------------|-----|----------------|----------|
| Structured morphology schema | `packs/schema.py` `MorphologyEntry`; `packs/data/he_*` | LIVE `_apply_morphology` | Manifold via `load_pack` | `test_morphology_registry`, holonomy morph | No paradigm generator | Medium if “full morph” | P0 (tags live) |
| Legacy triliteral / romanized root | `packs/compiler.py` `_is_hebrew_root`, `_triliteral_root` | LIVE feature | Manifold | holonomy same-root | Dual encodings | Low | P1 document/migrate |
| Root identity on resolve | `chat/pack_resolver.py` `resolve_entry`, `resolve_token_depths` | Lookup | LIVE PropGraph / recognition | `test_3lang_depth_capability`, `test_oov_pipeline` | Exact surface match only | Medium | P0 observed-HE |
| Root-sense disambiguation | Multi-surface rare; no sense graph | ABSENT | First-match | — | Homographs silent | **Critical** | **P0 prerequisite** |
| Binyan as licensed semantics | Sparse `binyan` tags (qal/piel) | LIVE as infl rotor | Manifold only | sparse data | No root-specific licensing | **Critical** if universalized | P2 after sense model |
| Verbal aspect | Sparse `aspect: perfect` | Tags only | Not clause-level | source DRAFT notes | Not temporal operators | High | P2–P3 |
| Voice / reflexive / passive | Essentially ABSENT in compiled HE | — | — | — | Niphal/Hithpael not systematic | High | P3 |
| Construct state | 1× construct tag; source `he:construct-chain` frame | Tag only | Frames DISCONNECTED | — | Not executable possession | High if genitive-universalized | P1 candidate |
| Nominal gender/number | LIVE tags | LIVE rotors | Tags | registry | No agreement checker | Medium | P2 |
| Pronominal suffixes | Sparse `suffix_chain` | LIVE | Ingest OOV decomp | ingest gate | Not full paradigm | Medium | P3 |
| Lexical valency / frames | `packs/he/frames.jsonl` (4 frames) | **DISCONNECTED** from `packs/data` | Not in `load_pack` | CLI validate only | No LIVE frame→BoundRelation | High | **P1 migration** |
| Semantic domains | lexicon `semantic_domains` | LIVE | Manifold | domain contracts | Not morph-derived | Low | Keep |
| Alignment edges | `alignment.jsonl` | Load nudge | Geometry | alignment tests | Data ≠ reasoning | Medium | Honest limits |
| Readback HE | `_assemble_he`, ADR-0030 hedges; no `readback_rules.py` | — | Limited assembly | identity depth tests | Fluency C01 only | High if “full readback” | See fluency honesty |
| Correction / contradiction | Teaching chains ADR-0102 | N/A | Teaching/eval | domain tests | Not morph-triggered | Medium | Tie morph to existing ops |
| OOV fail-closed | `gate_engaged=true`, `fail_closed` | Enforced | LIVE | pack runtime tests | OK | Low | Preserve |
| Fluency eval | `evals/hebrew_fluency` | — | Articulation | contract.md C01 | Not morph-constraint eval | High if misused | Separate lane |
| Executable morph→canonical relation | — | ABSENT | ABSENT | ABSENT | Core Logos gap | Critical | Next vertical slice |
---
## 3. Greek capability matrix
| Capability | Exact evidence | Compiler | Runtime | Proof/tests | Gap | Overclaim risk | Priority |
|------------|----------------|----------|---------|-------------|-----|----------------|----------|
| Lemma inventory | `packs/data/grc_*` (11 / 38) | LIVE | `resolve_entry` | registry, 3lang | Micro seed | Medium | P0 honesty |
| Sense disambiguation | No compiled `senses.jsonl` on data packs | ABSENT | First-match | — | Colwell needs sense+syntax | **Critical** | **P0 prerequisite** |
| Case tags | Nearly all **nominative** only | LIVE rotors | Tags only | registry order | No productive case paradigm | **Critical** if case→role | P1 observed forms only |
| Number/gender/agreement | Tags present | LIVE | No agreement engine | — | No S-V check | High | P2 |
| Prep + case | Source `el:relational-pros` | DISCONNECTED | ABSENT | — | Not enforced | High | After case diversity |
| Valency / frames | `packs/grc\|el/frames.jsonl` | DISCONNECTED | ABSENT | — | Dual-system gap | High | **P1 migration** |
| Aspect / tense-form | Present indicative dominant | Tags | Not operators | — | Aorist/perfect system ABSENT | High | P2–P3 licensed tables |
| Voice / mood | Sparse active/middle; indicative | Tags | — | — | No full mood system | High | P3 |
| Participles / infinitives | ABSENT as productive class | — | — | — | Out of seed scope | Medium | Defer |
| Semantic domains | `logos.core`, etc. | LIVE | Manifold | ADR-0102 | OK | Low | Keep |
| Alignment | grc↔he↔en edges | Load nudge | Geometry | alignment tests | Not robust holonomy | Medium | Honest limits |
| Colwell / anarthrous | Source frame + notes | DISCONNECTED | ABSENT | — | Not runtime | High if claimed LIVE | P2 fail-closed |
| Readback GRC | `_assemble_grc` | — | Limited | depth tests | Fluency-class | Medium | `evals/koine_greek_fluency` |
| Geometric signature for GRC | `resolve_geometric_signature` needs `senses.jsonl` | ABSENT for depth packs | EN math when present | fraction EN | HE/GRC cannot supply scale signatures | Medium | Domain concern |
| Executable morph→canonical relation | ABSENT | ABSENT | ABSENT | ABSENT | Core gap | Critical | Observed-GRC later |
---
## 4. Shared Logos substrate matrix
| Capability | Evidence | Compiler | Runtime | Proof/tests | Gap | Overclaim risk | Priority |
|------------|----------|----------|---------|-------------|-----|----------------|----------|
| Roles DEPTH_ROOT / DEPTH_RELATION | `LanguageRole`; manifests | Schema LIVE | Load | ADR-0015 | Source `el` vs runtime `grc` | Medium | Prefer `grc` |
| Checksum + OOV | `load_pack` | LIVE | ChatRuntime | pack tests | Source ADR-0005 gates never true | Low | Keep |
| Ordered morph composition | `_apply_morphology` | LIVE | Manifold | holonomy morph | Not semantic licensing | Medium | Keep |
| Alignment graph | `alignment/graph.py` | Load nudge | Geometry | alignment tests | Not clause reasoning | High if “reasoning” | Data + local geometry |
| Holonomy encode | `algebra/holonomy.py` | LIVE encode | Ingest path | **Honest-fail crown** | Crown **not robust** | **Critical** | Do not claim trilingual proof |
| PropGraph depth spine | `GraphNode` language/root/morphology_id | N/A | LIVE he/grc | 3lang tests | No BoundRelation emission | Medium | Bridge for constraints |
| Depth decorate vs decide | `enrich_assessments_with_depth` vs anti_unifier | — | Match LIVE; math decorate | 3lang, oov | EN math correctly inert | Low if labeled | Preserve |
| Canonical roles | `BoundRelation`, `RoleObligation` | N/A | Math LIVE (fraction) | problem_frame tests | Language-independent | Low | **Reuse only** |
| Teaching causal/contradiction | `hebrew_greek_textual_reasoning_chains_v1.jsonl` | N/A | Teaching/domain | ADR-0102 | Lexeme-level, not morph | Medium | Constraint consumer |
| GrammarAttractor | schema | Thin | DORMANT | — | Not hot path | Medium | Defer |
| `packs.evidence` | holonomy helpers | — | TEST-ONLY / dormant | holonomy tests | Not production | Medium | Don’t build runtime on it |
| Sensorium HE/GRC | `gate_engaged=False` default | Scaffold | Divergent from data True | — | Dual truth | Medium | Align or drop fiction |
| Domain contract | ADR-0091/0102 four packs | Manifest | Ledger reasoning-capable | ratification tests | Fluency ≠ morph-constraint | Medium | Keep honesty |
**Trilingual alignment:** curated alignment data + local manifold geometry; **clause holonomy crown proof is CONTRADICTED as robust** (`docs/analysis/holonomy-resonance-proof-not-robust-2026-06-14.md`, honest-fail tests).
---
## 5. Pack taxonomy (strict)
| Tier | Contents | Must not smuggle |
|------|----------|------------------|
| Universal canonical Logos primitives | `BoundRelation`, RoleObligations, epistemic status, VersorBinding/CGA, teaching operator families, shared `semantic_domains` | HE/GRC-only hidden reasoners |
| Hebrew surface + mapping rules | Observed surfaces, roots, tags, **authored** mapping rules + provenance | English→HE inference; universal binyan→semantics |
| Greek surface + mapping rules | Observed lemmas/forms, attested case tags, **authored** prep+case rules | Nominative=agent; English→case |
| English articulation | Operational base, realizer, fluency | Treating EN as depth proof |
| Domain packs | math/physics/etc. organs | Domain-private morph ontology |
| Anchor / lens / register | `packs/anchor_lens/he_*`, `grc_*` | Lens as morphology engine |
| Safety / identity | identity/safety/ethics; ADR-0030 hedges | Identity as role solver |
| Evaluation / mastery | fluency, fabrication_control, inference_closure, ablations | Fluency pass as morph-constraint pass |
---
## 6. Phased vertical-slice roadmap
### Phase 0 — Prerequisites (no bulk pack growth)
1. Freeze dual-system map: LIVE = `packs/data/*`; source `packs/{he,grc,el}` = draft/disconnected.
2. Sense/ambiguity policy: first-match is fail-open for meaning → multi-candidate or refuse on collision.
3. Entrypoint already LIVE: HE `resolve_entry` / `define אמת` path (or GRC lemma path).
4. No lexicon expansion until Phase 1 ablation.
### Phase 1 — Smallest observed-morphology constraint (separate implementation PR)
| Condition | Behavior |
|-----------|----------|
| Canonical only | Teaching operator/chain without morph constraint |
| Executable morph constraint | Authored rule with provenance → constraint or refuse |
| Metadata-only | `node_depths` / root notes; **must not** change outcome |
| Invalid / ambiguous | Homograph without sense rule → **refuse** |
**Consumer:** existing teaching/contradiction/abstention seams — **not** GSM8K, not new ontology.
**Inputs:** observed HE/GRC only.
**Success:** wrong=0; metadata inert; outcome changes only with provenance.
### Phase 2 — One morph phenomenon
- HE: root identity + ambiguity refuse → construct-state candidate → binyan only with license tables
- GRC: lemma+attested case as evidence only → prep+case after non-nominative data → aspect later
### Phase 3 — Schema/compiler extensions (only if Phase 1 shows value)
Optional `mapping_rules.jsonl`; sense inventory; compile-time rule validation; constraint readback.
### Phase 4 — Broader domains
Only after Phase 1–2 pass; never by inventing morph for English word problems.
---
## 7. No-go list
- Decorative packs not consumed by runtime/eval
- Unlicensed morph→semantics (nominative=agent, Hiphil=always cause, aorist=punctual, …)
- Parallel HE/GRC reasoner outside shared canonical operators
- English→Greek case / Hebrew binyan pseudo-evidence
- Coverage via guessing / relaxing refuse
- Claiming holonomy crown proof is LIVE
- Fluency C01 or ADR-0102 reasoning-capable as morph-constraint proof
- Broad lexicon growth before observed-morph ablation
- Filling all binyanim/case tables as “progress” without rules+eval
- Building production Logos on dormant `packs.evidence`
- Reopening English fraction_decrease ablation to “add depth value” without observed morph
---
## 8. Minimum contract for future domain packs
1. Shared language-independent constraint catalog IDs
2. Mapping rules: rule_id, language, features, preconditions, counterexamples, validity, fail-closed
3. Provenance: pack_id, version, morphology_id/entry_id, spans
4. Deterministic compile → immutable ordered candidates
5. Readback to surface features + canonical constraint
6. Contradiction with COHERENT evidence → abstain, never override
7. Eval: observed-language inputs + metadata-only + sealed holdout
8. No pack-private geometric meaning space (ADR-0005)
---
## 9. Future ablation/proof design (not authorized here)
| Arm | HE | GRC | Cross-lang | Canonical |
|-----|----|-----|------------|-----------|
| Input | Observed HE | Observed GRC | Authored paired clauses | Language-agnostic operator |
| Executable | Morph rule → constraint | Morph rule → constraint | Data-checked alignment (not crown holonomy until metric fixed) | Existing operators |
| Metadata-only | Roots on, rule off | Case tags on, rule off | Alignment labels off | N/A |
| Fail | Ambiguous root/sense | Ambiguous lemma/case | Conflicting alignments | Insufficient evidence |
---
## 10. Recommended next implementation unit
| Field | Value |
|-------|--------|
| **Branch** | `feat/observed-he-morph-constraint-v0` |
| **Worktree** | `../core-observed-he-morph-constraint` |
| **Why HE first** | LIVE root resolution + denser depth tests; GRC case inventory nominative-skewed |
| **PR-sized scope** | Ambiguity refuse; one authored mapping-rule type + provenance; consumer = teaching/contradiction/abstention; sealed observed-HE fixture; metadata-only control; no bulk lexicon; no English math; no holonomy crown “fix” |
| **Out of scope** | Pack expansion, GRC case tables, binyan universals, fluency C02–C13, dual ontology |
### Reviewer acceptance criteria (future PR)
- [ ] Observed Hebrew inputs (or explicit OOV refuse)
- [ ] Mapping rules with rule_id, preconditions, counterexamples, provenance
- [ ] Metadata-only outcome-identical to rule-off
- [ ] Ambiguous → refuse; wrong=0
- [ ] No English→HE morph invention
- [ ] No holonomy crown proof claim
- [ ] Does not modify relational-operator ablation fixtures/results
- [ ] Smoke + targeted tests green
---
## 11. One-line summary
Hebrew/Greek Logos packs are **LIVE as sparse compiled manifolds + depth recognition + domain teaching chains**, but **not** as executable, falsifiable morph→canonical-constraint systems. Holonomy crown proof is **not robust**. Next work is **prerequisite wiring + one observed-HE constraint ablation**, not a giant pack build.

152
docs/analysis/relational-operator-ablation-cartography-2026-07-19.md Normal file
View file

@ -0,0 +1,152 @@
# Relational Operator Ablation — Cartography Ledger & Go/No-Go
**Date:** 2026-07-19
**Branch:** `feat/deterministic-relational-operator-ablation`
**Worktree:** `../core-relational-operator-ablation`
**Authority commit:** `d8d62b8e` (main tip at branch creation)
**Status:** Evidence ledger for Phase 1–2; implementation of the narrow GO slice is companion code under `generate/relational_operator_ablation.py` and `evals/relational_operator_ablation/`.
**Related:** HE/GRC broader capability is **not** authorized by this ledger. See `docs/analysis/hebrew-koine-greek-logos-pack-capability-roadmap-2026-07-19.md` (verdict: READY ONLY FOR PREREQUISITE WIRING OR MIGRATION).
---
## 1. Executive decision (closed)
**Verdict for this work product:** **IMPLEMENTED AND MEASURED** (narrow slice only).
| Scope | Decision |
|-------|----------|
| Full hypothesis: ancient morphology automatically improves GSM8K arithmetic | **NO-GO** (scientific precondition: no valid English→depth mapping; anti-circularity) |
| Narrow fraction_decrease relational-operator ablation | **GO** — family already LIVE with ProblemFrame + VersorBinding + A2k |
| Depth as *executable* math contribution on English word problems | **NO-GO for implementation of new mapping** — metadata-only control proves inert |
**Relation family chosen from live evidence:** `proportional_change.decrease_to_fraction` / organ `fraction_decrease`
(`generate/construction_affordances.py` `_SERVING_AUTHORIZED_FAMILIES`; `generate/derivation/fraction_decrease.py`; Gate A2k in `generate/math_candidate_graph.py`).
### Decision criteria checklist
| Criterion | Pass/Fail | Evidence |
|-----------|-----------|----------|
| Canonical relation seam exists | **Pass** | `BoundRelation(relation_type="decrease_to_fraction")`, roles via `problem_frame_bound_relations` |
| Deterministic reconstructable operator | **Pass** | `VersorBinding` + CGA dilation; same text → same answer |
| Correction/readback or native integrity | **Pass (native)** | Self-verify + span provenance + `versor_condition < 1e-6`; no fabricated dual-adjoint |
| Testable eval lane | **Pass** | Sealed `evals/relational_operator_ablation/v1` + existing train_sample organ tests |
| Baseline vs operator vs depth vs metadata-only | **Pass** | Ablation conditions in harness |
| Fail-closed | **Pass** | Missing/ambiguous/hazard → refuse |
| Narrow non-claim scope | **Pass** | No full GSM8K claim; no EN→HE/GRC translation |
---
## 2. Live-path maps
### A. Text/chat — LIVE (not math serving)
```text
text → CognitiveTurnPipeline.run (core/cognition/pipeline.py)
→ tokenize / resolve_token_depths (chat/pack_resolver.py)
→ recognize (recognition/anti_unifier.py; root-canonical for he/grc)
→ ChatRuntime.chat (chat/runtime.py)
→ ingest.gate → FieldState
→ PropositionGraph (generate/graph_planner.py)
→ realizer / gates / refusal
→ CognitiveTurnResult.node_depths
```
`contract_assessment=None` at `resolve_surface` — ProblemFrame **DISCONNECTED** from chat spine.
### B. Language packs — dual layer
| Layer | Path | Status |
|-------|------|--------|
| Source ADR-0005 | `packs/{en,he,grc,el}/` status=draft, gates false | **DRAFT / DISCONNECTED** from compiler |
| Compiled | `packs/data/*` → `packs.compiler.load_pack` → `_apply_morphology` | **LIVE** |
| Depth IDs | `DEPTH_PACK_IDS` = he_core_cognition_v1, he_logos_micro_v1, grc_logos_* | **LIVE** |
### C. Problem-frame / math
```text
text → build_problem_frame (generate/problem_frame_builder.py) [DIAGNOSTIC IR + A2k input]
→ proposals / BoundRelation / BoundQuestionTarget
→ assess_fraction_decrease / assess_geometric_proposals
→ VersorBinding dilation
→ resolve_promotable_fraction_decrease (generate/derivation/fraction_decrease.py)
→ answer [LIVE Gate A2k]
parse_and_solve (generate/math_candidate_graph.py) [LIVE GSM8K serving orchestrator]
```
| Family | serving_allowed | Status |
|--------|-----------------|--------|
| fraction_decrease | **True** | **LIVE** A2k |
| percent_partition | False catalog; live prose A2l | **LIVE but DISCONNECTED** from frame |
| quantity_entity | False | **DIAGNOSTIC** |
| unary_delta | False | **DIAGNOSTIC** (no derivation organ) |
### D. Evaluation
| Split | N | correct / wrong / refused | Scorer |
|-------|---|---------------------------|--------|
| Sealed GSM8K test (recorded) | 1319 | **0 / 0 / 1319** | operator-only holdout |
| train_sample report.json | 50 | **30 / 0 / 20** | `_score_one_candidate_graph` |
| holdout_dev report.json | 500 | **5 / 0 / 495** | same |
| CORE public synthetic | 150 | **150 / 0 / 0** | classic `_score_one` |
**Note:** Order is always **correct / wrong / refused**. Prior prose that wrote `30/20/0` swapped wrong/refused.
---
## 3. Design truth matrix
| Claim | Spec | Implementation | Runtime | Tests | Status |
|-------|------|----------------|---------|-------|--------|
| Hebrew root morphology | ADR-0015 | `MorphologyEntry.root`; `_apply_morphology` | LIVE manifold + resolver | `test_morphology_registry`, holonomy | **LIVE (sparse)** |
| Hebrew binyan | ADR-0005 | Inflection tag only | geometric tag | sparse data | **Partial** |
| Greek case/aspect/voice | ADR-0015 | Compiled inflection tags | LIVE tags | morphology_registry | **Partial** |
| Anchor lens | pack inventory | `packs/anchor_lens/` | LIVE overlay | inventory | **LIVE** |
| PropGraph / depth | 3lang plan | `depth_canonical`, pipeline | LIVE he/grc input | `test_3lang_depth_capability` | **LIVE spine** |
| Depth → math answer | — | `enrich_assessments_with_depth` explanation only | decoration | same | **Inert on math** |
| Canonical math roles | kernel substrate | BoundRelation roles | A2k + diagnostic | problem_frame tests | **LIVE fraction** |
| VersorBinding | CGA construction boundary | dilation payload | LIVE A2k | fraction + 3lang tests | **LIVE** |
| Readback / integrity | organ self-verify | spans + verified derivation | LIVE | fraction_decrease | **Partial (native)** |
| GSM8K eval | ADR-0119+ | train_sample/holdout/sealed | LIVE open | runners | **LIVE** |
| serving_allowed all False | prior claim | one True family | — | construction_affordances | **Contradicted** |
| FGL | prior claim | — | — | — | **ABSENT** |
| ADR-0248 as Logos | prior claim | integrity handoffs | observe-only | ring3 | **Miscited** |
| ADR-0196 ancient packs | prior claim | Python/Rust/Zig | — | ADR text | **False** |
| EN GSM8K → GRC case / HE binyan | prior claim | — | — | — | **ABSENT** |
---
## 4. Claim corrections (prior synthesis)
1. **EN/HE/GRC roles** — Partially confirmed (ADRs + LIVE compiled packs; source packs draft).
2. **ADR-0005/0015** — Confirmed as architecture; implementation is `packs/data` compiler path.
3. **packs/he, packs/grc draft + compiler** — Confirmed split: draft source vs LIVE compiled.
4. **Ordered morph composition** — Confirmed LIVE+tested in `packs/compiler._apply_morphology`.
5. **depth_canonical / enrich** — Confirmed; enrich is **decoration**; recognize is **matching**.
6. **ProblemFrame symbols** — Confirmed under `generate/`.
7. **All serving_allowed=False** — **False**; fraction_decrease is True.
8. **No EN→case/binyan math path** — Confirmed absent.
9. **GSM8K all refused** — Sealed test **0/0/1319** recorded; train_sample **30/0/20**; holdout_dev **5/0/495**. claims_ledger C **7/43/0** is **stale**.
10. **FGL / ADR-0248 Logos** — FGL absent; ADR-0248 = integrity handoffs.
11. **ADR-0196** — Python/Rust/Zig, not ancient packs.
---
## 5. Implementation plan (narrow GO)
1. Pure condition runners: baseline scalar from frame roles; operator geometric; metadata-only depth; invalid refuse.
2. No new depth→role mapping from English.
3. Sealed fixture v1 (fraction_decrease + confusers).
4. Measurement report + tests.
5. Smoke gate.
---
## 6. Remaining gaps (not in this slice)
- Engineering: wire ProblemFrame into CognitiveTurnPipeline (optional future).
- Engineering: unify percent_partition prose organ with frame contracts.
- Scientific: whether *observed* he/grc morphology can constrain math on bilingual inputs (unproven; not this PR).
- Rejected: inventing Greek cases for English GSM8K.
- Deferred: expanding draft `packs/he|grc` gates.

154
docs/analysis/relational-operator-ablation-dossier-2026-07-19.md Normal file
View file

@ -0,0 +1,154 @@
# Deterministic Relational Operator Ablation — Final Engineering Dossier
**Branch:** `feat/deterministic-relational-operator-ablation`
**Worktree:** `/Users/kaizenpro/Projects/core-relational-operator-ablation`
**Date:** 2026-07-19
**Document set on this branch:**
| Doc | Role |
|-----|------|
| `docs/analysis/relational-operator-ablation-cartography-2026-07-19.md` | Phase 1 cartography, design-truth matrix, go/no-go |
| `docs/analysis/relational-operator-ablation-dossier-2026-07-19.md` | This file — ablation implementation + measurement |
| `docs/analysis/hebrew-koine-greek-logos-pack-capability-roadmap-2026-07-19.md` | Separate HE/GRC Logos audit (planning only; no pack build) |
---
## Executive verdict
**IMPLEMENTED AND MEASURED**
Narrow vertical slice only: family `proportional_change.decrease_to_fraction` / organ `fraction_decrease`.
Ancient-language → GSM8K arithmetic is **not** claimed and is **not** implemented (scientific NO-GO under anti-circularity).
**Depth on English-only cases was intentionally inert** (no observed-morphology basis). That does not test observed Hebrew/Greek input or the broader Logos-pack thesis — see the HE/GRC roadmap for the audited next path (`feat/observed-he-morph-constraint-v0`).
---
## Ground-truth corrections
| Prior claim | Classification | Evidence |
|-------------|----------------|----------|
| EN articulation / HE depth-root / GRC depth-relation | Partially confirmed | ADR-0005/0015; LIVE `packs/data/he_*`, `grc_*`; draft `packs/he\|grc` |
| ADR-0005/0015 architecture | Confirmed | Accepted ADRs; compiler path `packs.compiler._apply_morphology` |
| Source packs draft + morph compiler | Confirmed (split) | Source DISCONNECTED; compiled LIVE |
| Ordered morph composition | Confirmed LIVE | `MorphologyEntry`; tests holonomy/registry |
| depth_canonical / enrich | Confirmed; enrich=decoration | `recognition/depth_canonical.py`; math inert |
| ProblemFrame symbols | Confirmed | `generate/problem_frame*.py` |
| All `serving_allowed=False` | **False** | fraction_decrease **True** in `construction_affordances.py` |
| EN→Greek case/Hebrew binyan math path | **Absent** | No imports in math organs |
| GSM8K all refused | Split-dependent | Sealed **0/0/1319**; train_sample **30/0/20**; holdout_dev **5/0/495** (correct/wrong/refused) |
| FGL / ADR-0248 as Logos | **Absent / miscited** | ADR-0248 = integrity handoffs |
| ADR-0196 ancient packs | **False** | Python/Rust/Zig doctrine |
---
## Architecture map
See `docs/analysis/relational-operator-ablation-cartography-2026-07-19.md`.
Summary:
- **Chat path LIVE**; ProblemFrame **DISCONNECTED** (`contract_assessment=None`).
- **Math serving LIVE** via `parse_and_solve`; Gate A2k uses ProblemFrame + VersorBinding.
- **Depth LIVE** for he/grc recognition; **inert** for English math answers.
---
## Implemented mechanism
| Layer | Mechanism |
|-------|-----------|
| Canonical relation | `BoundRelation(relation_type="decrease_to_fraction")` with roles base_quantity, scale, state_entity, transition, unit |
| Baseline operator | Pure rational `base * (1 - scale)` from grounded frame roles + shared organ hazard refuse gates |
| Geometric operator | Live `resolve_promotable_fraction_decrease` → CGA dilation `VersorBinding` |
| Depth-derived mapping | **None executable** on English; recorded `inactive_no_mapping` with rule id |
| Metadata-only | `assess_contracts(..., depth=)` appends `[root:…]`; must not change answer |
| Provenance | Spans, scale surface, roles, mapping_rule_id, pack ids (empty when inactive) |
| Readback | `RelationReadback` reconstructs roles, scale, missing bindings, versor_error |
| Fail-closed | Missing/hazard/adversarial → refuse; wrong=0 on sealed fixture |
Axioms: geometry-first (CGA dilation), field/structured IR (ProblemFrame), propagation via versor_apply, native integrity (not invented dual-adjoint), reconstruction via readback, compilation-last (no new ontology), reality-over-inheritance (depth inert proven).
---
## Files changed
| Path | Purpose | Key symbols |
|------|---------|-------------|
| `docs/analysis/relational-operator-ablation-cartography-2026-07-19.md` | Evidence ledger + go/no-go | — |
| `docs/analysis/relational-operator-ablation-dossier-2026-07-19.md` | This dossier | — |
| `generate/relational_operator_ablation.py` | Condition runners | `run_baseline`, `run_operator`, `run_depth`, `run_metadata_only`, `run_invalid` |
| `evals/relational_operator_ablation/` | Sealed lane | `v1/cases.jsonl`, `v1/runner.py`, `v1/report.json` |
| `tests/test_relational_operator_ablation.py` | Unit/integration/adversarial proofs | 20 tests |
---
## Proof and test matrix
| Command | Result |
|---------|--------|
| `PYTHONPATH=. python3 -m evals.relational_operator_ablation.v1.runner` | passed; report_sha256 `53df6d17…` |
| `PYTHONPATH=. python3 -m pytest -q tests/test_relational_operator_ablation.py` | **20 passed** |
| `uv run core test --suite smoke -q` | **176 passed** (~132s) |
Key tests:
- baseline rational / operator geometric / metadata inert / depth inactive
- sealed gold correct; adversarial refuse not wrong
- preexisting scale>1 A2k gap isolated (not masked)
- determinism, no input mutation, readback provenance
---
## Ablation results (n=8 sealed)
Order: **correct / wrong / refused**
| Condition | correct | wrong | refused | coverage | Notes |
|-----------|---------|-------|---------|----------|-------|
| baseline | 2 | 0 | 6 | 2/8 | Frame roles + rational math |
| operator | 2 | 0 | 6 | 2/8 | Live VersorBinding path |
| depth | 2 | 0 | 6 | 2/8 | = operator; mapping inactive |
| metadata_only | 2 | 0 | 6 | 2/8 | Root notes present; answers match operator |
| invalid (adversarial subset n=6) | 0 | 0 | 6 | 0/6 | Prefer refuse over wrong |
**Findings:** metadata_only inert ✓; depth inactive equals operator ✓; wrong=0 ✓; deterministic ✓.
**Limitations:** n=8; not GSM8K-wide; depth executable unproven for English; chat spine still disconnected from ProblemFrame; pre-existing A2k scale>1 gap documented separately.
**Scientific conclusion:** Canonical relation operators (baseline scalar vs geometric) agree when both commit; depth **metadata does not improve arithmetic** on this English slice; executable depth mapping was correctly refused as inactive rather than fabricated.
---
## Remaining gaps
| Kind | Item |
|------|------|
| Engineering | Wire ProblemFrame into CognitiveTurnPipeline backpressure |
| Engineering | Unify percent_partition prose organ with frame contracts |
| Engineering | Fix A2k scale>1 admit (pre-existing; isolated test) |
| Scientific | Whether observed he/grc morphology can constrain bilingual math |
| Deferred | Expand draft source packs / holonomy crown proof |
| Rejected | Invent Greek case / Hebrew binyan from English GSM8K |
---
## PR-ready summary
**Branch:** `feat/deterministic-relational-operator-ablation`
**Title:** `feat(eval): deterministic relational operator ablation (fraction_decrease)`
**Description:**
Evidence-led cartography of language packs, depth spine, and ProblemFrame/math paths. Implements a narrow sealed ablation for `fraction_decrease` comparing math-frame baseline, geometric operator, inactive depth contribution, metadata-only control, and adversarial refuse. Proves depth metadata is inert on English arithmetic; does not invent ancient-language role mapping.
`[Verification]: Smoke suite passed locally (176 passed, ~132s); ablation tests 20 passed; ablation runner passed (n=8, wrong=0).`
**Risks:** Imports private hazard helpers from fraction_decrease for shared refuse gates; scale>1 organ gap remains.
**Rollback:** Revert branch commits; no serving flag changes.
**Reviewer checklist:**
- [ ] Cartography matrix rows cite live paths
- [ ] No English→case/binyan mapping introduced
- [ ] Metadata-only inert invariant holds
- [ ] wrong=0 on sealed fixture
- [ ] Preexisting A2k gap not masked

1
evals/relational_operator_ablation/__init__.py Normal file
View file

@ -0,0 +1 @@
"""Deterministic Relational Operator Ablation eval lane (fraction_decrease)."""

1
evals/relational_operator_ablation/v1/__init__.py Normal file
View file

@ -0,0 +1 @@
"""Sealed v1 fixture + runner for relational operator ablation."""

8
evals/relational_operator_ablation/v1/cases.jsonl Normal file
View file

@ -0,0 +1,8 @@
{"case_id": "roa-v1-0001", "problem": "In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. If the current temperature of the mountain is 84 degrees, what will the temperature decrease by?", "expected_answer": 21.0, "expected_unit": "degrees", "expected_outcome": "correct", "tags": ["fraction_decrease", "gold", "train_sample_0005"], "notes": "Canonical decrease-to-fraction delta; gold=21."}
{"case_id": "roa-v1-0002", "problem": "In two hours, Cedar peak's temperature will decrease to 2/3 of its temperature. If the current temperature of the peak is 60 degrees, what will the temperature decrease by?", "expected_answer": 20.0, "expected_unit": "degrees", "expected_outcome": "correct", "tags": ["fraction_decrease", "gold", "sibling"], "notes": "Sibling shape: 60*(1-2/3)=20."}
{"case_id": "roa-v1-0003", "problem": "In one hour, the lake's temperature will decrease to 3/4 of its temperature. If the current temperature of the lake is 80 degrees, what will the temperature be?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "final_value", "invalid"], "notes": "Final-value question must refuse delta organ (not guess 60)."}
{"case_id": "roa-v1-0004", "problem": "Yun had 20 paperclips initially, but then lost 12. Marion has 1/4 more than what Yun currently has, plus 7. How many paperclips does Marion have?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "affine_confuser", "invalid"], "notes": "Percent/fraction affine confuser; fraction_decrease must refuse."}
{"case_id": "roa-v1-0005", "problem": "A school has 100 students. Half of the students are girls, the other half are boys. 20% of the girls have dogs at home and 10% of the boys have dogs at home. How many students own dogs?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "percent_partition", "invalid"], "notes": "Percent partition is out of family scope for this ablation; must refuse on fraction_decrease conditions."}
{"case_id": "roa-v1-0006", "problem": "In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. If the current temperature of the mountain is 84 degrees and the cabin is 40 degrees, what will the temperature decrease by?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "distractor_quantity", "invalid"], "notes": "Multiple nearby temperatures; organ must refuse rather than pick one."}
{"case_id": "roa-v1-0007", "problem": "The mountain will decrease by 3/4 of its temperature. Current temperature is 84 degrees. What will the temperature decrease by?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "percent_change_vs_of", "invalid"], "notes": "decrease by fraction language (not decrease to); refuse."}
{"case_id": "roa-v1-0008", "problem": "In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature and then to 1/2 of its temperature. If the current temperature of the mountain is 84 degrees, what will the temperature decrease by?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "multi_fraction_hazard", "invalid"], "notes": "Multiple slash-fractions hazard; organ must refuse rather than pick one."}

1727
evals/relational_operator_ablation/v1/report.json Normal file
View file

File diff suppressed because it is too large Load diff

251
evals/relational_operator_ablation/v1/runner.py Normal file
View file

@ -0,0 +1,251 @@
"""Sealed measurement runner for Deterministic Relational Operator Ablation v1.
Conditions: baseline | operator | depth | metadata_only | invalid (adversarial cases).
Run:
PYTHONPATH=. python -m evals.relational_operator_ablation.v1.runner
Writes ``report.json`` beside this module (override via CORE_ROA_REPORT_PATH).
"""
from __future__ import annotations
import hashlib
import json
import os
import sys
from collections import Counter
from pathlib import Path
from typing import Any
from generate.relational_operator_ablation import (
AblationCase,
ConditionName,
answers_match,
run_all_conditions,
run_condition,
)
_HERE = Path(__file__).resolve().parent
_CASES = _HERE / "cases.jsonl"
_REPORT = Path(
os.environ.get("CORE_ROA_REPORT_PATH", str(_HERE / "report.json"))
)
_SCHEMA_VERSION = 1
_LANE = "relational_operator_ablation/v1"
_CONDITIONS: tuple[ConditionName, ...] = (
"baseline",
"operator",
"depth",
"metadata_only",
"invalid",
)
def _load_cases(path: Path = _CASES) -> list[AblationCase]:
cases: list[AblationCase] = []
for line in path.read_text(encoding="utf-8").splitlines():
line = line.strip()
if not line:
continue
raw = json.loads(line)
cases.append(
AblationCase(
case_id=raw["case_id"],
problem=raw["problem"],
expected_answer=raw.get("expected_answer"),
expected_unit=raw.get("expected_unit") or "",
expected_outcome=raw["expected_outcome"],
tags=tuple(raw.get("tags") or ()),
notes=raw.get("notes") or "",
)
)
return cases
def _empty_counts() -> dict[str, int]:
return {"correct": 0, "wrong": 0, "refused": 0, "n": 0, "coverage": 0}
def build_report(cases: list[AblationCase]) -> dict[str, Any]:
per_condition: dict[str, dict[str, int]] = {
c: _empty_counts() for c in _CONDITIONS
}
per_case: list[dict[str, Any]] = []
inert_checks: list[dict[str, Any]] = []
determinism: list[dict[str, Any]] = []
for case in cases:
row: dict[str, Any] = {"case_id": case.case_id, "tags": list(case.tags)}
results = run_all_conditions(case)
# Always score primary four conditions.
for name in ("baseline", "operator", "depth", "metadata_only"):
result = results[name]
bucket = per_condition[name]
bucket["n"] += 1
bucket[result.outcome] += 1
if result.answer is not None:
bucket["coverage"] += 1
row[name] = result.as_json()
# Invalid condition only for adversarial/refuse-gold cases.
if "invalid" in results:
result = results["invalid"]
bucket = per_condition["invalid"]
bucket["n"] += 1
bucket[result.outcome] += 1
if result.answer is not None:
bucket["coverage"] += 1
row["invalid"] = result.as_json()
else:
row["invalid"] = {
"condition": "invalid",
"case_id": case.case_id,
"outcome": "skipped_not_adversarial",
"note": "invalid condition reserved for refuse-gold/adversarial tags",
}
# Metadata-only must match operator answers (inert depth labels).
op = results["operator"]
meta = results["metadata_only"]
depth = results["depth"]
base = results["baseline"]
inert_checks.append(
{
"case_id": case.case_id,
"operator_eq_metadata": answers_match(op, meta),
"operator_eq_depth": answers_match(op, depth),
"baseline_eq_operator_when_both_commit": (
answers_match(base, op)
if base.answer is not None and op.answer is not None
else None
),
"metadata_has_root_note_when_runnable": meta.explanation_has_root_note
if meta.frame_runnable
else None,
}
)
# Determinism: re-run operator twice.
again = run_condition("operator", case)
determinism.append(
{
"case_id": case.case_id,
"operator_repeat_identical": answers_match(op, again)
and op.refusal_reason == again.refusal_reason,
}
)
per_case.append(row)
# Coverage fraction
for name, bucket in per_condition.items():
n = bucket["n"] or 1
bucket["coverage_rate"] = bucket["coverage"] / n
bucket["correct_rate"] = bucket["correct"] / n
bucket["wrong_rate"] = bucket["wrong"] / n
bucket["refused_rate"] = bucket["refused"] / n
inert_ok = all(c["operator_eq_metadata"] for c in inert_checks)
depth_inactive_ok = all(c["operator_eq_depth"] for c in inert_checks)
det_ok = all(d["operator_repeat_identical"] for d in determinism)
no_wrong_on_operator = per_condition["operator"]["wrong"] == 0
no_wrong_on_baseline = per_condition["baseline"]["wrong"] == 0
findings = {
"metadata_only_inert": inert_ok,
"depth_executable_inactive_equals_operator": depth_inactive_ok,
"operator_deterministic": det_ok,
"operator_wrong_zero": no_wrong_on_operator,
"baseline_wrong_zero": no_wrong_on_baseline,
"operator_solves_gold_fraction_cases": per_condition["operator"]["correct"]
>= 2,
"scientific_note": (
"Depth executable mapping is intentionally inactive on English-only "
"inputs (anti-circularity). Metadata-only he-root labels must not "
"change answers. Operator geometric path and baseline scalar path "
"should agree when both commit."
),
}
body = {
"schema_version": _SCHEMA_VERSION,
"lane": _LANE,
"family": "proportional_change.decrease_to_fraction",
"organ": "fraction_decrease",
"sample_size": len(cases),
"conditions": list(_CONDITIONS),
"counts": per_condition,
"findings": findings,
"inert_checks": inert_checks,
"determinism": determinism,
"per_case": per_case,
"limitations": [
"n=8 sealed cases; not a full GSM8K claim.",
"Depth-as-executable is unproven for English arithmetic; recorded inactive.",
"percent_partition dual-track is out of family scope for this slice.",
"Chat CognitiveTurnPipeline still passes contract_assessment=None.",
],
}
# Content-addressed digest of counts+findings+case outcomes (stable).
digest_src = json.dumps(
{
"counts": per_condition,
"findings": findings,
"outcomes": [
{
"id": row["case_id"],
"baseline": row["baseline"]["outcome"],
"operator": row["operator"]["outcome"],
"depth": row["depth"]["outcome"],
"metadata_only": row["metadata_only"]["outcome"],
}
for row in per_case
],
},
sort_keys=True,
separators=(",", ":"),
).encode("utf-8")
body["report_sha256"] = hashlib.sha256(digest_src).hexdigest()
return body
def write_report(report: dict[str, Any], path: Path = _REPORT) -> None:
path.write_text(
json.dumps(report, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
def main() -> int:
cases = _load_cases()
report = build_report(cases)
write_report(report)
findings = report["findings"]
ok = (
findings["metadata_only_inert"]
and findings["depth_executable_inactive_equals_operator"]
and findings["operator_deterministic"]
and findings["operator_wrong_zero"]
and findings["baseline_wrong_zero"]
and findings["operator_solves_gold_fraction_cases"]
)
print(
json.dumps(
{
"lane": _LANE,
"sample_size": report["sample_size"],
"counts": report["counts"],
"findings": findings,
"report_sha256": report["report_sha256"],
"passed": ok,
},
indent=2,
sort_keys=True,
)
)
return 0 if ok else 1
if __name__ == "__main__":
sys.exit(main())

582
generate/relational_operator_ablation.py Normal file
View file

@ -0,0 +1,582 @@
"""Deterministic Relational Operator Ablation — fraction_decrease vertical slice.
Scientific framing
------------------
Hypothesis under test (narrow): explicit canonical relation operators compiled
into the existing geometric (VersorBinding / CGA) layer are measurable against
a math-frame-only baseline, and depth *metadata* is inert unless an explicit
executable mapping is activated.
This module does **not**:
- translate English problems into Hebrew or Greek,
- invent case/binyan roles from English,
- relax refusal gates,
- or claim full-GSM8K competence.
Conditions
----------
- ``baseline``: ProblemFrame roles + pure rational arithmetic (no VersorBinding).
- ``operator``: live Gate-A2k geometric path (``resolve_promotable_fraction_decrease``).
- ``depth``: same as operator with an explicit depth-contribution record.
On English-only inputs there is **no** valid executable depth mapping
(anti-circularity); contribution is recorded as inactive.
- ``metadata_only``: depth labels passed through ``assess_contracts`` (decorate
explanations) then operator path — must not change the answer.
- ``invalid`` / adversarial: typed refusal, never a guessed answer.
Integrity model (project-native, not invented dual-adjoint jargon)
-------------------------------------------------------------------
- VersorBinding enforces ``versor_condition < 1e-6`` at construction.
- Organ self-verification requires grounded fraction token + base quantity.
- Readback reconstructs roles, scale surface, spans, and condition lineage.
"""
from __future__ import annotations
from dataclasses import asdict, dataclass, field
from fractions import Fraction
from typing import Any, Literal, Mapping
from generate.derivation.fraction_decrease import (
_asks_decrease_delta,
_has_hazard_surface,
compose_fraction_decrease,
resolve_promotable_fraction_decrease,
)
from generate.derivation.target import _question_clause
from generate.problem_frame_builder import build_problem_frame
from generate.problem_frame_contracts import (
ContractAssessment,
assess_contracts,
assess_fraction_decrease,
assess_geometric_proposals,
)
ConditionName = Literal[
"baseline",
"operator",
"depth",
"metadata_only",
"invalid",
]
OutcomeName = Literal["correct", "wrong", "refused"]
# Stable mapping-rule id for the (currently empty) English executable depth path.
DEPTH_EXECUTABLE_MAPPING_RULE: str = "depth_executable.fraction_decrease.v0_absent"
DEPTH_METADATA_RULE: str = "depth_metadata.enrich_assessments_with_depth.v1"
@dataclass(frozen=True, slots=True)
class DepthContribution:
"""Provenance for any depth-derived constraint (executable or inactive)."""
mapping_rule_id: str
source_pack_ids: tuple[str, ...]
source_record_ids: tuple[str, ...]
evidence_spans: tuple[tuple[int, int, str], ...]
validity: Literal["inactive_no_mapping", "metadata_only", "executable", "invalid"]
reason: str
def as_json(self) -> dict[str, Any]:
return asdict(self)
@dataclass(frozen=True, slots=True)
class RelationReadback:
"""Human-auditable reconstruction of the grounded relation."""
relation_type: str
roles: tuple[tuple[str, str], ...] # (role, target_id_or_surface)
scale_surface: str | None
base_surface: str | None
missing_bindings: tuple[str, ...]
explanation: str
has_versor_binding: bool
versor_error: float | None
def as_json(self) -> dict[str, Any]:
return asdict(self)
@dataclass(frozen=True, slots=True)
class AblationResult:
condition: ConditionName
case_id: str
outcome: OutcomeName
answer: float | None
answer_unit: str | None
expected_answer: float | None
refusal_reason: str
readback: RelationReadback | None
depth: DepthContribution | None
frame_runnable: bool
operator_bindings: int
explanation_has_root_note: bool = False
def as_json(self) -> dict[str, Any]:
payload = {
"condition": self.condition,
"case_id": self.case_id,
"outcome": self.outcome,
"answer": self.answer,
"answer_unit": self.answer_unit,
"expected_answer": self.expected_answer,
"refusal_reason": self.refusal_reason,
"frame_runnable": self.frame_runnable,
"operator_bindings": self.operator_bindings,
"explanation_has_root_note": self.explanation_has_root_note,
"readback": None if self.readback is None else self.readback.as_json(),
"depth": None if self.depth is None else self.depth.as_json(),
}
return payload
@dataclass(frozen=True, slots=True)
class AblationCase:
case_id: str
problem: str
expected_answer: float | None
expected_unit: str
expected_outcome: Literal["correct", "refused"]
tags: tuple[str, ...] = ()
depth_override: Mapping[str, Mapping[str, str]] | None = None
notes: str = ""
def _score(
*,
answer: float | None,
expected: float | None,
expected_outcome: str,
) -> OutcomeName:
if answer is None:
return "refused"
if expected is None:
# Gold is abstention; any committed answer is wrong.
return "wrong"
if abs(float(answer) - float(expected)) < 1e-6:
return "correct"
return "wrong"
def _readback_from_assessment(
assessment: ContractAssessment,
frame_roles: tuple[tuple[str, str], ...],
) -> RelationReadback:
scale_surface = None
versor_error = None
has_binding = bool(assessment.bindings)
if assessment.bindings:
scale_surface = assessment.bindings[0].semantic_identity
versor_error = float(assessment.bindings[0].versor_error)
return RelationReadback(
relation_type="decrease_to_fraction",
roles=frame_roles,
scale_surface=scale_surface,
base_surface=next((t for r, t in frame_roles if r == "base_quantity"), None),
missing_bindings=tuple(assessment.missing_bindings),
explanation=assessment.explanation,
has_versor_binding=has_binding,
versor_error=versor_error,
)
def _frame_roles(frame) -> tuple[tuple[str, str], ...]:
roles: list[tuple[str, str]] = []
for relation in frame.bound_relations:
if relation.relation_type != "decrease_to_fraction":
continue
for role in relation.roles:
roles.append((role.role, role.target_id))
return tuple(roles)
def _extract_base_and_scale(frame) -> tuple[Fraction, Fraction, str, str] | None:
"""Return (base, scale, base_surface, scale_surface) from grounded frame roles."""
quantities = {q.fact_id: q for q in frame.quantities}
mentions = {m.mention_id: m for m in frame.mentions}
# Map mention_id → quantity value via fact_id when available.
mention_value: dict[str, Fraction] = {}
for mention in frame.mentions:
if mention.kind != "quantity":
continue
if mention.fact_id and mention.fact_id in quantities:
value = quantities[mention.fact_id].value
mention_value[mention.mention_id] = (
value if isinstance(value, Fraction) else Fraction(str(value))
)
# Prefer role targets from the unique decrease_to_fraction relation.
relations = [
r for r in frame.bound_relations if r.relation_type == "decrease_to_fraction"
]
if len(relations) != 1:
return None
relation = relations[0]
base_id = next((r.target_id for r in relation.roles if r.role == "base_quantity"), None)
scale_id = next((r.target_id for r in relation.roles if r.role == "scale"), None)
if base_id is None or scale_id is None:
return None
if base_id not in mention_value or scale_id not in mention_value:
# Fall back: match by surface token against scalars.
return None
base = mention_value[base_id]
scale = mention_value[scale_id]
base_surface = mentions[base_id].surface if base_id in mentions else str(base)
scale_surface = mentions[scale_id].surface if scale_id in mentions else str(scale)
if not (0 < float(scale) < 1):
return None
return base, scale, base_surface, scale_surface
def run_baseline(case: AblationCase) -> AblationResult:
"""Canonical math-frame path: rational arithmetic from grounded roles only.
Shares organ *refusal* discipline (question shape + hazard surfaces) with
the live fraction_decrease organ so the ablation compares scalar vs geometric
*execution*, not divergent gate policy. Does **not** apply VersorBinding/CGA.
"""
frame = build_problem_frame(case.problem)
assessment = assess_fraction_decrease(frame)
roles = _frame_roles(frame)
readback = _readback_from_assessment(assessment, roles)
question = _question_clause(case.problem)
if not assessment.runnable:
return AblationResult(
condition="baseline",
case_id=case.case_id,
outcome=_score(answer=None, expected=case.expected_answer, expected_outcome=case.expected_outcome),
answer=None,
answer_unit=None,
expected_answer=case.expected_answer,
refusal_reason=assessment.explanation or "frame_not_runnable",
readback=readback,
depth=None,
frame_runnable=False,
operator_bindings=0,
)
# Shared refuse gates with generate.derivation.fraction_decrease (no geometry).
if not _asks_decrease_delta(question):
return AblationResult(
condition="baseline",
case_id=case.case_id,
outcome="refused",
answer=None,
answer_unit=None,
expected_answer=case.expected_answer,
refusal_reason="baseline_question_not_decrease_delta",
readback=readback,
depth=None,
frame_runnable=True,
operator_bindings=0,
)
if _has_hazard_surface(case.problem, question):
return AblationResult(
condition="baseline",
case_id=case.case_id,
outcome="refused",
answer=None,
answer_unit=None,
expected_answer=case.expected_answer,
refusal_reason="baseline_hazard_surface",
readback=readback,
depth=None,
frame_runnable=True,
operator_bindings=0,
)
extracted = _extract_base_and_scale(frame)
if extracted is None:
return AblationResult(
condition="baseline",
case_id=case.case_id,
outcome="refused",
answer=None,
answer_unit=None,
expected_answer=case.expected_answer,
refusal_reason="baseline_scale_or_base_unresolved",
readback=readback,
depth=None,
frame_runnable=True,
operator_bindings=0,
)
base, scale, _base_s, _scale_s = extracted
# decrease_by = base * (1 - scale) — pure rational, no geometric payload.
answer = float(base * (Fraction(1) - scale))
unit = ""
for mention in frame.mentions:
if mention.kind == "unit":
unit = mention.surface
break
return AblationResult(
condition="baseline",
case_id=case.case_id,
outcome=_score(
answer=answer,
expected=case.expected_answer,
expected_outcome=case.expected_outcome,
),
answer=answer,
answer_unit=unit or None,
expected_answer=case.expected_answer,
refusal_reason="",
readback=readback,
depth=None,
frame_runnable=True,
operator_bindings=0,
)
def run_operator(case: AblationCase) -> AblationResult:
"""Executable geometric operator path (live A2k organ)."""
frame = build_problem_frame(case.problem)
assessment = assess_fraction_decrease(frame)
geom = assess_geometric_proposals(frame)
geom_fd = next(
(g for g in geom if g.candidate_organ == "fraction_decrease"),
None,
)
roles = _frame_roles(frame)
# Prefer obligation-complete assessment for readback; fall back to geometric.
read_src = assessment if assessment.runnable else geom_fd
readback = (
_readback_from_assessment(read_src, roles)
if read_src is not None
else RelationReadback(
relation_type="decrease_to_fraction",
roles=roles,
scale_surface=None,
base_surface=None,
missing_bindings=(),
explanation="no_fraction_decrease_assessment",
has_versor_binding=False,
versor_error=None,
)
)
resolution = resolve_promotable_fraction_decrease(case.problem)
if resolution is None:
reason = "operator_refused"
if not assessment.runnable:
reason = assessment.explanation or reason
elif geom_fd is not None and not geom_fd.runnable:
reason = geom_fd.explanation or reason
return AblationResult(
condition="operator",
case_id=case.case_id,
outcome=_score(answer=None, expected=case.expected_answer, expected_outcome=case.expected_outcome),
answer=None,
answer_unit=None,
expected_answer=case.expected_answer,
refusal_reason=reason,
readback=readback,
depth=None,
frame_runnable=assessment.runnable,
operator_bindings=len(assessment.bindings) if assessment.bindings else (
len(geom_fd.bindings) if geom_fd else 0
),
)
return AblationResult(
condition="operator",
case_id=case.case_id,
outcome=_score(
answer=float(resolution.answer),
expected=case.expected_answer,
expected_outcome=case.expected_outcome,
),
answer=float(resolution.answer),
answer_unit=resolution.answer_unit or None,
expected_answer=case.expected_answer,
refusal_reason="",
readback=readback,
depth=None,
frame_runnable=assessment.runnable,
operator_bindings=len(assessment.bindings) or (
len(geom_fd.bindings) if geom_fd else 0
),
)
def run_depth(case: AblationCase) -> AblationResult:
"""Operator path + explicit depth-contribution provenance.
For English-only problems there is no authored executable depth mapping.
The contribution is recorded as ``inactive_no_mapping`` so the experiment
cannot claim depth value it did not earn. Answer equals the operator path.
"""
op = run_operator(case)
depth = DepthContribution(
mapping_rule_id=DEPTH_EXECUTABLE_MAPPING_RULE,
source_pack_ids=(),
source_record_ids=(),
evidence_spans=(),
validity="inactive_no_mapping",
reason=(
"No authored executable depth→relation mapping for English surface; "
"anti-circularity forbids inventing Greek case / Hebrew binyan from English."
),
)
return AblationResult(
condition="depth",
case_id=op.case_id,
outcome=op.outcome,
answer=op.answer,
answer_unit=op.answer_unit,
expected_answer=op.expected_answer,
refusal_reason=op.refusal_reason,
readback=op.readback,
depth=depth,
frame_runnable=op.frame_runnable,
operator_bindings=op.operator_bindings,
explanation_has_root_note=op.explanation_has_root_note,
)
def run_metadata_only(
case: AblationCase,
depth_labels: Mapping[str, Mapping[str, str]] | None = None,
) -> AblationResult:
"""Depth labels decorate assessments; operator answer must stay unchanged.
Labels default to a synthetic he root note that must not affect arithmetic.
"""
labels: dict[str, dict[str, str]] = {
k: dict(v)
for k, v in (
depth_labels
or case.depth_override
or {"p0": {"language": "he", "root": "א-מ-נ"}}
).items()
}
frame = build_problem_frame(case.problem)
assessments = assess_contracts(frame, depth=labels)
fd = next((a for a in assessments if a.candidate_organ == "fraction_decrease"), None)
has_root_note = bool(
fd is not None and "[root:" in (fd.explanation or "")
)
# Operator path is independent of assess_contracts decoration.
resolution = resolve_promotable_fraction_decrease(case.problem)
roles = _frame_roles(frame)
readback = (
_readback_from_assessment(fd, roles)
if fd is not None
else RelationReadback(
relation_type="decrease_to_fraction",
roles=roles,
scale_surface=None,
base_surface=None,
missing_bindings=(),
explanation="no_fraction_decrease_in_assess_contracts",
has_versor_binding=False,
versor_error=None,
)
)
depth = DepthContribution(
mapping_rule_id=DEPTH_METADATA_RULE,
source_pack_ids=("synthetic_metadata_only",),
source_record_ids=tuple(sorted(labels.keys())),
evidence_spans=(),
validity="metadata_only",
reason="Depth labels decorate ContractAssessment.explanation only; not executable.",
)
if resolution is None:
return AblationResult(
condition="metadata_only",
case_id=case.case_id,
outcome=_score(answer=None, expected=case.expected_answer, expected_outcome=case.expected_outcome),
answer=None,
answer_unit=None,
expected_answer=case.expected_answer,
refusal_reason=(fd.explanation if fd is not None else "operator_refused"),
readback=readback,
depth=depth,
frame_runnable=bool(fd and fd.runnable),
operator_bindings=len(fd.bindings) if fd and fd.bindings else 0,
explanation_has_root_note=has_root_note,
)
return AblationResult(
condition="metadata_only",
case_id=case.case_id,
outcome=_score(
answer=float(resolution.answer),
expected=case.expected_answer,
expected_outcome=case.expected_outcome,
),
answer=float(resolution.answer),
answer_unit=resolution.answer_unit or None,
expected_answer=case.expected_answer,
refusal_reason="",
readback=readback,
depth=depth,
frame_runnable=bool(fd and fd.runnable),
operator_bindings=len(fd.bindings) if fd and fd.bindings else 0,
explanation_has_root_note=has_root_note,
)
def run_invalid(case: AblationCase) -> AblationResult:
"""Force the invalid/adversarial condition: commit only on clean organ admit."""
# Same as operator scoring: confusers must refuse, not guess.
result = run_operator(case)
return AblationResult(
condition="invalid",
case_id=result.case_id,
outcome=result.outcome,
answer=result.answer,
answer_unit=result.answer_unit,
expected_answer=result.expected_answer,
refusal_reason=result.refusal_reason or (
"committed" if result.answer is not None else "refused"
),
readback=result.readback,
depth=DepthContribution(
mapping_rule_id="invalid.control",
source_pack_ids=(),
source_record_ids=(),
evidence_spans=(),
validity="invalid",
reason="Adversarial/invalid control case; refuse preferred over wrong.",
),
frame_runnable=result.frame_runnable,
operator_bindings=result.operator_bindings,
)
_CONDITION_RUNNERS = {
"baseline": run_baseline,
"operator": run_operator,
"depth": run_depth,
"metadata_only": run_metadata_only,
"invalid": run_invalid,
}
def run_condition(condition: ConditionName, case: AblationCase) -> AblationResult:
return _CONDITION_RUNNERS[condition](case)
def run_all_conditions(case: AblationCase) -> dict[ConditionName, AblationResult]:
# For gold-solve cases, skip invalid runner (same as operator); for refuse gold, use invalid.
results: dict[ConditionName, AblationResult] = {}
for name in ("baseline", "operator", "depth", "metadata_only"):
results[name] = run_condition(name, case) # type: ignore[arg-type]
if case.expected_outcome == "refused" or "adversarial" in case.tags or "invalid" in case.tags:
results["invalid"] = run_invalid(case)
return results
def answers_match(a: AblationResult, b: AblationResult, *, tol: float = 1e-6) -> bool:
"""True when both refuse, or both commit equal answers."""
if a.answer is None and b.answer is None:
return True
if a.answer is None or b.answer is None:
return False
return abs(float(a.answer) - float(b.answer)) < tol

318
tests/test_relational_operator_ablation.py Normal file
View file

@ -0,0 +1,318 @@
"""Unit, integration, and adversarial proofs for relational operator ablation."""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from generate.relational_operator_ablation import (
AblationCase,
DEPTH_EXECUTABLE_MAPPING_RULE,
DEPTH_METADATA_RULE,
answers_match,
run_all_conditions,
run_baseline,
run_condition,
run_depth,
run_metadata_only,
run_operator,
)
from algebra.versor import versor_condition
_CASES = (
Path(__file__).resolve().parents[1]
/ "evals"
/ "relational_operator_ablation"
/ "v1"
/ "cases.jsonl"
)
GOLD_0001 = AblationCase(
case_id="unit-gold-0001",
problem=(
"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. "
"If the current temperature of the mountain is 84 degrees, what will the temperature "
"decrease by?"
),
expected_answer=21.0,
expected_unit="degrees",
expected_outcome="correct",
tags=("fraction_decrease", "gold"),
)
FINAL_VALUE = AblationCase(
case_id="unit-final-value",
problem=(
"In one hour, the lake's temperature will decrease to 3/4 of its temperature. "
"If the current temperature of the lake is 80 degrees, what will the temperature be?"
),
expected_answer=None,
expected_unit="",
expected_outcome="refused",
tags=("adversarial", "invalid"),
)
MULTI_FRACTION = AblationCase(
case_id="unit-multi-fraction",
problem=(
"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature "
"and then to 1/2 of its temperature. If the current temperature of the mountain is 84 "
"degrees, what will the temperature decrease by?"
),
expected_answer=None,
expected_unit="",
expected_outcome="refused",
tags=("adversarial", "invalid"),
)
SCALE_OOR_PREEXISTING = AblationCase(
case_id="unit-scale-oor-preexisting",
problem=(
"In one hour, the river will decrease to 5/4 of its level. "
"If the current level is 40 feet, what will the level decrease by?"
),
expected_answer=None,
expected_unit="",
expected_outcome="refused",
tags=("preexisting_gap",),
)
def _load_sealed() -> list[AblationCase]:
cases: list[AblationCase] = []
for line in _CASES.read_text(encoding="utf-8").splitlines():
if not line.strip():
continue
raw = json.loads(line)
cases.append(
AblationCase(
case_id=raw["case_id"],
problem=raw["problem"],
expected_answer=raw.get("expected_answer"),
expected_unit=raw.get("expected_unit") or "",
expected_outcome=raw["expected_outcome"],
tags=tuple(raw.get("tags") or ()),
notes=raw.get("notes") or "",
)
)
return cases
# ---------------------------------------------------------------------------
# Unit
# ---------------------------------------------------------------------------
def test_baseline_pure_rational_no_versor_binding() -> None:
result = run_baseline(GOLD_0001)
assert result.outcome == "correct"
assert result.answer == pytest.approx(21.0)
assert result.operator_bindings == 0
assert result.readback is not None
assert result.readback.has_versor_binding is True or result.frame_runnable
# Baseline path itself does not *require* bindings to compute; frame may still
# produce them in assess_fraction_decrease for readback.
assert result.condition == "baseline"
def test_operator_commits_geometric_binding_with_versor_integrity() -> None:
result = run_operator(GOLD_0001)
assert result.outcome == "correct"
assert result.answer == pytest.approx(21.0, abs=1e-6)
assert result.operator_bindings >= 1
assert result.readback is not None
assert result.readback.scale_surface == "3/4"
assert result.readback.has_versor_binding is True
assert result.readback.versor_error is not None
assert result.readback.versor_error < 1e-6
def test_metadata_only_decorates_but_does_not_change_answer() -> None:
op = run_operator(GOLD_0001)
meta = run_metadata_only(GOLD_0001)
assert answers_match(op, meta)
assert meta.explanation_has_root_note is True
assert meta.depth is not None
assert meta.depth.validity == "metadata_only"
assert meta.depth.mapping_rule_id == DEPTH_METADATA_RULE
def test_depth_condition_records_inactive_mapping_and_matches_operator() -> None:
op = run_operator(GOLD_0001)
depth = run_depth(GOLD_0001)
assert answers_match(op, depth)
assert depth.depth is not None
assert depth.depth.validity == "inactive_no_mapping"
assert depth.depth.mapping_rule_id == DEPTH_EXECUTABLE_MAPPING_RULE
assert depth.depth.source_pack_ids == ()
def test_determinism_two_runs_identical() -> None:
a = run_operator(GOLD_0001)
b = run_operator(GOLD_0001)
assert answers_match(a, b)
assert a.refusal_reason == b.refusal_reason
assert a.operator_bindings == b.operator_bindings
def test_baseline_and_operator_agree_when_both_commit() -> None:
base = run_baseline(GOLD_0001)
op = run_operator(GOLD_0001)
assert base.answer is not None and op.answer is not None
assert answers_match(base, op)
# ---------------------------------------------------------------------------
# Integration — sealed fixture end-to-end
# ---------------------------------------------------------------------------
def test_sealed_fixture_has_expected_shape() -> None:
cases = _load_sealed()
assert len(cases) == 8
gold = [c for c in cases if c.expected_outcome == "correct"]
refuse = [c for c in cases if c.expected_outcome == "refused"]
assert len(gold) == 2
assert len(refuse) == 6
def test_sealed_gold_cases_operator_and_baseline_correct() -> None:
for case in _load_sealed():
if case.expected_outcome != "correct":
continue
base = run_baseline(case)
op = run_operator(case)
assert base.outcome == "correct", case.case_id
assert op.outcome == "correct", case.case_id
assert answers_match(base, op)
def test_sealed_adversarial_cases_refuse_not_wrong() -> None:
for case in _load_sealed():
if case.expected_outcome != "refused":
continue
op = run_operator(case)
base = run_baseline(case)
assert op.outcome == "refused", (case.case_id, op.outcome, op.answer)
assert base.outcome == "refused", (case.case_id, base.outcome, base.answer)
assert op.outcome != "wrong"
assert base.outcome != "wrong"
def test_runner_report_invariants() -> None:
from evals.relational_operator_ablation.v1.runner import build_report, _load_cases
report = build_report(_load_cases())
assert report["sample_size"] == 8
assert report["findings"]["metadata_only_inert"] is True
assert report["findings"]["depth_executable_inactive_equals_operator"] is True
assert report["findings"]["operator_deterministic"] is True
assert report["findings"]["operator_wrong_zero"] is True
assert report["findings"]["baseline_wrong_zero"] is True
assert report["counts"]["operator"]["correct"] >= 2
assert report["counts"]["operator"]["wrong"] == 0
assert report["report_sha256"]
# Deterministic digest across two builds.
report2 = build_report(_load_cases())
assert report["report_sha256"] == report2["report_sha256"]
# ---------------------------------------------------------------------------
# Adversarial
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
"case",
[FINAL_VALUE, MULTI_FRACTION],
ids=["final_value", "multi_fraction"],
)
def test_adversarial_refuse_closed(case: AblationCase) -> None:
results = run_all_conditions(case)
for name, result in results.items():
assert result.outcome == "refused", (name, result.outcome, result.answer)
assert result.answer is None
def test_preexisting_scale_out_of_range_gap_isolated() -> None:
"""Document pre-existing Gate A2k gap: scale>1 may still admit geometrically.
Frame assess_fraction_decrease marks scale_out_of_range (baseline refuses).
Geometric A2k path may still compute a negative delta — not introduced by
this ablation. Do not mask; pin the split so a future organ fix flips this.
"""
base = run_baseline(SCALE_OOR_PREEXISTING)
op = run_operator(SCALE_OOR_PREEXISTING)
assert base.outcome == "refused"
assert base.answer is None
# Pre-existing: operator may wrong-commit. If a future PR fixes A2k, this
# assertion should be updated to require refuse.
assert op.outcome in {"refused", "wrong"}
if op.outcome == "wrong":
assert op.answer is not None
def test_malformed_depth_labels_cannot_elevate_runnable() -> None:
"""Garbage depth must not create a runnable assessment that wasn't runnable."""
# Use a refuse-gold problem.
broken_depth = {
"p0": {"language": "he", "root": "NOT_A_REAL_ROOT_!!!@@@"},
"p1": {"language": "grc", "root": ""},
}
refuse_case = FINAL_VALUE
clean = run_operator(refuse_case)
polluted = run_metadata_only(refuse_case, depth_labels=broken_depth)
assert clean.outcome == "refused"
assert polluted.outcome == "refused"
assert polluted.answer is None
# Must not flip to correct via depth.
assert polluted.outcome != "correct"
def test_valid_depth_metadata_on_gold_does_not_change_answer() -> None:
depths = {
"p0": {"language": "he", "root": "א-מ-נ"},
"p1": {"language": "grc", "root": "λογ"},
}
op = run_operator(GOLD_0001)
meta = run_metadata_only(GOLD_0001, depth_labels=depths)
assert answers_match(op, meta)
assert meta.explanation_has_root_note is True
def test_distractor_quantity_case_refuses() -> None:
cases = {c.case_id: c for c in _load_sealed()}
case = cases["roa-v1-0006"]
assert run_operator(case).outcome == "refused"
assert run_baseline(case).outcome == "refused"
def test_percent_partition_out_of_family_refuses_on_fraction_organ() -> None:
cases = {c.case_id: c for c in _load_sealed()}
case = cases["roa-v1-0005"]
assert run_operator(case).outcome == "refused"
def test_affine_confuser_refuses() -> None:
cases = {c.case_id: c for c in _load_sealed()}
case = cases["roa-v1-0004"]
assert run_operator(case).outcome == "refused"
def test_no_mutation_of_input_case() -> None:
problem = GOLD_0001.problem
run_operator(GOLD_0001)
run_baseline(GOLD_0001)
run_metadata_only(GOLD_0001)
assert GOLD_0001.problem == problem
def test_readback_preserves_role_provenance_on_gold() -> None:
result = run_operator(GOLD_0001)
assert result.readback is not None
role_names = {r for r, _ in result.readback.roles}
assert "base_quantity" in role_names
assert "scale" in role_names
assert result.readback.scale_surface == "3/4"