Merge pull request 'feat(eval): deterministic relational operator ablation + HE/GRC Logos roadmap' (#87) from feat/deterministic-relational-operator-ablation into main
Reviewed-on: #87 --- Delivered ┌─────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────┐ │ Item │ Location │ ├─────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ Cartography ledger │ docs/analysis/relational-operator-ablation-cartography-2026-07-19.md │ ├─────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ Ablation dossier │ docs/analysis/relational-operator-ablation-dossier-2026-07-19.md │ ├─────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ HE/GRC Logos roadmap │ docs/analysis/hebrew-koine-greek-logos-pack-capability-roadmap-2026-07-19.md │ ├─────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ Ablation code + sealed eval │ generate/relational_operator_ablation.py, evals/relational_operator_ablation/v1/ │ ├─────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤ │ Tests │ tests/test_relational_operator_ablation.py │ └─────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────┘ Git Branch: feat/deterministic-relational-operator-ablation Remote: Forgejo core-labs/core (5 commits) 1.df029a84— cartography ledger 2.1d4d51bc— ablation runners 3.87b01868— sealed eval lane v1 4.84f7a6f7— tests + ablation dossier 5.08881ab6— HE/GRC Logos roadmap + cross-links Pull request #87 • Title: feat(eval): deterministic relational operator ablation + HE/GRC Logos roadmap • Mergeable: yes • [Verification]: smoke 176 + ablation 20 + runner n=8 wrong=0 documented on the PR Fresh start for next PR Roadmap recommends: ┌──────────┬──────────────────────────────────────────────────────────────────────────────┐ │ Field │ Value │ ├──────────┼──────────────────────────────────────────────────────────────────────────────┤ │ Branch │ feat/observed-he-morph-constraint-v0 │ ├──────────┼──────────────────────────────────────────────────────────────────────────────┤ │ Worktree │ ../core-observed-he-morph-constraint │ ├──────────┼──────────────────────────────────────────────────────────────────────────────┤ │ Scope │ Observed-Hebrew morph constraint only (not pack bulk-load, not English math) │ └──────────┴──────────────────────────────────────────────────────────────────────────────┘
This commit is contained in:
commit
5fae5a67f2
10 changed files with 3414 additions and 0 deletions
|
|
@ -0,0 +1,220 @@
|
||||||
|
# Hebrew and Koine Greek Logos Pack Capability Roadmap
|
||||||
|
|
||||||
|
**Date:** 2026-07-19
|
||||||
|
**Type:** No-implementation audit / planning dossier
|
||||||
|
**Authority tree:** `feat/deterministic-relational-operator-ablation`
|
||||||
|
**Companion docs:**
|
||||||
|
|
||||||
|
- `docs/analysis/relational-operator-ablation-cartography-2026-07-19.md` — Phase 1 cartography + go/no-go
|
||||||
|
- `docs/analysis/relational-operator-ablation-dossier-2026-07-19.md` — English `fraction_decrease` ablation (IMPLEMENTED AND MEASURED)
|
||||||
|
- This file — separate HE/GRC Logos capability audit (planning only)
|
||||||
|
|
||||||
|
**Boundary:** The relational-operator ablation remains closed. English-only `fraction_decrease` depth was intentionally inert (no observed-morphology basis). That result must not be generalized to “Hebrew/Greek never help,” nor used to authorize English→pseudo-depth. This roadmap does **not** authorize a giant pack build.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Executive verdict
|
||||||
|
|
||||||
|
**READY ONLY FOR PREREQUISITE WIRING OR MIGRATION**
|
||||||
|
|
||||||
|
| Label | Why |
|
||||||
|
|-------|-----|
|
||||||
|
| Not ready for broad Logos implementation | Dual pack systems; sparse morph tags; holonomy crown proof not robust; no executable morph→canonical-relation mapping with provenance |
|
||||||
|
| Not architecturally blocked | Canonical language-independent seams exist (`BoundRelation`, teaching operators, pack checksum/OOV, `resolve_entry` / `node_depths`) |
|
||||||
|
| Broader “Logos improves all domains” thesis | Scientifically under-specified until a narrow **observed-HE/GRC** constraint ablation exists |
|
||||||
|
|
||||||
|
### Ablation boundary (preserve)
|
||||||
|
|
||||||
|
English-only `fraction_decrease` sealed n=8: depth **intentionally inert**. Do not reopen ablation fixtures/results except for demonstrated defects. Do not claim HE/GRC help arithmetic without observed morph input.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Hebrew capability matrix
|
||||||
|
|
||||||
|
Status: **LIVE** | **EVALUABLE** | **DIAGNOSTIC** | **TEST-ONLY** | **DORMANT** | **SCAFFOLD** | **SPECIFIED** | **ABSENT** | **CONTRADICTED**
|
||||||
|
|
||||||
|
| Capability | Exact evidence | Compiler | Runtime | Proof/tests | Gap | Overclaim risk | Priority |
|
||||||
|
|------------|----------------|----------|---------|-------------|-----|----------------|----------|
|
||||||
|
| Structured morphology schema | `packs/schema.py` `MorphologyEntry`; `packs/data/he_*` | LIVE `_apply_morphology` | Manifold via `load_pack` | `test_morphology_registry`, holonomy morph | No paradigm generator | Medium if “full morph” | P0 (tags live) |
|
||||||
|
| Legacy triliteral / romanized root | `packs/compiler.py` `_is_hebrew_root`, `_triliteral_root` | LIVE feature | Manifold | holonomy same-root | Dual encodings | Low | P1 document/migrate |
|
||||||
|
| Root identity on resolve | `chat/pack_resolver.py` `resolve_entry`, `resolve_token_depths` | Lookup | LIVE PropGraph / recognition | `test_3lang_depth_capability`, `test_oov_pipeline` | Exact surface match only | Medium | P0 observed-HE |
|
||||||
|
| Root-sense disambiguation | Multi-surface rare; no sense graph | ABSENT | First-match | — | Homographs silent | **Critical** | **P0 prerequisite** |
|
||||||
|
| Binyan as licensed semantics | Sparse `binyan` tags (qal/piel) | LIVE as infl rotor | Manifold only | sparse data | No root-specific licensing | **Critical** if universalized | P2 after sense model |
|
||||||
|
| Verbal aspect | Sparse `aspect: perfect` | Tags only | Not clause-level | source DRAFT notes | Not temporal operators | High | P2–P3 |
|
||||||
|
| Voice / reflexive / passive | Essentially ABSENT in compiled HE | — | — | — | Niphal/Hithpael not systematic | High | P3 |
|
||||||
|
| Construct state | 1× construct tag; source `he:construct-chain` frame | Tag only | Frames DISCONNECTED | — | Not executable possession | High if genitive-universalized | P1 candidate |
|
||||||
|
| Nominal gender/number | LIVE tags | LIVE rotors | Tags | registry | No agreement checker | Medium | P2 |
|
||||||
|
| Pronominal suffixes | Sparse `suffix_chain` | LIVE | Ingest OOV decomp | ingest gate | Not full paradigm | Medium | P3 |
|
||||||
|
| Lexical valency / frames | `packs/he/frames.jsonl` (4 frames) | **DISCONNECTED** from `packs/data` | Not in `load_pack` | CLI validate only | No LIVE frame→BoundRelation | High | **P1 migration** |
|
||||||
|
| Semantic domains | lexicon `semantic_domains` | LIVE | Manifold | domain contracts | Not morph-derived | Low | Keep |
|
||||||
|
| Alignment edges | `alignment.jsonl` | Load nudge | Geometry | alignment tests | Data ≠ reasoning | Medium | Honest limits |
|
||||||
|
| Readback HE | `_assemble_he`, ADR-0030 hedges; no `readback_rules.py` | — | Limited assembly | identity depth tests | Fluency C01 only | High if “full readback” | See fluency honesty |
|
||||||
|
| Correction / contradiction | Teaching chains ADR-0102 | N/A | Teaching/eval | domain tests | Not morph-triggered | Medium | Tie morph to existing ops |
|
||||||
|
| OOV fail-closed | `gate_engaged=true`, `fail_closed` | Enforced | LIVE | pack runtime tests | OK | Low | Preserve |
|
||||||
|
| Fluency eval | `evals/hebrew_fluency` | — | Articulation | contract.md C01 | Not morph-constraint eval | High if misused | Separate lane |
|
||||||
|
| Executable morph→canonical relation | — | ABSENT | ABSENT | ABSENT | Core Logos gap | Critical | Next vertical slice |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Greek capability matrix
|
||||||
|
|
||||||
|
| Capability | Exact evidence | Compiler | Runtime | Proof/tests | Gap | Overclaim risk | Priority |
|
||||||
|
|------------|----------------|----------|---------|-------------|-----|----------------|----------|
|
||||||
|
| Lemma inventory | `packs/data/grc_*` (11 / 38) | LIVE | `resolve_entry` | registry, 3lang | Micro seed | Medium | P0 honesty |
|
||||||
|
| Sense disambiguation | No compiled `senses.jsonl` on data packs | ABSENT | First-match | — | Colwell needs sense+syntax | **Critical** | **P0 prerequisite** |
|
||||||
|
| Case tags | Nearly all **nominative** only | LIVE rotors | Tags only | registry order | No productive case paradigm | **Critical** if case→role | P1 observed forms only |
|
||||||
|
| Number/gender/agreement | Tags present | LIVE | No agreement engine | — | No S-V check | High | P2 |
|
||||||
|
| Prep + case | Source `el:relational-pros` | DISCONNECTED | ABSENT | — | Not enforced | High | After case diversity |
|
||||||
|
| Valency / frames | `packs/grc\|el/frames.jsonl` | DISCONNECTED | ABSENT | — | Dual-system gap | High | **P1 migration** |
|
||||||
|
| Aspect / tense-form | Present indicative dominant | Tags | Not operators | — | Aorist/perfect system ABSENT | High | P2–P3 licensed tables |
|
||||||
|
| Voice / mood | Sparse active/middle; indicative | Tags | — | — | No full mood system | High | P3 |
|
||||||
|
| Participles / infinitives | ABSENT as productive class | — | — | — | Out of seed scope | Medium | Defer |
|
||||||
|
| Semantic domains | `logos.core`, etc. | LIVE | Manifold | ADR-0102 | OK | Low | Keep |
|
||||||
|
| Alignment | grc↔he↔en edges | Load nudge | Geometry | alignment tests | Not robust holonomy | Medium | Honest limits |
|
||||||
|
| Colwell / anarthrous | Source frame + notes | DISCONNECTED | ABSENT | — | Not runtime | High if claimed LIVE | P2 fail-closed |
|
||||||
|
| Readback GRC | `_assemble_grc` | — | Limited | depth tests | Fluency-class | Medium | `evals/koine_greek_fluency` |
|
||||||
|
| Geometric signature for GRC | `resolve_geometric_signature` needs `senses.jsonl` | ABSENT for depth packs | EN math when present | fraction EN | HE/GRC cannot supply scale signatures | Medium | Domain concern |
|
||||||
|
| Executable morph→canonical relation | ABSENT | ABSENT | ABSENT | ABSENT | Core gap | Critical | Observed-GRC later |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Shared Logos substrate matrix
|
||||||
|
|
||||||
|
| Capability | Evidence | Compiler | Runtime | Proof/tests | Gap | Overclaim risk | Priority |
|
||||||
|
|------------|----------|----------|---------|-------------|-----|----------------|----------|
|
||||||
|
| Roles DEPTH_ROOT / DEPTH_RELATION | `LanguageRole`; manifests | Schema LIVE | Load | ADR-0015 | Source `el` vs runtime `grc` | Medium | Prefer `grc` |
|
||||||
|
| Checksum + OOV | `load_pack` | LIVE | ChatRuntime | pack tests | Source ADR-0005 gates never true | Low | Keep |
|
||||||
|
| Ordered morph composition | `_apply_morphology` | LIVE | Manifold | holonomy morph | Not semantic licensing | Medium | Keep |
|
||||||
|
| Alignment graph | `alignment/graph.py` | Load nudge | Geometry | alignment tests | Not clause reasoning | High if “reasoning” | Data + local geometry |
|
||||||
|
| Holonomy encode | `algebra/holonomy.py` | LIVE encode | Ingest path | **Honest-fail crown** | Crown **not robust** | **Critical** | Do not claim trilingual proof |
|
||||||
|
| PropGraph depth spine | `GraphNode` language/root/morphology_id | N/A | LIVE he/grc | 3lang tests | No BoundRelation emission | Medium | Bridge for constraints |
|
||||||
|
| Depth decorate vs decide | `enrich_assessments_with_depth` vs anti_unifier | — | Match LIVE; math decorate | 3lang, oov | EN math correctly inert | Low if labeled | Preserve |
|
||||||
|
| Canonical roles | `BoundRelation`, `RoleObligation` | N/A | Math LIVE (fraction) | problem_frame tests | Language-independent | Low | **Reuse only** |
|
||||||
|
| Teaching causal/contradiction | `hebrew_greek_textual_reasoning_chains_v1.jsonl` | N/A | Teaching/domain | ADR-0102 | Lexeme-level, not morph | Medium | Constraint consumer |
|
||||||
|
| GrammarAttractor | schema | Thin | DORMANT | — | Not hot path | Medium | Defer |
|
||||||
|
| `packs.evidence` | holonomy helpers | — | TEST-ONLY / dormant | holonomy tests | Not production | Medium | Don’t build runtime on it |
|
||||||
|
| Sensorium HE/GRC | `gate_engaged=False` default | Scaffold | Divergent from data True | — | Dual truth | Medium | Align or drop fiction |
|
||||||
|
| Domain contract | ADR-0091/0102 four packs | Manifest | Ledger reasoning-capable | ratification tests | Fluency ≠ morph-constraint | Medium | Keep honesty |
|
||||||
|
|
||||||
|
**Trilingual alignment:** curated alignment data + local manifold geometry; **clause holonomy crown proof is CONTRADICTED as robust** (`docs/analysis/holonomy-resonance-proof-not-robust-2026-06-14.md`, honest-fail tests).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Pack taxonomy (strict)
|
||||||
|
|
||||||
|
| Tier | Contents | Must not smuggle |
|
||||||
|
|------|----------|------------------|
|
||||||
|
| Universal canonical Logos primitives | `BoundRelation`, RoleObligations, epistemic status, VersorBinding/CGA, teaching operator families, shared `semantic_domains` | HE/GRC-only hidden reasoners |
|
||||||
|
| Hebrew surface + mapping rules | Observed surfaces, roots, tags, **authored** mapping rules + provenance | English→HE inference; universal binyan→semantics |
|
||||||
|
| Greek surface + mapping rules | Observed lemmas/forms, attested case tags, **authored** prep+case rules | Nominative=agent; English→case |
|
||||||
|
| English articulation | Operational base, realizer, fluency | Treating EN as depth proof |
|
||||||
|
| Domain packs | math/physics/etc. organs | Domain-private morph ontology |
|
||||||
|
| Anchor / lens / register | `packs/anchor_lens/he_*`, `grc_*` | Lens as morphology engine |
|
||||||
|
| Safety / identity | identity/safety/ethics; ADR-0030 hedges | Identity as role solver |
|
||||||
|
| Evaluation / mastery | fluency, fabrication_control, inference_closure, ablations | Fluency pass as morph-constraint pass |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Phased vertical-slice roadmap
|
||||||
|
|
||||||
|
### Phase 0 — Prerequisites (no bulk pack growth)
|
||||||
|
|
||||||
|
1. Freeze dual-system map: LIVE = `packs/data/*`; source `packs/{he,grc,el}` = draft/disconnected.
|
||||||
|
2. Sense/ambiguity policy: first-match is fail-open for meaning → multi-candidate or refuse on collision.
|
||||||
|
3. Entrypoint already LIVE: HE `resolve_entry` / `define אמת` path (or GRC lemma path).
|
||||||
|
4. No lexicon expansion until Phase 1 ablation.
|
||||||
|
|
||||||
|
### Phase 1 — Smallest observed-morphology constraint (separate implementation PR)
|
||||||
|
|
||||||
|
| Condition | Behavior |
|
||||||
|
|-----------|----------|
|
||||||
|
| Canonical only | Teaching operator/chain without morph constraint |
|
||||||
|
| Executable morph constraint | Authored rule with provenance → constraint or refuse |
|
||||||
|
| Metadata-only | `node_depths` / root notes; **must not** change outcome |
|
||||||
|
| Invalid / ambiguous | Homograph without sense rule → **refuse** |
|
||||||
|
|
||||||
|
**Consumer:** existing teaching/contradiction/abstention seams — **not** GSM8K, not new ontology.
|
||||||
|
**Inputs:** observed HE/GRC only.
|
||||||
|
**Success:** wrong=0; metadata inert; outcome changes only with provenance.
|
||||||
|
|
||||||
|
### Phase 2 — One morph phenomenon
|
||||||
|
|
||||||
|
- HE: root identity + ambiguity refuse → construct-state candidate → binyan only with license tables
|
||||||
|
- GRC: lemma+attested case as evidence only → prep+case after non-nominative data → aspect later
|
||||||
|
|
||||||
|
### Phase 3 — Schema/compiler extensions (only if Phase 1 shows value)
|
||||||
|
|
||||||
|
Optional `mapping_rules.jsonl`; sense inventory; compile-time rule validation; constraint readback.
|
||||||
|
|
||||||
|
### Phase 4 — Broader domains
|
||||||
|
|
||||||
|
Only after Phase 1–2 pass; never by inventing morph for English word problems.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. No-go list
|
||||||
|
|
||||||
|
- Decorative packs not consumed by runtime/eval
|
||||||
|
- Unlicensed morph→semantics (nominative=agent, Hiphil=always cause, aorist=punctual, …)
|
||||||
|
- Parallel HE/GRC reasoner outside shared canonical operators
|
||||||
|
- English→Greek case / Hebrew binyan pseudo-evidence
|
||||||
|
- Coverage via guessing / relaxing refuse
|
||||||
|
- Claiming holonomy crown proof is LIVE
|
||||||
|
- Fluency C01 or ADR-0102 reasoning-capable as morph-constraint proof
|
||||||
|
- Broad lexicon growth before observed-morph ablation
|
||||||
|
- Filling all binyanim/case tables as “progress” without rules+eval
|
||||||
|
- Building production Logos on dormant `packs.evidence`
|
||||||
|
- Reopening English fraction_decrease ablation to “add depth value” without observed morph
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Minimum contract for future domain packs
|
||||||
|
|
||||||
|
1. Shared language-independent constraint catalog IDs
|
||||||
|
2. Mapping rules: rule_id, language, features, preconditions, counterexamples, validity, fail-closed
|
||||||
|
3. Provenance: pack_id, version, morphology_id/entry_id, spans
|
||||||
|
4. Deterministic compile → immutable ordered candidates
|
||||||
|
5. Readback to surface features + canonical constraint
|
||||||
|
6. Contradiction with COHERENT evidence → abstain, never override
|
||||||
|
7. Eval: observed-language inputs + metadata-only + sealed holdout
|
||||||
|
8. No pack-private geometric meaning space (ADR-0005)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Future ablation/proof design (not authorized here)
|
||||||
|
|
||||||
|
| Arm | HE | GRC | Cross-lang | Canonical |
|
||||||
|
|-----|----|-----|------------|-----------|
|
||||||
|
| Input | Observed HE | Observed GRC | Authored paired clauses | Language-agnostic operator |
|
||||||
|
| Executable | Morph rule → constraint | Morph rule → constraint | Data-checked alignment (not crown holonomy until metric fixed) | Existing operators |
|
||||||
|
| Metadata-only | Roots on, rule off | Case tags on, rule off | Alignment labels off | N/A |
|
||||||
|
| Fail | Ambiguous root/sense | Ambiguous lemma/case | Conflicting alignments | Insufficient evidence |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Recommended next implementation unit
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|-------|--------|
|
||||||
|
| **Branch** | `feat/observed-he-morph-constraint-v0` |
|
||||||
|
| **Worktree** | `../core-observed-he-morph-constraint` |
|
||||||
|
| **Why HE first** | LIVE root resolution + denser depth tests; GRC case inventory nominative-skewed |
|
||||||
|
| **PR-sized scope** | Ambiguity refuse; one authored mapping-rule type + provenance; consumer = teaching/contradiction/abstention; sealed observed-HE fixture; metadata-only control; no bulk lexicon; no English math; no holonomy crown “fix” |
|
||||||
|
| **Out of scope** | Pack expansion, GRC case tables, binyan universals, fluency C02–C13, dual ontology |
|
||||||
|
|
||||||
|
### Reviewer acceptance criteria (future PR)
|
||||||
|
|
||||||
|
- [ ] Observed Hebrew inputs (or explicit OOV refuse)
|
||||||
|
- [ ] Mapping rules with rule_id, preconditions, counterexamples, provenance
|
||||||
|
- [ ] Metadata-only outcome-identical to rule-off
|
||||||
|
- [ ] Ambiguous → refuse; wrong=0
|
||||||
|
- [ ] No English→HE morph invention
|
||||||
|
- [ ] No holonomy crown proof claim
|
||||||
|
- [ ] Does not modify relational-operator ablation fixtures/results
|
||||||
|
- [ ] Smoke + targeted tests green
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. One-line summary
|
||||||
|
|
||||||
|
Hebrew/Greek Logos packs are **LIVE as sparse compiled manifolds + depth recognition + domain teaching chains**, but **not** as executable, falsifiable morph→canonical-constraint systems. Holonomy crown proof is **not robust**. Next work is **prerequisite wiring + one observed-HE constraint ablation**, not a giant pack build.
|
||||||
|
|
@ -0,0 +1,152 @@
|
||||||
|
# Relational Operator Ablation — Cartography Ledger & Go/No-Go
|
||||||
|
|
||||||
|
**Date:** 2026-07-19
|
||||||
|
**Branch:** `feat/deterministic-relational-operator-ablation`
|
||||||
|
**Worktree:** `../core-relational-operator-ablation`
|
||||||
|
**Authority commit:** `d8d62b8e` (main tip at branch creation)
|
||||||
|
**Status:** Evidence ledger for Phase 1–2; implementation of the narrow GO slice is companion code under `generate/relational_operator_ablation.py` and `evals/relational_operator_ablation/`.
|
||||||
|
|
||||||
|
**Related:** HE/GRC broader capability is **not** authorized by this ledger. See `docs/analysis/hebrew-koine-greek-logos-pack-capability-roadmap-2026-07-19.md` (verdict: READY ONLY FOR PREREQUISITE WIRING OR MIGRATION).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Executive decision (closed)
|
||||||
|
|
||||||
|
**Verdict for this work product:** **IMPLEMENTED AND MEASURED** (narrow slice only).
|
||||||
|
|
||||||
|
| Scope | Decision |
|
||||||
|
|-------|----------|
|
||||||
|
| Full hypothesis: ancient morphology automatically improves GSM8K arithmetic | **NO-GO** (scientific precondition: no valid English→depth mapping; anti-circularity) |
|
||||||
|
| Narrow fraction_decrease relational-operator ablation | **GO** — family already LIVE with ProblemFrame + VersorBinding + A2k |
|
||||||
|
| Depth as *executable* math contribution on English word problems | **NO-GO for implementation of new mapping** — metadata-only control proves inert |
|
||||||
|
|
||||||
|
**Relation family chosen from live evidence:** `proportional_change.decrease_to_fraction` / organ `fraction_decrease`
|
||||||
|
(`generate/construction_affordances.py` `_SERVING_AUTHORIZED_FAMILIES`; `generate/derivation/fraction_decrease.py`; Gate A2k in `generate/math_candidate_graph.py`).
|
||||||
|
|
||||||
|
### Decision criteria checklist
|
||||||
|
|
||||||
|
| Criterion | Pass/Fail | Evidence |
|
||||||
|
|-----------|-----------|----------|
|
||||||
|
| Canonical relation seam exists | **Pass** | `BoundRelation(relation_type="decrease_to_fraction")`, roles via `problem_frame_bound_relations` |
|
||||||
|
| Deterministic reconstructable operator | **Pass** | `VersorBinding` + CGA dilation; same text → same answer |
|
||||||
|
| Correction/readback or native integrity | **Pass (native)** | Self-verify + span provenance + `versor_condition < 1e-6`; no fabricated dual-adjoint |
|
||||||
|
| Testable eval lane | **Pass** | Sealed `evals/relational_operator_ablation/v1` + existing train_sample organ tests |
|
||||||
|
| Baseline vs operator vs depth vs metadata-only | **Pass** | Ablation conditions in harness |
|
||||||
|
| Fail-closed | **Pass** | Missing/ambiguous/hazard → refuse |
|
||||||
|
| Narrow non-claim scope | **Pass** | No full GSM8K claim; no EN→HE/GRC translation |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Live-path maps
|
||||||
|
|
||||||
|
### A. Text/chat — LIVE (not math serving)
|
||||||
|
|
||||||
|
```text
|
||||||
|
text → CognitiveTurnPipeline.run (core/cognition/pipeline.py)
|
||||||
|
→ tokenize / resolve_token_depths (chat/pack_resolver.py)
|
||||||
|
→ recognize (recognition/anti_unifier.py; root-canonical for he/grc)
|
||||||
|
→ ChatRuntime.chat (chat/runtime.py)
|
||||||
|
→ ingest.gate → FieldState
|
||||||
|
→ PropositionGraph (generate/graph_planner.py)
|
||||||
|
→ realizer / gates / refusal
|
||||||
|
→ CognitiveTurnResult.node_depths
|
||||||
|
```
|
||||||
|
|
||||||
|
`contract_assessment=None` at `resolve_surface` — ProblemFrame **DISCONNECTED** from chat spine.
|
||||||
|
|
||||||
|
### B. Language packs — dual layer
|
||||||
|
|
||||||
|
| Layer | Path | Status |
|
||||||
|
|-------|------|--------|
|
||||||
|
| Source ADR-0005 | `packs/{en,he,grc,el}/` status=draft, gates false | **DRAFT / DISCONNECTED** from compiler |
|
||||||
|
| Compiled | `packs/data/*` → `packs.compiler.load_pack` → `_apply_morphology` | **LIVE** |
|
||||||
|
| Depth IDs | `DEPTH_PACK_IDS` = he_core_cognition_v1, he_logos_micro_v1, grc_logos_* | **LIVE** |
|
||||||
|
|
||||||
|
### C. Problem-frame / math
|
||||||
|
|
||||||
|
```text
|
||||||
|
text → build_problem_frame (generate/problem_frame_builder.py) [DIAGNOSTIC IR + A2k input]
|
||||||
|
→ proposals / BoundRelation / BoundQuestionTarget
|
||||||
|
→ assess_fraction_decrease / assess_geometric_proposals
|
||||||
|
→ VersorBinding dilation
|
||||||
|
→ resolve_promotable_fraction_decrease (generate/derivation/fraction_decrease.py)
|
||||||
|
→ answer [LIVE Gate A2k]
|
||||||
|
|
||||||
|
parse_and_solve (generate/math_candidate_graph.py) [LIVE GSM8K serving orchestrator]
|
||||||
|
```
|
||||||
|
|
||||||
|
| Family | serving_allowed | Status |
|
||||||
|
|--------|-----------------|--------|
|
||||||
|
| fraction_decrease | **True** | **LIVE** A2k |
|
||||||
|
| percent_partition | False catalog; live prose A2l | **LIVE but DISCONNECTED** from frame |
|
||||||
|
| quantity_entity | False | **DIAGNOSTIC** |
|
||||||
|
| unary_delta | False | **DIAGNOSTIC** (no derivation organ) |
|
||||||
|
|
||||||
|
### D. Evaluation
|
||||||
|
|
||||||
|
| Split | N | correct / wrong / refused | Scorer |
|
||||||
|
|-------|---|---------------------------|--------|
|
||||||
|
| Sealed GSM8K test (recorded) | 1319 | **0 / 0 / 1319** | operator-only holdout |
|
||||||
|
| train_sample report.json | 50 | **30 / 0 / 20** | `_score_one_candidate_graph` |
|
||||||
|
| holdout_dev report.json | 500 | **5 / 0 / 495** | same |
|
||||||
|
| CORE public synthetic | 150 | **150 / 0 / 0** | classic `_score_one` |
|
||||||
|
|
||||||
|
**Note:** Order is always **correct / wrong / refused**. Prior prose that wrote `30/20/0` swapped wrong/refused.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Design truth matrix
|
||||||
|
|
||||||
|
| Claim | Spec | Implementation | Runtime | Tests | Status |
|
||||||
|
|-------|------|----------------|---------|-------|--------|
|
||||||
|
| Hebrew root morphology | ADR-0015 | `MorphologyEntry.root`; `_apply_morphology` | LIVE manifold + resolver | `test_morphology_registry`, holonomy | **LIVE (sparse)** |
|
||||||
|
| Hebrew binyan | ADR-0005 | Inflection tag only | geometric tag | sparse data | **Partial** |
|
||||||
|
| Greek case/aspect/voice | ADR-0015 | Compiled inflection tags | LIVE tags | morphology_registry | **Partial** |
|
||||||
|
| Anchor lens | pack inventory | `packs/anchor_lens/` | LIVE overlay | inventory | **LIVE** |
|
||||||
|
| PropGraph / depth | 3lang plan | `depth_canonical`, pipeline | LIVE he/grc input | `test_3lang_depth_capability` | **LIVE spine** |
|
||||||
|
| Depth → math answer | — | `enrich_assessments_with_depth` explanation only | decoration | same | **Inert on math** |
|
||||||
|
| Canonical math roles | kernel substrate | BoundRelation roles | A2k + diagnostic | problem_frame tests | **LIVE fraction** |
|
||||||
|
| VersorBinding | CGA construction boundary | dilation payload | LIVE A2k | fraction + 3lang tests | **LIVE** |
|
||||||
|
| Readback / integrity | organ self-verify | spans + verified derivation | LIVE | fraction_decrease | **Partial (native)** |
|
||||||
|
| GSM8K eval | ADR-0119+ | train_sample/holdout/sealed | LIVE open | runners | **LIVE** |
|
||||||
|
| serving_allowed all False | prior claim | one True family | — | construction_affordances | **Contradicted** |
|
||||||
|
| FGL | prior claim | — | — | — | **ABSENT** |
|
||||||
|
| ADR-0248 as Logos | prior claim | integrity handoffs | observe-only | ring3 | **Miscited** |
|
||||||
|
| ADR-0196 ancient packs | prior claim | Python/Rust/Zig | — | ADR text | **False** |
|
||||||
|
| EN GSM8K → GRC case / HE binyan | prior claim | — | — | — | **ABSENT** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Claim corrections (prior synthesis)
|
||||||
|
|
||||||
|
1. **EN/HE/GRC roles** — Partially confirmed (ADRs + LIVE compiled packs; source packs draft).
|
||||||
|
2. **ADR-0005/0015** — Confirmed as architecture; implementation is `packs/data` compiler path.
|
||||||
|
3. **packs/he, packs/grc draft + compiler** — Confirmed split: draft source vs LIVE compiled.
|
||||||
|
4. **Ordered morph composition** — Confirmed LIVE+tested in `packs/compiler._apply_morphology`.
|
||||||
|
5. **depth_canonical / enrich** — Confirmed; enrich is **decoration**; recognize is **matching**.
|
||||||
|
6. **ProblemFrame symbols** — Confirmed under `generate/`.
|
||||||
|
7. **All serving_allowed=False** — **False**; fraction_decrease is True.
|
||||||
|
8. **No EN→case/binyan math path** — Confirmed absent.
|
||||||
|
9. **GSM8K all refused** — Sealed test **0/0/1319** recorded; train_sample **30/0/20**; holdout_dev **5/0/495**. claims_ledger C **7/43/0** is **stale**.
|
||||||
|
10. **FGL / ADR-0248 Logos** — FGL absent; ADR-0248 = integrity handoffs.
|
||||||
|
11. **ADR-0196** — Python/Rust/Zig, not ancient packs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Implementation plan (narrow GO)
|
||||||
|
|
||||||
|
1. Pure condition runners: baseline scalar from frame roles; operator geometric; metadata-only depth; invalid refuse.
|
||||||
|
2. No new depth→role mapping from English.
|
||||||
|
3. Sealed fixture v1 (fraction_decrease + confusers).
|
||||||
|
4. Measurement report + tests.
|
||||||
|
5. Smoke gate.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Remaining gaps (not in this slice)
|
||||||
|
|
||||||
|
- Engineering: wire ProblemFrame into CognitiveTurnPipeline (optional future).
|
||||||
|
- Engineering: unify percent_partition prose organ with frame contracts.
|
||||||
|
- Scientific: whether *observed* he/grc morphology can constrain math on bilingual inputs (unproven; not this PR).
|
||||||
|
- Rejected: inventing Greek cases for English GSM8K.
|
||||||
|
- Deferred: expanding draft `packs/he|grc` gates.
|
||||||
154
docs/analysis/relational-operator-ablation-dossier-2026-07-19.md
Normal file
154
docs/analysis/relational-operator-ablation-dossier-2026-07-19.md
Normal file
|
|
@ -0,0 +1,154 @@
|
||||||
|
# Deterministic Relational Operator Ablation — Final Engineering Dossier
|
||||||
|
|
||||||
|
**Branch:** `feat/deterministic-relational-operator-ablation`
|
||||||
|
**Worktree:** `/Users/kaizenpro/Projects/core-relational-operator-ablation`
|
||||||
|
**Date:** 2026-07-19
|
||||||
|
|
||||||
|
**Document set on this branch:**
|
||||||
|
|
||||||
|
| Doc | Role |
|
||||||
|
|-----|------|
|
||||||
|
| `docs/analysis/relational-operator-ablation-cartography-2026-07-19.md` | Phase 1 cartography, design-truth matrix, go/no-go |
|
||||||
|
| `docs/analysis/relational-operator-ablation-dossier-2026-07-19.md` | This file — ablation implementation + measurement |
|
||||||
|
| `docs/analysis/hebrew-koine-greek-logos-pack-capability-roadmap-2026-07-19.md` | Separate HE/GRC Logos audit (planning only; no pack build) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Executive verdict
|
||||||
|
|
||||||
|
**IMPLEMENTED AND MEASURED**
|
||||||
|
|
||||||
|
Narrow vertical slice only: family `proportional_change.decrease_to_fraction` / organ `fraction_decrease`.
|
||||||
|
Ancient-language → GSM8K arithmetic is **not** claimed and is **not** implemented (scientific NO-GO under anti-circularity).
|
||||||
|
|
||||||
|
**Depth on English-only cases was intentionally inert** (no observed-morphology basis). That does not test observed Hebrew/Greek input or the broader Logos-pack thesis — see the HE/GRC roadmap for the audited next path (`feat/observed-he-morph-constraint-v0`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Ground-truth corrections
|
||||||
|
|
||||||
|
| Prior claim | Classification | Evidence |
|
||||||
|
|-------------|----------------|----------|
|
||||||
|
| EN articulation / HE depth-root / GRC depth-relation | Partially confirmed | ADR-0005/0015; LIVE `packs/data/he_*`, `grc_*`; draft `packs/he\|grc` |
|
||||||
|
| ADR-0005/0015 architecture | Confirmed | Accepted ADRs; compiler path `packs.compiler._apply_morphology` |
|
||||||
|
| Source packs draft + morph compiler | Confirmed (split) | Source DISCONNECTED; compiled LIVE |
|
||||||
|
| Ordered morph composition | Confirmed LIVE | `MorphologyEntry`; tests holonomy/registry |
|
||||||
|
| depth_canonical / enrich | Confirmed; enrich=decoration | `recognition/depth_canonical.py`; math inert |
|
||||||
|
| ProblemFrame symbols | Confirmed | `generate/problem_frame*.py` |
|
||||||
|
| All `serving_allowed=False` | **False** | fraction_decrease **True** in `construction_affordances.py` |
|
||||||
|
| EN→Greek case/Hebrew binyan math path | **Absent** | No imports in math organs |
|
||||||
|
| GSM8K all refused | Split-dependent | Sealed **0/0/1319**; train_sample **30/0/20**; holdout_dev **5/0/495** (correct/wrong/refused) |
|
||||||
|
| FGL / ADR-0248 as Logos | **Absent / miscited** | ADR-0248 = integrity handoffs |
|
||||||
|
| ADR-0196 ancient packs | **False** | Python/Rust/Zig doctrine |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture map
|
||||||
|
|
||||||
|
See `docs/analysis/relational-operator-ablation-cartography-2026-07-19.md`.
|
||||||
|
|
||||||
|
Summary:
|
||||||
|
|
||||||
|
- **Chat path LIVE**; ProblemFrame **DISCONNECTED** (`contract_assessment=None`).
|
||||||
|
- **Math serving LIVE** via `parse_and_solve`; Gate A2k uses ProblemFrame + VersorBinding.
|
||||||
|
- **Depth LIVE** for he/grc recognition; **inert** for English math answers.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Implemented mechanism
|
||||||
|
|
||||||
|
| Layer | Mechanism |
|
||||||
|
|-------|-----------|
|
||||||
|
| Canonical relation | `BoundRelation(relation_type="decrease_to_fraction")` with roles base_quantity, scale, state_entity, transition, unit |
|
||||||
|
| Baseline operator | Pure rational `base * (1 - scale)` from grounded frame roles + shared organ hazard refuse gates |
|
||||||
|
| Geometric operator | Live `resolve_promotable_fraction_decrease` → CGA dilation `VersorBinding` |
|
||||||
|
| Depth-derived mapping | **None executable** on English; recorded `inactive_no_mapping` with rule id |
|
||||||
|
| Metadata-only | `assess_contracts(..., depth=)` appends `[root:…]`; must not change answer |
|
||||||
|
| Provenance | Spans, scale surface, roles, mapping_rule_id, pack ids (empty when inactive) |
|
||||||
|
| Readback | `RelationReadback` reconstructs roles, scale, missing bindings, versor_error |
|
||||||
|
| Fail-closed | Missing/hazard/adversarial → refuse; wrong=0 on sealed fixture |
|
||||||
|
|
||||||
|
Axioms: geometry-first (CGA dilation), field/structured IR (ProblemFrame), propagation via versor_apply, native integrity (not invented dual-adjoint), reconstruction via readback, compilation-last (no new ontology), reality-over-inheritance (depth inert proven).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
| Path | Purpose | Key symbols |
|
||||||
|
|------|---------|-------------|
|
||||||
|
| `docs/analysis/relational-operator-ablation-cartography-2026-07-19.md` | Evidence ledger + go/no-go | — |
|
||||||
|
| `docs/analysis/relational-operator-ablation-dossier-2026-07-19.md` | This dossier | — |
|
||||||
|
| `generate/relational_operator_ablation.py` | Condition runners | `run_baseline`, `run_operator`, `run_depth`, `run_metadata_only`, `run_invalid` |
|
||||||
|
| `evals/relational_operator_ablation/` | Sealed lane | `v1/cases.jsonl`, `v1/runner.py`, `v1/report.json` |
|
||||||
|
| `tests/test_relational_operator_ablation.py` | Unit/integration/adversarial proofs | 20 tests |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Proof and test matrix
|
||||||
|
|
||||||
|
| Command | Result |
|
||||||
|
|---------|--------|
|
||||||
|
| `PYTHONPATH=. python3 -m evals.relational_operator_ablation.v1.runner` | passed; report_sha256 `53df6d17…` |
|
||||||
|
| `PYTHONPATH=. python3 -m pytest -q tests/test_relational_operator_ablation.py` | **20 passed** |
|
||||||
|
| `uv run core test --suite smoke -q` | **176 passed** (~132s) |
|
||||||
|
|
||||||
|
Key tests:
|
||||||
|
|
||||||
|
- baseline rational / operator geometric / metadata inert / depth inactive
|
||||||
|
- sealed gold correct; adversarial refuse not wrong
|
||||||
|
- preexisting scale>1 A2k gap isolated (not masked)
|
||||||
|
- determinism, no input mutation, readback provenance
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Ablation results (n=8 sealed)
|
||||||
|
|
||||||
|
Order: **correct / wrong / refused**
|
||||||
|
|
||||||
|
| Condition | correct | wrong | refused | coverage | Notes |
|
||||||
|
|-----------|---------|-------|---------|----------|-------|
|
||||||
|
| baseline | 2 | 0 | 6 | 2/8 | Frame roles + rational math |
|
||||||
|
| operator | 2 | 0 | 6 | 2/8 | Live VersorBinding path |
|
||||||
|
| depth | 2 | 0 | 6 | 2/8 | = operator; mapping inactive |
|
||||||
|
| metadata_only | 2 | 0 | 6 | 2/8 | Root notes present; answers match operator |
|
||||||
|
| invalid (adversarial subset n=6) | 0 | 0 | 6 | 0/6 | Prefer refuse over wrong |
|
||||||
|
|
||||||
|
**Findings:** metadata_only inert ✓; depth inactive equals operator ✓; wrong=0 ✓; deterministic ✓.
|
||||||
|
|
||||||
|
**Limitations:** n=8; not GSM8K-wide; depth executable unproven for English; chat spine still disconnected from ProblemFrame; pre-existing A2k scale>1 gap documented separately.
|
||||||
|
|
||||||
|
**Scientific conclusion:** Canonical relation operators (baseline scalar vs geometric) agree when both commit; depth **metadata does not improve arithmetic** on this English slice; executable depth mapping was correctly refused as inactive rather than fabricated.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Remaining gaps
|
||||||
|
|
||||||
|
| Kind | Item |
|
||||||
|
|------|------|
|
||||||
|
| Engineering | Wire ProblemFrame into CognitiveTurnPipeline backpressure |
|
||||||
|
| Engineering | Unify percent_partition prose organ with frame contracts |
|
||||||
|
| Engineering | Fix A2k scale>1 admit (pre-existing; isolated test) |
|
||||||
|
| Scientific | Whether observed he/grc morphology can constrain bilingual math |
|
||||||
|
| Deferred | Expand draft source packs / holonomy crown proof |
|
||||||
|
| Rejected | Invent Greek case / Hebrew binyan from English GSM8K |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PR-ready summary
|
||||||
|
|
||||||
|
**Branch:** `feat/deterministic-relational-operator-ablation`
|
||||||
|
**Title:** `feat(eval): deterministic relational operator ablation (fraction_decrease)`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Evidence-led cartography of language packs, depth spine, and ProblemFrame/math paths. Implements a narrow sealed ablation for `fraction_decrease` comparing math-frame baseline, geometric operator, inactive depth contribution, metadata-only control, and adversarial refuse. Proves depth metadata is inert on English arithmetic; does not invent ancient-language role mapping.
|
||||||
|
|
||||||
|
`[Verification]: Smoke suite passed locally (176 passed, ~132s); ablation tests 20 passed; ablation runner passed (n=8, wrong=0).`
|
||||||
|
|
||||||
|
**Risks:** Imports private hazard helpers from fraction_decrease for shared refuse gates; scale>1 organ gap remains.
|
||||||
|
**Rollback:** Revert branch commits; no serving flag changes.
|
||||||
|
**Reviewer checklist:**
|
||||||
|
- [ ] Cartography matrix rows cite live paths
|
||||||
|
- [ ] No English→case/binyan mapping introduced
|
||||||
|
- [ ] Metadata-only inert invariant holds
|
||||||
|
- [ ] wrong=0 on sealed fixture
|
||||||
|
- [ ] Preexisting A2k gap not masked
|
||||||
1
evals/relational_operator_ablation/__init__.py
Normal file
1
evals/relational_operator_ablation/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
"""Deterministic Relational Operator Ablation eval lane (fraction_decrease)."""
|
||||||
1
evals/relational_operator_ablation/v1/__init__.py
Normal file
1
evals/relational_operator_ablation/v1/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
"""Sealed v1 fixture + runner for relational operator ablation."""
|
||||||
8
evals/relational_operator_ablation/v1/cases.jsonl
Normal file
8
evals/relational_operator_ablation/v1/cases.jsonl
Normal file
|
|
@ -0,0 +1,8 @@
|
||||||
|
{"case_id": "roa-v1-0001", "problem": "In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. If the current temperature of the mountain is 84 degrees, what will the temperature decrease by?", "expected_answer": 21.0, "expected_unit": "degrees", "expected_outcome": "correct", "tags": ["fraction_decrease", "gold", "train_sample_0005"], "notes": "Canonical decrease-to-fraction delta; gold=21."}
|
||||||
|
{"case_id": "roa-v1-0002", "problem": "In two hours, Cedar peak's temperature will decrease to 2/3 of its temperature. If the current temperature of the peak is 60 degrees, what will the temperature decrease by?", "expected_answer": 20.0, "expected_unit": "degrees", "expected_outcome": "correct", "tags": ["fraction_decrease", "gold", "sibling"], "notes": "Sibling shape: 60*(1-2/3)=20."}
|
||||||
|
{"case_id": "roa-v1-0003", "problem": "In one hour, the lake's temperature will decrease to 3/4 of its temperature. If the current temperature of the lake is 80 degrees, what will the temperature be?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "final_value", "invalid"], "notes": "Final-value question must refuse delta organ (not guess 60)."}
|
||||||
|
{"case_id": "roa-v1-0004", "problem": "Yun had 20 paperclips initially, but then lost 12. Marion has 1/4 more than what Yun currently has, plus 7. How many paperclips does Marion have?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "affine_confuser", "invalid"], "notes": "Percent/fraction affine confuser; fraction_decrease must refuse."}
|
||||||
|
{"case_id": "roa-v1-0005", "problem": "A school has 100 students. Half of the students are girls, the other half are boys. 20% of the girls have dogs at home and 10% of the boys have dogs at home. How many students own dogs?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "percent_partition", "invalid"], "notes": "Percent partition is out of family scope for this ablation; must refuse on fraction_decrease conditions."}
|
||||||
|
{"case_id": "roa-v1-0006", "problem": "In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. If the current temperature of the mountain is 84 degrees and the cabin is 40 degrees, what will the temperature decrease by?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "distractor_quantity", "invalid"], "notes": "Multiple nearby temperatures; organ must refuse rather than pick one."}
|
||||||
|
{"case_id": "roa-v1-0007", "problem": "The mountain will decrease by 3/4 of its temperature. Current temperature is 84 degrees. What will the temperature decrease by?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "percent_change_vs_of", "invalid"], "notes": "decrease by fraction language (not decrease to); refuse."}
|
||||||
|
{"case_id": "roa-v1-0008", "problem": "In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature and then to 1/2 of its temperature. If the current temperature of the mountain is 84 degrees, what will the temperature decrease by?", "expected_answer": null, "expected_unit": "", "expected_outcome": "refused", "tags": ["adversarial", "multi_fraction_hazard", "invalid"], "notes": "Multiple slash-fractions hazard; organ must refuse rather than pick one."}
|
||||||
1727
evals/relational_operator_ablation/v1/report.json
Normal file
1727
evals/relational_operator_ablation/v1/report.json
Normal file
File diff suppressed because it is too large
Load diff
251
evals/relational_operator_ablation/v1/runner.py
Normal file
251
evals/relational_operator_ablation/v1/runner.py
Normal file
|
|
@ -0,0 +1,251 @@
|
||||||
|
"""Sealed measurement runner for Deterministic Relational Operator Ablation v1.
|
||||||
|
|
||||||
|
Conditions: baseline | operator | depth | metadata_only | invalid (adversarial cases).
|
||||||
|
|
||||||
|
Run:
|
||||||
|
PYTHONPATH=. python -m evals.relational_operator_ablation.v1.runner
|
||||||
|
|
||||||
|
Writes ``report.json`` beside this module (override via CORE_ROA_REPORT_PATH).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from collections import Counter
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from generate.relational_operator_ablation import (
|
||||||
|
AblationCase,
|
||||||
|
ConditionName,
|
||||||
|
answers_match,
|
||||||
|
run_all_conditions,
|
||||||
|
run_condition,
|
||||||
|
)
|
||||||
|
|
||||||
|
_HERE = Path(__file__).resolve().parent
|
||||||
|
_CASES = _HERE / "cases.jsonl"
|
||||||
|
_REPORT = Path(
|
||||||
|
os.environ.get("CORE_ROA_REPORT_PATH", str(_HERE / "report.json"))
|
||||||
|
)
|
||||||
|
_SCHEMA_VERSION = 1
|
||||||
|
_LANE = "relational_operator_ablation/v1"
|
||||||
|
_CONDITIONS: tuple[ConditionName, ...] = (
|
||||||
|
"baseline",
|
||||||
|
"operator",
|
||||||
|
"depth",
|
||||||
|
"metadata_only",
|
||||||
|
"invalid",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _load_cases(path: Path = _CASES) -> list[AblationCase]:
|
||||||
|
cases: list[AblationCase] = []
|
||||||
|
for line in path.read_text(encoding="utf-8").splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
raw = json.loads(line)
|
||||||
|
cases.append(
|
||||||
|
AblationCase(
|
||||||
|
case_id=raw["case_id"],
|
||||||
|
problem=raw["problem"],
|
||||||
|
expected_answer=raw.get("expected_answer"),
|
||||||
|
expected_unit=raw.get("expected_unit") or "",
|
||||||
|
expected_outcome=raw["expected_outcome"],
|
||||||
|
tags=tuple(raw.get("tags") or ()),
|
||||||
|
notes=raw.get("notes") or "",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return cases
|
||||||
|
|
||||||
|
|
||||||
|
def _empty_counts() -> dict[str, int]:
|
||||||
|
return {"correct": 0, "wrong": 0, "refused": 0, "n": 0, "coverage": 0}
|
||||||
|
|
||||||
|
|
||||||
|
def build_report(cases: list[AblationCase]) -> dict[str, Any]:
|
||||||
|
per_condition: dict[str, dict[str, int]] = {
|
||||||
|
c: _empty_counts() for c in _CONDITIONS
|
||||||
|
}
|
||||||
|
per_case: list[dict[str, Any]] = []
|
||||||
|
inert_checks: list[dict[str, Any]] = []
|
||||||
|
determinism: list[dict[str, Any]] = []
|
||||||
|
|
||||||
|
for case in cases:
|
||||||
|
row: dict[str, Any] = {"case_id": case.case_id, "tags": list(case.tags)}
|
||||||
|
results = run_all_conditions(case)
|
||||||
|
|
||||||
|
# Always score primary four conditions.
|
||||||
|
for name in ("baseline", "operator", "depth", "metadata_only"):
|
||||||
|
result = results[name]
|
||||||
|
bucket = per_condition[name]
|
||||||
|
bucket["n"] += 1
|
||||||
|
bucket[result.outcome] += 1
|
||||||
|
if result.answer is not None:
|
||||||
|
bucket["coverage"] += 1
|
||||||
|
row[name] = result.as_json()
|
||||||
|
|
||||||
|
# Invalid condition only for adversarial/refuse-gold cases.
|
||||||
|
if "invalid" in results:
|
||||||
|
result = results["invalid"]
|
||||||
|
bucket = per_condition["invalid"]
|
||||||
|
bucket["n"] += 1
|
||||||
|
bucket[result.outcome] += 1
|
||||||
|
if result.answer is not None:
|
||||||
|
bucket["coverage"] += 1
|
||||||
|
row["invalid"] = result.as_json()
|
||||||
|
else:
|
||||||
|
row["invalid"] = {
|
||||||
|
"condition": "invalid",
|
||||||
|
"case_id": case.case_id,
|
||||||
|
"outcome": "skipped_not_adversarial",
|
||||||
|
"note": "invalid condition reserved for refuse-gold/adversarial tags",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Metadata-only must match operator answers (inert depth labels).
|
||||||
|
op = results["operator"]
|
||||||
|
meta = results["metadata_only"]
|
||||||
|
depth = results["depth"]
|
||||||
|
base = results["baseline"]
|
||||||
|
inert_checks.append(
|
||||||
|
{
|
||||||
|
"case_id": case.case_id,
|
||||||
|
"operator_eq_metadata": answers_match(op, meta),
|
||||||
|
"operator_eq_depth": answers_match(op, depth),
|
||||||
|
"baseline_eq_operator_when_both_commit": (
|
||||||
|
answers_match(base, op)
|
||||||
|
if base.answer is not None and op.answer is not None
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
"metadata_has_root_note_when_runnable": meta.explanation_has_root_note
|
||||||
|
if meta.frame_runnable
|
||||||
|
else None,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Determinism: re-run operator twice.
|
||||||
|
again = run_condition("operator", case)
|
||||||
|
determinism.append(
|
||||||
|
{
|
||||||
|
"case_id": case.case_id,
|
||||||
|
"operator_repeat_identical": answers_match(op, again)
|
||||||
|
and op.refusal_reason == again.refusal_reason,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
per_case.append(row)
|
||||||
|
|
||||||
|
# Coverage fraction
|
||||||
|
for name, bucket in per_condition.items():
|
||||||
|
n = bucket["n"] or 1
|
||||||
|
bucket["coverage_rate"] = bucket["coverage"] / n
|
||||||
|
bucket["correct_rate"] = bucket["correct"] / n
|
||||||
|
bucket["wrong_rate"] = bucket["wrong"] / n
|
||||||
|
bucket["refused_rate"] = bucket["refused"] / n
|
||||||
|
|
||||||
|
inert_ok = all(c["operator_eq_metadata"] for c in inert_checks)
|
||||||
|
depth_inactive_ok = all(c["operator_eq_depth"] for c in inert_checks)
|
||||||
|
det_ok = all(d["operator_repeat_identical"] for d in determinism)
|
||||||
|
no_wrong_on_operator = per_condition["operator"]["wrong"] == 0
|
||||||
|
no_wrong_on_baseline = per_condition["baseline"]["wrong"] == 0
|
||||||
|
|
||||||
|
findings = {
|
||||||
|
"metadata_only_inert": inert_ok,
|
||||||
|
"depth_executable_inactive_equals_operator": depth_inactive_ok,
|
||||||
|
"operator_deterministic": det_ok,
|
||||||
|
"operator_wrong_zero": no_wrong_on_operator,
|
||||||
|
"baseline_wrong_zero": no_wrong_on_baseline,
|
||||||
|
"operator_solves_gold_fraction_cases": per_condition["operator"]["correct"]
|
||||||
|
>= 2,
|
||||||
|
"scientific_note": (
|
||||||
|
"Depth executable mapping is intentionally inactive on English-only "
|
||||||
|
"inputs (anti-circularity). Metadata-only he-root labels must not "
|
||||||
|
"change answers. Operator geometric path and baseline scalar path "
|
||||||
|
"should agree when both commit."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
body = {
|
||||||
|
"schema_version": _SCHEMA_VERSION,
|
||||||
|
"lane": _LANE,
|
||||||
|
"family": "proportional_change.decrease_to_fraction",
|
||||||
|
"organ": "fraction_decrease",
|
||||||
|
"sample_size": len(cases),
|
||||||
|
"conditions": list(_CONDITIONS),
|
||||||
|
"counts": per_condition,
|
||||||
|
"findings": findings,
|
||||||
|
"inert_checks": inert_checks,
|
||||||
|
"determinism": determinism,
|
||||||
|
"per_case": per_case,
|
||||||
|
"limitations": [
|
||||||
|
"n=8 sealed cases; not a full GSM8K claim.",
|
||||||
|
"Depth-as-executable is unproven for English arithmetic; recorded inactive.",
|
||||||
|
"percent_partition dual-track is out of family scope for this slice.",
|
||||||
|
"Chat CognitiveTurnPipeline still passes contract_assessment=None.",
|
||||||
|
],
|
||||||
|
}
|
||||||
|
# Content-addressed digest of counts+findings+case outcomes (stable).
|
||||||
|
digest_src = json.dumps(
|
||||||
|
{
|
||||||
|
"counts": per_condition,
|
||||||
|
"findings": findings,
|
||||||
|
"outcomes": [
|
||||||
|
{
|
||||||
|
"id": row["case_id"],
|
||||||
|
"baseline": row["baseline"]["outcome"],
|
||||||
|
"operator": row["operator"]["outcome"],
|
||||||
|
"depth": row["depth"]["outcome"],
|
||||||
|
"metadata_only": row["metadata_only"]["outcome"],
|
||||||
|
}
|
||||||
|
for row in per_case
|
||||||
|
],
|
||||||
|
},
|
||||||
|
sort_keys=True,
|
||||||
|
separators=(",", ":"),
|
||||||
|
).encode("utf-8")
|
||||||
|
body["report_sha256"] = hashlib.sha256(digest_src).hexdigest()
|
||||||
|
return body
|
||||||
|
|
||||||
|
|
||||||
|
def write_report(report: dict[str, Any], path: Path = _REPORT) -> None:
|
||||||
|
path.write_text(
|
||||||
|
json.dumps(report, indent=2, sort_keys=True) + "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
cases = _load_cases()
|
||||||
|
report = build_report(cases)
|
||||||
|
write_report(report)
|
||||||
|
findings = report["findings"]
|
||||||
|
ok = (
|
||||||
|
findings["metadata_only_inert"]
|
||||||
|
and findings["depth_executable_inactive_equals_operator"]
|
||||||
|
and findings["operator_deterministic"]
|
||||||
|
and findings["operator_wrong_zero"]
|
||||||
|
and findings["baseline_wrong_zero"]
|
||||||
|
and findings["operator_solves_gold_fraction_cases"]
|
||||||
|
)
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"lane": _LANE,
|
||||||
|
"sample_size": report["sample_size"],
|
||||||
|
"counts": report["counts"],
|
||||||
|
"findings": findings,
|
||||||
|
"report_sha256": report["report_sha256"],
|
||||||
|
"passed": ok,
|
||||||
|
},
|
||||||
|
indent=2,
|
||||||
|
sort_keys=True,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return 0 if ok else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
582
generate/relational_operator_ablation.py
Normal file
582
generate/relational_operator_ablation.py
Normal file
|
|
@ -0,0 +1,582 @@
|
||||||
|
"""Deterministic Relational Operator Ablation — fraction_decrease vertical slice.
|
||||||
|
|
||||||
|
Scientific framing
|
||||||
|
------------------
|
||||||
|
Hypothesis under test (narrow): explicit canonical relation operators compiled
|
||||||
|
into the existing geometric (VersorBinding / CGA) layer are measurable against
|
||||||
|
a math-frame-only baseline, and depth *metadata* is inert unless an explicit
|
||||||
|
executable mapping is activated.
|
||||||
|
|
||||||
|
This module does **not**:
|
||||||
|
- translate English problems into Hebrew or Greek,
|
||||||
|
- invent case/binyan roles from English,
|
||||||
|
- relax refusal gates,
|
||||||
|
- or claim full-GSM8K competence.
|
||||||
|
|
||||||
|
Conditions
|
||||||
|
----------
|
||||||
|
- ``baseline``: ProblemFrame roles + pure rational arithmetic (no VersorBinding).
|
||||||
|
- ``operator``: live Gate-A2k geometric path (``resolve_promotable_fraction_decrease``).
|
||||||
|
- ``depth``: same as operator with an explicit depth-contribution record.
|
||||||
|
On English-only inputs there is **no** valid executable depth mapping
|
||||||
|
(anti-circularity); contribution is recorded as inactive.
|
||||||
|
- ``metadata_only``: depth labels passed through ``assess_contracts`` (decorate
|
||||||
|
explanations) then operator path — must not change the answer.
|
||||||
|
- ``invalid`` / adversarial: typed refusal, never a guessed answer.
|
||||||
|
|
||||||
|
Integrity model (project-native, not invented dual-adjoint jargon)
|
||||||
|
-------------------------------------------------------------------
|
||||||
|
- VersorBinding enforces ``versor_condition < 1e-6`` at construction.
|
||||||
|
- Organ self-verification requires grounded fraction token + base quantity.
|
||||||
|
- Readback reconstructs roles, scale surface, spans, and condition lineage.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import asdict, dataclass, field
|
||||||
|
from fractions import Fraction
|
||||||
|
from typing import Any, Literal, Mapping
|
||||||
|
|
||||||
|
from generate.derivation.fraction_decrease import (
|
||||||
|
_asks_decrease_delta,
|
||||||
|
_has_hazard_surface,
|
||||||
|
compose_fraction_decrease,
|
||||||
|
resolve_promotable_fraction_decrease,
|
||||||
|
)
|
||||||
|
from generate.derivation.target import _question_clause
|
||||||
|
from generate.problem_frame_builder import build_problem_frame
|
||||||
|
from generate.problem_frame_contracts import (
|
||||||
|
ContractAssessment,
|
||||||
|
assess_contracts,
|
||||||
|
assess_fraction_decrease,
|
||||||
|
assess_geometric_proposals,
|
||||||
|
)
|
||||||
|
|
||||||
|
ConditionName = Literal[
|
||||||
|
"baseline",
|
||||||
|
"operator",
|
||||||
|
"depth",
|
||||||
|
"metadata_only",
|
||||||
|
"invalid",
|
||||||
|
]
|
||||||
|
|
||||||
|
OutcomeName = Literal["correct", "wrong", "refused"]
|
||||||
|
|
||||||
|
# Stable mapping-rule id for the (currently empty) English executable depth path.
|
||||||
|
DEPTH_EXECUTABLE_MAPPING_RULE: str = "depth_executable.fraction_decrease.v0_absent"
|
||||||
|
DEPTH_METADATA_RULE: str = "depth_metadata.enrich_assessments_with_depth.v1"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class DepthContribution:
|
||||||
|
"""Provenance for any depth-derived constraint (executable or inactive)."""
|
||||||
|
|
||||||
|
mapping_rule_id: str
|
||||||
|
source_pack_ids: tuple[str, ...]
|
||||||
|
source_record_ids: tuple[str, ...]
|
||||||
|
evidence_spans: tuple[tuple[int, int, str], ...]
|
||||||
|
validity: Literal["inactive_no_mapping", "metadata_only", "executable", "invalid"]
|
||||||
|
reason: str
|
||||||
|
|
||||||
|
def as_json(self) -> dict[str, Any]:
|
||||||
|
return asdict(self)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class RelationReadback:
|
||||||
|
"""Human-auditable reconstruction of the grounded relation."""
|
||||||
|
|
||||||
|
relation_type: str
|
||||||
|
roles: tuple[tuple[str, str], ...] # (role, target_id_or_surface)
|
||||||
|
scale_surface: str | None
|
||||||
|
base_surface: str | None
|
||||||
|
missing_bindings: tuple[str, ...]
|
||||||
|
explanation: str
|
||||||
|
has_versor_binding: bool
|
||||||
|
versor_error: float | None
|
||||||
|
|
||||||
|
def as_json(self) -> dict[str, Any]:
|
||||||
|
return asdict(self)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class AblationResult:
|
||||||
|
condition: ConditionName
|
||||||
|
case_id: str
|
||||||
|
outcome: OutcomeName
|
||||||
|
answer: float | None
|
||||||
|
answer_unit: str | None
|
||||||
|
expected_answer: float | None
|
||||||
|
refusal_reason: str
|
||||||
|
readback: RelationReadback | None
|
||||||
|
depth: DepthContribution | None
|
||||||
|
frame_runnable: bool
|
||||||
|
operator_bindings: int
|
||||||
|
explanation_has_root_note: bool = False
|
||||||
|
|
||||||
|
def as_json(self) -> dict[str, Any]:
|
||||||
|
payload = {
|
||||||
|
"condition": self.condition,
|
||||||
|
"case_id": self.case_id,
|
||||||
|
"outcome": self.outcome,
|
||||||
|
"answer": self.answer,
|
||||||
|
"answer_unit": self.answer_unit,
|
||||||
|
"expected_answer": self.expected_answer,
|
||||||
|
"refusal_reason": self.refusal_reason,
|
||||||
|
"frame_runnable": self.frame_runnable,
|
||||||
|
"operator_bindings": self.operator_bindings,
|
||||||
|
"explanation_has_root_note": self.explanation_has_root_note,
|
||||||
|
"readback": None if self.readback is None else self.readback.as_json(),
|
||||||
|
"depth": None if self.depth is None else self.depth.as_json(),
|
||||||
|
}
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class AblationCase:
|
||||||
|
case_id: str
|
||||||
|
problem: str
|
||||||
|
expected_answer: float | None
|
||||||
|
expected_unit: str
|
||||||
|
expected_outcome: Literal["correct", "refused"]
|
||||||
|
tags: tuple[str, ...] = ()
|
||||||
|
depth_override: Mapping[str, Mapping[str, str]] | None = None
|
||||||
|
notes: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
def _score(
|
||||||
|
*,
|
||||||
|
answer: float | None,
|
||||||
|
expected: float | None,
|
||||||
|
expected_outcome: str,
|
||||||
|
) -> OutcomeName:
|
||||||
|
if answer is None:
|
||||||
|
return "refused"
|
||||||
|
if expected is None:
|
||||||
|
# Gold is abstention; any committed answer is wrong.
|
||||||
|
return "wrong"
|
||||||
|
if abs(float(answer) - float(expected)) < 1e-6:
|
||||||
|
return "correct"
|
||||||
|
return "wrong"
|
||||||
|
|
||||||
|
|
||||||
|
def _readback_from_assessment(
|
||||||
|
assessment: ContractAssessment,
|
||||||
|
frame_roles: tuple[tuple[str, str], ...],
|
||||||
|
) -> RelationReadback:
|
||||||
|
scale_surface = None
|
||||||
|
versor_error = None
|
||||||
|
has_binding = bool(assessment.bindings)
|
||||||
|
if assessment.bindings:
|
||||||
|
scale_surface = assessment.bindings[0].semantic_identity
|
||||||
|
versor_error = float(assessment.bindings[0].versor_error)
|
||||||
|
return RelationReadback(
|
||||||
|
relation_type="decrease_to_fraction",
|
||||||
|
roles=frame_roles,
|
||||||
|
scale_surface=scale_surface,
|
||||||
|
base_surface=next((t for r, t in frame_roles if r == "base_quantity"), None),
|
||||||
|
missing_bindings=tuple(assessment.missing_bindings),
|
||||||
|
explanation=assessment.explanation,
|
||||||
|
has_versor_binding=has_binding,
|
||||||
|
versor_error=versor_error,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _frame_roles(frame) -> tuple[tuple[str, str], ...]:
|
||||||
|
roles: list[tuple[str, str]] = []
|
||||||
|
for relation in frame.bound_relations:
|
||||||
|
if relation.relation_type != "decrease_to_fraction":
|
||||||
|
continue
|
||||||
|
for role in relation.roles:
|
||||||
|
roles.append((role.role, role.target_id))
|
||||||
|
return tuple(roles)
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_base_and_scale(frame) -> tuple[Fraction, Fraction, str, str] | None:
|
||||||
|
"""Return (base, scale, base_surface, scale_surface) from grounded frame roles."""
|
||||||
|
quantities = {q.fact_id: q for q in frame.quantities}
|
||||||
|
mentions = {m.mention_id: m for m in frame.mentions}
|
||||||
|
# Map mention_id → quantity value via fact_id when available.
|
||||||
|
mention_value: dict[str, Fraction] = {}
|
||||||
|
for mention in frame.mentions:
|
||||||
|
if mention.kind != "quantity":
|
||||||
|
continue
|
||||||
|
if mention.fact_id and mention.fact_id in quantities:
|
||||||
|
value = quantities[mention.fact_id].value
|
||||||
|
mention_value[mention.mention_id] = (
|
||||||
|
value if isinstance(value, Fraction) else Fraction(str(value))
|
||||||
|
)
|
||||||
|
# Prefer role targets from the unique decrease_to_fraction relation.
|
||||||
|
relations = [
|
||||||
|
r for r in frame.bound_relations if r.relation_type == "decrease_to_fraction"
|
||||||
|
]
|
||||||
|
if len(relations) != 1:
|
||||||
|
return None
|
||||||
|
relation = relations[0]
|
||||||
|
base_id = next((r.target_id for r in relation.roles if r.role == "base_quantity"), None)
|
||||||
|
scale_id = next((r.target_id for r in relation.roles if r.role == "scale"), None)
|
||||||
|
if base_id is None or scale_id is None:
|
||||||
|
return None
|
||||||
|
if base_id not in mention_value or scale_id not in mention_value:
|
||||||
|
# Fall back: match by surface token against scalars.
|
||||||
|
return None
|
||||||
|
base = mention_value[base_id]
|
||||||
|
scale = mention_value[scale_id]
|
||||||
|
base_surface = mentions[base_id].surface if base_id in mentions else str(base)
|
||||||
|
scale_surface = mentions[scale_id].surface if scale_id in mentions else str(scale)
|
||||||
|
if not (0 < float(scale) < 1):
|
||||||
|
return None
|
||||||
|
return base, scale, base_surface, scale_surface
|
||||||
|
|
||||||
|
|
||||||
|
def run_baseline(case: AblationCase) -> AblationResult:
|
||||||
|
"""Canonical math-frame path: rational arithmetic from grounded roles only.
|
||||||
|
|
||||||
|
Shares organ *refusal* discipline (question shape + hazard surfaces) with
|
||||||
|
the live fraction_decrease organ so the ablation compares scalar vs geometric
|
||||||
|
*execution*, not divergent gate policy. Does **not** apply VersorBinding/CGA.
|
||||||
|
"""
|
||||||
|
frame = build_problem_frame(case.problem)
|
||||||
|
assessment = assess_fraction_decrease(frame)
|
||||||
|
roles = _frame_roles(frame)
|
||||||
|
readback = _readback_from_assessment(assessment, roles)
|
||||||
|
question = _question_clause(case.problem)
|
||||||
|
|
||||||
|
if not assessment.runnable:
|
||||||
|
return AblationResult(
|
||||||
|
condition="baseline",
|
||||||
|
case_id=case.case_id,
|
||||||
|
outcome=_score(answer=None, expected=case.expected_answer, expected_outcome=case.expected_outcome),
|
||||||
|
answer=None,
|
||||||
|
answer_unit=None,
|
||||||
|
expected_answer=case.expected_answer,
|
||||||
|
refusal_reason=assessment.explanation or "frame_not_runnable",
|
||||||
|
readback=readback,
|
||||||
|
depth=None,
|
||||||
|
frame_runnable=False,
|
||||||
|
operator_bindings=0,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Shared refuse gates with generate.derivation.fraction_decrease (no geometry).
|
||||||
|
if not _asks_decrease_delta(question):
|
||||||
|
return AblationResult(
|
||||||
|
condition="baseline",
|
||||||
|
case_id=case.case_id,
|
||||||
|
outcome="refused",
|
||||||
|
answer=None,
|
||||||
|
answer_unit=None,
|
||||||
|
expected_answer=case.expected_answer,
|
||||||
|
refusal_reason="baseline_question_not_decrease_delta",
|
||||||
|
readback=readback,
|
||||||
|
depth=None,
|
||||||
|
frame_runnable=True,
|
||||||
|
operator_bindings=0,
|
||||||
|
)
|
||||||
|
if _has_hazard_surface(case.problem, question):
|
||||||
|
return AblationResult(
|
||||||
|
condition="baseline",
|
||||||
|
case_id=case.case_id,
|
||||||
|
outcome="refused",
|
||||||
|
answer=None,
|
||||||
|
answer_unit=None,
|
||||||
|
expected_answer=case.expected_answer,
|
||||||
|
refusal_reason="baseline_hazard_surface",
|
||||||
|
readback=readback,
|
||||||
|
depth=None,
|
||||||
|
frame_runnable=True,
|
||||||
|
operator_bindings=0,
|
||||||
|
)
|
||||||
|
|
||||||
|
extracted = _extract_base_and_scale(frame)
|
||||||
|
if extracted is None:
|
||||||
|
return AblationResult(
|
||||||
|
condition="baseline",
|
||||||
|
case_id=case.case_id,
|
||||||
|
outcome="refused",
|
||||||
|
answer=None,
|
||||||
|
answer_unit=None,
|
||||||
|
expected_answer=case.expected_answer,
|
||||||
|
refusal_reason="baseline_scale_or_base_unresolved",
|
||||||
|
readback=readback,
|
||||||
|
depth=None,
|
||||||
|
frame_runnable=True,
|
||||||
|
operator_bindings=0,
|
||||||
|
)
|
||||||
|
|
||||||
|
base, scale, _base_s, _scale_s = extracted
|
||||||
|
# decrease_by = base * (1 - scale) — pure rational, no geometric payload.
|
||||||
|
answer = float(base * (Fraction(1) - scale))
|
||||||
|
unit = ""
|
||||||
|
for mention in frame.mentions:
|
||||||
|
if mention.kind == "unit":
|
||||||
|
unit = mention.surface
|
||||||
|
break
|
||||||
|
|
||||||
|
return AblationResult(
|
||||||
|
condition="baseline",
|
||||||
|
case_id=case.case_id,
|
||||||
|
outcome=_score(
|
||||||
|
answer=answer,
|
||||||
|
expected=case.expected_answer,
|
||||||
|
expected_outcome=case.expected_outcome,
|
||||||
|
),
|
||||||
|
answer=answer,
|
||||||
|
answer_unit=unit or None,
|
||||||
|
expected_answer=case.expected_answer,
|
||||||
|
refusal_reason="",
|
||||||
|
readback=readback,
|
||||||
|
depth=None,
|
||||||
|
frame_runnable=True,
|
||||||
|
operator_bindings=0,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def run_operator(case: AblationCase) -> AblationResult:
|
||||||
|
"""Executable geometric operator path (live A2k organ)."""
|
||||||
|
frame = build_problem_frame(case.problem)
|
||||||
|
assessment = assess_fraction_decrease(frame)
|
||||||
|
geom = assess_geometric_proposals(frame)
|
||||||
|
geom_fd = next(
|
||||||
|
(g for g in geom if g.candidate_organ == "fraction_decrease"),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
roles = _frame_roles(frame)
|
||||||
|
# Prefer obligation-complete assessment for readback; fall back to geometric.
|
||||||
|
read_src = assessment if assessment.runnable else geom_fd
|
||||||
|
readback = (
|
||||||
|
_readback_from_assessment(read_src, roles)
|
||||||
|
if read_src is not None
|
||||||
|
else RelationReadback(
|
||||||
|
relation_type="decrease_to_fraction",
|
||||||
|
roles=roles,
|
||||||
|
scale_surface=None,
|
||||||
|
base_surface=None,
|
||||||
|
missing_bindings=(),
|
||||||
|
explanation="no_fraction_decrease_assessment",
|
||||||
|
has_versor_binding=False,
|
||||||
|
versor_error=None,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
resolution = resolve_promotable_fraction_decrease(case.problem)
|
||||||
|
if resolution is None:
|
||||||
|
reason = "operator_refused"
|
||||||
|
if not assessment.runnable:
|
||||||
|
reason = assessment.explanation or reason
|
||||||
|
elif geom_fd is not None and not geom_fd.runnable:
|
||||||
|
reason = geom_fd.explanation or reason
|
||||||
|
return AblationResult(
|
||||||
|
condition="operator",
|
||||||
|
case_id=case.case_id,
|
||||||
|
outcome=_score(answer=None, expected=case.expected_answer, expected_outcome=case.expected_outcome),
|
||||||
|
answer=None,
|
||||||
|
answer_unit=None,
|
||||||
|
expected_answer=case.expected_answer,
|
||||||
|
refusal_reason=reason,
|
||||||
|
readback=readback,
|
||||||
|
depth=None,
|
||||||
|
frame_runnable=assessment.runnable,
|
||||||
|
operator_bindings=len(assessment.bindings) if assessment.bindings else (
|
||||||
|
len(geom_fd.bindings) if geom_fd else 0
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
return AblationResult(
|
||||||
|
condition="operator",
|
||||||
|
case_id=case.case_id,
|
||||||
|
outcome=_score(
|
||||||
|
answer=float(resolution.answer),
|
||||||
|
expected=case.expected_answer,
|
||||||
|
expected_outcome=case.expected_outcome,
|
||||||
|
),
|
||||||
|
answer=float(resolution.answer),
|
||||||
|
answer_unit=resolution.answer_unit or None,
|
||||||
|
expected_answer=case.expected_answer,
|
||||||
|
refusal_reason="",
|
||||||
|
readback=readback,
|
||||||
|
depth=None,
|
||||||
|
frame_runnable=assessment.runnable,
|
||||||
|
operator_bindings=len(assessment.bindings) or (
|
||||||
|
len(geom_fd.bindings) if geom_fd else 0
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def run_depth(case: AblationCase) -> AblationResult:
|
||||||
|
"""Operator path + explicit depth-contribution provenance.
|
||||||
|
|
||||||
|
For English-only problems there is no authored executable depth mapping.
|
||||||
|
The contribution is recorded as ``inactive_no_mapping`` so the experiment
|
||||||
|
cannot claim depth value it did not earn. Answer equals the operator path.
|
||||||
|
"""
|
||||||
|
op = run_operator(case)
|
||||||
|
depth = DepthContribution(
|
||||||
|
mapping_rule_id=DEPTH_EXECUTABLE_MAPPING_RULE,
|
||||||
|
source_pack_ids=(),
|
||||||
|
source_record_ids=(),
|
||||||
|
evidence_spans=(),
|
||||||
|
validity="inactive_no_mapping",
|
||||||
|
reason=(
|
||||||
|
"No authored executable depth→relation mapping for English surface; "
|
||||||
|
"anti-circularity forbids inventing Greek case / Hebrew binyan from English."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
return AblationResult(
|
||||||
|
condition="depth",
|
||||||
|
case_id=op.case_id,
|
||||||
|
outcome=op.outcome,
|
||||||
|
answer=op.answer,
|
||||||
|
answer_unit=op.answer_unit,
|
||||||
|
expected_answer=op.expected_answer,
|
||||||
|
refusal_reason=op.refusal_reason,
|
||||||
|
readback=op.readback,
|
||||||
|
depth=depth,
|
||||||
|
frame_runnable=op.frame_runnable,
|
||||||
|
operator_bindings=op.operator_bindings,
|
||||||
|
explanation_has_root_note=op.explanation_has_root_note,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def run_metadata_only(
|
||||||
|
case: AblationCase,
|
||||||
|
depth_labels: Mapping[str, Mapping[str, str]] | None = None,
|
||||||
|
) -> AblationResult:
|
||||||
|
"""Depth labels decorate assessments; operator answer must stay unchanged.
|
||||||
|
|
||||||
|
Labels default to a synthetic he root note that must not affect arithmetic.
|
||||||
|
"""
|
||||||
|
labels: dict[str, dict[str, str]] = {
|
||||||
|
k: dict(v)
|
||||||
|
for k, v in (
|
||||||
|
depth_labels
|
||||||
|
or case.depth_override
|
||||||
|
or {"p0": {"language": "he", "root": "א-מ-נ"}}
|
||||||
|
).items()
|
||||||
|
}
|
||||||
|
frame = build_problem_frame(case.problem)
|
||||||
|
assessments = assess_contracts(frame, depth=labels)
|
||||||
|
fd = next((a for a in assessments if a.candidate_organ == "fraction_decrease"), None)
|
||||||
|
has_root_note = bool(
|
||||||
|
fd is not None and "[root:" in (fd.explanation or "")
|
||||||
|
)
|
||||||
|
|
||||||
|
# Operator path is independent of assess_contracts decoration.
|
||||||
|
resolution = resolve_promotable_fraction_decrease(case.problem)
|
||||||
|
roles = _frame_roles(frame)
|
||||||
|
readback = (
|
||||||
|
_readback_from_assessment(fd, roles)
|
||||||
|
if fd is not None
|
||||||
|
else RelationReadback(
|
||||||
|
relation_type="decrease_to_fraction",
|
||||||
|
roles=roles,
|
||||||
|
scale_surface=None,
|
||||||
|
base_surface=None,
|
||||||
|
missing_bindings=(),
|
||||||
|
explanation="no_fraction_decrease_in_assess_contracts",
|
||||||
|
has_versor_binding=False,
|
||||||
|
versor_error=None,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
depth = DepthContribution(
|
||||||
|
mapping_rule_id=DEPTH_METADATA_RULE,
|
||||||
|
source_pack_ids=("synthetic_metadata_only",),
|
||||||
|
source_record_ids=tuple(sorted(labels.keys())),
|
||||||
|
evidence_spans=(),
|
||||||
|
validity="metadata_only",
|
||||||
|
reason="Depth labels decorate ContractAssessment.explanation only; not executable.",
|
||||||
|
)
|
||||||
|
|
||||||
|
if resolution is None:
|
||||||
|
return AblationResult(
|
||||||
|
condition="metadata_only",
|
||||||
|
case_id=case.case_id,
|
||||||
|
outcome=_score(answer=None, expected=case.expected_answer, expected_outcome=case.expected_outcome),
|
||||||
|
answer=None,
|
||||||
|
answer_unit=None,
|
||||||
|
expected_answer=case.expected_answer,
|
||||||
|
refusal_reason=(fd.explanation if fd is not None else "operator_refused"),
|
||||||
|
readback=readback,
|
||||||
|
depth=depth,
|
||||||
|
frame_runnable=bool(fd and fd.runnable),
|
||||||
|
operator_bindings=len(fd.bindings) if fd and fd.bindings else 0,
|
||||||
|
explanation_has_root_note=has_root_note,
|
||||||
|
)
|
||||||
|
|
||||||
|
return AblationResult(
|
||||||
|
condition="metadata_only",
|
||||||
|
case_id=case.case_id,
|
||||||
|
outcome=_score(
|
||||||
|
answer=float(resolution.answer),
|
||||||
|
expected=case.expected_answer,
|
||||||
|
expected_outcome=case.expected_outcome,
|
||||||
|
),
|
||||||
|
answer=float(resolution.answer),
|
||||||
|
answer_unit=resolution.answer_unit or None,
|
||||||
|
expected_answer=case.expected_answer,
|
||||||
|
refusal_reason="",
|
||||||
|
readback=readback,
|
||||||
|
depth=depth,
|
||||||
|
frame_runnable=bool(fd and fd.runnable),
|
||||||
|
operator_bindings=len(fd.bindings) if fd and fd.bindings else 0,
|
||||||
|
explanation_has_root_note=has_root_note,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def run_invalid(case: AblationCase) -> AblationResult:
|
||||||
|
"""Force the invalid/adversarial condition: commit only on clean organ admit."""
|
||||||
|
# Same as operator scoring: confusers must refuse, not guess.
|
||||||
|
result = run_operator(case)
|
||||||
|
return AblationResult(
|
||||||
|
condition="invalid",
|
||||||
|
case_id=result.case_id,
|
||||||
|
outcome=result.outcome,
|
||||||
|
answer=result.answer,
|
||||||
|
answer_unit=result.answer_unit,
|
||||||
|
expected_answer=result.expected_answer,
|
||||||
|
refusal_reason=result.refusal_reason or (
|
||||||
|
"committed" if result.answer is not None else "refused"
|
||||||
|
),
|
||||||
|
readback=result.readback,
|
||||||
|
depth=DepthContribution(
|
||||||
|
mapping_rule_id="invalid.control",
|
||||||
|
source_pack_ids=(),
|
||||||
|
source_record_ids=(),
|
||||||
|
evidence_spans=(),
|
||||||
|
validity="invalid",
|
||||||
|
reason="Adversarial/invalid control case; refuse preferred over wrong.",
|
||||||
|
),
|
||||||
|
frame_runnable=result.frame_runnable,
|
||||||
|
operator_bindings=result.operator_bindings,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
_CONDITION_RUNNERS = {
|
||||||
|
"baseline": run_baseline,
|
||||||
|
"operator": run_operator,
|
||||||
|
"depth": run_depth,
|
||||||
|
"metadata_only": run_metadata_only,
|
||||||
|
"invalid": run_invalid,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def run_condition(condition: ConditionName, case: AblationCase) -> AblationResult:
|
||||||
|
return _CONDITION_RUNNERS[condition](case)
|
||||||
|
|
||||||
|
|
||||||
|
def run_all_conditions(case: AblationCase) -> dict[ConditionName, AblationResult]:
|
||||||
|
# For gold-solve cases, skip invalid runner (same as operator); for refuse gold, use invalid.
|
||||||
|
results: dict[ConditionName, AblationResult] = {}
|
||||||
|
for name in ("baseline", "operator", "depth", "metadata_only"):
|
||||||
|
results[name] = run_condition(name, case) # type: ignore[arg-type]
|
||||||
|
if case.expected_outcome == "refused" or "adversarial" in case.tags or "invalid" in case.tags:
|
||||||
|
results["invalid"] = run_invalid(case)
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
def answers_match(a: AblationResult, b: AblationResult, *, tol: float = 1e-6) -> bool:
|
||||||
|
"""True when both refuse, or both commit equal answers."""
|
||||||
|
if a.answer is None and b.answer is None:
|
||||||
|
return True
|
||||||
|
if a.answer is None or b.answer is None:
|
||||||
|
return False
|
||||||
|
return abs(float(a.answer) - float(b.answer)) < tol
|
||||||
318
tests/test_relational_operator_ablation.py
Normal file
318
tests/test_relational_operator_ablation.py
Normal file
|
|
@ -0,0 +1,318 @@
|
||||||
|
"""Unit, integration, and adversarial proofs for relational operator ablation."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from generate.relational_operator_ablation import (
|
||||||
|
AblationCase,
|
||||||
|
DEPTH_EXECUTABLE_MAPPING_RULE,
|
||||||
|
DEPTH_METADATA_RULE,
|
||||||
|
answers_match,
|
||||||
|
run_all_conditions,
|
||||||
|
run_baseline,
|
||||||
|
run_condition,
|
||||||
|
run_depth,
|
||||||
|
run_metadata_only,
|
||||||
|
run_operator,
|
||||||
|
)
|
||||||
|
from algebra.versor import versor_condition
|
||||||
|
|
||||||
|
_CASES = (
|
||||||
|
Path(__file__).resolve().parents[1]
|
||||||
|
/ "evals"
|
||||||
|
/ "relational_operator_ablation"
|
||||||
|
/ "v1"
|
||||||
|
/ "cases.jsonl"
|
||||||
|
)
|
||||||
|
|
||||||
|
GOLD_0001 = AblationCase(
|
||||||
|
case_id="unit-gold-0001",
|
||||||
|
problem=(
|
||||||
|
"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. "
|
||||||
|
"If the current temperature of the mountain is 84 degrees, what will the temperature "
|
||||||
|
"decrease by?"
|
||||||
|
),
|
||||||
|
expected_answer=21.0,
|
||||||
|
expected_unit="degrees",
|
||||||
|
expected_outcome="correct",
|
||||||
|
tags=("fraction_decrease", "gold"),
|
||||||
|
)
|
||||||
|
|
||||||
|
FINAL_VALUE = AblationCase(
|
||||||
|
case_id="unit-final-value",
|
||||||
|
problem=(
|
||||||
|
"In one hour, the lake's temperature will decrease to 3/4 of its temperature. "
|
||||||
|
"If the current temperature of the lake is 80 degrees, what will the temperature be?"
|
||||||
|
),
|
||||||
|
expected_answer=None,
|
||||||
|
expected_unit="",
|
||||||
|
expected_outcome="refused",
|
||||||
|
tags=("adversarial", "invalid"),
|
||||||
|
)
|
||||||
|
|
||||||
|
MULTI_FRACTION = AblationCase(
|
||||||
|
case_id="unit-multi-fraction",
|
||||||
|
problem=(
|
||||||
|
"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature "
|
||||||
|
"and then to 1/2 of its temperature. If the current temperature of the mountain is 84 "
|
||||||
|
"degrees, what will the temperature decrease by?"
|
||||||
|
),
|
||||||
|
expected_answer=None,
|
||||||
|
expected_unit="",
|
||||||
|
expected_outcome="refused",
|
||||||
|
tags=("adversarial", "invalid"),
|
||||||
|
)
|
||||||
|
|
||||||
|
SCALE_OOR_PREEXISTING = AblationCase(
|
||||||
|
case_id="unit-scale-oor-preexisting",
|
||||||
|
problem=(
|
||||||
|
"In one hour, the river will decrease to 5/4 of its level. "
|
||||||
|
"If the current level is 40 feet, what will the level decrease by?"
|
||||||
|
),
|
||||||
|
expected_answer=None,
|
||||||
|
expected_unit="",
|
||||||
|
expected_outcome="refused",
|
||||||
|
tags=("preexisting_gap",),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _load_sealed() -> list[AblationCase]:
|
||||||
|
cases: list[AblationCase] = []
|
||||||
|
for line in _CASES.read_text(encoding="utf-8").splitlines():
|
||||||
|
if not line.strip():
|
||||||
|
continue
|
||||||
|
raw = json.loads(line)
|
||||||
|
cases.append(
|
||||||
|
AblationCase(
|
||||||
|
case_id=raw["case_id"],
|
||||||
|
problem=raw["problem"],
|
||||||
|
expected_answer=raw.get("expected_answer"),
|
||||||
|
expected_unit=raw.get("expected_unit") or "",
|
||||||
|
expected_outcome=raw["expected_outcome"],
|
||||||
|
tags=tuple(raw.get("tags") or ()),
|
||||||
|
notes=raw.get("notes") or "",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return cases
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Unit
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_baseline_pure_rational_no_versor_binding() -> None:
|
||||||
|
result = run_baseline(GOLD_0001)
|
||||||
|
assert result.outcome == "correct"
|
||||||
|
assert result.answer == pytest.approx(21.0)
|
||||||
|
assert result.operator_bindings == 0
|
||||||
|
assert result.readback is not None
|
||||||
|
assert result.readback.has_versor_binding is True or result.frame_runnable
|
||||||
|
# Baseline path itself does not *require* bindings to compute; frame may still
|
||||||
|
# produce them in assess_fraction_decrease for readback.
|
||||||
|
assert result.condition == "baseline"
|
||||||
|
|
||||||
|
|
||||||
|
def test_operator_commits_geometric_binding_with_versor_integrity() -> None:
|
||||||
|
result = run_operator(GOLD_0001)
|
||||||
|
assert result.outcome == "correct"
|
||||||
|
assert result.answer == pytest.approx(21.0, abs=1e-6)
|
||||||
|
assert result.operator_bindings >= 1
|
||||||
|
assert result.readback is not None
|
||||||
|
assert result.readback.scale_surface == "3/4"
|
||||||
|
assert result.readback.has_versor_binding is True
|
||||||
|
assert result.readback.versor_error is not None
|
||||||
|
assert result.readback.versor_error < 1e-6
|
||||||
|
|
||||||
|
|
||||||
|
def test_metadata_only_decorates_but_does_not_change_answer() -> None:
|
||||||
|
op = run_operator(GOLD_0001)
|
||||||
|
meta = run_metadata_only(GOLD_0001)
|
||||||
|
assert answers_match(op, meta)
|
||||||
|
assert meta.explanation_has_root_note is True
|
||||||
|
assert meta.depth is not None
|
||||||
|
assert meta.depth.validity == "metadata_only"
|
||||||
|
assert meta.depth.mapping_rule_id == DEPTH_METADATA_RULE
|
||||||
|
|
||||||
|
|
||||||
|
def test_depth_condition_records_inactive_mapping_and_matches_operator() -> None:
|
||||||
|
op = run_operator(GOLD_0001)
|
||||||
|
depth = run_depth(GOLD_0001)
|
||||||
|
assert answers_match(op, depth)
|
||||||
|
assert depth.depth is not None
|
||||||
|
assert depth.depth.validity == "inactive_no_mapping"
|
||||||
|
assert depth.depth.mapping_rule_id == DEPTH_EXECUTABLE_MAPPING_RULE
|
||||||
|
assert depth.depth.source_pack_ids == ()
|
||||||
|
|
||||||
|
|
||||||
|
def test_determinism_two_runs_identical() -> None:
|
||||||
|
a = run_operator(GOLD_0001)
|
||||||
|
b = run_operator(GOLD_0001)
|
||||||
|
assert answers_match(a, b)
|
||||||
|
assert a.refusal_reason == b.refusal_reason
|
||||||
|
assert a.operator_bindings == b.operator_bindings
|
||||||
|
|
||||||
|
|
||||||
|
def test_baseline_and_operator_agree_when_both_commit() -> None:
|
||||||
|
base = run_baseline(GOLD_0001)
|
||||||
|
op = run_operator(GOLD_0001)
|
||||||
|
assert base.answer is not None and op.answer is not None
|
||||||
|
assert answers_match(base, op)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Integration — sealed fixture end-to-end
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_sealed_fixture_has_expected_shape() -> None:
|
||||||
|
cases = _load_sealed()
|
||||||
|
assert len(cases) == 8
|
||||||
|
gold = [c for c in cases if c.expected_outcome == "correct"]
|
||||||
|
refuse = [c for c in cases if c.expected_outcome == "refused"]
|
||||||
|
assert len(gold) == 2
|
||||||
|
assert len(refuse) == 6
|
||||||
|
|
||||||
|
|
||||||
|
def test_sealed_gold_cases_operator_and_baseline_correct() -> None:
|
||||||
|
for case in _load_sealed():
|
||||||
|
if case.expected_outcome != "correct":
|
||||||
|
continue
|
||||||
|
base = run_baseline(case)
|
||||||
|
op = run_operator(case)
|
||||||
|
assert base.outcome == "correct", case.case_id
|
||||||
|
assert op.outcome == "correct", case.case_id
|
||||||
|
assert answers_match(base, op)
|
||||||
|
|
||||||
|
|
||||||
|
def test_sealed_adversarial_cases_refuse_not_wrong() -> None:
|
||||||
|
for case in _load_sealed():
|
||||||
|
if case.expected_outcome != "refused":
|
||||||
|
continue
|
||||||
|
op = run_operator(case)
|
||||||
|
base = run_baseline(case)
|
||||||
|
assert op.outcome == "refused", (case.case_id, op.outcome, op.answer)
|
||||||
|
assert base.outcome == "refused", (case.case_id, base.outcome, base.answer)
|
||||||
|
assert op.outcome != "wrong"
|
||||||
|
assert base.outcome != "wrong"
|
||||||
|
|
||||||
|
|
||||||
|
def test_runner_report_invariants() -> None:
|
||||||
|
from evals.relational_operator_ablation.v1.runner import build_report, _load_cases
|
||||||
|
|
||||||
|
report = build_report(_load_cases())
|
||||||
|
assert report["sample_size"] == 8
|
||||||
|
assert report["findings"]["metadata_only_inert"] is True
|
||||||
|
assert report["findings"]["depth_executable_inactive_equals_operator"] is True
|
||||||
|
assert report["findings"]["operator_deterministic"] is True
|
||||||
|
assert report["findings"]["operator_wrong_zero"] is True
|
||||||
|
assert report["findings"]["baseline_wrong_zero"] is True
|
||||||
|
assert report["counts"]["operator"]["correct"] >= 2
|
||||||
|
assert report["counts"]["operator"]["wrong"] == 0
|
||||||
|
assert report["report_sha256"]
|
||||||
|
# Deterministic digest across two builds.
|
||||||
|
report2 = build_report(_load_cases())
|
||||||
|
assert report["report_sha256"] == report2["report_sha256"]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Adversarial
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"case",
|
||||||
|
[FINAL_VALUE, MULTI_FRACTION],
|
||||||
|
ids=["final_value", "multi_fraction"],
|
||||||
|
)
|
||||||
|
def test_adversarial_refuse_closed(case: AblationCase) -> None:
|
||||||
|
results = run_all_conditions(case)
|
||||||
|
for name, result in results.items():
|
||||||
|
assert result.outcome == "refused", (name, result.outcome, result.answer)
|
||||||
|
assert result.answer is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_preexisting_scale_out_of_range_gap_isolated() -> None:
|
||||||
|
"""Document pre-existing Gate A2k gap: scale>1 may still admit geometrically.
|
||||||
|
|
||||||
|
Frame assess_fraction_decrease marks scale_out_of_range (baseline refuses).
|
||||||
|
Geometric A2k path may still compute a negative delta — not introduced by
|
||||||
|
this ablation. Do not mask; pin the split so a future organ fix flips this.
|
||||||
|
"""
|
||||||
|
base = run_baseline(SCALE_OOR_PREEXISTING)
|
||||||
|
op = run_operator(SCALE_OOR_PREEXISTING)
|
||||||
|
assert base.outcome == "refused"
|
||||||
|
assert base.answer is None
|
||||||
|
# Pre-existing: operator may wrong-commit. If a future PR fixes A2k, this
|
||||||
|
# assertion should be updated to require refuse.
|
||||||
|
assert op.outcome in {"refused", "wrong"}
|
||||||
|
if op.outcome == "wrong":
|
||||||
|
assert op.answer is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_malformed_depth_labels_cannot_elevate_runnable() -> None:
|
||||||
|
"""Garbage depth must not create a runnable assessment that wasn't runnable."""
|
||||||
|
# Use a refuse-gold problem.
|
||||||
|
broken_depth = {
|
||||||
|
"p0": {"language": "he", "root": "NOT_A_REAL_ROOT_!!!@@@"},
|
||||||
|
"p1": {"language": "grc", "root": ""},
|
||||||
|
}
|
||||||
|
refuse_case = FINAL_VALUE
|
||||||
|
clean = run_operator(refuse_case)
|
||||||
|
polluted = run_metadata_only(refuse_case, depth_labels=broken_depth)
|
||||||
|
assert clean.outcome == "refused"
|
||||||
|
assert polluted.outcome == "refused"
|
||||||
|
assert polluted.answer is None
|
||||||
|
# Must not flip to correct via depth.
|
||||||
|
assert polluted.outcome != "correct"
|
||||||
|
|
||||||
|
|
||||||
|
def test_valid_depth_metadata_on_gold_does_not_change_answer() -> None:
|
||||||
|
depths = {
|
||||||
|
"p0": {"language": "he", "root": "א-מ-נ"},
|
||||||
|
"p1": {"language": "grc", "root": "λογ"},
|
||||||
|
}
|
||||||
|
op = run_operator(GOLD_0001)
|
||||||
|
meta = run_metadata_only(GOLD_0001, depth_labels=depths)
|
||||||
|
assert answers_match(op, meta)
|
||||||
|
assert meta.explanation_has_root_note is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_distractor_quantity_case_refuses() -> None:
|
||||||
|
cases = {c.case_id: c for c in _load_sealed()}
|
||||||
|
case = cases["roa-v1-0006"]
|
||||||
|
assert run_operator(case).outcome == "refused"
|
||||||
|
assert run_baseline(case).outcome == "refused"
|
||||||
|
|
||||||
|
|
||||||
|
def test_percent_partition_out_of_family_refuses_on_fraction_organ() -> None:
|
||||||
|
cases = {c.case_id: c for c in _load_sealed()}
|
||||||
|
case = cases["roa-v1-0005"]
|
||||||
|
assert run_operator(case).outcome == "refused"
|
||||||
|
|
||||||
|
|
||||||
|
def test_affine_confuser_refuses() -> None:
|
||||||
|
cases = {c.case_id: c for c in _load_sealed()}
|
||||||
|
case = cases["roa-v1-0004"]
|
||||||
|
assert run_operator(case).outcome == "refused"
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_mutation_of_input_case() -> None:
|
||||||
|
problem = GOLD_0001.problem
|
||||||
|
run_operator(GOLD_0001)
|
||||||
|
run_baseline(GOLD_0001)
|
||||||
|
run_metadata_only(GOLD_0001)
|
||||||
|
assert GOLD_0001.problem == problem
|
||||||
|
|
||||||
|
|
||||||
|
def test_readback_preserves_role_provenance_on_gold() -> None:
|
||||||
|
result = run_operator(GOLD_0001)
|
||||||
|
assert result.readback is not None
|
||||||
|
role_names = {r for r, _ in result.readback.roles}
|
||||||
|
assert "base_quantity" in role_names
|
||||||
|
assert "scale" in role_names
|
||||||
|
assert result.readback.scale_surface == "3/4"
|
||||||
Loading…
Reference in a new issue