feat(demo): ADR-0074 — orthogonality tour (anchor-lens × register)
A single demo that walks the full 3 × 3 × 2 matrix (register × lens
× prompts, 18 cells) and pins five claims simultaneously, packaging
both single-axis invariants into one composition gate.
The single-axis tours assert opposite invariants:
register-tour : per (lens, prompt), trace_hash CONSTANT across
registers (R5 / ADR-0072).
anchor-lens-tour : per (register, prompt), engaged lens diverges
in trace_hash from the unanchored baseline
(L1.4 / ADR-0073d).
Orthogonality-tour packages both claims simultaneously across the
full matrix, plus three surface-level claims that pin the markers
operators actually see.
Composed claims (all five must hold)
A) inner_register_invariant_within_lens
For each (lens, prompt) cell, the three register runs share an
identical trace_hash. (R5 register-tour, applied 6 times:
3 lenses × 2 prompts.)
B) outer_lens_distinctness_within_register
For each (register, prompt) cell where any non-unanchored lens
engages, that engaged lens's trace_hash differs from the
unanchored baseline at the same (register, prompt).
(L1.4 anchor-lens-tour, applied 6 times: 3 registers × 2 prompts.)
C) surface_carries_register_marker_under_convivial
Every convivial cell with a non-empty surface has a non-empty
register_variant_id.
D) surface_carries_lens_annotation_when_engaged
Every engaged cell carries [lens(<id>):<mode>] in surface AND
a non-empty anchor_lens_mode_label.
E) no_substrate_glyph_leak_across_grid
No cell's surface contains Greek/Hebrew/Syriac/Arabic glyphs.
(ADR-0073c gate re-asserted across the full matrix.)
CLI wiring
core demo orthogonality-tour human-readable grid + claims
core demo orthogonality-tour --json structured report
Exit code 0 iff all five claims hold.
Files
evals/orthogonality_tour/__init__.py NEW
evals/orthogonality_tour/run_tour.py NEW
core/cli.py EDIT
- cmd_demo handler wires orthogonality-tour
- demo choices + EPILOG examples updated
tests/test_orthogonality_tour_demo.py NEW (9 tests)
docs/decisions/ADR-0074-orthogonality-tour.md NEW
Sanity check baked into tests
test_engaged_cells_appear_for_both_non_trivial_lenses pins that
grc_logos_v1 engages on knowledge in all 3 registers (3 cells)
and he_logos_v1 engages on truth in all 3 registers (3 cells).
Prevents the lift claims being vacuously satisfied by a future
engagement regression.
Lane evidence
- 9 new orthogonality-tour tests pass.
- core demo register-tour → all_claims_supported: True
- core demo anchor-lens-tour → all_claims_supported: True
- core demo orthogonality-tour → all_claims_supported: True
- python -m core.cli eval cognition → byte-identical 100/100/91.7/100.
- Full lane: 2745 passed / 4 skipped / 1 pre-existing failure
(+9 over L1.4's 2736; the one failure remains
test_all_preamble_explains_combined_run, unrelated).
No runtime / composer / loader / pack / schema changes. Pure demo
consumer of existing telemetry contracts.
This commit is contained in:
Shay
parent
1feec74b1c
commit
4426f387d1
5 changed files with 649 additions and 1 deletions
14
core/cli.py
14
core/cli.py
|
|
@ -23,7 +23,7 @@ _CORE_RS_DIR = _REPO_ROOT / "core-rs"
|
||||||
_CORE_RS_MANIFEST = _CORE_RS_DIR / "Cargo.toml"
|
_CORE_RS_MANIFEST = _CORE_RS_DIR / "Cargo.toml"
|
||||||
|
|
||||||
DESCRIPTION = "CORE versor engine command suite."
|
DESCRIPTION = "CORE versor engine command suite."
|
||||||
EPILOG = "Examples:\n core chat\n core pulse \"What is truth?\"\n core pulse --no-glove --json \"Compare knowledge and wisdom\"\n core bench\n core bench --suite all\n core bench --suite all --json --report bench_all.json\n core bench --suite determinism --runs 50\n core bench --suite speedup --json\n core trace \"word beginning truth\"\n core trace --output-language grc --frame-pack grc --json \"logos\"\n core rust status\n core rust build\n core oov covenant\n core pack list\n core pack verify en_minimal_v1\n core teaching audit\n core teaching audit --json\n core teaching gaps --top 10\n core teaching queue --threshold 3\n core teaching propose <candidate-jsonl-path>\n core teaching proposals --state pending\n core teaching review <proposal_id> --accept --review-date 2026-05-18\n core teaching supersede cause_light_reveals_truth --subject light --intent cause --connective grounds --object truth --review-date 2026-05-18\n core teaching supersessions\n core teaching supersessions --json\n core test --suite fast -q\n core test --suite pulse -q\n core test --suite proof -q\n core test --suite cognition -q\n core test -- tests/test_alignment_graph.py -q\n core demo audit-tour\n core demo register-tour\n core demo anchor-lens-tour\n core demo pack-measurements\n core demo long-context-comparison\n core demo anti-regression\n core demo learning-loop\n core demo articulation\n core demo conversation\n core demo conversation --no-stream\n core demo all\n core demo adr-0024-chain\n core eval --list\n core eval cognition\n core eval cognition --json --save\n core eval cognition --split dev --version v1\n core eval cognition --split holdout"
|
EPILOG = "Examples:\n core chat\n core pulse \"What is truth?\"\n core pulse --no-glove --json \"Compare knowledge and wisdom\"\n core bench\n core bench --suite all\n core bench --suite all --json --report bench_all.json\n core bench --suite determinism --runs 50\n core bench --suite speedup --json\n core trace \"word beginning truth\"\n core trace --output-language grc --frame-pack grc --json \"logos\"\n core rust status\n core rust build\n core oov covenant\n core pack list\n core pack verify en_minimal_v1\n core teaching audit\n core teaching audit --json\n core teaching gaps --top 10\n core teaching queue --threshold 3\n core teaching propose <candidate-jsonl-path>\n core teaching proposals --state pending\n core teaching review <proposal_id> --accept --review-date 2026-05-18\n core teaching supersede cause_light_reveals_truth --subject light --intent cause --connective grounds --object truth --review-date 2026-05-18\n core teaching supersessions\n core teaching supersessions --json\n core test --suite fast -q\n core test --suite pulse -q\n core test --suite proof -q\n core test --suite cognition -q\n core test -- tests/test_alignment_graph.py -q\n core demo audit-tour\n core demo register-tour\n core demo anchor-lens-tour\n core demo orthogonality-tour\n core demo pack-measurements\n core demo long-context-comparison\n core demo anti-regression\n core demo learning-loop\n core demo articulation\n core demo conversation\n core demo conversation --no-stream\n core demo all\n core demo adr-0024-chain\n core eval --list\n core eval cognition\n core eval cognition --json --save\n core eval cognition --split dev --version v1\n core eval cognition --split holdout"
|
||||||
|
|
||||||
_TEST_SUITES: dict[str, tuple[str, ...]] = {
|
_TEST_SUITES: dict[str, tuple[str, ...]] = {
|
||||||
"fast": (
|
"fast": (
|
||||||
|
|
@ -1981,6 +1981,14 @@ def cmd_demo(args: argparse.Namespace) -> int:
|
||||||
print(json.dumps(result, indent=2, sort_keys=True, default=str))
|
print(json.dumps(result, indent=2, sort_keys=True, default=str))
|
||||||
return 0 if result.get("all_claims_supported", False) else 1
|
return 0 if result.get("all_claims_supported", False) else 1
|
||||||
|
|
||||||
|
if target == "orthogonality-tour":
|
||||||
|
from evals.orthogonality_tour.run_tour import run_tour as run_ortho_tour
|
||||||
|
|
||||||
|
result = run_ortho_tour(emit_json=args.json)
|
||||||
|
if args.json:
|
||||||
|
print(json.dumps(result, indent=2, sort_keys=True, default=str))
|
||||||
|
return 0 if result.get("all_claims_supported", False) else 1
|
||||||
|
|
||||||
if target == "pack-measurements":
|
if target == "pack-measurements":
|
||||||
from scripts.publish_pack_measurements import (
|
from scripts.publish_pack_measurements import (
|
||||||
build_combined_report,
|
build_combined_report,
|
||||||
|
|
@ -2877,6 +2885,7 @@ def build_parser() -> argparse.ArgumentParser:
|
||||||
"audit-tour",
|
"audit-tour",
|
||||||
"register-tour",
|
"register-tour",
|
||||||
"anchor-lens-tour",
|
"anchor-lens-tour",
|
||||||
|
"orthogonality-tour",
|
||||||
"pack-measurements",
|
"pack-measurements",
|
||||||
"long-context-comparison",
|
"long-context-comparison",
|
||||||
"anti-regression",
|
"anti-regression",
|
||||||
|
|
@ -2901,6 +2910,9 @@ def build_parser() -> argparse.ArgumentParser:
|
||||||
"prompts × three lenses; trace_hash DISTINCT across lenses, "
|
"prompts × three lenses; trace_hash DISTINCT across lenses, "
|
||||||
"no substrate glyph leak. Opposite invariant from register-tour; "
|
"no substrate glyph leak. Opposite invariant from register-tour; "
|
||||||
"both must hold continuously. "
|
"both must hold continuously. "
|
||||||
|
"orthogonality-tour: ADR-0074 composition demo — full 3 × 3 × 2 "
|
||||||
|
"matrix (register × lens × prompts, 18 cells); pins five "
|
||||||
|
"claims simultaneously including both single-axis invariants. "
|
||||||
"pack-measurements: ADR-0043 — pack-layer claims → CI-enforced "
|
"pack-measurements: ADR-0043 — pack-layer claims → CI-enforced "
|
||||||
"numbers across the three ratified identity packs. "
|
"numbers across the three ratified identity packs. "
|
||||||
"long-context-comparison: ADR-0045 — CORE exact recall NIAH at "
|
"long-context-comparison: ADR-0045 — CORE exact recall NIAH at "
|
||||||
|
|
|
||||||
181
docs/decisions/ADR-0074-orthogonality-tour.md
Normal file
181
docs/decisions/ADR-0074-orthogonality-tour.md
Normal file
|
|
@ -0,0 +1,181 @@
|
||||||
|
# ADR-0074 — Orthogonality tour: anchor-lens × register composition demo
|
||||||
|
|
||||||
|
**Status:** Accepted
|
||||||
|
**Date:** 2026-05-19
|
||||||
|
**Ratified:** 2026-05-19
|
||||||
|
**Author:** Shay
|
||||||
|
**Builds on:** [ADR-0072](./ADR-0072-register-telemetry-operator-surface.md)
|
||||||
|
(register-tour, R5), [ADR-0073d](./ADR-0073d-anchor-lens-telemetry-tour.md)
|
||||||
|
(anchor-lens-tour, L1.4)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Two single-axis tours ship today:
|
||||||
|
|
||||||
|
* `core demo register-tour` — pins `trace_hash CONSTANT` across
|
||||||
|
registers within a fixed lens.
|
||||||
|
* `core demo anchor-lens-tour` — pins `trace_hash DISTINCT` across
|
||||||
|
lenses within a fixed register.
|
||||||
|
|
||||||
|
L1.3's `test_register_seam_within_lens_holds` already parametrizes
|
||||||
|
the inner claim over every lens choice. ADR-0074 packages **both
|
||||||
|
claims simultaneously** into a single demo that walks the full
|
||||||
|
register × lens × prompt matrix and asserts the orthogonality holds
|
||||||
|
turn-by-turn across the entire grid.
|
||||||
|
|
||||||
|
Without this composition demo, an operator can verify each axis
|
||||||
|
independently but has no single artifact answering the question:
|
||||||
|
|
||||||
|
> When I drive `core chat --register X --anchor-lens Y`, are the two
|
||||||
|
> axes actually independent, or does some unintended coupling
|
||||||
|
> sneak in?
|
||||||
|
|
||||||
|
ADR-0074 ships that artifact as `core demo orthogonality-tour`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
### Demo shape
|
||||||
|
|
||||||
|
A single 3 × 3 × 2 grid:
|
||||||
|
|
||||||
|
* **Lenses (3):** `default_unanchored_v1`, `grc_logos_v1`, `he_logos_v1`
|
||||||
|
* **Registers (3):** `default_neutral_v1`, `terse_v1`, `convivial_v1`
|
||||||
|
* **Prompts (2):** `"What is knowledge?"` (grc engages),
|
||||||
|
`"What is truth?"` (he engages)
|
||||||
|
|
||||||
|
That is 18 cells. For each cell the demo records:
|
||||||
|
|
||||||
|
```
|
||||||
|
surface, trace_hash, grounding_source,
|
||||||
|
register_id, register_variant_id,
|
||||||
|
anchor_lens_id, anchor_lens_mode_label
|
||||||
|
```
|
||||||
|
|
||||||
|
### Composed claims
|
||||||
|
|
||||||
|
```
|
||||||
|
A) inner_register_invariant_within_lens:
|
||||||
|
For each (lens, prompt) cell, the three register runs share an
|
||||||
|
identical trace_hash.
|
||||||
|
(R5 register-tour, applied 6 times: 3 lenses × 2 prompts.)
|
||||||
|
|
||||||
|
B) outer_lens_distinctness_within_register:
|
||||||
|
For each (register, prompt) cell where any non-unanchored lens
|
||||||
|
engages, that engaged lens's trace_hash differs from the
|
||||||
|
unanchored baseline's trace_hash for the same (register, prompt).
|
||||||
|
(L1.4 anchor-lens-tour, applied 6 times: 3 registers × 2 prompts.)
|
||||||
|
|
||||||
|
C) surface_carries_register_marker_under_convivial:
|
||||||
|
For every convivial cell with a non-empty surface,
|
||||||
|
register_variant_id is non-empty (convivial always picks an
|
||||||
|
opening from its non-empty bucket).
|
||||||
|
|
||||||
|
D) surface_carries_lens_annotation_when_engaged:
|
||||||
|
For every cell where a non-unanchored lens engages on the
|
||||||
|
prompt's en lemma, the surface contains `[lens(<id>):<mode>]`
|
||||||
|
and anchor_lens_mode_label is non-empty.
|
||||||
|
|
||||||
|
E) no_substrate_glyph_leak_across_grid:
|
||||||
|
No cell's surface contains Greek/Hebrew/Syriac/Arabic letter
|
||||||
|
blocks. (ADR-0073c gate re-asserted across the full matrix.)
|
||||||
|
```
|
||||||
|
|
||||||
|
The five claims together pin the orthogonality: A says register
|
||||||
|
varies the surface without changing the proposition; B says lens
|
||||||
|
varies the proposition; C and D say the user-visible markers behave
|
||||||
|
as expected; E says no axis leaks substrate glyphs even under
|
||||||
|
composition.
|
||||||
|
|
||||||
|
### Files
|
||||||
|
|
||||||
|
```
|
||||||
|
evals/orthogonality_tour/__init__.py NEW
|
||||||
|
evals/orthogonality_tour/run_tour.py NEW
|
||||||
|
|
||||||
|
core/cli.py EDIT
|
||||||
|
- cmd_demo handler wires orthogonality-tour
|
||||||
|
- demo target choices add "orthogonality-tour"
|
||||||
|
- cmd_demo description gains a line for orthogonality-tour
|
||||||
|
- EPILOG gains "core demo orthogonality-tour"
|
||||||
|
|
||||||
|
tests/test_orthogonality_tour_demo.py NEW
|
||||||
|
- Five claims pinned individually
|
||||||
|
- all_claims_supported overall
|
||||||
|
- grid shape sanity check (3 × 3 × 2 = 18 cells)
|
||||||
|
|
||||||
|
docs/decisions/ADR-0074-orthogonality-tour.md NEW (this file)
|
||||||
|
```
|
||||||
|
|
||||||
|
No runtime, composer, loader, pack, or schema changes. The demo is
|
||||||
|
a pure consumer of the existing telemetry contracts.
|
||||||
|
|
||||||
|
### Invariants pinned
|
||||||
|
|
||||||
|
```
|
||||||
|
register-tour (R5) — preserved
|
||||||
|
anchor-lens-tour (L1.4) — preserved
|
||||||
|
register_invariant_grounding (R3) — preserved
|
||||||
|
anchor_lens_byte_identity_null_lift (L1.2) — preserved
|
||||||
|
anchor_lens_lifts_proposition (L1.3) — preserved
|
||||||
|
anchor_lens_no_glyph_leak (L1.3) — preserved
|
||||||
|
|
||||||
|
invariant_orthogonality_tour_seam (NEW):
|
||||||
|
evals/orthogonality_tour/run_tour.py asserts the five composed
|
||||||
|
claims above on the full 3 × 3 × 2 grid. Exits non-zero on any
|
||||||
|
violation. Pinned by tests/test_orthogonality_tour_demo.py.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
### Capability unlocked
|
||||||
|
|
||||||
|
A single command answers "are register and anchor-lens genuinely
|
||||||
|
orthogonal under composition?" with structured evidence — 18 cells'
|
||||||
|
worth of surfaces, trace hashes, and telemetry fields, plus five
|
||||||
|
falsifiable claim booleans.
|
||||||
|
|
||||||
|
### Cognition lane
|
||||||
|
|
||||||
|
Unchanged. The demo is consumer-only; no production code path is
|
||||||
|
touched.
|
||||||
|
|
||||||
|
### Performance
|
||||||
|
|
||||||
|
The demo runs 18 chat turns sequentially. At current per-turn
|
||||||
|
latency this lands in single-digit seconds. No caching changes.
|
||||||
|
|
||||||
|
### Trust boundaries
|
||||||
|
|
||||||
|
* No new mutation surface. Demo only reads telemetry.
|
||||||
|
* Glyph-leak gate is re-asserted in the demo's own claim bundle so
|
||||||
|
it can never silently regress under composition.
|
||||||
|
* CLI choice list is the only operator-visible change; the
|
||||||
|
underlying flags (`--register`, `--anchor-lens`) are unchanged.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
```
|
||||||
|
python -m pytest tests/test_orthogonality_tour_demo.py -q N passed
|
||||||
|
core demo orthogonality-tour exit 0
|
||||||
|
core demo orthogonality-tour --json stable schema
|
||||||
|
|
||||||
|
Curated lanes (must remain green):
|
||||||
|
smoke / cognition / teaching / packs / runtime / algebra
|
||||||
|
|
||||||
|
Existing single-axis tours continue to pass:
|
||||||
|
core demo register-tour exit 0
|
||||||
|
core demo anchor-lens-tour exit 0
|
||||||
|
```
|
||||||
|
|
||||||
|
The orthogonality tour's exit code is the canonical composition
|
||||||
|
gate — if it ever exits non-zero in CI, the orthogonality between
|
||||||
|
the two axes has regressed and one of the single-axis tours is also
|
||||||
|
likely to be failing.
|
||||||
4
evals/orthogonality_tour/__init__.py
Normal file
4
evals/orthogonality_tour/__init__.py
Normal file
|
|
@ -0,0 +1,4 @@
|
||||||
|
"""Orthogonality tour — composes register × anchor-lens × prompts
|
||||||
|
into a single 3 × 3 × 2 grid and asserts five orthogonality claims
|
||||||
|
hold under composition. See ``run_tour.py``.
|
||||||
|
"""
|
||||||
345
evals/orthogonality_tour/run_tour.py
Normal file
345
evals/orthogonality_tour/run_tour.py
Normal file
|
|
@ -0,0 +1,345 @@
|
||||||
|
"""Orthogonality tour — composes register × anchor-lens × prompts in
|
||||||
|
one 3 × 3 × 2 grid (ADR-0074).
|
||||||
|
|
||||||
|
The single-axis tours assert opposite invariants:
|
||||||
|
|
||||||
|
register-tour : per (lens, prompt), trace_hash CONSTANT across
|
||||||
|
registers (R5 / ADR-0072).
|
||||||
|
anchor-lens-tour : per (register=neutral, prompt), trace_hash
|
||||||
|
DISTINCT across engaging lenses
|
||||||
|
(L1.4 / ADR-0073d).
|
||||||
|
|
||||||
|
This tour packages both claims simultaneously into a single demo
|
||||||
|
that walks the full register × lens × prompt matrix and asserts
|
||||||
|
the orthogonality holds turn-by-turn across all 18 cells.
|
||||||
|
|
||||||
|
Composed claims
|
||||||
|
---------------
|
||||||
|
|
||||||
|
* **A) inner_register_invariant_within_lens** — for each
|
||||||
|
(lens, prompt), the three register runs share an identical
|
||||||
|
trace_hash.
|
||||||
|
* **B) outer_lens_distinctness_within_register** — for each
|
||||||
|
(register, prompt) where any non-unanchored lens engages, the
|
||||||
|
engaged lens's trace_hash differs from the unanchored baseline.
|
||||||
|
* **C) surface_carries_register_marker_under_convivial** — every
|
||||||
|
convivial cell with a non-empty surface has a non-empty
|
||||||
|
``register_variant_id``.
|
||||||
|
* **D) surface_carries_lens_annotation_when_engaged** — every
|
||||||
|
engaged cell carries the ``[lens(<id>):<mode>]`` annotation and a
|
||||||
|
non-empty ``anchor_lens_mode_label``.
|
||||||
|
* **E) no_substrate_glyph_leak_across_grid** — no cell's surface
|
||||||
|
contains Greek/Hebrew/Syriac/Arabic glyphs.
|
||||||
|
|
||||||
|
Exit code 0 iff every claim holds.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from chat.runtime import ChatRuntime
|
||||||
|
from core.cognition.pipeline import CognitiveTurnPipeline
|
||||||
|
from core.config import RuntimeConfig
|
||||||
|
|
||||||
|
|
||||||
|
_REGISTERS = (
|
||||||
|
"default_neutral_v1",
|
||||||
|
"terse_v1",
|
||||||
|
"convivial_v1",
|
||||||
|
)
|
||||||
|
|
||||||
|
_LENSES = (
|
||||||
|
"default_unanchored_v1",
|
||||||
|
"grc_logos_v1",
|
||||||
|
"he_logos_v1",
|
||||||
|
)
|
||||||
|
|
||||||
|
_PROMPTS = (
|
||||||
|
"What is knowledge?",
|
||||||
|
"What is truth?",
|
||||||
|
)
|
||||||
|
|
||||||
|
_FORBIDDEN_BLOCKS: tuple[tuple[int, int, str], ...] = (
|
||||||
|
(0x0370, 0x03FF, "Greek and Coptic"),
|
||||||
|
(0x1F00, 0x1FFF, "Greek Extended"),
|
||||||
|
(0x0590, 0x05FF, "Hebrew"),
|
||||||
|
(0x0700, 0x074F, "Syriac"),
|
||||||
|
(0x0600, 0x06FF, "Arabic"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
_VERBOSE = True
|
||||||
|
|
||||||
|
|
||||||
|
def _say(*args: Any, **kwargs: Any) -> None:
|
||||||
|
if _VERBOSE:
|
||||||
|
print(*args, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def _print_header() -> None:
|
||||||
|
_say()
|
||||||
|
_say("=" * 76)
|
||||||
|
_say(" CORE Orthogonality Tour — register × anchor-lens × prompts")
|
||||||
|
_say("=" * 76)
|
||||||
|
_say(
|
||||||
|
" Walks the full 3 × 3 × 2 matrix (18 cells) and asserts:\n"
|
||||||
|
" A) varying register inside a fixed lens keeps trace_hash CONSTANT\n"
|
||||||
|
" B) varying lens inside a fixed register moves trace_hash where\n"
|
||||||
|
" the lens engages (DISTINCT from unanchored baseline)\n"
|
||||||
|
" C) convivial register attaches discourse markers\n"
|
||||||
|
" D) engaged lens emits [lens(<id>):<mode>] annotations\n"
|
||||||
|
" E) no substrate glyphs (Greek/Hebrew/…) leak into surfaces\n"
|
||||||
|
"\n"
|
||||||
|
" Together these pin orthogonality under composition — register and\n"
|
||||||
|
" anchor-lens never interfere even when both flags are set."
|
||||||
|
)
|
||||||
|
_say()
|
||||||
|
|
||||||
|
|
||||||
|
def _substrate_glyph_violations(surface: str) -> list[tuple[int, str, str]]:
|
||||||
|
out: list[tuple[int, str, str]] = []
|
||||||
|
for i, ch in enumerate(surface):
|
||||||
|
cp = ord(ch)
|
||||||
|
for start, end, label in _FORBIDDEN_BLOCKS:
|
||||||
|
if start <= cp <= end:
|
||||||
|
out.append((i, ch, label))
|
||||||
|
break
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _run_one_cell(register_id: str, lens_id: str, prompt: str) -> dict[str, Any]:
|
||||||
|
runtime = ChatRuntime(config=RuntimeConfig(
|
||||||
|
register_pack_id=register_id,
|
||||||
|
anchor_lens_id=lens_id,
|
||||||
|
))
|
||||||
|
pipeline = CognitiveTurnPipeline(runtime=runtime)
|
||||||
|
result = pipeline.run(prompt)
|
||||||
|
turn_event = runtime.turn_log[-1]
|
||||||
|
return {
|
||||||
|
"prompt": prompt,
|
||||||
|
"register_id": register_id,
|
||||||
|
"lens_id": lens_id,
|
||||||
|
"surface": turn_event.surface,
|
||||||
|
"trace_hash": result.trace_hash,
|
||||||
|
"grounding_source": getattr(turn_event, "grounding_source", ""),
|
||||||
|
"register_variant_id": getattr(turn_event, "register_variant_id", ""),
|
||||||
|
"anchor_lens_id": getattr(turn_event, "anchor_lens_id", ""),
|
||||||
|
"anchor_lens_mode_label": getattr(turn_event, "anchor_lens_mode_label", ""),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _build_grid() -> list[dict[str, Any]]:
|
||||||
|
cells: list[dict[str, Any]] = []
|
||||||
|
for register_id in _REGISTERS:
|
||||||
|
for lens_id in _LENSES:
|
||||||
|
for prompt in _PROMPTS:
|
||||||
|
cells.append(_run_one_cell(register_id, lens_id, prompt))
|
||||||
|
return cells
|
||||||
|
|
||||||
|
|
||||||
|
def _cells_by(
|
||||||
|
cells: list[dict[str, Any]],
|
||||||
|
*,
|
||||||
|
register_id: str | None = None,
|
||||||
|
lens_id: str | None = None,
|
||||||
|
prompt: str | None = None,
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
out: list[dict[str, Any]] = []
|
||||||
|
for c in cells:
|
||||||
|
if register_id is not None and c["register_id"] != register_id:
|
||||||
|
continue
|
||||||
|
if lens_id is not None and c["lens_id"] != lens_id:
|
||||||
|
continue
|
||||||
|
if prompt is not None and c["prompt"] != prompt:
|
||||||
|
continue
|
||||||
|
out.append(c)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _check_claims(cells: list[dict[str, Any]]) -> dict[str, Any]:
|
||||||
|
glyph_violations: list[str] = []
|
||||||
|
register_invariant_failures: list[str] = []
|
||||||
|
lens_distinctness_failures: list[str] = []
|
||||||
|
convivial_marker_failures: list[str] = []
|
||||||
|
lens_annotation_failures: list[str] = []
|
||||||
|
|
||||||
|
# Claim E — glyph leak across whole grid.
|
||||||
|
for c in cells:
|
||||||
|
for pos, ch, block in _substrate_glyph_violations(c["surface"]):
|
||||||
|
glyph_violations.append(
|
||||||
|
f"register={c['register_id']!r} lens={c['lens_id']!r} "
|
||||||
|
f"prompt={c['prompt']!r}: substrate glyph {ch!r} "
|
||||||
|
f"(block={block}) at pos {pos}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Claim A — register-tour seam inside every (lens, prompt) cell.
|
||||||
|
for lens_id in _LENSES:
|
||||||
|
for prompt in _PROMPTS:
|
||||||
|
triple = _cells_by(cells, lens_id=lens_id, prompt=prompt)
|
||||||
|
hashes = {c["trace_hash"] for c in triple}
|
||||||
|
if len(hashes) != 1:
|
||||||
|
register_invariant_failures.append(
|
||||||
|
f"lens={lens_id!r} prompt={prompt!r}: "
|
||||||
|
f"trace_hashes varied across registers: "
|
||||||
|
f"{sorted(hashes)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Claim B — lens distinctness inside every (register, prompt) cell.
|
||||||
|
# Engaged lenses' trace_hashes must differ from the unanchored
|
||||||
|
# baseline at the same (register, prompt).
|
||||||
|
for register_id in _REGISTERS:
|
||||||
|
for prompt in _PROMPTS:
|
||||||
|
unanchored_cell = _cells_by(
|
||||||
|
cells,
|
||||||
|
register_id=register_id,
|
||||||
|
lens_id="default_unanchored_v1",
|
||||||
|
prompt=prompt,
|
||||||
|
)[0]
|
||||||
|
for lens_id in _LENSES:
|
||||||
|
if lens_id == "default_unanchored_v1":
|
||||||
|
continue
|
||||||
|
cell = _cells_by(
|
||||||
|
cells,
|
||||||
|
register_id=register_id,
|
||||||
|
lens_id=lens_id,
|
||||||
|
prompt=prompt,
|
||||||
|
)[0]
|
||||||
|
if cell["anchor_lens_mode_label"]:
|
||||||
|
# Engaged → require trace_hash divergence.
|
||||||
|
if cell["trace_hash"] == unanchored_cell["trace_hash"]:
|
||||||
|
lens_distinctness_failures.append(
|
||||||
|
f"register={register_id!r} lens={lens_id!r} "
|
||||||
|
f"prompt={prompt!r}: engaged lens did not "
|
||||||
|
f"move trace_hash (still "
|
||||||
|
f"{cell['trace_hash'][:12]}...)"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Claim C — convivial register attaches discourse markers.
|
||||||
|
for c in _cells_by(cells, register_id="convivial_v1"):
|
||||||
|
if c["surface"] and not c["register_variant_id"]:
|
||||||
|
convivial_marker_failures.append(
|
||||||
|
f"convivial cell lens={c['lens_id']!r} "
|
||||||
|
f"prompt={c['prompt']!r} has empty register_variant_id "
|
||||||
|
f"despite non-empty surface"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Claim D — engaged lens emits annotation + non-empty mode_label.
|
||||||
|
for c in cells:
|
||||||
|
if c["anchor_lens_mode_label"]:
|
||||||
|
expected = f"[lens({c['lens_id']}):{c['anchor_lens_mode_label']}]"
|
||||||
|
if expected not in c["surface"]:
|
||||||
|
lens_annotation_failures.append(
|
||||||
|
f"register={c['register_id']!r} lens={c['lens_id']!r} "
|
||||||
|
f"prompt={c['prompt']!r}: mode_label="
|
||||||
|
f"{c['anchor_lens_mode_label']!r} but surface lacks "
|
||||||
|
f"annotation {expected!r}. Surface: {c['surface']!r}"
|
||||||
|
)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"inner_register_invariant_within_lens": not register_invariant_failures,
|
||||||
|
"outer_lens_distinctness_within_register": not lens_distinctness_failures,
|
||||||
|
"surface_carries_register_marker_under_convivial": (
|
||||||
|
not convivial_marker_failures
|
||||||
|
),
|
||||||
|
"surface_carries_lens_annotation_when_engaged": not lens_annotation_failures,
|
||||||
|
"no_substrate_glyph_leak_across_grid": not glyph_violations,
|
||||||
|
"register_invariant_failures": register_invariant_failures,
|
||||||
|
"lens_distinctness_failures": lens_distinctness_failures,
|
||||||
|
"convivial_marker_failures": convivial_marker_failures,
|
||||||
|
"lens_annotation_failures": lens_annotation_failures,
|
||||||
|
"glyph_violations": glyph_violations,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _print_grid(cells: list[dict[str, Any]]) -> None:
|
||||||
|
for prompt in _PROMPTS:
|
||||||
|
_say(f" Prompt: {prompt!r}")
|
||||||
|
for register_id in _REGISTERS:
|
||||||
|
_say(f" register = {register_id}")
|
||||||
|
for lens_id in _LENSES:
|
||||||
|
cell = _cells_by(
|
||||||
|
cells, register_id=register_id, lens_id=lens_id,
|
||||||
|
prompt=prompt,
|
||||||
|
)[0]
|
||||||
|
mode = cell["anchor_lens_mode_label"] or "-"
|
||||||
|
variant = cell["register_variant_id"] or "-"
|
||||||
|
_say(
|
||||||
|
f" lens={lens_id:24s} "
|
||||||
|
f"trace={cell['trace_hash'][:12]}… "
|
||||||
|
f"mode={mode:18s} variant={variant}"
|
||||||
|
)
|
||||||
|
_say()
|
||||||
|
|
||||||
|
|
||||||
|
def run_tour(*, emit_json: bool = False) -> dict[str, Any]:
|
||||||
|
"""Run the orthogonality tour and return a structured report."""
|
||||||
|
global _VERBOSE
|
||||||
|
_VERBOSE = not emit_json
|
||||||
|
|
||||||
|
if not emit_json:
|
||||||
|
_print_header()
|
||||||
|
_say(f" Building grid: {len(_REGISTERS)} registers × "
|
||||||
|
f"{len(_LENSES)} lenses × {len(_PROMPTS)} prompts = "
|
||||||
|
f"{len(_REGISTERS) * len(_LENSES) * len(_PROMPTS)} cells")
|
||||||
|
_say()
|
||||||
|
|
||||||
|
cells = _build_grid()
|
||||||
|
|
||||||
|
if not emit_json:
|
||||||
|
_say("-" * 76)
|
||||||
|
_say(" Composition grid")
|
||||||
|
_say("-" * 76)
|
||||||
|
_print_grid(cells)
|
||||||
|
|
||||||
|
claims = _check_claims(cells)
|
||||||
|
all_supported = all(
|
||||||
|
claims[k] for k in (
|
||||||
|
"inner_register_invariant_within_lens",
|
||||||
|
"outer_lens_distinctness_within_register",
|
||||||
|
"surface_carries_register_marker_under_convivial",
|
||||||
|
"surface_carries_lens_annotation_when_engaged",
|
||||||
|
"no_substrate_glyph_leak_across_grid",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if not emit_json:
|
||||||
|
_say("=" * 76)
|
||||||
|
_say(" Orthogonality claims (composition gate)")
|
||||||
|
_say("=" * 76)
|
||||||
|
for label, key in (
|
||||||
|
("A) inner_register_invariant_within_lens ",
|
||||||
|
"inner_register_invariant_within_lens"),
|
||||||
|
("B) outer_lens_distinctness_within_register ",
|
||||||
|
"outer_lens_distinctness_within_register"),
|
||||||
|
("C) surface_carries_register_marker_convivial",
|
||||||
|
"surface_carries_register_marker_under_convivial"),
|
||||||
|
("D) surface_carries_lens_annotation_engaged ",
|
||||||
|
"surface_carries_lens_annotation_when_engaged"),
|
||||||
|
("E) no_substrate_glyph_leak_across_grid ",
|
||||||
|
"no_substrate_glyph_leak_across_grid"),
|
||||||
|
):
|
||||||
|
_say(f" {label}: {claims[key]}")
|
||||||
|
_say()
|
||||||
|
_say(f" all_claims_supported : {all_supported}")
|
||||||
|
_say()
|
||||||
|
|
||||||
|
return {
|
||||||
|
"registers": list(_REGISTERS),
|
||||||
|
"lenses": list(_LENSES),
|
||||||
|
"prompts": list(_PROMPTS),
|
||||||
|
"cells": cells,
|
||||||
|
"claims": claims,
|
||||||
|
"all_claims_supported": all_supported,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__": # pragma: no cover
|
||||||
|
import sys
|
||||||
|
|
||||||
|
emit_json = "--json" in sys.argv
|
||||||
|
report = run_tour(emit_json=emit_json)
|
||||||
|
if emit_json:
|
||||||
|
print(json.dumps(report, indent=2, sort_keys=True, default=str))
|
||||||
|
sys.exit(0 if report["all_claims_supported"] else 1)
|
||||||
106
tests/test_orthogonality_tour_demo.py
Normal file
106
tests/test_orthogonality_tour_demo.py
Normal file
|
|
@ -0,0 +1,106 @@
|
||||||
|
"""Orthogonality tour demo — composition gate (ADR-0074).
|
||||||
|
|
||||||
|
Pins the five composed claims that the single-axis tours only assert
|
||||||
|
in isolation:
|
||||||
|
|
||||||
|
A) inner_register_invariant_within_lens
|
||||||
|
B) outer_lens_distinctness_within_register
|
||||||
|
C) surface_carries_register_marker_under_convivial
|
||||||
|
D) surface_carries_lens_annotation_when_engaged
|
||||||
|
E) no_substrate_glyph_leak_across_grid
|
||||||
|
|
||||||
|
Any one failing is the composition-regression signal.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from evals.orthogonality_tour.run_tour import (
|
||||||
|
_LENSES,
|
||||||
|
_PROMPTS,
|
||||||
|
_REGISTERS,
|
||||||
|
run_tour,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_tour_returns_structured_report():
|
||||||
|
report = run_tour(emit_json=True)
|
||||||
|
assert set(report) >= {
|
||||||
|
"registers", "lenses", "prompts", "cells", "claims",
|
||||||
|
"all_claims_supported",
|
||||||
|
}
|
||||||
|
assert list(report["registers"]) == list(_REGISTERS)
|
||||||
|
assert list(report["lenses"]) == list(_LENSES)
|
||||||
|
assert list(report["prompts"]) == list(_PROMPTS)
|
||||||
|
|
||||||
|
|
||||||
|
def test_grid_has_18_cells():
|
||||||
|
"""3 registers × 3 lenses × 2 prompts."""
|
||||||
|
report = run_tour(emit_json=True)
|
||||||
|
assert len(report["cells"]) == 18
|
||||||
|
|
||||||
|
|
||||||
|
def test_claim_A_register_invariant_within_each_lens():
|
||||||
|
"""Inner register-tour: for every (lens, prompt) the three
|
||||||
|
register runs share an identical trace_hash."""
|
||||||
|
report = run_tour(emit_json=True)
|
||||||
|
assert report["claims"]["inner_register_invariant_within_lens"] is True
|
||||||
|
assert report["claims"]["register_invariant_failures"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_claim_B_lens_distinctness_within_each_register():
|
||||||
|
"""Inner anchor-lens-tour: engaged lenses diverge from the
|
||||||
|
unanchored baseline at every (register, prompt) cell."""
|
||||||
|
report = run_tour(emit_json=True)
|
||||||
|
assert report["claims"]["outer_lens_distinctness_within_register"] is True
|
||||||
|
assert report["claims"]["lens_distinctness_failures"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_claim_C_convivial_carries_register_marker():
|
||||||
|
report = run_tour(emit_json=True)
|
||||||
|
assert report["claims"]["surface_carries_register_marker_under_convivial"] is True
|
||||||
|
assert report["claims"]["convivial_marker_failures"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_claim_D_engaged_lens_carries_annotation():
|
||||||
|
report = run_tour(emit_json=True)
|
||||||
|
assert report["claims"]["surface_carries_lens_annotation_when_engaged"] is True
|
||||||
|
assert report["claims"]["lens_annotation_failures"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_claim_E_no_substrate_glyph_leak_across_grid():
|
||||||
|
report = run_tour(emit_json=True)
|
||||||
|
assert report["claims"]["no_substrate_glyph_leak_across_grid"] is True
|
||||||
|
assert report["claims"]["glyph_violations"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_all_claims_supported():
|
||||||
|
"""Canonical composition gate — every claim must hold or exit non-zero."""
|
||||||
|
report = run_tour(emit_json=True)
|
||||||
|
assert report["all_claims_supported"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_engaged_cells_appear_for_both_non_trivial_lenses():
|
||||||
|
"""Sanity: the grid actually exercises engagement. grc_logos_v1
|
||||||
|
engages on knowledge; he_logos_v1 engages on truth. Each
|
||||||
|
non-trivial lens must produce at least one engaged cell
|
||||||
|
(otherwise the lift claim is vacuously satisfied)."""
|
||||||
|
report = run_tour(emit_json=True)
|
||||||
|
grc_engaged = [
|
||||||
|
c for c in report["cells"]
|
||||||
|
if c["lens_id"] == "grc_logos_v1"
|
||||||
|
and c["anchor_lens_mode_label"] == "systematic"
|
||||||
|
]
|
||||||
|
he_engaged = [
|
||||||
|
c for c in report["cells"]
|
||||||
|
if c["lens_id"] == "he_logos_v1"
|
||||||
|
and c["anchor_lens_mode_label"] == "covenant-verity"
|
||||||
|
]
|
||||||
|
# grc engages on knowledge in every register (3 cells); same for he on truth.
|
||||||
|
assert len(grc_engaged) == len(_REGISTERS), (
|
||||||
|
f"expected grc_logos_v1 engagement in {len(_REGISTERS)} cells "
|
||||||
|
f"(once per register on 'What is knowledge?'), got {len(grc_engaged)}"
|
||||||
|
)
|
||||||
|
assert len(he_engaged) == len(_REGISTERS), (
|
||||||
|
f"expected he_logos_v1 engagement in {len(_REGISTERS)} cells "
|
||||||
|
f"(once per register on 'What is truth?'), got {len(he_engaged)}"
|
||||||
|
)
|
||||||
Loading…
Reference in a new issue