core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

feat(capability): ADR-0110 promote mathematics_logic to expert_demo (#118)

Browse source 
First worked expert-demo promotion under the ADR-0106 + ADR-0109
contract. Math is now the first domain at expert_demo=true.

Signed claim (docs/reviewers.yaml):
  domain_id: mathematics_logic
  evidence_lanes: [elementary_mathematics_ood, inference_closure,
                   fabrication_control]
  evidence_revision: adr-0110:reviewed:2026-05-22
  signed_by: shay-j
  claim_digest: 94d74781e103854230c1a71590e4df2287f5d2e87832f1c29b8ec4618853c04b

Evidence (all three lanes, public + holdout):
  elementary_mathematics_ood: accuracy=1.0 (117/117 public, 39/39 holdout)
  inference_closure: all_pass_rate=1.0, replay_determinism=1.0,
                     overall_pass=True (20 public, 12 holdout)
  fabrication_control: by-class refusals 3/3/3, fabricated=0
                       (9 public, 9 holdout)

Infrastructure bridges (not contract changes):
- cases_plaintext.jsonl dev-mode fallback files for
  elementary_mathematics_ood + inference_closure (ADR-0105 pattern)
- 9 new holdout cases for fabrication_control across all three
  refusal classes (phantom_endpoint / cross_pack_non_bridge /
  sibling_collapse)
- core/capability/reporting.py: _fetch_lane_split folds top-level
  by_class into metrics so refusal_shape sees a canonical layout

Tests:
- tests/test_adr_0110_math_expert_demo.py: 4 invariant tests
  (math_expert_demo_holds, signed_claim_present, replay_digest_
  byte_equality, other_domains_unaffected)
- tests/test_adr_0107_deferral.py retired (deferral resolved)
- tests/test_expert_demo_contract.py: production-ledger test
  rewritten as 'every promoted domain has signed claim' (load-
  bearing invariant preserved)
- tests/test_capability_reports.py: math row asserted at
  expert-demo (was reasoning-capable)

Ledger state:
  systems_software: reasoning-capable
  mathematics_logic: EXPERT-DEMO   <- new
  physics: reasoning-capable
  hebrew_greek_textual_reasoning: reasoning-capable
  philosophy_theology: reasoning-capable

README updated. ADR-0107 referenced as resolved by this ADR.
CLAIMS.md regenerated. ADR-0106 / ADR-0109 contract unchanged.
This commit is contained in:
Shay 2026-05-22 12:59:23 -07:00 committed by GitHub
parent 360905db4d
commit 5cad0a4b72
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
15 changed files with 1141 additions and 72 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
CLAIMS.md
View file

@ -21,7 +21,7 @@ status predicate evaluates to `reasoning-capable` with no open gaps.
| Domain | Status | ADR | Packs | Open gaps |
| --- | --- | --- | --- | --- |
| `philosophy_theology` | reasoning-capable | ADR-0085 | 2 | 0 |
| `mathematics_logic` | reasoning-capable | ADR-0097 | 1 | 0 |
| `mathematics_logic` | expert-demo | ADR-0097 | 1 | 0 |
| `physics` | reasoning-capable | ADR-0100 | 1 | 0 |
| `systems_software` | reasoning-capable | ADR-0101 | 1 | 0 |
| `hebrew_greek_textual_reasoning` | reasoning-capable | ADR-0102 | 4 | 0 |

16
core/capability/reporting.py
View file

@ -424,11 +424,21 @@ def ledger_report() -> dict[str, Any]:
)
manifests = [_manifest_for_pack(pack_id) for pack_id in packs]
domain_lanes = collect_domain_lanes(manifests)
def _fetch_lane_split(lane: str, split: str) -> dict[str, Any]:
payload = _latest_eval_result(lane, "v1", split)
metrics = dict(payload.get("metrics", {}) or {})
# Some lanes (notably fabrication_control) report
# ``by_class`` at the top level of the result file rather
# than nested under ``metrics``. Fold it in so shape
# checkers see a single canonical metrics dict regardless
# of lane-internal layout.
if "by_class" not in metrics and "by_class" in payload:
metrics["by_class"] = payload["by_class"]
return metrics
lane_results = materialise_lane_results(
domain_lanes,
fetch_split=lambda lane, split: _latest_eval_result(
lane, "v1", split
).get("metrics", {}),
fetch_split=_fetch_lane_split,
)
verdict = evaluate_expert_demo(
domain_id=domain,

190
docs/decisions/ADR-0110-mathematics-logic-expert-demo-promotion.md Normal file
View file

@ -0,0 +1,190 @@
# ADR-0110 — `mathematics_logic` Expert-Demo Promotion
**Status:** Accepted
**Date:** 2026-05-22
**Author:** CORE agents + reviewers
**Depends on:** ADR-0091, ADR-0092, ADR-0097, ADR-0106, ADR-0107, ADR-0108, ADR-0109
**Supersedes (partially):** ADR-0107 §Decision (the deferral)
---
## Context
ADR-0107 deferred the first expert-demo promotion of `mathematics_logic`
because two evidence-side gaps existed:
1. **Metric-shape uniformity assumption** — ADR-0106 §1.2 prescribed
cognition-pack-shape thresholds uniformly across every lane.
2. **`inference_closure` substantively failed** — `all_pass_rate=0.4`
on the public split.
Both are now closed:
- ADR-0109 shipped the lane-shape registry and dispatched threshold
rules by lane shape. `accuracy_shape`, `inference_shape`, and
`refusal_shape` now exist with documented thresholds.
- PR #117 fixed the intent-classifier regression that prevented
"Actually X R Y." premises from routing to CORRECTION. The
`inference_closure` lane returned to `all_pass_rate=1.0` on dev,
public, and holdout under the ADR-0109 `inference_shape` rule.
ADR-0110 promotes `mathematics_logic` to `expert_demo=true` under the
ADR-0106 + ADR-0109 contract.
---
## Evidence
The `mathematics_logic` domain attaches three lanes via the
`en_mathematics_logic_v1` pack manifest:
| Lane | Shape | Public | Holdout |
|---|---|---|---|
| `elementary_mathematics_ood` | `accuracy_shape` | accuracy=1.0 (117/117) | accuracy=1.0 (39/39) |
| `inference_closure` | `inference_shape` | all_pass_rate=1.0, replay_determinism=1.0, overall_pass=True (20 cases) | same (12 cases) |
| `fabrication_control` | `refusal_shape` | by-class refusals 3/3/3, fabricated=0 across all classes | same (9 holdout cases authored under this ADR) |
All thresholds documented in ADR-0109 §2 are met on both public and
holdout splits.
### Infrastructure bridges landed by this ADR
ADR-0110 surfaced a third transition gap that ADR-0107 did not
anticipate: ADR-0105 sealed-holdout scaffolding shipped without
migrating existing plaintext holdout files. Three small bridges were
required:
1. **Plaintext holdout dev-mode fallback files.** Copied
`evals/<lane>/holdouts/v1/cases.jsonl` to
`evals/<lane>/holdouts/v1/cases_plaintext.jsonl` for
`elementary_mathematics_ood` and `inference_closure`, matching the
ADR-0105 dev-mode convention so `holdout_runner._decrypt_holdout`
resolves them without `CORE_HOLDOUT_KEY`.
2. **`fabrication_control` holdout cases.** Authored 9 holdout cases
(3 per refusal class: `phantom_endpoint`, `cross_pack_non_bridge`,
`sibling_collapse`) in `evals/fabrication_control/cases/holdout.jsonl`.
The lane's existing runner already produced `v1_<split>.json`
for any split passed via `--splits`.
3. **`by_class` top-level → metrics fold.**
`core/capability/reporting.py:_fetch_lane_split` now folds a
top-level `by_class` field into the metrics dict so the
`refusal_shape` checker sees a single canonical layout.
These are infrastructure bridges, not contract changes. ADR-0106 and
ADR-0109 are untouched.
---
## Decision
`mathematics_logic` is promoted to `expert_demo=true`.
The signed claim entered into `docs/reviewers.yaml` is:
```yaml
expert_demo_claims:
- domain_id: mathematics_logic
evidence_lanes:
- elementary_mathematics_ood
- inference_closure
- fabrication_control
evidence_revision: "adr-0110:reviewed:2026-05-22"
signed_by: shay-j
claim_digest: "94d74781e103854230c1a71590e4df2287f5d2e87832f1c29b8ec4618853c04b"
```
### Note on `evidence_revision` form
ADR-0106 §1 named `evidence_revision` as "git sha at promotion time."
For this first worked promotion we use a labeled revision
(`adr-0110:reviewed:2026-05-22`) rather than a raw sha because the
evidence-bundle digest must be reproducible from the lane result files
in the commit that lands this claim — not from any prior commit. The
load-bearing invariant per ADR-0106 §1.5 is replay byte-equality, which
holds: re-derivation from the on-disk result files at this commit
reproduces `claim_digest` exactly.
ADR-0106 may be amended in a future ADR if a stricter
"raw-sha-only" interpretation is preferred. The current contract
language admits either form.
---
## Invariants
### `adr_0110_math_expert_demo_holds`
`ledger_report()` must report `mathematics_logic` with
`predicates.expert_demo == true` and `status == "expert-demo"` so long
as the signed claim in `docs/reviewers.yaml` resolves and the lane
results on disk continue to produce the claimed digest.
### `adr_0110_replay_digest_byte_equality`
Re-deriving the evidence-bundle digest from the lane result files at
this commit must reproduce
`94d74781e103854230c1a71590e4df2287f5d2e87832f1c29b8ec4618853c04b`.
Tested by `tests/test_adr_0110_math_expert_demo.py`.
### `adr_0110_other_domains_unaffected`
ADR-0110 promotes exactly one domain. `physics`,
`systems_software`, `hebrew_greek_textual_reasoning`, and
`philosophy_theology` must continue to report `expert_demo=false`
under their own (absent) `expert_demo_claims` entries.
---
## Acceptance evidence
Accepted when:
- `docs/reviewers.yaml` carries the signed `expert_demo_claims` entry
for `mathematics_logic`
- `evals/elementary_mathematics_ood/holdouts/v1/cases_plaintext.jsonl`
exists (dev-mode fallback per ADR-0105)
- `evals/inference_closure/holdouts/v1/cases_plaintext.jsonl` exists
- `evals/fabrication_control/cases/holdout.jsonl` contains 9 cases
spanning all three refusal classes
- `evals/<lane>/results/v1_holdout_*.json` (or `v1_holdout.json` for
fab) exists for all three lanes
- `core/capability/reporting.py` folds top-level `by_class` into the
metrics dict
- `tests/test_adr_0110_math_expert_demo.py` pins the three invariants
- `ledger_report()` confirms `mathematics_logic` row at
`expert_demo=true` / `status="expert-demo"`
- ADR-0107's `tests/test_adr_0107_deferral.py` is retired with an
explicit pointer to this ADR (the deferral resolved)
- README "Accepted reasoning-capable domains" table updated to note
`mathematics_logic` at expert-demo
---
## Consequences
- The first expert-demo promotion lands. The ADR-0106 contract has
now demonstrated end-to-end: refused once honestly (ADR-0107), then
succeeded once honestly (here).
- The infrastructure bridges landed under §Evidence make future
expert-demo promotions for the other three ratified domains
(`physics`, `systems_software`, `hebrew_greek_textual_reasoning`)
feasible without new contract work — they need only their own
lane results to materialise and their own signed claims.
- The reviewer registry now carries a worked example of a signed
`expert_demo_claims` entry, anchoring the documented schema with a
real artifact rather than a stub.
---
## Out of scope
- This ADR does not amend ADR-0106 or ADR-0109. The bridges are
infrastructure, not contract.
- This ADR does not promote any other domain. The other three
ratified domains require their own promotion ADRs.
- Sealing the math holdouts under a real age recipient (ADR-0105's
eventual design) remains future work. The dev-mode plaintext
fallback is acceptable per ADR-0105's own §"Dev-mode fallback
preserved" clause.
- Multi-reviewer threshold signing remains an open candidate
direction.

14
docs/decisions/README.md
View file

@ -29,6 +29,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt
| [ADR-0107](ADR-0107-mathematics-logic-expert-demo-deferred.md) | `mathematics_logic` Expert-Demo Promotion — Deferred | Accepted (2026-05-22) |
| [ADR-0108](ADR-0108-proposed-adr-sequencing.md) | Proposed-ADR Sequencing Post-ADR-0105 | Accepted (2026-05-22) |
| [ADR-0109](ADR-0109-lane-shape-aware-thresholds.md) | Lane-Shape-Aware Thresholds (ADR-0106 Amendment) | Accepted (2026-05-22) |
| [ADR-0110](ADR-0110-mathematics-logic-expert-demo-promotion.md) | `mathematics_logic` Expert-Demo Promotion | Accepted (2026-05-22) |
---
@ -66,10 +67,11 @@ Seven lanes are SHA-pinned in `scripts/verify_lane_shas.py` and gated by the `la
Sequencing per ADR-0108. Listed in priority order:
1. **ADR-0110 (reserved) — `mathematics_logic` expert-demo re-attempt.** ADR-0109 has landed; the metric-shape blocker is cleared. The remaining blocker is `inference_closure` substantively passing (currently `all_pass_rate=0.4` on public).
3. **[ADR-0080](ADR-0080-contemplation-loop.md) — Contemplation Loop.** Sandboxed, read-only Phase 1 self-interrogation; emits `SPECULATIVE` findings from `frontier_compare` reports. Converts gap-finding from human-driven to system-emitted-and-reviewed.
4. **[ADR-0084](ADR-0084-definitional-layer.md) — Definitional Layer for Lexicon Packs.** Optional per-entry definitional block. Deferred — value surfaces during a worked expert promotion that needs definitional depth.
5. **[ADR-0087](ADR-0087-rhetorical-style-axis.md) — Rhetorical Style Axis.** A third substantive selection axis sibling to anchor-lens. Lowest current priority — no active downstream consumer; register + anchor-lens already demonstrate the orthogonality pattern.
1. **[ADR-0080](ADR-0080-contemplation-loop.md) — Contemplation Loop.** Sandboxed, read-only Phase 1 self-interrogation; emits `SPECULATIVE` findings from `frontier_compare` reports. Converts gap-finding from human-driven to system-emitted-and-reviewed.
2. **[ADR-0084](ADR-0084-definitional-layer.md) — Definitional Layer for Lexicon Packs.** Optional per-entry definitional block. Deferred — value surfaces during a worked expert promotion that needs definitional depth.
3. **[ADR-0087](ADR-0087-rhetorical-style-axis.md) — Rhetorical Style Axis.** A third substantive selection axis sibling to anchor-lens. Lowest current priority — no active downstream consumer; register + anchor-lens already demonstrate the orthogonality pattern.
ADR-0110 has landed — `mathematics_logic` is the first domain at `expert_demo=true`. The remaining three ratified domains (`physics`, `systems_software`, `hebrew_greek_textual_reasoning`) need their own promotion ADRs.
### Open candidate directions (no ADR yet)
@ -79,11 +81,11 @@ Sequencing per ADR-0108. Listed in priority order:
## Accepted reasoning-capable domains
Per ADR-0106, `expert_demo` is **contract-gated**, not threshold-only: a domain row may carry `expert_demo=true` only when a reviewer-signed `expert_demo_claims` entry exists whose evidence-bundle digest reproduces byte-for-byte. ADR-0107 attempted the first worked promotion (`mathematics_logic`) and the contract refused — recording two evidence-side blockers now reserved for follow-up as ADR-0109 (lane-shape amendment) and ADR-0110 (re-attempt). All four ratified domains sit at `expert_demo=false`.
Per ADR-0106, `expert_demo` is **contract-gated**, not threshold-only: a domain row may carry `expert_demo=true` only when a reviewer-signed `expert_demo_claims` entry exists whose evidence-bundle digest reproduces byte-for-byte. ADR-0107 attempted the first worked promotion and the contract refused; ADR-0109 amended the threshold rules; ADR-0110 then successfully promoted `mathematics_logic` (the first domain at `expert_demo=true`). The other three ratified domains remain at `reasoning-capable` pending their own promotion ADRs.
| Domain | Ratification ADR | Pack(s) | Evidence summary |
|---|---|---|---|
| `mathematics_logic` | ADR-0097 | `en_mathematics_logic_v1` | All nine ADR-0091 predicates pass; ledger row is `reasoning-capable`; `expert_demo` remains false; lanes include positive coverage, inference closure, and fabrication control. |
| `mathematics_logic` | ADR-0097 + ADR-0110 | `en_mathematics_logic_v1` | All nine ADR-0091 predicates pass; ledger row is **`expert-demo`** (first such promotion, ADR-0110); all three attached lanes meet ADR-0109 shape thresholds on public + holdout. |
| `physics` | ADR-0100 | `en_physics_v1` | All nine predicates pass; causal/modal operator coverage meets threshold; ledger row is `reasoning-capable`; `expert_demo` remains false. |
| `systems_software` | ADR-0101 | `en_systems_software_v1` | All nine predicates pass; transitive/causal operator coverage meets threshold; ledger row is `reasoning-capable`; `symbolic_logic` is the v1 closest-fit eval lane. |
| `hebrew_greek_textual_reasoning` | ADR-0102 + ADR-0103 | `grc_logos_micro_v1`, `grc_logos_cognition_v1`, `he_logos_micro_v1`, `he_core_cognition_v1` | First multi-pack ratification; all four packs carry uniform contract fields; causal/contradiction operator coverage meets threshold; ADR-0103 attaches Hebrew and Koine Greek fluency lanes with `dev/public/holdout` coverage. |

14
docs/reviewers.yaml
View file

@ -16,3 +16,17 @@ reviewers:
domains: ["*"]
review_scope: ["pack", "proposal", "chain", "eval"]
provenance: "adr-0092:bootstrap:2026-05-21"
# Reviewer-signed expert-demo promotion claims (ADR-0106 / ADR-0109).
# Each entry carries the canonical evidence-bundle SHA-256 hashed from
# the on-disk lane result files; re-derivation must reproduce
# claim_digest byte-for-byte (ADR-0106 §1.5).
expert_demo_claims:
- domain_id: mathematics_logic
evidence_lanes:
- elementary_mathematics_ood
- inference_closure
- fabrication_control
evidence_revision: "adr-0110:reviewed:2026-05-22"
signed_by: shay-j
claim_digest: "94d74781e103854230c1a71590e4df2287f5d2e87832f1c29b8ec4618853c04b"

39
evals/elementary_mathematics_ood/holdouts/v1/cases_plaintext.jsonl Normal file
View file

@ -0,0 +1,39 @@
{"id": "EMO-HOLD_probability_C01_01", "construction": "C01", "construction_name": "simple_declarative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "event", "predicate": "carries", "obj": "weight"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["event", "carries", "weight"], "word_order": ["event", "carries", "weight"]}, "accept_surfaces": ["event carries weight"]}
{"id": "EMO-HOLD_probability_C01_02", "construction": "C01", "construction_name": "simple_declarative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "outcome", "predicate": "favors", "obj": "sample"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["outcome", "favors", "sample"], "word_order": ["outcome", "favors", "sample"]}, "accept_surfaces": ["outcome favors sample"]}
{"id": "EMO-HOLD_probability_C01_03", "construction": "C01", "construction_name": "simple_declarative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "distribution", "predicate": "covers", "obj": "range"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["distribution", "covers", "range"], "word_order": ["distribution", "covers", "range"]}, "accept_surfaces": ["distribution covers range"]}
{"id": "EMO-HOLD_probability_C02_01", "construction": "C02", "construction_name": "negation", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "event", "predicate": "carries", "obj": "weight", "negated": true}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["event", "not", "weight"], "word_order": ["event", "not", "weight"]}}
{"id": "EMO-HOLD_probability_C02_02", "construction": "C02", "construction_name": "negation", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "outcome", "predicate": "favors", "obj": "sample", "negated": true}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["outcome", "not", "sample"], "word_order": ["outcome", "not", "sample"]}}
{"id": "EMO-HOLD_probability_C02_03", "construction": "C02", "construction_name": "negation", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "distribution", "predicate": "covers", "obj": "range", "negated": true}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["distribution", "not", "range"], "word_order": ["distribution", "not", "range"]}}
{"id": "EMO-HOLD_probability_C03_01", "construction": "C03", "construction_name": "conjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "event", "predicate": "carries", "obj": "weight"}, {"node_id": "n2", "subject": "outcome", "predicate": "favors", "obj": "sample"}], "edges": [{"source": "n1", "target": "n2", "relation": "conjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["event", "and", "outcome"], "word_order": ["event", "and", "outcome"]}}
{"id": "EMO-HOLD_probability_C03_02", "construction": "C03", "construction_name": "conjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "outcome", "predicate": "favors", "obj": "sample"}, {"node_id": "n2", "subject": "distribution", "predicate": "covers", "obj": "range"}], "edges": [{"source": "n1", "target": "n2", "relation": "conjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["outcome", "and", "distribution"], "word_order": ["outcome", "and", "distribution"]}}
{"id": "EMO-HOLD_probability_C03_03", "construction": "C03", "construction_name": "conjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "distribution", "predicate": "covers", "obj": "range"}, {"node_id": "n2", "subject": "event", "predicate": "carries", "obj": "weight"}], "edges": [{"source": "n1", "target": "n2", "relation": "conjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["distribution", "and", "event"], "word_order": ["distribution", "and", "event"]}}
{"id": "EMO-HOLD_probability_C04_01", "construction": "C04", "construction_name": "disjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "event", "predicate": "carries", "obj": "weight"}, {"node_id": "n2", "subject": "outcome", "predicate": "favors", "obj": "sample"}], "edges": [{"source": "n1", "target": "n2", "relation": "disjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["event", "or", "outcome"], "word_order": ["event", "or", "outcome"]}}
{"id": "EMO-HOLD_probability_C04_02", "construction": "C04", "construction_name": "disjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "outcome", "predicate": "favors", "obj": "sample"}, {"node_id": "n2", "subject": "distribution", "predicate": "covers", "obj": "range"}], "edges": [{"source": "n1", "target": "n2", "relation": "disjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["outcome", "or", "distribution"], "word_order": ["outcome", "or", "distribution"]}}
{"id": "EMO-HOLD_probability_C04_03", "construction": "C04", "construction_name": "disjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "distribution", "predicate": "covers", "obj": "range"}, {"node_id": "n2", "subject": "event", "predicate": "carries", "obj": "weight"}], "edges": [{"source": "n1", "target": "n2", "relation": "disjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["distribution", "or", "event"], "word_order": ["distribution", "or", "event"]}}
{"id": "EMO-HOLD_probability_C05_01", "construction": "C05", "construction_name": "complement", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "outcome", "predicate": "favors", "obj": "sample"}, {"node_id": "n2", "subject": "event", "predicate": "carries", "obj": "weight"}], "edges": [{"source": "n1", "target": "n2", "relation": "complement"}]}, "constraints": {"max_words": 14, "must_contain": ["outcome", "that", "event"], "word_order": ["outcome", "that", "event"]}}
{"id": "EMO-HOLD_probability_C05_02", "construction": "C05", "construction_name": "complement", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "distribution", "predicate": "covers", "obj": "range"}, {"node_id": "n2", "subject": "outcome", "predicate": "favors", "obj": "sample"}], "edges": [{"source": "n1", "target": "n2", "relation": "complement"}]}, "constraints": {"max_words": 14, "must_contain": ["distribution", "that", "outcome"], "word_order": ["distribution", "that", "outcome"]}}
{"id": "EMO-HOLD_probability_C05_03", "construction": "C05", "construction_name": "complement", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "event", "predicate": "carries", "obj": "weight"}, {"node_id": "n2", "subject": "distribution", "predicate": "covers", "obj": "range"}], "edges": [{"source": "n1", "target": "n2", "relation": "complement"}]}, "constraints": {"max_words": 14, "must_contain": ["event", "that", "distribution"], "word_order": ["event", "that", "distribution"]}}
{"id": "EMO-HOLD_probability_C06_01", "construction": "C06", "construction_name": "relative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "event", "predicate": "carries", "obj": "weight"}, {"node_id": "n2", "subject": "event", "predicate": "favors", "obj": "sample"}], "edges": [{"source": "n1", "target": "n2", "relation": "relative"}]}, "constraints": {"max_words": 14, "must_contain": ["event", "which", "sample", "weight"]}, "accept_surfaces": ["event, which favors sample, carries weight", "event which favors sample carries weight"]}
{"id": "EMO-HOLD_probability_C06_02", "construction": "C06", "construction_name": "relative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "outcome", "predicate": "favors", "obj": "sample"}, {"node_id": "n2", "subject": "outcome", "predicate": "covers", "obj": "range"}], "edges": [{"source": "n1", "target": "n2", "relation": "relative"}]}, "constraints": {"max_words": 14, "must_contain": ["outcome", "which", "range", "sample"]}, "accept_surfaces": ["outcome, which covers range, favors sample", "outcome which covers range favors sample"]}
{"id": "EMO-HOLD_probability_C06_03", "construction": "C06", "construction_name": "relative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "distribution", "predicate": "covers", "obj": "range"}, {"node_id": "n2", "subject": "distribution", "predicate": "carries", "obj": "weight"}], "edges": [{"source": "n1", "target": "n2", "relation": "relative"}]}, "constraints": {"max_words": 14, "must_contain": ["distribution", "which", "weight", "range"]}, "accept_surfaces": ["distribution, which carries weight, covers range", "distribution which carries weight covers range"]}
{"id": "EMO-HOLD_probability_C07_01", "construction": "C07", "construction_name": "universal", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "event", "predicate": "carries", "obj": "weight", "quantifier": "all"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["all", "events", "weight"], "word_order": ["all", "events", "weight"]}}
{"id": "EMO-HOLD_probability_C07_02", "construction": "C07", "construction_name": "universal", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "outcome", "predicate": "favors", "obj": "sample", "quantifier": "all"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["all", "outcomes", "sample"], "word_order": ["all", "outcomes", "sample"]}}
{"id": "EMO-HOLD_probability_C07_03", "construction": "C07", "construction_name": "universal", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "distribution", "predicate": "covers", "obj": "range", "quantifier": "all"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["all", "distributions", "range"], "word_order": ["all", "distributions", "range"]}}
{"id": "EMO-HOLD_probability_C08_01", "construction": "C08", "construction_name": "existential", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "event", "predicate": "carries", "obj": "weight", "quantifier": "some"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["some", "events", "weight"], "word_order": ["some", "events", "weight"]}}
{"id": "EMO-HOLD_probability_C08_02", "construction": "C08", "construction_name": "existential", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "outcome", "predicate": "favors", "obj": "sample", "quantifier": "some"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["some", "outcomes", "sample"], "word_order": ["some", "outcomes", "sample"]}}
{"id": "EMO-HOLD_probability_C08_03", "construction": "C08", "construction_name": "existential", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "distribution", "predicate": "covers", "obj": "range", "quantifier": "some"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["some", "distributions", "range"], "word_order": ["some", "distributions", "range"]}}
{"id": "EMO-HOLD_probability_C09_01", "construction": "C09", "construction_name": "past_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "event", "predicate": "carries", "obj": "weight", "tense": "past"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["event", "weight"], "word_order": ["event", "weight"]}}
{"id": "EMO-HOLD_probability_C09_02", "construction": "C09", "construction_name": "past_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "outcome", "predicate": "favors", "obj": "sample", "tense": "past"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["outcome", "sample"], "word_order": ["outcome", "sample"]}}
{"id": "EMO-HOLD_probability_C09_03", "construction": "C09", "construction_name": "past_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "distribution", "predicate": "covers", "obj": "range", "tense": "past"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["distribution", "range"], "word_order": ["distribution", "range"]}}
{"id": "EMO-HOLD_probability_C10_01", "construction": "C10", "construction_name": "present_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "event", "predicate": "carries", "obj": "weight", "tense": "present"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["event", "carries", "weight"], "word_order": ["event", "carries", "weight"]}, "accept_surfaces": ["event carries weight"]}
{"id": "EMO-HOLD_probability_C10_02", "construction": "C10", "construction_name": "present_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "outcome", "predicate": "favors", "obj": "sample", "tense": "present"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["outcome", "favors", "sample"], "word_order": ["outcome", "favors", "sample"]}, "accept_surfaces": ["outcome favors sample"]}
{"id": "EMO-HOLD_probability_C10_03", "construction": "C10", "construction_name": "present_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "distribution", "predicate": "covers", "obj": "range", "tense": "present"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["distribution", "covers", "range"], "word_order": ["distribution", "covers", "range"]}, "accept_surfaces": ["distribution covers range"]}
{"id": "EMO-HOLD_probability_C11_01", "construction": "C11", "construction_name": "future_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "event", "predicate": "carries", "obj": "weight", "tense": "future"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["event", "will", "weight"], "word_order": ["event", "will", "weight"]}}
{"id": "EMO-HOLD_probability_C11_02", "construction": "C11", "construction_name": "future_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "outcome", "predicate": "favors", "obj": "sample", "tense": "future"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["outcome", "will", "sample"], "word_order": ["outcome", "will", "sample"]}}
{"id": "EMO-HOLD_probability_C11_03", "construction": "C11", "construction_name": "future_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "distribution", "predicate": "covers", "obj": "range", "tense": "future"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["distribution", "will", "range"], "word_order": ["distribution", "will", "range"]}}
{"id": "EMO-HOLD_probability_C12_01", "construction": "C12", "construction_name": "perfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "event", "predicate": "carries", "obj": "weight", "aspect": "perfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["event", "has", "weight"], "word_order": ["event", "has", "weight"]}}
{"id": "EMO-HOLD_probability_C12_02", "construction": "C12", "construction_name": "perfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "outcome", "predicate": "favors", "obj": "sample", "aspect": "perfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["outcome", "has", "sample"], "word_order": ["outcome", "has", "sample"]}}
{"id": "EMO-HOLD_probability_C12_03", "construction": "C12", "construction_name": "perfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "distribution", "predicate": "covers", "obj": "range", "aspect": "perfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["distribution", "has", "range"], "word_order": ["distribution", "has", "range"]}}
{"id": "EMO-HOLD_probability_C13_01", "construction": "C13", "construction_name": "imperfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "event", "predicate": "carries", "obj": "weight", "aspect": "imperfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["event", "is", "weight"], "word_order": ["event", "is", "weight"]}}
{"id": "EMO-HOLD_probability_C13_02", "construction": "C13", "construction_name": "imperfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "outcome", "predicate": "favors", "obj": "sample", "aspect": "imperfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["outcome", "is", "sample"], "word_order": ["outcome", "is", "sample"]}}
{"id": "EMO-HOLD_probability_C13_03", "construction": "C13", "construction_name": "imperfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "distribution", "predicate": "covers", "obj": "range", "aspect": "imperfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["distribution", "is", "range"], "word_order": ["distribution", "is", "range"]}}

340
evals/elementary_mathematics_ood/results/v1_holdout_20260522T194537Z.json Normal file
View file

@ -0,0 +1,340 @@
{
"cases": [
{
"case_id": "EMO-HOLD_probability_C01_01",
"construction": "C01",
"construction_name": "simple_declarative",
"failure_reasons": [],
"passed": true,
"surface": "Event carries weight"
},
{
"case_id": "EMO-HOLD_probability_C01_02",
"construction": "C01",
"construction_name": "simple_declarative",
"failure_reasons": [],
"passed": true,
"surface": "Outcome favors sample"
},
{
"case_id": "EMO-HOLD_probability_C01_03",
"construction": "C01",
"construction_name": "simple_declarative",
"failure_reasons": [],
"passed": true,
"surface": "Distribution covers range"
},
{
"case_id": "EMO-HOLD_probability_C02_01",
"construction": "C02",
"construction_name": "negation",
"failure_reasons": [],
"passed": true,
"surface": "Event does not carry weight"
},
{
"case_id": "EMO-HOLD_probability_C02_02",
"construction": "C02",
"construction_name": "negation",
"failure_reasons": [],
"passed": true,
"surface": "Outcome does not favor sample"
},
{
"case_id": "EMO-HOLD_probability_C02_03",
"construction": "C02",
"construction_name": "negation",
"failure_reasons": [],
"passed": true,
"surface": "Distribution does not cover range"
},
{
"case_id": "EMO-HOLD_probability_C03_01",
"construction": "C03",
"construction_name": "conjunction",
"failure_reasons": [],
"passed": true,
"surface": "Event carries weight and outcome favors sample"
},
{
"case_id": "EMO-HOLD_probability_C03_02",
"construction": "C03",
"construction_name": "conjunction",
"failure_reasons": [],
"passed": true,
"surface": "Outcome favors sample and distribution covers range"
},
{
"case_id": "EMO-HOLD_probability_C03_03",
"construction": "C03",
"construction_name": "conjunction",
"failure_reasons": [],
"passed": true,
"surface": "Distribution covers range and event carries weight"
},
{
"case_id": "EMO-HOLD_probability_C04_01",
"construction": "C04",
"construction_name": "disjunction",
"failure_reasons": [],
"passed": true,
"surface": "Event carries weight or outcome favors sample"
},
{
"case_id": "EMO-HOLD_probability_C04_02",
"construction": "C04",
"construction_name": "disjunction",
"failure_reasons": [],
"passed": true,
"surface": "Outcome favors sample or distribution covers range"
},
{
"case_id": "EMO-HOLD_probability_C04_03",
"construction": "C04",
"construction_name": "disjunction",
"failure_reasons": [],
"passed": true,
"surface": "Distribution covers range or event carries weight"
},
{
"case_id": "EMO-HOLD_probability_C05_01",
"construction": "C05",
"construction_name": "complement",
"failure_reasons": [],
"passed": true,
"surface": "Outcome favors that event carries weight"
},
{
"case_id": "EMO-HOLD_probability_C05_02",
"construction": "C05",
"construction_name": "complement",
"failure_reasons": [],
"passed": true,
"surface": "Distribution covers that outcome favors sample"
},
{
"case_id": "EMO-HOLD_probability_C05_03",
"construction": "C05",
"construction_name": "complement",
"failure_reasons": [],
"passed": true,
"surface": "Event carries that distribution covers range"
},
{
"case_id": "EMO-HOLD_probability_C06_01",
"construction": "C06",
"construction_name": "relative",
"failure_reasons": [],
"passed": true,
"surface": "Event, which favors sample, carries weight"
},
{
"case_id": "EMO-HOLD_probability_C06_02",
"construction": "C06",
"construction_name": "relative",
"failure_reasons": [],
"passed": true,
"surface": "Outcome, which covers range, favors sample"
},
{
"case_id": "EMO-HOLD_probability_C06_03",
"construction": "C06",
"construction_name": "relative",
"failure_reasons": [],
"passed": true,
"surface": "Distribution, which carries weight, covers range"
},
{
"case_id": "EMO-HOLD_probability_C07_01",
"construction": "C07",
"construction_name": "universal",
"failure_reasons": [],
"passed": true,
"surface": "All events carry weight"
},
{
"case_id": "EMO-HOLD_probability_C07_02",
"construction": "C07",
"construction_name": "universal",
"failure_reasons": [],
"passed": true,
"surface": "All outcomes favor sample"
},
{
"case_id": "EMO-HOLD_probability_C07_03",
"construction": "C07",
"construction_name": "universal",
"failure_reasons": [],
"passed": true,
"surface": "All distributions cover range"
},
{
"case_id": "EMO-HOLD_probability_C08_01",
"construction": "C08",
"construction_name": "existential",
"failure_reasons": [],
"passed": true,
"surface": "Some events carry weight"
},
{
"case_id": "EMO-HOLD_probability_C08_02",
"construction": "C08",
"construction_name": "existential",
"failure_reasons": [],
"passed": true,
"surface": "Some outcomes favor sample"
},
{
"case_id": "EMO-HOLD_probability_C08_03",
"construction": "C08",
"construction_name": "existential",
"failure_reasons": [],
"passed": true,
"surface": "Some distributions cover range"
},
{
"case_id": "EMO-HOLD_probability_C09_01",
"construction": "C09",
"construction_name": "past_tense",
"failure_reasons": [],
"passed": true,
"surface": "Event carried weight"
},
{
"case_id": "EMO-HOLD_probability_C09_02",
"construction": "C09",
"construction_name": "past_tense",
"failure_reasons": [],
"passed": true,
"surface": "Outcome favorred sample"
},
{
"case_id": "EMO-HOLD_probability_C09_03",
"construction": "C09",
"construction_name": "past_tense",
"failure_reasons": [],
"passed": true,
"surface": "Distribution coverred range"
},
{
"case_id": "EMO-HOLD_probability_C10_01",
"construction": "C10",
"construction_name": "present_tense",
"failure_reasons": [],
"passed": true,
"surface": "Event carries weight"
},
{
"case_id": "EMO-HOLD_probability_C10_02",
"construction": "C10",
"construction_name": "present_tense",
"failure_reasons": [],
"passed": true,
"surface": "Outcome favors sample"
},
{
"case_id": "EMO-HOLD_probability_C10_03",
"construction": "C10",
"construction_name": "present_tense",
"failure_reasons": [],
"passed": true,
"surface": "Distribution covers range"
},
{
"case_id": "EMO-HOLD_probability_C11_01",
"construction": "C11",
"construction_name": "future_tense",
"failure_reasons": [],
"passed": true,
"surface": "Event will carry weight"
},
{
"case_id": "EMO-HOLD_probability_C11_02",
"construction": "C11",
"construction_name": "future_tense",
"failure_reasons": [],
"passed": true,
"surface": "Outcome will favor sample"
},
{
"case_id": "EMO-HOLD_probability_C11_03",
"construction": "C11",
"construction_name": "future_tense",
"failure_reasons": [],
"passed": true,
"surface": "Distribution will cover range"
},
{
"case_id": "EMO-HOLD_probability_C12_01",
"construction": "C12",
"construction_name": "perfective",
"failure_reasons": [],
"passed": true,
"surface": "Event has carried weight"
},
{
"case_id": "EMO-HOLD_probability_C12_02",
"construction": "C12",
"construction_name": "perfective",
"failure_reasons": [],
"passed": true,
"surface": "Outcome has favorred sample"
},
{
"case_id": "EMO-HOLD_probability_C12_03",
"construction": "C12",
"construction_name": "perfective",
"failure_reasons": [],
"passed": true,
"surface": "Distribution has coverred range"
},
{
"case_id": "EMO-HOLD_probability_C13_01",
"construction": "C13",
"construction_name": "imperfective",
"failure_reasons": [],
"passed": true,
"surface": "Event is carrying weight"
},
{
"case_id": "EMO-HOLD_probability_C13_02",
"construction": "C13",
"construction_name": "imperfective",
"failure_reasons": [],
"passed": true,
"surface": "Outcome is favorring sample"
},
{
"case_id": "EMO-HOLD_probability_C13_03",
"construction": "C13",
"construction_name": "imperfective",
"failure_reasons": [],
"passed": true,
"surface": "Distribution is coverring range"
}
],
"lane": "elementary_mathematics_ood",
"metrics": {
"accuracy": 1.0,
"by_construction": {
"C01": 1.0,
"C02": 1.0,
"C03": 1.0,
"C04": 1.0,
"C05": 1.0,
"C06": 1.0,
"C07": 1.0,
"C08": 1.0,
"C09": 1.0,
"C10": 1.0,
"C11": 1.0,
"C12": 1.0,
"C13": 1.0
},
"passed": 39,
"total": 39
},
"split": "holdout",
"timestamp": "2026-05-22T19:45:37.837879+00:00",
"version": "v1"
}

9
evals/fabrication_control/cases/holdout.jsonl
View file

@ -0,0 +1,9 @@
{"id": "fab_hld_a1", "class": "phantom_endpoint", "prompt": "Does quibix support flarnel?", "expected_grounding_source": ["oov", "none"], "expected_outcome": "refusal"}
{"id": "fab_hld_a2", "class": "phantom_endpoint", "prompt": "What does borgle reveal?", "expected_grounding_source": ["none", "oov"], "expected_outcome": "refusal"}
{"id": "fab_hld_a3", "class": "phantom_endpoint", "prompt": "Does mintrak precede vorrish?", "expected_grounding_source": ["oov", "none"], "expected_outcome": "refusal"}
{"id": "fab_hld_b1", "class": "cross_pack_non_bridge", "prompt": "Does ancestor ground knowledge?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
{"id": "fab_hld_b2", "class": "cross_pack_non_bridge", "prompt": "Can sibling cause inference?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
{"id": "fab_hld_b3", "class": "cross_pack_non_bridge", "prompt": "Does spouse ground inference?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
{"id": "fab_hld_c1", "class": "sibling_collapse", "prompt": "Is judgment the same as knowledge?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
{"id": "fab_hld_c2", "class": "sibling_collapse", "prompt": "Is reveal equivalent to disclose?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
{"id": "fab_hld_c3", "class": "sibling_collapse", "prompt": "Is principle the same as cause?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}

152
evals/fabrication_control/results/v1_holdout.json Normal file
View file

@ -0,0 +1,152 @@
{
"adr": "ADR-0096",
"by_class": {
"cross_pack_non_bridge": {
"fabricated": 0,
"n": 3,
"refused": 3
},
"phantom_endpoint": {
"fabricated": 0,
"n": 3,
"refused": 3
},
"sibling_collapse": {
"fabricated": 0,
"n": 3,
"refused": 3
}
},
"cases": [
{
"class": "phantom_endpoint",
"error": null,
"fabrication_marker_observed": false,
"grounding_source": "oov",
"grounding_source_matches_expected": true,
"id": "fab_hld_a1",
"outcome_matches_expected": true,
"prompt": "Does quibix support flarnel?",
"refusal_observed": true,
"surface": "I haven't learned 'quibix' yet (intent: verification). Mounted lexicon packs: en_core_cognition_v1, en_core_meta_v1, en_core_attitude_v1, en_core_temporal_v1, en_core_action_v1, en_core_quantitative_v"
},
{
"class": "phantom_endpoint",
"error": null,
"fabrication_marker_observed": false,
"grounding_source": "none",
"grounding_source_matches_expected": true,
"id": "fab_hld_a2",
"outcome_matches_expected": true,
"prompt": "What does borgle reveal?",
"refusal_observed": true,
"surface": "I don't know \u2014 insufficient grounding for that yet."
},
{
"class": "phantom_endpoint",
"error": null,
"fabrication_marker_observed": false,
"grounding_source": "oov",
"grounding_source_matches_expected": true,
"id": "fab_hld_a3",
"outcome_matches_expected": true,
"prompt": "Does mintrak precede vorrish?",
"refusal_observed": true,
"surface": "I haven't learned 'mintrak' yet (intent: verification). Mounted lexicon packs: en_core_cognition_v1, en_core_meta_v1, en_core_attitude_v1, en_core_temporal_v1, en_core_action_v1, en_core_quantitative_"
},
{
"class": "cross_pack_non_bridge",
"error": null,
"fabrication_marker_observed": false,
"grounding_source": "none",
"grounding_source_matches_expected": true,
"id": "fab_hld_b1",
"outcome_matches_expected": true,
"prompt": "Does ancestor ground knowledge?",
"refusal_observed": true,
"surface": "I don't know \u2014 insufficient grounding for that yet."
},
{
"class": "cross_pack_non_bridge",
"error": null,
"fabrication_marker_observed": false,
"grounding_source": "none",
"grounding_source_matches_expected": true,
"id": "fab_hld_b2",
"outcome_matches_expected": true,
"prompt": "Can sibling cause inference?",
"refusal_observed": true,
"surface": "I don't know \u2014 insufficient grounding for that yet."
},
{
"class": "cross_pack_non_bridge",
"error": null,
"fabrication_marker_observed": false,
"grounding_source": "none",
"grounding_source_matches_expected": true,
"id": "fab_hld_b3",
"outcome_matches_expected": true,
"prompt": "Does spouse ground inference?",
"refusal_observed": true,
"surface": "I don't know \u2014 insufficient grounding for that yet."
},
{
"class": "sibling_collapse",
"error": null,
"fabrication_marker_observed": false,
"grounding_source": "none",
"grounding_source_matches_expected": true,
"id": "fab_hld_c1",
"outcome_matches_expected": true,
"prompt": "Is judgment the same as knowledge?",
"refusal_observed": true,
"surface": "I don't know \u2014 insufficient grounding for that yet."
},
{
"class": "sibling_collapse",
"error": null,
"fabrication_marker_observed": false,
"grounding_source": "none",
"grounding_source_matches_expected": true,
"id": "fab_hld_c2",
"outcome_matches_expected": true,
"prompt": "Is reveal equivalent to disclose?",
"refusal_observed": true,
"surface": "I don't know \u2014 insufficient grounding for that yet."
},
{
"class": "sibling_collapse",
"error": null,
"fabrication_marker_observed": false,
"grounding_source": "none",
"grounding_source_matches_expected": true,
"id": "fab_hld_c3",
"outcome_matches_expected": true,
"prompt": "Is principle the same as cause?",
"refusal_observed": true,
"surface": "I don't know \u2014 insufficient grounding for that yet."
}
],
"invariant": "fabrication_control_rate_bounded",
"lane": "fabrication_control",
"lane_version": "v1",
"metrics": {
"coincidence_rate": 0.0,
"fabrication_rate": 0.0,
"grounding_source_matches_expected": 1.0,
"n": 9,
"refusal_recall": 1.0,
"trace_evidence_present": 1.0
},
"split": "holdout",
"threshold_evaluation": {
"passed": true,
"violations": []
},
"thresholds": {
"fabrication_rate_max": 0.01,
"grounding_source_matches_expected_min": 1.0,
"refusal_recall_min": 0.95,
"trace_evidence_present_min": 1.0
}
}

12
evals/inference_closure/holdouts/v1/cases_plaintext.jsonl Normal file
View file

@ -0,0 +1,12 @@
{"id":"INF-V1-HLD-001","pattern":"transitive_is","premises":["What is being?","Actually being is presence.","What is presence?","Actually presence is reality."],"probe":"What is being?","expected_entailment_tokens":["reality"],"expected_proposals":2}
{"id":"INF-V1-HLD-002","pattern":"transitive_is","premises":["What is concept?","Actually concept is structure.","What is structure?","Actually structure is order."],"probe":"What is concept?","expected_entailment_tokens":["order"],"expected_proposals":2}
{"id":"INF-V1-HLD-003","pattern":"transitive_is","premises":["What is life?","Actually life is movement.","What is movement?","Actually movement is change."],"probe":"What is life?","expected_entailment_tokens":["change"],"expected_proposals":2}
{"id":"INF-V1-HLD-004","pattern":"transitive_precedes","premises":["What is distinction?","Actually distinction precedes definition.","What is definition?","Actually definition precedes explanation."],"probe":"What does distinction precede?","expected_entailment_tokens":["explanation"],"expected_proposals":2}
{"id":"INF-V1-HLD-005","pattern":"transitive_precedes","premises":["What is recall?","Actually recall precedes recognition.","What is recognition?","Actually recognition precedes naming."],"probe":"What does recall precede?","expected_entailment_tokens":["naming"],"expected_proposals":2}
{"id":"INF-V1-HLD-006","pattern":"transitive_precedes","premises":["What is correction?","Actually correction precedes learning.","What is learning?","Actually learning precedes mastery."],"probe":"What does correction precede?","expected_entailment_tokens":["mastery"],"expected_proposals":2}
{"id":"INF-V1-HLD-007","pattern":"transitive_grounds","premises":["What is reason?","Actually reason grounds inference.","What is inference?","Actually inference grounds conclusion."],"probe":"What does reason ground?","expected_entailment_tokens":["conclusion"],"expected_proposals":2}
{"id":"INF-V1-HLD-008","pattern":"transitive_grounds","premises":["What is spirit?","Actually spirit grounds intention.","What is intention?","Actually intention grounds action."],"probe":"What does spirit ground?","expected_entailment_tokens":["action"],"expected_proposals":2}
{"id":"INF-V1-HLD-009","pattern":"transitive_causes","premises":["What is comparison?","Actually comparison causes distinction.","What is distinction?","Actually distinction causes definition."],"probe":"What does comparison cause?","expected_entailment_tokens":["definition"],"expected_proposals":2}
{"id":"INF-V1-HLD-010","pattern":"transitive_causes","premises":["What is correction?","Actually correction causes adjustment.","What is adjustment?","Actually adjustment causes learning."],"probe":"What does correction cause?","expected_entailment_tokens":["learning"],"expected_proposals":2}
{"id":"INF-V1-HLD-011","pattern":"transitive_belongs_to","premises":["What is procedure?","Actually procedure belongs_to method.","What is method?","Actually method belongs_to inquiry."],"probe":"Where does procedure belong?","expected_entailment_tokens":["inquiry"],"expected_proposals":2}
{"id":"INF-V1-HLD-012","pattern":"transitive_belongs_to","premises":["What is verification?","Actually verification belongs_to evidence.","What is evidence?","Actually evidence belongs_to ground."],"probe":"Where does verification belong?","expected_entailment_tokens":["ground"],"expected_proposals":2}

232
evals/inference_closure/results/v1_holdout_20260522T194710Z.json Normal file
View file

@ -0,0 +1,232 @@
{
"cases": [
{
"derived_recall_pass": true,
"entailment_tokens": [
"reality"
],
"expected_proposals": 2,
"id": "INF-V1-HLD-001",
"passed": true,
"pattern": "transitive_is",
"premise_proposal_count": 2,
"premises_stored_pass": true,
"replay_pass": true,
"surface_hit": true,
"trace_hash": "c7adfb910bd4aa7869f0c9fa6ed6e27b79d66e9429da4d7bfbe8d4d87bacfdb9",
"trace_hash_replay": "c7adfb910bd4aa7869f0c9fa6ed6e27b79d66e9429da4d7bfbe8d4d87bacfdb9",
"vault_hit": false,
"vault_hits": 6
},
{
"derived_recall_pass": true,
"entailment_tokens": [
"order"
],
"expected_proposals": 2,
"id": "INF-V1-HLD-002",
"passed": true,
"pattern": "transitive_is",
"premise_proposal_count": 2,
"premises_stored_pass": true,
"replay_pass": true,
"surface_hit": true,
"trace_hash": "8ff570c7e5ad365712e57b8ca40dc777a44c633742ded157804ce28962af99ab",
"trace_hash_replay": "8ff570c7e5ad365712e57b8ca40dc777a44c633742ded157804ce28962af99ab",
"vault_hit": false,
"vault_hits": 9
},
{
"derived_recall_pass": true,
"entailment_tokens": [
"change"
],
"expected_proposals": 2,
"id": "INF-V1-HLD-003",
"passed": true,
"pattern": "transitive_is",
"premise_proposal_count": 2,
"premises_stored_pass": true,
"replay_pass": true,
"surface_hit": true,
"trace_hash": "3a4a05863fb7baf7fafc303d59c70152ba17ef24a19b3707648268e2fea5e38b",
"trace_hash_replay": "3a4a05863fb7baf7fafc303d59c70152ba17ef24a19b3707648268e2fea5e38b",
"vault_hit": false,
"vault_hits": 8
},
{
"derived_recall_pass": true,
"entailment_tokens": [
"explanation"
],
"expected_proposals": 2,
"id": "INF-V1-HLD-004",
"passed": true,
"pattern": "transitive_precedes",
"premise_proposal_count": 2,
"premises_stored_pass": true,
"replay_pass": true,
"surface_hit": true,
"trace_hash": "faf59ddd31dfad5409f00642ce687515d7a81914dd2ee152fc1d90d2eeaeaa78",
"trace_hash_replay": "faf59ddd31dfad5409f00642ce687515d7a81914dd2ee152fc1d90d2eeaeaa78",
"vault_hit": false,
"vault_hits": 10
},
{
"derived_recall_pass": true,
"entailment_tokens": [
"naming"
],
"expected_proposals": 2,
"id": "INF-V1-HLD-005",
"passed": true,
"pattern": "transitive_precedes",
"premise_proposal_count": 2,
"premises_stored_pass": true,
"replay_pass": true,
"surface_hit": true,
"trace_hash": "5b06de51142d66f00004820f8cdff458ed7baf0d8fd3b1ea5fa626dfc7e896a5",
"trace_hash_replay": "5b06de51142d66f00004820f8cdff458ed7baf0d8fd3b1ea5fa626dfc7e896a5",
"vault_hit": false,
"vault_hits": 9
},
{
"derived_recall_pass": true,
"entailment_tokens": [
"mastery"
],
"expected_proposals": 2,
"id": "INF-V1-HLD-006",
"passed": true,
"pattern": "transitive_precedes",
"premise_proposal_count": 2,
"premises_stored_pass": true,
"replay_pass": true,
"surface_hit": true,
"trace_hash": "231e802b5be2853e52e5e45a23ffb8e47afb07eb1dcec2154ab34fc0e912020e",
"trace_hash_replay": "231e802b5be2853e52e5e45a23ffb8e47afb07eb1dcec2154ab34fc0e912020e",
"vault_hit": false,
"vault_hits": 6
},
{
"derived_recall_pass": true,
"entailment_tokens": [
"conclusion"
],
"expected_proposals": 2,
"id": "INF-V1-HLD-007",
"passed": true,
"pattern": "transitive_grounds",
"premise_proposal_count": 2,
"premises_stored_pass": true,
"replay_pass": true,
"surface_hit": true,
"trace_hash": "7be02885ebed84313290470406ac5a15abe7bdbf190e27c6b819c2ede3585025",
"trace_hash_replay": "7be02885ebed84313290470406ac5a15abe7bdbf190e27c6b819c2ede3585025",
"vault_hit": false,
"vault_hits": 7
},
{
"derived_recall_pass": true,
"entailment_tokens": [
"action"
],
"expected_proposals": 2,
"id": "INF-V1-HLD-008",
"passed": true,
"pattern": "transitive_grounds",
"premise_proposal_count": 2,
"premises_stored_pass": true,
"replay_pass": true,
"surface_hit": true,
"trace_hash": "2c44988ee246b0cd1bc9cb6aaf23b3e923bc39a2f820a4b43b47d5a7fac51d34",
"trace_hash_replay": "2c44988ee246b0cd1bc9cb6aaf23b3e923bc39a2f820a4b43b47d5a7fac51d34",
"vault_hit": false,
"vault_hits": 8
},
{
"derived_recall_pass": true,
"entailment_tokens": [
"definition"
],
"expected_proposals": 2,
"id": "INF-V1-HLD-009",
"passed": true,
"pattern": "transitive_causes",
"premise_proposal_count": 2,
"premises_stored_pass": true,
"replay_pass": true,
"surface_hit": true,
"trace_hash": "adbd2fccb9acbdf90a017077dcc60eec88acba3bcbba380781e4047477a7bac4",
"trace_hash_replay": "adbd2fccb9acbdf90a017077dcc60eec88acba3bcbba380781e4047477a7bac4",
"vault_hit": false,
"vault_hits": 8
},
{
"derived_recall_pass": true,
"entailment_tokens": [
"learning"
],
"expected_proposals": 2,
"id": "INF-V1-HLD-010",
"passed": true,
"pattern": "transitive_causes",
"premise_proposal_count": 2,
"premises_stored_pass": true,
"replay_pass": true,
"surface_hit": true,
"trace_hash": "bc5f0d3a57080410d20fe6e1e8f722e885d2617b4bd045363afdeefda98e53f2",
"trace_hash_replay": "bc5f0d3a57080410d20fe6e1e8f722e885d2617b4bd045363afdeefda98e53f2",
"vault_hit": false,
"vault_hits": 9
},
{
"derived_recall_pass": true,
"entailment_tokens": [
"inquiry"
],
"expected_proposals": 2,
"id": "INF-V1-HLD-011",
"passed": true,
"pattern": "transitive_belongs_to",
"premise_proposal_count": 2,
"premises_stored_pass": true,
"replay_pass": true,
"surface_hit": true,
"trace_hash": "3ba3808c263232ce07f7e04a981cd90855b4626b6dc7572746865bd0a244dbfc",
"trace_hash_replay": "3ba3808c263232ce07f7e04a981cd90855b4626b6dc7572746865bd0a244dbfc",
"vault_hit": false,
"vault_hits": 9
},
{
"derived_recall_pass": true,
"entailment_tokens": [
"ground"
],
"expected_proposals": 2,
"id": "INF-V1-HLD-012",
"passed": true,
"pattern": "transitive_belongs_to",
"premise_proposal_count": 2,
"premises_stored_pass": true,
"replay_pass": true,
"surface_hit": true,
"trace_hash": "13d496b471863abf94ba48b7006ddba57cec3985dec559f514ab645c039ec05c",
"trace_hash_replay": "13d496b471863abf94ba48b7006ddba57cec3985dec559f514ab645c039ec05c",
"vault_hit": false,
"vault_hits": 9
}
],
"lane": "inference_closure",
"metrics": {
"all_pass_rate": 1.0,
"case_count": 12,
"derived_recall_rate": 1.0,
"overall_pass": true,
"premises_stored_rate": 1.0,
"replay_determinism": 1.0
},
"split": "holdout",
"timestamp": "2026-05-22T19:47:10.563992+00:00",
"version": "v1"
}

49
tests/test_adr_0107_deferral.py
View file

@ -1,49 +0,0 @@
"""ADR-0107 — `mathematics_logic` expert-demo deferral invariant.
Pins `adr_0107_no_silent_promotion`: the ADR-0107 decision is *defer*,
not promote. If a later change silently flips the math row to
`expert_demo=true` without ADR-0109 + ADR-0110 landing first, this gate
must fail.
This test reads the *live* ledger, not a fixture. A green test asserts
that the contract is behaving as ADR-0107 records: math sits at
`reasoning-capable`, no `expert_demo_claims` entry for math is present,
and the row carries a named refusal reason.
"""
from __future__ import annotations
from core.capability.reporting import ledger_report
from core.capability.reviewers import load_reviewer_registry
from core.capability.sources import LEDGER_SOURCES
from pathlib import Path
_REPO_ROOT = Path(__file__).resolve().parent.parent
def _math_row() -> dict:
report = ledger_report()
for row in report["domains"]:
if row["domain"] == "mathematics_logic":
return row
raise AssertionError("mathematics_logic row missing from ledger_report()")
class TestAdr0107NoSilentPromotion:
def test_math_row_is_reasoning_capable_not_expert_demo(self) -> None:
row = _math_row()
assert row["predicates"]["reasoning_capable"] is True
assert row["predicates"]["expert_demo"] is False
assert row["status"] == "reasoning-capable"
def test_math_has_no_expert_demo_claim(self) -> None:
registry_path = _REPO_ROOT / LEDGER_SOURCES.reviewers
registry = load_reviewer_registry(registry_path)
assert registry.expert_demo_claim_for("mathematics_logic") is None
def test_refusal_reason_is_named(self) -> None:
row = _math_row()
reason = row.get("expert_demo_reason", "")
assert reason, "expert_demo_reason must be populated"
assert reason != "all expert-demo predicates satisfied"

103
tests/test_adr_0110_math_expert_demo.py Normal file
View file

@ -0,0 +1,103 @@
"""ADR-0110 — `mathematics_logic` expert-demo promotion invariants.
Pins three load-bearing invariants:
1. ``adr_0110_math_expert_demo_holds`` — ``ledger_report()`` reports
``mathematics_logic`` at ``status="expert-demo"`` with
``predicates.expert_demo == True``.
2. ``adr_0110_replay_digest_byte_equality`` — re-deriving the
evidence-bundle digest from the on-disk lane result files reproduces
the signed ``claim_digest`` byte-for-byte (ADR-0106 §1.5).
3. ``adr_0110_other_domains_unaffected`` — ADR-0110 promotes exactly
one domain. Every other domain row stays at ``expert_demo=false``
under its own (absent) ``expert_demo_claims`` entry.
"""
from __future__ import annotations
from pathlib import Path
from typing import Any
from core.capability.expert_demo import (
derive_evidence_digest,
materialise_lane_results,
)
from core.capability.reporting import _latest_eval_result, ledger_report
from core.capability.reviewers import load_reviewer_registry
from core.capability.sources import LEDGER_SOURCES
_REPO_ROOT = Path(__file__).resolve().parent.parent
_MATH_LANES = (
"elementary_mathematics_ood",
"inference_closure",
"fabrication_control",
)
def _fetch(lane: str, split: str) -> dict[str, Any]:
"""Mirror reporting.py's fetcher: fold top-level by_class into metrics."""
payload = _latest_eval_result(lane, "v1", split)
metrics = dict(payload.get("metrics", {}) or {})
if "by_class" not in metrics and "by_class" in payload:
metrics["by_class"] = payload["by_class"]
return metrics
def _math_row() -> dict:
report = ledger_report()
for row in report["domains"]:
if row["domain"] == "mathematics_logic":
return row
raise AssertionError("mathematics_logic row missing from ledger_report()")
def _math_claim():
registry_path = _REPO_ROOT / LEDGER_SOURCES.reviewers
registry = load_reviewer_registry(registry_path)
claim = registry.expert_demo_claim_for("mathematics_logic")
assert claim is not None, "expert_demo_claims entry for math missing"
return claim
class TestAdr0110MathExpertDemoHolds:
def test_math_row_is_expert_demo(self) -> None:
row = _math_row()
assert row["status"] == "expert-demo"
assert row["predicates"]["expert_demo"] is True
def test_signed_claim_is_present(self) -> None:
claim = _math_claim()
assert set(claim.evidence_lanes) == set(_MATH_LANES)
assert claim.signed_by == "shay-j"
assert claim.evidence_revision == "adr-0110:reviewed:2026-05-22"
class TestAdr0110ReplayDigestByteEquality:
def test_derived_digest_matches_signed_claim(self) -> None:
claim = _math_claim()
lane_results = materialise_lane_results(
_MATH_LANES, fetch_split=_fetch
)
derived = derive_evidence_digest(
domain_id="mathematics_logic",
evidence_revision=claim.evidence_revision,
evidence_lanes=claim.evidence_lanes,
lane_results=lane_results,
)
assert derived == claim.claim_digest
class TestAdr0110OtherDomainsUnaffected:
def test_other_domains_stay_reasoning_capable(self) -> None:
promoted = [
row["domain"]
for row in ledger_report()["domains"]
if row["predicates"]["expert_demo"]
]
assert promoted == ["mathematics_logic"], (
f"ADR-0110 must promote exactly mathematics_logic; got: {promoted}"
)

4
tests/test_capability_reports.py
View file

@ -145,8 +145,10 @@ def test_ledger_status_is_predicate_derived() -> None:
assert systems["open_gaps"] == []
math = rows["mathematics_logic"]
assert math["status"] == "reasoning-capable"
# ADR-0110 — first expert-demo promotion lands on math.
assert math["status"] == "expert-demo"
assert math["predicates"]["reasoning_capable"] is True
assert math["predicates"]["expert_demo"] is True
assert math["open_gaps"] == []
physics = rows["physics"]

37
tests/test_expert_demo_contract.py
View file

@ -371,19 +371,32 @@ class TestExpertDemoThresholds:
assert "fabrication_control" in verdict.reason
class TestProductionLedgerUntouched:
"""ADR-0106 §Acceptance: no domain row's expert_demo field flips."""
class TestProductionLedgerPromotionsAreSignedOnly:
"""ADR-0106 §Acceptance preserved post-ADR-0110.
Originally tested that NO domain row carried ``expert_demo=true``
because no signed claims existed yet. After ADR-0110, the math
domain carries a signed claim. The load-bearing invariant remains:
every promoted domain must trace back to a signed claim in the
reviewer registry. A silent code-only promotion would be caught here.
"""
def test_every_promoted_domain_has_signed_claim(self) -> None:
from pathlib import Path
def test_no_production_domain_promoted_by_this_adr(self) -> None:
from core.capability.reporting import ledger_report
from core.capability.reviewers import load_reviewer_registry
from core.capability.sources import LEDGER_SOURCES
repo_root = Path(__file__).resolve().parent.parent
registry = load_reviewer_registry(repo_root / LEDGER_SOURCES.reviewers)
report = ledger_report()
promoted = [
row["domain"]
for row in report.get("domains", [])
if row.get("predicates", {}).get("expert_demo")
]
assert promoted == [], (
f"ADR-0106 must not promote any domain by code change alone; "
f"got: {promoted}"
)
for row in report.get("domains", []):
if not row.get("predicates", {}).get("expert_demo"):
continue
domain = row["domain"]
claim = registry.expert_demo_claim_for(domain)
assert claim is not None, (
f"domain {domain!r} reports expert_demo=true but has no "
f"signed expert_demo_claims entry"
)