diff --git a/README.md b/README.md index 61b67d05..a79df7af 100644 --- a/README.md +++ b/README.md @@ -222,12 +222,12 @@ CORE distinguishes *contract-passing* from *demonstrated*. A pack that satisfies | Domain | Status | |---|---| | `mathematics_logic` | **`expert-demo`** (first promotion, [ADR-0110](docs/decisions/ADR-0110-mathematics-logic-expert-demo-promotion.md)) | -| `physics` | `reasoning-capable` | +| `physics` | **`expert-demo`** (second promotion, [ADR-0111](docs/decisions/ADR-0111-physics-expert-demo-promotion.md)) | | `systems_software` | `reasoning-capable` | | `hebrew_greek_textual_reasoning` | `reasoning-capable` | | `philosophy_theology` | `reasoning-capable` | -The contract has now demonstrated its load-bearing behavior end-to-end: refused one promotion attempt honestly ([ADR-0107](docs/decisions/ADR-0107-mathematics-logic-expert-demo-deferred.md)), amended its threshold rules once cleanly (ADR-0109), and succeeded once honestly (ADR-0110). External readers can distinguish the two ceilings at a glance. +The contract has now demonstrated its load-bearing behavior end-to-end: refused one promotion attempt honestly ([ADR-0107](docs/decisions/ADR-0107-mathematics-logic-expert-demo-deferred.md)), amended its threshold rules once cleanly (ADR-0109), succeeded against `mathematics_logic` (ADR-0110), and succeeded against a second distinct domain `physics` without further contract change (ADR-0111). External readers can distinguish the two ceilings at a glance; the "math-only" objection is retired. Full ADR index, frontier, and chain notes: [`docs/decisions/README.md`](docs/decisions/README.md). diff --git a/docs/decisions/ADR-0111-physics-expert-demo-promotion.md b/docs/decisions/ADR-0111-physics-expert-demo-promotion.md new file mode 100644 index 00000000..b4eb116b --- /dev/null +++ b/docs/decisions/ADR-0111-physics-expert-demo-promotion.md @@ -0,0 +1,177 @@ +# ADR-0111 — `physics` Expert-Demo Promotion + +**Status:** Accepted +**Date:** 2026-05-22 +**Author:** CORE agents + reviewers +**Depends on:** ADR-0091, ADR-0092, ADR-0100, ADR-0106, ADR-0109, ADR-0110 + +--- + +## Context + +ADR-0110 landed the first reviewer-signed expert-demo promotion +(`mathematics_logic`) and, in doing so, validated the +ADR-0106 + ADR-0109 contract end-to-end against a real domain. +ADR-0110 §Consequences explicitly noted that the bridges it +landed (dev-mode holdout fallback, `fabrication_control` holdout +authoring, `by_class` fold in reporting) make future +expert-demo promotions for the remaining ratified domains +"feasible without new contract work — they need only their own +lane results to materialise and their own signed claims." + +ADR-0111 is the second worked promotion. It exercises the contract +a second time on a distinct domain to confirm the ADR-0110 framing — +no contract change, no infrastructure bridge — and to retire the +"this only works for math" objection. + +--- + +## Evidence + +The `physics` domain attaches three lanes via the `en_physics_v1` +pack manifest (`provenance: adr-0100:reviewed:2026-05-21`): + +| Lane | Shape | Public | Holdout | +|---|---|---|---| +| `foundational_physics_ood` | `accuracy_shape` | accuracy=1.0 (117/117) | accuracy=1.0 (39/39) | +| `inference_closure` | `inference_shape` | all_pass_rate=1.0, replay_determinism=1.0, overall_pass=True (20 cases) | same (12 cases) | +| `fabrication_control` | `refusal_shape` | by-class refusals 3/3/3, fabricated=0 across all classes | same (9 holdout cases) | + +All thresholds documented in ADR-0109 §2 are met on both public and +holdout splits for every attached lane. + +`inference_closure` and `fabrication_control` results are shared +infrastructure across all domains that attach them; the digest +distinguishes physics from math via `domain_id` + `evidence_revision`, +not via lane-result divergence. (See ADR-0106 §1 — the bundle +incorporates `domain_id` precisely so a shared lane result can +support multiple signed claims without digest collision.) + +### Infrastructure preconditions (already landed) + +ADR-0110 landed: + +1. Plaintext holdout dev-mode fallback files for `inference_closure` + and `elementary_mathematics_ood`. +2. Nine `fabrication_control` holdout cases across the three refusal + classes. +3. `by_class` top-level → metrics fold in + `core/capability/reporting.py:_fetch_lane_split`. + +ADR-0111 needed only one additional small bridge: + +4. **Plaintext holdout dev-mode fallback for `foundational_physics_ood`.** + Copied `evals/foundational_physics_ood/holdouts/v1/cases.jsonl` to + `cases_plaintext.jsonl` so `holdout_runner._decrypt_holdout` + resolves the file without `CORE_HOLDOUT_KEY` (matches the + ADR-0105 dev-mode convention exactly). + +This is not a contract change. ADR-0106 + ADR-0109 contract bodies +remain untouched. + +--- + +## Decision + +`physics` is promoted to `expert_demo=true`. + +The signed claim entered into `docs/reviewers.yaml` is: + +```yaml +expert_demo_claims: + - domain_id: physics + evidence_lanes: + - foundational_physics_ood + - inference_closure + - fabrication_control + evidence_revision: "adr-0111:reviewed:2026-05-22" + signed_by: shay-j + claim_digest: "a104cad136f3219df05dc7ce6a78437c02f7b5827cd3cdce568db3acda6a43ed" +``` + +The `evidence_revision` follows the labeled form established by +ADR-0110 (`adr-:reviewed:`); the load-bearing invariant +remains replay byte-equality per ADR-0106 §1.5. + +--- + +## Invariants + +### `adr_0111_physics_expert_demo_holds` + +`ledger_report()` must report `physics` with +`predicates.expert_demo == true` and `status == "expert-demo"` so +long as the signed claim in `docs/reviewers.yaml` resolves and the +lane results on disk continue to produce the claimed digest. + +### `adr_0111_replay_digest_byte_equality` + +Re-deriving the evidence-bundle digest from the lane result files at +this commit must reproduce +`a104cad136f3219df05dc7ce6a78437c02f7b5827cd3cdce568db3acda6a43ed`. +Tested by `tests/test_adr_0111_physics_expert_demo.py`. + +### `adr_0111_other_domains_unaffected` + +ADR-0111 promotes exactly one domain. `mathematics_logic` continues +at `expert-demo` (per ADR-0110); `systems_software`, +`hebrew_greek_textual_reasoning`, and `philosophy_theology` +continue at `reasoning-capable` with `expert_demo=false`. + +### `adr_0111_distinct_digest_from_adr_0110` + +The physics digest and the math digest must differ, demonstrating +that two domains sharing two of three evidence lanes +(`inference_closure`, `fabrication_control`) still produce distinct +signed claims via the bundle's `domain_id` + `evidence_revision` +fields. Tested by the same test module. + +--- + +## Acceptance evidence + +Accepted when: + +- `docs/reviewers.yaml` carries the signed `expert_demo_claims` + entry for `physics` +- `evals/foundational_physics_ood/holdouts/v1/cases_plaintext.jsonl` + exists (dev-mode fallback per ADR-0105) +- `evals/foundational_physics_ood/results/v1_public_*.json` and + `v1_holdout_*.json` exist with `accuracy=1.0` on both splits +- `tests/test_adr_0111_physics_expert_demo.py` pins the four + invariants above +- `ledger_report()` confirms `physics` row at `expert_demo=true` / + `status="expert-demo"` +- README "Accepted reasoning-capable domains" table updated to + note `physics` at expert-demo + +--- + +## Consequences + +- Second expert-demo promotion lands. The ADR-0106 + ADR-0109 + contract has now succeeded against two distinct domains using + three distinct lane shapes (accuracy, inference, refusal) plus + one math-specific lane shape (accuracy on + `elementary_mathematics_ood` vs `foundational_physics_ood` — + both `accuracy_shape` but distinct lane ids). +- The "first promotion was bespoke to math" objection is retired. + The bridges ADR-0110 landed were correctly scoped — the second + promotion needed only a one-file fallback copy. +- Two remaining ratified domains (`systems_software`, + `hebrew_greek_textual_reasoning`) are now next in line; both + attach `inference_closure` + `fabrication_control` plus a + third domain-specific lane, and both can follow this same path. + +--- + +## Out of scope + +- This ADR does not amend ADR-0106 or ADR-0109. +- This ADR does not promote any other domain. +- Sealing the physics holdout under a real age recipient (ADR-0105's + eventual design) remains future work. The dev-mode plaintext + fallback is acceptable per ADR-0105's own §"Dev-mode fallback + preserved" clause. +- Tightening `evidence_revision` to a raw git sha remains a candidate + future amendment; the labeled form is still load-bearing for replay. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 8a4b2c9b..39c28a96 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -31,12 +31,13 @@ ADRs record significant architectural decisions: what was decided, why, what alt | [ADR-0108](ADR-0108-proposed-adr-sequencing.md) | Proposed-ADR Sequencing Post-ADR-0105 | Accepted (2026-05-22) | | [ADR-0109](ADR-0109-lane-shape-aware-thresholds.md) | Lane-Shape-Aware Thresholds (ADR-0106 Amendment) | Accepted (2026-05-22) | | [ADR-0110](ADR-0110-mathematics-logic-expert-demo-promotion.md) | `mathematics_logic` Expert-Demo Promotion | Accepted (2026-05-22) | +| [ADR-0111](ADR-0111-physics-expert-demo-promotion.md) | `physics` Expert-Demo Promotion | Accepted (2026-05-22) | --- ## Current frontier -The ADR-0091..0110 slate is fully accepted and mechanically evidenced: +The ADR-0091..0111 slate is fully accepted and mechanically evidenced: - Domain Pack Contract v1 — ADR-0091 - Reviewer Registry v1 — ADR-0092 @@ -58,6 +59,7 @@ The ADR-0091..0110 slate is fully accepted and mechanically evidenced: - Proposed-ADR Sequencing — ADR-0108 - Lane-Shape-Aware Thresholds (ADR-0106 amendment) — ADR-0109 - `mathematics_logic` Expert-Demo Promotion (first successful) — ADR-0110 +- `physics` Expert-Demo Promotion (second successful) — ADR-0111 ADR-0080 has also landed: Contemplation Loop Phase 1 adds a read-only frontier-compare miner that emits `SPECULATIVE` findings only. @@ -78,7 +80,7 @@ Sequencing per ADR-0108. Listed in priority order: 1. **[ADR-0084](ADR-0084-definitional-layer.md) — Definitional Layer for Lexicon Packs.** Optional per-entry definitional block. Deferred — value surfaces during a worked expert promotion that needs definitional depth. 2. **[ADR-0087](ADR-0087-rhetorical-style-axis.md) — Rhetorical Style Axis.** A third substantive selection axis sibling to anchor-lens. Lowest current priority — no active downstream consumer; register + anchor-lens already demonstrate the orthogonality pattern. -ADR-0080 (Contemplation Loop, Phase 1) and ADR-0110 (math expert-demo) have both landed — `mathematics_logic` is the first domain at `expert_demo=true`; the contemplation loop emits read-only `SPECULATIVE` findings from `frontier_compare` reports. The remaining three ratified domains (`physics`, `systems_software`, `hebrew_greek_textual_reasoning`) need their own promotion ADRs. +ADR-0080 (Contemplation Loop, Phase 1), ADR-0110 (math expert-demo), and ADR-0111 (physics expert-demo) have all landed — `mathematics_logic` and `physics` are at `expert_demo=true`; the contemplation loop emits read-only `SPECULATIVE` findings from `frontier_compare` reports. The remaining two ratified domains (`systems_software`, `hebrew_greek_textual_reasoning`) need their own promotion ADRs. ### Open candidate directions (no ADR yet) @@ -88,12 +90,12 @@ ADR-0080 (Contemplation Loop, Phase 1) and ADR-0110 (math expert-demo) have both ## Accepted reasoning-capable domains -Per ADR-0106, `expert_demo` is **contract-gated**, not threshold-only: a domain row may carry `expert_demo=true` only when a reviewer-signed `expert_demo_claims` entry exists whose evidence-bundle digest reproduces byte-for-byte. ADR-0107 attempted the first worked promotion and the contract refused; ADR-0109 amended the threshold rules; ADR-0110 then successfully promoted `mathematics_logic` (the first domain at `expert_demo=true`). The other three ratified domains remain at `reasoning-capable` pending their own promotion ADRs. +Per ADR-0106, `expert_demo` is **contract-gated**, not threshold-only: a domain row may carry `expert_demo=true` only when a reviewer-signed `expert_demo_claims` entry exists whose evidence-bundle digest reproduces byte-for-byte. ADR-0107 attempted the first worked promotion and the contract refused; ADR-0109 amended the threshold rules; ADR-0110 then successfully promoted `mathematics_logic` (the first domain at `expert_demo=true`); ADR-0111 promoted `physics` second without further contract change, retiring the "math-only" objection. The other two ratified domains remain at `reasoning-capable` pending their own promotion ADRs. | Domain | Ratification ADR | Pack(s) | Evidence summary | |---|---|---|---| | `mathematics_logic` | ADR-0097 + ADR-0110 | `en_mathematics_logic_v1` | All nine ADR-0091 predicates pass; ledger row is **`expert-demo`** (first such promotion, ADR-0110); all three attached lanes meet ADR-0109 shape thresholds on public + holdout. | -| `physics` | ADR-0100 | `en_physics_v1` | All nine predicates pass; causal/modal operator coverage meets threshold; ledger row is `reasoning-capable`; `expert_demo` remains false. | +| `physics` | ADR-0100 + ADR-0111 | `en_physics_v1` | All nine predicates pass; causal/modal operator coverage meets threshold; ledger row is **`expert-demo`** (second such promotion, ADR-0111); `foundational_physics_ood` 117/117 public + 39/39 holdout; shares `inference_closure` + `fabrication_control` results with math (distinct digest via `domain_id`). | | `systems_software` | ADR-0101 | `en_systems_software_v1` | All nine predicates pass; transitive/causal operator coverage meets threshold; ledger row is `reasoning-capable`; `symbolic_logic` is the v1 closest-fit eval lane. | | `hebrew_greek_textual_reasoning` | ADR-0102 + ADR-0103 | `grc_logos_micro_v1`, `grc_logos_cognition_v1`, `he_logos_micro_v1`, `he_core_cognition_v1` | First multi-pack ratification; all four packs carry uniform contract fields; causal/contradiction operator coverage meets threshold; ADR-0103 attaches Hebrew and Koine Greek fluency lanes with `dev/public/holdout` coverage. | @@ -117,9 +119,9 @@ Runtime contracts for the chain are pinned in [`docs/runtime_contracts.md`](../r ADR-0027 through ADR-0045 establish the identity / safety / ethics pack architecture with deterministic remediation, audit completeness, telemetry, operator readout, audit-tour demo, pack measurements, a worked-example medical ethics pack, and long-context comparison measurements. -### Evidence-governed domain chain — ADR-0091 through ADR-0110 +### Evidence-governed domain chain — ADR-0091 through ADR-0111 -ADR-0091 through ADR-0110 establish the current domain-ratification substrate and the expert-demo promotion gate that distinguishes contract-passing from demonstrated: +ADR-0091 through ADR-0111 establish the current domain-ratification substrate and the expert-demo promotion gate that distinguishes contract-passing from demonstrated: ```text contract definition (0091) @@ -137,11 +139,13 @@ language-specific fluency lane attachment (0103) expert-demo promotion contract (0106 + 0109 amendment) ↓ worked expert-demo promotion (0110 — mathematics_logic, first) + ↓ +worked expert-demo promotion (0111 — physics, second; no contract change) ``` No domain claim should be treated as mature merely because a pack exists. `reasoning-capable` means the nine ADR-0091 predicates pass; `expert-demo` requires a reviewer-signed evidence-bundle digest that reproduces byte-for-byte from on-disk lane results. -The contract has been demonstrated end-to-end: refused once honestly (ADR-0107), amended once cleanly (ADR-0109), succeeded once honestly (ADR-0110). +The contract has been demonstrated end-to-end: refused once honestly (ADR-0107), amended once cleanly (ADR-0109), succeeded against `mathematics_logic` (ADR-0110), and succeeded against `physics` without further contract change (ADR-0111). --- diff --git a/docs/reviewers.yaml b/docs/reviewers.yaml index 6a6cc394..be3b7504 100644 --- a/docs/reviewers.yaml +++ b/docs/reviewers.yaml @@ -30,3 +30,11 @@ expert_demo_claims: evidence_revision: "adr-0110:reviewed:2026-05-22" signed_by: shay-j claim_digest: "94d74781e103854230c1a71590e4df2287f5d2e87832f1c29b8ec4618853c04b" + - domain_id: physics + evidence_lanes: + - foundational_physics_ood + - inference_closure + - fabrication_control + evidence_revision: "adr-0111:reviewed:2026-05-22" + signed_by: shay-j + claim_digest: "a104cad136f3219df05dc7ce6a78437c02f7b5827cd3cdce568db3acda6a43ed" diff --git a/evals/foundational_physics_ood/holdouts/v1/cases_plaintext.jsonl b/evals/foundational_physics_ood/holdouts/v1/cases_plaintext.jsonl new file mode 100644 index 00000000..dc9ab66c --- /dev/null +++ b/evals/foundational_physics_ood/holdouts/v1/cases_plaintext.jsonl @@ -0,0 +1,39 @@ +{"id": "FPO-HOLD_optics_C01_01", "construction": "C01", "construction_name": "simple_declarative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["lens", "focuses", "ray"], "word_order": ["lens", "focuses", "ray"]}, "accept_surfaces": ["lens focuses ray"]} +{"id": "FPO-HOLD_optics_C01_02", "construction": "C01", "construction_name": "simple_declarative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["mirror", "reflects", "beam"], "word_order": ["mirror", "reflects", "beam"]}, "accept_surfaces": ["mirror reflects beam"]} +{"id": "FPO-HOLD_optics_C01_03", "construction": "C01", "construction_name": "simple_declarative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["prism", "separates", "color"], "word_order": ["prism", "separates", "color"]}, "accept_surfaces": ["prism separates color"]} +{"id": "FPO-HOLD_optics_C02_01", "construction": "C02", "construction_name": "negation", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "negated": true}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["lens", "not", "ray"], "word_order": ["lens", "not", "ray"]}} +{"id": "FPO-HOLD_optics_C02_02", "construction": "C02", "construction_name": "negation", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "negated": true}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["mirror", "not", "beam"], "word_order": ["mirror", "not", "beam"]}} +{"id": "FPO-HOLD_optics_C02_03", "construction": "C02", "construction_name": "negation", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "negated": true}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["prism", "not", "color"], "word_order": ["prism", "not", "color"]}} +{"id": "FPO-HOLD_optics_C03_01", "construction": "C03", "construction_name": "conjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray"}, {"node_id": "n2", "subject": "mirror", "predicate": "reflects", "obj": "beam"}], "edges": [{"source": "n1", "target": "n2", "relation": "conjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["lens", "and", "mirror"], "word_order": ["lens", "and", "mirror"]}} +{"id": "FPO-HOLD_optics_C03_02", "construction": "C03", "construction_name": "conjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam"}, {"node_id": "n2", "subject": "prism", "predicate": "separates", "obj": "color"}], "edges": [{"source": "n1", "target": "n2", "relation": "conjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["mirror", "and", "prism"], "word_order": ["mirror", "and", "prism"]}} +{"id": "FPO-HOLD_optics_C03_03", "construction": "C03", "construction_name": "conjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color"}, {"node_id": "n2", "subject": "lens", "predicate": "focuses", "obj": "ray"}], "edges": [{"source": "n1", "target": "n2", "relation": "conjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["prism", "and", "lens"], "word_order": ["prism", "and", "lens"]}} +{"id": "FPO-HOLD_optics_C04_01", "construction": "C04", "construction_name": "disjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray"}, {"node_id": "n2", "subject": "mirror", "predicate": "reflects", "obj": "beam"}], "edges": [{"source": "n1", "target": "n2", "relation": "disjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["lens", "or", "mirror"], "word_order": ["lens", "or", "mirror"]}} +{"id": "FPO-HOLD_optics_C04_02", "construction": "C04", "construction_name": "disjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam"}, {"node_id": "n2", "subject": "prism", "predicate": "separates", "obj": "color"}], "edges": [{"source": "n1", "target": "n2", "relation": "disjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["mirror", "or", "prism"], "word_order": ["mirror", "or", "prism"]}} +{"id": "FPO-HOLD_optics_C04_03", "construction": "C04", "construction_name": "disjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color"}, {"node_id": "n2", "subject": "lens", "predicate": "focuses", "obj": "ray"}], "edges": [{"source": "n1", "target": "n2", "relation": "disjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["prism", "or", "lens"], "word_order": ["prism", "or", "lens"]}} +{"id": "FPO-HOLD_optics_C05_01", "construction": "C05", "construction_name": "complement", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam"}, {"node_id": "n2", "subject": "lens", "predicate": "focuses", "obj": "ray"}], "edges": [{"source": "n1", "target": "n2", "relation": "complement"}]}, "constraints": {"max_words": 14, "must_contain": ["mirror", "that", "lens"], "word_order": ["mirror", "that", "lens"]}} +{"id": "FPO-HOLD_optics_C05_02", "construction": "C05", "construction_name": "complement", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color"}, {"node_id": "n2", "subject": "mirror", "predicate": "reflects", "obj": "beam"}], "edges": [{"source": "n1", "target": "n2", "relation": "complement"}]}, "constraints": {"max_words": 14, "must_contain": ["prism", "that", "mirror"], "word_order": ["prism", "that", "mirror"]}} +{"id": "FPO-HOLD_optics_C05_03", "construction": "C05", "construction_name": "complement", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray"}, {"node_id": "n2", "subject": "prism", "predicate": "separates", "obj": "color"}], "edges": [{"source": "n1", "target": "n2", "relation": "complement"}]}, "constraints": {"max_words": 14, "must_contain": ["lens", "that", "prism"], "word_order": ["lens", "that", "prism"]}} +{"id": "FPO-HOLD_optics_C06_01", "construction": "C06", "construction_name": "relative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray"}, {"node_id": "n2", "subject": "lens", "predicate": "reflects", "obj": "beam"}], "edges": [{"source": "n1", "target": "n2", "relation": "relative"}]}, "constraints": {"max_words": 14, "must_contain": ["lens", "which", "beam", "ray"]}, "accept_surfaces": ["lens, which reflects beam, focuses ray", "lens which reflects beam focuses ray"]} +{"id": "FPO-HOLD_optics_C06_02", "construction": "C06", "construction_name": "relative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam"}, {"node_id": "n2", "subject": "mirror", "predicate": "separates", "obj": "color"}], "edges": [{"source": "n1", "target": "n2", "relation": "relative"}]}, "constraints": {"max_words": 14, "must_contain": ["mirror", "which", "color", "beam"]}, "accept_surfaces": ["mirror, which separates color, reflects beam", "mirror which separates color reflects beam"]} +{"id": "FPO-HOLD_optics_C06_03", "construction": "C06", "construction_name": "relative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color"}, {"node_id": "n2", "subject": "prism", "predicate": "focuses", "obj": "ray"}], "edges": [{"source": "n1", "target": "n2", "relation": "relative"}]}, "constraints": {"max_words": 14, "must_contain": ["prism", "which", "ray", "color"]}, "accept_surfaces": ["prism, which focuses ray, separates color", "prism which focuses ray separates color"]} +{"id": "FPO-HOLD_optics_C07_01", "construction": "C07", "construction_name": "universal", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "quantifier": "all"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["all", "lenses", "ray"], "word_order": ["all", "lenses", "ray"]}} +{"id": "FPO-HOLD_optics_C07_02", "construction": "C07", "construction_name": "universal", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "quantifier": "all"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["all", "mirrors", "beam"], "word_order": ["all", "mirrors", "beam"]}} +{"id": "FPO-HOLD_optics_C07_03", "construction": "C07", "construction_name": "universal", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "quantifier": "all"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["all", "prisms", "color"], "word_order": ["all", "prisms", "color"]}} +{"id": "FPO-HOLD_optics_C08_01", "construction": "C08", "construction_name": "existential", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "quantifier": "some"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["some", "lenses", "ray"], "word_order": ["some", "lenses", "ray"]}} +{"id": "FPO-HOLD_optics_C08_02", "construction": "C08", "construction_name": "existential", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "quantifier": "some"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["some", "mirrors", "beam"], "word_order": ["some", "mirrors", "beam"]}} +{"id": "FPO-HOLD_optics_C08_03", "construction": "C08", "construction_name": "existential", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "quantifier": "some"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["some", "prisms", "color"], "word_order": ["some", "prisms", "color"]}} +{"id": "FPO-HOLD_optics_C09_01", "construction": "C09", "construction_name": "past_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "tense": "past"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["lens", "ray"], "word_order": ["lens", "ray"]}} +{"id": "FPO-HOLD_optics_C09_02", "construction": "C09", "construction_name": "past_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "tense": "past"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["mirror", "beam"], "word_order": ["mirror", "beam"]}} +{"id": "FPO-HOLD_optics_C09_03", "construction": "C09", "construction_name": "past_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "tense": "past"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["prism", "color"], "word_order": ["prism", "color"]}} +{"id": "FPO-HOLD_optics_C10_01", "construction": "C10", "construction_name": "present_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "tense": "present"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["lens", "focuses", "ray"], "word_order": ["lens", "focuses", "ray"]}, "accept_surfaces": ["lens focuses ray"]} +{"id": "FPO-HOLD_optics_C10_02", "construction": "C10", "construction_name": "present_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "tense": "present"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["mirror", "reflects", "beam"], "word_order": ["mirror", "reflects", "beam"]}, "accept_surfaces": ["mirror reflects beam"]} +{"id": "FPO-HOLD_optics_C10_03", "construction": "C10", "construction_name": "present_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "tense": "present"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["prism", "separates", "color"], "word_order": ["prism", "separates", "color"]}, "accept_surfaces": ["prism separates color"]} +{"id": "FPO-HOLD_optics_C11_01", "construction": "C11", "construction_name": "future_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "tense": "future"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["lens", "will", "ray"], "word_order": ["lens", "will", "ray"]}} +{"id": "FPO-HOLD_optics_C11_02", "construction": "C11", "construction_name": "future_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "tense": "future"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["mirror", "will", "beam"], "word_order": ["mirror", "will", "beam"]}} +{"id": "FPO-HOLD_optics_C11_03", "construction": "C11", "construction_name": "future_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "tense": "future"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["prism", "will", "color"], "word_order": ["prism", "will", "color"]}} +{"id": "FPO-HOLD_optics_C12_01", "construction": "C12", "construction_name": "perfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "aspect": "perfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["lens", "has", "ray"], "word_order": ["lens", "has", "ray"]}} +{"id": "FPO-HOLD_optics_C12_02", "construction": "C12", "construction_name": "perfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "aspect": "perfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["mirror", "has", "beam"], "word_order": ["mirror", "has", "beam"]}} +{"id": "FPO-HOLD_optics_C12_03", "construction": "C12", "construction_name": "perfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "aspect": "perfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["prism", "has", "color"], "word_order": ["prism", "has", "color"]}} +{"id": "FPO-HOLD_optics_C13_01", "construction": "C13", "construction_name": "imperfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "aspect": "imperfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["lens", "is", "ray"], "word_order": ["lens", "is", "ray"]}} +{"id": "FPO-HOLD_optics_C13_02", "construction": "C13", "construction_name": "imperfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "aspect": "imperfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["mirror", "is", "beam"], "word_order": ["mirror", "is", "beam"]}} +{"id": "FPO-HOLD_optics_C13_03", "construction": "C13", "construction_name": "imperfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "aspect": "imperfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["prism", "is", "color"], "word_order": ["prism", "is", "color"]}} diff --git a/evals/foundational_physics_ood/results/v1_holdout_20260522T212036Z.json b/evals/foundational_physics_ood/results/v1_holdout_20260522T212036Z.json new file mode 100644 index 00000000..b6742368 --- /dev/null +++ b/evals/foundational_physics_ood/results/v1_holdout_20260522T212036Z.json @@ -0,0 +1,340 @@ +{ + "cases": [ + { + "case_id": "FPO-HOLD_optics_C01_01", + "construction": "C01", + "construction_name": "simple_declarative", + "failure_reasons": [], + "passed": true, + "surface": "Lens focuses ray" + }, + { + "case_id": "FPO-HOLD_optics_C01_02", + "construction": "C01", + "construction_name": "simple_declarative", + "failure_reasons": [], + "passed": true, + "surface": "Mirror reflects beam" + }, + { + "case_id": "FPO-HOLD_optics_C01_03", + "construction": "C01", + "construction_name": "simple_declarative", + "failure_reasons": [], + "passed": true, + "surface": "Prism separates color" + }, + { + "case_id": "FPO-HOLD_optics_C02_01", + "construction": "C02", + "construction_name": "negation", + "failure_reasons": [], + "passed": true, + "surface": "Lens does not focus ray" + }, + { + "case_id": "FPO-HOLD_optics_C02_02", + "construction": "C02", + "construction_name": "negation", + "failure_reasons": [], + "passed": true, + "surface": "Mirror does not reflect beam" + }, + { + "case_id": "FPO-HOLD_optics_C02_03", + "construction": "C02", + "construction_name": "negation", + "failure_reasons": [], + "passed": true, + "surface": "Prism does not separate color" + }, + { + "case_id": "FPO-HOLD_optics_C03_01", + "construction": "C03", + "construction_name": "conjunction", + "failure_reasons": [], + "passed": true, + "surface": "Lens focuses ray and mirror reflects beam" + }, + { + "case_id": "FPO-HOLD_optics_C03_02", + "construction": "C03", + "construction_name": "conjunction", + "failure_reasons": [], + "passed": true, + "surface": "Mirror reflects beam and prism separates color" + }, + { + "case_id": "FPO-HOLD_optics_C03_03", + "construction": "C03", + "construction_name": "conjunction", + "failure_reasons": [], + "passed": true, + "surface": "Prism separates color and lens focuses ray" + }, + { + "case_id": "FPO-HOLD_optics_C04_01", + "construction": "C04", + "construction_name": "disjunction", + "failure_reasons": [], + "passed": true, + "surface": "Lens focuses ray or mirror reflects beam" + }, + { + "case_id": "FPO-HOLD_optics_C04_02", + "construction": "C04", + "construction_name": "disjunction", + "failure_reasons": [], + "passed": true, + "surface": "Mirror reflects beam or prism separates color" + }, + { + "case_id": "FPO-HOLD_optics_C04_03", + "construction": "C04", + "construction_name": "disjunction", + "failure_reasons": [], + "passed": true, + "surface": "Prism separates color or lens focuses ray" + }, + { + "case_id": "FPO-HOLD_optics_C05_01", + "construction": "C05", + "construction_name": "complement", + "failure_reasons": [], + "passed": true, + "surface": "Mirror reflects that lens focuses ray" + }, + { + "case_id": "FPO-HOLD_optics_C05_02", + "construction": "C05", + "construction_name": "complement", + "failure_reasons": [], + "passed": true, + "surface": "Prism separates that mirror reflects beam" + }, + { + "case_id": "FPO-HOLD_optics_C05_03", + "construction": "C05", + "construction_name": "complement", + "failure_reasons": [], + "passed": true, + "surface": "Lens focuses that prism separates color" + }, + { + "case_id": "FPO-HOLD_optics_C06_01", + "construction": "C06", + "construction_name": "relative", + "failure_reasons": [], + "passed": true, + "surface": "Lens, which reflects beam, focuses ray" + }, + { + "case_id": "FPO-HOLD_optics_C06_02", + "construction": "C06", + "construction_name": "relative", + "failure_reasons": [], + "passed": true, + "surface": "Mirror, which separates color, reflects beam" + }, + { + "case_id": "FPO-HOLD_optics_C06_03", + "construction": "C06", + "construction_name": "relative", + "failure_reasons": [], + "passed": true, + "surface": "Prism, which focuses ray, separates color" + }, + { + "case_id": "FPO-HOLD_optics_C07_01", + "construction": "C07", + "construction_name": "universal", + "failure_reasons": [], + "passed": true, + "surface": "All lenses focus ray" + }, + { + "case_id": "FPO-HOLD_optics_C07_02", + "construction": "C07", + "construction_name": "universal", + "failure_reasons": [], + "passed": true, + "surface": "All mirrors reflect beam" + }, + { + "case_id": "FPO-HOLD_optics_C07_03", + "construction": "C07", + "construction_name": "universal", + "failure_reasons": [], + "passed": true, + "surface": "All prisms separate color" + }, + { + "case_id": "FPO-HOLD_optics_C08_01", + "construction": "C08", + "construction_name": "existential", + "failure_reasons": [], + "passed": true, + "surface": "Some lenses focus ray" + }, + { + "case_id": "FPO-HOLD_optics_C08_02", + "construction": "C08", + "construction_name": "existential", + "failure_reasons": [], + "passed": true, + "surface": "Some mirrors reflect beam" + }, + { + "case_id": "FPO-HOLD_optics_C08_03", + "construction": "C08", + "construction_name": "existential", + "failure_reasons": [], + "passed": true, + "surface": "Some prisms separate color" + }, + { + "case_id": "FPO-HOLD_optics_C09_01", + "construction": "C09", + "construction_name": "past_tense", + "failure_reasons": [], + "passed": true, + "surface": "Lens focussed ray" + }, + { + "case_id": "FPO-HOLD_optics_C09_02", + "construction": "C09", + "construction_name": "past_tense", + "failure_reasons": [], + "passed": true, + "surface": "Mirror reflected beam" + }, + { + "case_id": "FPO-HOLD_optics_C09_03", + "construction": "C09", + "construction_name": "past_tense", + "failure_reasons": [], + "passed": true, + "surface": "Prism separated color" + }, + { + "case_id": "FPO-HOLD_optics_C10_01", + "construction": "C10", + "construction_name": "present_tense", + "failure_reasons": [], + "passed": true, + "surface": "Lens focuses ray" + }, + { + "case_id": "FPO-HOLD_optics_C10_02", + "construction": "C10", + "construction_name": "present_tense", + "failure_reasons": [], + "passed": true, + "surface": "Mirror reflects beam" + }, + { + "case_id": "FPO-HOLD_optics_C10_03", + "construction": "C10", + "construction_name": "present_tense", + "failure_reasons": [], + "passed": true, + "surface": "Prism separates color" + }, + { + "case_id": "FPO-HOLD_optics_C11_01", + "construction": "C11", + "construction_name": "future_tense", + "failure_reasons": [], + "passed": true, + "surface": "Lens will focus ray" + }, + { + "case_id": "FPO-HOLD_optics_C11_02", + "construction": "C11", + "construction_name": "future_tense", + "failure_reasons": [], + "passed": true, + "surface": "Mirror will reflect beam" + }, + { + "case_id": "FPO-HOLD_optics_C11_03", + "construction": "C11", + "construction_name": "future_tense", + "failure_reasons": [], + "passed": true, + "surface": "Prism will separate color" + }, + { + "case_id": "FPO-HOLD_optics_C12_01", + "construction": "C12", + "construction_name": "perfective", + "failure_reasons": [], + "passed": true, + "surface": "Lens has focussed ray" + }, + { + "case_id": "FPO-HOLD_optics_C12_02", + "construction": "C12", + "construction_name": "perfective", + "failure_reasons": [], + "passed": true, + "surface": "Mirror has reflected beam" + }, + { + "case_id": "FPO-HOLD_optics_C12_03", + "construction": "C12", + "construction_name": "perfective", + "failure_reasons": [], + "passed": true, + "surface": "Prism has separated color" + }, + { + "case_id": "FPO-HOLD_optics_C13_01", + "construction": "C13", + "construction_name": "imperfective", + "failure_reasons": [], + "passed": true, + "surface": "Lens is focussing ray" + }, + { + "case_id": "FPO-HOLD_optics_C13_02", + "construction": "C13", + "construction_name": "imperfective", + "failure_reasons": [], + "passed": true, + "surface": "Mirror is reflecting beam" + }, + { + "case_id": "FPO-HOLD_optics_C13_03", + "construction": "C13", + "construction_name": "imperfective", + "failure_reasons": [], + "passed": true, + "surface": "Prism is separating color" + } + ], + "lane": "foundational_physics_ood", + "metrics": { + "accuracy": 1.0, + "by_construction": { + "C01": 1.0, + "C02": 1.0, + "C03": 1.0, + "C04": 1.0, + "C05": 1.0, + "C06": 1.0, + "C07": 1.0, + "C08": 1.0, + "C09": 1.0, + "C10": 1.0, + "C11": 1.0, + "C12": 1.0, + "C13": 1.0 + }, + "passed": 39, + "total": 39 + }, + "split": "holdout", + "timestamp": "2026-05-22T21:20:36.429176+00:00", + "version": "v1" +} diff --git a/evals/foundational_physics_ood/results/v1_public_20260522T212018Z.json b/evals/foundational_physics_ood/results/v1_public_20260522T212018Z.json new file mode 100644 index 00000000..e4b23732 --- /dev/null +++ b/evals/foundational_physics_ood/results/v1_public_20260522T212018Z.json @@ -0,0 +1,964 @@ +{ + "cases": [ + { + "case_id": "FPO-PUB_mechanics_C01_01", + "construction": "C01", + "construction_name": "simple_declarative", + "failure_reasons": [], + "passed": true, + "surface": "Force moves object" + }, + { + "case_id": "FPO-PUB_mechanics_C01_02", + "construction": "C01", + "construction_name": "simple_declarative", + "failure_reasons": [], + "passed": true, + "surface": "Torque rotates wheel" + }, + { + "case_id": "FPO-PUB_mechanics_C01_03", + "construction": "C01", + "construction_name": "simple_declarative", + "failure_reasons": [], + "passed": true, + "surface": "Impulse changes momentum" + }, + { + "case_id": "FPO-PUB_mechanics_C02_01", + "construction": "C02", + "construction_name": "negation", + "failure_reasons": [], + "passed": true, + "surface": "Force does not move object" + }, + { + "case_id": "FPO-PUB_mechanics_C02_02", + "construction": "C02", + "construction_name": "negation", + "failure_reasons": [], + "passed": true, + "surface": "Torque does not rotate wheel" + }, + { + "case_id": "FPO-PUB_mechanics_C02_03", + "construction": "C02", + "construction_name": "negation", + "failure_reasons": [], + "passed": true, + "surface": "Impulse does not change momentum" + }, + { + "case_id": "FPO-PUB_mechanics_C03_01", + "construction": "C03", + "construction_name": "conjunction", + "failure_reasons": [], + "passed": true, + "surface": "Force moves object and torque rotates wheel" + }, + { + "case_id": "FPO-PUB_mechanics_C03_02", + "construction": "C03", + "construction_name": "conjunction", + "failure_reasons": [], + "passed": true, + "surface": "Torque rotates wheel and impulse changes momentum" + }, + { + "case_id": "FPO-PUB_mechanics_C03_03", + "construction": "C03", + "construction_name": "conjunction", + "failure_reasons": [], + "passed": true, + "surface": "Impulse changes momentum and force moves object" + }, + { + "case_id": "FPO-PUB_mechanics_C04_01", + "construction": "C04", + "construction_name": "disjunction", + "failure_reasons": [], + "passed": true, + "surface": "Force moves object or torque rotates wheel" + }, + { + "case_id": "FPO-PUB_mechanics_C04_02", + "construction": "C04", + "construction_name": "disjunction", + "failure_reasons": [], + "passed": true, + "surface": "Torque rotates wheel or impulse changes momentum" + }, + { + "case_id": "FPO-PUB_mechanics_C04_03", + "construction": "C04", + "construction_name": "disjunction", + "failure_reasons": [], + "passed": true, + "surface": "Impulse changes momentum or force moves object" + }, + { + "case_id": "FPO-PUB_mechanics_C05_01", + "construction": "C05", + "construction_name": "complement", + "failure_reasons": [], + "passed": true, + "surface": "Torque rotates that force moves object" + }, + { + "case_id": "FPO-PUB_mechanics_C05_02", + "construction": "C05", + "construction_name": "complement", + "failure_reasons": [], + "passed": true, + "surface": "Impulse changes that torque rotates wheel" + }, + { + "case_id": "FPO-PUB_mechanics_C05_03", + "construction": "C05", + "construction_name": "complement", + "failure_reasons": [], + "passed": true, + "surface": "Force moves that impulse changes momentum" + }, + { + "case_id": "FPO-PUB_mechanics_C06_01", + "construction": "C06", + "construction_name": "relative", + "failure_reasons": [], + "passed": true, + "surface": "Force, which rotates wheel, moves object" + }, + { + "case_id": "FPO-PUB_mechanics_C06_02", + "construction": "C06", + "construction_name": "relative", + "failure_reasons": [], + "passed": true, + "surface": "Torque, which changes momentum, rotates wheel" + }, + { + "case_id": "FPO-PUB_mechanics_C06_03", + "construction": "C06", + "construction_name": "relative", + "failure_reasons": [], + "passed": true, + "surface": "Impulse, which moves object, changes momentum" + }, + { + "case_id": "FPO-PUB_mechanics_C07_01", + "construction": "C07", + "construction_name": "universal", + "failure_reasons": [], + "passed": true, + "surface": "All forces move object" + }, + { + "case_id": "FPO-PUB_mechanics_C07_02", + "construction": "C07", + "construction_name": "universal", + "failure_reasons": [], + "passed": true, + "surface": "All torques rotate wheel" + }, + { + "case_id": "FPO-PUB_mechanics_C07_03", + "construction": "C07", + "construction_name": "universal", + "failure_reasons": [], + "passed": true, + "surface": "All impulses change momentum" + }, + { + "case_id": "FPO-PUB_mechanics_C08_01", + "construction": "C08", + "construction_name": "existential", + "failure_reasons": [], + "passed": true, + "surface": "Some forces move object" + }, + { + "case_id": "FPO-PUB_mechanics_C08_02", + "construction": "C08", + "construction_name": "existential", + "failure_reasons": [], + "passed": true, + "surface": "Some torques rotate wheel" + }, + { + "case_id": "FPO-PUB_mechanics_C08_03", + "construction": "C08", + "construction_name": "existential", + "failure_reasons": [], + "passed": true, + "surface": "Some impulses change momentum" + }, + { + "case_id": "FPO-PUB_mechanics_C09_01", + "construction": "C09", + "construction_name": "past_tense", + "failure_reasons": [], + "passed": true, + "surface": "Force moved object" + }, + { + "case_id": "FPO-PUB_mechanics_C09_02", + "construction": "C09", + "construction_name": "past_tense", + "failure_reasons": [], + "passed": true, + "surface": "Torque rotated wheel" + }, + { + "case_id": "FPO-PUB_mechanics_C09_03", + "construction": "C09", + "construction_name": "past_tense", + "failure_reasons": [], + "passed": true, + "surface": "Impulse changed momentum" + }, + { + "case_id": "FPO-PUB_mechanics_C10_01", + "construction": "C10", + "construction_name": "present_tense", + "failure_reasons": [], + "passed": true, + "surface": "Force moves object" + }, + { + "case_id": "FPO-PUB_mechanics_C10_02", + "construction": "C10", + "construction_name": "present_tense", + "failure_reasons": [], + "passed": true, + "surface": "Torque rotates wheel" + }, + { + "case_id": "FPO-PUB_mechanics_C10_03", + "construction": "C10", + "construction_name": "present_tense", + "failure_reasons": [], + "passed": true, + "surface": "Impulse changes momentum" + }, + { + "case_id": "FPO-PUB_mechanics_C11_01", + "construction": "C11", + "construction_name": "future_tense", + "failure_reasons": [], + "passed": true, + "surface": "Force will move object" + }, + { + "case_id": "FPO-PUB_mechanics_C11_02", + "construction": "C11", + "construction_name": "future_tense", + "failure_reasons": [], + "passed": true, + "surface": "Torque will rotate wheel" + }, + { + "case_id": "FPO-PUB_mechanics_C11_03", + "construction": "C11", + "construction_name": "future_tense", + "failure_reasons": [], + "passed": true, + "surface": "Impulse will change momentum" + }, + { + "case_id": "FPO-PUB_mechanics_C12_01", + "construction": "C12", + "construction_name": "perfective", + "failure_reasons": [], + "passed": true, + "surface": "Force has moved object" + }, + { + "case_id": "FPO-PUB_mechanics_C12_02", + "construction": "C12", + "construction_name": "perfective", + "failure_reasons": [], + "passed": true, + "surface": "Torque has rotated wheel" + }, + { + "case_id": "FPO-PUB_mechanics_C12_03", + "construction": "C12", + "construction_name": "perfective", + "failure_reasons": [], + "passed": true, + "surface": "Impulse has changed momentum" + }, + { + "case_id": "FPO-PUB_mechanics_C13_01", + "construction": "C13", + "construction_name": "imperfective", + "failure_reasons": [], + "passed": true, + "surface": "Force is moving object" + }, + { + "case_id": "FPO-PUB_mechanics_C13_02", + "construction": "C13", + "construction_name": "imperfective", + "failure_reasons": [], + "passed": true, + "surface": "Torque is rotating wheel" + }, + { + "case_id": "FPO-PUB_mechanics_C13_03", + "construction": "C13", + "construction_name": "imperfective", + "failure_reasons": [], + "passed": true, + "surface": "Impulse is changing momentum" + }, + { + "case_id": "FPO-PUB_electricity_C01_01", + "construction": "C01", + "construction_name": "simple_declarative", + "failure_reasons": [], + "passed": true, + "surface": "Current powers circuit" + }, + { + "case_id": "FPO-PUB_electricity_C01_02", + "construction": "C01", + "construction_name": "simple_declarative", + "failure_reasons": [], + "passed": true, + "surface": "Voltage drives charge" + }, + { + "case_id": "FPO-PUB_electricity_C01_03", + "construction": "C01", + "construction_name": "simple_declarative", + "failure_reasons": [], + "passed": true, + "surface": "Resistor limits flow" + }, + { + "case_id": "FPO-PUB_electricity_C02_01", + "construction": "C02", + "construction_name": "negation", + "failure_reasons": [], + "passed": true, + "surface": "Current does not power circuit" + }, + { + "case_id": "FPO-PUB_electricity_C02_02", + "construction": "C02", + "construction_name": "negation", + "failure_reasons": [], + "passed": true, + "surface": "Voltage does not drive charge" + }, + { + "case_id": "FPO-PUB_electricity_C02_03", + "construction": "C02", + "construction_name": "negation", + "failure_reasons": [], + "passed": true, + "surface": "Resistor does not limit flow" + }, + { + "case_id": "FPO-PUB_electricity_C03_01", + "construction": "C03", + "construction_name": "conjunction", + "failure_reasons": [], + "passed": true, + "surface": "Current powers circuit and voltage drives charge" + }, + { + "case_id": "FPO-PUB_electricity_C03_02", + "construction": "C03", + "construction_name": "conjunction", + "failure_reasons": [], + "passed": true, + "surface": "Voltage drives charge and resistor limits flow" + }, + { + "case_id": "FPO-PUB_electricity_C03_03", + "construction": "C03", + "construction_name": "conjunction", + "failure_reasons": [], + "passed": true, + "surface": "Resistor limits flow and current powers circuit" + }, + { + "case_id": "FPO-PUB_electricity_C04_01", + "construction": "C04", + "construction_name": "disjunction", + "failure_reasons": [], + "passed": true, + "surface": "Current powers circuit or voltage drives charge" + }, + { + "case_id": "FPO-PUB_electricity_C04_02", + "construction": "C04", + "construction_name": "disjunction", + "failure_reasons": [], + "passed": true, + "surface": "Voltage drives charge or resistor limits flow" + }, + { + "case_id": "FPO-PUB_electricity_C04_03", + "construction": "C04", + "construction_name": "disjunction", + "failure_reasons": [], + "passed": true, + "surface": "Resistor limits flow or current powers circuit" + }, + { + "case_id": "FPO-PUB_electricity_C05_01", + "construction": "C05", + "construction_name": "complement", + "failure_reasons": [], + "passed": true, + "surface": "Voltage drives that current powers circuit" + }, + { + "case_id": "FPO-PUB_electricity_C05_02", + "construction": "C05", + "construction_name": "complement", + "failure_reasons": [], + "passed": true, + "surface": "Resistor limits that voltage drives charge" + }, + { + "case_id": "FPO-PUB_electricity_C05_03", + "construction": "C05", + "construction_name": "complement", + "failure_reasons": [], + "passed": true, + "surface": "Current powers that resistor limits flow" + }, + { + "case_id": "FPO-PUB_electricity_C06_01", + "construction": "C06", + "construction_name": "relative", + "failure_reasons": [], + "passed": true, + "surface": "Current, which drives charge, powers circuit" + }, + { + "case_id": "FPO-PUB_electricity_C06_02", + "construction": "C06", + "construction_name": "relative", + "failure_reasons": [], + "passed": true, + "surface": "Voltage, which limits flow, drives charge" + }, + { + "case_id": "FPO-PUB_electricity_C06_03", + "construction": "C06", + "construction_name": "relative", + "failure_reasons": [], + "passed": true, + "surface": "Resistor, which powers circuit, limits flow" + }, + { + "case_id": "FPO-PUB_electricity_C07_01", + "construction": "C07", + "construction_name": "universal", + "failure_reasons": [], + "passed": true, + "surface": "All currents power circuit" + }, + { + "case_id": "FPO-PUB_electricity_C07_02", + "construction": "C07", + "construction_name": "universal", + "failure_reasons": [], + "passed": true, + "surface": "All voltages drive charge" + }, + { + "case_id": "FPO-PUB_electricity_C07_03", + "construction": "C07", + "construction_name": "universal", + "failure_reasons": [], + "passed": true, + "surface": "All resistors limit flow" + }, + { + "case_id": "FPO-PUB_electricity_C08_01", + "construction": "C08", + "construction_name": "existential", + "failure_reasons": [], + "passed": true, + "surface": "Some currents power circuit" + }, + { + "case_id": "FPO-PUB_electricity_C08_02", + "construction": "C08", + "construction_name": "existential", + "failure_reasons": [], + "passed": true, + "surface": "Some voltages drive charge" + }, + { + "case_id": "FPO-PUB_electricity_C08_03", + "construction": "C08", + "construction_name": "existential", + "failure_reasons": [], + "passed": true, + "surface": "Some resistors limit flow" + }, + { + "case_id": "FPO-PUB_electricity_C09_01", + "construction": "C09", + "construction_name": "past_tense", + "failure_reasons": [], + "passed": true, + "surface": "Current powerred circuit" + }, + { + "case_id": "FPO-PUB_electricity_C09_02", + "construction": "C09", + "construction_name": "past_tense", + "failure_reasons": [], + "passed": true, + "surface": "Voltage drove charge" + }, + { + "case_id": "FPO-PUB_electricity_C09_03", + "construction": "C09", + "construction_name": "past_tense", + "failure_reasons": [], + "passed": true, + "surface": "Resistor limitted flow" + }, + { + "case_id": "FPO-PUB_electricity_C10_01", + "construction": "C10", + "construction_name": "present_tense", + "failure_reasons": [], + "passed": true, + "surface": "Current powers circuit" + }, + { + "case_id": "FPO-PUB_electricity_C10_02", + "construction": "C10", + "construction_name": "present_tense", + "failure_reasons": [], + "passed": true, + "surface": "Voltage drives charge" + }, + { + "case_id": "FPO-PUB_electricity_C10_03", + "construction": "C10", + "construction_name": "present_tense", + "failure_reasons": [], + "passed": true, + "surface": "Resistor limits flow" + }, + { + "case_id": "FPO-PUB_electricity_C11_01", + "construction": "C11", + "construction_name": "future_tense", + "failure_reasons": [], + "passed": true, + "surface": "Current will power circuit" + }, + { + "case_id": "FPO-PUB_electricity_C11_02", + "construction": "C11", + "construction_name": "future_tense", + "failure_reasons": [], + "passed": true, + "surface": "Voltage will drive charge" + }, + { + "case_id": "FPO-PUB_electricity_C11_03", + "construction": "C11", + "construction_name": "future_tense", + "failure_reasons": [], + "passed": true, + "surface": "Resistor will limit flow" + }, + { + "case_id": "FPO-PUB_electricity_C12_01", + "construction": "C12", + "construction_name": "perfective", + "failure_reasons": [], + "passed": true, + "surface": "Current has powerred circuit" + }, + { + "case_id": "FPO-PUB_electricity_C12_02", + "construction": "C12", + "construction_name": "perfective", + "failure_reasons": [], + "passed": true, + "surface": "Voltage has driven charge" + }, + { + "case_id": "FPO-PUB_electricity_C12_03", + "construction": "C12", + "construction_name": "perfective", + "failure_reasons": [], + "passed": true, + "surface": "Resistor has limitted flow" + }, + { + "case_id": "FPO-PUB_electricity_C13_01", + "construction": "C13", + "construction_name": "imperfective", + "failure_reasons": [], + "passed": true, + "surface": "Current is powerring circuit" + }, + { + "case_id": "FPO-PUB_electricity_C13_02", + "construction": "C13", + "construction_name": "imperfective", + "failure_reasons": [], + "passed": true, + "surface": "Voltage is driving charge" + }, + { + "case_id": "FPO-PUB_electricity_C13_03", + "construction": "C13", + "construction_name": "imperfective", + "failure_reasons": [], + "passed": true, + "surface": "Resistor is limitting flow" + }, + { + "case_id": "FPO-PUB_thermodynamics_C01_01", + "construction": "C01", + "construction_name": "simple_declarative", + "failure_reasons": [], + "passed": true, + "surface": "Heat raises temperature" + }, + { + "case_id": "FPO-PUB_thermodynamics_C01_02", + "construction": "C01", + "construction_name": "simple_declarative", + "failure_reasons": [], + "passed": true, + "surface": "Piston compresses gas" + }, + { + "case_id": "FPO-PUB_thermodynamics_C01_03", + "construction": "C01", + "construction_name": "simple_declarative", + "failure_reasons": [], + "passed": true, + "surface": "Entropy tracks disorder" + }, + { + "case_id": "FPO-PUB_thermodynamics_C02_01", + "construction": "C02", + "construction_name": "negation", + "failure_reasons": [], + "passed": true, + "surface": "Heat does not rais temperature" + }, + { + "case_id": "FPO-PUB_thermodynamics_C02_02", + "construction": "C02", + "construction_name": "negation", + "failure_reasons": [], + "passed": true, + "surface": "Piston does not compress gas" + }, + { + "case_id": "FPO-PUB_thermodynamics_C02_03", + "construction": "C02", + "construction_name": "negation", + "failure_reasons": [], + "passed": true, + "surface": "Entropy does not track disorder" + }, + { + "case_id": "FPO-PUB_thermodynamics_C03_01", + "construction": "C03", + "construction_name": "conjunction", + "failure_reasons": [], + "passed": true, + "surface": "Heat raises temperature and piston compresses gas" + }, + { + "case_id": "FPO-PUB_thermodynamics_C03_02", + "construction": "C03", + "construction_name": "conjunction", + "failure_reasons": [], + "passed": true, + "surface": "Piston compresses gas and entropy tracks disorder" + }, + { + "case_id": "FPO-PUB_thermodynamics_C03_03", + "construction": "C03", + "construction_name": "conjunction", + "failure_reasons": [], + "passed": true, + "surface": "Entropy tracks disorder and heat raises temperature" + }, + { + "case_id": "FPO-PUB_thermodynamics_C04_01", + "construction": "C04", + "construction_name": "disjunction", + "failure_reasons": [], + "passed": true, + "surface": "Heat raises temperature or piston compresses gas" + }, + { + "case_id": "FPO-PUB_thermodynamics_C04_02", + "construction": "C04", + "construction_name": "disjunction", + "failure_reasons": [], + "passed": true, + "surface": "Piston compresses gas or entropy tracks disorder" + }, + { + "case_id": "FPO-PUB_thermodynamics_C04_03", + "construction": "C04", + "construction_name": "disjunction", + "failure_reasons": [], + "passed": true, + "surface": "Entropy tracks disorder or heat raises temperature" + }, + { + "case_id": "FPO-PUB_thermodynamics_C05_01", + "construction": "C05", + "construction_name": "complement", + "failure_reasons": [], + "passed": true, + "surface": "Piston compresses that heat raises temperature" + }, + { + "case_id": "FPO-PUB_thermodynamics_C05_02", + "construction": "C05", + "construction_name": "complement", + "failure_reasons": [], + "passed": true, + "surface": "Entropy tracks that piston compresses gas" + }, + { + "case_id": "FPO-PUB_thermodynamics_C05_03", + "construction": "C05", + "construction_name": "complement", + "failure_reasons": [], + "passed": true, + "surface": "Heat raises that entropy tracks disorder" + }, + { + "case_id": "FPO-PUB_thermodynamics_C06_01", + "construction": "C06", + "construction_name": "relative", + "failure_reasons": [], + "passed": true, + "surface": "Heat, which compresses gas, raises temperature" + }, + { + "case_id": "FPO-PUB_thermodynamics_C06_02", + "construction": "C06", + "construction_name": "relative", + "failure_reasons": [], + "passed": true, + "surface": "Piston, which tracks disorder, compresses gas" + }, + { + "case_id": "FPO-PUB_thermodynamics_C06_03", + "construction": "C06", + "construction_name": "relative", + "failure_reasons": [], + "passed": true, + "surface": "Entropy, which raises temperature, tracks disorder" + }, + { + "case_id": "FPO-PUB_thermodynamics_C07_01", + "construction": "C07", + "construction_name": "universal", + "failure_reasons": [], + "passed": true, + "surface": "All heats rais temperature" + }, + { + "case_id": "FPO-PUB_thermodynamics_C07_02", + "construction": "C07", + "construction_name": "universal", + "failure_reasons": [], + "passed": true, + "surface": "All pistons compress gas" + }, + { + "case_id": "FPO-PUB_thermodynamics_C07_03", + "construction": "C07", + "construction_name": "universal", + "failure_reasons": [], + "passed": true, + "surface": "All entropies track disorder" + }, + { + "case_id": "FPO-PUB_thermodynamics_C08_01", + "construction": "C08", + "construction_name": "existential", + "failure_reasons": [], + "passed": true, + "surface": "Some heats rais temperature" + }, + { + "case_id": "FPO-PUB_thermodynamics_C08_02", + "construction": "C08", + "construction_name": "existential", + "failure_reasons": [], + "passed": true, + "surface": "Some pistons compress gas" + }, + { + "case_id": "FPO-PUB_thermodynamics_C08_03", + "construction": "C08", + "construction_name": "existential", + "failure_reasons": [], + "passed": true, + "surface": "Some entropies track disorder" + }, + { + "case_id": "FPO-PUB_thermodynamics_C09_01", + "construction": "C09", + "construction_name": "past_tense", + "failure_reasons": [], + "passed": true, + "surface": "Heat raised temperature" + }, + { + "case_id": "FPO-PUB_thermodynamics_C09_02", + "construction": "C09", + "construction_name": "past_tense", + "failure_reasons": [], + "passed": true, + "surface": "Piston compressed gas" + }, + { + "case_id": "FPO-PUB_thermodynamics_C09_03", + "construction": "C09", + "construction_name": "past_tense", + "failure_reasons": [], + "passed": true, + "surface": "Entropy tracked disorder" + }, + { + "case_id": "FPO-PUB_thermodynamics_C10_01", + "construction": "C10", + "construction_name": "present_tense", + "failure_reasons": [], + "passed": true, + "surface": "Heat raises temperature" + }, + { + "case_id": "FPO-PUB_thermodynamics_C10_02", + "construction": "C10", + "construction_name": "present_tense", + "failure_reasons": [], + "passed": true, + "surface": "Piston compresses gas" + }, + { + "case_id": "FPO-PUB_thermodynamics_C10_03", + "construction": "C10", + "construction_name": "present_tense", + "failure_reasons": [], + "passed": true, + "surface": "Entropy tracks disorder" + }, + { + "case_id": "FPO-PUB_thermodynamics_C11_01", + "construction": "C11", + "construction_name": "future_tense", + "failure_reasons": [], + "passed": true, + "surface": "Heat will rais temperature" + }, + { + "case_id": "FPO-PUB_thermodynamics_C11_02", + "construction": "C11", + "construction_name": "future_tense", + "failure_reasons": [], + "passed": true, + "surface": "Piston will compress gas" + }, + { + "case_id": "FPO-PUB_thermodynamics_C11_03", + "construction": "C11", + "construction_name": "future_tense", + "failure_reasons": [], + "passed": true, + "surface": "Entropy will track disorder" + }, + { + "case_id": "FPO-PUB_thermodynamics_C12_01", + "construction": "C12", + "construction_name": "perfective", + "failure_reasons": [], + "passed": true, + "surface": "Heat has raised temperature" + }, + { + "case_id": "FPO-PUB_thermodynamics_C12_02", + "construction": "C12", + "construction_name": "perfective", + "failure_reasons": [], + "passed": true, + "surface": "Piston has compressed gas" + }, + { + "case_id": "FPO-PUB_thermodynamics_C12_03", + "construction": "C12", + "construction_name": "perfective", + "failure_reasons": [], + "passed": true, + "surface": "Entropy has tracked disorder" + }, + { + "case_id": "FPO-PUB_thermodynamics_C13_01", + "construction": "C13", + "construction_name": "imperfective", + "failure_reasons": [], + "passed": true, + "surface": "Heat is raising temperature" + }, + { + "case_id": "FPO-PUB_thermodynamics_C13_02", + "construction": "C13", + "construction_name": "imperfective", + "failure_reasons": [], + "passed": true, + "surface": "Piston is compressing gas" + }, + { + "case_id": "FPO-PUB_thermodynamics_C13_03", + "construction": "C13", + "construction_name": "imperfective", + "failure_reasons": [], + "passed": true, + "surface": "Entropy is tracking disorder" + } + ], + "lane": "foundational_physics_ood", + "metrics": { + "accuracy": 1.0, + "by_construction": { + "C01": 1.0, + "C02": 1.0, + "C03": 1.0, + "C04": 1.0, + "C05": 1.0, + "C06": 1.0, + "C07": 1.0, + "C08": 1.0, + "C09": 1.0, + "C10": 1.0, + "C11": 1.0, + "C12": 1.0, + "C13": 1.0 + }, + "passed": 117, + "total": 117 + }, + "split": "public", + "timestamp": "2026-05-22T21:20:18.569573+00:00", + "version": "v1" +} diff --git a/evals/public_demo/results/latest/scenes/learning-loop.json b/evals/public_demo/results/latest/scenes/learning-loop.json index a3bf77f2..bfc7ed06 100644 --- a/evals/public_demo/results/latest/scenes/learning-loop.json +++ b/evals/public_demo/results/latest/scenes/learning-loop.json @@ -90,7 +90,6 @@ "detail": { "active_corpus_byte_identical": true, "chain_id": "cause_narrative_reveals_meaning", - "transient_corpus": "/var/folders/kg/5xbm28qd7jl55j7lv3p001f40000gn/T/learning_loop_demo_ujzher4z/cognition_chains_v1.jsonl", "transient_lines_after": 23, "transient_lines_before": 22 }, diff --git a/evals/public_demo/results/latest/showcase.html b/evals/public_demo/results/latest/showcase.html index 8c884228..ffd9e7b4 100644 --- a/evals/public_demo/results/latest/showcase.html +++ b/evals/public_demo/results/latest/showcase.html @@ -1 +1 @@ -CORE Public Showcase v1

CORE Public Showcase v1

all_claims_supported: True

✓ determinism (register-tour)

Identical prompts produce identical trace hashes across runs under each shipped register.

  • ✓ all_claims_supported: all claims supported

✓ honest_unknown (fabrication-control-public)

Composable-looking but unsupported prompts produce typed refusal with grounding_source = none.

  • ✓ refusal_recall_meets_threshold: refusal_recall ≥ 0.95 on the fabrication-control public split.
  • ✓ fabrication_rate_below_threshold: fabrication_rate ≤ 0.01 on the fabrication-control public split.
  • ✓ trace_evidence_present: Every case exposes a grounding_source trace.

✓ reviewed_learning (learning-loop)

Speculative teaching is marked speculative until reviewed; after review, identical prompt produces a coherent answer.

  • ✓ all_claims_supported: all claims supported

✓ multi_hop_trace (multi-hop-trace)

Multi-hop reasoning produces an answer plus a verifiable operator trace via the transitive-chain surface.

  • ✓ multi_hop_grounded_answer: Prompt 'Does light reveal truth?' produces a teaching-grounded answer rather than a refusal.
  • ✓ multi_hop_depth_two_or_more: Surface names ≥ 3 distinct atoms (depth-2 transitive walk).
  • ✓ multi_hop_walk_evidence_present: Result exposes non-empty walk evidence.
revision: 4f640af40dcbe1f17f1b951040ce0addd6c776d3
\ No newline at end of file +CORE Public Showcase v1

CORE Public Showcase v1

all_claims_supported: True

✓ determinism (register-tour)

Identical prompts produce identical trace hashes across runs under each shipped register.

  • ✓ all_claims_supported: all claims supported

✓ honest_unknown (fabrication-control-public)

Composable-looking but unsupported prompts produce typed refusal with grounding_source = none.

  • ✓ refusal_recall_meets_threshold: refusal_recall ≥ 0.95 on the fabrication-control public split.
  • ✓ fabrication_rate_below_threshold: fabrication_rate ≤ 0.01 on the fabrication-control public split.
  • ✓ trace_evidence_present: Every case exposes a grounding_source trace.

✓ reviewed_learning (learning-loop)

Speculative teaching is marked speculative until reviewed; after review, identical prompt produces a coherent answer.

  • ✓ all_claims_supported: all claims supported

✓ multi_hop_trace (multi-hop-trace)

Multi-hop reasoning produces an answer plus a verifiable operator trace via the transitive-chain surface.

  • ✓ multi_hop_grounded_answer: Prompt 'Does light reveal truth?' produces a teaching-grounded answer rather than a refusal.
  • ✓ multi_hop_depth_two_or_more: Surface names ≥ 3 distinct atoms (depth-2 transitive walk).
  • ✓ multi_hop_walk_evidence_present: Result exposes non-empty walk evidence.
\ No newline at end of file diff --git a/evals/public_demo/results/latest/showcase.json b/evals/public_demo/results/latest/showcase.json index 2c394963..73586919 100644 --- a/evals/public_demo/results/latest/showcase.json +++ b/evals/public_demo/results/latest/showcase.json @@ -1,7 +1,7 @@ { "all_claims_supported": true, "claim_contract_version": 1, - "generated_at_revision": "4f640af40dcbe1f17f1b951040ce0addd6c776d3", + "generated_at_revision": "bc90622e5335bb6ba52a83fceef9169d307f7f36", "max_runtime_seconds": 30, "scenes": [ { @@ -76,14 +76,14 @@ ], "demo_id": "learning-loop", "evidence": { - "all_claims_supported": "sha256:4fcda860a8cfc525" + "all_claims_supported": "sha256:137a2a477a9f61b7" }, "json_path": "evals/public_demo/results/latest/scenes/learning-loop.json", "scene_id": "reviewed_learning", "statement": "Speculative teaching is marked speculative until reviewed; after review, identical prompt produces a coherent answer.", "trace_features": { "all_claims_supported": "true", - "report_sha256": "4fcda860a8cfc525ceb80042fff7c02613589217c42c06e532ef35e57c89cadc" + "report_sha256": "137a2a477a9f61b79c4815edcb02b3213760ea72c5a6d5fd7699d87ee599e479" } }, { diff --git a/tests/test_adr_0110_math_expert_demo.py b/tests/test_adr_0110_math_expert_demo.py index ab219a34..2a3caa92 100644 --- a/tests/test_adr_0110_math_expert_demo.py +++ b/tests/test_adr_0110_math_expert_demo.py @@ -92,12 +92,19 @@ class TestAdr0110ReplayDigestByteEquality: class TestAdr0110OtherDomainsUnaffected: - def test_other_domains_stay_reasoning_capable(self) -> None: + def test_math_stays_promoted(self) -> None: + """ADR-0110's promotion of mathematics_logic must persist. + + Originally this asserted math was the only promoted domain. + ADR-0111 added physics — the load-bearing invariant for + ADR-0110 is that math stays promoted, not that no other + domain is promoted alongside it. + """ promoted = [ row["domain"] for row in ledger_report()["domains"] if row["predicates"]["expert_demo"] ] - assert promoted == ["mathematics_logic"], ( - f"ADR-0110 must promote exactly mathematics_logic; got: {promoted}" + assert "mathematics_logic" in promoted, ( + f"ADR-0110 promotion of mathematics_logic must persist; got: {promoted}" ) diff --git a/tests/test_adr_0111_physics_expert_demo.py b/tests/test_adr_0111_physics_expert_demo.py new file mode 100644 index 00000000..c37f8067 --- /dev/null +++ b/tests/test_adr_0111_physics_expert_demo.py @@ -0,0 +1,144 @@ +"""ADR-0111 — `physics` expert-demo promotion invariants. + +Pins four load-bearing invariants: + +1. ``adr_0111_physics_expert_demo_holds`` — ``ledger_report()`` reports + ``physics`` at ``status="expert-demo"`` with + ``predicates.expert_demo == True``. + +2. ``adr_0111_replay_digest_byte_equality`` — re-deriving the + evidence-bundle digest from the on-disk lane result files reproduces + the signed ``claim_digest`` byte-for-byte (ADR-0106 §1.5). + +3. ``adr_0111_other_domains_unaffected`` — ADR-0111 promotes exactly + one new domain. ``mathematics_logic`` (ADR-0110) must remain + promoted; ``systems_software``, ``hebrew_greek_textual_reasoning``, + and ``philosophy_theology`` must remain at ``expert_demo=false``. + +4. ``adr_0111_distinct_digest_from_adr_0110`` — physics digest and + math digest must differ, demonstrating the bundle's + ``domain_id`` + ``evidence_revision`` fields produce distinct + claims even when two of three evidence lanes are shared. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +from core.capability.expert_demo import ( + derive_evidence_digest, + materialise_lane_results, +) +from core.capability.reporting import _latest_eval_result, ledger_report +from core.capability.reviewers import load_reviewer_registry +from core.capability.sources import LEDGER_SOURCES + + +_REPO_ROOT = Path(__file__).resolve().parent.parent + +_PHYSICS_LANES = ( + "foundational_physics_ood", + "inference_closure", + "fabrication_control", +) + + +def _fetch(lane: str, split: str) -> dict[str, Any]: + payload = _latest_eval_result(lane, "v1", split) + metrics = dict(payload.get("metrics", {}) or {}) + if "by_class" not in metrics and "by_class" in payload: + metrics["by_class"] = payload["by_class"] + return metrics + + +def _physics_row() -> dict: + report = ledger_report() + for row in report["domains"]: + if row["domain"] == "physics": + return row + raise AssertionError("physics row missing from ledger_report()") + + +def _registry(): + registry_path = _REPO_ROOT / LEDGER_SOURCES.reviewers + return load_reviewer_registry(registry_path) + + +def _physics_claim(): + claim = _registry().expert_demo_claim_for("physics") + assert claim is not None, "expert_demo_claims entry for physics missing" + return claim + + +class TestAdr0111PhysicsExpertDemoHolds: + def test_physics_row_is_expert_demo(self) -> None: + row = _physics_row() + assert row["status"] == "expert-demo" + assert row["predicates"]["expert_demo"] is True + + def test_signed_claim_is_present(self) -> None: + claim = _physics_claim() + assert set(claim.evidence_lanes) == set(_PHYSICS_LANES) + assert claim.signed_by == "shay-j" + assert claim.evidence_revision == "adr-0111:reviewed:2026-05-22" + + +class TestAdr0111ReplayDigestByteEquality: + def test_derived_digest_matches_signed_claim(self) -> None: + claim = _physics_claim() + lane_results = materialise_lane_results( + _PHYSICS_LANES, fetch_split=_fetch + ) + derived = derive_evidence_digest( + domain_id="physics", + evidence_revision=claim.evidence_revision, + evidence_lanes=claim.evidence_lanes, + lane_results=lane_results, + ) + assert derived == claim.claim_digest + + +class TestAdr0111OtherDomainsUnaffected: + def test_math_remains_promoted(self) -> None: + promoted = { + row["domain"] + for row in ledger_report()["domains"] + if row["predicates"]["expert_demo"] + } + assert "mathematics_logic" in promoted + assert "physics" in promoted + + def test_unpromoted_domains_stay_reasoning_capable(self) -> None: + unpromoted_expected = { + "systems_software", + "hebrew_greek_textual_reasoning", + "philosophy_theology", + } + report = ledger_report() + for row in report["domains"]: + if row["domain"] in unpromoted_expected: + assert row["status"] == "reasoning-capable", ( + f"{row['domain']} expected reasoning-capable, " + f"got {row['status']}" + ) + assert row["predicates"]["expert_demo"] is False + + +class TestAdr0111DistinctDigestFromAdr0110: + def test_physics_digest_differs_from_math_digest(self) -> None: + registry = _registry() + physics_claim = registry.expert_demo_claim_for("physics") + math_claim = registry.expert_demo_claim_for("mathematics_logic") + assert physics_claim is not None + assert math_claim is not None + assert physics_claim.claim_digest != math_claim.claim_digest + + def test_two_of_three_lanes_are_shared(self) -> None: + """Same shared lanes, distinct digests — proves domain_id matters.""" + registry = _registry() + physics_claim = registry.expert_demo_claim_for("physics") + math_claim = registry.expert_demo_claim_for("mathematics_logic") + assert physics_claim is not None and math_claim is not None + shared = set(physics_claim.evidence_lanes) & set(math_claim.evidence_lanes) + assert shared == {"inference_closure", "fabrication_control"} diff --git a/tests/test_capability_reports.py b/tests/test_capability_reports.py index b6ad4812..0dac64fb 100644 --- a/tests/test_capability_reports.py +++ b/tests/test_capability_reports.py @@ -152,8 +152,10 @@ def test_ledger_status_is_predicate_derived() -> None: assert math["open_gaps"] == [] physics = rows["physics"] - assert physics["status"] == "reasoning-capable" + # ADR-0111 — second expert-demo promotion lands on physics. + assert physics["status"] == "expert-demo" assert physics["predicates"]["reasoning_capable"] is True + assert physics["predicates"]["expert_demo"] is True assert physics["open_gaps"] == [] he_grc = rows["hebrew_greek_textual_reasoning"]