feat: ADR-0111 physics expert-demo promotion (second successful)
Second worked promotion exercising the ADR-0106 + ADR-0109 contract on a domain distinct from mathematics_logic. No contract change. Evidence: - foundational_physics_ood: accuracy=1.0 (117/117 public, 39/39 holdout) - inference_closure: all_pass_rate=1.0 (shared with math, distinct digest via domain_id) - fabrication_control: refused=n, fabricated=0 across all classes (shared) Signed claim digest: a104cad136f3219df05dc7ce6a78437c02f7b5827cd3cdce568db3acda6a43ed Bridge landed: cases_plaintext.jsonl dev-mode fallback for foundational_physics_ood (matches ADR-0105 convention; analogous to the math/inference bridges in ADR-0110). One small file, not a contract change. Tests: - tests/test_adr_0111_physics_expert_demo.py — 4 invariants, 6 cases - tests/test_adr_0110_math_expert_demo.py — relaxed "only math promoted" to "math stays promoted" (load-bearing for ADR-0110 is persistence) - tests/test_capability_reports.py — physics row now expert-demo Retires the "first promotion was math-specific" objection: the bridges ADR-0110 landed were correctly scoped, and the contract holds across two distinct domains using shared lane infrastructure with distinct digests. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Shay
parent
bc90622e53
commit
45272a7bb2
13 changed files with 1702 additions and 18 deletions
|
|
@ -222,12 +222,12 @@ CORE distinguishes *contract-passing* from *demonstrated*. A pack that satisfies
|
|||
| Domain | Status |
|
||||
|---|---|
|
||||
| `mathematics_logic` | **`expert-demo`** (first promotion, [ADR-0110](docs/decisions/ADR-0110-mathematics-logic-expert-demo-promotion.md)) |
|
||||
| `physics` | `reasoning-capable` |
|
||||
| `physics` | **`expert-demo`** (second promotion, [ADR-0111](docs/decisions/ADR-0111-physics-expert-demo-promotion.md)) |
|
||||
| `systems_software` | `reasoning-capable` |
|
||||
| `hebrew_greek_textual_reasoning` | `reasoning-capable` |
|
||||
| `philosophy_theology` | `reasoning-capable` |
|
||||
|
||||
The contract has now demonstrated its load-bearing behavior end-to-end: refused one promotion attempt honestly ([ADR-0107](docs/decisions/ADR-0107-mathematics-logic-expert-demo-deferred.md)), amended its threshold rules once cleanly (ADR-0109), and succeeded once honestly (ADR-0110). External readers can distinguish the two ceilings at a glance.
|
||||
The contract has now demonstrated its load-bearing behavior end-to-end: refused one promotion attempt honestly ([ADR-0107](docs/decisions/ADR-0107-mathematics-logic-expert-demo-deferred.md)), amended its threshold rules once cleanly (ADR-0109), succeeded against `mathematics_logic` (ADR-0110), and succeeded against a second distinct domain `physics` without further contract change (ADR-0111). External readers can distinguish the two ceilings at a glance; the "math-only" objection is retired.
|
||||
|
||||
Full ADR index, frontier, and chain notes: [`docs/decisions/README.md`](docs/decisions/README.md).
|
||||
|
||||
|
|
|
|||
177
docs/decisions/ADR-0111-physics-expert-demo-promotion.md
Normal file
177
docs/decisions/ADR-0111-physics-expert-demo-promotion.md
Normal file
|
|
@ -0,0 +1,177 @@
|
|||
# ADR-0111 — `physics` Expert-Demo Promotion
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-05-22
|
||||
**Author:** CORE agents + reviewers
|
||||
**Depends on:** ADR-0091, ADR-0092, ADR-0100, ADR-0106, ADR-0109, ADR-0110
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0110 landed the first reviewer-signed expert-demo promotion
|
||||
(`mathematics_logic`) and, in doing so, validated the
|
||||
ADR-0106 + ADR-0109 contract end-to-end against a real domain.
|
||||
ADR-0110 §Consequences explicitly noted that the bridges it
|
||||
landed (dev-mode holdout fallback, `fabrication_control` holdout
|
||||
authoring, `by_class` fold in reporting) make future
|
||||
expert-demo promotions for the remaining ratified domains
|
||||
"feasible without new contract work — they need only their own
|
||||
lane results to materialise and their own signed claims."
|
||||
|
||||
ADR-0111 is the second worked promotion. It exercises the contract
|
||||
a second time on a distinct domain to confirm the ADR-0110 framing —
|
||||
no contract change, no infrastructure bridge — and to retire the
|
||||
"this only works for math" objection.
|
||||
|
||||
---
|
||||
|
||||
## Evidence
|
||||
|
||||
The `physics` domain attaches three lanes via the `en_physics_v1`
|
||||
pack manifest (`provenance: adr-0100:reviewed:2026-05-21`):
|
||||
|
||||
| Lane | Shape | Public | Holdout |
|
||||
|---|---|---|---|
|
||||
| `foundational_physics_ood` | `accuracy_shape` | accuracy=1.0 (117/117) | accuracy=1.0 (39/39) |
|
||||
| `inference_closure` | `inference_shape` | all_pass_rate=1.0, replay_determinism=1.0, overall_pass=True (20 cases) | same (12 cases) |
|
||||
| `fabrication_control` | `refusal_shape` | by-class refusals 3/3/3, fabricated=0 across all classes | same (9 holdout cases) |
|
||||
|
||||
All thresholds documented in ADR-0109 §2 are met on both public and
|
||||
holdout splits for every attached lane.
|
||||
|
||||
`inference_closure` and `fabrication_control` results are shared
|
||||
infrastructure across all domains that attach them; the digest
|
||||
distinguishes physics from math via `domain_id` + `evidence_revision`,
|
||||
not via lane-result divergence. (See ADR-0106 §1 — the bundle
|
||||
incorporates `domain_id` precisely so a shared lane result can
|
||||
support multiple signed claims without digest collision.)
|
||||
|
||||
### Infrastructure preconditions (already landed)
|
||||
|
||||
ADR-0110 landed:
|
||||
|
||||
1. Plaintext holdout dev-mode fallback files for `inference_closure`
|
||||
and `elementary_mathematics_ood`.
|
||||
2. Nine `fabrication_control` holdout cases across the three refusal
|
||||
classes.
|
||||
3. `by_class` top-level → metrics fold in
|
||||
`core/capability/reporting.py:_fetch_lane_split`.
|
||||
|
||||
ADR-0111 needed only one additional small bridge:
|
||||
|
||||
4. **Plaintext holdout dev-mode fallback for `foundational_physics_ood`.**
|
||||
Copied `evals/foundational_physics_ood/holdouts/v1/cases.jsonl` to
|
||||
`cases_plaintext.jsonl` so `holdout_runner._decrypt_holdout`
|
||||
resolves the file without `CORE_HOLDOUT_KEY` (matches the
|
||||
ADR-0105 dev-mode convention exactly).
|
||||
|
||||
This is not a contract change. ADR-0106 + ADR-0109 contract bodies
|
||||
remain untouched.
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
`physics` is promoted to `expert_demo=true`.
|
||||
|
||||
The signed claim entered into `docs/reviewers.yaml` is:
|
||||
|
||||
```yaml
|
||||
expert_demo_claims:
|
||||
- domain_id: physics
|
||||
evidence_lanes:
|
||||
- foundational_physics_ood
|
||||
- inference_closure
|
||||
- fabrication_control
|
||||
evidence_revision: "adr-0111:reviewed:2026-05-22"
|
||||
signed_by: shay-j
|
||||
claim_digest: "a104cad136f3219df05dc7ce6a78437c02f7b5827cd3cdce568db3acda6a43ed"
|
||||
```
|
||||
|
||||
The `evidence_revision` follows the labeled form established by
|
||||
ADR-0110 (`adr-<id>:reviewed:<date>`); the load-bearing invariant
|
||||
remains replay byte-equality per ADR-0106 §1.5.
|
||||
|
||||
---
|
||||
|
||||
## Invariants
|
||||
|
||||
### `adr_0111_physics_expert_demo_holds`
|
||||
|
||||
`ledger_report()` must report `physics` with
|
||||
`predicates.expert_demo == true` and `status == "expert-demo"` so
|
||||
long as the signed claim in `docs/reviewers.yaml` resolves and the
|
||||
lane results on disk continue to produce the claimed digest.
|
||||
|
||||
### `adr_0111_replay_digest_byte_equality`
|
||||
|
||||
Re-deriving the evidence-bundle digest from the lane result files at
|
||||
this commit must reproduce
|
||||
`a104cad136f3219df05dc7ce6a78437c02f7b5827cd3cdce568db3acda6a43ed`.
|
||||
Tested by `tests/test_adr_0111_physics_expert_demo.py`.
|
||||
|
||||
### `adr_0111_other_domains_unaffected`
|
||||
|
||||
ADR-0111 promotes exactly one domain. `mathematics_logic` continues
|
||||
at `expert-demo` (per ADR-0110); `systems_software`,
|
||||
`hebrew_greek_textual_reasoning`, and `philosophy_theology`
|
||||
continue at `reasoning-capable` with `expert_demo=false`.
|
||||
|
||||
### `adr_0111_distinct_digest_from_adr_0110`
|
||||
|
||||
The physics digest and the math digest must differ, demonstrating
|
||||
that two domains sharing two of three evidence lanes
|
||||
(`inference_closure`, `fabrication_control`) still produce distinct
|
||||
signed claims via the bundle's `domain_id` + `evidence_revision`
|
||||
fields. Tested by the same test module.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance evidence
|
||||
|
||||
Accepted when:
|
||||
|
||||
- `docs/reviewers.yaml` carries the signed `expert_demo_claims`
|
||||
entry for `physics`
|
||||
- `evals/foundational_physics_ood/holdouts/v1/cases_plaintext.jsonl`
|
||||
exists (dev-mode fallback per ADR-0105)
|
||||
- `evals/foundational_physics_ood/results/v1_public_*.json` and
|
||||
`v1_holdout_*.json` exist with `accuracy=1.0` on both splits
|
||||
- `tests/test_adr_0111_physics_expert_demo.py` pins the four
|
||||
invariants above
|
||||
- `ledger_report()` confirms `physics` row at `expert_demo=true` /
|
||||
`status="expert-demo"`
|
||||
- README "Accepted reasoning-capable domains" table updated to
|
||||
note `physics` at expert-demo
|
||||
|
||||
---
|
||||
|
||||
## Consequences
|
||||
|
||||
- Second expert-demo promotion lands. The ADR-0106 + ADR-0109
|
||||
contract has now succeeded against two distinct domains using
|
||||
three distinct lane shapes (accuracy, inference, refusal) plus
|
||||
one math-specific lane shape (accuracy on
|
||||
`elementary_mathematics_ood` vs `foundational_physics_ood` —
|
||||
both `accuracy_shape` but distinct lane ids).
|
||||
- The "first promotion was bespoke to math" objection is retired.
|
||||
The bridges ADR-0110 landed were correctly scoped — the second
|
||||
promotion needed only a one-file fallback copy.
|
||||
- Two remaining ratified domains (`systems_software`,
|
||||
`hebrew_greek_textual_reasoning`) are now next in line; both
|
||||
attach `inference_closure` + `fabrication_control` plus a
|
||||
third domain-specific lane, and both can follow this same path.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope
|
||||
|
||||
- This ADR does not amend ADR-0106 or ADR-0109.
|
||||
- This ADR does not promote any other domain.
|
||||
- Sealing the physics holdout under a real age recipient (ADR-0105's
|
||||
eventual design) remains future work. The dev-mode plaintext
|
||||
fallback is acceptable per ADR-0105's own §"Dev-mode fallback
|
||||
preserved" clause.
|
||||
- Tightening `evidence_revision` to a raw git sha remains a candidate
|
||||
future amendment; the labeled form is still load-bearing for replay.
|
||||
|
|
@ -31,12 +31,13 @@ ADRs record significant architectural decisions: what was decided, why, what alt
|
|||
| [ADR-0108](ADR-0108-proposed-adr-sequencing.md) | Proposed-ADR Sequencing Post-ADR-0105 | Accepted (2026-05-22) |
|
||||
| [ADR-0109](ADR-0109-lane-shape-aware-thresholds.md) | Lane-Shape-Aware Thresholds (ADR-0106 Amendment) | Accepted (2026-05-22) |
|
||||
| [ADR-0110](ADR-0110-mathematics-logic-expert-demo-promotion.md) | `mathematics_logic` Expert-Demo Promotion | Accepted (2026-05-22) |
|
||||
| [ADR-0111](ADR-0111-physics-expert-demo-promotion.md) | `physics` Expert-Demo Promotion | Accepted (2026-05-22) |
|
||||
|
||||
---
|
||||
|
||||
## Current frontier
|
||||
|
||||
The ADR-0091..0110 slate is fully accepted and mechanically evidenced:
|
||||
The ADR-0091..0111 slate is fully accepted and mechanically evidenced:
|
||||
|
||||
- Domain Pack Contract v1 — ADR-0091
|
||||
- Reviewer Registry v1 — ADR-0092
|
||||
|
|
@ -58,6 +59,7 @@ The ADR-0091..0110 slate is fully accepted and mechanically evidenced:
|
|||
- Proposed-ADR Sequencing — ADR-0108
|
||||
- Lane-Shape-Aware Thresholds (ADR-0106 amendment) — ADR-0109
|
||||
- `mathematics_logic` Expert-Demo Promotion (first successful) — ADR-0110
|
||||
- `physics` Expert-Demo Promotion (second successful) — ADR-0111
|
||||
|
||||
ADR-0080 has also landed: Contemplation Loop Phase 1 adds a read-only frontier-compare miner that emits `SPECULATIVE` findings only.
|
||||
|
||||
|
|
@ -78,7 +80,7 @@ Sequencing per ADR-0108. Listed in priority order:
|
|||
1. **[ADR-0084](ADR-0084-definitional-layer.md) — Definitional Layer for Lexicon Packs.** Optional per-entry definitional block. Deferred — value surfaces during a worked expert promotion that needs definitional depth.
|
||||
2. **[ADR-0087](ADR-0087-rhetorical-style-axis.md) — Rhetorical Style Axis.** A third substantive selection axis sibling to anchor-lens. Lowest current priority — no active downstream consumer; register + anchor-lens already demonstrate the orthogonality pattern.
|
||||
|
||||
ADR-0080 (Contemplation Loop, Phase 1) and ADR-0110 (math expert-demo) have both landed — `mathematics_logic` is the first domain at `expert_demo=true`; the contemplation loop emits read-only `SPECULATIVE` findings from `frontier_compare` reports. The remaining three ratified domains (`physics`, `systems_software`, `hebrew_greek_textual_reasoning`) need their own promotion ADRs.
|
||||
ADR-0080 (Contemplation Loop, Phase 1), ADR-0110 (math expert-demo), and ADR-0111 (physics expert-demo) have all landed — `mathematics_logic` and `physics` are at `expert_demo=true`; the contemplation loop emits read-only `SPECULATIVE` findings from `frontier_compare` reports. The remaining two ratified domains (`systems_software`, `hebrew_greek_textual_reasoning`) need their own promotion ADRs.
|
||||
|
||||
### Open candidate directions (no ADR yet)
|
||||
|
||||
|
|
@ -88,12 +90,12 @@ ADR-0080 (Contemplation Loop, Phase 1) and ADR-0110 (math expert-demo) have both
|
|||
|
||||
## Accepted reasoning-capable domains
|
||||
|
||||
Per ADR-0106, `expert_demo` is **contract-gated**, not threshold-only: a domain row may carry `expert_demo=true` only when a reviewer-signed `expert_demo_claims` entry exists whose evidence-bundle digest reproduces byte-for-byte. ADR-0107 attempted the first worked promotion and the contract refused; ADR-0109 amended the threshold rules; ADR-0110 then successfully promoted `mathematics_logic` (the first domain at `expert_demo=true`). The other three ratified domains remain at `reasoning-capable` pending their own promotion ADRs.
|
||||
Per ADR-0106, `expert_demo` is **contract-gated**, not threshold-only: a domain row may carry `expert_demo=true` only when a reviewer-signed `expert_demo_claims` entry exists whose evidence-bundle digest reproduces byte-for-byte. ADR-0107 attempted the first worked promotion and the contract refused; ADR-0109 amended the threshold rules; ADR-0110 then successfully promoted `mathematics_logic` (the first domain at `expert_demo=true`); ADR-0111 promoted `physics` second without further contract change, retiring the "math-only" objection. The other two ratified domains remain at `reasoning-capable` pending their own promotion ADRs.
|
||||
|
||||
| Domain | Ratification ADR | Pack(s) | Evidence summary |
|
||||
|---|---|---|---|
|
||||
| `mathematics_logic` | ADR-0097 + ADR-0110 | `en_mathematics_logic_v1` | All nine ADR-0091 predicates pass; ledger row is **`expert-demo`** (first such promotion, ADR-0110); all three attached lanes meet ADR-0109 shape thresholds on public + holdout. |
|
||||
| `physics` | ADR-0100 | `en_physics_v1` | All nine predicates pass; causal/modal operator coverage meets threshold; ledger row is `reasoning-capable`; `expert_demo` remains false. |
|
||||
| `physics` | ADR-0100 + ADR-0111 | `en_physics_v1` | All nine predicates pass; causal/modal operator coverage meets threshold; ledger row is **`expert-demo`** (second such promotion, ADR-0111); `foundational_physics_ood` 117/117 public + 39/39 holdout; shares `inference_closure` + `fabrication_control` results with math (distinct digest via `domain_id`). |
|
||||
| `systems_software` | ADR-0101 | `en_systems_software_v1` | All nine predicates pass; transitive/causal operator coverage meets threshold; ledger row is `reasoning-capable`; `symbolic_logic` is the v1 closest-fit eval lane. |
|
||||
| `hebrew_greek_textual_reasoning` | ADR-0102 + ADR-0103 | `grc_logos_micro_v1`, `grc_logos_cognition_v1`, `he_logos_micro_v1`, `he_core_cognition_v1` | First multi-pack ratification; all four packs carry uniform contract fields; causal/contradiction operator coverage meets threshold; ADR-0103 attaches Hebrew and Koine Greek fluency lanes with `dev/public/holdout` coverage. |
|
||||
|
||||
|
|
@ -117,9 +119,9 @@ Runtime contracts for the chain are pinned in [`docs/runtime_contracts.md`](../r
|
|||
|
||||
ADR-0027 through ADR-0045 establish the identity / safety / ethics pack architecture with deterministic remediation, audit completeness, telemetry, operator readout, audit-tour demo, pack measurements, a worked-example medical ethics pack, and long-context comparison measurements.
|
||||
|
||||
### Evidence-governed domain chain — ADR-0091 through ADR-0110
|
||||
### Evidence-governed domain chain — ADR-0091 through ADR-0111
|
||||
|
||||
ADR-0091 through ADR-0110 establish the current domain-ratification substrate and the expert-demo promotion gate that distinguishes contract-passing from demonstrated:
|
||||
ADR-0091 through ADR-0111 establish the current domain-ratification substrate and the expert-demo promotion gate that distinguishes contract-passing from demonstrated:
|
||||
|
||||
```text
|
||||
contract definition (0091)
|
||||
|
|
@ -137,11 +139,13 @@ language-specific fluency lane attachment (0103)
|
|||
expert-demo promotion contract (0106 + 0109 amendment)
|
||||
↓
|
||||
worked expert-demo promotion (0110 — mathematics_logic, first)
|
||||
↓
|
||||
worked expert-demo promotion (0111 — physics, second; no contract change)
|
||||
```
|
||||
|
||||
No domain claim should be treated as mature merely because a pack exists. `reasoning-capable` means the nine ADR-0091 predicates pass; `expert-demo` requires a reviewer-signed evidence-bundle digest that reproduces byte-for-byte from on-disk lane results.
|
||||
|
||||
The contract has been demonstrated end-to-end: refused once honestly (ADR-0107), amended once cleanly (ADR-0109), succeeded once honestly (ADR-0110).
|
||||
The contract has been demonstrated end-to-end: refused once honestly (ADR-0107), amended once cleanly (ADR-0109), succeeded against `mathematics_logic` (ADR-0110), and succeeded against `physics` without further contract change (ADR-0111).
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -30,3 +30,11 @@ expert_demo_claims:
|
|||
evidence_revision: "adr-0110:reviewed:2026-05-22"
|
||||
signed_by: shay-j
|
||||
claim_digest: "94d74781e103854230c1a71590e4df2287f5d2e87832f1c29b8ec4618853c04b"
|
||||
- domain_id: physics
|
||||
evidence_lanes:
|
||||
- foundational_physics_ood
|
||||
- inference_closure
|
||||
- fabrication_control
|
||||
evidence_revision: "adr-0111:reviewed:2026-05-22"
|
||||
signed_by: shay-j
|
||||
claim_digest: "a104cad136f3219df05dc7ce6a78437c02f7b5827cd3cdce568db3acda6a43ed"
|
||||
|
|
|
|||
|
|
@ -0,0 +1,39 @@
|
|||
{"id": "FPO-HOLD_optics_C01_01", "construction": "C01", "construction_name": "simple_declarative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["lens", "focuses", "ray"], "word_order": ["lens", "focuses", "ray"]}, "accept_surfaces": ["lens focuses ray"]}
|
||||
{"id": "FPO-HOLD_optics_C01_02", "construction": "C01", "construction_name": "simple_declarative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["mirror", "reflects", "beam"], "word_order": ["mirror", "reflects", "beam"]}, "accept_surfaces": ["mirror reflects beam"]}
|
||||
{"id": "FPO-HOLD_optics_C01_03", "construction": "C01", "construction_name": "simple_declarative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["prism", "separates", "color"], "word_order": ["prism", "separates", "color"]}, "accept_surfaces": ["prism separates color"]}
|
||||
{"id": "FPO-HOLD_optics_C02_01", "construction": "C02", "construction_name": "negation", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "negated": true}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["lens", "not", "ray"], "word_order": ["lens", "not", "ray"]}}
|
||||
{"id": "FPO-HOLD_optics_C02_02", "construction": "C02", "construction_name": "negation", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "negated": true}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["mirror", "not", "beam"], "word_order": ["mirror", "not", "beam"]}}
|
||||
{"id": "FPO-HOLD_optics_C02_03", "construction": "C02", "construction_name": "negation", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "negated": true}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["prism", "not", "color"], "word_order": ["prism", "not", "color"]}}
|
||||
{"id": "FPO-HOLD_optics_C03_01", "construction": "C03", "construction_name": "conjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray"}, {"node_id": "n2", "subject": "mirror", "predicate": "reflects", "obj": "beam"}], "edges": [{"source": "n1", "target": "n2", "relation": "conjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["lens", "and", "mirror"], "word_order": ["lens", "and", "mirror"]}}
|
||||
{"id": "FPO-HOLD_optics_C03_02", "construction": "C03", "construction_name": "conjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam"}, {"node_id": "n2", "subject": "prism", "predicate": "separates", "obj": "color"}], "edges": [{"source": "n1", "target": "n2", "relation": "conjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["mirror", "and", "prism"], "word_order": ["mirror", "and", "prism"]}}
|
||||
{"id": "FPO-HOLD_optics_C03_03", "construction": "C03", "construction_name": "conjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color"}, {"node_id": "n2", "subject": "lens", "predicate": "focuses", "obj": "ray"}], "edges": [{"source": "n1", "target": "n2", "relation": "conjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["prism", "and", "lens"], "word_order": ["prism", "and", "lens"]}}
|
||||
{"id": "FPO-HOLD_optics_C04_01", "construction": "C04", "construction_name": "disjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray"}, {"node_id": "n2", "subject": "mirror", "predicate": "reflects", "obj": "beam"}], "edges": [{"source": "n1", "target": "n2", "relation": "disjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["lens", "or", "mirror"], "word_order": ["lens", "or", "mirror"]}}
|
||||
{"id": "FPO-HOLD_optics_C04_02", "construction": "C04", "construction_name": "disjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam"}, {"node_id": "n2", "subject": "prism", "predicate": "separates", "obj": "color"}], "edges": [{"source": "n1", "target": "n2", "relation": "disjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["mirror", "or", "prism"], "word_order": ["mirror", "or", "prism"]}}
|
||||
{"id": "FPO-HOLD_optics_C04_03", "construction": "C04", "construction_name": "disjunction", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color"}, {"node_id": "n2", "subject": "lens", "predicate": "focuses", "obj": "ray"}], "edges": [{"source": "n1", "target": "n2", "relation": "disjunction"}]}, "constraints": {"max_words": 14, "must_contain": ["prism", "or", "lens"], "word_order": ["prism", "or", "lens"]}}
|
||||
{"id": "FPO-HOLD_optics_C05_01", "construction": "C05", "construction_name": "complement", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam"}, {"node_id": "n2", "subject": "lens", "predicate": "focuses", "obj": "ray"}], "edges": [{"source": "n1", "target": "n2", "relation": "complement"}]}, "constraints": {"max_words": 14, "must_contain": ["mirror", "that", "lens"], "word_order": ["mirror", "that", "lens"]}}
|
||||
{"id": "FPO-HOLD_optics_C05_02", "construction": "C05", "construction_name": "complement", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color"}, {"node_id": "n2", "subject": "mirror", "predicate": "reflects", "obj": "beam"}], "edges": [{"source": "n1", "target": "n2", "relation": "complement"}]}, "constraints": {"max_words": 14, "must_contain": ["prism", "that", "mirror"], "word_order": ["prism", "that", "mirror"]}}
|
||||
{"id": "FPO-HOLD_optics_C05_03", "construction": "C05", "construction_name": "complement", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray"}, {"node_id": "n2", "subject": "prism", "predicate": "separates", "obj": "color"}], "edges": [{"source": "n1", "target": "n2", "relation": "complement"}]}, "constraints": {"max_words": 14, "must_contain": ["lens", "that", "prism"], "word_order": ["lens", "that", "prism"]}}
|
||||
{"id": "FPO-HOLD_optics_C06_01", "construction": "C06", "construction_name": "relative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray"}, {"node_id": "n2", "subject": "lens", "predicate": "reflects", "obj": "beam"}], "edges": [{"source": "n1", "target": "n2", "relation": "relative"}]}, "constraints": {"max_words": 14, "must_contain": ["lens", "which", "beam", "ray"]}, "accept_surfaces": ["lens, which reflects beam, focuses ray", "lens which reflects beam focuses ray"]}
|
||||
{"id": "FPO-HOLD_optics_C06_02", "construction": "C06", "construction_name": "relative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam"}, {"node_id": "n2", "subject": "mirror", "predicate": "separates", "obj": "color"}], "edges": [{"source": "n1", "target": "n2", "relation": "relative"}]}, "constraints": {"max_words": 14, "must_contain": ["mirror", "which", "color", "beam"]}, "accept_surfaces": ["mirror, which separates color, reflects beam", "mirror which separates color reflects beam"]}
|
||||
{"id": "FPO-HOLD_optics_C06_03", "construction": "C06", "construction_name": "relative", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color"}, {"node_id": "n2", "subject": "prism", "predicate": "focuses", "obj": "ray"}], "edges": [{"source": "n1", "target": "n2", "relation": "relative"}]}, "constraints": {"max_words": 14, "must_contain": ["prism", "which", "ray", "color"]}, "accept_surfaces": ["prism, which focuses ray, separates color", "prism which focuses ray separates color"]}
|
||||
{"id": "FPO-HOLD_optics_C07_01", "construction": "C07", "construction_name": "universal", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "quantifier": "all"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["all", "lenses", "ray"], "word_order": ["all", "lenses", "ray"]}}
|
||||
{"id": "FPO-HOLD_optics_C07_02", "construction": "C07", "construction_name": "universal", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "quantifier": "all"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["all", "mirrors", "beam"], "word_order": ["all", "mirrors", "beam"]}}
|
||||
{"id": "FPO-HOLD_optics_C07_03", "construction": "C07", "construction_name": "universal", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "quantifier": "all"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["all", "prisms", "color"], "word_order": ["all", "prisms", "color"]}}
|
||||
{"id": "FPO-HOLD_optics_C08_01", "construction": "C08", "construction_name": "existential", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "quantifier": "some"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["some", "lenses", "ray"], "word_order": ["some", "lenses", "ray"]}}
|
||||
{"id": "FPO-HOLD_optics_C08_02", "construction": "C08", "construction_name": "existential", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "quantifier": "some"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["some", "mirrors", "beam"], "word_order": ["some", "mirrors", "beam"]}}
|
||||
{"id": "FPO-HOLD_optics_C08_03", "construction": "C08", "construction_name": "existential", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "quantifier": "some"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["some", "prisms", "color"], "word_order": ["some", "prisms", "color"]}}
|
||||
{"id": "FPO-HOLD_optics_C09_01", "construction": "C09", "construction_name": "past_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "tense": "past"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["lens", "ray"], "word_order": ["lens", "ray"]}}
|
||||
{"id": "FPO-HOLD_optics_C09_02", "construction": "C09", "construction_name": "past_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "tense": "past"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["mirror", "beam"], "word_order": ["mirror", "beam"]}}
|
||||
{"id": "FPO-HOLD_optics_C09_03", "construction": "C09", "construction_name": "past_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "tense": "past"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["prism", "color"], "word_order": ["prism", "color"]}}
|
||||
{"id": "FPO-HOLD_optics_C10_01", "construction": "C10", "construction_name": "present_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "tense": "present"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["lens", "focuses", "ray"], "word_order": ["lens", "focuses", "ray"]}, "accept_surfaces": ["lens focuses ray"]}
|
||||
{"id": "FPO-HOLD_optics_C10_02", "construction": "C10", "construction_name": "present_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "tense": "present"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["mirror", "reflects", "beam"], "word_order": ["mirror", "reflects", "beam"]}, "accept_surfaces": ["mirror reflects beam"]}
|
||||
{"id": "FPO-HOLD_optics_C10_03", "construction": "C10", "construction_name": "present_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "tense": "present"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["prism", "separates", "color"], "word_order": ["prism", "separates", "color"]}, "accept_surfaces": ["prism separates color"]}
|
||||
{"id": "FPO-HOLD_optics_C11_01", "construction": "C11", "construction_name": "future_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "tense": "future"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["lens", "will", "ray"], "word_order": ["lens", "will", "ray"]}}
|
||||
{"id": "FPO-HOLD_optics_C11_02", "construction": "C11", "construction_name": "future_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "tense": "future"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["mirror", "will", "beam"], "word_order": ["mirror", "will", "beam"]}}
|
||||
{"id": "FPO-HOLD_optics_C11_03", "construction": "C11", "construction_name": "future_tense", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "tense": "future"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["prism", "will", "color"], "word_order": ["prism", "will", "color"]}}
|
||||
{"id": "FPO-HOLD_optics_C12_01", "construction": "C12", "construction_name": "perfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "aspect": "perfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["lens", "has", "ray"], "word_order": ["lens", "has", "ray"]}}
|
||||
{"id": "FPO-HOLD_optics_C12_02", "construction": "C12", "construction_name": "perfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "aspect": "perfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["mirror", "has", "beam"], "word_order": ["mirror", "has", "beam"]}}
|
||||
{"id": "FPO-HOLD_optics_C12_03", "construction": "C12", "construction_name": "perfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "aspect": "perfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["prism", "has", "color"], "word_order": ["prism", "has", "color"]}}
|
||||
{"id": "FPO-HOLD_optics_C13_01", "construction": "C13", "construction_name": "imperfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "lens", "predicate": "focuses", "obj": "ray", "aspect": "imperfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["lens", "is", "ray"], "word_order": ["lens", "is", "ray"]}}
|
||||
{"id": "FPO-HOLD_optics_C13_02", "construction": "C13", "construction_name": "imperfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "mirror", "predicate": "reflects", "obj": "beam", "aspect": "imperfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["mirror", "is", "beam"], "word_order": ["mirror", "is", "beam"]}}
|
||||
{"id": "FPO-HOLD_optics_C13_03", "construction": "C13", "construction_name": "imperfective", "proposition_graph": {"nodes": [{"node_id": "n1", "subject": "prism", "predicate": "separates", "obj": "color", "aspect": "imperfective"}], "edges": []}, "constraints": {"max_words": 12, "must_contain": ["prism", "is", "color"], "word_order": ["prism", "is", "color"]}}
|
||||
|
|
@ -0,0 +1,340 @@
|
|||
{
|
||||
"cases": [
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C01_01",
|
||||
"construction": "C01",
|
||||
"construction_name": "simple_declarative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Lens focuses ray"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C01_02",
|
||||
"construction": "C01",
|
||||
"construction_name": "simple_declarative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Mirror reflects beam"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C01_03",
|
||||
"construction": "C01",
|
||||
"construction_name": "simple_declarative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Prism separates color"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C02_01",
|
||||
"construction": "C02",
|
||||
"construction_name": "negation",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Lens does not focus ray"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C02_02",
|
||||
"construction": "C02",
|
||||
"construction_name": "negation",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Mirror does not reflect beam"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C02_03",
|
||||
"construction": "C02",
|
||||
"construction_name": "negation",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Prism does not separate color"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C03_01",
|
||||
"construction": "C03",
|
||||
"construction_name": "conjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Lens focuses ray and mirror reflects beam"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C03_02",
|
||||
"construction": "C03",
|
||||
"construction_name": "conjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Mirror reflects beam and prism separates color"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C03_03",
|
||||
"construction": "C03",
|
||||
"construction_name": "conjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Prism separates color and lens focuses ray"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C04_01",
|
||||
"construction": "C04",
|
||||
"construction_name": "disjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Lens focuses ray or mirror reflects beam"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C04_02",
|
||||
"construction": "C04",
|
||||
"construction_name": "disjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Mirror reflects beam or prism separates color"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C04_03",
|
||||
"construction": "C04",
|
||||
"construction_name": "disjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Prism separates color or lens focuses ray"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C05_01",
|
||||
"construction": "C05",
|
||||
"construction_name": "complement",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Mirror reflects that lens focuses ray"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C05_02",
|
||||
"construction": "C05",
|
||||
"construction_name": "complement",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Prism separates that mirror reflects beam"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C05_03",
|
||||
"construction": "C05",
|
||||
"construction_name": "complement",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Lens focuses that prism separates color"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C06_01",
|
||||
"construction": "C06",
|
||||
"construction_name": "relative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Lens, which reflects beam, focuses ray"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C06_02",
|
||||
"construction": "C06",
|
||||
"construction_name": "relative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Mirror, which separates color, reflects beam"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C06_03",
|
||||
"construction": "C06",
|
||||
"construction_name": "relative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Prism, which focuses ray, separates color"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C07_01",
|
||||
"construction": "C07",
|
||||
"construction_name": "universal",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "All lenses focus ray"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C07_02",
|
||||
"construction": "C07",
|
||||
"construction_name": "universal",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "All mirrors reflect beam"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C07_03",
|
||||
"construction": "C07",
|
||||
"construction_name": "universal",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "All prisms separate color"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C08_01",
|
||||
"construction": "C08",
|
||||
"construction_name": "existential",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Some lenses focus ray"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C08_02",
|
||||
"construction": "C08",
|
||||
"construction_name": "existential",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Some mirrors reflect beam"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C08_03",
|
||||
"construction": "C08",
|
||||
"construction_name": "existential",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Some prisms separate color"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C09_01",
|
||||
"construction": "C09",
|
||||
"construction_name": "past_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Lens focussed ray"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C09_02",
|
||||
"construction": "C09",
|
||||
"construction_name": "past_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Mirror reflected beam"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C09_03",
|
||||
"construction": "C09",
|
||||
"construction_name": "past_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Prism separated color"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C10_01",
|
||||
"construction": "C10",
|
||||
"construction_name": "present_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Lens focuses ray"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C10_02",
|
||||
"construction": "C10",
|
||||
"construction_name": "present_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Mirror reflects beam"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C10_03",
|
||||
"construction": "C10",
|
||||
"construction_name": "present_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Prism separates color"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C11_01",
|
||||
"construction": "C11",
|
||||
"construction_name": "future_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Lens will focus ray"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C11_02",
|
||||
"construction": "C11",
|
||||
"construction_name": "future_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Mirror will reflect beam"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C11_03",
|
||||
"construction": "C11",
|
||||
"construction_name": "future_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Prism will separate color"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C12_01",
|
||||
"construction": "C12",
|
||||
"construction_name": "perfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Lens has focussed ray"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C12_02",
|
||||
"construction": "C12",
|
||||
"construction_name": "perfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Mirror has reflected beam"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C12_03",
|
||||
"construction": "C12",
|
||||
"construction_name": "perfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Prism has separated color"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C13_01",
|
||||
"construction": "C13",
|
||||
"construction_name": "imperfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Lens is focussing ray"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C13_02",
|
||||
"construction": "C13",
|
||||
"construction_name": "imperfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Mirror is reflecting beam"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-HOLD_optics_C13_03",
|
||||
"construction": "C13",
|
||||
"construction_name": "imperfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Prism is separating color"
|
||||
}
|
||||
],
|
||||
"lane": "foundational_physics_ood",
|
||||
"metrics": {
|
||||
"accuracy": 1.0,
|
||||
"by_construction": {
|
||||
"C01": 1.0,
|
||||
"C02": 1.0,
|
||||
"C03": 1.0,
|
||||
"C04": 1.0,
|
||||
"C05": 1.0,
|
||||
"C06": 1.0,
|
||||
"C07": 1.0,
|
||||
"C08": 1.0,
|
||||
"C09": 1.0,
|
||||
"C10": 1.0,
|
||||
"C11": 1.0,
|
||||
"C12": 1.0,
|
||||
"C13": 1.0
|
||||
},
|
||||
"passed": 39,
|
||||
"total": 39
|
||||
},
|
||||
"split": "holdout",
|
||||
"timestamp": "2026-05-22T21:20:36.429176+00:00",
|
||||
"version": "v1"
|
||||
}
|
||||
|
|
@ -0,0 +1,964 @@
|
|||
{
|
||||
"cases": [
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C01_01",
|
||||
"construction": "C01",
|
||||
"construction_name": "simple_declarative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Force moves object"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C01_02",
|
||||
"construction": "C01",
|
||||
"construction_name": "simple_declarative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Torque rotates wheel"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C01_03",
|
||||
"construction": "C01",
|
||||
"construction_name": "simple_declarative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Impulse changes momentum"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C02_01",
|
||||
"construction": "C02",
|
||||
"construction_name": "negation",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Force does not move object"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C02_02",
|
||||
"construction": "C02",
|
||||
"construction_name": "negation",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Torque does not rotate wheel"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C02_03",
|
||||
"construction": "C02",
|
||||
"construction_name": "negation",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Impulse does not change momentum"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C03_01",
|
||||
"construction": "C03",
|
||||
"construction_name": "conjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Force moves object and torque rotates wheel"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C03_02",
|
||||
"construction": "C03",
|
||||
"construction_name": "conjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Torque rotates wheel and impulse changes momentum"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C03_03",
|
||||
"construction": "C03",
|
||||
"construction_name": "conjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Impulse changes momentum and force moves object"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C04_01",
|
||||
"construction": "C04",
|
||||
"construction_name": "disjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Force moves object or torque rotates wheel"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C04_02",
|
||||
"construction": "C04",
|
||||
"construction_name": "disjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Torque rotates wheel or impulse changes momentum"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C04_03",
|
||||
"construction": "C04",
|
||||
"construction_name": "disjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Impulse changes momentum or force moves object"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C05_01",
|
||||
"construction": "C05",
|
||||
"construction_name": "complement",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Torque rotates that force moves object"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C05_02",
|
||||
"construction": "C05",
|
||||
"construction_name": "complement",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Impulse changes that torque rotates wheel"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C05_03",
|
||||
"construction": "C05",
|
||||
"construction_name": "complement",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Force moves that impulse changes momentum"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C06_01",
|
||||
"construction": "C06",
|
||||
"construction_name": "relative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Force, which rotates wheel, moves object"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C06_02",
|
||||
"construction": "C06",
|
||||
"construction_name": "relative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Torque, which changes momentum, rotates wheel"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C06_03",
|
||||
"construction": "C06",
|
||||
"construction_name": "relative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Impulse, which moves object, changes momentum"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C07_01",
|
||||
"construction": "C07",
|
||||
"construction_name": "universal",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "All forces move object"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C07_02",
|
||||
"construction": "C07",
|
||||
"construction_name": "universal",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "All torques rotate wheel"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C07_03",
|
||||
"construction": "C07",
|
||||
"construction_name": "universal",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "All impulses change momentum"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C08_01",
|
||||
"construction": "C08",
|
||||
"construction_name": "existential",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Some forces move object"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C08_02",
|
||||
"construction": "C08",
|
||||
"construction_name": "existential",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Some torques rotate wheel"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C08_03",
|
||||
"construction": "C08",
|
||||
"construction_name": "existential",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Some impulses change momentum"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C09_01",
|
||||
"construction": "C09",
|
||||
"construction_name": "past_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Force moved object"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C09_02",
|
||||
"construction": "C09",
|
||||
"construction_name": "past_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Torque rotated wheel"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C09_03",
|
||||
"construction": "C09",
|
||||
"construction_name": "past_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Impulse changed momentum"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C10_01",
|
||||
"construction": "C10",
|
||||
"construction_name": "present_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Force moves object"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C10_02",
|
||||
"construction": "C10",
|
||||
"construction_name": "present_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Torque rotates wheel"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C10_03",
|
||||
"construction": "C10",
|
||||
"construction_name": "present_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Impulse changes momentum"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C11_01",
|
||||
"construction": "C11",
|
||||
"construction_name": "future_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Force will move object"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C11_02",
|
||||
"construction": "C11",
|
||||
"construction_name": "future_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Torque will rotate wheel"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C11_03",
|
||||
"construction": "C11",
|
||||
"construction_name": "future_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Impulse will change momentum"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C12_01",
|
||||
"construction": "C12",
|
||||
"construction_name": "perfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Force has moved object"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C12_02",
|
||||
"construction": "C12",
|
||||
"construction_name": "perfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Torque has rotated wheel"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C12_03",
|
||||
"construction": "C12",
|
||||
"construction_name": "perfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Impulse has changed momentum"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C13_01",
|
||||
"construction": "C13",
|
||||
"construction_name": "imperfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Force is moving object"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C13_02",
|
||||
"construction": "C13",
|
||||
"construction_name": "imperfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Torque is rotating wheel"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_mechanics_C13_03",
|
||||
"construction": "C13",
|
||||
"construction_name": "imperfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Impulse is changing momentum"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C01_01",
|
||||
"construction": "C01",
|
||||
"construction_name": "simple_declarative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Current powers circuit"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C01_02",
|
||||
"construction": "C01",
|
||||
"construction_name": "simple_declarative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Voltage drives charge"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C01_03",
|
||||
"construction": "C01",
|
||||
"construction_name": "simple_declarative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Resistor limits flow"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C02_01",
|
||||
"construction": "C02",
|
||||
"construction_name": "negation",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Current does not power circuit"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C02_02",
|
||||
"construction": "C02",
|
||||
"construction_name": "negation",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Voltage does not drive charge"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C02_03",
|
||||
"construction": "C02",
|
||||
"construction_name": "negation",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Resistor does not limit flow"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C03_01",
|
||||
"construction": "C03",
|
||||
"construction_name": "conjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Current powers circuit and voltage drives charge"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C03_02",
|
||||
"construction": "C03",
|
||||
"construction_name": "conjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Voltage drives charge and resistor limits flow"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C03_03",
|
||||
"construction": "C03",
|
||||
"construction_name": "conjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Resistor limits flow and current powers circuit"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C04_01",
|
||||
"construction": "C04",
|
||||
"construction_name": "disjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Current powers circuit or voltage drives charge"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C04_02",
|
||||
"construction": "C04",
|
||||
"construction_name": "disjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Voltage drives charge or resistor limits flow"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C04_03",
|
||||
"construction": "C04",
|
||||
"construction_name": "disjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Resistor limits flow or current powers circuit"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C05_01",
|
||||
"construction": "C05",
|
||||
"construction_name": "complement",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Voltage drives that current powers circuit"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C05_02",
|
||||
"construction": "C05",
|
||||
"construction_name": "complement",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Resistor limits that voltage drives charge"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C05_03",
|
||||
"construction": "C05",
|
||||
"construction_name": "complement",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Current powers that resistor limits flow"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C06_01",
|
||||
"construction": "C06",
|
||||
"construction_name": "relative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Current, which drives charge, powers circuit"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C06_02",
|
||||
"construction": "C06",
|
||||
"construction_name": "relative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Voltage, which limits flow, drives charge"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C06_03",
|
||||
"construction": "C06",
|
||||
"construction_name": "relative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Resistor, which powers circuit, limits flow"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C07_01",
|
||||
"construction": "C07",
|
||||
"construction_name": "universal",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "All currents power circuit"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C07_02",
|
||||
"construction": "C07",
|
||||
"construction_name": "universal",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "All voltages drive charge"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C07_03",
|
||||
"construction": "C07",
|
||||
"construction_name": "universal",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "All resistors limit flow"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C08_01",
|
||||
"construction": "C08",
|
||||
"construction_name": "existential",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Some currents power circuit"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C08_02",
|
||||
"construction": "C08",
|
||||
"construction_name": "existential",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Some voltages drive charge"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C08_03",
|
||||
"construction": "C08",
|
||||
"construction_name": "existential",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Some resistors limit flow"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C09_01",
|
||||
"construction": "C09",
|
||||
"construction_name": "past_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Current powerred circuit"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C09_02",
|
||||
"construction": "C09",
|
||||
"construction_name": "past_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Voltage drove charge"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C09_03",
|
||||
"construction": "C09",
|
||||
"construction_name": "past_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Resistor limitted flow"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C10_01",
|
||||
"construction": "C10",
|
||||
"construction_name": "present_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Current powers circuit"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C10_02",
|
||||
"construction": "C10",
|
||||
"construction_name": "present_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Voltage drives charge"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C10_03",
|
||||
"construction": "C10",
|
||||
"construction_name": "present_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Resistor limits flow"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C11_01",
|
||||
"construction": "C11",
|
||||
"construction_name": "future_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Current will power circuit"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C11_02",
|
||||
"construction": "C11",
|
||||
"construction_name": "future_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Voltage will drive charge"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C11_03",
|
||||
"construction": "C11",
|
||||
"construction_name": "future_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Resistor will limit flow"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C12_01",
|
||||
"construction": "C12",
|
||||
"construction_name": "perfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Current has powerred circuit"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C12_02",
|
||||
"construction": "C12",
|
||||
"construction_name": "perfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Voltage has driven charge"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C12_03",
|
||||
"construction": "C12",
|
||||
"construction_name": "perfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Resistor has limitted flow"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C13_01",
|
||||
"construction": "C13",
|
||||
"construction_name": "imperfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Current is powerring circuit"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C13_02",
|
||||
"construction": "C13",
|
||||
"construction_name": "imperfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Voltage is driving charge"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_electricity_C13_03",
|
||||
"construction": "C13",
|
||||
"construction_name": "imperfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Resistor is limitting flow"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C01_01",
|
||||
"construction": "C01",
|
||||
"construction_name": "simple_declarative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Heat raises temperature"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C01_02",
|
||||
"construction": "C01",
|
||||
"construction_name": "simple_declarative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Piston compresses gas"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C01_03",
|
||||
"construction": "C01",
|
||||
"construction_name": "simple_declarative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Entropy tracks disorder"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C02_01",
|
||||
"construction": "C02",
|
||||
"construction_name": "negation",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Heat does not rais temperature"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C02_02",
|
||||
"construction": "C02",
|
||||
"construction_name": "negation",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Piston does not compress gas"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C02_03",
|
||||
"construction": "C02",
|
||||
"construction_name": "negation",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Entropy does not track disorder"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C03_01",
|
||||
"construction": "C03",
|
||||
"construction_name": "conjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Heat raises temperature and piston compresses gas"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C03_02",
|
||||
"construction": "C03",
|
||||
"construction_name": "conjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Piston compresses gas and entropy tracks disorder"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C03_03",
|
||||
"construction": "C03",
|
||||
"construction_name": "conjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Entropy tracks disorder and heat raises temperature"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C04_01",
|
||||
"construction": "C04",
|
||||
"construction_name": "disjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Heat raises temperature or piston compresses gas"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C04_02",
|
||||
"construction": "C04",
|
||||
"construction_name": "disjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Piston compresses gas or entropy tracks disorder"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C04_03",
|
||||
"construction": "C04",
|
||||
"construction_name": "disjunction",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Entropy tracks disorder or heat raises temperature"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C05_01",
|
||||
"construction": "C05",
|
||||
"construction_name": "complement",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Piston compresses that heat raises temperature"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C05_02",
|
||||
"construction": "C05",
|
||||
"construction_name": "complement",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Entropy tracks that piston compresses gas"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C05_03",
|
||||
"construction": "C05",
|
||||
"construction_name": "complement",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Heat raises that entropy tracks disorder"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C06_01",
|
||||
"construction": "C06",
|
||||
"construction_name": "relative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Heat, which compresses gas, raises temperature"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C06_02",
|
||||
"construction": "C06",
|
||||
"construction_name": "relative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Piston, which tracks disorder, compresses gas"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C06_03",
|
||||
"construction": "C06",
|
||||
"construction_name": "relative",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Entropy, which raises temperature, tracks disorder"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C07_01",
|
||||
"construction": "C07",
|
||||
"construction_name": "universal",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "All heats rais temperature"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C07_02",
|
||||
"construction": "C07",
|
||||
"construction_name": "universal",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "All pistons compress gas"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C07_03",
|
||||
"construction": "C07",
|
||||
"construction_name": "universal",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "All entropies track disorder"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C08_01",
|
||||
"construction": "C08",
|
||||
"construction_name": "existential",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Some heats rais temperature"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C08_02",
|
||||
"construction": "C08",
|
||||
"construction_name": "existential",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Some pistons compress gas"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C08_03",
|
||||
"construction": "C08",
|
||||
"construction_name": "existential",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Some entropies track disorder"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C09_01",
|
||||
"construction": "C09",
|
||||
"construction_name": "past_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Heat raised temperature"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C09_02",
|
||||
"construction": "C09",
|
||||
"construction_name": "past_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Piston compressed gas"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C09_03",
|
||||
"construction": "C09",
|
||||
"construction_name": "past_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Entropy tracked disorder"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C10_01",
|
||||
"construction": "C10",
|
||||
"construction_name": "present_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Heat raises temperature"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C10_02",
|
||||
"construction": "C10",
|
||||
"construction_name": "present_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Piston compresses gas"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C10_03",
|
||||
"construction": "C10",
|
||||
"construction_name": "present_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Entropy tracks disorder"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C11_01",
|
||||
"construction": "C11",
|
||||
"construction_name": "future_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Heat will rais temperature"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C11_02",
|
||||
"construction": "C11",
|
||||
"construction_name": "future_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Piston will compress gas"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C11_03",
|
||||
"construction": "C11",
|
||||
"construction_name": "future_tense",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Entropy will track disorder"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C12_01",
|
||||
"construction": "C12",
|
||||
"construction_name": "perfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Heat has raised temperature"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C12_02",
|
||||
"construction": "C12",
|
||||
"construction_name": "perfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Piston has compressed gas"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C12_03",
|
||||
"construction": "C12",
|
||||
"construction_name": "perfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Entropy has tracked disorder"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C13_01",
|
||||
"construction": "C13",
|
||||
"construction_name": "imperfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Heat is raising temperature"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C13_02",
|
||||
"construction": "C13",
|
||||
"construction_name": "imperfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Piston is compressing gas"
|
||||
},
|
||||
{
|
||||
"case_id": "FPO-PUB_thermodynamics_C13_03",
|
||||
"construction": "C13",
|
||||
"construction_name": "imperfective",
|
||||
"failure_reasons": [],
|
||||
"passed": true,
|
||||
"surface": "Entropy is tracking disorder"
|
||||
}
|
||||
],
|
||||
"lane": "foundational_physics_ood",
|
||||
"metrics": {
|
||||
"accuracy": 1.0,
|
||||
"by_construction": {
|
||||
"C01": 1.0,
|
||||
"C02": 1.0,
|
||||
"C03": 1.0,
|
||||
"C04": 1.0,
|
||||
"C05": 1.0,
|
||||
"C06": 1.0,
|
||||
"C07": 1.0,
|
||||
"C08": 1.0,
|
||||
"C09": 1.0,
|
||||
"C10": 1.0,
|
||||
"C11": 1.0,
|
||||
"C12": 1.0,
|
||||
"C13": 1.0
|
||||
},
|
||||
"passed": 117,
|
||||
"total": 117
|
||||
},
|
||||
"split": "public",
|
||||
"timestamp": "2026-05-22T21:20:18.569573+00:00",
|
||||
"version": "v1"
|
||||
}
|
||||
|
|
@ -90,7 +90,6 @@
|
|||
"detail": {
|
||||
"active_corpus_byte_identical": true,
|
||||
"chain_id": "cause_narrative_reveals_meaning",
|
||||
"transient_corpus": "/var/folders/kg/5xbm28qd7jl55j7lv3p001f40000gn/T/learning_loop_demo_ujzher4z/cognition_chains_v1.jsonl",
|
||||
"transient_lines_after": 23,
|
||||
"transient_lines_before": 22
|
||||
},
|
||||
|
|
|
|||
|
|
@ -1 +1 @@
|
|||
<!doctype html><html><head><meta charset='utf-8'><title>CORE Public Showcase v1</title><style>body{font-family:system-ui;max-width:880px;margin:2rem auto;padding:0 1rem;}h1{font-size:1.5rem;}h2{font-size:1.1rem;margin-top:1.5rem;}.claim{margin:.25rem 0;list-style:none;}.claim.fail{color:#a33;}.mark{font-weight:bold;display:inline-block;width:1.2rem;}code{background:#f3f3f3;padding:.05rem .2rem;border-radius:3px;}footer{margin-top:2rem;color:#888;font-size:.85rem;}</style></head><body><h1>CORE Public Showcase v1</h1><p>all_claims_supported: <strong>True</strong></p><section><h2>✓ determinism <small>(register-tour)</small></h2><p>Identical prompts produce identical trace hashes across runs under each shipped register.</p><ul><li class="claim ok"><span class="mark">✓</span> <code>all_claims_supported</code>: all claims supported</li></ul></section><section><h2>✓ honest_unknown <small>(fabrication-control-public)</small></h2><p>Composable-looking but unsupported prompts produce typed refusal with grounding_source = none.</p><ul><li class="claim ok"><span class="mark">✓</span> <code>refusal_recall_meets_threshold</code>: refusal_recall ≥ 0.95 on the fabrication-control public split.</li><li class="claim ok"><span class="mark">✓</span> <code>fabrication_rate_below_threshold</code>: fabrication_rate ≤ 0.01 on the fabrication-control public split.</li><li class="claim ok"><span class="mark">✓</span> <code>trace_evidence_present</code>: Every case exposes a grounding_source trace.</li></ul></section><section><h2>✓ reviewed_learning <small>(learning-loop)</small></h2><p>Speculative teaching is marked speculative until reviewed; after review, identical prompt produces a coherent answer.</p><ul><li class="claim ok"><span class="mark">✓</span> <code>all_claims_supported</code>: all claims supported</li></ul></section><section><h2>✓ multi_hop_trace <small>(multi-hop-trace)</small></h2><p>Multi-hop reasoning produces an answer plus a verifiable operator trace via the transitive-chain surface.</p><ul><li class="claim ok"><span class="mark">✓</span> <code>multi_hop_grounded_answer</code>: Prompt 'Does light reveal truth?' produces a teaching-grounded answer rather than a refusal.</li><li class="claim ok"><span class="mark">✓</span> <code>multi_hop_depth_two_or_more</code>: Surface names ≥ 3 distinct atoms (depth-2 transitive walk).</li><li class="claim ok"><span class="mark">✓</span> <code>multi_hop_walk_evidence_present</code>: Result exposes non-empty walk evidence.</li></ul></section><footer>revision: <code>4f640af40dcbe1f17f1b951040ce0addd6c776d3</code></footer></body></html>
|
||||
<!doctype html><html><head><meta charset='utf-8'><title>CORE Public Showcase v1</title><style>body{font-family:system-ui;max-width:880px;margin:2rem auto;padding:0 1rem;}h1{font-size:1.5rem;}h2{font-size:1.1rem;margin-top:1.5rem;}.claim{margin:.25rem 0;list-style:none;}.claim.fail{color:#a33;}.mark{font-weight:bold;display:inline-block;width:1.2rem;}code{background:#f3f3f3;padding:.05rem .2rem;border-radius:3px;}footer{margin-top:2rem;color:#888;font-size:.85rem;}</style></head><body><h1>CORE Public Showcase v1</h1><p>all_claims_supported: <strong>True</strong></p><section><h2>✓ determinism <small>(register-tour)</small></h2><p>Identical prompts produce identical trace hashes across runs under each shipped register.</p><ul><li class="claim ok"><span class="mark">✓</span> <code>all_claims_supported</code>: all claims supported</li></ul></section><section><h2>✓ honest_unknown <small>(fabrication-control-public)</small></h2><p>Composable-looking but unsupported prompts produce typed refusal with grounding_source = none.</p><ul><li class="claim ok"><span class="mark">✓</span> <code>refusal_recall_meets_threshold</code>: refusal_recall ≥ 0.95 on the fabrication-control public split.</li><li class="claim ok"><span class="mark">✓</span> <code>fabrication_rate_below_threshold</code>: fabrication_rate ≤ 0.01 on the fabrication-control public split.</li><li class="claim ok"><span class="mark">✓</span> <code>trace_evidence_present</code>: Every case exposes a grounding_source trace.</li></ul></section><section><h2>✓ reviewed_learning <small>(learning-loop)</small></h2><p>Speculative teaching is marked speculative until reviewed; after review, identical prompt produces a coherent answer.</p><ul><li class="claim ok"><span class="mark">✓</span> <code>all_claims_supported</code>: all claims supported</li></ul></section><section><h2>✓ multi_hop_trace <small>(multi-hop-trace)</small></h2><p>Multi-hop reasoning produces an answer plus a verifiable operator trace via the transitive-chain surface.</p><ul><li class="claim ok"><span class="mark">✓</span> <code>multi_hop_grounded_answer</code>: Prompt 'Does light reveal truth?' produces a teaching-grounded answer rather than a refusal.</li><li class="claim ok"><span class="mark">✓</span> <code>multi_hop_depth_two_or_more</code>: Surface names ≥ 3 distinct atoms (depth-2 transitive walk).</li><li class="claim ok"><span class="mark">✓</span> <code>multi_hop_walk_evidence_present</code>: Result exposes non-empty walk evidence.</li></ul></section><footer>revision: <code>bc90622e5335bb6ba52a83fceef9169d307f7f36</code></footer></body></html>
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
{
|
||||
"all_claims_supported": true,
|
||||
"claim_contract_version": 1,
|
||||
"generated_at_revision": "4f640af40dcbe1f17f1b951040ce0addd6c776d3",
|
||||
"generated_at_revision": "bc90622e5335bb6ba52a83fceef9169d307f7f36",
|
||||
"max_runtime_seconds": 30,
|
||||
"scenes": [
|
||||
{
|
||||
|
|
@ -76,14 +76,14 @@
|
|||
],
|
||||
"demo_id": "learning-loop",
|
||||
"evidence": {
|
||||
"all_claims_supported": "sha256:4fcda860a8cfc525"
|
||||
"all_claims_supported": "sha256:137a2a477a9f61b7"
|
||||
},
|
||||
"json_path": "evals/public_demo/results/latest/scenes/learning-loop.json",
|
||||
"scene_id": "reviewed_learning",
|
||||
"statement": "Speculative teaching is marked speculative until reviewed; after review, identical prompt produces a coherent answer.",
|
||||
"trace_features": {
|
||||
"all_claims_supported": "true",
|
||||
"report_sha256": "4fcda860a8cfc525ceb80042fff7c02613589217c42c06e532ef35e57c89cadc"
|
||||
"report_sha256": "137a2a477a9f61b79c4815edcb02b3213760ea72c5a6d5fd7699d87ee599e479"
|
||||
}
|
||||
},
|
||||
{
|
||||
|
|
|
|||
|
|
@ -92,12 +92,19 @@ class TestAdr0110ReplayDigestByteEquality:
|
|||
|
||||
|
||||
class TestAdr0110OtherDomainsUnaffected:
|
||||
def test_other_domains_stay_reasoning_capable(self) -> None:
|
||||
def test_math_stays_promoted(self) -> None:
|
||||
"""ADR-0110's promotion of mathematics_logic must persist.
|
||||
|
||||
Originally this asserted math was the only promoted domain.
|
||||
ADR-0111 added physics — the load-bearing invariant for
|
||||
ADR-0110 is that math stays promoted, not that no other
|
||||
domain is promoted alongside it.
|
||||
"""
|
||||
promoted = [
|
||||
row["domain"]
|
||||
for row in ledger_report()["domains"]
|
||||
if row["predicates"]["expert_demo"]
|
||||
]
|
||||
assert promoted == ["mathematics_logic"], (
|
||||
f"ADR-0110 must promote exactly mathematics_logic; got: {promoted}"
|
||||
assert "mathematics_logic" in promoted, (
|
||||
f"ADR-0110 promotion of mathematics_logic must persist; got: {promoted}"
|
||||
)
|
||||
|
|
|
|||
144
tests/test_adr_0111_physics_expert_demo.py
Normal file
144
tests/test_adr_0111_physics_expert_demo.py
Normal file
|
|
@ -0,0 +1,144 @@
|
|||
"""ADR-0111 — `physics` expert-demo promotion invariants.
|
||||
|
||||
Pins four load-bearing invariants:
|
||||
|
||||
1. ``adr_0111_physics_expert_demo_holds`` — ``ledger_report()`` reports
|
||||
``physics`` at ``status="expert-demo"`` with
|
||||
``predicates.expert_demo == True``.
|
||||
|
||||
2. ``adr_0111_replay_digest_byte_equality`` — re-deriving the
|
||||
evidence-bundle digest from the on-disk lane result files reproduces
|
||||
the signed ``claim_digest`` byte-for-byte (ADR-0106 §1.5).
|
||||
|
||||
3. ``adr_0111_other_domains_unaffected`` — ADR-0111 promotes exactly
|
||||
one new domain. ``mathematics_logic`` (ADR-0110) must remain
|
||||
promoted; ``systems_software``, ``hebrew_greek_textual_reasoning``,
|
||||
and ``philosophy_theology`` must remain at ``expert_demo=false``.
|
||||
|
||||
4. ``adr_0111_distinct_digest_from_adr_0110`` — physics digest and
|
||||
math digest must differ, demonstrating the bundle's
|
||||
``domain_id`` + ``evidence_revision`` fields produce distinct
|
||||
claims even when two of three evidence lanes are shared.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from core.capability.expert_demo import (
|
||||
derive_evidence_digest,
|
||||
materialise_lane_results,
|
||||
)
|
||||
from core.capability.reporting import _latest_eval_result, ledger_report
|
||||
from core.capability.reviewers import load_reviewer_registry
|
||||
from core.capability.sources import LEDGER_SOURCES
|
||||
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
|
||||
_PHYSICS_LANES = (
|
||||
"foundational_physics_ood",
|
||||
"inference_closure",
|
||||
"fabrication_control",
|
||||
)
|
||||
|
||||
|
||||
def _fetch(lane: str, split: str) -> dict[str, Any]:
|
||||
payload = _latest_eval_result(lane, "v1", split)
|
||||
metrics = dict(payload.get("metrics", {}) or {})
|
||||
if "by_class" not in metrics and "by_class" in payload:
|
||||
metrics["by_class"] = payload["by_class"]
|
||||
return metrics
|
||||
|
||||
|
||||
def _physics_row() -> dict:
|
||||
report = ledger_report()
|
||||
for row in report["domains"]:
|
||||
if row["domain"] == "physics":
|
||||
return row
|
||||
raise AssertionError("physics row missing from ledger_report()")
|
||||
|
||||
|
||||
def _registry():
|
||||
registry_path = _REPO_ROOT / LEDGER_SOURCES.reviewers
|
||||
return load_reviewer_registry(registry_path)
|
||||
|
||||
|
||||
def _physics_claim():
|
||||
claim = _registry().expert_demo_claim_for("physics")
|
||||
assert claim is not None, "expert_demo_claims entry for physics missing"
|
||||
return claim
|
||||
|
||||
|
||||
class TestAdr0111PhysicsExpertDemoHolds:
|
||||
def test_physics_row_is_expert_demo(self) -> None:
|
||||
row = _physics_row()
|
||||
assert row["status"] == "expert-demo"
|
||||
assert row["predicates"]["expert_demo"] is True
|
||||
|
||||
def test_signed_claim_is_present(self) -> None:
|
||||
claim = _physics_claim()
|
||||
assert set(claim.evidence_lanes) == set(_PHYSICS_LANES)
|
||||
assert claim.signed_by == "shay-j"
|
||||
assert claim.evidence_revision == "adr-0111:reviewed:2026-05-22"
|
||||
|
||||
|
||||
class TestAdr0111ReplayDigestByteEquality:
|
||||
def test_derived_digest_matches_signed_claim(self) -> None:
|
||||
claim = _physics_claim()
|
||||
lane_results = materialise_lane_results(
|
||||
_PHYSICS_LANES, fetch_split=_fetch
|
||||
)
|
||||
derived = derive_evidence_digest(
|
||||
domain_id="physics",
|
||||
evidence_revision=claim.evidence_revision,
|
||||
evidence_lanes=claim.evidence_lanes,
|
||||
lane_results=lane_results,
|
||||
)
|
||||
assert derived == claim.claim_digest
|
||||
|
||||
|
||||
class TestAdr0111OtherDomainsUnaffected:
|
||||
def test_math_remains_promoted(self) -> None:
|
||||
promoted = {
|
||||
row["domain"]
|
||||
for row in ledger_report()["domains"]
|
||||
if row["predicates"]["expert_demo"]
|
||||
}
|
||||
assert "mathematics_logic" in promoted
|
||||
assert "physics" in promoted
|
||||
|
||||
def test_unpromoted_domains_stay_reasoning_capable(self) -> None:
|
||||
unpromoted_expected = {
|
||||
"systems_software",
|
||||
"hebrew_greek_textual_reasoning",
|
||||
"philosophy_theology",
|
||||
}
|
||||
report = ledger_report()
|
||||
for row in report["domains"]:
|
||||
if row["domain"] in unpromoted_expected:
|
||||
assert row["status"] == "reasoning-capable", (
|
||||
f"{row['domain']} expected reasoning-capable, "
|
||||
f"got {row['status']}"
|
||||
)
|
||||
assert row["predicates"]["expert_demo"] is False
|
||||
|
||||
|
||||
class TestAdr0111DistinctDigestFromAdr0110:
|
||||
def test_physics_digest_differs_from_math_digest(self) -> None:
|
||||
registry = _registry()
|
||||
physics_claim = registry.expert_demo_claim_for("physics")
|
||||
math_claim = registry.expert_demo_claim_for("mathematics_logic")
|
||||
assert physics_claim is not None
|
||||
assert math_claim is not None
|
||||
assert physics_claim.claim_digest != math_claim.claim_digest
|
||||
|
||||
def test_two_of_three_lanes_are_shared(self) -> None:
|
||||
"""Same shared lanes, distinct digests — proves domain_id matters."""
|
||||
registry = _registry()
|
||||
physics_claim = registry.expert_demo_claim_for("physics")
|
||||
math_claim = registry.expert_demo_claim_for("mathematics_logic")
|
||||
assert physics_claim is not None and math_claim is not None
|
||||
shared = set(physics_claim.evidence_lanes) & set(math_claim.evidence_lanes)
|
||||
assert shared == {"inference_closure", "fabrication_control"}
|
||||
|
|
@ -152,8 +152,10 @@ def test_ledger_status_is_predicate_derived() -> None:
|
|||
assert math["open_gaps"] == []
|
||||
|
||||
physics = rows["physics"]
|
||||
assert physics["status"] == "reasoning-capable"
|
||||
# ADR-0111 — second expert-demo promotion lands on physics.
|
||||
assert physics["status"] == "expert-demo"
|
||||
assert physics["predicates"]["reasoning_capable"] is True
|
||||
assert physics["predicates"]["expert_demo"] is True
|
||||
assert physics["open_gaps"] == []
|
||||
|
||||
he_grc = rows["hebrew_greek_textual_reasoning"]
|
||||
|
|
|
|||
Loading…
Reference in a new issue