6a31559921
New evals/deduction_serve/ lane scores the PRODUCTION serving decider (the exact comprehend -> to_deductive_logic -> evaluate_entailment_with_trace pipeline chat/deduction_surface.py runs) end-to-end from raw text -- distinct from evals/deductive_logic (bare engine vs formula strings) and evals/comprehension/propositional_runner.py (reader fidelity vs the independent oracle, not the production engine). This is the only lane proving the capability core chat actually serves. 27 hand-authored cases (gold computed by independent logical reasoning, not copied from a first engine run), 4 classes: entailed/refuted/unknown/ declined. 27/27 correct, wrong=0. Wired into core test --suite deductive (tests/test_deduction_serve_lane.py) and SHA-pinned in scripts/verify_lane_shas.py (deduction_serve_v1). Honesty check during authoring: a case intended as 'entailed' (contraposition, 'Therefore if not q then not p') actually declined -- tracing why found a genuine reader-grammar boundary (negation cannot nest inside an if/then clause; generate/meaning_graph/reader.py's _chunk rejects it). Reclassified to declined/out_of_band_nested_negation (documented in contract.md) rather than forcing an artificial pass, and added a replacement entailed case (three-hop chain) to keep coverage. Out-of-scope finding (documented, not fixed here): running scripts/verify_lane_shas.py --update to compute this lane's pin also re-executed every OTHER registered lane and surfaced two pre-existing, unrelated problems on main -- miner_loop_closure/curriculum_loop_closure/ demo_composition regenerate non-deterministic content IDs (their committed pins don't reproduce even on a clean checkout), and public_demo errors outright (matches the known env-timeout flake in project memory). Reverted all four lanes' results/*.json and PINNED_SHAS entries to their original committed values -- this PR's only PINNED_SHAS change is the new deduction_serve_v1 entry. [Verification]: smoke 180 passed; cognition 122 passed/1 skipped; core test --suite deductive 25 passed; evals.deduction_serve.runner 27/27 wrong=0; pinned SHA independently re-verified against the committed report.json bytes.
118 lines
6.4 KiB
Markdown
118 lines
6.4 KiB
Markdown
# Deduction-serve arc — Phase 2 (end-to-end eval lane + wrong=0 gate), 2026-07-23
|
|
|
|
**Base:** `main` @ `6a54d27a`. **Branch:** `feat/deduction-serve-phase1` (stacked
|
|
on top of the Phase 1 commit, same branch — Phase 2 is additive to Phase 1's
|
|
serving path, not a separate capability).
|
|
**Depends on:** Phase 1 (`chat/deduction_surface.py`, the composer this lane scores).
|
|
|
|
## What shipped
|
|
|
|
A new, dedicated, SHA-pinnable capability lane —
|
|
**`evals/deduction_serve/`** — that scores the *production serving
|
|
decider* end-to-end from raw text, closing the gap neither existing lane
|
|
covers:
|
|
|
|
| Lane | What it scores | Decision procedure used |
|
|
|---|---|---|
|
|
| `evals/deductive_logic` | the bare `entail.py` engine | `evaluate_entailment` on hand-authored **formula strings** |
|
|
| `evals/comprehension/propositional_runner.py` | reader **fidelity** | the **independent oracle**, not the production engine |
|
|
| `evals/deduction_serve` (new) | the **production serving pipeline**, raw text in | `comprehend → to_deductive_logic → evaluate_entailment_with_trace` — the exact calls `chat/deduction_surface.py` makes |
|
|
|
|
## New files
|
|
|
|
- **`evals/deduction_serve/v1/cases.jsonl`** — 27 hand-authored cases, gold
|
|
computed independently by direct logical reasoning (not copied from a
|
|
first engine run — see "Honesty check" below). Four gold classes:
|
|
`entailed` (8), `refuted` (5), `unknown` (5), `declined` (9 — covering
|
|
inconsistent premises, categorical/syllogism shape, multi-word English,
|
|
and nested negation).
|
|
- **`evals/deduction_serve/runner.py`** — `decide(text)` calls the exact
|
|
pipeline the composer runs (typed outcome, not rendered prose — keeps
|
|
the pin stable against wording-only changes); `build_report` /
|
|
`build_combined_report` / `write_combined_report` mirror
|
|
`evals/deductive_logic/runner.py`'s pattern exactly.
|
|
- **`evals/deduction_serve/report.json`** — the committed, SHA-pinnable
|
|
artifact: `27/27 correct, wrong=0, all_correct=true`.
|
|
- **`evals/deduction_serve/contract.md`** — the lane contract, including
|
|
the three known Band v1 boundaries this corpus documents.
|
|
- **`tests/test_deduction_serve_lane.py`** — wrong=0 gate + a negative
|
|
control (a deliberately wrong gold label correctly flags `wrong=1`,
|
|
proving the gate isn't vacuously green).
|
|
|
|
## Changed files
|
|
|
|
- **`core/cli_test.py`** — `"deductive"` suite now runs
|
|
`tests/test_deduction_serve_lane.py` alongside the existing
|
|
`test_deductive_logic_entail.py` (a natural sibling).
|
|
- **`scripts/verify_lane_shas.py`** — new `LaneSpec("deduction_serve_v1", ...)`
|
|
+ pinned SHA (computed via the script's own `--update`, the documented
|
|
procedure — not hand-typed).
|
|
|
|
## Honesty check — a real authoring bug the lane itself caught
|
|
|
|
The first draft of the corpus included a case intended as **entailed**
|
|
(contraposition: `"If p then q. Therefore if not q then not p."`). Running
|
|
the actual runner (not just eyeballing it) surfaced a mismatch: the
|
|
pipeline **declined** it. Tracing why (not assuming the case was right):
|
|
`generate/meaning_graph/reader.py`'s `_parse_propositional` only accepts
|
|
`not P` as a **top-level** clause — `_chunk`'s reserved-word guard rejects
|
|
`"not q"` / `"not p"` *inside* an `if/then` slot (`not` is in `_RESERVED`).
|
|
This is a genuine, narrower-than-assumed reader-grammar boundary, not a
|
|
runner bug. Fixed honestly: reclassified the case's gold to `declined`
|
|
(`out_of_band_nested_negation`, now documented in the contract) and added
|
|
a different, in-band **entailed** case (a three-hop chain,
|
|
`"If p then q. If q then r. If r then s. p. Therefore s."`) to keep
|
|
coverage. This is exactly the kind of finding Phase 0/1's "honest wrinkle"
|
|
discipline is meant to surface — caught here because the lane was actually
|
|
*run*, not because the corpus was authored perfectly the first time.
|
|
|
|
## Out-of-scope finding: three lanes emit non-deterministic committed artifacts; `public_demo` errors
|
|
|
|
Running the documented `scripts/verify_lane_shas.py --update` procedure to
|
|
compute this lane's pin **re-executes every registered lane's runner**,
|
|
which rewrites their committed `results/v1_dev.json` files on disk (not
|
|
merely re-hashes the existing ones). Doing so surfaced two pre-existing,
|
|
unrelated problems on `main @ 6a54d27a`:
|
|
|
|
- **`miner_loop_closure`, `curriculum_loop_closure`, `demo_composition`
|
|
regenerate non-deterministic content IDs** (e.g. `miner_loop_closure`'s
|
|
`proposal_id`/`finding_id` fields differ byte-for-byte between two
|
|
consecutive runs on identical input — `be5a8f06893b0575` vs
|
|
`4272ff76eb8a9e16` for the same `positive_basic` case). Their committed
|
|
`PINNED_SHAS` values therefore do not reproduce even on an unmodified
|
|
checkout — a genuine determinism bug in those lanes, independent of
|
|
anything this arc touched.
|
|
- **`public_demo` errors outright** (`ERROR after 85s`) — matches the
|
|
already-known "`public_demo` flake = env timeout" gotcha in standing
|
|
project memory.
|
|
|
|
Neither is caused by, or in scope for, the deduction-serve arc. Reverted
|
|
all four lanes' `results/*.json` and `PINNED_SHAS` entries to their
|
|
original committed values before finalizing this PR — the only change to
|
|
`scripts/verify_lane_shas.py` is the new `deduction_serve_v1` `LaneSpec` +
|
|
pin. Worth a dedicated follow-up outside this arc: those three lanes need
|
|
a deterministic-ID fix before their pins can ever be legitimately
|
|
refreshed via `--update` again.
|
|
|
|
## Verification
|
|
|
|
```
|
|
uv run python -m evals.deduction_serve.runner # 27/27, wrong=0
|
|
uv run python -m pytest tests/test_deduction_serve_lane.py -q # 2 passed
|
|
uv run core test --suite deductive -q # 25 passed
|
|
uv run core test --suite smoke -q # 180 passed
|
|
uv run core test --suite cognition -q # 122 passed, 1 skipped
|
|
uv run python scripts/verify_lane_shas.py --update # pin computed + written
|
|
```
|
|
|
|
## Verdict
|
|
|
|
Phase 2 complete. The deduction-serve capability now has a durable,
|
|
SHA-pinned, `core test`-wired proof of correctness distinct from (and
|
|
stronger than) the two lanes that could be mistaken for covering the same
|
|
ground — this lane is the only one that proves the *production* decider
|
|
gets raw natural-language arguments right, not just its parts in
|
|
isolation. Combined with Phase 1, the arc's original goal is met: a user
|
|
can ask a basic logic question in `core chat` and get a decided,
|
|
articulated, wrong=0-verified answer, with a committed regression gate
|
|
protecting it going forward.
|