docs(assessment): independent verification of the Tier-S arc assessment
Re-checks every load-bearing claim of an external architectural assessment (18 observations + a five-pillar blueprint) against the tree, by reading the implicated code and, where possible, executing it. Headline: Workbench records a proved deduction answer as ungrounded. With deduction_serving_enabled ratified ON, chat/runtime.py stamps TurnEvent.grounding_source="deduction", and workbench/api.py's _coerce_grounding_source whitelist silently rewrites it to "none". Verified by running workbench.api._run_chat_turn, not by reading. Scope is one field: epistemic_state still reads epistemic_state_needed (honest) because workbench/api.py:818 prefers the TurnEvent's own value. The unregistered path degrades honestly; the hand-copied whitelist asserts a falsehood. Falsified items, with evidence: - parents[n] audit — run: 168 sites, 0 outside the repo root. - realizer-guard exemption — documented at both guard sites. - T12 — present in the weekly-audit stragglers TODO. - assert_corpus_sound — already domain-parameterized and called unconditionally. - promotion oracle — already the wrong=0 lane gate, per tier-s-housekeeping §3. - accrue_realized_knowledge — a session-memory deployment profile, not a debt clock. - cross-subject proof — resolve_domain fails closed on ambiguity. Also corrects the blueprint's flagship item: EntailmentTrace carries BDD node keys, not derivation steps, so multi-step articulation is engine work (proof-term extraction), not a rendering-layer gap. And records that the "second subject arena" premise is falsified by the curriculum-volume measurement — 11 bands across 4 subjects already route, all 24x-73x short. Companion Apple Silicon/MLX blueprint assessed: both premises contradicted by the repo's own benchmark report (no MLX in any runtime path; Rust backend opt-in and off). One of five items adopted (typestate), one rejected on numerics (bf16 against a 1e-6 gate). [Verification]: docs-only change; no code paths touched. Findings themselves were executed against a scratch venv — the repo pins ==3.12.13, which uv cannot fetch for linux-x86_64, so the pin was left untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FduW6Krm3PPQv3P5iwBYtx
This commit is contained in:
Claude
parent
6a06cb2929
commit
676b6555b4
1 changed files with 186 additions and 0 deletions
186
docs/research/architecture-assessment-verification-2026-07-25.md
Normal file
186
docs/research/architecture-assessment-verification-2026-07-25.md
Normal file
|
|
@ -0,0 +1,186 @@
|
|||
# Independent verification of the Tier-S arc assessment (2026-07-25)
|
||||
|
||||
**Input:** an external architectural assessment of the deduction-serve generalization arc
|
||||
(18 numbered observations) plus a synthesized five-pillar execution blueprint built on it.
|
||||
**Method:** every load-bearing claim re-checked against the tree at `6a06cb2` by reading the
|
||||
implicated code — and, where possible, by executing it. **Result:** roughly a third of the
|
||||
blueprint's work items are falsified by the codebase; one flagship item is diagnosed at the wrong
|
||||
layer; and the most urgent live defect appears in neither document.
|
||||
|
||||
The forward-discipline praise is accurate and was confirmed mechanically:
|
||||
`chat/data/deduction_serve_ledger.json` holds 25 sealed bands / 18,000 cases / `wrong=0`, and
|
||||
`evals/capability_index/baseline.json` reports breadth 13, `wrong_total=0`,
|
||||
`assert_mode_valid=true`. What follows concerns only what the assessment proposed to do *next*.
|
||||
|
||||
## 1. The live defect neither document names
|
||||
|
||||
**Workbench records a proved deduction answer as ungrounded.** Verified by execution, not by
|
||||
reading:
|
||||
|
||||
```
|
||||
_run_chat_turn('If it rains then the ground is wet. It rains. Therefore the ground is wet.')
|
||||
|
||||
surface = 'Given: if it rains then the ground is wet; it rains.
|
||||
Your premises entail: the ground is wet.'
|
||||
grounding_source = 'none' <- TurnEvent carried 'deduction'
|
||||
epistemic_state = 'epistemic_state_needed'
|
||||
```
|
||||
|
||||
The chain:
|
||||
|
||||
1. `workbench/api.py:646` — the live HTTP chat route calls `_run_chat_turn(prompt)`.
|
||||
2. `workbench/api.py:773` — constructs a bare `ChatRuntime()`, i.e. `DEFAULT_CONFIG`.
|
||||
3. `core/config.py:397` — `deduction_serving_enabled: bool = True` (ratified ON, 2026-07-24).
|
||||
4. `chat/runtime.py:1748-1757` — the composer commits and stamps
|
||||
`TurnEvent.grounding_source = "deduction"`.
|
||||
5. `workbench/api.py:714-719` — `_coerce_grounding_source` whitelists
|
||||
`{pack, teaching, vault, partial, oov, none}` and returns `"none"` for anything else.
|
||||
|
||||
`workbench/api.py` contains no occurrence of `"deduction"` or `"curriculum"`.
|
||||
|
||||
**Scope of the falsification — one field, not two.** `workbench/api.py:818` prefers
|
||||
`TurnEvent.epistemic_state` over the grounding-source fallback, and the runtime stamps
|
||||
`epistemic_state_needed` for the unregistered label. That is honest. So `grounding_source` is
|
||||
falsified; `epistemic_state` is not.
|
||||
|
||||
The asymmetry is the useful part: the *unregistered* path degrades honestly
|
||||
(`epistemic_state_needed` — "this label needs determination"), while the *hand-copied whitelist*
|
||||
asserts a falsehood (`none` = ungrounded). The coercion is strictly worse than doing nothing.
|
||||
|
||||
**This inverts the assessment's reading.** `chat/runtime.py:477-481` argues the unregistered
|
||||
literal is "inert today" because "core chat REPL turns do not flow through Workbench's
|
||||
`CognitivePipelineRecord` path." True, and irrelevant — the traffic flows the other way. Workbench
|
||||
turns flow through the *deduction* path. That comment has been stale since the flag was ratified ON.
|
||||
|
||||
## 2. Falsified items — no session should be spent on these
|
||||
|
||||
| Claim | What the code says |
|
||||
|---|---|
|
||||
| `parents[n]` audit is "a 30-minute grep with significant payoff" | **Audit run.** 168 `Path(__file__).parents[n]` sites repo-wide; **0** resolve outside the repo root. The `derived_close_proposals.py` bug was isolated. |
|
||||
| Realizer-guard exemption is undocumented at the guard site | **False.** Documented at *both* sites — `chat/runtime.py:2371-2375`, `:3007-3010` — with the invariant and a worked example (`pack open=VERB` rejecting "the door is not open"). |
|
||||
| T12 weekly-ledger entry is "completely missing" | **False.** `docs/analysis/weekly-audit-2026-07-22-stragglers-todo.md:170` — an open pattern-watch item. The claim came from grepping commit messages only. |
|
||||
| `assert_corpus_sound()` has one impl; a second subject must reimplement it | **False.** `evals/curriculum_serve/runner.py:71` is already `assert_corpus_sound(domain, cases)`, and `build_report()` calls it plus `assert_provenance` plus `assert_anti_recall_coverage` unconditionally (`:105-108`). Structural, not discipline-based. |
|
||||
| Manual promotion sweeps need an automated Promotion Oracle | **Already exists.** `docs/research/tier-s-housekeeping-2026-07-24.md` §3: "*not a one-time manual check… The sweep's proof is the passing gate, not a separate pass.*" `ds-mem-0020` surfaced *as* a lane failure. |
|
||||
| `accrue_realized_knowledge` is "dark," a closing temporal-consistency window | **Misread.** `core/config.py:316-321` gates **session** memory, is `False` for eval/one-shot runtimes, and is enabled by the production L10 process. A deployment profile, not an epistemic debt clock. |
|
||||
| Cross-subject proof could "emerge accidentally" | **Structurally impossible.** `curriculum_surface.resolve_domain` requires both terms in one subject's vocabulary and refuses `ambiguous_reading` rather than picking. Deduction has no subject notion at all. |
|
||||
|
||||
## 3. The flagship item is diagnosed at the wrong layer
|
||||
|
||||
The blueprint states that v5-VP and v6-EX "produce intermediate proof chains" that
|
||||
`render_entailment_verb` "drops to print a terminal verdict."
|
||||
|
||||
`generate/proof_chain/entail.py:125-190` decides entailment by canonicalizing
|
||||
`(P1 & … & Pn) -> Q` and asking `is_tautology` — a BDD equivalence check, not a derivation.
|
||||
`EntailmentTrace` (`:79-99`) carries `outcome`, `reason`, and five canonical BDD node keys —
|
||||
opaque hashes. There are no intermediate steps in the object. The renderers (`render.py:62+`)
|
||||
already emit every premise plus the verdict.
|
||||
|
||||
Nothing is dropped because nothing exists to drop. Multi-step articulation is **engine** work —
|
||||
proof-term extraction, or a derivation calculus over the ROBDD — with a real soundness surface.
|
||||
Scoping it as "design the trace object, then write the renderer" would produce a renderer with
|
||||
nothing to render.
|
||||
|
||||
## 4. The "second subject arena" premise
|
||||
|
||||
The blueprint orders its work as "harden lateral contracts before spinning up the second subject
|
||||
arena." `docs/research/curriculum-volume-quantification-2026-07-24.md` §2 measures otherwise:
|
||||
**11 bands across 4 subjects already route questions today**, every one is 24×–73× short of the
|
||||
entailed-bucket floor, and the document concludes in bold that **no served subject has a family
|
||||
close enough to flag as "next."**
|
||||
|
||||
There is no subject to spin up; the wiring is done. What physics has and the others lack is
|
||||
ratified content volume. That makes the Ratification Ceremony not a third-tier item but the
|
||||
binding one.
|
||||
|
||||
## 5. Confirmed, but materially rescoped
|
||||
|
||||
- **`missing_ok`** — real and narrow. Exactly one production call site passes `True`
|
||||
(`chat/curriculum_serve_license.py:46`), correctly, because
|
||||
`chat/data/curriculum_serve_ledger.json` does not exist yet. Default is `False`. A small
|
||||
manifest closes it.
|
||||
- **Telemetry spine** — the proposed fix is unimplementable as written. `pipeline.py:799` hashes
|
||||
`hash_surface`; `runtime.py:1212` stamps the *served* surface; `TurnEvent` never carries
|
||||
`hash_surface`, so `trace_hash` is not merely divergent from telemetry but **not reconstructable
|
||||
from it**. Both documents also miss the larger sibling flagged in-code at `runtime.py:1235-1241`:
|
||||
the external telemetry sink still receives the **pre-override** event (audit-ledger R7).
|
||||
- **Discovery-yield stratification** — ~4× the implied cost. `teaching/discovery_yield.py:67-70`
|
||||
reads a single monotonic `turn_count`; there is no per-source counter to stratify on. Requires
|
||||
new persisted manifest fields. The denominator has always included pack/vault/none turns —
|
||||
deduction adds to a pre-existing dilution rather than creating it.
|
||||
- **Band-level capability index** — `index.py` scores via `coverage_geomean`. Adding 25 near-1.0
|
||||
band entries would mechanically inflate the headline `capability_score`, against the index's own
|
||||
anti-gaming property. Needs a scoring decision before a schema change.
|
||||
- **`engine_refused` as a standing gate** — `tests/test_vocab_trigger_instrument.py` is already in
|
||||
the `deductive` suite. The instrument is defined as a before/after admissions counter; promoting
|
||||
a measurement to a gate means pinning an expected value, which is precisely the stale-exact-pin
|
||||
failure mode the assessment's own test-freshness item exists to prevent. The two requests collide.
|
||||
- **`_prior_surface`** — mostly false as diagnosed. `pipeline.py:719` sets
|
||||
`_prior_surface = hash_surface`, which *is* updated in lockstep with the logos-morph override
|
||||
(`:526`) and the speculative marker (`:676`); the realizer guard runs upstream. The register
|
||||
exclusion is deliberate and documented (`:715-717`). A pin is still cheap insurance.
|
||||
- **Test freshness** — the cited instance was already fixed in `a4d6868` with a docstring
|
||||
explaining the failure mode. Remaining `== (` assertions in `tests/test_cli_test_suites.py` pin
|
||||
argv prefixes, which is appropriate. A repo-wide de-pin sweep is low yield.
|
||||
- **Math Phase 4.2 case-first** — fully accurate, and the one strategic item both documents get
|
||||
right. Adopt `docs/research/math-reader-phase-4-1-status-2026-07-24.md` verbatim.
|
||||
|
||||
## 6. The Apple Silicon / MLX substrate blueprint
|
||||
|
||||
A companion document proposes five bare-metal optimizations. Both of its foundational premises are
|
||||
contradicted by this repo's own measured report.
|
||||
|
||||
- *"MLX executes tensor operations on the Neural Engine"* is `README.md:82` — a design aspiration.
|
||||
`evals/reports/apple_uma_mechanical_sympathy_latest.md` §10 explicitly disclaims it: "No MLX
|
||||
semantic-backend claim. No Neural Engine acceleration claim. No CoreML acceleration claim."
|
||||
MLX appears in exactly two files, both under `benchmarks/`. **No MLX in any runtime path.**
|
||||
- *"Rust remains the incumbent native algebra backend"* — `algebra/backend.py:6-8`: "Pure Python is
|
||||
the deterministic default. Rust is an explicit opt-in backend via `CORE_BACKEND=rust`." The
|
||||
benchmark run records `core_rs import: False`, `using_rust(): False`, status `python_fallback`.
|
||||
|
||||
The algebra is running in pure Python/NumPy. The document proposes GPU shader work for a layer
|
||||
that has not switched on the CPU kernel it already wrote — `core-rs/src/cl41.rs` already hardcodes
|
||||
the Cl(4,1) Cayley table at compile time via bitmask anticommutation-parity and metric contraction
|
||||
(`[u8;1024]` index + `[i8;1024]` sign), which is its own recommendation 1 minus the GPU.
|
||||
|
||||
Meanwhile `chat/runtime.py` calls `versor_condition(field_state.F)` three times per turn
|
||||
(`:2313`, `:2464`, `:2514`) at a measured Python p50 of 0.536 ms — ≈1.6 ms/turn against a
|
||||
FrameVerdict TTFV of 0.151 ms. The per-turn geometric cost is ~10× the entire proof latency, and
|
||||
the remedy is already written and merged.
|
||||
|
||||
| Proposal | Verdict |
|
||||
|---|---|
|
||||
| Rust typestate (`UnverifiedClaim` → `VersorClaim`) | **Adopt.** No `PhantomData`/typestate anywhere in the tree — greenfield, not "recently explored." The only proposal with zero determinism cost, and it needs no GPU. |
|
||||
| SoA memory layout | **Resequence.** AoS confirmed (`vault.rs:111,149`). But the copy truth-table names the marshalling copy as the cost (`extract_f32_slice`, `lib.rs:348`), while batch paths are already zero-copy (`lib.rs:282`). SoA pays off only after zero-copy scalar bindings. |
|
||||
| Custom sparse `.metal` Cl(4,1) kernel | **Premature.** The table exists. Rust is off by default *for determinism*; a GPU kernel makes determinism harder. Per `AGENTS.md:276` the `ubuntu-latest:host` label is native macOS on one machine, so a shader would have a single validation host against a local-first merge bar. |
|
||||
| MLX lazy kernel fusion of the versor invariant | **Blocked on doctrine.** JIT fusion reassociates float ops; `versor_condition < 1e-6` is a non-negotiable (`algebra/versor.py:100`) and `core-rs/tests/test_crdt_hash_parity.rs` pins bit-exact parity. Needs a ratifying ADR, not a ticket. |
|
||||
| bfloat16 asymmetric precision routing | **Reject as specified.** bf16 ε ≈ 7.8e-3 — four orders coarser than the 1e-6 gate; upcasting inside the shader cannot recover bits lost at storage. `cga.rs:embed_point_raw` computes `0.5*(r²∓1)`, cancellation-prone at bf16 for r ≈ 1 — exactly the unit-normalized regime. |
|
||||
|
||||
The actionable question the document missed: **why is `CORE_BACKEND=rust` not the default?**
|
||||
Establish whether `core_rs` still holds bit-exact parity; if it does, flipping the default is the
|
||||
order-of-magnitude win being reached for.
|
||||
|
||||
None of this is on the critical path of the arc that just closed — FrameVerdict TTFV is 0.151 ms
|
||||
with `producer=proof_chain.entail`, and the deduction/curriculum serving path is pure-Python ROBDD
|
||||
that never touches CGA.
|
||||
|
||||
## 7. Resulting order of work
|
||||
|
||||
1. **Stop the provenance falsification.** Red test; register `deduction`/`curriculum` in
|
||||
`core/epistemic_state.py` + the TS contract in one commit (`enumCoverage.test.ts` already forces
|
||||
atomicity); derive `_coerce_grounding_source`'s whitelist from the enum rather than a hand-copied
|
||||
set; delete the stale comment at `chat/runtime.py:471-481`.
|
||||
2. **Build the Ratification Ceremony** — `proposal → chain record → corpus commit → arena queue
|
||||
entry → ledger reseal`, reusing `core/ratified_ledger.py`'s sealers and the `core proposal-queue`
|
||||
CLI, with the vocab-trigger admissions delta as the acceptance metric.
|
||||
3. **Batch the cheap closures** — `missing_ok` manifest; telemetry contract documented honestly
|
||||
plus audit-ledger R7 raised; the `_prior_surface` pin; deductive suite into the pre-push gate;
|
||||
ADR domain index and an arc-close brief template.
|
||||
4. **Math Phase 4.2 case-first**, in parallel — cases `0000`/`0001`/`0148`/`0082`, no
|
||||
`NEUTRAL_COUNT_VERBS` growth (ADR-0251).
|
||||
5. **Substrate track**, behind the above — Rust-default parity question, then typestate.
|
||||
|
||||
Deferred with stated reasons: multi-step articulation (re-scope as engine work), yield
|
||||
stratification (blocked on manifest counters), band-level index (blocked on a scoring decision),
|
||||
`engine_refused` gate (bucket empty), cross-subject ADR (`resolve_domain` fails closed).
|
||||
|
||||
Relates to [[project-generalization-arc]].
|
||||
Loading…
Reference in a new issue