core/docs/decisions/SESSION-2026-05-27-tier3-sequencing.md

SESSION 2026-05-27 — Tier 3 sequencing, eval-surface discipline, the ADR-0166 principle

Participants: Shay, Claude (orchestrator), an earlier strategic-analysis agent (external) Outputs: ADR-0166 — Measurement-Capability Sequencing Discipline Anchor: thesis-decoding-not-generating

---

What triggered the session

The operator shared a substantive strategic analysis from another agent. That analysis surveyed CORE's current eval surface, identified that the Tier 3 learning-curve lanes (multi_step_reasoning, symbolic_logic, cross_domain_transfer, zero_code_domain_acquisition, compositionality, inference_closure) are all TBD, and proposed a four-phase plan to advance general capability:

1. Phase 1 (immediate): Run all existing Tier 3 lanes; populate the TBD rows.
2. Phase 2 (architectural): Build the missing structural-pattern recognizer + cross-subdomain transfer operator named in evals/cross_domain_transfer/gaps.md.
3. Phase 3 (parallel to Phase 2): Expand the OOD eval surface (chemistry, propositional logic data, historical sequence, geometry, multi-step causal, analogical).
4. Phase 4: External benchmarks (ARC, BBH subset, GPQA-Diamond).

The analysis declared its single most impactful next commit: "run all Tier 3 lanes and fill the TBD rows."

The operator's question: where does this go — ADR, scope doc, or something else?

What the analysis got right

Three load-bearing instincts:

1. Don't skip to MMLU-Pro or GSM8K-MATH. Sequence respects CORE's actual architecture; harder math comes later, after structural reasoning is in place.

2. Geometry-first as strategic bet. CGA gives CORE a structural advantage in spatial reasoning that no transformer has. Author the operator that produces the advantage, and chase the moat instead of the leaderboard. This was the sharpest call.

3. TBD rows are data debt. A measurement table full of TBD plateaus the entire diagnostic surface. They need numbers.

What the analysis missed

Two blind spots, one of them load-bearing.

Blind spot #1 — GSM8K-math treated as solved

The analysis lists gsm8k_math matter-of-factly alongside the OOD lanes, as if the lane has a real, healthy capability behind it. It does not. At session start (post #331):

gsm8k_math/train_sample/v1:  correct=3  refused=47  wrong=0
exit_criterion:              correct_min=10  →  not passed

The comprehension-reader Phase 1 wiring landed (ADR-0164 / #326 / #331). The measurement confirmed: question-only reader scope cannot move correct off the baseline because every question-refused case also has statement-side barriers. Phase 2 (Brief 10, statement-frame reader) is the load-bearing next dispatch.

Against that backdrop, "the single most impactful next commit is to run all Tier 3 lanes" reads differently. It is measurement of a capability surface that has not yet cleared its first gate. The Tier 3 lanes will refuse uniformly until the math substrate handles its training set; their TBD rows would be filled with zeros that don't distinguish "the lane is hard" from "the engine isn't finished."

Blind spot #2 — the comprehension-reader pivot isn't in the analysis

ADR-0164 / 0165 landed mid-session. The analysis's call for a "structural-pattern recognizer" is in significant part what the comprehension reader's update-rule tables become when their composition generalizes beyond math. The Phase 2 statement reader is the structural-pattern recognizer — under a different name — that the cross-domain gaps.md asks for. Authoring a separate "structural-pattern recognizer" component without checking whether the reader already is it (or will be, with Phase 2 + Phase 3) risks duplicating work.

This isn't the analysis's fault — it didn't have the in-session context. But it means the proposed Phase 2 ("build the missing component") and the actual in-flight Phase 2 (Brief 10) collide. The correct read is that Brief 10 IS the cross-domain step the analysis names, embedded in the comprehension-reader architecture.

Mid-session finding that sharpened the diagnostic further

While drafting this log, PR #332 (a wrong=0 guard + adapter hardening pass over the Phase 1 reader) landed its measurement and produced a deeper diagnostic than the Phase 1 result alone:

All 47 refused cases either have incomplete graphs (recognized-but-not-injected statements — ADR-0163 recognizers flag the shape but the injector produces structurally incomplete output) or question structures beyond Phase 1 scope (aggregate "they", numeric target values, hypernym units).

The bottleneck is not statement parsing per se — it's the ADR-0163 recognizer injectors emitting incomplete data, which the reader correctly refuses to admit (wrong=0 by construction via the new guard). Two paths now open:

• Brief 10 (Phase 2 reader) — bypass the inadequate injectors by replacing statement parsing with the reader's compositional rules. This is the long-term destination per ADR-0164.3 §Phase 3.
• ADR-0163 Phase E injector fixes — fix the existing recognizer pipeline to emit complete graphs. Shorter-term win, but doesn't reduce the regex sentence-template surface (ADR-0165 and ADR-0164 call for eventual deletion).

Brief 10 dominates here. Phase 2 reader fixes the structural problem; injector patches fix the symptom.

The honest re-sequence

NOW              Brief 10 (Phase 2 statement-frame reader)
                   ↓ measure on gsm8k_math train_sample
                 If correct ≥ 25: architecture proven at scale.
                 If 4–24: name the specific gap → Phase 2.1 sub-brief.
                 If 3:    deeper architectural problem; reassess.

PARALLEL         Run existing Tier 3 lanes ONCE as a snapshot.
(cheap)          Populate the TBD rows with whatever they read at
                 today's capability. Re-run after Phase 2 lands.
                 The snapshot is diagnostic, not strategic.

NEXT             Cross-domain transfer operator (per gaps.md).
                 BUT first verify whether the comprehension reader's
                 Phase 2 / Phase 3 update rules already are this,
                 under a different name. Don't duplicate work that
                 is in flight.

THEN             Geometry path:
                   1. Build CGA → spatial-inference operators.
                   2. Author spatial_geometry_ood.
                   3. Run, measure.
                 In that order. Lane after operator, per ADR-0166.

LAST             External benchmarks (ARC, BBH, GPQA-Diamond).
                 After the reader is genuinely domain-general — not
                 just math — these become reachable targets.

The principle being formalized

The fourth governing principle that came out of this exchange:

Build the mechanism, measure the mechanism, then expand scope. Don't expand the eval surface ahead of the capability that produces signal on it. Lanes that refuse uniformly generate noise, not data — and noise drowns the signal you actually want to diagnose by.

This is now ADR-0166 (Measurement-Capability Sequencing Discipline), sitting alongside ADR-0114a (anti-overfitting) and ADR-0165 (regex scope rule) as a structural invariant the project enforces by convention rather than by case-by-case judgment.

The ADR's three-question test mechanizes the principle: every new eval lane PR answers "does the capability exist?", "has at least one case admitted?", "will the lane distinguish presence from absence?" A "no" on any of the three defers the lane until the capability lands.

What this session did not decide

• Which lane authoring follows Phase 2 measurement. That depends on the measurement outcome. If Phase 2 lifts correct ≥ 25, the OOD expansion can begin. If not, Phase 2.1 narrows the gap first.
• Whether the ADR-0163 Phase E injector path is worth doing at all. Brief 10 makes it largely obsolete; the question is whether there's enough short-term value to do it in parallel. Defer until Brief 10's measurement.
• The structural-pattern recognizer's exact relationship to the comprehension reader. Audit recommended after Brief 10. If the reader IS the recognizer in a different idiom, the cross_domain_transfer/gaps.md text gets amended; if not, a separate component is authored after the reader stabilizes.
• Geometry timeline. Strategic bet, no timeline. Worth doing when the operators are an obvious next move under existing architecture; not worth forcing.

Closing observation

The most useful thing the external analysis did was not its recommended action (run all Tier 3 lanes). It was the question its existence forced: where is signal, where is noise, where is data debt, where is wishlist? Sorting those four categories is the work. Once they're sorted, the next commit is mechanical: build the next mechanism the data actually needs, in the order the architecture supports.

The other useful thing — almost incidental — was that the analysis was unaware of the comprehension-reader pivot. That gap forced us to articulate why the pivot supersedes a strict reading of the analysis's Phase 2. ADR-0166 codifies that articulation so it survives this conversation: capability before measurement, measurement before expansion, no lane authored ahead of its operator.

The analysis itself was substantive and worth engaging. The amendments it required are documented here in the spirit of "we read it, we adopted what was right, we corrected what assumed unbuilt capabilities were built." That spirit is what ADR-0166 preserves going forward.