From 834d1ae57787ece44d908eb6e1b157a71934d7ae Mon Sep 17 00:00:00 2001 From: Shay Date: Tue, 26 May 2026 22:19:40 -0700 Subject: [PATCH] docs(ADR-0166): measurement-capability sequencing discipline (#334) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the fourth governing principle to the family of structural- invariant ADRs (alongside ADR-0114a anti-overfitting, ADR-0165 regex scope rule, CLAUDE.md versor invariant). The rule, stated negatively: do not author eval lanes ahead of the operators those lanes test, and do not expand the eval surface ahead of the capability that produces signal on it. Three-question test for new eval lanes: 1. Does the capability this lane probes exist on main today? 2. Has at least one case admitted end-to-end through that capability? 3. Will running this lane distinguish capability-presence from capability-absence? A "no" on any defers the lane until the capability lands. Tier 3 TBD rows are data debt; running existing lanes to populate them is permitted (snapshot of current capability) but is NOT a substitute for capability work. Why now: a strategic-analysis exchange this session proposed authoring spatial_geometry_ood, historical_sequence_ood, and other new lanes while GSM8K-math sits at 3/47/0 and the comprehension reader (ADR-0164) is mid-build. The proposal's "most impactful next commit is to run all Tier 3 lanes" framing would have generated noise (lanes refusing uniformly because their underlying operators don't exist) rather than the diagnostic signal that justifies prioritization. ADR-0166 mechanizes the constraint that prevents that pattern. Session log SESSION-2026-05-27-tier3-sequencing.md captures the narrative: what the analysis got right (geometry-first as strategic bet, sequencing instincts), what it missed (GSM8K-math treated as solved; comprehension reader pivot not in context), and the honest re-sequence (Brief 10 first; Tier 3 snapshot in parallel; cross-domain transfer after verifying whether the reader IS the requested structural-pattern recognizer under a different name). The session also surfaced a mid-flight diagnostic from PR #332: the actual GSM8K bottleneck is the ADR-0163 recognizer injector emitting incomplete graphs, which the reader correctly refuses to admit (wrong=0 by construction via the new guard). Brief 10 (Phase 2 reader) dominates here because it replaces the inadequate injector surface entirely. No code changes. ADRs only. Refs ADR-0114a, ADR-0165, CLAUDE.md §"Non-Negotiable Field Invariant". --- ...-0166-measurement-capability-sequencing.md | 282 ++++++++++++++++++ .../SESSION-2026-05-27-tier3-sequencing.md | 218 ++++++++++++++ 2 files changed, 500 insertions(+) create mode 100644 docs/decisions/ADR-0166-measurement-capability-sequencing.md create mode 100644 docs/decisions/SESSION-2026-05-27-tier3-sequencing.md diff --git a/docs/decisions/ADR-0166-measurement-capability-sequencing.md b/docs/decisions/ADR-0166-measurement-capability-sequencing.md new file mode 100644 index 00000000..506e85bc --- /dev/null +++ b/docs/decisions/ADR-0166-measurement-capability-sequencing.md @@ -0,0 +1,282 @@ +# ADR-0166 — Measurement-Capability Sequencing Discipline + +**Status:** Proposed +**Date:** 2026-05-27 +**Author:** Shay +**Anchor:** [[thesis-decoding-not-generating]] +**Companions:** [ADR-0114a — Anti-overfitting proof obligations](./ADR-0114a-anti-overfitting-proof-obligations.md), [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md), [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md) + +--- + +## Context — why the order matters + +CORE accrues two distinct kinds of artifact over time: + +1. **Capability artifacts** — the operators, recognizers, registries, and + composition rules that make the engine *able to* admit a class of + problems (e.g. the comprehension reader, the binding-graph + admissibility check, the deterministic solver). + +2. **Measurement artifacts** — the eval lanes, runners, scoring rules, + and baselines that quantify *whether* the engine handles a class of + problems (e.g. `evals/gsm8k_math/`, `evals/cross_domain_transfer/`, + the capability-axis lanes G1–G5). + +Both are load-bearing. Both must exist for an honest claim of capability +to be made. But they only produce signal when they are sequenced +correctly: a measurement artifact that runs against a non-existent +capability produces refusals — and refusals at 100% generate noise, not +data. + +The session that produced this ADR identified a concrete instance: +a strategic plan proposed authoring `spatial_geometry_ood`, +`historical_sequence_ood`, and other new eval lanes ahead of the +operators that those lanes would test. The recommended "single most +impactful next commit" was to populate the Tier 3 TBD rows by running +existing lanes — a step that is legitimate but is in fact a snapshot, +not advancement. Meanwhile the engine's actual capability bottleneck +(`gsm8k_math` train_sample at 3/47/0) was treated as adjacent rather +than as the gating concern. + +That kind of sequencing inversion is what this ADR prevents. It is in +the same family of structural invariants as the field-state versor +condition (CLAUDE.md §"Non-Negotiable Field Invariant"), ADR-0165's +regex scope rule, and ADR-0114a's anti-overfitting obligations: a +typed boundary the project enforces by convention, not a guideline. + +--- + +## Rule + +> **Capability lands before the measurement that depends on it.** +> An eval lane may only be authored when the operators it probes exist +> on main and at least one case admits end-to-end through those +> operators. An OOD lane for a domain may only be authored when the +> in-domain capability it probes has a non-trivial baseline. +> +> An existing TBD measurement row may be populated at any time by +> running the lane; this is a snapshot of current capability and is +> permitted independently. But "populating TBD rows" is **not** a +> substitute for the capability work that would make those numbers +> rise. +> +> Expanding the eval surface ahead of the capability that produces +> signal on it is rejected at PR review. + +--- + +## Three-question test (apply to every new eval lane PR) + +When reviewing or proposing a new eval lane, answer three questions: + +1. **Does the capability this lane probes exist on main today?** + Name the modules. If you cannot point to specific code that + implements the operator, recognizer, or composition rule the lane + would exercise, the lane is being authored ahead of its capability. + +2. **Has at least one case admitted end-to-end through that + capability?** "Admitted" means produced a non-refused output that + passed admissibility. A lane whose every case will predictably + refuse generates no signal — it just stamps the same refusal mode + N times. + +3. **Will running this lane distinguish capability-presence from + capability-absence?** A lane that returns 0/N at capability=absent + and 0/N at capability=partial is not measuring; it is decorating + the data debt. + +A lane that fails any of these three is being authored ahead of its +capability and must wait. The waiting period is not arbitrary: it +ends when the answers shift to (1) yes, (2) yes, (3) yes — at which +point the lane lands and immediately produces signal. + +--- + +## Legitimate work (capability-before-lane) + +These are the canonical examples of correct sequencing. + +| Sequence | Example | +|---|---| +| Operator lands → lane authored that probes it → measure | ADR-0118 stepped realizer landed → `evals/articulation_of_status` authored → real metrics emerged | +| Reader landed → coexistence wiring → measurement lane updated to use it | ADR-0164 Phase 1 reader landed → coexistence wiring (#331) → `reader_phase1_delta.json` produced | +| Solver landed → capability axis lane → byte-stable baseline | ADR-0116 deterministic solver → capability-axis lanes G1–G5, S1 each at 100% on controlled cases | +| Pack landed → pack-test landed → enum-coverage gate | `en_core_math_v1` (#322) → `tests/test_en_core_math_v1_pack.py` → manifest-checksum + lemma-count regression net | + +In each case, the measurement artifact was authored *after* there was +something for it to measure, and produced signal the moment it ran. + +## Forbidden work (lane-before-capability) + +These are the patterns this ADR rejects. + +``` +# FORBIDDEN — authoring an OOD lane without the operator: + +mkdir evals/spatial_geometry_ood +# But CGA-field → spatial-inference operators do not exist. +# Every case will refuse. The lane will register 0/N pass for +# however long the operators take to land, generating no signal +# in the interim. +``` + +``` +# FORBIDDEN — authoring an external benchmark integration without +# the prerequisite capability: + +mkdir evals/arc_easy +# But the comprehension reader is currently math-specific in its +# composition rules. ARC reasoning prose requires narrative-frame +# composition rules that do not exist. The lane will refuse on +# every case until that capability lands. +``` + +``` +# FORBIDDEN — adding TBD rows to the Tier 3 table as if they were +# work: + +# tier3: +# multi_step_reasoning: TBD +# symbolic_logic: TBD +# cross_domain_transfer: TBD +# spatial_geometry_ood: TBD <-- new row, capability missing +``` + +``` +# FORBIDDEN — substituting measurement work for capability work: + +# "The most impactful next commit is to run all Tier 3 lanes and fill +# the TBD rows." +# +# Wrong, when GSM8K-math is at 3/47/0 and the reader scope that +# would lift it has not landed. The numbers Tier 3 would produce +# are dominated by the unbuilt-reader noise floor. Populate the +# rows AFTER the reader work clears the floor, not before. +``` + +--- + +## Population — how the eval surface grows + +The closed sequencing rule is not static; it advances as capability +advances. New lanes enter the surface through a specific pathway: + +1. **A capability commits to main.** Operator / recognizer / + composition rule lands with tests and a non-trivial admission case. + +2. **The contemplation corridor (ADR-0150 / 0152 / 0155 / 0161) + identifies measurement gaps the new capability now makes + testable.** This is the same corridor that grows the lexicon and + the primitive registry under ADR-0164 / 0165 — but it now also + surfaces "this capability admits but has no lane that measures it" + as a proposal type. + +3. **A new lane is authored.** Its first run produces real numbers + (not zeros), satisfying the three-question test from the moment + of landing. + +4. **The lane is added to the Tier 3 table or domain ledger as a + real row** with a real baseline — never as TBD. + +A TBD row exists *only* for lanes that were already authored under a +prior capability and have not been re-run since. TBD is a re-run +debt, not a placeholder for unbuilt lanes. + +--- + +## Consequences + +### Positive + +1. **Signal-to-noise ratio of the eval surface stays high.** Every + lane that exists produces interpretable numbers. The number of + lanes that consistently report 0/N is bounded. +2. **Strategic prioritization becomes legible.** When someone proposes + "what should we do next?" the answer is constrained by current + capability rather than by aspirational lane authoring. The order + "Brief 10 (Phase 2 reader) before any new OOD lane" is mechanical + under this rule. +3. **Data debt has a fixed retirement path.** Run the lane → number + appears → no debt. There's no slow accrual of zeros that hides + whether progress is real. +4. **The eval surface stays a *measurement* of CORE rather than a + *wishlist* for CORE.** + +### Negative / tradeoffs + +1. **Some lane authoring is deferred that an enthusiastic operator + might want to do "for completeness."** This is intentional; the + completeness gain is illusory if the lanes refuse uniformly. +2. **The rule requires PR reviewers to verify the three-question test + — slight review overhead per new-lane PR.** The test is fast (one + grep + one trial run). +3. **A new operator and a new lane cannot land in the same PR** + unless the operator's admission case is demonstrated *first*, + before the lane is added. (Allowed: operator + admission test in + PR N, lane in PR N+1. Forbidden: operator + lane in same PR with + no demonstrated admission.) + +--- + +## Boundaries — what this ADR does **not** say + +1. **It does not forbid measurement.** Running existing lanes to + snapshot current numbers is always permitted and often + diagnostically valuable. The ADR governs *authoring*, not + *running*. + +2. **It does not require all lanes to pass at 100%.** A lane that + reports e.g. `correct=3 refused=47 wrong=0` is producing signal — + the refusals are themselves interpretable when wrong=0 is intact. + The ADR's "0/N pattern" forbidding refers to lanes that refuse + uniformly because their capability is wholly absent. + +3. **It does not constrain experimental probes.** A throwaway + measurement script under `scratch/` or `evals/_lab/` exploring + whether a capability would admit a class of input is fine. The + constraint applies to lanes that enter the canonical Tier 3 / + domain-ledger / capability-axis surfaces. + +4. **It does not specify ordering among capabilities.** Whether to + build Phase 2 statement reader before geometric-inference + operators is a separate strategic decision (see + `SESSION-2026-05-27-tier3-sequencing.md`). The ADR only says: in + whichever order capability is built, the lane that measures it + follows, not leads. + +5. **It does not constrain the contemplation corridor.** The + corridor's proposals (lexicon entries, primitives, recognizer + candidates) are not eval lanes; they ride a separate pathway + under ADR-0057 / 0150 / 0152 / 0161. + +--- + +## Cross-references + +- **Sibling structural invariants** (same family, different domain): + - [ADR-0114a](./ADR-0114a-anti-overfitting-proof-obligations.md) — + Anti-overfitting proof obligations + - [ADR-0165](./ADR-0165-regex-scope-rule.md) — Regex scope rule + - CLAUDE.md §"Non-Negotiable Field Invariant" — + `versor_condition(F) < 1e-6` + - CLAUDE.md §"Normalization Rules" — allowed sites only +- **The capability path that motivates this ADR**: + - [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) — the + architectural pivot that, mid-build, makes the "run all Tier 3 + lanes" framing premature + - [ADR-0114](./ADR-0114-expert-capability-roadmap-gsm8k-first.md) — + the roadmap that established GSM8K as the first capability gate +- **Narrative record**: + - `SESSION-2026-05-27-tier3-sequencing.md` — how this rule was + reached, the prior analysis it amends, the honest re-sequence +- **Population mechanism**: + - [ADR-0150](./ADR-0150-autonomous-inter-session-contemplation.md), + [ADR-0152](./ADR-0152-learning-arc-demo.md), + [ADR-0155](./ADR-0155-ci-contemplation-runner.md), + [ADR-0161](./ADR-0161-hitl-async-queue.md) — the corridor that + surfaces measurement gaps as proposals +- **Anchor**: `[[thesis-decoding-not-generating]]` — measurement that + precedes capability is not decoding the engine's state; it is + generating a wishlist of states the engine has not yet entered. + The ADR keeps the eval surface aligned with what the engine + actually does. diff --git a/docs/decisions/SESSION-2026-05-27-tier3-sequencing.md b/docs/decisions/SESSION-2026-05-27-tier3-sequencing.md new file mode 100644 index 00000000..5d3e50be --- /dev/null +++ b/docs/decisions/SESSION-2026-05-27-tier3-sequencing.md @@ -0,0 +1,218 @@ +# SESSION 2026-05-27 — Tier 3 sequencing, eval-surface discipline, the ADR-0166 principle + +**Participants:** Shay, Claude (orchestrator), an earlier strategic-analysis agent (external) +**Outputs:** [ADR-0166 — Measurement-Capability Sequencing Discipline](./ADR-0166-measurement-capability-sequencing.md) +**Anchor:** [[thesis-decoding-not-generating]] + +--- + +## What triggered the session + +The operator shared a substantive strategic analysis from another +agent. That analysis surveyed CORE's current eval surface, identified +that the Tier 3 learning-curve lanes (`multi_step_reasoning`, +`symbolic_logic`, `cross_domain_transfer`, +`zero_code_domain_acquisition`, `compositionality`, +`inference_closure`) are all TBD, and proposed a four-phase plan to +advance general capability: + +1. **Phase 1 (immediate):** Run all existing Tier 3 lanes; populate + the TBD rows. +2. **Phase 2 (architectural):** Build the missing structural-pattern + recognizer + cross-subdomain transfer operator named in + `evals/cross_domain_transfer/gaps.md`. +3. **Phase 3 (parallel to Phase 2):** Expand the OOD eval surface + (chemistry, propositional logic data, historical sequence, + geometry, multi-step causal, analogical). +4. **Phase 4:** External benchmarks (ARC, BBH subset, GPQA-Diamond). + +The analysis declared its single most impactful next commit: +**"run all Tier 3 lanes and fill the TBD rows."** + +The operator's question: where does this go — ADR, scope doc, or +something else? + +## What the analysis got right + +Three load-bearing instincts: + +1. **Don't skip to MMLU-Pro or GSM8K-MATH.** Sequence respects + CORE's actual architecture; harder math comes later, after + structural reasoning is in place. + +2. **Geometry-first as strategic bet.** CGA gives CORE a structural + advantage in spatial reasoning that no transformer has. + Author the operator that produces the advantage, and chase the + moat instead of the leaderboard. This was the sharpest call. + +3. **TBD rows are data debt.** A measurement table full of TBD + plateaus the entire diagnostic surface. They need numbers. + +## What the analysis missed + +Two blind spots, one of them load-bearing. + +### Blind spot #1 — GSM8K-math treated as solved + +The analysis lists `gsm8k_math` matter-of-factly alongside the OOD +lanes, as if the lane has a real, healthy capability behind it. It +does not. At session start (post #331): + +``` +gsm8k_math/train_sample/v1: correct=3 refused=47 wrong=0 +exit_criterion: correct_min=10 → not passed +``` + +The comprehension-reader Phase 1 wiring landed (ADR-0164 / #326 / +#331). The measurement confirmed: question-only reader scope cannot +move correct off the baseline because every question-refused case +also has statement-side barriers. Phase 2 (Brief 10, statement-frame +reader) is the load-bearing next dispatch. + +Against that backdrop, "the single most impactful next commit is to +run all Tier 3 lanes" reads differently. It is measurement of a +capability surface that has not yet cleared its first gate. The +Tier 3 lanes will refuse uniformly until the math substrate handles +its training set; their TBD rows would be filled with zeros that +don't distinguish "the lane is hard" from "the engine isn't +finished." + +### Blind spot #2 — the comprehension-reader pivot isn't in the analysis + +ADR-0164 / 0165 landed mid-session. The analysis's call for a +"structural-pattern recognizer" is in significant part what the +comprehension reader's update-rule tables *become* when their +composition generalizes beyond math. The Phase 2 statement reader is +the structural-pattern recognizer — under a different name — that +the cross-domain gaps.md asks for. Authoring a separate +"structural-pattern recognizer" component without checking whether +the reader already is it (or will be, with Phase 2 + Phase 3) risks +duplicating work. + +This isn't the analysis's fault — it didn't have the in-session +context. But it means the proposed Phase 2 ("build the missing +component") and the actual in-flight Phase 2 (Brief 10) collide. +The correct read is that Brief 10 IS the cross-domain step the +analysis names, embedded in the comprehension-reader architecture. + +## Mid-session finding that sharpened the diagnostic further + +While drafting this log, PR #332 (a wrong=0 guard + adapter +hardening pass over the Phase 1 reader) landed its measurement and +produced a deeper diagnostic than the Phase 1 result alone: + +> All 47 refused cases either have **incomplete graphs** +> (recognized-but-not-injected statements — ADR-0163 recognizers +> flag the shape but the injector produces structurally incomplete +> output) or **question structures beyond Phase 1 scope** (aggregate +> "they", numeric target values, hypernym units). + +The bottleneck is not statement *parsing* per se — it's the +ADR-0163 recognizer **injectors** emitting incomplete data, which +the reader correctly refuses to admit (wrong=0 by construction via +the new guard). Two paths now open: + +- **Brief 10 (Phase 2 reader)** — bypass the inadequate injectors + by replacing statement parsing with the reader's compositional + rules. This is the long-term destination per ADR-0164.3 §Phase 3. +- **ADR-0163 Phase E injector fixes** — fix the existing recognizer + pipeline to emit complete graphs. Shorter-term win, but doesn't + reduce the regex sentence-template surface (ADR-0165 and ADR-0164 + call for eventual deletion). + +Brief 10 dominates here. Phase 2 reader fixes the structural +problem; injector patches fix the symptom. + +## The honest re-sequence + +``` +NOW Brief 10 (Phase 2 statement-frame reader) + ↓ measure on gsm8k_math train_sample + If correct ≥ 25: architecture proven at scale. + If 4–24: name the specific gap → Phase 2.1 sub-brief. + If 3: deeper architectural problem; reassess. + +PARALLEL Run existing Tier 3 lanes ONCE as a snapshot. +(cheap) Populate the TBD rows with whatever they read at + today's capability. Re-run after Phase 2 lands. + The snapshot is diagnostic, not strategic. + +NEXT Cross-domain transfer operator (per gaps.md). + BUT first verify whether the comprehension reader's + Phase 2 / Phase 3 update rules already are this, + under a different name. Don't duplicate work that + is in flight. + +THEN Geometry path: + 1. Build CGA → spatial-inference operators. + 2. Author spatial_geometry_ood. + 3. Run, measure. + In that order. Lane after operator, per ADR-0166. + +LAST External benchmarks (ARC, BBH, GPQA-Diamond). + After the reader is genuinely domain-general — not + just math — these become reachable targets. +``` + +## The principle being formalized + +The fourth governing principle that came out of this exchange: + +> **Build the mechanism, measure the mechanism, then expand scope.** +> Don't expand the eval surface ahead of the capability that +> produces signal on it. Lanes that refuse uniformly generate +> noise, not data — and noise drowns the signal you actually want +> to diagnose by. + +This is now ADR-0166 (Measurement-Capability Sequencing Discipline), +sitting alongside ADR-0114a (anti-overfitting) and ADR-0165 (regex +scope rule) as a structural invariant the project enforces by +convention rather than by case-by-case judgment. + +The ADR's three-question test mechanizes the principle: every new +eval lane PR answers "does the capability exist?", "has at least one +case admitted?", "will the lane distinguish presence from absence?" +A "no" on any of the three defers the lane until the capability +lands. + +## What this session did not decide + +- **Which lane authoring follows Phase 2 measurement.** That depends + on the measurement outcome. If Phase 2 lifts correct ≥ 25, the OOD + expansion can begin. If not, Phase 2.1 narrows the gap first. +- **Whether the ADR-0163 Phase E injector path is worth doing at + all.** Brief 10 makes it largely obsolete; the question is whether + there's enough short-term value to do it in parallel. Defer until + Brief 10's measurement. +- **The structural-pattern recognizer's exact relationship to the + comprehension reader.** Audit recommended after Brief 10. If the + reader IS the recognizer in a different idiom, the + `cross_domain_transfer/gaps.md` text gets amended; if not, a + separate component is authored *after* the reader stabilizes. +- **Geometry timeline.** Strategic bet, no timeline. Worth doing + when the operators are an obvious next move under existing + architecture; not worth forcing. + +## Closing observation + +The most useful thing the external analysis did was *not* its +recommended action (run all Tier 3 lanes). It was the question its +existence forced: where is signal, where is noise, where is data +debt, where is wishlist? Sorting those four categories is the +work. Once they're sorted, the next commit is mechanical: build +the next mechanism the data actually needs, in the order the +architecture supports. + +The other useful thing — almost incidental — was that the analysis +was unaware of the comprehension-reader pivot. That gap forced us +to articulate why the pivot supersedes a strict reading of the +analysis's Phase 2. ADR-0166 codifies that articulation so it +survives this conversation: capability before measurement, +measurement before expansion, no lane authored ahead of its +operator. + +The analysis itself was substantive and worth engaging. The +amendments it required are documented here in the spirit of "we +read it, we adopted what was right, we corrected what assumed +unbuilt capabilities were built." That spirit is what ADR-0166 +preserves going forward.