core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

docs(ADR-0166): measurement-capability sequencing discipline (#334)

Browse source 
Add the fourth governing principle to the family of structural-
invariant ADRs (alongside ADR-0114a anti-overfitting, ADR-0165 regex
scope rule, CLAUDE.md versor invariant). The rule, stated negatively:
do not author eval lanes ahead of the operators those lanes test, and
do not expand the eval surface ahead of the capability that produces
signal on it.

Three-question test for new eval lanes:

  1. Does the capability this lane probes exist on main today?
  2. Has at least one case admitted end-to-end through that capability?
  3. Will running this lane distinguish capability-presence from
     capability-absence?

A "no" on any defers the lane until the capability lands. Tier 3 TBD
rows are data debt; running existing lanes to populate them is
permitted (snapshot of current capability) but is NOT a substitute for
capability work.

Why now: a strategic-analysis exchange this session proposed authoring
spatial_geometry_ood, historical_sequence_ood, and other new lanes
while GSM8K-math sits at 3/47/0 and the comprehension reader (ADR-0164)
is mid-build. The proposal's "most impactful next commit is to run all
Tier 3 lanes" framing would have generated noise (lanes refusing
uniformly because their underlying operators don't exist) rather than
the diagnostic signal that justifies prioritization. ADR-0166 mechanizes
the constraint that prevents that pattern.

Session log SESSION-2026-05-27-tier3-sequencing.md captures the
narrative: what the analysis got right (geometry-first as strategic
bet, sequencing instincts), what it missed (GSM8K-math treated as
solved; comprehension reader pivot not in context), and the honest
re-sequence (Brief 10 first; Tier 3 snapshot in parallel; cross-domain
transfer after verifying whether the reader IS the requested
structural-pattern recognizer under a different name).

The session also surfaced a mid-flight diagnostic from PR #332: the
actual GSM8K bottleneck is the ADR-0163 recognizer injector emitting
incomplete graphs, which the reader correctly refuses to admit
(wrong=0 by construction via the new guard). Brief 10 (Phase 2 reader)
dominates here because it replaces the inadequate injector surface
entirely.

No code changes. ADRs only.

Refs ADR-0114a, ADR-0165, CLAUDE.md §"Non-Negotiable Field Invariant".
This commit is contained in:
Shay 2026-05-26 22:19:40 -07:00 committed by GitHub
parent b3dbde94b4
commit 834d1ae577
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 500 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

282
docs/decisions/ADR-0166-measurement-capability-sequencing.md Normal file
View file

@ -0,0 +1,282 @@
# ADR-0166 — Measurement-Capability Sequencing Discipline
**Status:** Proposed
**Date:** 2026-05-27
**Author:** Shay
**Anchor:** [[thesis-decoding-not-generating]]
**Companions:** [ADR-0114a — Anti-overfitting proof obligations](./ADR-0114a-anti-overfitting-proof-obligations.md), [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md), [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md)
---
## Context — why the order matters
CORE accrues two distinct kinds of artifact over time:
1. **Capability artifacts** — the operators, recognizers, registries, and
composition rules that make the engine *able to* admit a class of
problems (e.g. the comprehension reader, the binding-graph
admissibility check, the deterministic solver).
2. **Measurement artifacts** — the eval lanes, runners, scoring rules,
and baselines that quantify *whether* the engine handles a class of
problems (e.g. `evals/gsm8k_math/`, `evals/cross_domain_transfer/`,
the capability-axis lanes G1–G5).
Both are load-bearing. Both must exist for an honest claim of capability
to be made. But they only produce signal when they are sequenced
correctly: a measurement artifact that runs against a non-existent
capability produces refusals — and refusals at 100% generate noise, not
data.
The session that produced this ADR identified a concrete instance:
a strategic plan proposed authoring `spatial_geometry_ood`,
`historical_sequence_ood`, and other new eval lanes ahead of the
operators that those lanes would test. The recommended "single most
impactful next commit" was to populate the Tier 3 TBD rows by running
existing lanes — a step that is legitimate but is in fact a snapshot,
not advancement. Meanwhile the engine's actual capability bottleneck
(`gsm8k_math` train_sample at 3/47/0) was treated as adjacent rather
than as the gating concern.
That kind of sequencing inversion is what this ADR prevents. It is in
the same family of structural invariants as the field-state versor
condition (CLAUDE.md §"Non-Negotiable Field Invariant"), ADR-0165's
regex scope rule, and ADR-0114a's anti-overfitting obligations: a
typed boundary the project enforces by convention, not a guideline.
---
## Rule
> **Capability lands before the measurement that depends on it.**
> An eval lane may only be authored when the operators it probes exist
> on main and at least one case admits end-to-end through those
> operators. An OOD lane for a domain may only be authored when the
> in-domain capability it probes has a non-trivial baseline.
>
> An existing TBD measurement row may be populated at any time by
> running the lane; this is a snapshot of current capability and is
> permitted independently. But "populating TBD rows" is **not** a
> substitute for the capability work that would make those numbers
> rise.
>
> Expanding the eval surface ahead of the capability that produces
> signal on it is rejected at PR review.
---
## Three-question test (apply to every new eval lane PR)
When reviewing or proposing a new eval lane, answer three questions:
1. **Does the capability this lane probes exist on main today?**
Name the modules. If you cannot point to specific code that
implements the operator, recognizer, or composition rule the lane
would exercise, the lane is being authored ahead of its capability.
2. **Has at least one case admitted end-to-end through that
capability?** "Admitted" means produced a non-refused output that
passed admissibility. A lane whose every case will predictably
refuse generates no signal — it just stamps the same refusal mode
N times.
3. **Will running this lane distinguish capability-presence from
capability-absence?** A lane that returns 0/N at capability=absent
and 0/N at capability=partial is not measuring; it is decorating
the data debt.
A lane that fails any of these three is being authored ahead of its
capability and must wait. The waiting period is not arbitrary: it
ends when the answers shift to (1) yes, (2) yes, (3) yes — at which
point the lane lands and immediately produces signal.
---
## Legitimate work (capability-before-lane)
These are the canonical examples of correct sequencing.
| Sequence | Example |
|---|---|
| Operator lands → lane authored that probes it → measure | ADR-0118 stepped realizer landed → `evals/articulation_of_status` authored → real metrics emerged |
| Reader landed → coexistence wiring → measurement lane updated to use it | ADR-0164 Phase 1 reader landed → coexistence wiring (#331) → `reader_phase1_delta.json` produced |
| Solver landed → capability axis lane → byte-stable baseline | ADR-0116 deterministic solver → capability-axis lanes G1–G5, S1 each at 100% on controlled cases |
| Pack landed → pack-test landed → enum-coverage gate | `en_core_math_v1` (#322) → `tests/test_en_core_math_v1_pack.py` → manifest-checksum + lemma-count regression net |
In each case, the measurement artifact was authored *after* there was
something for it to measure, and produced signal the moment it ran.
## Forbidden work (lane-before-capability)
These are the patterns this ADR rejects.
```
# FORBIDDEN — authoring an OOD lane without the operator:
mkdir evals/spatial_geometry_ood
# But CGA-field → spatial-inference operators do not exist.
# Every case will refuse. The lane will register 0/N pass for
# however long the operators take to land, generating no signal
# in the interim.
```
```
# FORBIDDEN — authoring an external benchmark integration without
# the prerequisite capability:
mkdir evals/arc_easy
# But the comprehension reader is currently math-specific in its
# composition rules. ARC reasoning prose requires narrative-frame
# composition rules that do not exist. The lane will refuse on
# every case until that capability lands.
```
```
# FORBIDDEN — adding TBD rows to the Tier 3 table as if they were
# work:
# tier3:
# multi_step_reasoning: TBD
# symbolic_logic: TBD
# cross_domain_transfer: TBD
# spatial_geometry_ood: TBD <-- new row, capability missing
```
```
# FORBIDDEN — substituting measurement work for capability work:
# "The most impactful next commit is to run all Tier 3 lanes and fill
# the TBD rows."
#
# Wrong, when GSM8K-math is at 3/47/0 and the reader scope that
# would lift it has not landed. The numbers Tier 3 would produce
# are dominated by the unbuilt-reader noise floor. Populate the
# rows AFTER the reader work clears the floor, not before.
```
---
## Population — how the eval surface grows
The closed sequencing rule is not static; it advances as capability
advances. New lanes enter the surface through a specific pathway:
1. **A capability commits to main.** Operator / recognizer /
composition rule lands with tests and a non-trivial admission case.
2. **The contemplation corridor (ADR-0150 / 0152 / 0155 / 0161)
identifies measurement gaps the new capability now makes
testable.** This is the same corridor that grows the lexicon and
the primitive registry under ADR-0164 / 0165 — but it now also
surfaces "this capability admits but has no lane that measures it"
as a proposal type.
3. **A new lane is authored.** Its first run produces real numbers
(not zeros), satisfying the three-question test from the moment
of landing.
4. **The lane is added to the Tier 3 table or domain ledger as a
real row** with a real baseline — never as TBD.
A TBD row exists *only* for lanes that were already authored under a
prior capability and have not been re-run since. TBD is a re-run
debt, not a placeholder for unbuilt lanes.
---
## Consequences
### Positive
1. **Signal-to-noise ratio of the eval surface stays high.** Every
lane that exists produces interpretable numbers. The number of
lanes that consistently report 0/N is bounded.
2. **Strategic prioritization becomes legible.** When someone proposes
"what should we do next?" the answer is constrained by current
capability rather than by aspirational lane authoring. The order
"Brief 10 (Phase 2 reader) before any new OOD lane" is mechanical
under this rule.
3. **Data debt has a fixed retirement path.** Run the lane → number
appears → no debt. There's no slow accrual of zeros that hides
whether progress is real.
4. **The eval surface stays a *measurement* of CORE rather than a
*wishlist* for CORE.**
### Negative / tradeoffs
1. **Some lane authoring is deferred that an enthusiastic operator
might want to do "for completeness."** This is intentional; the
completeness gain is illusory if the lanes refuse uniformly.
2. **The rule requires PR reviewers to verify the three-question test
— slight review overhead per new-lane PR.** The test is fast (one
grep + one trial run).
3. **A new operator and a new lane cannot land in the same PR**
unless the operator's admission case is demonstrated *first*,
before the lane is added. (Allowed: operator + admission test in
PR N, lane in PR N+1. Forbidden: operator + lane in same PR with
no demonstrated admission.)
---
## Boundaries — what this ADR does **not** say
1. **It does not forbid measurement.** Running existing lanes to
snapshot current numbers is always permitted and often
diagnostically valuable. The ADR governs *authoring*, not
*running*.
2. **It does not require all lanes to pass at 100%.** A lane that
reports e.g. `correct=3 refused=47 wrong=0` is producing signal —
the refusals are themselves interpretable when wrong=0 is intact.
The ADR's "0/N pattern" forbidding refers to lanes that refuse
uniformly because their capability is wholly absent.
3. **It does not constrain experimental probes.** A throwaway
measurement script under `scratch/` or `evals/_lab/` exploring
whether a capability would admit a class of input is fine. The
constraint applies to lanes that enter the canonical Tier 3 /
domain-ledger / capability-axis surfaces.
4. **It does not specify ordering among capabilities.** Whether to
build Phase 2 statement reader before geometric-inference
operators is a separate strategic decision (see
`SESSION-2026-05-27-tier3-sequencing.md`). The ADR only says: in
whichever order capability is built, the lane that measures it
follows, not leads.
5. **It does not constrain the contemplation corridor.** The
corridor's proposals (lexicon entries, primitives, recognizer
candidates) are not eval lanes; they ride a separate pathway
under ADR-0057 / 0150 / 0152 / 0161.
---
## Cross-references
- **Sibling structural invariants** (same family, different domain):
- [ADR-0114a](./ADR-0114a-anti-overfitting-proof-obligations.md) —
Anti-overfitting proof obligations
- [ADR-0165](./ADR-0165-regex-scope-rule.md) — Regex scope rule
- CLAUDE.md §"Non-Negotiable Field Invariant" —
`versor_condition(F) < 1e-6`
- CLAUDE.md §"Normalization Rules" — allowed sites only
- **The capability path that motivates this ADR**:
- [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) — the
architectural pivot that, mid-build, makes the "run all Tier 3
lanes" framing premature
- [ADR-0114](./ADR-0114-expert-capability-roadmap-gsm8k-first.md) —
the roadmap that established GSM8K as the first capability gate
- **Narrative record**:
- `SESSION-2026-05-27-tier3-sequencing.md` — how this rule was
reached, the prior analysis it amends, the honest re-sequence
- **Population mechanism**:
- [ADR-0150](./ADR-0150-autonomous-inter-session-contemplation.md),
[ADR-0152](./ADR-0152-learning-arc-demo.md),
[ADR-0155](./ADR-0155-ci-contemplation-runner.md),
[ADR-0161](./ADR-0161-hitl-async-queue.md) — the corridor that
surfaces measurement gaps as proposals
- **Anchor**: `[[thesis-decoding-not-generating]]` — measurement that
precedes capability is not decoding the engine's state; it is
generating a wishlist of states the engine has not yet entered.
The ADR keeps the eval surface aligned with what the engine
actually does.

218
docs/decisions/SESSION-2026-05-27-tier3-sequencing.md Normal file
View file

@ -0,0 +1,218 @@
# SESSION 2026-05-27 — Tier 3 sequencing, eval-surface discipline, the ADR-0166 principle
**Participants:** Shay, Claude (orchestrator), an earlier strategic-analysis agent (external)
**Outputs:** [ADR-0166 — Measurement-Capability Sequencing Discipline](./ADR-0166-measurement-capability-sequencing.md)
**Anchor:** [[thesis-decoding-not-generating]]
---
## What triggered the session
The operator shared a substantive strategic analysis from another
agent. That analysis surveyed CORE's current eval surface, identified
that the Tier 3 learning-curve lanes (`multi_step_reasoning`,
`symbolic_logic`, `cross_domain_transfer`,
`zero_code_domain_acquisition`, `compositionality`,
`inference_closure`) are all TBD, and proposed a four-phase plan to
advance general capability:
1. **Phase 1 (immediate):** Run all existing Tier 3 lanes; populate
the TBD rows.
2. **Phase 2 (architectural):** Build the missing structural-pattern
recognizer + cross-subdomain transfer operator named in
`evals/cross_domain_transfer/gaps.md`.
3. **Phase 3 (parallel to Phase 2):** Expand the OOD eval surface
(chemistry, propositional logic data, historical sequence,
geometry, multi-step causal, analogical).
4. **Phase 4:** External benchmarks (ARC, BBH subset, GPQA-Diamond).
The analysis declared its single most impactful next commit:
**"run all Tier 3 lanes and fill the TBD rows."**
The operator's question: where does this go — ADR, scope doc, or
something else?
## What the analysis got right
Three load-bearing instincts:
1. **Don't skip to MMLU-Pro or GSM8K-MATH.** Sequence respects
CORE's actual architecture; harder math comes later, after
structural reasoning is in place.
2. **Geometry-first as strategic bet.** CGA gives CORE a structural
advantage in spatial reasoning that no transformer has.
Author the operator that produces the advantage, and chase the
moat instead of the leaderboard. This was the sharpest call.
3. **TBD rows are data debt.** A measurement table full of TBD
plateaus the entire diagnostic surface. They need numbers.
## What the analysis missed
Two blind spots, one of them load-bearing.
### Blind spot #1 — GSM8K-math treated as solved
The analysis lists `gsm8k_math` matter-of-factly alongside the OOD
lanes, as if the lane has a real, healthy capability behind it. It
does not. At session start (post #331):
```
gsm8k_math/train_sample/v1: correct=3 refused=47 wrong=0
exit_criterion: correct_min=10 → not passed
```
The comprehension-reader Phase 1 wiring landed (ADR-0164 / #326 /
#331). The measurement confirmed: question-only reader scope cannot
move correct off the baseline because every question-refused case
also has statement-side barriers. Phase 2 (Brief 10, statement-frame
reader) is the load-bearing next dispatch.
Against that backdrop, "the single most impactful next commit is to
run all Tier 3 lanes" reads differently. It is measurement of a
capability surface that has not yet cleared its first gate. The
Tier 3 lanes will refuse uniformly until the math substrate handles
its training set; their TBD rows would be filled with zeros that
don't distinguish "the lane is hard" from "the engine isn't
finished."
### Blind spot #2 — the comprehension-reader pivot isn't in the analysis
ADR-0164 / 0165 landed mid-session. The analysis's call for a
"structural-pattern recognizer" is in significant part what the
comprehension reader's update-rule tables *become* when their
composition generalizes beyond math. The Phase 2 statement reader is
the structural-pattern recognizer — under a different name — that
the cross-domain gaps.md asks for. Authoring a separate
"structural-pattern recognizer" component without checking whether
the reader already is it (or will be, with Phase 2 + Phase 3) risks
duplicating work.
This isn't the analysis's fault — it didn't have the in-session
context. But it means the proposed Phase 2 ("build the missing
component") and the actual in-flight Phase 2 (Brief 10) collide.
The correct read is that Brief 10 IS the cross-domain step the
analysis names, embedded in the comprehension-reader architecture.
## Mid-session finding that sharpened the diagnostic further
While drafting this log, PR #332 (a wrong=0 guard + adapter
hardening pass over the Phase 1 reader) landed its measurement and
produced a deeper diagnostic than the Phase 1 result alone:
> All 47 refused cases either have **incomplete graphs**
> (recognized-but-not-injected statements — ADR-0163 recognizers
> flag the shape but the injector produces structurally incomplete
> output) or **question structures beyond Phase 1 scope** (aggregate
> "they", numeric target values, hypernym units).
The bottleneck is not statement *parsing* per se — it's the
ADR-0163 recognizer **injectors** emitting incomplete data, which
the reader correctly refuses to admit (wrong=0 by construction via
the new guard). Two paths now open:
- **Brief 10 (Phase 2 reader)** — bypass the inadequate injectors
by replacing statement parsing with the reader's compositional
rules. This is the long-term destination per ADR-0164.3 §Phase 3.
- **ADR-0163 Phase E injector fixes** — fix the existing recognizer
pipeline to emit complete graphs. Shorter-term win, but doesn't
reduce the regex sentence-template surface (ADR-0165 and ADR-0164
call for eventual deletion).
Brief 10 dominates here. Phase 2 reader fixes the structural
problem; injector patches fix the symptom.
## The honest re-sequence
```
NOW Brief 10 (Phase 2 statement-frame reader)
↓ measure on gsm8k_math train_sample
If correct ≥ 25: architecture proven at scale.
If 4–24: name the specific gap → Phase 2.1 sub-brief.
If 3: deeper architectural problem; reassess.
PARALLEL Run existing Tier 3 lanes ONCE as a snapshot.
(cheap) Populate the TBD rows with whatever they read at
today's capability. Re-run after Phase 2 lands.
The snapshot is diagnostic, not strategic.
NEXT Cross-domain transfer operator (per gaps.md).
BUT first verify whether the comprehension reader's
Phase 2 / Phase 3 update rules already are this,
under a different name. Don't duplicate work that
is in flight.
THEN Geometry path:
1. Build CGA → spatial-inference operators.
2. Author spatial_geometry_ood.
3. Run, measure.
In that order. Lane after operator, per ADR-0166.
LAST External benchmarks (ARC, BBH, GPQA-Diamond).
After the reader is genuinely domain-general — not
just math — these become reachable targets.
```
## The principle being formalized
The fourth governing principle that came out of this exchange:
> **Build the mechanism, measure the mechanism, then expand scope.**
> Don't expand the eval surface ahead of the capability that
> produces signal on it. Lanes that refuse uniformly generate
> noise, not data — and noise drowns the signal you actually want
> to diagnose by.
This is now ADR-0166 (Measurement-Capability Sequencing Discipline),
sitting alongside ADR-0114a (anti-overfitting) and ADR-0165 (regex
scope rule) as a structural invariant the project enforces by
convention rather than by case-by-case judgment.
The ADR's three-question test mechanizes the principle: every new
eval lane PR answers "does the capability exist?", "has at least one
case admitted?", "will the lane distinguish presence from absence?"
A "no" on any of the three defers the lane until the capability
lands.
## What this session did not decide
- **Which lane authoring follows Phase 2 measurement.** That depends
on the measurement outcome. If Phase 2 lifts correct ≥ 25, the OOD
expansion can begin. If not, Phase 2.1 narrows the gap first.
- **Whether the ADR-0163 Phase E injector path is worth doing at
all.** Brief 10 makes it largely obsolete; the question is whether
there's enough short-term value to do it in parallel. Defer until
Brief 10's measurement.
- **The structural-pattern recognizer's exact relationship to the
comprehension reader.** Audit recommended after Brief 10. If the
reader IS the recognizer in a different idiom, the
`cross_domain_transfer/gaps.md` text gets amended; if not, a
separate component is authored *after* the reader stabilizes.
- **Geometry timeline.** Strategic bet, no timeline. Worth doing
when the operators are an obvious next move under existing
architecture; not worth forcing.
## Closing observation
The most useful thing the external analysis did was *not* its
recommended action (run all Tier 3 lanes). It was the question its
existence forced: where is signal, where is noise, where is data
debt, where is wishlist? Sorting those four categories is the
work. Once they're sorted, the next commit is mechanical: build
the next mechanism the data actually needs, in the order the
architecture supports.
The other useful thing — almost incidental — was that the analysis
was unaware of the comprehension-reader pivot. That gap forced us
to articulate why the pivot supersedes a strict reading of the
analysis's Phase 2. ADR-0166 codifies that articulation so it
survives this conversation: capability before measurement,
measurement before expansion, no lane authored ahead of its
operator.
The analysis itself was substantive and worth engaging. The
amendments it required are documented here in the spirit of "we
read it, we adopted what was right, we corrected what assumed
unbuilt capabilities were built." That spirit is what ADR-0166
preserves going forward.