docs(plan): add CORE general advancement path (#314)

2026-05-26 18:32:08 -07:00 · 2026-05-26 18:32:08 -07:00 · 0b4a87beae
commit 0b4a87beae
parent da70919f94
1 changed files with 491 additions and 0 deletions
--- a/docs/plans/core-general-advancement-plan.md
+++ b/docs/plans/core-general-advancement-plan.md
@ -0,0 +1,491 @@
+# CORE General Advancement Plan
+
+**Status:** Proposed  
+**Date:** 2026-05-26  
+**Scope:** Documentation / planning only  
+**Anchor context:** ADR-0163 GSM8K corridor, ADR-0161 HITL queue, ADR-0160/0162 Workbench, existing eval methodology
+
+---
+
+## Executive summary
+
+CORE should not advance by chasing a broad external benchmark suite first.  The next master path is a three-lane sequence:
+
+1. **Exploit the live ADR-0163 GSM8K corridor now** because it is the freshest, proven, end-to-end capability-growth mechanism.
+2. **Instrument broader Tier-3 / generalization lanes in parallel** so general advancement is measured rather than assumed.
+3. **Delay broad external benchmarks until they are diagnostic adapters**, not vanity refusal generators.
+
+The governing principle is:
+
+```text
+truth before coverage
+refusal before confabulation
+replay before claims
+```
+
+CORE's current advantage is not raw benchmark breadth.  Its current advantage is that capability growth can be measured under deterministic replay, typed refusal, operator-ratified mutation, and `wrong == 0` discipline.  The plan below preserves that advantage while creating a route toward broader capability.
+
+---
+
+## Current pinned state
+
+The current mainline state after the ADR-0163 corridor work is:
+
+```text
+GSM8K train_sample_v1:
+  correct: 3
+  refused: 47
+  wrong: 0
+  exit: failed, correct_min=10
+```
+
+Interpretation:
+
+- This is the first measurable non-zero GSM8K lift.
+- The `wrong == 0` invariant remains intact.
+- The result is still far from an expert-capability claim.
+- The latest question-grammar extension moved some cases from question-level refusal into solver-side refusal, which means the current bottleneck is no longer only grammar/admission; it is increasingly **solver composition over admitted anchors**.
+
+The current refusal profile says the next capability work should target *composition semantics*, not broad benchmark breadth.
+
+---
+
+## Comparison of candidate plans
+
+Three planning directions were considered:
+
+| Direction | Strength | Failure mode |
+|---|---|---|
+| GSM8K-first corridor | Grounded in the freshest live mechanism; has clear measurable gates | Can become narrow if it ignores generalization lanes |
+| Refusal-taxonomy / parser expansion | Correctly protects `wrong == 0` and focuses on admission gaps | Stale if it assumes the current result is still full refusal; newest work has moved some failures downstream |
+| Tier-3 / generalization-first | Correctly asks whether progress transfers beyond math | Can become premature if lanes are stubbed or if operators are built before measured gaps are pinned |
+
+The superior plan is a fused route:
+
+```text
+A. Finish the active GSM8K corridor to prove safe capability growth.
+B. Build a CORE General Panel over existing lanes and Tier-3 readiness.
+C. Populate missing Tier-3/generalization lanes with real numbers.
+D. Build only the operators revealed by those measurements.
+E. Add external benchmark adapters once they produce actionable deltas.
+```
+
+---
+
+## Phase 1 — ADR-0163 Phase E: GSM8K lift history
+
+### Branch recommendation
+
+```bash
+feat/adr-0163-phase-e-gsm8k-lift-history
+```
+
+### Purpose
+
+Make every GSM8K improvement undeniable, versioned, and reviewable before further solver changes land.
+
+### Deliverables
+
+```text
+evals/gsm8k_math/train_sample/v1/baselines/
+evals/gsm8k_math/train_sample/v1/history/
+evals/gsm8k_math/train_sample/v1/lift_report.schema.json
+evals/gsm8k_math/train_sample/v1/build_lift_report.py
+```
+
+Each run should emit a `LiftReport` shaped roughly as:
+
+```json
+{
+  "schema_version": 1,
+  "base_sha": "...",
+  "head_sha": "...",
+  "counts_before": {"correct": 3, "refused": 47, "wrong": 0},
+  "counts_after": {"correct": 0, "refused": 0, "wrong": 0},
+  "delta": {"correct": 0, "refused": 0, "wrong": 0},
+  "newly_correct_cases": [],
+  "newly_wrong_cases": [],
+  "refusal_shift_histogram": {},
+  "trace_hash_stability": 1.0,
+  "wrong_zero_preserved": true
+}
+```
+
+### Acceptance gates
+
+```text
+wrong == 0
+trace/run determinism stable
+append-only history
+case-level refusal shifts visible
+current baseline pinned at 3/47/0
+```
+
+### Why this comes first
+
+Without this, the next solver lift becomes narrative rather than evidence.  ADR-0163's power is the corridor: measure, widen, replay, ratify, re-measure.  Phase E makes that loop durable.
+
+---
+
+## Phase 2 — ADR-0163 D.5: solver composition over admitted anchors
+
+### Branch recommendation
+
+```bash
+feat/adr-0163-d5-solver-composition
+```
+
+### Purpose
+
+Convert already-admitted question/statement anchors into solvable graph state.
+
+Recent question grammar work widened admissibility but did not lift correctness beyond `3/47/0`.  Several cases now reach `no branch produced a solvable graph`, showing the bottleneck has moved from grammar into solver-side composition.
+
+### Target composition shapes, in priority order
+
+| Target | Why first |
+|---|---|
+| earnings-rate composition | Common and checkable: `makes $18/hour`, `earns X per Y` |
+| profit-target composition | Cost/revenue/target problems become solvable only when target arithmetic composes |
+| unit partition composition | Common GSM8K surface: split into sections, groups, packs, bags |
+| fractional transfer/change | Needed for `1/4 of`, `decrease to 3/4`, `half of` |
+| comparative delta semantics | Must stay gated until `how many more` computes a difference, not a total |
+
+### Acceptance gate
+
+```text
+GSM8K train_sample_v1:
+  correct >= 10
+  wrong == 0
+```
+
+This is ADR-0163 Round 1 exit.
+
+### Non-negotiables
+
+- No answer-producing fast path that bypasses verifier discipline.
+- No broad regex widening that raises wrong risk without solver semantics.
+- Comparative surfaces remain detection-gated until delta semantics are implemented.
+- Any candidate that would increase `wrong` is rejected by replay, not accepted by operator judgment.
+
+---
+
+## Phase 3 — CORE General Panel v0
+
+### Branch recommendation
+
+```bash
+feat/evals-core-general-v0
+```
+
+### Purpose
+
+Create a broad internal measurement panel before using broad external benchmarks as steering targets.
+
+### Proposed command
+
+```bash
+core eval panel core_general_v0 --json
+```
+
+### Panel contents
+
+```text
+core_general_v0:
+  gsm8k_train_sample_v1
+  math_capability_axes_G1_G5_S1
+  cognition
+  provenance
+  calibration
+  monotonic_learning
+  symbolic_logic
+  adversarial_identity
+  refusal_taxonomy
+  realizer_guard
+  workbench_chat_smoke
+  tier3_readiness
+```
+
+### Panel metrics
+
+```text
+correct
+wrong
+refused
+decoded_unarticulated
+trace_hash_stability
+replay_equivalence_rate
+versor_condition_max
+unknown_domain_gate_honored
+proposal_mutation_count
+unratified_mutation_count
+cross_case_determinism
+```
+
+### Acceptance gates
+
+```text
+single JSON report emitted
+all included lane reports referenced by path/SHA
+no fake pass for missing lanes
+TBD/stub lanes represented as readiness gaps, not success
+wrong == 0 across answer-producing lanes
+```
+
+### Why this matters
+
+This creates an honest general dashboard without pretending CORE is already broad-benchmark-ready.  It gives the operator and engineers one panel to inspect capability growth, safety invariants, and lane readiness.
+
+---
+
+## Phase 4 — Tier-3 readiness and first numbers
+
+### Branch recommendation
+
+```bash
+feat/evals-tier3-readiness-and-first-numbers
+```
+
+### Purpose
+
+General advancement cannot be judged by GSM8K alone.  The missing question is whether CORE's learning and reasoning transfer.
+
+### Lanes to classify and/or populate
+
+| Lane | Question answered |
+|---|---|
+| `inference_closure` | Can CORE derive consequences rather than merely recall premises? |
+| `multi_step_reasoning` | Can CORE preserve context through chained operations? |
+| `symbolic_logic_v3+` | Can proposition structure become actual inference correctness? |
+| `cross_domain_transfer` | Can a learned structure in one domain map into another? |
+| `compositionality` | Can known smaller relations compose into unseen larger ones? |
+| `sample_efficiency` | How many reviewed examples are required before lift? |
+
+### Required classification
+
+Every Tier-3 lane should be classified as one of:
+
+```text
+runnable_with_numbers
+scaffold_only
+missing_cases
+missing_runner
+missing_operator
+```
+
+### Deliverables
+
+```text
+1. classify every Tier-3 lane
+2. run every runnable lane
+3. for non-runnable lanes, add contract.md / minimal dev cases / gaps.md as appropriate
+4. update core_general_v0 with tier3_readiness summary
+```
+
+### Acceptance gates
+
+```text
+no TBD row represented as progress
+all runnable lanes have deterministic reports
+all non-runnable lanes have explicit reason and next engineering dependency
+```
+
+---
+
+## Phase 5 — Structural pattern recognizer v1
+
+### Branch recommendation
+
+```bash
+feat/structural-pattern-recognizer-v1
+```
+
+### Purpose
+
+Enable measured cross-domain transfer without hand-waving.
+
+### Build order
+
+1. **Structural pattern recognizer over PropositionGraph**
+   - relation-shape extraction
+   - variable role labeling
+   - source/target domain separation
+   - deterministic canonical pattern digest
+
+2. **Matched-control transfer eval**
+   - same structure, different vocabulary/domain
+   - A-arm seeded, B-arm unseeded
+   - require B-arm improvement only when structural isomorphism exists
+
+3. **Cross-domain transfer operator**
+   - only after matched-control evidence identifies the exact transfer gap
+
+### Acceptance gates
+
+```text
+B-arm lift > 0
+A-arm unchanged
+wrong == 0
+no unratified corpus mutation
+trace hashes stable
+canonical pattern digests stable
+```
+
+### Guardrail
+
+Do not build a grand transfer operator before the structural recognizer and matched-control eval prove the shape of the gap.
+
+---
+
+## Phase 6 — Spatial / geometry OOD lane
+
+### Branch recommendation
+
+```bash
+feat/evals-spatial-geometry-ood-v1
+```
+
+### Purpose
+
+Test a domain where CORE's substrate should eventually have structural advantage rather than merely chase transformer-style text benchmarks.
+
+### Start text-only
+
+Do **not** start with image geometry.  Start with text-only spatial/geometric reasoning.
+
+### Proposed cases
+
+```text
+relative position
+containment
+intersection
+distance/order relations
+simple geometric transformations
+diagram-free Euclidean word problems
+```
+
+### Deliverables
+
+```text
+evals/spatial_geometry_ood/contract.md
+evals/spatial_geometry_ood/dev/cases.jsonl
+evals/spatial_geometry_ood/public/v1/cases.jsonl
+evals/spatial_geometry_ood/runner.py
+```
+
+---
+
+## External benchmark adapter ladder
+
+External benchmarks should validate the internal substrate; they should not blindly steer development while the substrate is still missing operators.
+
+| Stage | External benchmark | Use |
+|---|---|---|
+| Now | None as primary | Existing internal signals are more actionable |
+| After GSM8K `>=10/50/0` | GSM8K public split | Same substrate, broader sample |
+| After Tier-3 numbers exist | BBH-lite | Reasoning-shape diagnostic |
+| After factuality lane exists | SimpleQA-lite | Maps cleanly to correct / incorrect / not attempted |
+| After science packs mature | GPQA subset | Science reasoning diagnostic |
+| After broad seeded knowledge exists | MMLU-Pro | Otherwise mostly measures missing lexicon |
+| After code substrate exists | HumanEval / SWE-bench Verified | Requires code-generation / repo-repair substrate |
+| After visual ingestion exists | MMMU | Requires multimodal substrate |
+| After durable tool runtime exists | Tau-bench | Requires agent/tool action semantics |
+
+---
+
+## What not to do yet
+
+1. **Do not make MMLU-Pro the next primary objective.**
+   - It will mostly measure missing broad lexicon/domain packs at this stage.
+
+2. **Do not run LiveBench as a primary steering metric yet.**
+   - It is valuable later, but currently too broad to identify CORE-specific engineering deltas.
+
+3. **Do not build SWE-bench / HumanEval lanes before code substrate exists.**
+   - Otherwise the result is predictable refusal rather than useful diagnosis.
+
+4. **Do not build MMMU before visual ingestion exists.**
+   - It would test missing modality infrastructure, not reasoning.
+
+5. **Do not broaden answer surfaces by weakening refusal discipline.**
+   - A higher correct count with `wrong > 0` is a regression, not progress.
+
+6. **Do not represent stubbed Tier-3 lanes as green.**
+   - Readiness classification is acceptable; fake success is not.
+
+---
+
+## Recommended immediate PR sequence
+
+### PR 1 — GSM8K Phase E lift history
+
+```bash
+feat/adr-0163-phase-e-gsm8k-lift-history
+```
+
+Exit:
+
+```text
+LiftReport exists
+history append-only
+current baseline pinned at 3/47/0
+wrong == 0 preserved
+```
+
+### PR 2 — D.5 solver composition
+
+```bash
+feat/adr-0163-d5-solver-composition
+```
+
+Exit:
+
+```text
+GSM8K train_sample_v1 correct >= 10
+wrong == 0
+```
+
+### PR 3 — CORE General Panel + Tier-3 readiness
+
+```bash
+feat/evals-core-general-v0-tier3-readiness
+```
+
+Exit:
+
+```text
+core eval panel core_general_v0 --json
+Tier-3 lane readiness classified
+all runnable Tier-3 lanes measured
+no fake TBD rows represented as progress
+```
+
+### PR 4 — Structural recognizer v1
+
+```bash
+feat/structural-pattern-recognizer-v1
+```
+
+Exit:
+
+```text
+canonical pattern digests
+matched-control transfer cases
+no cross-domain wrong answers
+```
+
+---
+
+## Final doctrine
+
+The master path is:
+
+```text
+GSM8K corridor proves safe capability growth.
+core_general_v0 proves broad internal measurement.
+Tier-3 lanes reveal generalization bottlenecks.
+Structural recognizer + transfer operator attack the measured bottleneck.
+External benchmarks validate once they become actionable.
+```
+
+This keeps CORE aligned with its strongest architectural claim: deterministic, traceable, refusal-first cognition whose capability growth is replay-verifiable rather than merely asserted.