core-labs/core
core-labs/core
1
0
Fork 0
Code Issues 12 Pull requests Projects Releases Packages Wiki Activity Actions
core/docs/plans/core-general-advancement-plan.md

13 KiB
Raw Permalink Blame History

CORE General Advancement Plan

Status: Proposed
Date: 2026-05-26
Scope: Documentation / planning only
Anchor context: ADR-0163 GSM8K corridor, ADR-0161 HITL queue, ADR-0160/0162 Workbench, existing eval methodology

Executive summary

CORE should not advance by chasing a broad external benchmark suite first. The next master path is a three-lane sequence:

  1. Exploit the live ADR-0163 GSM8K corridor now because it is the freshest, proven, end-to-end capability-growth mechanism.
  2. Instrument broader Tier-3 / generalization lanes in parallel so general advancement is measured rather than assumed.
  3. Delay broad external benchmarks until they are diagnostic adapters, not vanity refusal generators.

The governing principle is:

truth before coverage
refusal before confabulation
replay before claims

CORE's current advantage is not raw benchmark breadth. Its current advantage is that capability growth can be measured under deterministic replay, typed refusal, operator-ratified mutation, and wrong == 0 discipline. The plan below preserves that advantage while creating a route toward broader capability.

Current pinned state

The current mainline state after the ADR-0163 corridor work is:

GSM8K train_sample_v1:
  correct: 3
  refused: 47
  wrong: 0
  exit: failed, correct_min=10

Interpretation:

  • This is the first measurable non-zero GSM8K lift.
  • The wrong == 0 invariant remains intact.
  • The result is still far from an expert-capability claim.
  • The latest question-grammar extension moved some cases from question-level refusal into solver-side refusal, which means the current bottleneck is no longer only grammar/admission; it is increasingly solver composition over admitted anchors.

The current refusal profile says the next capability work should target composition semantics, not broad benchmark breadth.

Comparison of candidate plans

Three planning directions were considered:

Direction Strength Failure mode
GSM8K-first corridor Grounded in the freshest live mechanism; has clear measurable gates Can become narrow if it ignores generalization lanes
Refusal-taxonomy / parser expansion Correctly protects wrong == 0 and focuses on admission gaps Stale if it assumes the current result is still full refusal; newest work has moved some failures downstream
Tier-3 / generalization-first Correctly asks whether progress transfers beyond math Can become premature if lanes are stubbed or if operators are built before measured gaps are pinned

The superior plan is a fused route:

A. Finish the active GSM8K corridor to prove safe capability growth.
B. Build a CORE General Panel over existing lanes and Tier-3 readiness.
C. Populate missing Tier-3/generalization lanes with real numbers.
D. Build only the operators revealed by those measurements.
E. Add external benchmark adapters once they produce actionable deltas.

Phase 1 — ADR-0163 Phase E: GSM8K lift history

Branch recommendation

feat/adr-0163-phase-e-gsm8k-lift-history

Purpose

Make every GSM8K improvement undeniable, versioned, and reviewable before further solver changes land.

Deliverables

evals/gsm8k_math/train_sample/v1/baselines/
evals/gsm8k_math/train_sample/v1/history/
evals/gsm8k_math/train_sample/v1/lift_report.schema.json
evals/gsm8k_math/train_sample/v1/build_lift_report.py

Each run should emit a LiftReport shaped roughly as:

{
  "schema_version": 1,
  "base_sha": "...",
  "head_sha": "...",
  "counts_before": {"correct": 3, "refused": 47, "wrong": 0},
  "counts_after": {"correct": 0, "refused": 0, "wrong": 0},
  "delta": {"correct": 0, "refused": 0, "wrong": 0},
  "newly_correct_cases": [],
  "newly_wrong_cases": [],
  "refusal_shift_histogram": {},
  "trace_hash_stability": 1.0,
  "wrong_zero_preserved": true
}

Acceptance gates

wrong == 0
trace/run determinism stable
append-only history
case-level refusal shifts visible
current baseline pinned at 3/47/0

Why this comes first

Without this, the next solver lift becomes narrative rather than evidence. ADR-0163's power is the corridor: measure, widen, replay, ratify, re-measure. Phase E makes that loop durable.

Phase 2 — ADR-0163 D.5: solver composition over admitted anchors

Branch recommendation

feat/adr-0163-d5-solver-composition

Purpose

Convert already-admitted question/statement anchors into solvable graph state.

Recent question grammar work widened admissibility but did not lift correctness beyond 3/47/0. Several cases now reach no branch produced a solvable graph, showing the bottleneck has moved from grammar into solver-side composition.

Target composition shapes, in priority order

Target Why first
earnings-rate composition Common and checkable: makes $18/hour, earns X per Y
profit-target composition Cost/revenue/target problems become solvable only when target arithmetic composes
unit partition composition Common GSM8K surface: split into sections, groups, packs, bags
fractional transfer/change Needed for 1/4 of, decrease to 3/4, half of
comparative delta semantics Must stay gated until how many more computes a difference, not a total

Acceptance gate

GSM8K train_sample_v1:
  correct >= 10
  wrong == 0

This is ADR-0163 Round 1 exit.

Non-negotiables

  • No answer-producing fast path that bypasses verifier discipline.
  • No broad regex widening that raises wrong risk without solver semantics.
  • Comparative surfaces remain detection-gated until delta semantics are implemented.
  • Any candidate that would increase wrong is rejected by replay, not accepted by operator judgment.

Phase 3 — CORE General Panel v0

Branch recommendation

feat/evals-core-general-v0

Purpose

Create a broad internal measurement panel before using broad external benchmarks as steering targets.

Proposed command

core eval panel core_general_v0 --json

Panel contents

core_general_v0:
  gsm8k_train_sample_v1
  math_capability_axes_G1_G5_S1
  cognition
  provenance
  calibration
  monotonic_learning
  symbolic_logic
  adversarial_identity
  refusal_taxonomy
  realizer_guard
  workbench_chat_smoke
  tier3_readiness

Panel metrics

correct
wrong
refused
decoded_unarticulated
trace_hash_stability
replay_equivalence_rate
versor_condition_max
unknown_domain_gate_honored
proposal_mutation_count
unratified_mutation_count
cross_case_determinism

Acceptance gates

single JSON report emitted
all included lane reports referenced by path/SHA
no fake pass for missing lanes
TBD/stub lanes represented as readiness gaps, not success
wrong == 0 across answer-producing lanes

Why this matters

This creates an honest general dashboard without pretending CORE is already broad-benchmark-ready. It gives the operator and engineers one panel to inspect capability growth, safety invariants, and lane readiness.

Phase 4 — Tier-3 readiness and first numbers

Branch recommendation

feat/evals-tier3-readiness-and-first-numbers

Purpose

General advancement cannot be judged by GSM8K alone. The missing question is whether CORE's learning and reasoning transfer.

Lanes to classify and/or populate

Lane Question answered
inference_closure Can CORE derive consequences rather than merely recall premises?
multi_step_reasoning Can CORE preserve context through chained operations?
symbolic_logic_v3+ Can proposition structure become actual inference correctness?
cross_domain_transfer Can a learned structure in one domain map into another?
compositionality Can known smaller relations compose into unseen larger ones?
sample_efficiency How many reviewed examples are required before lift?

Required classification

Every Tier-3 lane should be classified as one of:

runnable_with_numbers
scaffold_only
missing_cases
missing_runner
missing_operator

Deliverables

1. classify every Tier-3 lane
2. run every runnable lane
3. for non-runnable lanes, add contract.md / minimal dev cases / gaps.md as appropriate
4. update core_general_v0 with tier3_readiness summary

Acceptance gates

no TBD row represented as progress
all runnable lanes have deterministic reports
all non-runnable lanes have explicit reason and next engineering dependency

Phase 5 — Structural pattern recognizer v1

Branch recommendation

feat/structural-pattern-recognizer-v1

Purpose

Enable measured cross-domain transfer without hand-waving.

Build order

  1. Structural pattern recognizer over PropositionGraph

    • relation-shape extraction
    • variable role labeling
    • source/target domain separation
    • deterministic canonical pattern digest

  2. Matched-control transfer eval

    • same structure, different vocabulary/domain
    • A-arm seeded, B-arm unseeded
    • require B-arm improvement only when structural isomorphism exists

  3. Cross-domain transfer operator

    • only after matched-control evidence identifies the exact transfer gap

Acceptance gates

B-arm lift > 0
A-arm unchanged
wrong == 0
no unratified corpus mutation
trace hashes stable
canonical pattern digests stable

Guardrail

Do not build a grand transfer operator before the structural recognizer and matched-control eval prove the shape of the gap.

Phase 6 — Spatial / geometry OOD lane

Branch recommendation

feat/evals-spatial-geometry-ood-v1

Purpose

Test a domain where CORE's substrate should eventually have structural advantage rather than merely chase transformer-style text benchmarks.

Start text-only

Do not start with image geometry. Start with text-only spatial/geometric reasoning.

Proposed cases

relative position
containment
intersection
distance/order relations
simple geometric transformations
diagram-free Euclidean word problems

Deliverables

evals/spatial_geometry_ood/contract.md
evals/spatial_geometry_ood/dev/cases.jsonl
evals/spatial_geometry_ood/public/v1/cases.jsonl
evals/spatial_geometry_ood/runner.py

External benchmark adapter ladder

External benchmarks should validate the internal substrate; they should not blindly steer development while the substrate is still missing operators.

Stage External benchmark Use
Now None as primary Existing internal signals are more actionable
After GSM8K >=10/50/0 GSM8K public split Same substrate, broader sample
After Tier-3 numbers exist BBH-lite Reasoning-shape diagnostic
After factuality lane exists SimpleQA-lite Maps cleanly to correct / incorrect / not attempted
After science packs mature GPQA subset Science reasoning diagnostic
After broad seeded knowledge exists MMLU-Pro Otherwise mostly measures missing lexicon
After code substrate exists HumanEval / SWE-bench Verified Requires code-generation / repo-repair substrate
After visual ingestion exists MMMU Requires multimodal substrate
After durable tool runtime exists Tau-bench Requires agent/tool action semantics

What not to do yet

  1. Do not make MMLU-Pro the next primary objective.

    • It will mostly measure missing broad lexicon/domain packs at this stage.

  2. Do not run LiveBench as a primary steering metric yet.

    • It is valuable later, but currently too broad to identify CORE-specific engineering deltas.

  3. Do not build SWE-bench / HumanEval lanes before code substrate exists.

    • Otherwise the result is predictable refusal rather than useful diagnosis.

  4. Do not build MMMU before visual ingestion exists.

    • It would test missing modality infrastructure, not reasoning.

  5. Do not broaden answer surfaces by weakening refusal discipline.

    • A higher correct count with wrong > 0 is a regression, not progress.

  6. Do not represent stubbed Tier-3 lanes as green.

    • Readiness classification is acceptable; fake success is not.

Recommended immediate PR sequence

PR 1 — GSM8K Phase E lift history

feat/adr-0163-phase-e-gsm8k-lift-history

Exit:

LiftReport exists
history append-only
current baseline pinned at 3/47/0
wrong == 0 preserved

PR 2 — D.5 solver composition

feat/adr-0163-d5-solver-composition

Exit:

GSM8K train_sample_v1 correct >= 10
wrong == 0

PR 3 — CORE General Panel + Tier-3 readiness

feat/evals-core-general-v0-tier3-readiness

Exit:

core eval panel core_general_v0 --json
Tier-3 lane readiness classified
all runnable Tier-3 lanes measured
no fake TBD rows represented as progress

PR 4 — Structural recognizer v1

feat/structural-pattern-recognizer-v1

Exit:

canonical pattern digests
matched-control transfer cases
no cross-domain wrong answers

Final doctrine

The master path is:

GSM8K corridor proves safe capability growth.
core_general_v0 proves broad internal measurement.
Tier-3 lanes reveal generalization bottlenecks.
Structural recognizer + transfer operator attack the measured bottleneck.
External benchmarks validate once they become actionable.

This keeps CORE aligned with its strongest architectural claim: deterministic, traceable, refusal-first cognition whose capability growth is replay-verifiable rather than merely asserted.