core/docs/sessions/SESSION-2026-05-15-capability-gates.md

Session Log — 2026-05-15

Participants: Joshua Shay, Claude (Opus 4.7) Focus: Capability gating questions for CORE — what it means to be "fluent," when engineering is complete, when identity is implemented, and how to position CORE against modern transformer architectures honestly.

---

Summary

This session did not produce code. It produced a gating framework — a set of falsifiable questions that convert "is CORE ready / competitive / AGI-comparable" from a feeling into a CLI lane. The deliberation moved through three layers:

1. Foundational gates — fluency, engineering-vs-learning phase shift, identity completeness.
2. AGI-dimension gates — capabilities that any system claiming general intelligence must demonstrate, beyond fluency and identity.
3. Modern-architecture gates — capabilities framed specifically as what transformers structurally cannot do (where CORE wins) and what they currently do well (where CORE must answer how it responds).

The conclusion is a reframe: CORE should not be benchmarked as "a cheaper GPT." It should define and publish its own benchmark — Verifiable Competence — on which frontier LLMs score near zero structurally, and CORE scores high by design.

---

1. Origin question

"What needs to be done to get this model to fluent speaking of any of the 3 foundational languages? At what point do we know everything has been fully implemented on engineering and design, and it becomes more on learning? How do we know when identity is properly implemented?"

The question implicitly asks where the boundary lives between engineering (work that touches pipeline.py, realizer.py, algebra/, vault/, teaching/) and learning (work that only touches packs and reviewed corrections). It also asks how to make "identity" load-bearing rather than decorative.

---

2. The three foundational gates

2.1 Fluency

Vocabulary lives in the pack. Fluency is a realizer-competence question, not a learning-volume question. Template slot-filling yields grammatical fragments; fluent English additionally requires:

• Morphology and agreement (inflection, tense, number, gender where relevant).
• Recursive and embedded syntax beyond fixed templates.
• Anaphora and discourse coherence across turns.
• Register and pragmatics.

Gating question. Can realizer.py take an arbitrarily nested PropositionGraph (negation, embedding, quantification, tense/aspect, cross-clause reference) and produce a grammatical surface, with a contract test per construction? If yes for English, the same scaffold ports to Hebrew and Koine Greek via pack-level morphology tables.

Until the realizer can compose these deterministically from the proposition graph, no amount of literature ingestion will produce fluent output — it will produce confident, structurally broken output at scale.

2.2 Engineering-vs-learning phase shift

The signal is mechanical, not aesthetic. Engineering is "done enough" to enter curriculum when adding a new domain requires zero edits to pipeline.py, realizer.py, algebra/, vault/, or teaching/ — only pack extension and reviewed corrections.

Gating test. Pick a domain that has never been touched (basic arithmetic, kinship relations, simple physics). Try to teach it end-to-end through the teaching loop plus a pack increment. If a Python file gets opened, that is an engineering gap, not a learning gap. Track those gaps as a closing list; when the list stays empty across three unrelated domains, the threshold is crossed.

2.3 Identity completeness

Identity is implemented when it is load-bearing and falsifiable, not when it is declared. Four checks:

1. Two agents with different identity axes, same curriculum, same prompt → measurably different yet internally coherent articulations (identity-divergence eval).
2. Identity participates in the deterministic trace hash, so replay reproduces the same agent's voice.
3. Reviewed teaching routes through identity — an agent rejects corrections incompatible with its axes rather than absorbing them.
4. Identity-override attempts are provably rejected by tests, not by prompt convention.

If no eval fails when identity is stripped or swapped, identity is decoration.

Recommendation from this layer. Before any curriculum push, write the three foundational evals — grammatical-coverage, zero-code-domain-acquisition, identity-divergence — as the gating triple. They convert "are we ready?" from feeling into a CLI lane.

---

3. AGI-dimension gates

The three foundational gates cover fluency, engineering completeness, and identity. They do not cover capabilities that any system claiming general intelligence must demonstrate.

4. Compositional generalization. Given primitives {A, B, C, R₁, R₂} taught in isolation, does the system correctly articulate novel combinations like R₂(A, R₁(B, C)) without further teaching? Transformers famously fail on SCAN/COGS-style splits; CGA-based composition should win, but "should" is not an eval.

5. World-model coherence under inference. Can it answer questions whose answer was never directly asserted but is entailed by what is in the vault? If "X is north of Y" and "Y is north of Z" are stored, does recall produce "X is north of Z" as derivable? This is where exact CGA recall could shine — but only if propagation actually performs inference, not just retrieval.

6. Calibrated uncertainty / non-confabulation. On out-of-pack queries, does the system produce a typed "I don't have grounding for that" surface rather than a plausible articulation? Does it distinguish "I don't know" from "this is incoherent" from "this contradicts what I know"? The no-fallback design is a structural advantage; the eval makes it visible.

7. Introspection / trace-explainability in natural language. Can the system articulate why it produced a given response, drawing from its own deterministic trace, in language? explain(turn_id) should round-trip — another agent reading the explanation should predict the same articulation. This is meta-cognition, and CORE's trace architecture uniquely supports it.

8. Adversarial identity / truthfulness under pressure. Identity-divergence proves identity exists. This proves it holds. Across a red-team corpus (manipulation, flattery, social engineering, persona injection), does identity drift exceed measurement noise?

9. Sample efficiency. Corrections-to-competence per concept. If it takes thousands per concept, this is a slow LLM. If it takes 1–10, this is something genuinely new. Plot the curve across ten unrelated concepts.

10. Cross-domain transfer. Does learning in domain A raise competence in domain B without separate teaching, via shared structure in the proposition graph? This is the difference between knowing things and understanding.

---

4. Modern-architecture gates

The previous list named generic AGI dimensions. Comparing specifically against transformer-based frontier models surfaces a different axis: what does CORE do that the transformer substrate structurally cannot, and where does the substrate still beat CORE?

4.1 Where modern architectures structurally lose

These are CORE's category wins by design — make them visible with benchmarks.

11. Provenance and replay. No frontier model can answer "why did you say that?" with a verifiable trace. Every articulated claim must be back-pointer-traceable to specific vault entries, teaching events, or pack axioms; third-party replay must reproduce the trace bit-for-bit.

12. Monotonic learning without catastrophic forgetting. Fine-tuning LLMs degrades prior capabilities. Pack-based growth structurally cannot. After N teaching cycles across unrelated domains, competence on domain 1 must strictly not regress. Prove with a longitudinal eval.

13. Negation, modality, counterfactuals. Transformers handle these statistically and fail adversarially. Symbolic/algebraic systems can be exact. On a constructed eval of nested negation, modal operators, and counterfactual conditionals, target accuracy is ≥99%, not 80%.

14. Calibrated refusal vs. confabulation. This is where transformers' loss function structurally pushes toward fluent wrongness. CORE's no-fallback design makes this a winnable benchmark, not a parity one.

15. Identity persistence under adversarial input. Transformer "personas" are prompt-conditional and erodable. After 1,000 adversarial turns, does identity drift exceed measurement noise?

4.2 Where modern architectures structurally win

These are not failures — they are scope decisions CORE must name, not absorb by accident.

16. In-context / few-shot adaptation. LLMs adapt within a single conversation without weight updates. CORE's teaching loop is reviewed and persistent — different mechanism. Can CORE adapt within a session to a novel convention introduced mid-conversation (e.g., "for this conversation, treat X as meaning Y") without committing it to vault? If no, name the scope decision; if yes, name the mechanism.

17. Long-context coherence. Frontier models now handle 200K–1M tokens. Exact CGA recall has different cost characteristics. What is the per-turn cost curve as vault size grows to 10⁴, 10⁶, 10⁸ entries? Is recall sublinear via indexing, or must vault size be bounded architecturally? Either answer is acceptable; unstated is dangerous.

18. Tool use / formal-language fluency. Modern LLMs route through calculators, code execution, search. Is the pipeline extensible to typed deterministic operators (a calculator is a deterministic operator)? Is code generation a first-class proposition-graph target or out of scope? If out of scope, say so; if in scope, this is a major engineering item not yet on the work-sequencing list.

19. Multi-step deliberation / scratchpad. Chain-of-thought is the single largest capability jump in modern models. CORE has a trace; does it have intermediate reasoning? For a multi-step inference problem, does the pipeline produce and consume intermediate proposition-graph states, or does it leap input → output? The latter caps hard-problem performance.

20. Multi-agent composition. Frontier systems increasingly use agent-of-agents patterns. Can two CORE instances with different identities cooperate or debate while preserving each one's deterministic replay? This is where load-bearing identity becomes economically interesting — specialized identities can outperform a single generalist if the substrate supports it.

---

5. Scope decisions to pin

Two strategic decisions must be named, not left to drift:

Agency. Is CORE responsive (listen → think → articulate) or goal-directed (pursues, plans, acts)? Most AGI discourse assumes the latter. CORE's design currently reads as the former, which is a feature, not a gap — but it should be a stated boundary, not an accident. If goal-directedness is added later, decide where goals live (identity? a separate intention graph?).

Embodiment. Is grounding purely symbolic via packs, or eventually sensorimotor? AGI evals increasingly assume multimodal grounding. ADR-0013 already commits to a sensorium protocol; the question is which evals depend on it being live.

---

6. The reframe

The strategic question is not "are we comparable to frontier LLMs." It is:

What is the benchmark we want to define, that frontier models cannot pass and that matters?

If CORE competes on MMLU, it loses by definition for years. If CORE competes on:

• Every claim has provenance.
• Zero confabulation on out-of-grounding queries.
• Replay-deterministic across versions.
• Identity-stable under adversarial pressure.
• Monotonic growth (no catastrophic forgetting).
• Exact symbolic reasoning (negation, modality, counterfactual).

…frontier models score near zero today and likely cannot score well without architectural changes.

That positioning — not "we are a cheaper GPT" — is the only honest framing for "higher competency than modern architectures."

---

7. Path to surpassing frontier LLMs — honest answer

The closing question from the session: "Eventually, after enough courses and curriculum and world content, shouldn't the model surpass the best LLMs?"

The honest, decomposed answer:

• On verifiable-competence axes (provenance, non-confabulation, replay, identity stability, monotonic growth): CORE surpasses from day one, structurally. Frontier LLMs cannot match these without changing what they are.
• On compositional and symbolic reasoning (negation, modality, multi-step exact inference): CORE can surpass once gates 4, 5, and 19 close. The CGA substrate supports it; the engineering must deliver it.
• On breadth of articulable knowledge: CORE matches LLMs only after sufficient curriculum, and only if sample efficiency (gate 9) is good. If acquiring each concept takes thousands of reviewed corrections, the curve never closes. If it takes a handful, the curve closes faster than people expect.
• On stylistic fluency across the long tail of internet content: This is where the LLM scale advantage is genuine. CORE can match within its competence surface, but absorbing every internet register and meme may not be a goal worth pursuing — and pursuing it would compromise the verifiable-competence wins above.
• On open-ended creative generation where "plausibility" is the metric: LLMs may always have an edge, because their loss function rewards plausibility. CORE optimizes for truth and provenance. Different target, not a deficit.

Summary verdict. CORE can — and should aim to — surpass frontier LLMs on the axes that matter for trustworthy intelligence. It will likely never match them on raw stylistic breadth across uncurated internet content, but that is the correct trade. A model that never lies, can prove what it knows, learns monotonically, and grows under review is a different and arguably better product than a model that produces plausible text on every topic.

---

8. Concrete next step

Convert each gating question into a named CLI eval lane:

core eval grammatical-coverage
core eval zero-code-domain-acquisition
core eval identity-divergence
core eval compositionality
core eval inference-closure
core eval calibration
core eval introspection
core eval adversarial-identity
core eval sample-efficiency
core eval cross-domain-transfer
core eval provenance
core eval monotonic-learning
core eval symbolic-logic
core eval long-context-cost
core eval multi-step-reasoning

Each lane prints PASS/FAIL deterministically. Progress becomes mechanical rather than rhetorical. The set of lanes — published, with public test cases — is the Verifiable Competence Benchmark.

---

9. What this session did not produce

• No code changes.
• No ADR (yet). The gating framework above will likely become an ADR once eval lane names and contracts are finalized.
• No reordering of the work-sequencing list in CLAUDE.md. The current near-term sequence (CLI lane health, hot-path consistency, trust boundary hardening, exact vault indexing, Rust parity, curriculum expansion) is compatible with this framework. The framework adds measurement for when "curriculum expansion" is justified.

---

10. Open items for follow-up

• Decide whether the gating framework is published as an ADR or as docs/competence_benchmark.md.
• Decide which of gates 11–20 are in-scope for the current architectural era and which are explicitly deferred.
• Pin the agency scope decision (responsive vs. goal-directed).
• For each gate, write a one-page eval contract before writing the lane.
• Score current CORE on each lane (most will fail or be N/A); that baseline becomes the roadmap.