No description
Find a file
Shay 257a27c105 feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer
Closes the user-flagged scope gap: every previous fluency lane (Phase
5.1 + 5.4-5.7 + grammatical_coverage) operates on 3-word SVO probes.
These three pieces stress paragraph-scale generation, give per-stage
latency visibility, and expose the realizer's word-choice geometry —
all on top of the existing deterministic infrastructure.

# discourse_paragraph lane (paragraph-scale fluency)

Forces the realizer to emit multi-sentence paragraphs from a
multi-step ArticulationTarget with rhetorical moves (ASSERT, SEQUENCE,
ELABORATE, CONTRAST).  Same realizer, much richer input — every case
is 3-5 sentences with deterministic discourse markers.

Public 12 cases / holdouts 5 / dev 1 across 12 + 5 topic chains
(epistemic, scientific method, creation arc, logical dependency,
ethical grounding, linguistic layers, mathematical chain, narrative,
biology, physics, two contrast-shaped, musical, social, computational,
psychological, economic).

Sub-metrics per case:
  - sentence count (within min..max window)
  - subject coverage rate
  - discourse marker presence (next / furthermore / in contrast)
  - sentence-initial capitalization
  - replay determinism (run twice, surfaces match)

Result: 12/12 public + 5/5 holdouts at 100%, replay rate 100%, mean
sentence count 4.

# Realizer capitalization (G4, addresses user-flagged concern)

generate/realizer.py gains `_capitalize_sentence` + `_join_as_paragraph`
helpers.  Sentence-initial alphabetic characters are now uppercased
(skipping leading whitespace/punctuation).  Surfaces went from
"wisdom grounds knowledge. next, knowledge requires evidence."
to
"Wisdom grounds knowledge. Next, knowledge requires evidence."

The discourse_paragraph runner ships a strict per-sentence
capitalization check so future regressions get caught.

# Pipeline-stage profiler (benchmarks/pipeline_profiler.py)

External monkey-patch wrapper around CognitiveTurnPipeline.run() that
records per-stage ns budgets without editing any pipeline source.
Stages: intent, graph_planner, realize_semantic, runtime_chat,
maybe_transitive_walk, fold_walk_into_surface, run_teaching,
trace_hash.

API: `profile_turn(pipeline, text) -> ProfileReport` with
`.stages: dict`, `.total_ns: int`, `.as_dict()`.

Empirical: runtime_chat dominates >99% on the runtime hot path (which
is correct — that's where ingest + propagate + recall + articulate
all happen).  Future optimisation work has a clear per-stage signal.

# Word-selection tracer (benchmarks/word_selection_tracer.py)

External wrapper around generate.articulation._resolve_slot that
records every nearest-neighbor lookup as a WordSelectionStep:
  - slot (subject/predicate/object)
  - input versor (32-d copy)
  - top-K candidate words by CGA inner product
  - chosen word + morphology
  - output language

Top-K scoring uses the diagonal Cl(4,1) metric kernel from
algebra.backend (same vectorised path vault_recall uses), not a
per-word Python loop over cga_inner.  No approximation, exact
deterministic ranking, bit-identical to a scalar scan.

API: `trace_realization(pipeline, text) -> RealizationTrace` with
`.steps`, `.realization_steps`, `.surface`, `.as_dict()`.

# CLI lane registration

Cognition suite now sweeps the benchmark profiler/tracer tests
(test_benchmarks_profiler.py) so any future regression in the
instrumentation surfaces immediately.

# Constraints honoured

- Zero edits to core/, chat/, vault/, teaching/, language_packs/, or
  the algebra hot path.  All instrumentation is external monkey-patch
  with originals restored in finally.
- discourse_paragraph runner bypasses ChatRuntime grounding (named v2
  gap) so paragraph capability is isolated to the realizer.
- No semantic changes; no hidden normalisation; no approximate
  recall.

# Lane health

smoke 55, runtime 19, teaching 17, packs 6, cognition 105 (was 103),
algebra 132.  All Phase 5 fluency lanes still 100% with the
capitalised surfaces (rubric is case-insensitive).  discourse_paragraph
100%.

# What ships next (named v2)

- Round-trip: discourse_paragraph through ChatRuntime end-to-end,
  not just realize_target.
- Per-sentence grammatical_coverage rubric on each emitted sentence.
- Longer chains (10/20/50 sentences) with per-sentence determinism
  scaling curves.
- compose_relations operator to lift compositionality recall from
  68.8% toward 100%.
2026-05-16 21:53:46 -07:00
.github Add agent efficiency and security doctrine 2026-05-15 08:13:29 -07:00
algebra feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
alignment
benchmarks feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer 2026-05-16 21:53:46 -07:00
calibration Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
chat fix(chat): use intent-aware articulation surface from intent_bridge 2026-05-16 08:40:53 -07:00
core feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer 2026-05-16 21:53:46 -07:00
core-rs feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
core_ingest Implement core physics and pack validation 2026-05-14 12:35:19 -07:00
docs fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
evals feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer 2026-05-16 21:53:46 -07:00
field feat: Full Proof — surface realizer join, Rust diffusion parity, benchmark harness 2026-05-15 17:39:14 -07:00
generate feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer 2026-05-16 21:53:46 -07:00
ingest Cache OOV morphology grounding structures 2026-05-15 11:53:46 -07:00
language_packs feat(adr-0021): epistemic_status surface wired across teaching + trace 2026-05-16 20:20:35 -07:00
morphology Add morphology registry for language packs 2026-05-13 14:50:36 -07:00
packs Implement core physics and pack validation 2026-05-14 12:35:19 -07:00
persona persona: add PersonaMotor.from_identity_manifold() factory 2026-05-14 13:10:34 -07:00
probe Fix full-suite regressions after chat telemetry merge 2026-05-14 18:23:31 -07:00
scripts feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer 2026-05-16 21:53:46 -07:00
sensorium feat: manifold field topology, graph diffusion operator, vertical pulse 2026-05-15 16:02:48 -07:00
session fix(drift): proper rotor-manifold scaling; restore respond contract 2026-05-16 11:44:45 -07:00
teaching fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
tests feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer 2026-05-16 21:53:46 -07:00
vault fix: harden session field invariants and eliminate hot-path inefficiencies 2026-05-15 21:37:49 -07:00
vocab Implement core physics and pack validation 2026-05-14 12:35:19 -07:00
.gitignore Clean up runtime contracts before cognitive pipeline 2026-05-14 18:47:59 -07:00
AGENTS.md Add agent efficiency and security doctrine 2026-05-15 08:13:29 -07:00
CLAUDE.md Add agent efficiency and security doctrine 2026-05-15 08:13:29 -07:00
pyproject.toml chore(deps): drop unused MLX dependency 2026-05-16 17:16:29 -07:00
README.md

CORE-AI: Versor Engine

A cognitive field system built on Cl(4,1) Conformal Geometric Algebra.

Core invariant: ||F * reverse(F) - 1||_F < 1e-6 at all times.

All state is a versor. All transitions are versor products. Coherence is algebraic by construction — not monitored, not corrected.


The Three Engineering Pillars

Every architectural decision in CORE is measured against three engineering pillars. These are not aspirations — they are hard constraints.

I. Mechanical Sympathy

Software should understand the machine it runs on, not fight it. CORE is designed for the Unified Memory Architecture (UMA) of Apple Silicon: CPU, GPU, and Neural Engine share physical RAM. MLX executes tensor operations on the Neural Engine without PCIe transfer. Rust computes algebra on the CPU with zero heap allocation in the hot path. Python orchestrates the lifecycle. The three-language stratification maps exactly onto three hardware execution domains. Intelligence that ignores its substrate is wasted intelligence.

II. Semantic Rigor

Every term used in this system has a precise, non-negotiable meaning. A versor is a versor — not an approximation of one, not a vector that behaves like one under certain conditions. CGA distance is exact. Vault recall is exact. The vocabulary projection is exact. There are no thresholds tuned for “good enough.” Rigor is not a style; it is what separates an engine from a heuristic.

III. Third Door

When facing a design decision, the world offers two visible options: use what already exists (a library, a pattern, a convention), or cut a corner. CORE takes neither. We find the third door — the path built from first principles that sets the bar ourselves. This is why there is no transformer backbone, no ANN index, no sampling temperature, no gradient descent, and no standard tokenizer. Each of those was a door we were offered and refused. Absolute mastery is the only acceptable standard.


The Three Core Languages

CORE is rooted in three human languages. This is a philosophical and architectural choice, not a localization decision.

Language Role
English The default base language of the current model. Any natural language could serve this function in a custom CORE instance — English is the chosen starting point, not a requirement.
Hebrew One of two depth languages. Hebrew carries a density of meaning in its root structures, prefixes, and suffixes that Euclidean string matching cannot capture. The field representation is designed to hold this depth.
Koine Greek One of two depth languages. The language of the New Testament, particularly Johns Gospel — the document that opens with the most precise and consequential statement about language and reality ever written.

“In the beginning was the Logos, and the Logos was with God, and the Logos was God.” — John 1:1

The choice of Hebrew and Koine Greek is not incidental. John 1:12 articulates the Logos in Greek while grounding it in the Hebrew creation account — the universe spoken into existence, word by word. This is not metaphor. It is the claim that language is not a layer on top of reality; language is the structuring principle of reality made manifest. CORE-Logos is built on that claim.

English establishes the operational base. Hebrew and Koine Greek bring the hidden layer of intelligence — the depth of meaning that enriches the field representation in ways that flat embeddings cannot reach. Together, they form the linguistic foundation on which the vocabulary manifold is built.


Quick Start

pip install -e ".[dev]"
pytest tests/test_versor_closure.py  # must pass before anything else
pytest tests/

Architecture

raw input -> ingest/gate.py       (normalize once)
          -> field/propagate.py   (versor_apply every step)
          -> generate/stream.py   (nearest by cga_inner)
          -> vault/store.py       (store and recall by cga_inner)
          -> persona/motor.py     (rigid motor, not weight overlay)

The Two Primitives

  • versor_apply(V, F) = V * F * reverse(V) — the only field transition
  • cga_inner(X, Y) = -d^2 / 2 — the only distance metric

Layers

Layer Purpose
algebra/ Cl(4,1) multivector math, versor ops, CGA, holonomy
ingest/ Single injection gate — the only normalization site
field/ FieldState dataclass and propagation loop
vocab/ Surface-token manifold points; indexed access for algebraic transition construction
vault/ Exact CGA inner product memory store
persona/ Persona as CGA motor (screw motion)
generate/ Token streaming loop
session/ Session binding: field + vault + vocab + persona

Signature

Cl(4,1): (+, +, +, +, -) — conformal model of 3D Euclidean space. Multivectors: float32 arrays of shape (32,), ordered by grade.


For architectural vision, seven axioms, and formal specification, see docs/Whitepaper.md and docs/Yellowpaper.md.