From f973e61bc2ee87464b00e4b18c28ba37d2f4b14b Mon Sep 17 00:00:00 2001 From: Shay Date: Fri, 15 May 2026 08:13:29 -0700 Subject: [PATCH] Add agent efficiency and security doctrine - update AGENTS.md with standing efficiency/performance and security doctrine - align CLAUDE.md with current performance/security expectations - update Copilot/Codex instructions with hot-path, trust-boundary, and CLI validation defaults - refresh work sequencing now that eval and calibration are on main --- .github/copilot-instructions.md | 49 +++++++++++++++++-- AGENTS.md | 87 +++++++++++++++++++++++++++++---- CLAUDE.md | 83 ++++++++++++++++++++++++++++--- 3 files changed, 199 insertions(+), 20 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 94663794..68826d88 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -24,6 +24,8 @@ The cognitive path is centered on: - `generate/graph_planner.py` - `generate/realizer.py` - `teaching/correction.py`, `teaching/review.py`, `teaching/store.py` +- `evals/*` +- `calibration/*` - `language_packs/data/en_core_cognition_v1` The runtime response contract is documented in `docs/runtime_contracts.md`. @@ -53,6 +55,36 @@ Forbidden hot-path repair sites: Do not add grade monitors, drift timers, watchdog repair functions, ANN/HNSW, cosine similarity, or approximate recall. +## Efficiency and Performance + +Treat performance as part of the architecture. Slow feedback causes poor +engineering decisions and hides regressions. + +When touching hot paths, prefer: + +- backend-dispatched algebra when semantics match +- import hoisting and removal of repeated structure-building +- deterministic immutable caches or safe copied data +- exact CGA batching/vectorization instead of approximate search +- small validation lanes and bounded eval cases for iterative work + +Do not improve speed by weakening invariants, skipping construction checks, +adding hot-path repair, using approximate recall, or mutating shared cached state +unsafely. + +## Security and Trust Boundaries + +When touching user-controlled text, dynamic imports, filesystem paths, CLI +reports, pack validators, or logs, enforce and test the trust boundary. + +Required defaults: + +- arbitrary-code execution must be explicit and opt-in +- unsafe pack IDs and path traversal must be rejected +- raw user text should not be leaked in expanded logging unless local/debug is explicit +- pack mutations stay proposal-only unless a reviewed path applies them +- report/file writes must be bounded to caller-specified paths with clear behavior + ## Surface Contract Keep these separate: @@ -90,20 +122,31 @@ core test --suite packs -q core test --suite runtime -q core test --suite algebra -q core test --suite full -q +core eval cognition ``` For a feature PR, run the smallest relevant suite and then `full` when practical. +## Current Work Sequence + +1. Keep CLI lanes and `core eval cognition` green. +2. Tighten hot-path backend consistency and semantics-preserving performance. +3. Harden pack/OOV/logging trust boundaries. +4. Add exact vault recall indexing/batching without approximate search. +5. Add Rust backend parity only after Python semantics are locked by tests. +6. Expand curriculum teaching after replay/eval/calibration remain deterministic. + ## PR Standard Every change should state: ```text -Capability added/protected: +Capability/performance/security boundary added or protected: Invariant protected: -CLI suite run: -No hidden normalization / stochastic fallback / unreviewed mutation: +CLI suite/eval run: +No hidden normalization / stochastic fallback / approximate recall / unreviewed mutation: +Trust boundary enforced when relevant: ``` Prefer small PRs. Do not combine baseline repair, feature work, and broad diff --git a/AGENTS.md b/AGENTS.md index deabe9a7..c5c81e81 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -118,8 +118,71 @@ Important modules: - `generate/realizer.py` / `generate/templates.py` — deterministic realization. - `teaching/*` — reviewed teaching/correction lifecycle. - `language_packs/data/en_core_cognition_v1` — compact cognition seed pack. +- `evals/*` — deterministic cognition evidence harness. +- `calibration/*` — bounded replay-based operator calibration. - `docs/runtime_contracts.md` — runtime response, memory, identity, and testing contracts. +## Efficiency and Performance Doctrine + +Performance is an architectural property. Do not treat it as an afterthought +that will be cleaned up after features land. + +Before modifying hot paths, identify whether the change touches: + +- algebra backend dispatch (`algebra/backend.py`) +- versor application / closure (`algebra/versor.py`) +- propagation (`field/propagate.py`) +- injection / OOV grounding (`ingest/gate.py`) +- vault recall/storage (`vault/store.py`) +- session turn loop (`session/context.py`) +- runtime/eval loops (`chat/runtime.py`, `core/cognition/*`, `evals/*`) + +Required approach: + +1. Prefer semantics-preserving cleanup before new knobs. +2. Route hot-path algebra through `algebra.backend` when semantics are identical. +3. Hoist repeated imports and repeated structure-building out of tight loops. +4. Cache only deterministic, immutable, or safely copied structures. +5. Keep exact CGA recall exact; optimize scans with batching/vectorization, not approximation. +6. Prove speed-oriented changes through existing CLI lanes and, when practical, small benchmark/eval evidence. + +Never improve speed by: + +- weakening `versor_condition` thresholds +- skipping closure checks at construction boundaries +- adding hot-path repair/normalization +- replacing exact CGA with cosine/ANN/HNSW +- hiding failures behind retry loops without telemetry +- mutating shared cached state unsafely + +For test speed, prefer better validation lanes, small-case eval tests, fixture reuse where safe, and pack/load caching with immutability guarantees. Do not delete meaningful tests just because the full suite is slow. + +## Security and Trust-Boundary Doctrine + +Every agent must identify user-controlled input and dynamic execution surfaces. +Security hardening should be built into the same PRs that touch those surfaces. + +High-risk surfaces: + +- `core pack validate` dynamic validator execution +- language/source pack loading +- OOV token grounding and logs +- CLI commands that echo user input +- report/eval output paths +- pack mutation proposals +- any future file/network/database integration + +Required approach: + +1. Make arbitrary-code execution explicit and opt-in. +2. Reject path traversal and unsafe pack IDs before filesystem access. +3. Centralize display/log handling for user-controlled strings when expanding logging. +4. Keep pack mutation proposal-only unless an explicit reviewed path applies it. +5. Avoid leaking raw sensitive tokens in errors/reports unless the command is explicitly local/debug. +6. Preserve deterministic replay evidence for security-relevant decisions. + +Do not add hidden background execution, dynamic imports from untrusted paths, shell passthroughs, or broad filesystem writes without an explicit trust boundary and tests. + ## Chat Surface Contract Do not collapse these fields: @@ -171,11 +234,12 @@ Never compute a manifest checksum from a pre-serialization Python string. Current capability sequence: -1. Keep CLI test suites green. -2. Integrate semantic seed surfaces into realizer/cognition quality. -3. Add cognitive eval harness. -4. Add operator calibration from deterministic replay evidence. -5. Expand curriculum teaching only after the loop remains deterministic. +1. Keep CLI test suites and `core eval cognition` green. +2. Tighten hot-path backend consistency and semantics-preserving performance. +3. Harden pack/OOV/logging trust boundaries. +4. Add exact vault recall indexing/batching without approximate search. +5. Add Rust backend parity only after Python semantics are locked by tests. +6. Expand curriculum teaching only after replay/eval/calibration remain deterministic. Do not add dashboards, broad infra, or large test matrices unless they directly protect or unlock one of the above capabilities. @@ -192,6 +256,7 @@ core test --suite packs -q core test --suite runtime -q core test --suite algebra -q core test --suite full -q +core eval cognition ``` For targeted work, run the smallest relevant suite first, then `full` before @@ -206,6 +271,9 @@ Good tests protect: - identity protection - reviewed correction safety - semantic pack loadability and deterministic ordering +- eval/calibration determinism +- hot-path performance semantics +- explicit security trust boundaries Bad tests preserve private helper shapes, stale constructors, punctuation trivia outside documented contracts, or legacy behavior that contradicts the current @@ -216,10 +284,11 @@ architecture. Every PR must answer: ```text -What cognitive capability did this add or protect? +What cognitive capability, performance property, or security boundary did this add or protect? What invariant proves it did not corrupt the field? -Which CLI suite proves the relevant lane? -Did it avoid hidden normalization, stochastic fallback, and unreviewed mutation? +Which CLI suite/eval proves the relevant lane? +Did it avoid hidden normalization, stochastic fallback, approximate recall, and unreviewed mutation? +If it touches user input, files, dynamic imports, or logs, what trust boundary was enforced? ``` Prefer small, load-bearing PRs. Do not mix baseline fixes, feature work, and @@ -230,4 +299,4 @@ large reorganization unless the coupling is unavoidable. Raw input becomes a closed versor field once; thought evolves through exact versor transitions and CGA recall; cognition is structured as intent, proposition graph, articulation target, deterministic realization, reviewed -memory, and replayable trace. +memory, eval/calibration replay, and traceable evidence. diff --git a/CLAUDE.md b/CLAUDE.md index 79428ff7..5ee0a8d9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -22,6 +22,7 @@ CognitiveTurnPipeline -> deterministic realizer -> generation walk telemetry -> reviewed teaching loop + -> deterministic eval/calibration replay -> deterministic trace hash ``` @@ -99,8 +100,71 @@ runtime path. Vault recall is exact and deterministic. - `generate/realizer.py` and `generate/templates.py` — deterministic surface realization. - `teaching/correction.py`, `teaching/review.py`, `teaching/store.py` — reviewed teaching loop. - `language_packs/data/en_core_cognition_v1` — core cognition semantic seed pack. +- `evals/*` — deterministic cognition eval harness. +- `calibration/*` — bounded replay-based calibration. - `docs/runtime_contracts.md` — response, telemetry, memory, identity, and testing contracts. +## Efficiency and Performance Doctrine + +Performance is part of correctness for this project because slow feedback hides +regressions and encourages unsafe shortcuts. Do not defer obvious hot-path or +validation-lane issues until “later.” + +Before changing hot paths, identify whether the change touches: + +- algebra backend dispatch +- versor application / closure +- propagation +- injection / OOV grounding +- vault recall/storage +- session turn loop +- runtime/eval loops + +Required approach: + +1. Prefer semantics-preserving cleanup before new knobs. +2. Use `algebra.backend` for hot-path algebra when semantics are identical. +3. Hoist repeated imports and repeated structure-building out of tight loops. +4. Cache deterministic immutable data only, or return safe copies. +5. Keep exact CGA recall exact; use batching/vectorization, not approximation. +6. Validate speed-oriented changes through CLI lanes and `core eval cognition`. + +Never improve speed by weakening closure thresholds, skipping construction +checks, adding hot-path repair, replacing exact CGA with approximate metrics, or +mutating shared cached state unsafely. + +For test speed, prefer curated CLI lanes, small-case eval tests, safe fixture +reuse, and immutable pack/load caching. Do not delete meaningful tests just +because the full suite is slow. + +## Security and Trust Boundaries + +Any change that touches user-controlled text, filesystem paths, dynamic imports, +reports, pack validators, or logs must state the trust boundary. + +High-risk surfaces: + +- `core pack validate` dynamic validator execution. +- language/source pack loading. +- OOV token grounding and error messages. +- CLI commands that echo user content. +- eval/report output paths. +- pack mutation proposals. +- future file/network/database integrations. + +Required approach: + +1. Make arbitrary-code execution explicit and opt-in. +2. Reject path traversal and unsafe pack IDs before filesystem access. +3. Centralize safe display/log handling before increasing logging. +4. Keep pack mutation proposal-only unless a reviewed path applies it. +5. Avoid leaking raw sensitive tokens unless the command is explicitly local/debug. +6. Preserve deterministic replay evidence for security-relevant decisions. + +Do not add hidden background execution, dynamic imports from untrusted paths, +shell passthroughs, or broad filesystem writes without tests and a documented +trust boundary. + ## Runtime Surface Contract Keep these distinct: @@ -155,6 +219,7 @@ core test --suite packs -q core test --suite runtime -q core test --suite algebra -q core test --suite full -q +core eval cognition ``` Run the smallest relevant suite first, then `full` before merge when practical. @@ -163,11 +228,12 @@ Run the smallest relevant suite first, then `full` before merge when practical. Current near-term sequence: -1. Keep CLI lanes green. -2. Integrate semantic seed relations into realizer/cognition quality. -3. Add cognitive eval harness. -4. Add deterministic operator calibration from replay evidence. -5. Expand curriculum teaching after the loop is stable. +1. Keep CLI lanes and `core eval cognition` green. +2. Tighten hot-path backend consistency and semantics-preserving performance. +3. Harden pack/OOV/logging trust boundaries. +4. Add exact vault recall indexing/batching without approximate search. +5. Add Rust backend parity only after Python semantics are locked by tests. +6. Expand curriculum teaching after replay/eval/calibration remain deterministic. Avoid broad docs-first churn, dashboard work, or large infrastructure unless it unlocks one of these steps. @@ -177,10 +243,11 @@ unlocks one of these steps. Before opening or merging, answer: ```text -What capability did this add or protect? +What capability, performance property, or security boundary did this add/protect? Which invariant proves the field remains valid? -Which CLI suite proves the lane? -Did this avoid hidden normalization, stochastic fallback, and unreviewed mutation? +Which CLI suite/eval proves the lane? +Did this avoid hidden normalization, stochastic fallback, approximate recall, and unreviewed mutation? +If it touches user input, files, dynamic imports, or logs, what trust boundary was enforced? ``` Prefer small, load-bearing PRs with clear evidence.