core-labs/core
core-labs/core
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
core/core/cli_test.py
Shay cfda71dc1c feat(evals): grammar round-trip instrument (Phase 1) 
The measurement foundation for docs/plans/grammar-unification-2026-07-26.md.

WHY: evals/deterministic_fluency reports 1.00 on all six predicates and
still passes "banana does the.", "wet ground rains the is." and
"is is is is." — it checks terminal punctuation, presence of a verb-shaped
token, and two anti-shape regexes. Heuristic predicates will always have
that failure mode, because grammaticality cannot be measured without a
grammar. So this lane measures agreement between the two halves of CORE
that already encode grammar, and requires the measurement to FAIL on salad.

Two directions, reported separately because they fail for different
reasons and have different remedies:

  G-round-trip  graph -> realize_target -> surface -> comprehend -> graph
  S-round-trip  surface -> comprehend -> graph -> categorical renderer -> surface

v1 baseline on main @ 9696443a:

  graph_cases            280      surface_cases            8
  g_write_rate         1.000      s_read_rate          1.000
  g_read_rate          0.000      s_renderable_rate    0.625
  g_exact_rate         0.000      s_surface_match_rate 0.000
                                  negative_cases          16
                                  reject_rate          1.000

g_read_rate and s_surface_match_rate are pins on measured DEFECTS, not
goals; they may be revised upward only. s_surface_match_rate = 0 is the
§1.7 categorical render defect caught by construction — the lane found it
without being told to look.

g_args_rate and g_predicates_rate are deliberately separate: high argument
agreement with low predicate agreement would mean the grammars align and
only the vocabulary is split, a materially different remedy from both
being low. That distinction decides the arc's direction (plan §6).

Design notes:
- The committed cases.jsonl is the SINGLE source for authored surfaces —
  no in-module duplicate, since a second copy of a corpus is the defect
  this arc exists to remove. Negative shuffles are DERIVED at run time so
  they cannot drift from the positives.
- The shuffles are lexically identical to positives (same vocabulary, same
  length, order destroyed) so the lane cannot pass by vocabulary-checking.
- Fixed rotation, not a PRNG, so reject_rate is byte-reproducible.
- The lane keeps a local copy of the reader's quantifier map ON PURPOSE so
  it never becomes a consumer of what it measures;
  test_quantifier_map_matches_reader fails loudly if the reader changes.
- _render_categorical deliberately reaches a private serving function: a
  lane measuring a private copy would measure what users never see.

Every guarantee is paired with a mutation test. The load-bearing one is
test_reject_rate_goes_red_when_the_reader_accepts_everything: an
accept-everything reader must drive reject_rate to 0.0. Without it,
reject_rate == 1.0 would be unfalsifiable — precisely the defect that
makes the existing fluency lane decoration.

Also documents plainly what round-trip does NOT prove: it measures mutual
intelligibility, not English quality. english_fluency_ood accepts "river
flows valley" and round-trip would be happy with it. No metric here may be
cited as evidence of prose quality.

scripts/measure_grammar_seam.py reproduces every number in the plan's §1
so a reader can check them instead of trusting them.

[Verification]: in-worktree on CPython 3.12.13, uv sync --locked —
smoke 621 (unchanged), deductive 364 (349 + 15 new). Lane SHA pins
verified separately. No serving code touched; new files plus one suite
registration line.
2026-07-26 16:30:49 -07:00

348 lines
14 KiB
Python
Raw Blame History

"""Curated pytest and ruff CLI command handlers."""
from __future__ import annotations
import argparse
import subprocess
import sys
from collections.abc import Sequence
from pathlib import Path
from typing import Protocol
TEST_SUITES: dict[str, tuple[str, ...]] = {
"fast": (
"tests/test_cli_test_suites.py",
"tests/test_cli_runner_contract.py",
"tests/test_runtime_config.py",
"tests/test_core_semantic_seed_pack.py",
"tests/test_intent_proposition_graph.py",
"tests/test_articulation_realizer_v2.py",
"tests/test_reviewed_teaching_loop.py",
"tests/test_cognitive_eval_harness.py",
),
"smoke": (
"tests/test_chat_runtime.py",
"tests/test_achat.py",
"tests/test_runtime_config.py",
"tests/test_cognitive_turn_pipeline.py",
"tests/test_audit_ledger_r7.py",
"tests/test_architectural_invariants.py",
"tests/test_cli_runner_contract.py",
# Audio sensorium lane — part of the smoke.yml PR gate (compiler,
# CRDT merge, eval gates, pack manifest, mount, teachers; ~3s).
# Listed explicitly so the local-first pre-push gate (AGENTS.md
# protocol) equals the CI gate rather than silently narrowing it.
"tests/test_audio_compiler.py",
"tests/test_audio_crdt_merge.py",
"tests/test_audio_eval_gates.py",
"tests/test_audio_pack_manifest.py",
"tests/test_audio_sensorium_mount.py",
"tests/test_audio_teachers.py",
# ADR-0043 — identity falsifiability: ratified identity packs must
# produce distinct, directionally-correct articulations, with a
# pack-invariant grounding/refusal floor and zero fabrication. Lives
# only under ``full`` historically, so a divergence regression cleared
# the PR gate and surfaced only post-merge. Promoted into smoke so
# the falsifiability claim blocks-on-regression rather than
# detect-after-merge.
"tests/test_pack_measurements_phase2.py",
# ADR-0253 dual-pack boundary — draft he/grc trees must not be serve imports.
"tests/test_pack_draft_serve_boundary.py",
# Register axis (ADR-0069/0071/0077) — the e2e tests here are the
# falsifiable contract that terse/convivial registers actually differ
# from neutral on pipeline-served surfaces. Red on main outside every
# gate for 2026-07-20..24 (the register-axis serving regression,
# docs/research/lane-drift-investigation-2026-07-24.md) because this
# file lived only under `full`; the lane-shas job was the sole gate
# that runs the demo lanes this axis shows up in, and its failures
# were conflated with an unrelated flake. Promoted so this axis can
# never silently regress again.
"tests/test_register_substantive_consumption.py",
# Deduction serving is LIVE (ADR-0256 ratified the flag ON), so its
# cross-stack provenance contract is a serving-path contract, not a
# capability-under-development one. Workbench's live chat route builds
# a bare ChatRuntime(), which means these composers decide Workbench
# turns too — and while their labels were unregistered the API coerced
# them to "none" and recorded a proved answer as ungrounded. ~3s.
"tests/test_workbench_deduction_provenance.py",
# Correction binding anchors to hash_surface, in lockstep with every
# substantive override of the served surface. Cheap, and the axis is
# invisible until it breaks silently.
"tests/test_prior_surface_deduction_binding.py",
# ADR governance. An ADR that governs a default-ON flag records a
# decision already in force, so leaving it "Proposed" makes the
# governance record assert something false about the running system —
# which is what happened to ADR-0256 while deduction serving was live.
# Filesystem + regex only, ~1.2s, and it is the record for a serving
# path, so it belongs on the pre-push gate rather than under `full`.
# test_adr_index.py rides along: it landed in #113 registered in NO
# curated suite, which is the same silent-red shape it exists to catch.
"tests/test_adr_status_governance.py",
"tests/test_adr_index.py",
# Volume honesty (ADR-0264 R9). The Wilson floor assumes independent
# trials; a deterministic pipeline replaying one case N times supplies
# one trial's evidence, not N. This pins the measured exposure in the
# ratified deduction ledger (21 of 25 bands short on distinct evidence)
# in BOTH directions, and forces any new licensed capability to register
# an audit source. It gates a live, flag-ON serving path, so it belongs
# on the pre-push gate. ~1.5s.
"tests/test_volume_honesty.py",
# ADR-0264 R1-R4/R8 — taught curriculum NEGATIVES. Before this, a row
# authored to refute compiled to a premise ASSERTING the atom, and the
# independent oracle ignored polarity too, so gold agreed and wrong=0
# stayed green. Corpus-level epistemology is unfalsifiable by every
# other gate here, so this one belongs on the pre-push gate. ~0.4s.
"tests/test_curriculum_polarity.py",
),
"runtime": (
"tests/test_chat_runtime.py",
"tests/test_achat.py",
"tests/test_runtime_config.py",
"tests/test_session_coherence.py",
),
"cognition": (
"tests/test_intent_proposition_graph.py",
"tests/test_cognitive_turn_pipeline.py",
"tests/test_audit_ledger_r7.py",
"tests/test_articulation_realizer_v2.py",
"tests/test_semantic_realizer_integration.py",
"tests/test_cognitive_eval_harness.py",
"tests/test_deterministic_hash.py",
"tests/test_morphology_irregular.py",
"tests/test_realizer_quantifier_agreement.py",
"tests/test_benchmarks_profiler.py",
"tests/test_compose_relations.py",
"tests/test_replay_vs_llm_benchmark.py",
),
"teaching": (
"tests/test_reviewed_teaching_loop.py",
"tests/test_pipeline_teaching_integration.py",
"tests/test_epistemic_invariants.py",
"tests/test_adr_0172_w2_decomposer.py",
"tests/test_adr_0172_w5_inference_proposal.py",
"tests/test_math_frame_ratification.py",
"tests/test_math_composition_ratification.py",
"tests/test_teaching_coverage_cli.py",
"tests/test_proposal_queue.py",
# The ratification ceremony is the only path that turns a reviewed
# proposal into a routable ratified chain — i.e. the one mechanism that
# can move the curriculum-volume constraint. It landed in #113 in NO
# curated suite, so its 14 tests ran only under `full`.
"tests/test_ratification_ceremony.py",
),
"packs": (
"tests/test_pack_draft_serve_boundary.py",
"tests/test_core_semantic_seed_pack.py",
"tests/test_adr_0127_pack_ratification.py",
"tests/test_frame_registry_load.py",
"tests/test_composition_registry_load.py",
"tests/test_composition_consult_in_injector.py",
"tests/test_consumption_case_0050_hazard_pin.py",
"tests/test_consumption_empty_registry_no_op.py",
"tests/test_consumption_partition.py",
"tests/test_matcher_extension_currency_per_unit.py",
"tests/test_matcher_extension_case_0050_hazard_pin.py",
"tests/test_matcher_extension_end_to_end_admission.py",
"tests/test_me2_cross_sentence_subject.py",
"tests/test_me2_case_0019_admits.py",
"tests/test_me3_additive_composition.py",
"tests/test_me4_subtractive_composition.py",
"tests/test_me5_all_categories_integration.py",
"tests/test_rat1_end_to_end_admission.py",
"tests/test_wave_a_multiplicative_aggregation_injector.py",
),
"algebra": (
"tests/test_stage2_physics_hardening.py",
"tests/test_geometric_convergence_checklist.py",
"tests/test_versor_closure.py",
"tests/test_holonomy.py",
"tests/test_holonomy_resonance.py",
"tests/test_energy.py",
"tests/test_motor.py",
"tests/test_null_cone.py",
"tests/test_vault_recall.py",
"tests/test_vault_recall_vectorised.py",
"tests/test_vault_recall_rust_parity.py",
"tests/test_cga_inner_rust_parity.py",
"tests/test_geometric_product_rust_parity.py",
"tests/test_versor_condition_rust_parity.py",
"tests/test_versor_apply_rust_parity.py",
),
"sensorium": (
"tests/test_sensorium_compiler_delta.py",
"tests/test_audio_compiler.py",
"tests/test_audio_crdt_merge.py",
"tests/test_audio_eval_gates.py",
"tests/test_audio_pack_manifest.py",
"tests/test_audio_sensorium_mount.py",
"tests/test_vision_compiler.py",
"tests/test_event_vision_compiler.py",
"tests/test_vision_crdt_merge.py",
"tests/test_vision_eval_gates.py",
"tests/test_vision_sensorium_mount.py",
"tests/test_sensorimotor_contract.py",
"tests/test_sensorimotor_pack_manifest.py",
"tests/test_observation_frame_contract.py",
"tests/test_observation_frame_harness.py",
"tests/test_environment_falsification.py",
"tests/test_environment_falsification_eval_cli.py",
"tests/test_witness_log_importer.py",
"tests/test_tabletop_lab_protocol.py",
"tests/test_sensorium_eval_cli.py",
"tests/test_efferent_gate.py",
),
"pulse": (
"tests/test_pulse_integration.py",
"tests/test_graph_diffusion.py",
),
"formation": ("tests/formation",),
"proof": ("tests/test_proof_properties.py",),
# ADR-0024 chain suites (Phases 2-6). Each phase has its own contract
# tests so reviewers can run them independently; ``adr-0024`` runs the full
# chain end-to-end.
"refusal": ("tests/test_refusal_contract.py",),
"margin": ("tests/test_margin_admissibility.py",),
"rotor": ("tests/test_rotor_admissibility.py",),
"inner-loop": (
"tests/test_inner_loop_admissibility.py",
"tests/test_inner_loop_phase2.py",
"tests/test_inner_loop_phase3.py",
"tests/test_inner_loop_phase4.py",
),
"phase5": ("tests/test_phase5_corpus.py",),
"phase6": ("tests/test_phase6_demo.py",),
"adr-0024": (
"tests/test_refusal_contract.py",
"tests/test_margin_admissibility.py",
"tests/test_rotor_admissibility.py",
"tests/test_inner_loop_admissibility.py",
"tests/test_inner_loop_phase2.py",
"tests/test_inner_loop_phase3.py",
"tests/test_inner_loop_phase4.py",
"tests/test_phase5_corpus.py",
"tests/test_phase6_demo.py",
),
# ADR-0126 P6 — measurement harness for the GSM8K candidate-graph parser
# exit criterion. ``wrong == 0`` is a hard gate (Obligation #4: refuse
# rather than confabulate).
"math": ("tests/test_adr_0126_train_sample_runner.py",),
"deductive": (
"tests/test_deductive_logic_entail.py",
"tests/test_deduction_serve_lane.py",
"tests/test_deduction_serve_license.py",
"tests/test_categorical_decider.py",
"tests/test_english_argument_reader.py",
"tests/test_member_argument_reader.py",
"tests/test_cond_member_argument_reader.py",
"tests/test_verb_argument_reader.py",
"tests/test_exist_argument_reader.py",
"tests/test_deduction_serve_e2e.py",
"tests/test_curriculum_serve.py",
"tests/test_curriculum_practice.py",
"tests/test_curriculum_polarity.py",
"tests/test_ledger_reseal.py",
"tests/test_ratified_ledger_bridge.py",
"tests/test_vocab_trigger_instrument.py",
"tests/test_grammar_roundtrip.py",
),
"full": ("tests/",),
}
class CommandRunner(Protocol):
def __call__(
self,
*args: str,
check: bool = False,
cwd: Path | None = None,
env: dict[str, str] | None = None,
) -> int: ...
def run_command(
*args: str,
check: bool = False,
cwd: Path | None = None,
env: dict[str, str] | None = None,
) -> int:
completed = subprocess.run(args, check=check, text=True, cwd=cwd, env=env)
return int(completed.returncode)
def pytest_args_for_suite(suite: str, extra_args: Sequence[str]) -> list[str]:
paths = TEST_SUITES[suite]
forwarded = list(extra_args)
if forwarded and forwarded[0] == "--":
forwarded = forwarded[1:]
return [*paths, *forwarded]
def xdist_available() -> bool:
"""Return True iff ``pytest-xdist`` is importable."""
try:
import xdist # noqa: F401
except ImportError:
return False
return True
def maybe_inject_xdist(forwarded: list[str], suite: str | None) -> list[str]:
"""Inject ``-n auto`` for suites large enough to benefit from parallelism."""
if not xdist_available():
return forwarded
# Honour explicit operator override.
if any(a.startswith("-n") or a == "--dist" for a in forwarded):
return forwarded
if suite == "full":
return ["-n", "auto", *forwarded]
return forwarded
def cmd_test(
args: argparse.Namespace,
*,
run: CommandRunner = run_command,
python_executable: str = sys.executable,
) -> int:
"""Run pytest through curated suite aliases or direct passthrough args."""
default_args = ["-q", "--tb=short"]
if args.list_suites:
for name in sorted(TEST_SUITES):
print(name)
return 0
if args.suite:
forwarded = pytest_args_for_suite(args.suite, args.args or default_args)
else:
forwarded = list(args.args or default_args)
if forwarded and forwarded[0] == "--":
forwarded = forwarded[1:]
forwarded = maybe_inject_xdist(forwarded, args.suite)
return run(python_executable, "-m", "pytest", *forwarded)
def cmd_check(
args: argparse.Namespace,
*,
run: CommandRunner = run_command,
python_executable: str = sys.executable,
) -> int:
"""Run ruff over selected project paths."""
targets = args.paths or [
"algebra",
"alignment",
"chat",
"core",
"field",
"generate",
"ingest",
"packs",
"morphology",
"persona",
"sensorium",
"session",
"vault",
"vocab",
"tests",
]
return run(python_executable, "-m", "ruff", "check", *targets)
Reference in a new issue View git blame Copy permalink