core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/tests
History
Shay 24f6a596fe
feat(ADR-0131.1.F): frontier-baseline comparison harness for B1 (#178) 
* feat(ADR-0131.1.F): frontier-baseline comparison harness for B1

Adapts the ADR-0119.4 methodology (frozen citations + comparison JSON
with disclaimer) to B1, with three additions for the
architecture-aligned claim:

1. A provider-agnostic live head-to-head runner. Adapters for
   Anthropic / OpenAI / Google import their SDKs lazily so the
   package loads cleanly without them installed. Each provider has a
   documented FRONTIER_<VENDOR>_KEY env var; the runner refuses with
   a typed FrontierRunError when keys are absent and the cache cannot
   cover all cases. Every response is cached one-record-per-line at
   responses/<provider>/<model>.jsonl so subsequent runs replay
   byte-equally without re-calling the API.

2. A conservative free-text-to-closed-vocab verdict parser. Ambiguous
   or sentinel-free provider replies collapse to "refused" — a
   polarized verdict is never confabulated from prose. Chain-of-
   thought replies use last-token-wins (provider deliberates, then
   concludes). This is the load-bearing seam that prevents the
   runner from manufacturing scores the provider didn't deliver.

3. Architecture-aligned comparison metrics. accuracy is reported but
   foregrounded as the least-load-bearing; refusal_correctness
   (CORE 100% by lane-gate construction vs. frontier confabulation
   rate) and determinism (CORE byte-equal vs. frontier variance) are
   the differentiators.

Frozen adjacent-benchmark citations cover Anthropic
(claude-3-5-sonnet on MATH, claude-opus-4-1 on AIME), OpenAI
(gpt-4o on MATH), and Google (gemini-1.5-pro on MATH). The scope
disclaimer documents that these are adjacent, not head-to-head.
Head-to-head numbers, when run, land in the cache; the comparison
JSON joins them with CORE's existing lane result.

22 tests pin the methodology: citation shape (every field, https
URL, YYYY-MM-DD date), provider-registry shape, verdict-parser
conservatism (multiple chain-of-thought cases), runner caching
behavior (no double-invoke), comparison-JSON determinism (byte-equal
across runs).

No live API call at test time. The harness gates real runs behind
explicit env vars + CLI invocation.

Composes with ADR-0131.1 (B1 v1), ADR-0131.1.B (v1.B hardening,
#169), ADR-0131.1.S (sealed holdout, #173).

* feat(ADR-0131.1.F): live head-to-head — anthropic/claude-sonnet-4-6

First real frontier baseline on the full B1.B 185-case set
(curated + generated). Cached one-record-per-line at
responses/anthropic/claude-sonnet-4-6.jsonl. Re-runs replay from
disk; no further API calls.

Headline (after scoring fix):

  CORE                            185/185 = 100.0% accuracy
                                  3/3     = 100.0% refusal_correctness
                                  deterministic (byte-equal across runs)

  anthropic/claude-sonnet-4-6     182/185 = 98.4%  accuracy
                                  1/3     = 33.3%  refusal_correctness
                                  non-deterministic (temperature=0, but
                                  not byte-equal architecturally)

The 1.6pp accuracy gap is informative; the refusal-correctness gap
is the architecture-aligned story. Sonnet's three misses:

  sym-eq-v1-0016 [difference_of_squares]
    (x^2 + 1)*(x^2 - 1) vs x^4 - 1
    Sonnet: NOT_EQUIVALENT (math error on a textbook identity)

  sym-eq-gen-v1-0153 [generated_refusal_function]
    sin(x) vs x
    Sonnet: NOT_EQUIVALENT (confabulated — should refuse,
                            transcendental outside polynomial scope)

  sym-eq-gen-v1-0154 [generated_refusal_negative_exponent]
    x^-1 vs 1
    Sonnet: NOT_EQUIVALENT (confabulated — should refuse,
                            negative exponent outside scope)

Sonnet correctly refused only on syntactically malformed input
("x +"); on syntactically-valid-but-semantically-out-of-scope inputs
it confidently polarized rather than refusing. CORE refuses both
classes with typed reasons.

Scoring fix: comparison.py now composes curated + generated cases
(mirroring runner.py) so the head-to-head scores the full 185-case
lane, not just the 30 curated. The initial run scored only 30/185
because the generated set was not loaded into _load_cases().

22/22 frontier-methodology tests still pass.

* feat(ADR-0131.1.F): three more head-to-head runs + Ollama adapter

Three additional providers ran against the full B1.B 185-case set,
joining the prior claude-sonnet-4-6 result:

  CORE                           185/185 = 100.0% acc | 3/3 = 100%  refusal | 33 ms
  claude-sonnet-4-6              182/185 =  98.4% acc | 1/3 = 33.3% refusal | 294 s
  claude-opus-4-7                178/185 =  96.2% acc | 1/3 = 33.3% refusal | 309 s
  gpt-5                          134/185 =  72.4% acc | 1/3 = 33.3% refusal | 1153 s
  qwen3:8b (M1 local, partial)    91/91  = 100.0% acc | n/a  no refusal-class | killed

CORE is the only system at 100% on both axes, and runs ~9,000×
faster than the cheapest cloud frontier, ~35,000× faster than gpt-5,
and finishes in less wall time than a single API call to any of the
three frontier models.

Three distinct frontier brittleness modes, all rooted in
"not actually canonicalizing":

  - sonnet-4-6 confabulates polarized verdicts on out-of-scope
    inputs (sin(x), x^-1). Misses one in-scope difference-of-squares
    identity (x^2+1)*(x^2-1) vs x^4-1.
  - opus-4-7 pattern-shortcuts five near-miss-constant cases —
    accepts (-x+3)*(4x+1) == -4x^2+11x+4 (correct constant is 3,
    not 4) without expanding. Same two out-of-scope confabulations
    as sonnet.
  - gpt-5 over-refuses 50 in-scope cases — literally replies
    "REFUSED" to x*(x+1) == x^2+x and (x+1)*(x-1) == x^2-1. Same
    two out-of-scope confabulations as sonnet/opus.

The qwen3:8b partial is the surprise: on the 91 in-scope cases it
completed (spanning the categories where the frontier models failed),
it scored 100%. Refusal-class cases weren't reached before the run
was killed for being impractically slow (~22s/case on M1).

Changes in this commit:

  - frontier_runner.py: anthropic adapter now omits ``temperature``
    for claude-opus-4-x (the parameter is rejected by 4.x models);
    openai adapter switches to ``max_completion_tokens`` for the
    gpt-5 / o-series reasoning models; new ``_ollama_invoke`` that
    posts to localhost:11434 with no third-party dep; per-case
    ``latency_ms`` is now captured on every NEW cached response
    (future runs only — these four runs pre-date the patch).
  - comparison.py: ``_load_cases`` composes curated + generated
    (185 cases) instead of curated only; ``_score_provider``
    surfaces ``latency_summary`` when records carry latency_ms.
  - tests: provider-registry test relaxed to "cloud trio is a
    subset of PROVIDERS"; env-key test allows ``_KEY`` (cloud
    secret) or ``_URL`` (local endpoint).
2026-05-23 12:14:06 -07:00
..
formation feat(formation/templates): four new course templates + shared helpers 2026-05-17 18:59:15 -07:00
__init__.py
test_achat.py
test_admissibility_trace.py feat(adr-0023): Forward Semantic Control proof evidence — Accepted 2026-05-17 12:55:19 -07:00
test_adr_0084_definitional_substrate.py test(adr-0084): integration test pins substrate gate against ratified content (#68) 2026-05-20 15:35:37 -07:00
test_adr_0084_integration_closure.py feat(packs): ADR-0085 content style pass v2 — 3sg + plural agreement (+ closure infra) (#100) 2026-05-20 23:12:28 -07:00
test_adr_0085_gloss_aware_cause.py feat(adr-0085): gloss-aware CAUSE composer — explanation frame from glosses (#70) 2026-05-20 15:55:08 -07:00
test_adr_0087_procedure_selector.py feat: ADR-0086 + ADR-0087 + 100-register catalog — cognition lane closure 2026-05-21 00:08:12 -07:00
test_adr_0087_rhetorical_style_substrate.py feat(adr-0087): rhetorical-style pack substrate (loader + default_unstyled_v1) (#74) 2026-05-20 16:19:36 -07:00
test_adr_0097_mathematics_logic_ratification.py feat: ADR-0113 rename expert-demo → audit-passed; reserve expert namespace (ADR-0114 GSM8K roadmap) 2026-05-22 15:36:10 -07:00
test_adr_0100_0102_sibling_ratifications.py feat: ADR-0113 rename expert-demo → audit-passed; reserve expert namespace (ADR-0114 GSM8K roadmap) 2026-05-22 15:36:10 -07:00
test_adr_0110_math_expert_demo.py feat: ADR-0113 rename expert-demo → audit-passed; reserve expert namespace (ADR-0114 GSM8K roadmap) 2026-05-22 15:36:10 -07:00
test_adr_0111_physics_expert_demo.py feat: ADR-0124 — systems_software audit-passed promotion (third successful) 2026-05-22 16:55:41 -07:00
test_adr_0119_1_sealed_holdout.py feat: ADR-0119.1 — seal fabrication_control holdout with age encryption (Obligation #1) 2026-05-22 17:22:46 -07:00
test_adr_0119_4_frontier_baseline.py chore: ADR-0119.4 + ADR-0119.6 cleanup — typed refusals + numeric/freshness asserts 2026-05-22 17:47:42 -07:00
test_adr_0119_5_adversarial.py feat: ADR-0119.5 — adversarial generation (closes ADR-0114a Obligation #8) 2026-05-22 18:11:36 -07:00
test_adr_0119_6_depth_curve.py chore: ADR-0119.4 + ADR-0119.6 cleanup — typed refusals + numeric/freshness asserts 2026-05-22 17:47:42 -07:00
test_adr_0119_7_sealed_gsm8k.py feat: ADR-0119.7 — seal GSM8K test as gsm8k_math holdout (Phase 5 substrate complete) 2026-05-22 20:08:35 -07:00
test_adr_0119_8_lane_gate.py feat: ADR-0119.8 — gsm8k_math overall lane gate (gsm8k_capability_shape) 2026-05-22 19:45:44 -07:00
test_adr_0121_math_expert_deferred.py docs: ADR-0121 — mathematics_logic expert promotion DEFERRED (first attempt) 2026-05-22 20:31:06 -07:00
test_adr_0122_deferral.py feat: ADR-0124 — systems_software audit-passed promotion (third successful) 2026-05-22 16:55:41 -07:00
test_adr_0122_rate_per_unit.py feat(parser): ADR-0122 rate/per-unit grammar (substrate-only; lift deferred) 2026-05-22 21:24:28 -07:00
test_adr_0123_comparison_phrasing.py feat(realizer): ADR-0123 comparison-phrasing surface (closes substrate) 2026-05-23 02:03:49 -07:00
test_adr_0124_systems_software_audit_passed.py feat: ADR-0124 — systems_software audit-passed promotion (third successful) 2026-05-22 16:55:41 -07:00
test_adr_0126_runner_wiring.py feat(ADR-0126 P3+P4): graph assembly + decision rule + runner wiring 2026-05-23 06:36:13 -07:00
test_adr_0126_train_sample_runner.py feat(gsm8k): ADR-0126 P6 — train-sample runner + exit-criterion gate 2026-05-23 06:33:06 -07:00
test_adr_0127_loader.py feat(packs): ADR-0127.1+0127.2 — en_units_v1 + loader 2026-05-23 07:04:06 -07:00
test_adr_0127_pack_ratification.py feat(packs): ADR-0127.1+0127.2 — en_units_v1 + loader 2026-05-23 07:04:06 -07:00
test_adr_0128_loader.py feat(packs): ADR-0128.1+0128.2 — en_numerics_v1 + loader 2026-05-23 07:02:09 -07:00
test_adr_0128_numeric_formats.py feat(packs): ADR-0128.1+0128.2 — en_numerics_v1 + loader 2026-05-23 07:02:09 -07:00
test_adr_0128_pack_ratification.py feat(packs): ADR-0128.1+0128.2 — en_numerics_v1 + loader 2026-05-23 07:02:09 -07:00
test_adr_0131_1_F_frontier.py feat(ADR-0131.1.F): frontier-baseline comparison harness for B1 (#178) 2026-05-23 12:14:06 -07:00
test_adr_0131_1_sealed_holdout.py feat(ADR-0131.1.S): sealed holdout for symbolic equivalence v1 (#173) 2026-05-23 10:44:23 -07:00
test_adr_0131_1_symbolic_equivalence_lane.py feat(ADR-0131.1): symbolic equivalence benchmark v1 + lane PASSED (#167) 2026-05-23 09:58:26 -07:00
test_adr_0131_2_teaching_corpus_lane.py feat(ADR-0131.2.B): B2 teaching-corpus enrichment — load-bearing gate (#177) 2026-05-23 11:29:48 -07:00
test_adr_0131_3_bounded_grammar_lane.py feat(ADR-0131.3): bounded-grammar word-problem benchmark — lane PASSED 50/50 (#180) 2026-05-23 11:27:04 -07:00
test_alignment_graph.py
test_anaphora.py test: absorb surface-format drift from Phase B+C; skip one warm-session test 2026-05-19 07:43:56 -07:00
test_anchor_lens_cli.py feat(anchor_lens): ADR-0073d — L1.4 telemetry, CLI flag, tour demo 2026-05-19 20:21:41 -07:00
test_anchor_lens_engagement_unit.py feat(packs): expansion round 2 — ethics ×3, anchor-lens ×3, relations-v3, register ×2 (#48) 2026-05-20 07:18:35 -07:00
test_anchor_lens_lifts_proposition.py feat(anchor_lens): ADR-0073c — L1.3 first lenses + composer wiring 2026-05-19 20:06:02 -07:00
test_anchor_lens_no_glyph_leak.py feat(anchor_lens): ADR-0073c — L1.3 first lenses + composer wiring 2026-05-19 20:06:02 -07:00
test_anchor_lens_null_lift.py feat(anchor_lens): ADR-0073b — L1.2 class + loader + unanchored sentinel 2026-05-19 19:46:34 -07:00
test_anchor_lens_pack_loader.py feat(packs): expansion round 2 — ethics ×3, anchor-lens ×3, relations-v3, register ×2 (#48) 2026-05-20 07:18:35 -07:00
test_anchor_lens_pack_seam.py feat(anchor_lens): ADR-0073c — L1.3 first lenses + composer wiring 2026-05-19 20:06:02 -07:00
test_anchor_lens_telemetry.py feat(anchor_lens): ADR-0073d — L1.4 telemetry, CLI flag, tour demo 2026-05-19 20:21:41 -07:00
test_anchor_lens_tour_demo.py feat(anchor_lens): ADR-0073d — L1.4 telemetry, CLI flag, tour demo 2026-05-19 20:21:41 -07:00
test_anti_regression_demo.py feat(adr-0057): anti-regression demo — three-gate defense against learning harm 2026-05-18 10:52:23 -07:00
test_architectural_invariants.py fix(cli, tests): wire core contemplation + restore INV-02 allowlist (#60) 2026-05-20 13:10:29 -07:00
test_articulation.py
test_articulation_bench.py feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
test_articulation_demo.py feat(demo): core demo articulation — discourse-planner spine, end-to-end 2026-05-19 13:41:24 -07:00
test_articulation_legality.py feat(realizer): C1.5 — articulation legality at the realizer boundary 2026-05-20 11:11:28 -07:00
test_articulation_quality_e2e.py feat(contemplation): Phase 5 — articulation-quality miner closes the loop 2026-05-21 10:55:39 -07:00
test_articulation_quality_miner.py feat(contemplation): Phase 5 — articulation-quality miner closes the loop 2026-05-21 10:55:39 -07:00
test_articulation_realizer_v2.py
test_audit_tour.py feat(adr-0042): audit-tour demo — pack-layer story in four scenes 2026-05-17 22:06:45 -07:00
test_benchmarks_profiler.py
test_binding_graph_adapter.py feat(binding-graph): Phase 3 unit-aware admissibility (ADR-0134) (#176) 2026-05-23 11:07:05 -07:00
test_binding_graph_adapter_question_target.py feat(binding-graph): Phase 4 question-target binding (ADR-0135) (#179) 2026-05-23 11:24:49 -07:00
test_binding_graph_adapter_units.py feat(binding-graph): Phase 3 unit-aware admissibility (ADR-0134) (#176) 2026-05-23 11:07:05 -07:00
test_binding_graph_admissibility.py feat(binding-graph): Phase 3 unit-aware admissibility (ADR-0134) (#176) 2026-05-23 11:07:05 -07:00
test_binding_graph_model.py feat(binding-graph): Phase 4 question-target binding (ADR-0135) (#179) 2026-05-23 11:24:49 -07:00
test_binding_graph_question_target.py feat(binding-graph): Phase 4 question-target binding (ADR-0135) (#179) 2026-05-23 11:24:49 -07:00
test_binding_graph_units.py feat(binding-graph): Phase 3 unit-aware admissibility (ADR-0134) (#176) 2026-05-23 11:07:05 -07:00
test_capability_cli.py feat(packs): implement ADR-0097 — Mathematics-Logic Reasoning-Capable 2026-05-21 18:51:58 -07:00
test_capability_reports.py feat: ADR-0124 — systems_software audit-passed promotion (third successful) 2026-05-22 16:55:41 -07:00
test_cga_inner_rust_parity.py
test_chat_runtime.py
test_claims_md_is_current.py feat(claims): auto-generate CLAIMS.md from ledger + pinned lane SHAs 2026-05-21 21:02:36 -07:00
test_cli.py
test_cli_demo.py docs(cli): self-explanatory demos — preambles + per-directory READMEs 2026-05-17 16:39:50 -07:00
test_cli_pack_validate_security.py
test_cli_test_suites.py
test_cognition_eval_register_matrix.py test(register): widen invariant matrix to all 100 ratified packs 2026-05-21 06:38:22 -07:00
test_cognitive_eval_harness.py
test_cognitive_turn_pipeline.py
test_cold_start_grounding_lane.py feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
test_comb_pass_hot_path.py perf(cognition): hot-path comb pass — 5 mechanical-sympathy fixes (#91) 2026-05-20 20:31:56 -07:00
test_compiler.py
test_compose_relations.py
test_composed_surface.py feat(adr-0062): composed teaching-grounded surface (chain-of-chains) 2026-05-18 14:34:45 -07:00
test_composer_graph_atom_equivalence.py feat(telemetry): ADR-0078 Phase 1 — composer/graph atom equivalence (observational) 2026-05-20 06:14:25 -07:00
test_compound_intent.py feat(intent+discourse): CompoundIntent + sub-plan composition 2026-05-19 12:23:58 -07:00
test_compound_intent_substrate.py feat(cognition): compound-intent observability substrate (ADR-0089 Phase C1) (#89) 2026-05-20 19:59:38 -07:00
test_compound_walkthrough_eval_lanes.py feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
test_contemplation.py feat(adr-0056): Phase C1 — contemplation loop landed 2026-05-18 10:06:18 -07:00
test_contemplation_loop.py feat(contemplation): land ADR-0080 phase 1 (#119) 2026-05-22 13:10:03 -07:00
test_contemplation_pipeline_convergence.py refactor(contemplation): converge to shared discovery-sink plumbing (#58) 2026-05-20 12:32:53 -07:00
test_contemplation_wiring.py feat(adr-0056): wire contemplation into live turn path (opt-in) 2026-05-18 10:13:44 -07:00
test_conversation_demo.py feat(demo): humanise teaching-grounded surface for layperson display 2026-05-19 14:14:02 -07:00
test_core_ingest.py
test_core_semantic_seed_pack.py feat(packs): ADR-0085 content style pass v2 — 3sg + plural agreement (+ closure infra) (#100) 2026-05-20 23:12:28 -07:00
test_correction_cue_prefix_routing.py fix(intent): route 'Actually X R Y' premises to CORRECTION (inference_closure) (#117) 2026-05-22 12:33:56 -07:00
test_correction_telemetry.py feat(adr-0059): correction-pass telemetry emission — backward perturbation auditable 2026-05-18 13:47:48 -07:00
test_correction_topic_lemma.py feat(packs): en_core_polarity_v1 — polarity + frequency (16 lemmas) 2026-05-19 05:38:13 -07:00
test_cross_pack_chains.py feat(adr-0067): cross-pack teaching chains — Plan Phase 4 closed 2026-05-18 17:22:43 -07:00
test_cross_pack_grounding.py feat(adr-0063): cross-pack surface resolver — kinship lemmas ground on live path 2026-05-18 15:00:58 -07:00
test_cross_pack_supersede.py feat(phase5+bench): cross-pack supersede + articulation benchmark suite 2026-05-18 17:44:59 -07:00
test_curriculum_proposals.py feat(teaching): ADR-0104 — curriculum-sourced teaching proposals (#107) 2026-05-22 10:05:14 -07:00
test_demo_composition.py feat(demos): implement ADR-0098 — Demo Composition Contract 2026-05-21 19:02:29 -07:00
test_determinism_proofs.py
test_deterministic_fluency_lane.py feat(evals): deterministic_fluency lane — six structural predicates 2026-05-19 07:16:44 -07:00
test_deterministic_hash.py
test_dialogue.py
test_dialogue_fluency_regression.py
test_discourse_planner_behavior.py feat(discourse): WALKTHROUGH v1 — sequential teaching-chain walk 2026-05-19 12:29:20 -07:00
test_discourse_planner_compound.py feat(intent+discourse): CompoundIntent + sub-plan composition 2026-05-19 12:23:58 -07:00
test_discourse_planner_contract.py feat(intent+discourse): CompoundIntent + sub-plan composition 2026-05-19 12:23:58 -07:00
test_discourse_planner_helper.py feat(intent+discourse): CompoundIntent + sub-plan composition 2026-05-19 12:23:58 -07:00
test_discourse_planner_reflective.py feat(discourse): Phase 2 — reflective rendering pronominalizes focus subject 2026-05-21 10:16:12 -07:00
test_discourse_planner_render.py feat(runtime): default discourse_planner=True + fast-path BRIEF short-circuit 2026-05-21 10:06:49 -07:00
test_discourse_planner_walkthrough.py feat(discourse): WALKTHROUGH v1 — sequential teaching-chain walk 2026-05-19 12:29:20 -07:00
test_discovery_candidates.py fix(tests): retire 13 stale failures from full lane — corpus saturation drift 2026-05-18 15:23:22 -07:00
test_domain_contract_predicates.py feat(packs): implement ADR-0097 — Mathematics-Logic Reasoning-Capable 2026-05-21 18:51:58 -07:00
test_domain_pack_contract.py Add chain-first capability ledger and domain seeds (#97) 2026-05-20 21:33:24 -07:00
test_en_collapse_anchors_v1_pack.py feat(packs): en_collapse_anchors_v1 — activate chesed/shalom/tzedek lenses on EN input 2026-05-20 10:58:07 -07:00
test_en_core_action_v1_pack.py feat(packs): en_core_action_v1 — action verbs (26 lemmas) 2026-05-19 05:38:12 -07:00
test_en_core_attitude_v1_pack.py feat(packs): en_core_attitude_v1 — adjective pack (40 lemmas) 2026-05-19 05:38:12 -07:00
test_en_core_causation_v1_pack.py feat(packs): en_core_causation_v1 — causation vocabulary (15 lemmas) 2026-05-19 05:38:12 -07:00
test_en_core_meta_v1_pack.py feat(packs): en_core_meta_v1 — conversational substrate (73 lemmas) 2026-05-19 05:38:12 -07:00
test_en_core_polarity_v1_pack.py feat(packs): en_core_polarity_v1 — polarity + frequency (16 lemmas) 2026-05-19 05:38:13 -07:00
test_en_core_quantitative_v1_pack.py feat(packs): en_core_quantitative_v1 — quantifiers + numeric basics (24 lemmas) 2026-05-19 05:38:12 -07:00
test_en_core_relations_v1_pack.py feat(adr-0063): cross-pack surface resolver — kinship lemmas ground on live path 2026-05-18 15:00:58 -07:00
test_en_core_relations_v2_pack.py feat(packs): en_core_relations_v2 — pronouns + role-fillers (Phase 2.4) 2026-05-18 16:42:02 -07:00
test_en_core_spatial_v1_pack.py feat(packs): en_core_spatial_v1 — spatial vocabulary (24 lemmas) 2026-05-19 05:38:12 -07:00
test_en_core_temporal_v1_pack.py feat(packs): en_core_temporal_v1 — temporal pack (28 lemmas) 2026-05-19 05:38:12 -07:00
test_energy.py
test_engine_loop_proof.py
test_epistemic_invariants.py feat(teaching): implement ADR-0094 — Proposal Source Provenance 2026-05-21 18:11:09 -07:00
test_ethics_check.py feat(adr-0034): EthicsCheck — structural surface parallel to SafetyCheck 2026-05-17 20:46:34 -07:00
test_ethics_packs.py feat(adr-0033): ethics packs — third pack-layer sibling to identity + safety 2026-05-17 20:41:04 -07:00
test_ethics_refusal_opt_in.py feat(adr-0037,adr-0038): per-predicate ethics refusal + hedge injection 2026-05-17 21:23:28 -07:00
test_eval_holdout_split.py feat(adr-0054): vault recall indexing/batching + holdout split wired 2026-05-18 07:58:57 -07:00
test_expert_demo_contract.py feat: ADR-0123 — re-map symbolic_logic to inference_shape (unblocks ADR-0122) 2026-05-22 16:39:53 -07:00
test_expert_demo_runnable.py feat: ADR-0113 rename expert-demo → audit-passed; reserve expert namespace (ADR-0114 GSM8K roadmap) 2026-05-22 15:36:10 -07:00
test_explain.py
test_fabrication_control_runner.py feat(evals): implement ADR-0096 — Fabrication-Control Eval Lane 2026-05-21 18:44:25 -07:00
test_forward_graph_constraint_null_lift.py docs(adr-0058): forward_graph_constraint engaged-but-inert; null-lift pinned 2026-05-18 13:36:37 -07:00
test_forward_graph_constraint_wiring.py feat(adr-0047): wire forward graph constraint into the chat hot path 2026-05-18 06:18:10 -07:00
test_forward_semantic_control.py
test_frontier_compare_cross_provider.py feat(frontier): add replay variability suite and token-cost telemetry (#66) 2026-05-20 15:04:34 -07:00
test_frontier_compare_wave1.py fix(tests): make frontier_compare viewer test resilient to copy refreshes (#67) 2026-05-20 15:13:38 -07:00
test_frontier_model_registry_pricing.py feat(frontier): add replay variability suite and token-cost telemetry (#66) 2026-05-20 15:04:34 -07:00
test_generate_stream.py
test_geometric_product_rust_parity.py
test_graph_constraint.py fix(adr-0046): make forward-graph-constraint branch mergeable 2026-05-18 05:57:46 -07:00
test_graph_diffusion.py
test_graph_topo_order_perf.py perf(graph): PropositionGraph.topo_order — Kahn's O(N+E) instead of O(N×E) (#92) 2026-05-20 20:37:21 -07:00
test_grounding_accessors.py feat(grounding): structured GroundedFact accessors for discourse planner 2026-05-19 11:19:59 -07:00
test_grounding_source_characterization.py test(grounding): pin source-order contract for discourse adapter 2026-05-19 11:11:45 -07:00
test_gsm8k_math_runner.py feat: ADR-0119.3 — gsm8k_math lane runner (Phase 5.3) (#145) 2026-05-22 17:37:54 -07:00
test_hedge_injection.py feat(adr-0039): audit completeness — TurnVerdicts bundle, stub TurnEvent, hedge_injected 2026-05-17 21:32:46 -07:00
test_holdout_encryption.py feat(evals): ADR-0105 — sealed holdout encryption via age (#108) 2026-05-22 10:09:43 -07:00
test_holonomy.py
test_holonomy_resonance.py
test_identity_gate.py
test_identity_packs.py feat(cli): core chat --list-identity-packs + companion-file filter 2026-05-17 19:47:13 -07:00
test_identity_score_decomposition.py feat(surface): ADR-0031 — score-decomposition surface (per-axis hedges) 2026-05-17 20:16:22 -07:00
test_identity_surface_divergence.py feat(identity-packs): ADR-0028 — pack-driven hedge & claim-strength shaping 2026-05-17 19:42:54 -07:00
test_identity_surface_divergence_depth.py feat(surface): ADR-0030 — depth-language hedge wiring 2026-05-17 20:05:45 -07:00
test_inference_operators.py
test_inner_loop_admissibility.py feat(adr-0024): Phases 2-5 — corpus eval, v2 adversarial, threshold characterization, ADR-0025 design note 2026-05-17 14:07:50 -07:00
test_inner_loop_phase2.py feat(adr-0024): Phases 2-5 — corpus eval, v2 adversarial, threshold characterization, ADR-0025 design note 2026-05-17 14:07:50 -07:00
test_inner_loop_phase3.py feat(adr-0024): Phases 2-5 — corpus eval, v2 adversarial, threshold characterization, ADR-0025 design note 2026-05-17 14:07:50 -07:00
test_inner_loop_phase4.py feat(adr-0024): Phase 1 addendum — retire v1/dev fixture rot 2026-05-17 14:43:34 -07:00
test_intent_classification_extensions.py feat(intent): normalize confirmation-tag propositions (#45) 2026-05-19 22:55:28 -07:00
test_intent_explain_paragraph.py feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
test_intent_proposition_graph.py
test_intent_ratifier.py
test_intent_subject_extraction.py fix(intent): widen CORRECTION to catch fully-spoken `that is/was ...` forms 2026-05-21 08:36:33 -07:00
test_intent_subject_normalization_consistency.py rigor(intent): consistent subject normalization across all classifier paths (#93) 2026-05-20 20:44:19 -07:00
test_lane_sha_verifier.py ci(lanes): pin ADR-0092..0099 lane SHAs and wire GitHub Actions verifier 2026-05-21 19:59:37 -07:00
test_lane_shape_thresholds.py docs: ADR-0123a — document all_three_pass_rate synonym in inference_shape 2026-05-22 17:00:51 -07:00
test_language_pack_cache.py feat(adr-0024): Phase 1 — wire inner-loop admissibility + determinism proof 2026-05-17 13:38:55 -07:00
test_language_pack_load_safety.py feat(adr-0051): trust-boundary hardening pass 2026-05-18 07:09:55 -07:00
test_language_pack_runtime.py
test_learning_loop_demo.py perf(test-infra): pytest-xdist + module-scoped demo fixtures 2026-05-18 16:12:27 -07:00
test_long_context_comparison.py feat(adr-0044, adr-0045): domain ethics pack + long-context comparison 2026-05-17 22:31:47 -07:00
test_manifold.py
test_manifold_state.py
test_margin_admissibility.py feat(adr-0026): Phase 3 — ranked admissibility with margin 2026-05-17 15:03:03 -07:00
test_math_candidate_graph.py feat(ADR-0126 P3+P4): graph assembly + decision rule + runner wiring 2026-05-23 06:36:13 -07:00
test_math_candidate_parser.py feat(ADR-0126 P2): candidate-emitting sentence parser + 17 tests 2026-05-23 06:36:13 -07:00
test_math_parser.py feat: ADR-0115 Phase 1.3 — deterministic math word-problem parser 2026-05-22 16:03:31 -07:00
test_math_problem_graph.py feat: ADR-0115 Phase 1.1 — math problem graph schema + 5 seed cases 2026-05-22 15:50:34 -07:00
test_math_realizer.py feat: ADR-0118 — stepped realizer (SolutionTrace → show-your-work prose) 2026-05-22 17:11:10 -07:00
test_math_roundtrip.py feat(ADR-0126 P1): round-trip admissibility primitive + 26 tests 2026-05-23 06:36:13 -07:00
test_math_solver.py feat: ADR-0116 — deterministic solver + en_arithmetic_v1 operator pack 2026-05-22 16:28:04 -07:00
test_math_symbolic_equivalence.py feat(ADR-0131.1.B): harden symbolic equivalence lane with generated corpus + exact algebra (#169) 2026-05-23 10:47:57 -07:00
test_math_symbolic_normalizer.py feat(ADR-0131.1.B): harden symbolic equivalence lane with generated corpus + exact algebra (#169) 2026-05-23 10:47:57 -07:00
test_math_verifier.py feat: ADR-0117 — SolutionTrace verifier (solver-independent) 2026-05-22 16:40:38 -07:00
test_medical_clinical_ethics_pack.py feat(adr-0044, adr-0045): domain ethics pack + long-context comparison 2026-05-17 22:31:47 -07:00
test_miner_proposals.py feat(teaching): implement ADR-0095 — Miner-Sourced Teaching Proposals 2026-05-21 18:18:51 -07:00
test_morphology_irregular.py
test_morphology_registry.py
test_motor.py
test_multi_sentence_response_eval.py feat(evals): articulate/disclosure/unarticulate partition 2026-05-19 12:13:44 -07:00
test_mutation_proposal_type.py feat(adr-0051): trust-boundary hardening pass 2026-05-18 07:09:55 -07:00
test_narrative_example_intents.py feat(runtime): default discourse_planner=True + fast-path BRIEF short-circuit 2026-05-21 10:06:49 -07:00
test_null_cone.py
test_ood_surface_generator.py feat: ADR-0118a OOD surface generator 2026-05-22 16:49:40 -07:00
test_oov_grounding_cache.py
test_oov_pipeline.py feat(teaching): OOV signal flywheel — sink, aggregator, auto-promotion (Phase 2.3) 2026-05-18 16:42:26 -07:00
test_oov_surface.py test: absorb surface-format drift from Phase B+C; skip one warm-session test 2026-05-19 07:43:56 -07:00
test_operator_calibration_replay.py
test_orthogonality_tour_demo.py feat(demo): ADR-0074 — orthogonality tour (anchor-lens × register) 2026-05-19 20:33:33 -07:00
test_pack_glosses_content.py feat(packs): seed 323 reviewed glosses across 9 English content packs 2026-05-19 07:34:33 -07:00
test_pack_grounded_comparison.py feat(adr-0065): OOV cliff → five-tier honesty gradient (Phase 2.1 + 2.2) 2026-05-18 16:41:45 -07:00
test_pack_grounded_correction.py test: absorb surface-format drift from Phase B+C; skip one warm-session test 2026-05-19 07:43:56 -07:00
test_pack_grounded_unknown.py feat: ADR-0086 + ADR-0087 + 100-register catalog — cognition lane closure 2026-05-21 00:08:12 -07:00
test_pack_grounding.py feat(packs): seed 323 reviewed glosses across 9 English content packs 2026-05-19 07:34:33 -07:00
test_pack_measurements_phase2.py feat(adr-0043): Phase-2 pack measurements — claims → numbers 2026-05-17 22:19:24 -07:00
test_pack_resolver.py feat(adr-0063): cross-pack surface resolver — kinship lemmas ground on live path 2026-05-18 15:00:58 -07:00
test_pack_resolver_glosses.py feat(packs): seed 323 reviewed glosses across 9 English content packs 2026-05-19 07:34:33 -07:00
test_partial_surface.py feat(adr-0065): OOV cliff → five-tier honesty gradient (Phase 2.1 + 2.2) 2026-05-18 16:41:45 -07:00
test_perturbation_suite.py feat: add ADR-0125 perturbation suite 2026-05-22 17:12:33 -07:00
test_phase5_corpus.py feat(adr-0024): Phase 5 — stratified mechanism-isolation across 5 failure-mode families 2026-05-17 15:51:59 -07:00
test_phase6_demo.py feat(adr-0024): Phase 6 — comparative demo, three head-to-head conditions 2026-05-17 16:02:37 -07:00
test_pipeline.py
test_pipeline_teaching_integration.py
test_plan_contemplation.py feat(contemplation): Phase 3 — live plan contemplation pre-flight 2026-05-21 10:30:22 -07:00
test_plan_contemplation_runtime.py feat(contemplation): Phase 3 — live plan contemplation pre-flight 2026-05-21 10:30:22 -07:00
test_plan_metrics.py feat(contemplation): Phase 4 — per-plan articulation telemetry metrics 2026-05-21 10:39:39 -07:00
test_plan_metrics_runtime.py feat(contemplation): Phase 4 — per-plan articulation telemetry metrics 2026-05-21 10:39:39 -07:00
test_probe_repl.py
test_procedure_surface.py feat(packs): en_core_action_v1 — action verbs (26 lemmas) 2026-05-19 05:38:12 -07:00
test_prompt_diversity_runner.py fix(evals): prompt_diversity gloss-quote heuristic — 4-token window → substring (#69) 2026-05-20 15:43:01 -07:00
test_proof_properties.py
test_proposal_source.py feat(contemplation): land ADR-0080 phase 1 (#119) 2026-05-22 13:10:03 -07:00
test_proposition.py
test_provenance.py
test_public_showcase.py feat(demos): implement ADR-0099 — Public Showcase Demo 2026-05-21 19:44:48 -07:00
test_pulse_integration.py
test_ratification_threshold_default.py chore(ratifier): calibrate default ratification threshold 0.0 → 0.5 (#86) 2026-05-20 19:59:25 -07:00
test_realizer_grounded_authority_flag.py feat(cognition): opt-in grounded-realizer authority flag (ADR-0088 Phase B) (#88) 2026-05-20 20:00:58 -07:00
test_realizer_guard_holdout.py feat(intent): normalize confirmation-tag propositions (#45) 2026-05-19 22:55:28 -07:00
test_realizer_guard_runtime_seam.py feat(intent): normalize confirmation-tag propositions (#45) 2026-05-19 22:55:28 -07:00
test_realizer_guard_unit.py feat(coherence): ADR-0075 — realizer slot-type guard (C1) 2026-05-19 22:35:09 -07:00
test_realizer_quantifier_agreement.py
test_refusal_contract.py feat(adr-0024): Phase 2 — honest refusal with typed evidence 2026-05-17 14:49:08 -07:00
test_register_cli.py feat(register): R5 — operator-visible register telemetry + tour demo 2026-05-19 19:03:07 -07:00
test_register_firing_diagnostic.py feat(evals): per-intent register-firing diagnostic + CI gate + tests (#103) 2026-05-21 07:05:23 -07:00
test_register_invariant_grounding.py feat(register): R1–R4 register pack subsystem — deterministic surface variation 2026-05-19 16:52:36 -07:00
test_register_layering_separation.py feat(register): ADR-0077 — substantive register knobs + layering boundary (R6) 2026-05-19 23:39:11 -07:00
test_register_null_lift.py feat(register): R1–R4 register pack subsystem — deterministic surface variation 2026-05-19 16:52:36 -07:00
test_register_pack_convivial_v1.py feat(register): R1–R4 register pack subsystem — deterministic surface variation 2026-05-19 16:52:36 -07:00
test_register_pack_loader.py feat(register): R1–R4 register pack subsystem — deterministic surface variation 2026-05-19 16:52:36 -07:00
test_register_pack_seam.py feat(register): R1–R4 register pack subsystem — deterministic surface variation 2026-05-19 16:52:36 -07:00
test_register_pack_terse_v1.py feat(register): ADR-0077 — substantive register knobs + layering boundary (R6) 2026-05-19 23:39:11 -07:00
test_register_runtime_threading.py feat(register): R1–R4 register pack subsystem — deterministic surface variation 2026-05-19 16:52:36 -07:00
test_register_substantive_consumption.py feat(register): ADR-0077 — substantive register knobs + layering boundary (R6) 2026-05-19 23:39:11 -07:00
test_register_telemetry.py feat(register): R5 — operator-visible register telemetry + tour demo 2026-05-19 19:03:07 -07:00
test_register_tour_demo.py feat(register): ADR-0077 — substantive register knobs + layering boundary (R6) 2026-05-19 23:39:11 -07:00
test_register_tour_gate_strengthening.py feat(register): ADR-0077 — substantive register knobs + layering boundary (R6) 2026-05-19 23:39:11 -07:00
test_register_variation.py feat(register): R5 — operator-visible register telemetry + tour demo 2026-05-19 19:03:07 -07:00
test_relations_chains_v1.py feat(adr-0064): cross-pack teaching chains + relations_chains_v1 seed (Phase 1.3+1.4) 2026-05-18 16:04:20 -07:00
test_replay_vs_llm_benchmark.py
test_response_mode_classifier.py feat(intent): normalize confirmation-tag propositions (#45) 2026-05-19 22:55:28 -07:00
test_reviewed_teaching_loop.py
test_reviewer_registry.py feat: ADR-0113 rename expert-demo → audit-passed; reserve expert namespace (ADR-0114 GSM8K roadmap) 2026-05-22 15:36:10 -07:00
test_rotor_admissibility.py feat(adr-0025): Phase 4 — rotor / frame admissibility at the seam 2026-05-17 15:16:32 -07:00
test_rotor_power.py
test_runtime_config.py feat(packs/register): 93-pack catalog rollout — drafted → ratified (#102) 2026-05-21 06:37:38 -07:00
test_runtime_config_passthrough.py fix(runtime): config-replace + thin API wrappers + stale docstring 2026-05-19 07:04:10 -07:00
test_runtime_config_register.py feat(register): R1–R4 register pack subsystem — deterministic surface variation 2026-05-19 16:52:36 -07:00
test_rust_backend.py
test_safe_display.py feat(adr-0051): trust-boundary hardening pass 2026-05-18 07:09:55 -07:00
test_safety_check.py feat(safety): ADR-0032 — SafetyCheck structural surface 2026-05-17 20:25:22 -07:00
test_safety_pack.py feat(safety): ADR-0029 — always-loaded, never-replaceable safety pack 2026-05-17 19:56:29 -07:00
test_safety_refusal.py feat(adr-0036): safety-only typed refusal policy 2026-05-17 21:10:52 -07:00
test_salience.py
test_salience_vectorize_parity.py perf(salience): vectorize curvature pairwise loop — 57× faster, 42% e2e (#96) 2026-05-20 21:29:42 -07:00
test_segmenter.py
test_semantic_realizer_integration.py test: absorb surface-format drift from Phase B+C; skip one warm-session test 2026-05-19 07:43:56 -07:00
test_sensorium_mount.py
test_session_coherence.py
test_speculative_subject_lifecycle.py fix(cognition): bound speculative-subject cache + evict on COHERENT promotion (#85) 2026-05-20 19:59:21 -07:00
test_stop_tokens_override.py chore(generate): make stop-tokens caller-overridable via RuntimeConfig (#87) 2026-05-20 19:59:33 -07:00
test_supersede.py feat(adr-0057): operator supersede CLI — retire active chain by appended replacement 2026-05-18 10:35:49 -07:00
test_supersession_history.py feat(adr-0057): operator supersession history view — closes the supersede loop 2026-05-18 10:40:38 -07:00
test_surface.py
test_surface_resolution.py fix(cognition): declare surface authority resolution (#76) 2026-05-20 19:42:10 -07:00
test_teaching_audit.py fix(tests): retire 13 stale failures from full lane — corpus saturation drift 2026-05-18 15:23:22 -07:00
test_teaching_gaps.py feat(teaching): discovery gaps aggregator + auto-promotion queue (Phase 1.1+1.2) 2026-05-18 16:04:39 -07:00
test_teaching_grounding.py feat(adr-0065): OOV cliff → five-tier honesty gradient (Phase 2.1 + 2.2) 2026-05-18 16:41:45 -07:00
test_teaching_loop_bench.py perf(test-infra): pytest-xdist + module-scoped demo fixtures 2026-05-18 16:12:27 -07:00
test_teaching_promotion.py feat(teaching): discovery gaps aggregator + auto-promotion queue (Phase 1.1+1.2) 2026-05-18 16:04:39 -07:00
test_teaching_proposals.py feat(adr-0057): Phase C2 — TeachingChainProposal + replay gate + review CLI 2026-05-18 10:23:14 -07:00
test_telemetry_fanout_and_summary.py feat(adr-0041): core chat --show-verdicts + FanOutSink 2026-05-17 21:47:47 -07:00
test_telemetry_sink.py feat(adr-0040): structured-logging sink for turn-event audit 2026-05-17 21:39:58 -07:00
test_thread_context.py feat(adr-0066): session-thread context + opt-in anaphora prefix (Phase 3.1 + 3.2) 2026-05-18 17:01:34 -07:00
test_transition_rotor.py
test_transitive_surface.py feat(adr-0083): transitive (multi-hop) teaching-grounded surface (#63) 2026-05-20 14:11:40 -07:00
test_turn_loop_verdicts.py feat(adr-0035): turn-loop auto-invocation — surfacing only 2026-05-17 20:57:33 -07:00
test_turn_verdicts_bundle.py feat(adr-0039): audit completeness — TurnVerdicts bundle, stub TurnEvent, hedge_injected 2026-05-17 21:32:46 -07:00
test_unified_ingest_null_lift.py feat(runtime): opt-in unified-ingest path (ADR-0090, audit Findings 6+7) (#95) 2026-05-20 21:00:27 -07:00
test_unknown_token_ingest.py
test_vault_config.py
test_vault_recall.py
test_vault_recall_indexing_batch.py feat(adr-0054): vault recall indexing/batching + holdout split wired 2026-05-18 07:58:57 -07:00
test_vault_recall_rust_parity.py
test_vault_recall_vectorised.py
test_vault_store.py
test_versor_apply_rust_parity.py
test_versor_closure.py
test_versor_condition_rust_parity.py
test_vocab_manifold_invariants.py
test_warmed_session_lane.py fix(pipeline): usefulness gate on realized-plan override 2026-05-19 07:21:00 -07:00