core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/docs/decisions/ADR-0131.2-teaching-corpus-eval.md

2.8 KiB
Raw Blame History

ADR-0131.2 — Benchmark 2: CORE-native teaching-corpus eval (lane gate)

Status: Accepted Date: 2026-05-23 Author: CORE agents + reviewers Depends on: ADR-0131 (composite math promotion gate), ADR-0064 (cross-pack teaching)

Context

ADR-0131 establishes a composite expert-promotion gate for the mathematics_logic domain, superseding the GSM8K-coverage gate with three architecture-aligned benchmarks.

This decision document addresses the design and implementation of Benchmark 2 (teaching-corpus math eval). The purpose of this benchmark is to ensure that the CORE-native teaching/replay loop is internally consistent and capable of carrying mathematical knowledge end-to-end (propose → ratify → replay-equivalent) without regressing existing cognition capabilities.

ADR-0064 serves as structural precedent by introducing the multi-corpus, pack-residency validation rules, allowing the system to isolate different domains into their own vocabulary packs (e.g., binding math chains specifically to en_mathematics_logic_v1 or en_arithmetic_v1).

Decision

We establish the Benchmark 2 eval lane substrate:

  1. New Math Teaching Corpus: A hand-curated dataset teaching/math_corpora/math_teaching_v1.jsonl containing 30 mathematical logic chains in propose-shape (DiscoveryCandidate). The chains are ratification-ready, meaning they contain valid connectives, intents, polarities, and evidence pointers.
  2. Evaluative Bounded Domain: The corpus lemmas are strictly restricted to the en_mathematics_logic_v1 language pack. This ensures that the math domain is orthogonal to the core cognition domain (en_core_cognition_v1).
  3. Transient Replay Loop Validation: We implement an evaluation runner evals/math_teaching_corpus/v1/runner.py that dynamically registers the math corpus spec, runs candidate proposals through propose_from_candidate, checks for regression against the public cognition split, and ratifies them via accept_proposal into a transient math corpus.
  4. Deterministic Report Writeback: The runner emits a deterministic report.json reporting the status of each case.
  5. Strict Pass Criterion:
    • correct_rate == 1.0 (all cases must be proposed, verified as replay-equivalent, and ratified successfully).
    • wrong == 0 (no case may fail the replay gate or raise errors).

Consequences

  • The teaching/replay loop is validated as a robust vehicle for mathematical logic relations.
  • Pack-residency rules (ADR-0064) prevent cross-domain leakage; math logic terms are validated to belong strictly to en_mathematics_logic_v1 and are ignored by the cognition split.
  • Benchmark 2 provides the internal consistency gate for mathematical logic, clearing the way for Benchmark 3 (bounded-grammar word-problem set) and the composite gate integration.