e831ed2615
Phase 2a r2/r3/r4 of the redefined plan: the general comprehension reader now
reads THREE independent-gold reasoning domains end-to-end (prose -> MeaningGraph
-> projection -> independent oracle -> answer vs gold), all wrong=0, and all
three are wired into the capability index.
reader.py — new domain-agnostic templates (function words + order; parse-or-refuse):
- categorical E/I/O: "no Xs are Ys"->disjoint, "some Xs are Ys"->intersects,
"some Xs are not Ys"->some_not (A "all Xs are Ys"->subset already existed)
- "therefore <categorical>" -> conclusion QUERY (same neutral predicate vocab)
- comparative facts: "<X> [is] <comp> [than] <Y>" -> less(...), closed
less/greater comparator lexicon, elided-copula support
- sort query ("sort ascending|descending", "... order from <low> to <high>")
and compare query ("compare <X> with <Y>")
- clause-splitting on commas / leading and|or for multi-clause sentences
projectors.py — to_syllogism (premises + validity conclusion, finite-model size 3)
and to_total_ordering (less-facts + sort/compare). Both return None when nothing
is honestly askable of their oracle (caller treats as refusal).
capability_index — wire 3 comprehension lanes into ADAPTERS; re-freeze baseline
breadth 3->6, capability_score 0.919641->0.814356 (geomean falls BY DESIGN as
honest partial-coverage domains join; wrong_total stays 0). digest 0a98b9b4...
Scores: set_membership 8/8, syllogism 6/8, total_ordering 4/8 — all wrong=0.
Multi-word NP handling is DEFERRED on purpose, not missed: the gold lanes
canonicalize multi-word NPs three contradictory ways ("North station"->"north",
"Level one"->"level_one", "metal objects"->"metal"), so no single general rule is
wrong=0-safe. The reader refuses multi-word NPs until the gold lanes carry a
canonicalization contract. Every refusal is a genuine harder phenomenon
(multi-word NP, adjectival predicate, trailing tokens) — never a readable case
silently dropped.
Tests: reader templates, projector unit tests, syllogism/total_ordering
end-to-end wrong=0 with pinned counts, capability breadth 3->6. 138 targeted +
87 smoke green. Lane SHAs 8/9 (sole miss = public_demo env wall-clock flake).
95 lines
3.2 KiB
Python
95 lines
3.2 KiB
Python
"""Per-lane adapters — normalize each independent-gold lane to a DomainResult.
|
|
|
|
These are thin COUNT extractors, not capability logic: each calls a lane's own
|
|
self-loading runner and reads its correct/wrong/refused counts. A lane that fails
|
|
to run is recorded as ``not_covered`` (no silent drop), never faked.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from dataclasses import dataclass
|
|
|
|
from evals.capability_index.index import DomainResult
|
|
|
|
|
|
def _counts(report: dict) -> tuple[int, int, int]:
|
|
c = report.get("counts", report)
|
|
return int(c["correct"]), int(c["wrong"]), int(c["refused"])
|
|
|
|
|
|
def deductive_logic_result() -> DomainResult:
|
|
from evals.deductive_logic.runner import build_combined_report
|
|
|
|
agg = build_combined_report()["aggregate"] # {n, correct, wrong, refused}
|
|
return DomainResult(
|
|
"deductive_logic", int(agg["correct"]), int(agg["wrong"]), int(agg["refused"])
|
|
)
|
|
|
|
|
|
def relational_metric_result() -> DomainResult:
|
|
from evals.relational_metric.runner import run
|
|
|
|
r = run()
|
|
return DomainResult(
|
|
"relational_metric", int(r["correct"]), int(r["wrong"]), int(r["refused"])
|
|
)
|
|
|
|
|
|
def dimensional_result() -> DomainResult:
|
|
from evals.dimensional.runner import _ROOT, _load, build_report
|
|
|
|
correct, wrong, refused = _counts(build_report(_load(_ROOT / "v1" / "cases.jsonl")))
|
|
return DomainResult("dimensional", correct, wrong, refused)
|
|
|
|
|
|
def comprehension_set_membership_result() -> DomainResult:
|
|
from evals.comprehension.set_membership_runner import run
|
|
|
|
c, w, r = _counts(run())
|
|
return DomainResult("comprehension_set_membership", c, w, r)
|
|
|
|
|
|
def comprehension_syllogism_result() -> DomainResult:
|
|
from evals.comprehension.syllogism_runner import run
|
|
|
|
c, w, r = _counts(run())
|
|
return DomainResult("comprehension_syllogism", c, w, r)
|
|
|
|
|
|
def comprehension_total_ordering_result() -> DomainResult:
|
|
from evals.comprehension.total_ordering_runner import run
|
|
|
|
c, w, r = _counts(run())
|
|
return DomainResult("comprehension_total_ordering", c, w, r)
|
|
|
|
|
|
#: The reasoning domains currently composed into the index (self-loading lanes).
|
|
#: The three ``comprehension_*`` lanes score the GENERAL comprehension reader
|
|
#: (prose -> MeaningGraph -> projection -> independent oracle), so the index now
|
|
#: measures comprehension breadth, not just structured-input reasoning.
|
|
ADAPTERS = (
|
|
deductive_logic_result,
|
|
relational_metric_result,
|
|
dimensional_result,
|
|
comprehension_set_membership_result,
|
|
comprehension_syllogism_result,
|
|
comprehension_total_ordering_result,
|
|
)
|
|
|
|
|
|
@dataclass(frozen=True, slots=True)
|
|
class Collection:
|
|
results: tuple[DomainResult, ...]
|
|
not_covered: tuple[tuple[str, str], ...] # (adapter_name, error) — no silent drop
|
|
|
|
|
|
def collect_domain_results() -> Collection:
|
|
"""Run every adapter; surface any that fail rather than dropping them."""
|
|
results: list[DomainResult] = []
|
|
not_covered: list[tuple[str, str]] = []
|
|
for adapter in ADAPTERS:
|
|
try:
|
|
results.append(adapter())
|
|
except Exception as exc: # noqa: BLE001 — surfacing is the contract
|
|
not_covered.append((adapter.__name__, repr(exc)))
|
|
return Collection(results=tuple(results), not_covered=tuple(not_covered))
|