6f70e834ff
* feat(determine): one-hop sound relational entailment — inverse/converse + symmetric Mastery-v2 Step 3 lead capability (core). DETERMINE could perceive relational structure (16 predicates, breadth-complete) but could not derive the simplest entailed relational facts — it read only the stored direction. This adds two SOUND one-hop rules that read a stored edge in its other lawful direction: INVERSE/converse greater_than(a,b) <= told less_than(b,a) SYMMETRIC sibling_of(b,a) <= told sibling_of(a,b) Strictly scoped (per review): OPEN-WORLD (asserts only True, never False — the answer=False path stays unbuilt, INV-30's firewall holds), ONE hop (NO transitive chaining), DECLARED rules only. Ontology/metadata-driven, not prose intuition: - generate/meaning_graph/relational.py: declarative algebra tables — _INVERSE_PAIRS (the converse edges; the pack carries no inverse metadata) + INVERSE_OF (derived, an involution) + SYMMETRIC_PREDICATES (mirrors the pack's graph.edge.symmetric tag). load_relational_pack_symmetric() reads the pack ontology; a test pins the constant equal to it (no silent divergence). - generate/determine/determine.py: _relational_one_hop() between direct entailment and transitive subsumption; new Determined.rule provenance field (direct/inverse/ symmetric/subsumption) — replay-safe (render reads only basis; trace hashes surface). wrong=0 confuser block proves the rule cannot over-fire: less_than is not self-inverse, sibling_of does not imply parent_of, greater_than does not imply equal_to, NO transitive chain (direct or through-inverse), and no answer=False is ever emitted. An obsolete pin (test_symmetric_converse_is_not_faked, which asserted the pre-rule direct-only floor) is updated to the new sound behavior and a preserved asymmetric-converse bite. Tests: test_determine_relational_inference.py (13, the capability contract) + updated test_relational_reader.py. Broad relational/determine/grounding + capability_index baseline sweep: 114 green; baseline digest UNCHANGED (this is a determination capability; the comprehension lanes are untouched). REMAINING for the full lead PR (next): the measurement lane — an independent-oracle evals/relational_inference lane + a capability_index adapter + baseline re-freeze (breadth 9->10, wrong_total still 0), so the capability registers on the yardstick. INTEGRATION NOTE (cross-PR): this adds 2 Determined() construction sites (now 4: direct/inverse/symmetric/subsumption, ALL answer=True). When rebased onto main with INV-30 (PR #770), update INV-30's test_determine_construction_sites_are_visible count 2 -> 4 and confirm all four assert True. Merge order: #770 -> this. * feat(capability-index): relational-inference lane — breadth 9->10, wrong=0 Mastery-v2 Step 3 lead, measurement half. Puts the one-hop relational-inference capability ON the capability index with an independent-oracle lane: - evals/relational_inference/v1/: cases.jsonl (13 positive: 8 inverse + 5 symmetric, authored from the relation algebra INDEPENDENTLY of determine/relational per INV-25/27), refusals.jsonl (8 confusers that MUST refuse), provenance.md (incl. the honest overlaps_event reader-surface coverage gap). - evals/comprehension/relational_inference_runner.py: told fact(s) -> determine(query) vs gold; refusal = coverage miss, disagreement = wrong (structurally 0 on positives). - evals/capability_index/adapters.py: comprehension_relational_inference_result added to ADAPTERS. - baseline.json re-frozen: breadth 9 -> 10, capability_score 0.94403 -> 0.949483, wrong_total still 0, digest deliberately changed (35dea2b2...). - tests/test_relational_inference_lane.py: SHA-pinned gold, positive wrong=0 + coverage>0, and the confuser wrong=0 BITE (every confuser must refuse). Also reconciles INV-30 (now in main via #770): determine.py grew from 2 to 3 Determined() construction sites (direct, the shared relational one-hop constructor, transitive subsumption) — ALL answer=True. test_determine_construction_sites_are_visible updated 2 -> 3 (correcting the prior commit note's overcount of 4; inverse and symmetric share the single _relational_determined constructor, so it is ONE new site). Also fixes a .gitignore gap: ADR-0219 gen-dir checkpoints (engine_state/current, engine_state/gen-*/) were not ignored (only the old flat engine_state/*.json patterns were), so runtime checkpoints could be committed by accident. Verification: new lane 13/0/0 coverage 1.0; capability_baseline digest matches the re-freeze; INV-30 + architectural invariants green; smoke 99/99. * docs+test: reconcile stale relational-inference docstrings; assert rule provenance (review #775) Addresses two review-requested cleanups on #775: 1. Stale docstrings. generate/meaning_graph/relational.py said the symmetric converse is "a sound-but-incomplete refusal at DETERMINE" and there is "NO transitive/ symmetric/rule inference"; generate/determine/determine.py said "symmetric-converse questions are Undetermined", "no transitive/symmetric/rule inference", and "asserts only answer=True on a direct hit". All false after the one-hop relational algebra landed. Updated: the reader does direct-reading only; DETERMINE applies declared one-hop inverse/converse + pack-declared symmetric (plus the existing member/subset subsumption), still open-world / never answer=False; transitive relational closure, negation, and closed-world falsehood remain out of scope. 2. Rule-provenance gold now meaningful. The relational_inference runner compared only (answer, predicate, subject, object), ignoring res.rule though the gold carries "rule": "inverse"/"symmetric". Added res.rule to the got/gold tuple so a "right answer via the wrong rule path" can no longer pass silently. Lane still 13/0/0; baseline counts (and digest) unchanged. Verification: runner 13/0/0; lane + capability_baseline + unit + INV-30 green (22).
94 lines
1.9 KiB
JSON
94 lines
1.9 KiB
JSON
{
|
|
"capability_score": 0.949483,
|
|
"coverage_geomean": 0.949483,
|
|
"coverage_micro": 0.993932,
|
|
"accuracy_micro": 1.0,
|
|
"breadth": 10,
|
|
"min_domain_coverage": 0.833333,
|
|
"wrong_total": 0,
|
|
"assert_mode_valid": true,
|
|
"deterministic_digest": "35dea2b23084de9d56bd97225ccee6d9c07e650551559dd26c4fed7324806ffc",
|
|
"domains": [
|
|
{
|
|
"domain": "comprehension_propositional",
|
|
"correct": 12,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"coverage": 1.0,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "comprehension_relational_inference",
|
|
"correct": 13,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"coverage": 1.0,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "comprehension_relational_metric",
|
|
"correct": 15,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"coverage": 1.0,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "comprehension_relational_predicate",
|
|
"correct": 17,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"coverage": 1.0,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "comprehension_set_membership",
|
|
"correct": 8,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"coverage": 1.0,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "comprehension_syllogism",
|
|
"correct": 7,
|
|
"wrong": 0,
|
|
"refused": 1,
|
|
"coverage": 0.875,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "comprehension_total_ordering",
|
|
"correct": 7,
|
|
"wrong": 0,
|
|
"refused": 1,
|
|
"coverage": 0.875,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "deductive_logic",
|
|
"correct": 716,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"coverage": 1.0,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "dimensional",
|
|
"correct": 10,
|
|
"wrong": 0,
|
|
"refused": 2,
|
|
"coverage": 0.833333,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "relational_metric",
|
|
"correct": 14,
|
|
"wrong": 0,
|
|
"refused": 1,
|
|
"coverage": 0.933333,
|
|
"accuracy": 1.0
|
|
}
|
|
],
|
|
"not_covered": []
|
|
}
|