a005a92fed
The binding-graph's FIRST comprehension consumer (doctrine-aligned: quantities live in binding_graph, NOT the MeaningGraph). generate/quantitative_comprehension.py reads arithmetic prose into SymbolBinding/BoundFact/BoundEquation and runs the REAL check_admissibility (shell -> verify -> rebuild with the actual UnitProof) — there is NO stamped "admitted": an equation is admitted only if its operand units verify. Then to_relational_metric projects the binding-graph to the independent relational_metric oracle for the verdict. Templates (digits only; non-digit quantity REFUSES): "<X> has <N> <unit>" -> BoundFact(X = N) "<Y> has <N> more <unit> than <X>" -> BoundEquation(Y = X + N) op=add "<Y> has <N> fewer <unit> than <X>" -> BoundEquation(Y = X - N) op=subtract "How many <unit> does <Y> have" -> ask Y "How many <unit> do <X> and <Y> have"-> total = X + Y; ask total Unit modelling (honest, not faked): a noun the closed en_units_v1 pack knows is used verbatim (dollars -> dollar/money); an UNKNOWN sortal noun (stickers, coins) is a count of discrete objects -> the existing 'item' lemma (dimension count). So admissibility stays a REAL check: count+count admits, count+money (a mixed-unit sum) REFUSES with unit_mismatch — verified to bite. comprehension_relational_metric: 15/15 wrong=0 (full coverage). Located OUTSIDE generate/meaning_graph (it targets binding_graph, not the MeaningGraph) so INV-28 neutrality stays intact; oracle imports none of the SUT (new INV-25 lane). Capability index breadth 7->8, score 0.928622 -> 0.937258, wrong_total 0, digest 50e0675b… Tests: reader templates + count/known-unit modelling + admissibility-bite (mixed unit refuses) + non-digit refusal; end-to-end full-coverage wrong=0; arithmetic added to the structure-preservation generative panel (projected relations+query == ground truth); capability breadth 7->8; INV-25 arithmetic lane. 93 targeted + 90 smoke green; lane SHAs 8/9 (sole miss = public_demo env flake; deductive_logic + math_teaching unchanged -> no GSM8K coupling).
78 lines
1.6 KiB
JSON
78 lines
1.6 KiB
JSON
{
|
|
"capability_score": 0.937258,
|
|
"coverage_geomean": 0.937258,
|
|
"coverage_micro": 0.993703,
|
|
"accuracy_micro": 1.0,
|
|
"breadth": 8,
|
|
"min_domain_coverage": 0.833333,
|
|
"wrong_total": 0,
|
|
"assert_mode_valid": true,
|
|
"deterministic_digest": "50e0675bd69938ce5747b5d47592504b9cf143027dd6c1c5e410da0add25341c",
|
|
"domains": [
|
|
{
|
|
"domain": "comprehension_propositional",
|
|
"correct": 12,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"coverage": 1.0,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "comprehension_relational_metric",
|
|
"correct": 15,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"coverage": 1.0,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "comprehension_set_membership",
|
|
"correct": 8,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"coverage": 1.0,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "comprehension_syllogism",
|
|
"correct": 7,
|
|
"wrong": 0,
|
|
"refused": 1,
|
|
"coverage": 0.875,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "comprehension_total_ordering",
|
|
"correct": 7,
|
|
"wrong": 0,
|
|
"refused": 1,
|
|
"coverage": 0.875,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "deductive_logic",
|
|
"correct": 716,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"coverage": 1.0,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "dimensional",
|
|
"correct": 10,
|
|
"wrong": 0,
|
|
"refused": 2,
|
|
"coverage": 0.833333,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "relational_metric",
|
|
"correct": 14,
|
|
"wrong": 0,
|
|
"refused": 1,
|
|
"coverage": 0.933333,
|
|
"accuracy": 1.0
|
|
}
|
|
],
|
|
"not_covered": []
|
|
}
|