Commit graph

1404 commits

Author SHA1 Message Date
Shay
9337223733
Merge pull request #626 from AssetOverflow/feat/r2-solver-verifier
feat(constraint): R2 Pack B — exact integer solver + answer-choice verifier
2026-06-07 08:28:12 -07:00
Shay
ae16c0cae7
Merge pull request #625 from AssetOverflow/feat/r2-ruler
feat(constraint): R2 Pack A — constraint IR + setup oracle + gold + ADR-0211 (the ruler)
2026-06-07 08:19:31 -07:00
Shay
8d2c270144
Merge pull request #624 from AssetOverflow/feat/r1-inverse-frame
feat(comprehension): R1 inverse reader frame — close R1 at 7/0/3 (PR-7b)
2026-06-07 08:04:23 -07:00
Shay
a3a26b8fc5 test(constraint): pin the three Pack-C review hazards as regression tests
Per review: (H1) exactly-two-category binding already has its meaningful-fail twin; (H2) a weighted total whose unit does not match the coefficient unit refuses missing_weighted_total (never a cross-unit sum); (H3) answer-choice ties EXACTLY or refuses no_matching_option (never nearest). All three were already handled by construction; these pin them so they stay handled.
2026-06-07 07:49:10 -07:00
Shay
faecb21c37 docs(analysis): R2 inventory ledger — reader 10/0/0, 6 families read+solved+verified (R2 C9)
Records the final R2 v1 state: reader setup_correct 10 / setup_wrong 0 + 3 correct reader-refusals; 7 solved answers + 3 solver-refused + 3 reader-refused; gold 13/13. Per-fixture table, the four wrong=0 gates, the reader-vs-solver division of labor (equal coefficients are a solver refusal, not a reader one), covered families, and R3 deferrals. R1 7/0/3 and 15-case 15/0/0 unchanged.
2026-06-07 07:36:17 -07:00
Shay
4ddaef4d00 feat(constraint): two-category reader — prose to ConstraintProblem (R2 C5-C9)
generate/constraint_comprehension/reader.py: read_constraint_problem comprehends two-category constraint prose into a typed ConstraintProblem via five recognizers — C5 category-pair (from coefficient clauses; >2 -> too_many_categories), C6 coefficient (Each <cat> holds/has/costs/is-worth <N> <unit>; shared measured unit), C7 total-count (collective-unit sentence -> x+y=N; absent -> missing_total_count), C8 weighted-total (measured-unit sentence -> ax+by=T; absent -> missing_weighted_total), C9 query-target (How many <cat> -> the asked unknown). Refuses, never mis-assembles.

Reconciles the sketch's 'no equal coefficients' note: equal coefficients are a SOLVER refusal (indistinguishable_weights), not a reader one — the reader reads r2-010 setup_correct and the solver refuses it. run_reader lane + 'reader' CLI subcommand. R2 reader: setup_correct 10 / setup_wrong 0 / refused_correct 3; R1 7/0/3 and 15-case 15/0/0 unchanged. Off-serving. 8 reader tests incl. full read->solve->verify chain + meaningful-fail guards.
2026-06-07 07:34:55 -07:00
Shay
121362c52a feat(answer-choices): multiple-choice verifier with contradiction flag (R2 C4)
generate/answer_choices/{parse,verify}.py: parse_options normalizes {label:value} to {label:int} (int or single-integer string; ambiguous/empty refuse). verify_answer_choice ties a PROVEN value to exactly one option -> ChoiceVerdict(consistent); a disagreeing key -> ChoiceVerdict(contradiction) naming both the consistent answer and the wrong key (truth discipline, not a refusal). Refuses no_matching_option / ambiguous_options / unknown_provided_label.

End-to-end with C2 gold + C3 solver: every solved fixture solves, ties to its labeled answer, confirms consistent. Off-serving. 9 tests incl. contradiction-flag meaningful-fail.
2026-06-07 07:26:00 -07:00
Shay
babcf2fdb2 feat(constraint): exact integer 2-var solver — Cramer's rule, refusal-first (R2 C3)
generate/constraint_comprehension/solver.py: solve_two_var_linear (order-independent 2x2 integer Cramer's rule over typed constraints), the solve_two_var_count_weight specialization, and solve/answer_constraint_problem driving it from a ConstraintProblem. Four typed refusals: indistinguishable_weights (det==0), non_integer_solution (numer%det!=0, never rounds), negative_solution, verification_failed (identity backstop).

Ties to the C2 gold: solves all 7 solved fixtures to their gold value and refuses all 3 solver_refuses fixtures with EXACTLY the gold-claimed reason (the gold's reason is now solver-verified, not just annotation). Per-refusal meaningful-fail + positive re-substitution. Off-serving. 9 tests.
2026-06-07 07:23:23 -07:00
Shay
a098519bc1 feat(constraint): R2 setup oracle + gold + ADR-0211 — the ruler (R2 C2)
evals/constraint_oracle: span-free canonical setup signature (unknowns/facts/constraints/query; terms merged+sorted, lhs constant folded into rhs), a 13-fixture gold (7 solved / 3 solver_refuses / 3 reader_refuses) with a closed expect+refusal taxonomy, and a gold-validation runner. No reader yet — this proves the ruler is coherent (13/13 valid), not capability.

ADR-0211 ratifies the R2 finite-integer two-category constraint compiler: the IR, the canonical signature, the closed taxonomy, the four wrong=0 gates (setup oracle / reader refusal / exact solver / answer-choice contradiction flag), off-serving disjointness, and the C1-C9 ladder. Cross-refs ADR-0207/0175/0083/0055-0057; reuses the R1 setup_oracle pattern. Off-serving (no generate.derivation / reliability_gate). 13 oracle tests incl. per-branch meaningful-fail.
2026-06-07 07:19:29 -07:00
Shay
e71531c0c9 feat(constraint): R2 linear-constraint IR — typed problem model (R2 C1)
generate/constraint_comprehension/{expr,model}.py: frozen, slots'd dataclasses, no behavior. expr = LinearExpr (sum(coeff*symbol)+constant) + LinearConstraint (lhs eq rhs, optional source_span). model = Unknown (symbol/entity/unit/finite-integer domain), AttributeFact (per-category coefficient provenance), ConstraintQuery (symbol+unit), ConstraintProblem (unknowns/facts/constraints/query).

Terms pinned as (symbol, coefficient) to match the gold serialization. Query is a minimal dedicated type, not R1's BoundUnknown (no degenerate fit). Off-serving package; no generate.derivation / reliability_gate import. 9 IR tests (shape + frozen + defaults).
2026-06-07 07:10:20 -07:00
Shay
60b40d3e3a feat(comprehension): inverse reader frame — base of a more/fewer-than (PR-7b / R2 C0)
r1-07 now reads setup-correct and answers 6 — 'Nia has 9 more beads than Omar. Nia has 15. -> Omar = 6'. The reader binds the unknown base's unit FROM the relation when its subject is a known fact and its referent is the otherwise-ungrounded query target, so the equation is admissible; the answer oracle reverse-solves it (PR-7a). Bounded: single base == query target (no chains), known subject value, base not otherwise grounded, <=1 inverse (multiple_inverse_bases else), never over times/divide.

R1 setup 6/0/4 -> 7/0/3; R1 answers -> 7 correct / 0 wrong; 15-case 15/0/0; setup_wrong stays 0. Off-serving. Refreshes the R1 ledger to 7/0/3 (R1 closed; the 3 remaining refusals are wrong=0 boundaries).
2026-06-07 07:06:26 -07:00
Shay
0e6a7f9ac0
Merge pull request #619 from AssetOverflow/feat/inverse-solve-contract
feat: narrow reverse-solve oracle contract (PR-7a, contract only)
2026-06-07 06:44:07 -07:00
Shay
036ed8c659 feat(oracle): narrow reverse-solve contract — base of one more/fewer_than (PR-7a)
Pin the EXACT reverse-solve semantics in the independent answer oracle before the
reader learns the inverse frame (PR-7b). A single more_than/fewer_than whose entity
is grounded and whose ref is the (otherwise ungrounded) query target inverts to the
base: base = entity -/+ delta (always integer-exact for +/-).

Refuses everything wider (each a meaningful-fail oracle test): more than one inverse
constraint, an inverse base that is not the query target (a chain), an already-grounded
base, a negative count result, or an inverse over times/divide. The duplicate-definition
guard moved into the per-kind branches so a grounded entity on a more/fewer_than is read
as an inverse constraint, not a redefinition; forward more/fewer/times/divide/sum paths
are unchanged (regression-guarded).

Contract only — the reader is NOT touched: r1-07 still refuses (admissibility_refused),
so R1 stays 6 / 0 / 4 and 15-case 15 / 0 / 0. r1-07 gold drops the non-consumed ask_base
(aligning with the reader's future projection) and gains gold=6, dormant until PR-7b.

Off-serving (only evals/tests import the oracle). SHA gate 8/9 (public_demo env flake).
2026-06-07 06:22:36 -07:00
Shay
a8707d3bac
Merge pull request #618 from AssetOverflow/feat/aggregate-query-variants
feat: additive aggregate query variants — 'altogether' / 'in total'
2026-06-07 05:59:16 -07:00
Shay
ef06923866 feat(comprehension): additive aggregate query variants — 'altogether' / 'in total'
Widen the aggregate-query recognizer so a multi-part total may be asked with a
trailing qualifier after 'have': 'How many X do A and B have altogether?' and
'... in total?'. The qualifier is stripped and honored ONLY for the multi-part
(sumquery) form — a single-entity query carrying it refuses, guarded by
'not aggregate'.

Phrasing-only: no new arithmetic, no new relation kind, no inverse solving, no
distractor/pronoun handling. The parts still flow through the existing sum_of;
an ungrounded part (unit unbound) or a unit-incompatible part (unit mismatch) is
refused downstream by the REAL admissibility check, so the recognizer cannot
over-read. Off-serving organ only (no generate.derivation / reliability_gate).

Flips r1-03 (more+altogether -> 25) and r1-04 (fewer+in total -> 34):
  R1 setup   6 / 0 / 4    R1 answers 6 / 0 / 4  (setup_wrong 0, gold_error 0)
  15-case 15 / 0 / 0      29 quant tests, 102 affected-file tests green

Tests are meaningful-fail: the single-entity-qualifier, ungrounded-part, and
unit-incompatible-part refusals each fail loudly if their guard is removed.
2026-06-07 05:49:28 -07:00
Shay
82b14065f2
Merge pull request #617 from AssetOverflow/docs/r1-inventory-ledger
docs: R1 comprehension inventory ledger (4/0/6 baseline)
2026-06-07 05:49:12 -07:00
Shay
261c0e4c2d docs(analysis): R1 comprehension inventory ledger
Baseline decision artifact at main@5ada1392 (post-#616). Records the R1
4/0/6 state per-fixture: the 4 admitted semantic families and the invariant
protecting each, and a classification of the 6 refusals — 2 pure phrasing
gaps (r1-03/r1-04 aggregate-query), 1 real reverse-solve capability (r1-07),
and 3 correct wrong=0 boundaries that must stay refused (r1-08/09/10).

Source-verified against the merged reader (refusal codes + line refs) and
the live lanes (setup 4/0/6, answers 4/0/6 setup_wrong/gold_error 0,
15-case 15/0/0). Docs-only; no code, no serving lane, no SHA-lane content.
2026-06-07 05:30:14 -07:00
Shay
5ada1392a5
Merge pull request #616 from AssetOverflow/feat/partition-aggregate-divide
feat(comprehension): aggregate-then-divide partition frame — "split equally into N boxes" (PR-6d)
2026-06-07 05:24:50 -07:00
Shay
f9ef9e56a4 feat(comprehension): aggregate-then-divide partition frame — "split equally into N boxes" (PR-6d)
PR-6d adds the partition frame: combine all parts into a total, then split that
total equally into N containers. r1-06-subtotal-reused moves refused → correct —
the FIRST case where the divisor applies to a DERIVED symbol (the total), not a
directly given fact. That is real progress toward GSM8K setup comprehension,
where intermediate quantities are the norm.

Scope (kept narrow on purpose):
  No new relation kind.
  No new arithmetic operation.
  No rational support.
  No rounding/flooring.
  No serving path touched.

The frame reuses the already-ratified pieces — SumOf(parts) + Div(Symbol(total),
Literal(N)) → divide_by — so this PR is reader-only (no IR / admissibility /
oracle / signature change).

Frame grammar:
  "They combine their <unit> and split them equally into N <containers>."
  + "How many <unit> are in each <container>?"
  -> total = sum(all facts); per_<container> = total / N; ask per_<container>.

wrong=0 boundaries:
- Exact-divisibility still gates the ANSWER, now over a derived total: 5+6=11,
  11/3 is non-exact -> the setup reads correctly but the answer REFUSES (never
  floors). Setup comprehension and answer exactness are cleanly separated.
- Partition/query coherence: a partition is read ONLY together with its
  "in each <container>" query (and vice versa); container mismatch (box vs jar)
  refuses. Prevents over-reading a story detail into an unused derived value.
  Meaningful-fail verified: disabling the guard makes a dangling partition
  wrongly comprehend.

Gates:
  R1 setup:   4 correct / 0 wrong / 6 refused
  R1 answers: 4 correct / 0 wrong / 6 refused / setup_wrong 0 / gold_error 0
  15-case setup: 15 / 0 / 0
  97 PR-6d tests + 99 relational/invariant tests green. Reader is off-serving
  (no generate.derivation / core.reliability_gate import).
2026-06-06 21:07:19 -07:00
Shay
409615fcef
Merge pull request #615 from AssetOverflow/feat/half-as-many-divide
feat(comprehension): the divisive comparative frame — "half as many" as exact integer division (PR-6c)
2026-06-06 20:44:52 -07:00
Shay
0951d80e04 feat(comprehension): the divisive comparative frame — "half as many" as exact integer division (PR-6c)
PR-6c adds the divisive comparative frame: "half as many" read as EXACT INTEGER
DIVISION. It is the divisor twin of PR-5c's multiplicative frame, and moves the
independent R1 gold's r1-02-half from refused → correct.

No serving path touched. No rational/fractional answer support added. Non-exact
division refuses.

Design (ADR-0134 amended — divide made symmetric with multiply):
- `_check_divide` now admits a SINGLE-DEP divide-by-dimensionless-literal
  (item / dimensionless = item), the exact twin of single-dep multiply. The
  2-dep rate-divide path is untouched. This keeps the IR's "literal operands
  are not deps" invariant (proven in PR-6a) uniform across Mul AND Div, so the
  reader builds both without a per-op special case and WITHOUT synthesizing a
  divisor symbol that would pollute the setup-oracle's unit signature.
- `Div(Symbol, Literal)` IR node: "ref / divisor", operation_kind "divide",
  projects to `divide_by`. Divisor-only contract mirrors the scalar-only one.
- Reader: `_DIVISOR_WORDS={half:2}` slots into the same 8-token "<WORD> as many"
  template as the factor words; graph carries only the two entities.
- Gold reconciliation: r1-02 placeholder `times_as_many factor 0.5` → exact
  `divide_by divisor 2` (gold 4). Makes the INDEPENDENT gold integer-faithful.

The wrong=0 boundary — exact divisibility:
  the oracle admits `divide_by` only when `base % divisor == 0`. An odd base
  halved REFUSES (gold_error), never floors to a wrong integer. Divisor must be
  a nonzero int (0, 0.5, 1.5, bool all refuse); divisor=1 is intentionally the
  identity (pinned). admissibility proves DIMENSION; the oracle proves EXACT VALUE.

Meaningful-fail (CLAUDE.md Schema-Defined Proof Obligations), both verified red:
- drop the `% divisor` guard → test_oracle_refuses_non_exact_division fails (returns 3).
- disable the single-dep divide branch → the admissibility test AND the reader's
  `half` test fail (admissibility refuses → reader refuses → half stays refused).

Gates:
  R1 setup:   3 correct / 0 wrong / 7 refused
  R1 answers: 3 correct / 0 wrong / 7 refused / setup_wrong 0 / gold_error 0
  15-case setup: 15 / 0 / 0
  91 PR-6c tests + 60 relational lanes + 56 architectural invariants + 502
  binding-graph/proof-chain/adapter tests green. All 8 SHA-content lanes match
  (serving unmoved; admissibility has no generate.derivation/reliability_gate consumer).
2026-06-06 20:18:39 -07:00
Shay
5ac1526536
Merge pull request #614 from AssetOverflow/pr-6b-r1-answer-oracle-followup
test(comprehension): prove R1 times-as-many answer oracle (PR-6b follow-up)
2026-06-06 19:35:57 -07:00
Shay
f4cf490e8d test(comprehension): prove R1 answer-oracle lane 2026-06-06 18:32:21 -07:00
Shay
15b0b583aa test(comprehension): expose R1 answer lane CLI 2026-06-06 18:31:41 -07:00
Shay
56260629b8 test(comprehension): add answer gold for setup-correct R1 cases 2026-06-06 18:31:12 -07:00
Shay
88d8950d3e test(comprehension): expose R1 answer lane 2026-06-06 18:28:49 -07:00
Shay
c364d434da feat(comprehension): add R1 answer-oracle lane 2026-06-06 18:26:30 -07:00
Shay
ce8897e84f feat(comprehension): add off-serving times_as_many answer oracle 2026-06-06 18:20:23 -07:00
Shay
2786b32d62
Merge pull request #613 from AssetOverflow/feat/scalar-multiply-invariant 2026-06-06 18:08:41 -07:00
Shay
c2b97f40bf test(comprehension): prove the scalar-multiply contract (PR-6a)
The multiplicative comparative frame (PR-5c) admits exactly one shape —
Mul(Symbol, Literal), a unit-bearing symbol times a dimensionless integer
(count × scalar = count). That contract was held by OMISSION: to_relation's
`case _: return None` refused every other Mul shape, but no test would fail
if the guard were loosened, and no doc stated where the guarantee lives.

This makes the obligation meaningfully-failing (CLAUDE.md Schema-Defined
Proof Obligations), with no runtime logic change:

- test_mul_projection_admits_only_symbol_times_literal — Mul(Symbol, Symbol)
  (a count×count product), a commuted factor, and compound factors all REFUSE
  (to_relation → None). Verified to go red when a Mul(Symbol, Symbol) projection
  arm is injected.
- test_literal_factor_is_dimensionless_by_construction — Literal has exactly
  one field (value); a unit-bearing literal multiplication is unrepresentable,
  not merely unchecked.
- test_scalar_only_guard_is_load_bearing — check_admissibility's `multiply`
  dispatch products operand units generally (count×count → count², no refusal),
  so it would NOT catch the masquerade. The projection arm is the sole boundary.

Docstrings on Mul and to_relation now state the scalar-only contract and that
it is enforced at the projection boundary, not in the dimensional checker.

Gates unchanged: setup-oracle 15-case 15/0/0 and R1 2/0/8 (setup_wrong=0);
77 expr/admissibility/reader/setup-oracle tests + 56 architectural invariants
green. No serving path touched.
2026-06-06 17:42:22 -07:00
Shay
9a285dfbdb
Merge pull request #612 from AssetOverflow/feat/r1-setup-gold
feat(comprehension): R1 gold (PR-5b) + the multiplicative comparative frame (PR-5c)
2026-06-06 17:38:35 -07:00
Shay
e9cbe65d77 feat(comprehension): the multiplicative comparative frame — first R1 capability (PR-5c)
The first capability slice on the R1 arc, gated by the setup-oracle: turn the
"twice / N times as many" reading from REFUSED into a correct setup, without a single
misread. Builds on the typed IR (PR-4) and the R1 gold (PR-5b).

- IR: a Mul(symbol, literal-factor) node — to_canonical_string "ref * factor",
  operation_kind "multiply", dependencies {ref}, to_relation -> times_as_many. The
  product keeps the symbol's unit (count * scalar = count), admitted by the REAL
  check_admissibility multiply path (the literal factor is dimensionless, not a dep).
- Reader: a multiplicative template "Y has <factor> as many <unit> as X" (factor word:
  twice/double/triple/quadruple) and "Y has <N> times as many <unit> as X", checked
  BEFORE the digit gate (the factor may be a word). 'half' (a /2) is deliberately
  deferred — divide-by-literal is a separate admissibility path.
- setup-oracle: relation_signature now canonicalizes times_as_many.

Setup-oracle R1 result: 2 setup_correct (r1-01 twice; r1-05 the multi-step chain
ivy/jon=3*ivy/kim=jon+2), 0 setup_WRONG, 8 setup_refused. Every hard negative stays a
safe refusal: missing-base (Rosa ungrounded), ambiguous referent, distractor, inverse,
partition, 'altogether'/'in total' phrasings, and 'half' (divide). wrong=0 held through
the first capability addition.

Gates green: setup-oracle R1 setup_wrong=0; 15-case setup gate 15/15 setup_wrong=0;
relational_metric answer lane 15/15 wrong=0; binding-graph admissibility + realize +
architectural invariants + chat-runtime + pipeline (122+). No serving path touched (this
reader feeds the relational_metric / setup-oracle lanes, not the candidate-graph serving).
2026-06-06 17:29:23 -07:00
Shay
5b193280a5 feat(setup-oracle): independent R1 gold + the reader refuses, never misreads (PR-5b)
Pure-gold. Defines what "correct" means for the R1 (comparative / derived-symbol /
multi-step) shapes the reader cannot read yet, and verifies the current reader REFUSES
them rather than misreading them. NO frame extraction, no parser change, no serving.

- evals/setup_oracle/r1_gold.jsonl — 10 INDEPENDENT, self-contained fixtures (id, text,
  relations, expected_units, query, notes) covering: twice/half-as-many (multiplicative),
  more/fewer-than -> total, a multi-step derived chain (jon=3*ivy; kim=jon+2), a derived
  subtotal reused downstream (total=lee+mae; per_box=total/3), an inverse target, and the
  hard negatives (ambiguous referent, missing base quantity, distractor quantity). The
  gold is the authority — each fixture is reviewable without joining files.
- run_r1() scores the CURRENT reader against this gold.

Investigative result (the deliverable): **10/10 setup_refused, setup_wrong=0, setup_correct=0.**
The reader refuses every R1 shape with a TYPED reason (non_digit_quantity /
no_quantity_template / unreadable_quantity_clause / admissibility_refused) — it NEVER
misreads an R1 case as a simpler form. So the wrong=0 posture holds all the way down, and
PR-5c (the R1 frame) starts from a clean foundation: every R1 case is currently a safe
refusal, and the frame's job is refusal -> correct, with the oracle ensuring no
setup_wrong is ever introduced.

`python -m evals.setup_oracle r1` prints the auditable R1 report (exit 0 iff
setup_wrong==0). The 15-case setup gate + relational_metric answer lane are untouched.
2026-06-06 17:18:45 -07:00
Shay
6c8f2b8332
Merge pull request #611 from AssetOverflow/feat/setup-oracle-units
feat(setup-oracle): make the ruler UNIT-AWARE (setup-oracle v2, PR-5a)
2026-06-06 17:13:20 -07:00
Shay
4056069152 feat(setup-oracle): make the ruler UNIT-AWARE (setup-oracle v2, PR-5a)
Strengthen the ruler before it judges real GSM8K frames — NOT a capability change, no
new parser behavior, no serving path. The setup-oracle now grades UNITS, read from the
binding-graph itself, against an independent expected-unit gold.

- New per-symbol unit axis (symbol_unit_signature) — read from the GRAPH's symbols
  (reader_symbol_units), not the answer projection; covers fact + relation-operand units
  (operands are symbols). A None unit canonicalizes to "unset" so an unmodelled symbol
  never silently matches a declared gold unit.
- The question-target signature gains the target's expected_unit (reader from
  BoundUnknown.expected_unit; gold from the fixture).
- expected_units.json — independent, hand-authored MODELING unit per entity for the 15
  cases (a discrete sortal count -> the generic count unit "item"; money -> "dollars").
  Authored from the PROBLEM, not copied from the reader.
- The reader's per-symbol units match the independent gold on all 15 (setup_wrong=0) —
  agreement, not circularity (both apply the same correct count->item / money modelling).
- Meaningful-fail: a reading whose STRUCTURE matches but whose UNIT diverges now FAILS
  (test_unit_mismatch_is_caught, test_target_unit_mismatch_is_caught). The ruler is not
  decoration on the unit axis either.

Gate held: setup-oracle 15/15 setup_wrong=0 (now structure + units + target). The
relational_metric answer lane is untouched (oracle-only change). No generate/ change.

This is the unit-aware ruler PR-5b (independent R1 gold) and PR-5c (the first R1
comparative/derived-symbol frame) must pass before serving.
2026-06-06 17:09:45 -07:00
Shay
3d006d2e19
Merge pull request #610 from AssetOverflow/feat/quant-expression-ir
refactor(comprehension): typed expression IR as the source of meaning (PR-4)
2026-06-06 17:03:21 -07:00
Shay
06450928c9 refactor(comprehension): typed expression IR as the source of meaning (PR-4)
Internal hygiene + future-proofing. No serving path, no new capability — the 15-case
reader is structurally cleaner and the projection no longer recovers meaning by
re-parsing a string.

- New generate/quantitative_expr.py: a typed Expr IR (Literal/Symbol/Add/Sub/SumOf) with
  to_canonical_string (BYTE-IDENTICAL to the legacy "ref + delta" / "ref - delta" / "a + b"
  format), dependencies, operation_kind, and to_relation (the structured projection).
- The reader builds the Expr per equation as the SOURCE OF MEANING; rhs_canonical,
  dependencies, and operation_kind are all DERIVED from it (BoundEquation stays a string —
  the binding-graph's deliberate decoupling layer is untouched). QuantComprehension carries
  the IR as equation_exprs.
- to_relational_metric reads the IR via to_relation — the rhs_canonical string-reparse is
  GONE. An unhandled equation shape refuses (None).
- The dead _rhs string builder is removed.

Gates held: relational_metric answer lane 15/15 wrong=0; setup-oracle 15/15 setup_wrong=0;
malformed-target refusals intact; realize-binding-graph + architectural invariants green
(95). rhs_canonical is byte-identical, so the binding-graph + downstream hashes are
unchanged. No model change.

This is the foundation PR-5's R1 frames build on — structured equations, not string parsing
— and only after independent R1 gold is hand-authored.
2026-06-06 16:57:53 -07:00
Shay
0882a4f8ef
Merge pull request #609 from AssetOverflow/feat/drop-quantquery-read-unknowns
refactor(comprehension): drop QuantQuery — consumers read graph.unknowns (PR-3)
2026-06-06 16:52:10 -07:00
Shay
0d32a655f1 refactor(comprehension): drop QuantQuery — consumers read the target from graph.unknowns (PR-3)
Completes the PR-1 migration. The question target now has a single source of truth:
the binding-graph's sole BoundUnknown. The sidecar QuantQuery dataclass + the
QuantComprehension.query field are DELETED.

- New helper single_unknown(graph) -> BoundUnknown | None: returns the sole target, or
  None on a graph that does not carry exactly one. Zero unknowns (no question) and
  multiple unknowns (ambiguous) both REFUSE — the consumer must never pick one.
- to_relational_metric reads the query from single_unknown(graph) (refuses on None).
- realize_quantitative reads the asked symbol from single_unknown(bg) (NotRealized on None).
- Tests: the .query assertions move to single_unknown; new malformed-graph tests prove
  0 and >1 unknowns REFUSE rather than pick one (the wrong=0 boundary).

Byte-identical where it matters: relational_metric answer lane 15/15 wrong=0, setup-oracle
15/15 setup_wrong=0, realize-binding-graph + architectural invariants green. No serving
path touched. No dangling QuantQuery reference remains.
2026-06-06 16:49:09 -07:00
Shay
a700b4503c
Merge pull request #608 from AssetOverflow/feat/quant-bound-unknown-and-setup-oracle
feat(comprehension): question target in the graph (PR-1) + setup-oracle lane
2026-06-06 16:42:52 -07:00
Shay
59974865ef feat(comprehension): question target in the graph (PR-1) + setup-oracle lane (grade the reading)
Two coupled, additive, off-serving changes toward the typed math-comprehension organ.
No serving path touched; the relational_metric answer lane stays 15/15 wrong=0.

PR-1 — QuantQuery → BoundUnknown. comprehend_quantitative now emits the question
target as a BoundUnknown INSIDE the binding-graph (symbol_id, state_index="terminal",
question_form "count"|"total", expected_unit), so the graph is a real question-bearing
mathematical object and its canonical serialization carries the target. The external
QuantQuery is RETAINED, consistent-by-construction, so the two consumers
(to_relational_metric, realize/quantitative) are byte-identical; a follow-up rewires
them onto graph.unknowns and drops the duplicate field.

Setup-oracle lane (evals/setup_oracle) — grade the READING, not the answer. The
relational_metric lane scores answers, which can bless a semantically-wrong derivation
that coincidentally lands on the right number (the exact hazard the held-out
measurements + the 2/87 resolve_pooled probe exposed). The setup-oracle compares the
reader's comprehended STRUCTURE — a span-free signature of facts + typed equations +
the BoundUnknown target — against the INDEPENDENT gold structure (the relational_metric
cases' own relations/query, authored separately from the binding-graph reader). A
structural mismatch is setup_wrong, the wrong=0-critical count, even when the answer
would be right. v1 grades structure (units deferred — covered by admissibility). The
reader reads all 15 cases with the gold structure (setup_wrong=0); a meaningful-fail
test proves the oracle catches a right-answer/wrong-structure reading (it is not
decoration). `python -m evals.setup_oracle` exits nonzero iff setup_wrong > 0.

This is the measurement rig BEFORE investing in frame families: setup_wrong=0 is the
gate; serving must not move while setup_wrong > 0. It is the first milestone of the
math-comprehension organ, not a path to "solve GSM8K".

Verified: setup-oracle 15/15 setup_correct wrong=0; quantitative + setup-oracle unit
tests (17); realize-binding-graph + binding-graph + architectural invariants (183).
2026-06-06 16:40:15 -07:00
Shay
61ac070c91
Merge pull request #607 from AssetOverflow/feat/verified-canonical-comparison
docs(analysis): VERIFIED producer scoping — validate-first verdict is DO NOT BUILD
2026-06-06 16:18:14 -07:00
Shay
6f2053bcb7 docs(analysis): VERIFIED producer — validate-first verdict is DO NOT BUILD (comprehension-bound)
Scoping + holdout validation of the VERIFIED canonical-comparison producer (the
ADR-0206 §5 math-serving widening's gate). Records the empirical verdict so it is
not re-chased: on holdout_dev the fold-derivation reader is 2 correct / 87 wrong on
candidate-graph-refused cases, and a pure-chain certifier would admit 37 wrong (a
wrong=0 breach) — the train_sample 3/7 was overfit. math_verifier.verify is
solver-replay soundness, not correctness; the R1 graph reader is nested (0 flips).
Math serving is comprehension-bound; the math seam correctly stays inert; the real
lever is the general COMPREHEND organ, not a narrow GSM8K certifier. No code change.
2026-06-06 16:09:41 -07:00
Shay
cbaa3e72d7
Merge pull request #606 from AssetOverflow/feat/math-serving-reach-seam
feat(derivation): math-serving reach seam — select_self_verified is policy-aware (ADR-0206 §5)
2026-06-06 15:56:21 -07:00
Shay
c5f0c90738 feat(derivation): math-serving reach seam — select_self_verified is policy-aware (ADR-0206 §5)
Completes the ADR-0206 §5 math-serving deferral in its own careful PR. The GSM8K
wrong=0 serving gate, select_self_verified, now takes a ReachPolicy. This changes
NO serving behavior today — it is the safe, byte-identical, sanctioned first edit to
the most wrong=0-critical line, with a live-wiring test, so a future VERIFIED
widening has a precise, proven integration point.

Why it can't actually widen yet (the tension scoping surfaced): GSM8K wrong=0 is
ABSOLUTE (zero wrong, ever); a reliability license is STATISTICAL (a 0.99 Wilson
floor); math answers aren't disclosed like the cognition path (E). So widening the
math serve on a statistical license would eventually serve a SILENT wrong. ADR-0206
§4 foresaw this: VERIFIED is "the only state that will license widening past gold,"
and it is reserved pending a canonical-comparison pass (the soundness≠correctness
gap) that is unbuilt.

Design (safe by construction):
- select_self_verified(..., policy=STRICT_POLICY). STRICT (the default every one of
  the 5 callers passes) is the prior logic verbatim — unique answer → Resolution;
  zero-verify or disagreement → refuse. Byte-identical: the pinned serving-lane SHAs
  (demo_composition 3a3d09f3, fabrication 01e1b6b7, math_teaching_corpus eaf160d1)
  are unchanged; Resolution is unchanged; no import cycle.
- A wider reach resolves a disagreement ONLY via _canonically_verified — the VERIFIED
  gate — whose body returns None (capability unbuilt). So the widening is
  STRUCTURALLY inert: a disagreement refuses regardless of policy. wrong=0 holds by
  construction, not caller discipline. A statistical reliability license is NOT
  consulted by the math path (the cognition/math asymmetry is deliberate: cognition
  discloses, math is absolute).
- test_seam_is_live_wiring injects the gate to prove the consumer fires under a wider
  reach — and STRICT still refuses even then.

Verified: smoke (90), invariants (56), governance + selfverify + ms2 (354 incl. the
new seam tests), GSM8K serving regression byte-identical (the one red,
test_serving_unchanged_by_search expecting stale 6/44, fails identically on clean
main — a pre-existing stale artifact, not this change). Three-lens adversarial review
(byte-identity/seal, structural wrong=0 inertness, no-overclaim): all held.

The real unlock — a VERIFIED canonical-comparison producer — is scoped in
docs/analysis/VERIFIED-canonical-comparison-scoping-2026-06-06.md (recommended:
back-substitution / constraint-satisfaction on a checkable problem class). Still
deferred: SITUATE (stakes), the live FEED-BACK loop, reach_level JSONL emission.
2026-06-06 15:46:14 -07:00
Shay
546045eb4d
Merge pull request #605 from AssetOverflow/feat/learned-estimation
feat(determine): calibrated disclosed estimation — the engine earns the right to guess (Step E)
2026-06-06 13:57:56 -07:00
Shay
7cb826a548 feat(determine): calibrated disclosed estimation — the engine earns the right to guess (Step E)
The final AGI-spine step (A INSTRUMENT → B WIRE → C DEEPEN → D CLOSE → E ESTIMATION).
The engine may now SERVE a DISCLOSED estimate for a query it would otherwise refuse —
but only for a predicate-class that has measured itself reliable, and never as fact.

This executes the ADR-0206 §5 cognition-path widening: the bridge's LICENSE node
(reliability_gate.license_for), previously "built — not yet called from serving", is now
called. govern_response returns APPROXIMATE iff a genuine licensed Action.SERVE
LicenseDecision is passed (STRICT for every other input — so every existing serving call
site is byte-identical); shape_surface DISCLOSES the estimate as "[approximate] …".

Mechanism:
- generate/determine/estimate.py — a BLIND converse-guesser: told p(a,b), asked p(b,a),
  it commits the converse. It never reads the pack's symmetry metadata; whether the guess
  is right is MEASURED, not assumed.
- evals/determination_estimation/ — the gold lane: run_practice (sealed, ADR-0199) folds
  the converse-guesser over symmetric (sibling_of) vs directed (parent_of) cases, scored
  against the pack's graph.edge.symmetric truth (gold independent of the solver). The gate
  DISCRIMINATES: sibling_of earns SERVE (660 correct → Wilson floor 0.990046 ≥ θ_SERVE),
  parent_of does not (660 wrong → 0.0). The license is earned by VOLUME — 657 perfect
  commits is the exact θ_SERVE=0.99 threshold (656 is below).
- generate/determine/data/estimation_ledger.json — the ratified committed ledger,
  hash-verified on load (a hand-edited ledger raises RatifiedLedgerError); it IS the
  deterministic sealed-practice output (a GSM8K-style --check test pins this).
- chat/runtime.py — when a converse query is refused and the class holds a SERVE license,
  the disclosed estimate is surfaced through the bridge (gated by config.estimation_enabled,
  default OFF; only meaningful with accrue_realized_knowledge).

Invariants:
- wrong=0 by construction — an estimate is ALWAYS disclosed ([approximate]), never a silent
  commit (UNVERIFIED_POSSIBLE is never in APPROXIMATE's admissible set), and only a genuine
  ratified license widens (a forged {"licensed":True} dict / a PROPOSE license / an
  unlicensed SERVE all stay STRICT). Defense-in-depth: type-gate ∧ admissible-set ∧
  hardcoded disclosed state.
- never self-authored — ceilings stay at safe defaults (θ_SERVE=0.99); the engine cannot
  raise its own bar. The ledger is sealed practice, hash-verified.
- session/serving only — no corpus/pack/identity/proposal/vault mutation; the HITL teaching
  path is untouched. Deterministic; no clock/random.
- byte-identical for every non-E turn (the 2643 govern_response call passes no license).

Out of scope (separate ADR-0206 §5 PRs): the math-serving seam (select_self_verified,
touches the sealed metric), SITUATE (stakes), and the live FEED-BACK loop.

Verified green: smoke (90), architectural invariants (56), response_governance (321,
incl. the new license-gated widening test), the determination-estimation lane (12), and
the B/D/determine regression net. Four-lens adversarial review (disclosure/wrong=0,
calibration integrity, byte-identity, boundary/determinism): all held. Design:
docs/analysis/E-estimation-design-2026-06-06.md.
2026-06-06 13:49:07 -07:00
Shay
7373183dc0
Merge pull request #604 from AssetOverflow/review/codex-env-falsification
feat(sensorium): environment-falsification roadmap (6 phases, afferent) — ratified + hardened
2026-06-06 13:15:04 -07:00
Shay
7f5958dbd5 chore(sensorium): ratify environment-falsification roadmap — witness-importer hardening + ADR-0198 reconciliation
Review/ratification pass over the 6-phase afferent environment-falsification
roadmap (rebased onto D-inclusive main). The roadmap validated green: smoke (90),
architectural invariants (56), the full sensorium suite, all phase tests, and the
four eval lanes (environment-falsification / event-vision / vision / sensorimotor)
report failed:0. A five-lens adversarial review found the efferent gate
fail-closed (default-deny, conjunctive safety∧ethics∧tool_scope, refuses before
the decoder, hash-only traces, no real decoder, no overclaim), the afferent
Phases 1-5 read-only (runtime import-guards + frozen dataclasses), and the
expected-artifact obligations deterministic + meaningfully-failing. Two
reconciliations from that review:

1. Witness-log importer trust boundary (sensorium/logs/importer.py). The offline
   JSONL importer now rejects malformed input with a deterministic ValueError
   instead of a raw JSONDecodeError/KeyError, names the 1-based line on a bad
   JSON line, rejects non-object records and non-integer ticks, and size-caps the
   file (8 MiB) so an oversized log is refused rather than read whole. The trust
   boundary is now stated in the module + function docstrings. Behavior for VALID
   logs is byte-identical (the environment-falsification lane hashes are
   unchanged). New rejection-path tests added.

2. ADR-0198 reconciliation. The §3 verdict-enforcing GATE mechanism
   (VerdictEnforcingEfferentGate + MotorActionIntent) has now landed, so the
   Implementation Status is amended to say so — while being explicit that the §3
   lowering itself (deriving verdicts from the ADR-0029/0033/0036/0037 governance
   packs), the motor decoder (every adapter is decoder=None), and ratification by
   the dedicated motor governance ADR (now drafted as ADR-0216, Proposed) all
   remain deferred. No physical motor decode is authorized.

Disjoint from the GSM8K serving path and from Step D (generate/determine|realize,
chat/runtime, core/config) — clean rebase, no overlap.
2026-06-06 12:54:31 -07:00
Shay
d67c83b794 Add motor verdict enforcing gate 2026-06-06 12:37:57 -07:00