feat(kernel): operationalize ProblemFrame and deprecate legacy parsing
Make #829 kernel substrate the preferred construction path via build_problem_frame, legacy parsing audit, no-new-legacy agent rules, morphology planner v2, and guard tests. No serving score or report changes.
This commit is contained in:
parent
95350276e1
commit
ed379f4982
12 changed files with 1188 additions and 12 deletions
19
AGENTS.md
19
AGENTS.md
|
|
@ -323,6 +323,25 @@ If it touches user input, files, dynamic imports, or logs, what trust boundary w
|
|||
Prefer small, load-bearing PRs. Do not mix baseline fixes, feature work, and
|
||||
large reorganization unless the coupling is unavoidable.
|
||||
|
||||
## Kernel Substrate / No-New-Legacy Rule
|
||||
|
||||
After PR #829, the preferred math comprehension construction path is:
|
||||
|
||||
```text
|
||||
raw problem text → KernelFacts → ProblemFrame → contract-backed derivation organs
|
||||
```
|
||||
|
||||
- Use `generate/problem_frame_builder.py::build_problem_frame` for substrate-backed
|
||||
fact extraction (scalars, units, hazards, process-frame candidates).
|
||||
- New derivation capabilities must consume ProblemFrame facts where the substrate can
|
||||
represent the needed meaning.
|
||||
- New raw-prose/local-regex parsing inside a derivation organ requires an explicit
|
||||
`LEGACY_EXCEPTION` comment and migration rationale.
|
||||
- Guard test: `tests/test_kernel_no_new_legacy_derivation_surfaces.py`.
|
||||
- Audit map: `docs/analysis/kernel-substrate-deprecation-audit-2026-06-18.md`.
|
||||
|
||||
Do not add another isolated benchmark organ with a local prose parser.
|
||||
|
||||
## Architecture in One Sentence
|
||||
|
||||
Raw input becomes a closed versor field once; thought evolves through exact
|
||||
|
|
|
|||
14
CLAUDE.md
14
CLAUDE.md
|
|
@ -123,6 +123,20 @@ runtime path. Vault recall is exact and deterministic.
|
|||
- `calibration/*` — bounded replay-based calibration.
|
||||
- `docs/runtime_contracts.md` — response, telemetry, memory, identity, and testing contracts.
|
||||
|
||||
### Kernel substrate / ProblemFrame (operational path after PR #829)
|
||||
|
||||
New derivation capabilities must consume `KernelFacts` / `ProblemFrame` facts where the
|
||||
substrate can represent the needed meaning (`generate/problem_frame_builder.py`).
|
||||
|
||||
```text
|
||||
raw problem text → KernelFacts → ProblemFrame → contract-backed derivation organs
|
||||
```
|
||||
|
||||
New raw-prose/local-regex parsing inside a derivation organ requires an explicit
|
||||
`LEGACY_EXCEPTION` note and a migration rationale. Guard:
|
||||
`tests/test_kernel_no_new_legacy_derivation_surfaces.py`. Migration map:
|
||||
`docs/analysis/kernel-substrate-deprecation-audit-2026-06-18.md`.
|
||||
|
||||
### GSM8K math comprehension substrate (sealed; serving `7/43/0`, wrong=0 — moves only via ratified PRs)
|
||||
|
||||
- `core/reliability_gate/` — calibrated-learning ledger + gate (ADR-0175): `ClassTally` counts, `conservative_floor` (one-sided Wilson, N_MIN=10), θ ceilings.
|
||||
|
|
|
|||
18
GROK.md
18
GROK.md
|
|
@ -284,6 +284,24 @@ This is not optional. It is the only continuity mechanism across your stateless
|
|||
|
||||
---
|
||||
|
||||
## Kernel Substrate / ProblemFrame Doctrine
|
||||
|
||||
New derivation capabilities must consume `KernelFacts` / `ProblemFrame` facts where the
|
||||
substrate can represent the needed meaning (`generate/problem_frame_builder.py`).
|
||||
|
||||
```text
|
||||
raw problem text → KernelFacts → ProblemFrame → contract-backed derivation organs
|
||||
```
|
||||
|
||||
New raw-prose/local-regex parsing inside a derivation organ requires an explicit
|
||||
`LEGACY_EXCEPTION` note and a migration rationale. Guard:
|
||||
`tests/test_kernel_no_new_legacy_derivation_surfaces.py`.
|
||||
|
||||
Do not add isolated benchmark organs with local prose parsers. Do not treat #829
|
||||
substrate modules as optional helpers.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Summary
|
||||
|
||||
Raw input becomes a closed versor field once; thought evolves through exact versor transitions and CGA recall; cognition is structured as intent, proposition graph, articulation target, deterministic realization, reviewed memory, eval/calibration replay, and traceable evidence.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,186 @@
|
|||
# Kernel Operationalization Implementation (2026-06-18)
|
||||
|
||||
Records the operationalization pass after PR #829 (Kernel Substrate Tranche 1).
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Make the #829 kernel substrate the **preferred operational path**:
|
||||
|
||||
```text
|
||||
raw problem text → KernelFacts → ProblemFrame → contract-backed derivation organs
|
||||
```
|
||||
|
||||
Begin retiring legacy `raw text → local regex organ → one-off answer` habits without
|
||||
breaking serving (`train_sample: 30 / 20 / 0`, `wrong_ids: []`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Relationship to #829
|
||||
|
||||
PR #829 (`58a94c8e`) added:
|
||||
|
||||
- `generate/kernel_facts.py`
|
||||
- `language_packs/scalar_equivalence.py`
|
||||
- `language_packs/unit_dimensions.py`
|
||||
- `language_packs/ambiguity_hazards.py`
|
||||
- `generate/process_frames.py`
|
||||
- `generate/problem_frame.py` (IR skeleton)
|
||||
- `scripts/gsm8k_substrate_morphology.py` (v1 labels)
|
||||
|
||||
This PR **operationalizes** those modules via `build_problem_frame`, adds deprecation
|
||||
audit + no-new-legacy guardrails, and upgrades morphology to planner v2. No serving
|
||||
organ rewrite in this PR.
|
||||
|
||||
---
|
||||
|
||||
## 3. Legacy audit summary
|
||||
|
||||
See `docs/analysis/kernel-substrate-deprecation-audit-2026-06-18.md`.
|
||||
|
||||
Key findings:
|
||||
|
||||
- **Shared debt:** `generate/derivation/extract.py` fans out to most organs.
|
||||
- **Serving path (keep):** `math_candidate_parser.py`, `math_candidate_graph.py`.
|
||||
- **First migration targets:** `percent_partition`, `nested_fraction_remainder_total`, `fraction_decrease`, `temporal_tariff`.
|
||||
- **Recommended first organ:** `percent_partition`.
|
||||
|
||||
---
|
||||
|
||||
## 4. No-new-legacy rule
|
||||
|
||||
Added to agent/architecture guidance:
|
||||
|
||||
> New derivation capabilities must consume KernelFacts / ProblemFrame facts where the
|
||||
> substrate can represent the needed meaning. New raw-prose/local-regex parsing inside a
|
||||
> derivation organ requires an explicit `LEGACY_EXCEPTION` note and a migration rationale.
|
||||
|
||||
**Locations:**
|
||||
|
||||
- `AGENTS.md`
|
||||
- `GROK.md`
|
||||
- `CLAUDE.md`
|
||||
- `docs/architecture/kernel-knowledge-layer-v1.md`
|
||||
- `docs/runtime_contracts.md`
|
||||
|
||||
**Guard test:** `tests/test_kernel_no_new_legacy_derivation_surfaces.py`
|
||||
|
||||
---
|
||||
|
||||
## 5. ProblemFrame builder behavior
|
||||
|
||||
**Module:** `generate/problem_frame_builder.py`
|
||||
|
||||
**Entry point:** `build_problem_frame(problem_text: str) -> ProblemFrame`
|
||||
|
||||
Pipeline:
|
||||
|
||||
1. `extract_scalar_candidates` → scalar facts with exact spans (ordinal contexts like `third place` refused as fractions)
|
||||
2. Unit token scan + `classify_dimension` → `GroundedUnit` with `problem_text` spans
|
||||
3. `ambiguity_hazards` registry scan (+ `%` → percent hazards)
|
||||
4. `process_frames` trigger scan → candidate frames (not conclusions)
|
||||
5. `CandidateRelation` per matched process frame
|
||||
6. Optional `QuestionTarget` from `how many` / `how much` / `?`
|
||||
|
||||
**Non-goals enforced:** no answer derivation, no case-id behavior, no ADR-0128 broadening
|
||||
for unsupported surfaces (`.5`, `1 / 2`).
|
||||
|
||||
**Tests:** `tests/test_problem_frame_builder.py` (8 scenarios from brief)
|
||||
|
||||
---
|
||||
|
||||
## 6. Morphology planner v2 behavior
|
||||
|
||||
**Module:** `scripts/gsm8k_substrate_morphology.py`
|
||||
|
||||
**New API:** `plan_substrate_case(...)`, `recommend_migration_target(...)`
|
||||
|
||||
**CLI:** `--planner` emits v2 records; optional `--verdicts` attaches serving verdicts.
|
||||
|
||||
Output fields:
|
||||
|
||||
| Field | Source |
|
||||
|---|---|
|
||||
| `case_id` | caller / cases JSONL |
|
||||
| `current_verdict` | optional report JSON |
|
||||
| `recognized_scalars` | `build_problem_frame` |
|
||||
| `recognized_units` | `build_problem_frame` |
|
||||
| `recognized_process_frames` | `build_problem_frame` |
|
||||
| `recognized_hazards` | `build_problem_frame` |
|
||||
| `missing_substrate_labels` | `classify_missing_substrate` (v1) |
|
||||
| `legacy_parser_dependency` | heuristic organ/module map |
|
||||
| `recommended_migration_target` | organ or `substrate:*` extension |
|
||||
|
||||
No pack mutation, no sealed artifact analysis, no answer mining, no report rebaseline.
|
||||
|
||||
---
|
||||
|
||||
## 7. First organ migration recommendation
|
||||
|
||||
**Target:** `generate/derivation/percent_partition.py`
|
||||
|
||||
See audit doc § "Recommended first migration: percent_partition".
|
||||
|
||||
---
|
||||
|
||||
## 8. Serving integration status
|
||||
|
||||
| Surface | Status |
|
||||
|---|---|
|
||||
| `build_problem_frame` | **Available** — inspection/diagnostics/tests only |
|
||||
| Morphology planner v2 | **Available** — CLI diagnostics |
|
||||
| Serving answer admission from ProblemFrame | **Not wired** — intentional |
|
||||
| `report.json` | **Unchanged** |
|
||||
| Sealed artifacts | **Unchanged** |
|
||||
| Train/holdout scores | **Unchanged** (validated below) |
|
||||
|
||||
---
|
||||
|
||||
## 9. Validation
|
||||
|
||||
```bash
|
||||
git diff --check origin/main...HEAD
|
||||
pytest tests/test_kernel_facts.py -q
|
||||
pytest tests/test_problem_frame_skeleton.py -q
|
||||
pytest tests/test_problem_frame_builder.py -q
|
||||
pytest tests/test_gsm8k_morphology_missing_kernel_labels.py -q
|
||||
pytest tests/test_kernel_no_new_legacy_derivation_surfaces.py -q
|
||||
pytest tests/test_adr_0128_numeric_formats.py -q
|
||||
pytest tests/test_math_candidate_graph_xhigh_sprint13_lift.py -q
|
||||
pytest tests/test_math_candidate_graph_sprint12_singleton_contract_lift.py -q
|
||||
pytest tests/test_math_candidate_graph_sprint11_cluster_contract_lift.py -q
|
||||
```
|
||||
|
||||
Capability stability (expected `train_sample: 30 20 0`, `wrong_ids: []`):
|
||||
|
||||
```bash
|
||||
uv run python - <<'PY'
|
||||
from evals.gsm8k_math.train_sample.v1.runner import _CASES_PATH, _load_cases, build_report
|
||||
r = build_report(_load_cases(_CASES_PATH))
|
||||
c = r["counts"]
|
||||
print("train_sample:", c["correct"], c["refused"], c["wrong"])
|
||||
print("wrong_ids:", sorted(x["case_id"] for x in r["per_case"] if x["verdict"] == "wrong"))
|
||||
PY
|
||||
```
|
||||
|
||||
Holdout safety (expected `wrong_ids: []`):
|
||||
|
||||
```bash
|
||||
uv run python - <<'PY'
|
||||
from evals.gsm8k_math.holdout_dev.v1.runner import build_report
|
||||
r = build_report()
|
||||
print("wrong_ids:", [x["case_id"] for x in r["per_case"] if x["verdict"] == "wrong"])
|
||||
PY
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Next PR recommendation
|
||||
|
||||
**`feat(kernel): migrate percent_partition to ProblemFrame`**
|
||||
|
||||
- Consume `build_problem_frame` facts inside `percent_partition`
|
||||
- Keep verifier gate and `wrong == 0`
|
||||
- No `report.json` rebaseline unless ratified
|
||||
- Remove redundant local percent regex only after parity tests pass
|
||||
151
docs/analysis/kernel-substrate-deprecation-audit-2026-06-18.md
Normal file
151
docs/analysis/kernel-substrate-deprecation-audit-2026-06-18.md
Normal file
|
|
@ -0,0 +1,151 @@
|
|||
# Kernel Substrate Legacy Deprecation Audit (2026-06-18)
|
||||
|
||||
Migration map for retiring raw-prose / local-regex derivation habits after PR #829
|
||||
(Kernel Substrate Tranche 1). This is not a blame list — it classifies what still
|
||||
serves, what should migrate, and what can be deleted after migration.
|
||||
|
||||
## Audit method
|
||||
|
||||
Searches run on `origin/main` at merge `58a94c8e` plus operationalization branch
|
||||
additions:
|
||||
|
||||
```bash
|
||||
git grep -n "re.compile" generate/derivation generate/math_candidate_graph.py \
|
||||
generate/math_candidate_parser.py generate/math_completeness.py generate/math_roundtrip.py
|
||||
git grep -n "half\|quarter\|third\|percent\|per\|each\|remaining\|altogether" generate/ tests/
|
||||
git grep -n "case_id" generate/ evals/ tests/
|
||||
```
|
||||
|
||||
## Classification legend
|
||||
|
||||
| Class | Meaning |
|
||||
|---|---|
|
||||
| `current_runtime_dependency` | Still on the serving path; do not break during migration |
|
||||
| `migrate_to_problemframe` | Should consume `ProblemFrame` / `KernelFacts` instead of scraping prose |
|
||||
| `wrap_with_substrate_adapter` | Can be fed by substrate facades without full organ rewrite yet |
|
||||
| `delete_after_migration` | One-off helper that should disappear once ProblemFrame path owns the surface |
|
||||
| `allowed_non_derivation_regex` | Harmless test/doc/format regex outside derivation admission |
|
||||
|
||||
---
|
||||
|
||||
## Shared legacy substrate (highest leverage)
|
||||
|
||||
| File | Class | Notes |
|
||||
|---|---|---|
|
||||
| `generate/derivation/extract.py` | `migrate_to_problemframe` | Central quantity/word-qty regex (`_HALF_OF_RE`, `_FRACTION_OF_RE`, `_WORD_QTY_RE`). Every organ importing `extract_quantities` inherits this debt. |
|
||||
| `generate/derivation/clauses.py` | `wrap_with_substrate_adapter` | Sentence split regex only; keep until clause segmentation moves behind a substrate contract. |
|
||||
| `generate/derivation/comparatives.py` | `migrate_to_problemframe` | Local comparative scalar extraction (`half`, `times`, `more than`). |
|
||||
| `generate/math_candidate_parser.py` | `current_runtime_dependency` | 47 regex sites; recognizer/candidate-graph serving path. Migrate incrementally via ProblemFrame inspection, not big-bang delete. |
|
||||
| `generate/math_candidate_graph.py` | `current_runtime_dependency` | Organ dispatch hub; must keep until each organ migrates. |
|
||||
| `generate/math_roundtrip.py` | `current_runtime_dependency` | Tokenization helper used by organs and verifier. |
|
||||
| `generate/math_completeness.py` | `wrap_with_substrate_adapter` | Completeness checks; not a prose parser but coupled to legacy graph shape. |
|
||||
|
||||
---
|
||||
|
||||
## First-wave organ migration candidates
|
||||
|
||||
| Organ | Class | Local parsing debt | ProblemFrame feed |
|
||||
|---|---|---|---|
|
||||
| `generate/derivation/percent_partition.py` | `migrate_to_problemframe` | `_PERCENT_OF_GROUP_RE`, `_FRACTION_RE`, half-split phrase scan | `extract_scalar_candidates`, partition/consumption process frames, percent hazards |
|
||||
| `generate/derivation/nested_fraction_remainder_total.py` | `migrate_to_problemframe` | `_OUTER_HALF_RE`, `_INNER_QUARTER_RE`, morning/afternoon regex | partition frame + grounded scalars + remainder hazards |
|
||||
| `generate/derivation/fraction_decrease.py` | `migrate_to_problemframe` | `_DECREASE_TO_FRACTION_RE`, extra-fraction scan | scalar candidates + consumption/partition frames |
|
||||
| `generate/derivation/temporal_tariff.py` | `migrate_to_problemframe` | hourly rate / threshold shift regex | labor_rate frame + unit dimensions (time/money) |
|
||||
|
||||
All four have train-sample coverage, perform local scalar/phrase parsing today, can be
|
||||
fed by `build_problem_frame`, and can preserve `wrong == 0` when migrated under
|
||||
contract-backed admission.
|
||||
|
||||
### Recommended first migration: `percent_partition`
|
||||
|
||||
**Before**
|
||||
|
||||
```text
|
||||
raw-prose local parsing inside organ
|
||||
→ _PERCENT_OF_GROUP_RE / half-split phrase heuristics
|
||||
→ extract_quantities + comparative_step
|
||||
→ answer derivation
|
||||
```
|
||||
|
||||
**After (next PR)**
|
||||
|
||||
```text
|
||||
build_problem_frame(problem_text)
|
||||
→ grounded scalars (50%, half)
|
||||
→ partition + consumption process-frame candidates
|
||||
→ percent / remainder hazards preserved
|
||||
→ explicit percent_partition contract consumes ProblemFrame facts
|
||||
→ answer derivation (unchanged verifier gate)
|
||||
```
|
||||
|
||||
**Why first:** smallest surface area among the four candidates, clear scalar/frame
|
||||
coverage in substrate, existing train coverage, and morphology planner v2 already
|
||||
routes half+percent split problems to this organ.
|
||||
|
||||
---
|
||||
|
||||
## Other derivation organs (deferred)
|
||||
|
||||
| File | Class | Notes |
|
||||
|---|---|---|
|
||||
| `generate/derivation/affine_fraction_delta.py` | `migrate_to_problemframe` | Fraction decrease/increase affine cues |
|
||||
| `generate/derivation/piecewise_daily_hours_total.py` | `migrate_to_problemframe` | Calendar + halfway-month local tables |
|
||||
| `generate/derivation/loose_crayon_box_capacity.py` | `migrate_to_problemframe` | Container/box regex family |
|
||||
| `generate/derivation/r1_reconstruction.py` | `current_runtime_dependency` | Broad fact-reconstruction regex suite |
|
||||
| `generate/derivation/sequential_comparative_scale.py` | `migrate_to_problemframe` | Reader/scale regex |
|
||||
| `generate/derivation/round_trip_trip_duration.py` | `migrate_to_problemframe` | Travel fraction cues |
|
||||
| `generate/derivation/bounded_rate_projection.py` | `migrate_to_problemframe` | Percent projection regex |
|
||||
| `generate/derivation/calendar_grounding.py` | `wrap_with_substrate_adapter` | Month-name regex pending `kernel_calendar` pack |
|
||||
| `generate/derivation/state/bind.py` | `delete_after_migration` | Word token regex subsumed by substrate spans |
|
||||
| `generate/derivation/state/source.py` | `delete_after_migration` | Conjunction split regex |
|
||||
|
||||
---
|
||||
|
||||
## New substrate path (target design)
|
||||
|
||||
| File | Class | Notes |
|
||||
|---|---|---|
|
||||
| `generate/problem_frame_builder.py` | **target path** | `build_problem_frame` — substrate-backed, no solving |
|
||||
| `language_packs/scalar_equivalence.py` | **target path** | ADR-0128 facade; `extract_scalar_candidates` |
|
||||
| `language_packs/unit_dimensions.py` | **target path** | ADR-0127 facade |
|
||||
| `language_packs/ambiguity_hazards.py` | **target path** | Hazard registry |
|
||||
| `generate/process_frames.py` | **target path** | Declarative process-frame candidates |
|
||||
| `generate/kernel_facts.py` | **target path** | Typed fact/provenance model |
|
||||
| `generate/problem_frame.py` | **target path** | ProblemFrame IR skeleton |
|
||||
| `scripts/gsm8k_substrate_morphology.py` | **target path** | Planner v2 diagnostics |
|
||||
|
||||
---
|
||||
|
||||
## Phrase-surface grep summary (`generate/`)
|
||||
|
||||
Surfaces like `half`, `quarter`, `percent`, `remaining` appear in:
|
||||
|
||||
- **Substrate (keep):** `scalar_equivalence.py`, `ambiguity_hazards.py`, `process_frames.py`, `problem_frame_builder.py`
|
||||
- **Legacy organs (migrate):** `derivation/extract.py`, `percent_partition.py`, `nested_fraction_remainder_total.py`, `fraction_decrease.py`, `comparatives.py`, and sprint lift organs
|
||||
- **Serving graph (keep temporarily):** `math_candidate_graph.py`, `math_candidate_parser.py`
|
||||
- **Non-derivation (allowed):** `process_frames.py` trigger tables, tests, docs
|
||||
|
||||
---
|
||||
|
||||
## `case_id` usage
|
||||
|
||||
| Area | Class | Notes |
|
||||
|---|---|---|
|
||||
| `evals/gsm8k_math/**` | `allowed_non_derivation_regex` | Case identifiers for measurement harnesses only |
|
||||
| `tests/test_math_candidate_graph_*` | `allowed_non_derivation_regex` | Fixture pinning by case_id |
|
||||
| `generate/derivation/**` | **none found** | Organs do not hardcode case_id (good) |
|
||||
| `generate/problem_frame_builder.py` | **none** | Builder explicitly case-agnostic |
|
||||
|
||||
---
|
||||
|
||||
## Guardrails added in this PR
|
||||
|
||||
- **No-new-legacy rule** in `AGENTS.md`, `GROK.md`, `CLAUDE.md`, `docs/architecture/kernel-knowledge-layer-v1.md`, `docs/runtime_contracts.md`
|
||||
- **`tests/test_kernel_no_new_legacy_derivation_surfaces.py`** — allowlisted legacy regex files; new regex in derivation paths requires `LEGACY_EXCEPTION`
|
||||
|
||||
---
|
||||
|
||||
## Next PR recommendation
|
||||
|
||||
Migrate **`percent_partition`** to consume `ProblemFrame` facts while keeping the
|
||||
existing verifier gate and `wrong == 0` invariant. Do not change `report.json` or
|
||||
sealed artifacts in the migration PR unless a ratified score change is intended.
|
||||
|
|
@ -75,6 +75,17 @@ To prevent architectural drift and maintain a clean separation of concerns, the
|
|||
2. **Dimension Tables:** Standard conversion coefficients (e.g., $60 \text{ minutes} = 1 \text{ hour}$) must reside in the units pack conversions, not in organ-level multiplication constants.
|
||||
3. **Calendar Metrics:** Day-counts and calendar logic must be resolved strictly through the calendar pack.
|
||||
|
||||
### No-new-legacy rule (enforced after operationalization PR)
|
||||
|
||||
New derivation capabilities must consume `KernelFacts` / `ProblemFrame` facts where the
|
||||
substrate can represent the needed meaning. Construction entry point:
|
||||
`generate/problem_frame_builder.py::build_problem_frame`.
|
||||
|
||||
New raw-prose/local-regex parsing inside a derivation organ requires an explicit
|
||||
`LEGACY_EXCEPTION` comment and migration rationale. CI guard:
|
||||
`tests/test_kernel_no_new_legacy_derivation_surfaces.py`. Legacy inventory:
|
||||
`docs/analysis/kernel-substrate-deprecation-audit-2026-06-18.md`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Kernel Pack Model
|
||||
|
|
|
|||
|
|
@ -812,6 +812,28 @@ from ADR-0119.5 and ADR-0114a Obligation #8:
|
|||
The `subtle_in_grammar` family (4 cases, all `correct`) proves the gate
|
||||
is not trivially satisfied by refusing everything.
|
||||
|
||||
## Kernel substrate / ProblemFrame construction contract
|
||||
|
||||
Diagnostics and future organ migrations may construct a `ProblemFrame` from raw
|
||||
problem text via `generate/problem_frame_builder.py::build_problem_frame`.
|
||||
|
||||
Required properties:
|
||||
|
||||
- Deterministic ordering of scalars, units, hazards, and process-frame candidates.
|
||||
- Exact `problem_text` source spans where grounding is available.
|
||||
- Hazards preserved from `language_packs.ambiguity_hazards` and scalar candidates.
|
||||
- Process frames attached as **candidates**, not conclusions.
|
||||
- No answer derivation, no case-id behavior, no serving admission from ProblemFrame alone.
|
||||
|
||||
No-new-legacy rule: new derivation capabilities must consume ProblemFrame facts where
|
||||
the substrate can represent the needed meaning. New raw-prose/local-regex parsing
|
||||
inside a derivation organ requires an explicit `LEGACY_EXCEPTION` note and migration
|
||||
rationale (`tests/test_kernel_no_new_legacy_derivation_surfaces.py`).
|
||||
|
||||
Serving policy (current): ProblemFrame construction is inspection/diagnostic only.
|
||||
Changing `train_sample` / holdout scores via ProblemFrame requires a ratified organ
|
||||
migration PR with verifier parity evidence.
|
||||
|
||||
### Evidence location
|
||||
|
||||
- Lane runner: `evals/gsm8k_math/runner.py`
|
||||
|
|
|
|||
346
generate/problem_frame_builder.py
Normal file
346
generate/problem_frame_builder.py
Normal file
|
|
@ -0,0 +1,346 @@
|
|||
"""ProblemFrame builder — substrate-backed construction from raw problem text.
|
||||
|
||||
Operationalizes the #829 kernel substrate path:
|
||||
|
||||
raw text → scalar/unit/hazard/process-frame facts → ProblemFrame
|
||||
|
||||
Non-goals:
|
||||
- answer derivation
|
||||
- case-id behavior
|
||||
- serving admission
|
||||
- guessing unsupported or ambiguous surfaces
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from fractions import Fraction
|
||||
|
||||
from generate.kernel_facts import (
|
||||
CandidateRelation,
|
||||
GroundedScalar,
|
||||
GroundedUnit,
|
||||
KernelHazard,
|
||||
KernelProvenance,
|
||||
RelationRole,
|
||||
SourceSpan,
|
||||
)
|
||||
from generate.problem_frame import ProblemFrame, ProblemFrameBuilder, QuestionTarget
|
||||
from generate.process_frames import ProcessFrame, all_frames, lookup_frame
|
||||
from language_packs.ambiguity_hazards import (
|
||||
AmbiguityHazard,
|
||||
all_registered_surfaces,
|
||||
lookup_hazards,
|
||||
)
|
||||
from language_packs.scalar_equivalence import (
|
||||
ScalarCandidate,
|
||||
extract_scalar_candidates,
|
||||
list_unsupported_surfaces,
|
||||
)
|
||||
from language_packs.unit_dimensions import classify_dimension
|
||||
|
||||
_UNIT_TOKEN_RE: re.Pattern[str] = re.compile(r"\b\d+(?:\.\d+)?\s+([a-zA-Z]+)\b")
|
||||
|
||||
_UNIT_STOPWORDS: frozenset[str] = frozenset({
|
||||
"more", "less", "times", "percent", "percentage", "of", "and", "or",
|
||||
"the", "a", "an", "in", "to", "for", "with", "at", "by", "from",
|
||||
"each", "per", "way", "ways",
|
||||
})
|
||||
|
||||
_ORDINAL_SUFFIX_RE: re.Pattern[str] = re.compile(
|
||||
r"\b(half|third|quarter)\s+(place|position|grade|rank)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def _surface_in_text(surface: str, text_lower: str) -> bool:
|
||||
token = surface.lower()
|
||||
padded = f" {text_lower} "
|
||||
return (
|
||||
f" {token} " in padded
|
||||
or text_lower.startswith(f"{token} ")
|
||||
or text_lower.endswith(f" {token}")
|
||||
or text_lower == token
|
||||
)
|
||||
|
||||
|
||||
def _hazard_to_kernel(hazard: AmbiguityHazard) -> KernelHazard:
|
||||
return KernelHazard(
|
||||
hazard_id=hazard.hazard_id,
|
||||
category=hazard.category,
|
||||
surface=hazard.surface,
|
||||
description=hazard.description,
|
||||
context_required=hazard.context_required,
|
||||
)
|
||||
|
||||
|
||||
def _extract_unit_candidates(text: str) -> tuple[GroundedUnit, ...]:
|
||||
units: list[GroundedUnit] = []
|
||||
seen: set[tuple[str, int, int]] = set()
|
||||
|
||||
for match in _UNIT_TOKEN_RE.finditer(text):
|
||||
token = match.group(1)
|
||||
token_lower = token.lower()
|
||||
if token_lower in _UNIT_STOPWORDS:
|
||||
continue
|
||||
dim_fact = classify_dimension(token_lower)
|
||||
if dim_fact is None:
|
||||
continue
|
||||
start = match.start(1)
|
||||
end = match.end(1)
|
||||
key = (token_lower, start, end)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
span = SourceSpan(text[start:end], start, end)
|
||||
provenance = KernelProvenance(kind="problem_text", source_spans=(span,))
|
||||
units.append(
|
||||
GroundedUnit(
|
||||
fact_id=f"unit-{len(units):04d}",
|
||||
surface=token_lower,
|
||||
dimension=dim_fact.dimension,
|
||||
singular=dim_fact.singular,
|
||||
provenance=provenance,
|
||||
)
|
||||
)
|
||||
|
||||
return tuple(sorted(units, key=lambda u: (u.provenance.source_spans[0].start, u.surface)))
|
||||
|
||||
|
||||
def _extract_hazards(text: str) -> tuple[KernelHazard, ...]:
|
||||
text_lower = text.lower()
|
||||
hazards: list[KernelHazard] = []
|
||||
seen: set[str] = set()
|
||||
|
||||
for surface in all_registered_surfaces():
|
||||
if not _surface_in_text(surface, text_lower):
|
||||
continue
|
||||
for hazard in lookup_hazards(surface):
|
||||
if hazard.hazard_id in seen:
|
||||
continue
|
||||
seen.add(hazard.hazard_id)
|
||||
hazards.append(_hazard_to_kernel(hazard))
|
||||
|
||||
if "%" in text:
|
||||
for hazard in lookup_hazards("percent"):
|
||||
if hazard.hazard_id in seen:
|
||||
continue
|
||||
seen.add(hazard.hazard_id)
|
||||
hazards.append(_hazard_to_kernel(hazard))
|
||||
|
||||
return tuple(sorted(hazards, key=lambda h: h.hazard_id))
|
||||
|
||||
|
||||
def _is_ordinal_scalar_span(text: str, start: int, end: int) -> bool:
|
||||
"""Refuse fraction readings for ordinals like ``third place``."""
|
||||
window_start = max(0, start - 20)
|
||||
window_end = min(len(text), end + 20)
|
||||
window = text[window_start:window_end]
|
||||
for match in _ORDINAL_SUFFIX_RE.finditer(window):
|
||||
abs_start = window_start + match.start()
|
||||
abs_end = window_start + match.end()
|
||||
if start >= abs_start and end <= abs_end:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _filter_scalar_candidates(
|
||||
text: str,
|
||||
candidates: tuple[ScalarCandidate, ...],
|
||||
) -> tuple[ScalarCandidate, ...]:
|
||||
kept: list[ScalarCandidate] = []
|
||||
for candidate in candidates:
|
||||
if candidate.source_span is None:
|
||||
kept.append(candidate)
|
||||
continue
|
||||
start, end = candidate.source_span
|
||||
if _is_ordinal_scalar_span(text, start, end):
|
||||
continue
|
||||
kept.append(candidate)
|
||||
return tuple(kept)
|
||||
|
||||
|
||||
def _trigger_span(text: str, trigger: str) -> SourceSpan | None:
|
||||
text_lower = text.lower()
|
||||
trigger_lower = trigger.lower()
|
||||
idx = text_lower.find(trigger_lower)
|
||||
if idx < 0:
|
||||
return None
|
||||
return SourceSpan(text[idx:idx + len(trigger_lower)], idx, idx + len(trigger_lower))
|
||||
|
||||
|
||||
def _extract_process_frame_candidates(text: str) -> tuple[ProcessFrame, ...]:
|
||||
text_lower = text.lower()
|
||||
matched: dict[str, ProcessFrame] = {}
|
||||
|
||||
for frame in all_frames():
|
||||
for trigger in frame.trigger_surfaces:
|
||||
if _surface_in_text(trigger, text_lower):
|
||||
matched[frame.name] = frame
|
||||
break
|
||||
|
||||
return tuple(matched[name] for name in sorted(matched))
|
||||
|
||||
|
||||
def _frame_roles(frame: ProcessFrame) -> tuple[RelationRole, ...]:
|
||||
roles: list[RelationRole] = []
|
||||
for role in frame.required_roles:
|
||||
roles.append(RelationRole(role.name, True, role.description))
|
||||
for role in frame.optional_roles:
|
||||
roles.append(RelationRole(role.name, False, role.description))
|
||||
return tuple(roles)
|
||||
|
||||
|
||||
def _extract_candidate_relations(
|
||||
text: str,
|
||||
frames: tuple[ProcessFrame, ...],
|
||||
) -> tuple[CandidateRelation, ...]:
|
||||
relations: list[CandidateRelation] = []
|
||||
|
||||
for frame in frames:
|
||||
span: SourceSpan | None = None
|
||||
for trigger in frame.trigger_surfaces:
|
||||
span = _trigger_span(text, trigger)
|
||||
if span is not None:
|
||||
break
|
||||
provenance = (
|
||||
KernelProvenance(kind="problem_text", source_spans=(span,))
|
||||
if span is not None
|
||||
else None
|
||||
)
|
||||
frame_hazards = tuple(
|
||||
KernelHazard(
|
||||
hazard_id=f"frame-{frame.name}-{category}",
|
||||
category=category,
|
||||
surface=frame.name,
|
||||
description=f"Process frame {frame.name} hazard {category}",
|
||||
)
|
||||
for category in frame.hazards
|
||||
)
|
||||
relations.append(
|
||||
CandidateRelation(
|
||||
relation_id=f"rel-{frame.name}",
|
||||
relation_type=frame.candidate_relation,
|
||||
roles=_frame_roles(frame),
|
||||
provenance=provenance,
|
||||
hazards=frame_hazards,
|
||||
)
|
||||
)
|
||||
|
||||
return tuple(relations)
|
||||
|
||||
|
||||
def _scalar_to_grounded(
|
||||
candidate: ScalarCandidate,
|
||||
text: str,
|
||||
index: int,
|
||||
) -> GroundedScalar | None:
|
||||
if candidate.source_span is None or candidate.source_surface is None:
|
||||
return None
|
||||
|
||||
start, end = candidate.source_span
|
||||
span = SourceSpan(candidate.source_surface, start, end)
|
||||
provenance = KernelProvenance(kind="problem_text", source_spans=(span,))
|
||||
hazards = tuple(
|
||||
KernelHazard(
|
||||
hazard_id=hid,
|
||||
category=hid,
|
||||
surface=candidate.surface,
|
||||
description=f"Scalar hazard {hid}",
|
||||
)
|
||||
for hid in candidate.hazards
|
||||
)
|
||||
return GroundedScalar(
|
||||
fact_id=f"scalar-{index:04d}",
|
||||
surface=candidate.surface,
|
||||
value=candidate.canonical,
|
||||
provenance=provenance,
|
||||
hazards=hazards,
|
||||
)
|
||||
|
||||
|
||||
def _detect_question_target(text: str) -> QuestionTarget | None:
|
||||
text_lower = text.lower()
|
||||
if "how many" in text_lower:
|
||||
return QuestionTarget("how many", "count")
|
||||
if "how much" in text_lower:
|
||||
return QuestionTarget("how much", "quantity")
|
||||
if "?" in text:
|
||||
return QuestionTarget("?", "unknown")
|
||||
return None
|
||||
|
||||
|
||||
def _has_unsupported_scalar_surface(text: str) -> bool:
|
||||
for surface in list_unsupported_surfaces():
|
||||
if surface in text:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def build_problem_frame(problem_text: str) -> ProblemFrame:
|
||||
"""Build a substrate-backed ProblemFrame from raw problem text.
|
||||
|
||||
Deterministic ordering; preserves hazards and provenance; does not derive
|
||||
answers or bind case-specific behavior.
|
||||
"""
|
||||
builder = ProblemFrameBuilder()
|
||||
|
||||
scalars = _filter_scalar_candidates(problem_text, extract_scalar_candidates(problem_text))
|
||||
for scalar in scalars:
|
||||
builder.add_scalar(scalar)
|
||||
|
||||
for index, scalar in enumerate(scalars):
|
||||
grounded = _scalar_to_grounded(scalar, problem_text, index)
|
||||
if grounded is not None:
|
||||
builder.add_quantity(grounded)
|
||||
|
||||
for unit in _extract_unit_candidates(problem_text):
|
||||
builder.add_unit(unit)
|
||||
|
||||
for hazard in _extract_hazards(problem_text):
|
||||
builder.add_hazard(hazard)
|
||||
|
||||
frames = _extract_process_frame_candidates(problem_text)
|
||||
for frame in frames:
|
||||
builder.add_process_frame(frame)
|
||||
|
||||
for relation in _extract_candidate_relations(problem_text, frames):
|
||||
builder.add_relation(relation)
|
||||
|
||||
question_target = _detect_question_target(problem_text)
|
||||
if question_target is not None:
|
||||
builder.set_question_target(question_target)
|
||||
|
||||
# Unsupported scalar tokenisations remain absent from scalars; callers can
|
||||
# consult list_unsupported_surfaces() — we do not broaden ADR-0128 here.
|
||||
_ = _has_unsupported_scalar_surface(problem_text)
|
||||
|
||||
return builder.build()
|
||||
|
||||
|
||||
def recognized_scalar_surfaces(frame: ProblemFrame) -> tuple[str, ...]:
|
||||
"""Return sorted scalar surfaces recognized in a ProblemFrame."""
|
||||
surfaces = {s.surface for s in frame.scalars}
|
||||
surfaces.update(q.surface for q in frame.quantities)
|
||||
return tuple(sorted(surfaces))
|
||||
|
||||
|
||||
def recognized_unit_surfaces(frame: ProblemFrame) -> tuple[str, ...]:
|
||||
"""Return sorted unit surfaces recognized in a ProblemFrame."""
|
||||
return tuple(sorted({u.surface for u in frame.units}))
|
||||
|
||||
|
||||
def recognized_process_frame_names(frame: ProblemFrame) -> tuple[str, ...]:
|
||||
"""Return sorted process-frame names attached as candidates."""
|
||||
return tuple(sorted({f.name for f in frame.process_frames}))
|
||||
|
||||
|
||||
def recognized_hazard_ids(frame: ProblemFrame) -> tuple[str, ...]:
|
||||
"""Return sorted hazard IDs preserved on the frame."""
|
||||
return tuple(sorted({h.hazard_id for h in frame.hazards}))
|
||||
|
||||
|
||||
def scalar_canonical_values(frame: ProblemFrame) -> tuple[Fraction, ...]:
|
||||
"""Return canonical scalar values in deterministic order."""
|
||||
values = [s.canonical for s in frame.scalars]
|
||||
values.extend(q.value for q in frame.quantities)
|
||||
return tuple(sorted(values, key=lambda v: (v.denominator, v.numerator)))
|
||||
|
|
@ -1,7 +1,9 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Classify GSM8K problems by missing substrate category.
|
||||
"""Classify GSM8K problems by missing substrate category and plan migrations.
|
||||
|
||||
Tranche 1 — broad base-layer foundations.
|
||||
Planner v2 — operationalization pass: recognize substrate facts and recommend
|
||||
legacy-parser migration targets without answer mining or pack mutation.
|
||||
|
||||
Labels are semantically honest: ``missing_*`` categories fire only when a
|
||||
needed substrate lookup actually fails, not merely because a trigger
|
||||
|
|
@ -13,8 +15,15 @@ import argparse
|
|||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Sequence
|
||||
from typing import Any, Sequence
|
||||
|
||||
from generate.problem_frame_builder import (
|
||||
build_problem_frame,
|
||||
recognized_hazard_ids,
|
||||
recognized_process_frame_names,
|
||||
recognized_scalar_surfaces,
|
||||
recognized_unit_surfaces,
|
||||
)
|
||||
from language_packs.scalar_equivalence import list_unsupported_surfaces
|
||||
from language_packs.unit_dimensions import classify_dimension
|
||||
from language_packs.loader import lookup_container
|
||||
|
|
@ -157,11 +166,141 @@ def classify_missing_substrate(problem_text: str) -> tuple[str, ...]:
|
|||
return tuple(sorted(labels))
|
||||
|
||||
|
||||
_FIRST_MIGRATION_ORGANS: tuple[str, ...] = (
|
||||
"percent_partition",
|
||||
"nested_fraction_remainder_total",
|
||||
"fraction_decrease",
|
||||
"temporal_tariff",
|
||||
)
|
||||
|
||||
_ORGAN_MODULE_PATHS: dict[str, str] = {
|
||||
"percent_partition": "generate/derivation/percent_partition.py",
|
||||
"nested_fraction_remainder_total": "generate/derivation/nested_fraction_remainder_total.py",
|
||||
"fraction_decrease": "generate/derivation/fraction_decrease.py",
|
||||
"temporal_tariff": "generate/derivation/temporal_tariff.py",
|
||||
"extract_shared": "generate/derivation/extract.py",
|
||||
"math_candidate_parser": "generate/math_candidate_parser.py",
|
||||
}
|
||||
|
||||
|
||||
def _legacy_parser_dependency(
|
||||
problem_text: str,
|
||||
process_frames: tuple[str, ...],
|
||||
missing_labels: tuple[str, ...],
|
||||
) -> tuple[str, ...]:
|
||||
"""Map problem surfaces to currently-serving legacy parser modules."""
|
||||
deps: set[str] = set()
|
||||
lowered = problem_text.lower()
|
||||
|
||||
if "%" in problem_text or "percent" in lowered:
|
||||
deps.add(_ORGAN_MODULE_PATHS["percent_partition"])
|
||||
if "other half" in lowered:
|
||||
deps.add(_ORGAN_MODULE_PATHS["percent_partition"])
|
||||
if "remaining" in lowered and ("half" in lowered or "quarter" in lowered):
|
||||
deps.add(_ORGAN_MODULE_PATHS["nested_fraction_remainder_total"])
|
||||
if any(word in lowered for word in ("decrease", "decreased", "decreases")):
|
||||
deps.add(_ORGAN_MODULE_PATHS["fraction_decrease"])
|
||||
if any(
|
||||
token in lowered
|
||||
for token in ("hour", "hours", "per hour", "overtime", "threshold")
|
||||
):
|
||||
deps.add(_ORGAN_MODULE_PATHS["temporal_tariff"])
|
||||
if "labor_rate" in process_frames:
|
||||
deps.add(_ORGAN_MODULE_PATHS["temporal_tariff"])
|
||||
|
||||
if re.search(r"\d", problem_text):
|
||||
deps.add(_ORGAN_MODULE_PATHS["extract_shared"])
|
||||
if "missing_scalar_equivalence" in missing_labels:
|
||||
deps.add(_ORGAN_MODULE_PATHS["math_candidate_parser"])
|
||||
|
||||
return tuple(sorted(deps))
|
||||
|
||||
|
||||
def recommend_migration_target(
|
||||
problem_text: str,
|
||||
process_frames: tuple[str, ...],
|
||||
missing_labels: tuple[str, ...],
|
||||
) -> str:
|
||||
"""Recommend the next organ or substrate extension for this problem."""
|
||||
lowered = problem_text.lower()
|
||||
|
||||
if "%" in problem_text and ("half" in lowered or "partition" in process_frames):
|
||||
return "percent_partition"
|
||||
if "other half" in lowered and "%" in problem_text:
|
||||
return "percent_partition"
|
||||
|
||||
if "missing_scalar_equivalence" in missing_labels:
|
||||
return "substrate:scalar_equivalence"
|
||||
if "missing_unit_dimension" in missing_labels:
|
||||
return "substrate:unit_dimensions"
|
||||
if "blocked_provenance_gap" in missing_labels:
|
||||
return "substrate:kernel_calendar"
|
||||
if "remaining" in lowered and ("half" in lowered or "quarter" in lowered):
|
||||
return "nested_fraction_remainder_total"
|
||||
if any(word in lowered for word in ("decrease", "decreased")):
|
||||
return "fraction_decrease"
|
||||
if "labor_rate" in process_frames or any(
|
||||
token in lowered for token in ("per hour", "hourly", "overtime")
|
||||
):
|
||||
return "temporal_tariff"
|
||||
if "blocked_ambiguity_hazard" in missing_labels:
|
||||
return "substrate:ambiguity_hazards"
|
||||
|
||||
if process_frames:
|
||||
return process_frames[0]
|
||||
|
||||
return "substrate:problem_frame_builder"
|
||||
|
||||
|
||||
def plan_substrate_case(
|
||||
*,
|
||||
case_id: str,
|
||||
problem_text: str,
|
||||
current_verdict: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Planner v2 record for one problem — diagnostics only, no solving."""
|
||||
frame = build_problem_frame(problem_text)
|
||||
missing_labels = classify_missing_substrate(problem_text)
|
||||
process_frames = recognized_process_frame_names(frame)
|
||||
|
||||
return {
|
||||
"case_id": case_id,
|
||||
"current_verdict": current_verdict,
|
||||
"recognized_scalars": recognized_scalar_surfaces(frame),
|
||||
"recognized_units": recognized_unit_surfaces(frame),
|
||||
"recognized_process_frames": process_frames,
|
||||
"recognized_hazards": recognized_hazard_ids(frame),
|
||||
"missing_substrate_labels": missing_labels,
|
||||
"legacy_parser_dependency": _legacy_parser_dependency(
|
||||
problem_text,
|
||||
process_frames,
|
||||
missing_labels,
|
||||
),
|
||||
"recommended_migration_target": recommend_migration_target(
|
||||
problem_text,
|
||||
process_frames,
|
||||
missing_labels,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Classify GSM8K problems by missing substrate.")
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Classify GSM8K problems by missing substrate and plan migrations.",
|
||||
)
|
||||
parser.add_argument("--cases", type=str, help="Path to JSONL cases file")
|
||||
parser.add_argument("--out", type=str, help="Path to write classified output JSONL")
|
||||
parser.add_argument("--limit", type=int, help="Limit number of cases to process")
|
||||
parser.add_argument(
|
||||
"--planner",
|
||||
action="store_true",
|
||||
help="Emit morphology planner v2 records (recognized substrate + migration targets)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--verdicts",
|
||||
type=str,
|
||||
help="Optional JSON report with per_case verdicts keyed by case_id",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
|
|
@ -174,7 +313,18 @@ def main() -> None:
|
|||
print(f"Cases file not found at {args.cases}")
|
||||
return
|
||||
|
||||
out_lines = []
|
||||
verdicts: dict[str, str] = {}
|
||||
if args.verdicts:
|
||||
report_path = Path(args.verdicts)
|
||||
if report_path.exists():
|
||||
report = json.loads(report_path.read_text(encoding="utf-8"))
|
||||
for row in report.get("per_case", []):
|
||||
cid = row.get("case_id")
|
||||
verdict = row.get("verdict")
|
||||
if cid and verdict:
|
||||
verdicts[cid] = verdict
|
||||
|
||||
out_lines: list[dict[str, Any]] = []
|
||||
count = 0
|
||||
with cases_path.open("r", encoding="utf-8") as f:
|
||||
for line in f:
|
||||
|
|
@ -185,13 +335,21 @@ def main() -> None:
|
|||
if not problem_text:
|
||||
continue
|
||||
|
||||
labels = classify_missing_substrate(problem_text)
|
||||
case_id = case.get("case_id") or f"case_{count}"
|
||||
out_lines.append({
|
||||
"case_id": case_id,
|
||||
"problem_text": problem_text,
|
||||
"missing_substrate_labels": labels
|
||||
})
|
||||
if args.planner:
|
||||
record = plan_substrate_case(
|
||||
case_id=case_id,
|
||||
problem_text=problem_text,
|
||||
current_verdict=verdicts.get(case_id),
|
||||
)
|
||||
else:
|
||||
labels = classify_missing_substrate(problem_text)
|
||||
record = {
|
||||
"case_id": case_id,
|
||||
"problem_text": problem_text,
|
||||
"missing_substrate_labels": labels,
|
||||
}
|
||||
out_lines.append(record)
|
||||
|
||||
count += 1
|
||||
if args.limit and count >= args.limit:
|
||||
|
|
|
|||
|
|
@ -1,7 +1,11 @@
|
|||
"""Tests for scripts/gsm8k_substrate_morphology.py."""
|
||||
from __future__ import annotations
|
||||
|
||||
from scripts.gsm8k_substrate_morphology import classify_missing_substrate
|
||||
from scripts.gsm8k_substrate_morphology import (
|
||||
classify_missing_substrate,
|
||||
plan_substrate_case,
|
||||
recommend_migration_target,
|
||||
)
|
||||
|
||||
|
||||
def test_classify_missing_substrate_labels() -> None:
|
||||
|
|
@ -69,4 +73,40 @@ def test_deterministic_and_sorted() -> None:
|
|||
assert "missing_unit_dimension" in labels1
|
||||
assert "missing_part_whole_frame" not in labels1
|
||||
assert "missing_container_frame" not in labels1
|
||||
assert "blocked_provenance_gap" in labels1
|
||||
assert "blocked_provenance_gap" in labels1
|
||||
|
||||
|
||||
def test_planner_v2_recognizes_substrate_without_solving() -> None:
|
||||
record = plan_substrate_case(
|
||||
case_id="test-0001",
|
||||
problem_text="Mia spent 50% of her money.",
|
||||
current_verdict="refused",
|
||||
)
|
||||
|
||||
assert record["case_id"] == "test-0001"
|
||||
assert record["current_verdict"] == "refused"
|
||||
assert record["recognized_scalars"]
|
||||
assert "consumption" in record["recognized_process_frames"]
|
||||
assert record["recognized_hazards"]
|
||||
assert isinstance(record["legacy_parser_dependency"], tuple)
|
||||
assert record["recommended_migration_target"] in {
|
||||
"percent_partition",
|
||||
"substrate:scalar_equivalence",
|
||||
"substrate:ambiguity_hazards",
|
||||
"consumption",
|
||||
"substrate:problem_frame_builder",
|
||||
}
|
||||
|
||||
|
||||
def test_planner_v2_recommends_percent_partition_for_half_percent_split() -> None:
|
||||
text = (
|
||||
"There are 100 students. Half are girls and the other half are boys. "
|
||||
"30% of the girls own pets and 20% of the boys own pets. "
|
||||
"How many students own pets?"
|
||||
)
|
||||
target = recommend_migration_target(
|
||||
text,
|
||||
("partition", "consumption"),
|
||||
classify_missing_substrate(text),
|
||||
)
|
||||
assert target == "percent_partition"
|
||||
106
tests/test_kernel_no_new_legacy_derivation_surfaces.py
Normal file
106
tests/test_kernel_no_new_legacy_derivation_surfaces.py
Normal file
|
|
@ -0,0 +1,106 @@
|
|||
"""Guard against new legacy raw-text parsing in derivation paths."""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
# Current serving legacy surfaces — explicit allowlist while migrations proceed.
|
||||
ALLOWLISTED_LEGACY_DERIVATION_FILES: frozenset[str] = frozenset({
|
||||
"generate/derivation/affine_comparative_inversion_total.py",
|
||||
"generate/derivation/affine_fraction_delta.py",
|
||||
"generate/derivation/bounded_rate_projection.py",
|
||||
"generate/derivation/calendar_grounding.py",
|
||||
"generate/derivation/clauses.py",
|
||||
"generate/derivation/closed_reference_affine_aggregate.py",
|
||||
"generate/derivation/comparatives.py",
|
||||
"generate/derivation/compose.py",
|
||||
"generate/derivation/duration_segment_total.py",
|
||||
"generate/derivation/extract.py",
|
||||
"generate/derivation/fraction_decrease.py",
|
||||
"generate/derivation/giveaway_target_residual.py",
|
||||
"generate/derivation/goal_residual.py",
|
||||
"generate/derivation/loose_crayon_box_capacity.py",
|
||||
"generate/derivation/multistep.py",
|
||||
"generate/derivation/nested_fraction_remainder_total.py",
|
||||
"generate/derivation/percent_partition.py",
|
||||
"generate/derivation/piecewise_daily_hours_total.py",
|
||||
"generate/derivation/product_bridge.py",
|
||||
"generate/derivation/question_bound_product.py",
|
||||
"generate/derivation/r1_reconstruction.py",
|
||||
"generate/derivation/round_trip_trip_duration.py",
|
||||
"generate/derivation/search.py",
|
||||
"generate/derivation/sequential_comparative_scale.py",
|
||||
"generate/derivation/state/bind.py",
|
||||
"generate/derivation/state/source.py",
|
||||
"generate/derivation/survey_rate_earnings.py",
|
||||
"generate/derivation/target.py",
|
||||
"generate/derivation/temporal_tariff.py",
|
||||
"generate/math_candidate_parser.py",
|
||||
"generate/math_candidate_graph.py",
|
||||
"generate/math_completeness.py",
|
||||
"generate/math_roundtrip.py",
|
||||
})
|
||||
|
||||
_REGEX_USAGE = re.compile(
|
||||
r"\bre\.(?:compile|search|match|findall|finditer|sub|split)\b"
|
||||
)
|
||||
|
||||
_SCAN_ROOTS = (
|
||||
_REPO_ROOT / "generate" / "derivation",
|
||||
_REPO_ROOT / "generate" / "math_candidate_parser.py",
|
||||
_REPO_ROOT / "generate" / "math_candidate_graph.py",
|
||||
_REPO_ROOT / "generate" / "math_completeness.py",
|
||||
_REPO_ROOT / "generate" / "math_roundtrip.py",
|
||||
)
|
||||
|
||||
|
||||
def _iter_scanned_files() -> list[Path]:
|
||||
files: list[Path] = []
|
||||
for root in _SCAN_ROOTS:
|
||||
if root.is_file():
|
||||
files.append(root)
|
||||
continue
|
||||
for path in sorted(root.rglob("*.py")):
|
||||
if path.name == "__init__.py":
|
||||
continue
|
||||
files.append(path)
|
||||
return files
|
||||
|
||||
|
||||
def test_no_new_legacy_derivation_regex_without_exception() -> None:
|
||||
violations: list[str] = []
|
||||
|
||||
for path in _iter_scanned_files():
|
||||
rel = path.relative_to(_REPO_ROOT).as_posix()
|
||||
text = path.read_text(encoding="utf-8")
|
||||
if not _REGEX_USAGE.search(text):
|
||||
continue
|
||||
if rel in ALLOWLISTED_LEGACY_DERIVATION_FILES:
|
||||
continue
|
||||
if "LEGACY_EXCEPTION" in text:
|
||||
continue
|
||||
violations.append(rel)
|
||||
|
||||
assert not violations, (
|
||||
"New legacy raw-text regex surfaces detected. Either migrate to "
|
||||
"ProblemFrame/substrate extraction or add an explicit LEGACY_EXCEPTION "
|
||||
f"with migration rationale:\n" + "\n".join(violations)
|
||||
)
|
||||
|
||||
|
||||
def test_allowlist_covers_current_legacy_derivation_regex_files() -> None:
|
||||
"""Ensure the allowlist stays aligned with scanned legacy regex files."""
|
||||
regex_files = {
|
||||
path.relative_to(_REPO_ROOT).as_posix()
|
||||
for path in _iter_scanned_files()
|
||||
if _REGEX_USAGE.search(path.read_text(encoding="utf-8"))
|
||||
}
|
||||
missing = sorted(regex_files - ALLOWLISTED_LEGACY_DERIVATION_FILES)
|
||||
assert not missing, (
|
||||
"Allowlist missing current legacy regex files — update "
|
||||
"ALLOWLISTED_LEGACY_DERIVATION_FILES:\n" + "\n".join(missing)
|
||||
)
|
||||
105
tests/test_problem_frame_builder.py
Normal file
105
tests/test_problem_frame_builder.py
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
"""Tests for generate/problem_frame_builder.py."""
|
||||
from __future__ import annotations
|
||||
|
||||
from fractions import Fraction
|
||||
|
||||
from generate.problem_frame_builder import build_problem_frame
|
||||
from language_packs.scalar_equivalence import extract_scalar_candidates, list_unsupported_surfaces
|
||||
|
||||
|
||||
def _frame_names(text: str) -> tuple[str, ...]:
|
||||
return tuple(f.name for f in build_problem_frame(text).process_frames)
|
||||
|
||||
|
||||
def _hazard_categories(text: str) -> tuple[str, ...]:
|
||||
return tuple(sorted({h.category for h in build_problem_frame(text).hazards}))
|
||||
|
||||
|
||||
def test_percent_text_produces_scalar_facts_and_hazards_without_solving() -> None:
|
||||
text = "Mia spent 50% of her money."
|
||||
frame = build_problem_frame(text)
|
||||
|
||||
assert frame.scalars
|
||||
assert any(s.canonical == Fraction(1, 2) for s in frame.scalars)
|
||||
assert "percent_change_vs_percent_of" in _hazard_categories(text)
|
||||
assert frame.question_target is None
|
||||
|
||||
|
||||
def test_transfer_text_produces_transfer_process_frame_without_solving() -> None:
|
||||
text = "Tom gave Ana 3 marbles."
|
||||
frame = build_problem_frame(text)
|
||||
|
||||
assert "transfer" in _frame_names(text)
|
||||
assert any(r.relation_type == "transfer" for r in frame.candidate_relations)
|
||||
assert not frame.question_target
|
||||
|
||||
|
||||
def test_container_text_produces_container_process_frame_without_solving() -> None:
|
||||
text = "There are 4 full boxes and 3 loose crayons."
|
||||
frame = build_problem_frame(text)
|
||||
|
||||
assert "container_packing" in _frame_names(text)
|
||||
assert frame.units or frame.scalars
|
||||
|
||||
|
||||
def test_travel_text_produces_travel_process_frame_without_solving() -> None:
|
||||
text = "A car drove 30 miles each way."
|
||||
frame = build_problem_frame(text)
|
||||
|
||||
assert "travel" in _frame_names(text)
|
||||
|
||||
|
||||
def test_ambiguous_quarter_surfaces_carry_hazards() -> None:
|
||||
text = "A quarter of the class left."
|
||||
frame = build_problem_frame(text)
|
||||
|
||||
categories = _hazard_categories(text)
|
||||
assert "unbound_base_quantity" in categories
|
||||
assert any(
|
||||
h.surface == "quarter" or h.category.startswith("quarter_")
|
||||
for h in frame.hazards
|
||||
)
|
||||
|
||||
|
||||
def test_third_ordinal_carries_hazard() -> None:
|
||||
text = "Sam finished in third place."
|
||||
frame = build_problem_frame(text)
|
||||
|
||||
assert "third_ordinal" in _hazard_categories(text)
|
||||
assert not any(s.canonical == Fraction(1, 3) for s in frame.scalars)
|
||||
|
||||
|
||||
def test_unsupported_scalar_surfaces_do_not_broaden_adr_0128() -> None:
|
||||
text = "The remaining amount is .5 of the total."
|
||||
frame = build_problem_frame(text)
|
||||
|
||||
assert frame.scalars == ()
|
||||
assert ".5" in list_unsupported_surfaces()
|
||||
assert extract_scalar_candidates(text) == ()
|
||||
|
||||
|
||||
def test_exact_spans_slice_original_text() -> None:
|
||||
text = "Mia spent 50% of her money."
|
||||
frame = build_problem_frame(text)
|
||||
|
||||
for scalar in frame.scalars:
|
||||
assert scalar.source_span is not None
|
||||
assert scalar.source_surface is not None
|
||||
start, end = scalar.source_span
|
||||
assert text[start:end] == scalar.source_surface
|
||||
|
||||
for quantity in frame.quantities:
|
||||
span = quantity.provenance.source_spans[0]
|
||||
assert text[span.start:span.end] == span.text
|
||||
|
||||
|
||||
def test_deterministic_ordering_across_repeated_runs() -> None:
|
||||
text = "Tom gave Ana 3 marbles and spent 50% of her money."
|
||||
frame_a = build_problem_frame(text)
|
||||
frame_b = build_problem_frame(text)
|
||||
|
||||
assert frame_a.scalars == frame_b.scalars
|
||||
assert frame_a.units == frame_b.units
|
||||
assert frame_a.process_frames == frame_b.process_frames
|
||||
assert frame_a.hazards == frame_b.hazards
|
||||
assert frame_a.candidate_relations == frame_b.candidate_relations
|
||||
Loading…
Reference in a new issue