core/evals
Shay 80240ea9b8
feat(derivation): Workstream A inc 1 — lexeme reader + cleaned exemplars + proxy rebaseline (#796)
* feat(derivation): Workstream A inc 1 — lexeme-only reader components for fractions/comparatives + cleaned Phase B exemplars + proxy rebaseline (ratified, post-review fixes)

- generate/derivation/extract.py: lexeme-only passes (surface tokens only; no synthesized values or non-surface source_tokens for "half of", "X/Y of", "X more/less than"). Composition deferred per module contract (ADR-0165). Hygiene and EX-6 preserved. Docstring updated.
- teaching/admissibility_exemplars/*_v1.jsonl: grown + cleaned (removed category-polluted seeds, deduplicated ma IDs, fixed comparative exemplar to surface tokens per lexeme rule).
- evals/gsm8k_math/train_sample/v1/report.json: rebaselined after fixes (runner executed; wrong=0 held).
- docs/analysis/*-2026-06-1[6-7].md + lookback: ratif + honest post-review lookback (local paths removed, scope claims match actual clean diff, extractor/exemplar fixes documented).

Governing: the two 2026-06-16/17 ratifications (included; local session path replaced with stable ref). Scope per ratif: reader (lexeme) + exemplar growth (clean) + rebaseline + tests + lookback. No sealed, no CLOSE (deferred), 0 wrong.

Added tests in test_adr_0179_extract.py for new lexeme paths + grounding.

Branch prepared clean from origin/main; diff limited to scoped files only (no posture ratif, no runtime_contracts drift).

See updated lookback and PR body for full evidence (6/44/0 or current post-fix,  tests green, etc.).

(Addresses review: Semantic Rigor restored, branch hygiene, exemplar quality, tests, docs.)

* fix(review): resolve all verifiability blockers

- generate/derivation/extract.py: conflict markers removed (was from stash); lexeme-only implementation confirmed clean.
- tests/test_adr_0179_extract.py: added TestWorkstreamAReaderLexemeOnly class with direct tests for half-of, fraction-of, more/less components, source_token grounding, no-synthesis invariants.
- evals/gsm8k_math/train_sample/v1/report.json: fresh rebaseline via runner with fixed code (6/44/0, wrong=0).
- docs/analysis/gsm8k-workstream-a-increment-1-lookback-2026-06-17.md: updated to exactly match the actual diff and actual report numbers on this head.
- No conflict markers remain in extract, tests, lookback, ratifs, or exemplars (verified by grep).
- Branch head now clean and matches all claims in PR body/lookback.

This is the verifiable head for the lead engineer review.

* fix(test): make lexeme-only half test case-insensitive on source_token to match actual extraction (preserves input casing like other EX tests)

* fix(review): commit current report.json (6/44/0 from runner) and updated lookback so they appear in branch diff; lookback now exactly matches actual files in git diff --name-only and actual test count (29) and report numbers.

* fix(review): precise fractional doc in extract contract (normalizes factors while preserving source_token; comparative deferred); remove stale comparative sum/diff comments; lookback exact reality (29/29 tests, no deep-dive in diff, no sum/diff claim); mark remaining exemplars REVIEW-REQUIRED/ambiguous with explicit notes. Small final patch.

* fix(exemplars): remove three REVIEW-REQUIRED ambiguous-but-machine-admissible seeds (rwc-v1-0022, rwc-v1-0023, ma-v1-0024) from ingested JSONL to eliminate synthesis-poisoning risk per final review. Update lookback wording for actual seed count and rationale. Small final patch.

* docs(lookback): align exemplar category list with actual diff (only dcs/rwc/ma touched in this increment; no ca/ta/dsnq changes)
2026-06-16 21:57:05 -07:00
..
adversarial_identity
anchor_lens_tour
anti_regression feat: strengthen visibility and measurement of CLOSE flywheel proposal review/ratification side (#794) 2026-06-16 19:33:25 -07:00
articulation
articulation_of_status
audio_sensorium
audit_tour
calibration
capability_index feat(reader): add overlaps_event finite-verb reader surface (B3) (#783) 2026-06-15 14:56:09 -07:00
classical_literature_ood
close_derived_climb feat: strengthen visibility and measurement of CLOSE flywheel proposal review/ratification side (#794) 2026-06-16 19:33:25 -07:00
cognition
cold_start_grounding
combined_rate_oracle docs(cmb): lookback review + fix single-agent-attribution hygiene hazard (H3) 2026-06-08 13:27:17 -07:00
compositionality
compound_intent_decomposition
comprehension feat: one-hop sound relational entailment (inverse/symmetric) + capability-index lane (#775) 2026-06-15 11:39:41 -07:00
constraint_oracle feat(verified): P1-C — split bound_slots_digest into separate obligation 2026-06-08 17:17:31 -07:00
contemplation_quality
contradiction_detection
conversation
conversational_thread_coherence
cross_domain_transfer
curriculum_loop_closure
deductive_logic
demo_composition
determination_closure feat(determine): idle deductive consolidation — the loop learns from determined facts (Step D) 2026-06-06 12:28:09 -07:00
determination_estimation feat(determine): calibrated disclosed estimation — the engine earns the right to guess (Step E) 2026-06-06 13:49:07 -07:00
deterministic_fluency
dimensional
discourse_paragraph
domain_contract_validation
edge_budget feat(edge): edge-deployment budget gate — deterministic per-turn persistence cost 2026-06-06 10:27:10 -07:00
elementary_mathematics_ood
english_fluency_ood
environment_falsification Add offline witness log importer 2026-06-06 12:37:57 -07:00
event_vision_sensorium Add event vision sensorium lane 2026-06-06 12:37:57 -07:00
fabrication_control
field_incidence work done towards furthering comprehension, contemplation, learning, and also began working on fixing some failing tests 2026-06-16 12:27:59 -07:00
flywheel_demo
forward_semantic_control
foundational_biology_ood
foundational_physics_ood
frame_verdict_text_cwa feat(frame-verdict): closed-world FrameVerdict substrate — PR-1..4 + hardening (ADR-0222 B4) (#787) 2026-06-16 06:23:03 -07:00
frontier_compare
grammatical_coverage
gsm8k_math feat(derivation): Workstream A inc 1 — lexeme reader + cleaned exemplars + proxy rebaseline (#796) 2026-06-16 21:57:05 -07:00
gsm8k_parser_dev
hebrew_fluency
identity_divergence
industry_demos
inference_closure
introspection
koine_greek_fluency
l10_always_on feat(identity): split engine identity from build provenance (ADR-0220 PR C) (#774) 2026-06-15 11:38:04 -07:00
l10_continuity test(l10): W2-R arbitrary-interruption recovery harness (ADR-0219) 2026-06-15 02:45:29 -07:00
lab
learning_arc
learning_loop
long_context_cost
math_bounded_grammar/v1
math_capability_axes
math_expert_claims/v1
math_symbolic_equivalence
math_teaching_corpus/v1
miner_loop_closure
monotonic_learning
multi_agent_composition
multi_sentence_response
multi_step_reasoning
obligation_2_ood_ratio
obligation_5_perturbation
obligation_6_depth_curve
obligation_8_adversarial
obligation_10_pack_provenance
orthogonality_tour
prompt_diversity
proofwriter_owa feat(eval): ProofWriter-OWA refusal-floor lane (B1) — independent oracle, measure-only (#779) 2026-06-15 12:28:20 -07:00
propositional_logic feat(comprehend): propositional-logic comprehension (4th domain, flagship oracle) 2026-06-05 23:24:54 -07:00
provenance
public_demo docs(public_demo): add known environment caveat for wall-clock budget flake 2026-06-11 18:12:00 -07:00
rate_oracle test(rate-oracle): port _canonical_outcome non-vacuous validation to R3 (R3-vac) 2026-06-08 07:38:20 -07:00
realizer_guard
refusal_calibration feat(identity): L11 identity continuity — same identity across reboot, not just same bytes 2026-06-05 13:52:57 -07:00
refusal_taxonomy
register_diagnostics
register_tour
relational feat(reader): add overlaps_event finite-verb reader surface (B3) (#783) 2026-06-15 14:56:09 -07:00
relational_inference/v1 feat(reader): add overlaps_event finite-verb reader surface (B3) (#783) 2026-06-15 14:56:09 -07:00
relational_metric feat(oracle): narrow reverse-solve contract — base of one more/fewer_than (PR-7a) 2026-06-07 06:22:36 -07:00
relational_transitive feat(determine): add transitive strict-order relational inference (#781) 2026-06-15 14:20:30 -07:00
reports
results
reviewer_registry
sample_efficiency
self_consistency_over_time
sensorimotor_sensorium
sensorium Add event vision sensorium lane 2026-06-06 12:37:57 -07:00
set_membership evals: add staged independent gold lanes 2026-06-05 16:03:26 -07:00
setup_oracle feat(comprehension): inverse reader frame — base of a more/fewer-than (PR-7b / R2 C0) 2026-06-07 07:06:26 -07:00
syllogism feat(comprehend): multi-word NP chunking under a canonicalization contract 2026-06-05 22:47:34 -07:00
symbolic_logic
teaching_injection_resistance
total_ordering feat(comprehend): multi-word NP chunking under a canonicalization contract 2026-06-05 22:47:34 -07:00
vision_sensorium
walkthrough_chain
warmed_session_consistency
zero_code_domain_acquisition
__init__.py
_parallel.py
baseline_runner.py
CLAIMS.md
cognition_cases.jsonl
framework.py
holdout_runner.py
metrics.py
parallel.py
run_cognition_eval.py