core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

chore(eval): mark candidate-graph runner aggregation as needing audit (#213)

Browse source 
Adds a 3-line TODO comment above `_score_one_candidate_graph` in
evals/gsm8k_math/runner.py. No behavior change.

Flags that `report.json` metrics may not credit candidate-graph
admissions routed through this branch (Stage 1 candidate-graph
parse + internal solve path) the same way `_score_one` admissions are
credited. Aggregation in calling code needs an audit before the
canonical run.honest_runner.json artifact can be trusted for
cross-phase comparison.

This is Piece A of a three-piece hygiene split. The MEMORY.md
compaction and worktree audit pieces are deferred — they need
human judgment (re-shaping vs. truncating) and an OS-correct date
predicate (BSD vs. GNU), neither of which fits a one-shot script
pass.

No tests run — this change is comment-only and has zero runtime
effect.
This commit is contained in:
Shay 2026-05-24 06:57:23 -07:00 committed by GitHub
parent 6e072f95be
commit 7d0803b457
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 3 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
evals/gsm8k_math/runner.py
View file

@ -205,6 +205,9 @@ def _score_one(case: dict[str, Any]) -> CaseOutcome:
)
# TODO(ADR-future): report.json metrics may not credit candidate-graph admissions
# routed through this branch. Aggregation in calling code needs an audit before
# the canonical run.honest_runner.json artifact can be trusted for cross-phase comparison.
def _score_one_candidate_graph(case: dict[str, Any]) -> CaseOutcome:
"""ADR-0126 P4 — score one case via the candidate-graph pipeline.