c504796165
Extends ADR-0022 with inspection/telemetry surfaces that turn the forward-semantic-control claim from "mechanism exists" into "mechanism is causally load-bearing, isolated, and replayable." Changes (zero runtime semantics change beyond a pipeline bug fix): - AdmissibilityTraceStep + GenerationResult.admissibility_trace — per-transition record of region label, candidates before/after, selected destination, and the typed AdmissibilityVerdict. - ChatResponse + CognitiveTurnResult expose admissibility_trace, admissibility_trace_hash, ratification_outcome, region_was_unconstrained. - hash_admissibility_trace + compute_trace_hash fold the new fields only when they carry non-default values, so pre-ADR-0023 turn hashes remain byte-preserved. - Same-path ablation leg in evals/forward_semantic_control/runner.py: generate(..., region=None) vs generate(..., region=R) on the same runtime/vocab/field/persona/prompt — isolates the region as cause. - Lane expansion: 8 dev cases across 4 relation axes (cause, means, precedes, part_of) including 2 adversarial distractor cases. - Lane metrics now report region_only_constrained_rate / region_only_gap / ratified_rate / demoted_rate / passthrough_rate / passthrough_on_scored. - Bug fix surfaced by the new accounting: _ratify_intent looked up runtime.vocab (always None) instead of runtime.session.vocab — every production turn was silently PASSTHROUGH. Fixed; ratifier now actually gates intent classification. - tests/test_admissibility_trace.py: hash determinism + pre-ADR-0023 byte-preservation tests. Lane evidence (dev, 8 cases): - constrained_pass_rate=0.80, causality_gap=0.80 - region_only_gap=1.00 (5/5 with region, 0/5 without — same path) - ratified_rate=1.00, passthrough_on_scored=false - overall_pass=true Bench: 9.41s / 20 turns (~470ms/turn), well inside the +5% budget. Full pytest: 922 passed, 1 pre-existing failure (test_language_pack_cache, unrelated to ADR-0023).
53 lines
2 KiB
JSON
53 lines
2 KiB
JSON
{
|
|
"cloud_reference": {
|
|
"hourly_usd": 0.0416,
|
|
"name": "AWS t3.medium (2 vCPU, 4 GiB)",
|
|
"region": "us-east-1, on-demand, Linux",
|
|
"source_note": "aws.amazon.com/ec2/instance-types/t3 — public on-demand rate, captured 2026-05-17. Update source_note + hourly_usd if the price page changes."
|
|
},
|
|
"cpu_seconds_total": 9.410622,
|
|
"cpu_utilization": 0.9996,
|
|
"energy_disclosure": "Joules per turn is not reported. Honest energy measurement requires RAPL (Linux) or IOKit/powermetrics (macOS) with privileged access. cpu_seconds_total is the available CPU-time proxy.",
|
|
"frontier_pricing_comparison": [
|
|
{
|
|
"core_cheaper_by_x": 121.3,
|
|
"frontier_usd_per_1000_turns": 0.66,
|
|
"input_usd_per_million_tokens": 3.0,
|
|
"name": "Anthropic Claude Sonnet 4.5 (API)",
|
|
"output_usd_per_million_tokens": 15.0,
|
|
"source_note": "anthropic.com/pricing — public API rate, captured 2026-05-17."
|
|
},
|
|
{
|
|
"core_cheaper_by_x": 82.7,
|
|
"frontier_usd_per_1000_turns": 0.45,
|
|
"input_usd_per_million_tokens": 2.5,
|
|
"name": "OpenAI GPT-4o (API)",
|
|
"output_usd_per_million_tokens": 10.0,
|
|
"source_note": "openai.com/api/pricing — public API rate, captured 2026-05-17."
|
|
},
|
|
{
|
|
"core_cheaper_by_x": 40.4,
|
|
"frontier_usd_per_1000_turns": 0.22,
|
|
"input_usd_per_million_tokens": 1.0,
|
|
"name": "Anthropic Claude Haiku 4.5 (API)",
|
|
"output_usd_per_million_tokens": 5.0,
|
|
"source_note": "anthropic.com/pricing — public API rate, captured 2026-05-17."
|
|
}
|
|
],
|
|
"frontier_token_assumption": {
|
|
"input_tokens_per_turn": 20,
|
|
"note": "Conservative short-prompt / short-answer turn. Frontier $/1000-turns scales linearly with these counts.",
|
|
"output_tokens_per_turn": 40
|
|
},
|
|
"latency": {
|
|
"max_ms": 597.419,
|
|
"median_ms": 549.86,
|
|
"min_ms": 4.027,
|
|
"p95_ms": 556.098
|
|
},
|
|
"throughput_turns_per_second": 2.1244,
|
|
"turns": 20,
|
|
"usd_per_1000_turns": 0.005439,
|
|
"wall_seconds_total": 9.414349,
|
|
"warmup_turns": 5
|
|
}
|