How to measure the real cost of an agent evaluation

An evaluation run spends tokens in two places: the system being tested and the system grading it. The second category is easy to miss because one answer can be checked by several rubrics, and a nondeterministic rubric may be sampled more than once. A useful cost report makes every multiplier explicit and marks work that cannot be attributed to a test case.

Model the call graph before estimating cost

This section turns an evaluation plan into a count of model calls. The purpose is not to predict a bill to the cent; it is to expose which decisions multiply the bill.

For a suite with (C) cases, (M) model-judged metrics, and (S) judge samples, the first approximation is:

judge calls = C × M × S
total calls = agent calls + judge calls + retries

That formula needs one refinement. Some metrics make more than one judgment per sample, while deterministic checks make none. Represent each metric with a calls_per_sample value instead of treating every check alike.

The accounting model can remain small and explicit:

from dataclasses import dataclass
from decimal import Decimal


@dataclass(frozen=True)
class ModelRate:
    input_per_million: Decimal
    output_per_million: Decimal


@dataclass(frozen=True)
class Usage:
    input_tokens: int
    output_tokens: int

    def cost(self, rate: ModelRate) -> Decimal:
        million = Decimal(1_000_000)
        return (
            Decimal(self.input_tokens) * rate.input_per_million / million
            + Decimal(self.output_tokens) * rate.output_per_million / million
        )


@dataclass(frozen=True)
class JudgedMetric:
    name: str
    samples: int
    calls_per_sample: int = 1

    def calls_for(self, case_count: int) -> int:
        return case_count * self.samples * self.calls_per_sample


def expected_judge_calls(
    case_count: int,
    metrics: list[JudgedMetric],
) -> dict[str, int]:
    return {metric.name: metric.calls_for(case_count) for metric in metrics}

Keep rates in deployment configuration, not in the article, source fixtures, or scorecard schema. Provider prices change; recorded token counts remain useful after the rate changes.

Attribute only the work you can trace

This section separates exact per-case cost from aggregate cost. The distinction matters in systems where one user turn produces several model calls.

A request may route through an orchestrator, call a specialist, invoke a tool, and ask another model to format the result. If every hop records only model usage, the run total can be correct while the per-case total is fiction. Exact attribution requires the same case identifier to cross every process and model boundary.

Use a small immutable context object rather than passing identifiers as unrelated keyword arguments. It needs a run identifier, case identifier, and trace identifier, and it must cross every process and model boundary involved in answering or judging the case.

Every model-call event should include run_id, case_id, trace_id, model name, token usage, and call role. The call role might be answer, route, or judge; it should describe function, not an internal agent name. A tool or hosted runtime that drops the context breaks exact attribution at that boundary.

Report the break instead of hiding it. A scorecard can expose attribution: complete, attribution: partial, or attribution: aggregate_only. Under partial attribution, show the traced per-case amount and the unattributed run total separately. Do not divide the missing amount evenly across cases and label the result measured.

Record cost as events, then calculate views

This section defines a storage shape that supports both case-level debugging and run-level budgeting. Raw usage events are more durable than a precomputed cost_per_case field.

Each event should record the available run, case, and trace identifiers; the call’s functional role; model identifier; input and output tokens; and observation time. Missing context remains explicit instead of being assigned to a convenient case. The event schema should describe measurement, not a provider’s current rate card.

Calculate money when rendering the report. That keeps the usage record independent of a provider contract and lets historical reports be recalculated under a new rate card.

Treat retries and concurrency differently

This section covers two controls that are often grouped together even though they affect different outcomes. Retries increase call count; concurrency changes elapsed time.

Retry the smallest failed unit. If one judge response is malformed, rerun that metric sample for that case. A suite-level retry repeats valid work and makes the cost of grader instability hard to see. Record the original attempt and retry as separate events so the report can show how much evaluation failure costs.

Concurrency should be bounded by provider quotas and local resources. Raising it can shorten a run without reducing tokens. If throttling causes retries, excessive concurrency can increase both time and cost, so the useful measurement is completed cases per minute alongside retry rate—not concurrency in isolation.

Batch execution and prompt caching can reduce cost when a provider supports them, but measure their effect against captured usage rather than copying advertised savings into a forecast. A cached rubric still needs a version identifier so a prompt change cannot reuse a result produced under a different grading definition.

Decide which measurements can become gates

Token use and answer length are valuable trend signals, but they are rarely correctness properties. Record them on every run and alert on meaningful changes after you have enough history to define normal variation. Avoid a universal word limit when cases range from short confirmations to explanatory answers.

The practical outcome is a report with three honest numbers: fully attributed case cost, unattributed run cost, and judge overhead. Together they explain what was measured, what was not, and which part of the evaluation design is responsible for the bill.