Make your evaluation harness run the agent you ship

An agent is more than its model and prompt. Context assembly, session state, tool adapters, and lifecycle hooks all affect the answer. An evaluation that bypasses those paths can use production model code and still measure behavior that no user will encounter.

The safest design is to keep production control flow and replace only external data. Synthetic fixtures provide reproducibility; shared adapters preserve fidelity.

Put context assembly behind one interface

This section creates a seam that production and evaluation can share. The harness should not reproduce prompt blocks in test code because the copy will drift as fields and formatting change.

The production boundary can be expressed as a narrow interface:

from dataclasses import dataclass
from datetime import datetime
from typing import Protocol


@dataclass(frozen=True)
class AccountSnapshot:
    display_name: str
    plan: str
    projects: tuple[str, ...]
    updated_at: datetime


class SnapshotSource(Protocol):
    async def load(self, account_id: str) -> AccountSnapshot: ...


class ContextAssembler:
    def __init__(self, source: SnapshotSource) -> None:
        self.source = source

    async def build(self, account_id: str) -> str:
        snapshot = await self.source.load(account_id)
        projects = ", ".join(snapshot.projects) or "none"
        return (
            "<account_context>\n"
            f"name: {snapshot.display_name}\n"
            f"plan: {snapshot.plan}\n"
            f"projects: {projects}\n"
            f"updated_at: {snapshot.updated_at.isoformat()}\n"
            "</account_context>"
        )

The application creates ContextAssembler with a database-backed source. The evaluation runner creates it with an in-memory source. Both call the same build method, so escaping, field selection, and prompt structure cannot diverge.

Replace data sources, not request shape

This section builds a synthetic source without changing what the user says. A realistic case should contain natural language, not internal identifiers inserted to help the harness.

A fixture adapter can implement that same interface without changing request shape:

from datetime import datetime, timedelta, timezone

class FixtureSnapshotSource:
    def __init__(self, now: datetime) -> None:
        self.now = now

    async def load(self, account_id: str) -> AccountSnapshot:
        fixtures = {
            "standard": AccountSnapshot(
                display_name="Morgan",
                plan="active",
                projects=("Atlas", "Beacon"),
                updated_at=self.now - timedelta(minutes=5),
            ),
            "empty": AccountSnapshot(
                display_name="Riley",
                plan="trial",
                projects=(),
                updated_at=self.now - timedelta(minutes=5),
            ),
        }
        return fixtures[account_id]


FIXED_NOW = datetime(2030, 1, 15, 12, 0, tzinfo=timezone.utc)

An evaluation can now ask “Summarize Morgan’s projects” and expect name resolution to use the assembled context. If the test message contains a database ID that a real user never sees, the case is exercising an easier and different task.

The fixture values above are invented and domain-neutral. Public examples should never contain transformed production records: changing a name does not make an account snapshot anonymous when dates, relationships, and rare states remain recognizable.

Freeze asserted values and control the clock

This section prevents fixtures from expiring while keeping expected answers stable. Tests that mention “today” or “last week” need a clock supplied by the harness.

Do not calculate relative dates from the machine clock inside fixture modules. Pass a clock into both the application and the fixture source, then pin it for the run. Every relative window should derive its start and end from that same clock and calendar policy.

There are two valid approaches. A frozen clock makes every timestamp and assertion identical across runs. A moving clock can keep calendar language current while fixed values remain comparable, but only if every date derives from the same injected clock. Mixing real time and fixture time creates failures that appear only around midnight, daylight-saving transitions, or the first day of a month.

Design fixtures around shape

This section chooses cases based on structural risk rather than row count. More records do not compensate for missing states.

A small fixture set should cover a normal account, an empty result, a maximum-size payload, a partially completed flow, an ambiguous name, and a stale snapshot. Each shape should exist because it exercises a branch in context compilation or tool behavior. Keep the objects synthetic and minimal; unused fields make maintenance harder and may reveal more of the private data model than the lesson requires.

Encode the reason beside each fixture so future editors know what must not be simplified. A short purpose statement such as “returns an explicit no-data answer without fabrication” is more useful than a generic fixture name.

The empty result deserves a positive assertion. A language model often tries to fill a gap with a plausible summary; a fixture with no records verifies that the system says there is nothing to summarize.

Prove the harness crosses production boundaries

This section adds tests for the harness itself. A green behavioral score means little if lifecycle hooks or context assembly never ran.

Instrument the shared boundary with observable events such as context.loaded, context.rendered, and tool.called. Each evaluation case should declare the production steps it expects and fail when any are missing.

This assertion catches a runner that invokes the model directly and skips the application lifecycle. It also makes the fidelity claim testable without publishing internal class names or the exact production topology.

Detect fixture drift

Fixture staleness is a dependency problem. Store a public contract version with every case and fail loading when it does not match the version expected by the context assembler or tool schema. The version should change when field meaning changes, not whenever fixture prose is edited.

Contract versions do not prove that an expected answer is still good, so keep review dates and ownership in fixture metadata. Re-run human review after a prompt policy, output contract, or tool schema changes.

A faithful evaluation differs from production in values, credentials, and side effects. It should not differ in context shape, lifecycle order, routing boundaries, or output parsing. That constraint produces scores that describe the shipped system while keeping private data and implementation details out of the test corpus.