Move the model out of the request path

Putting a model call directly inside an HTTP request couples user latency to inference latency. It also makes retries expensive, outages contagious, and the same page capable of producing a different answer each time it loads.

Many AI features do not need synchronous generation. When the underlying evidence changes less often than the interface is read, generate a versioned artifact in the background and make the request path a predictable lookup.

Treat generated output as an artifact

This section defines a stored result with enough context to decide whether it is reusable. Generated prose alone is not a cache entry because its meaning depends on source data, instructions, and model behavior.

A reusable artifact needs enough context to explain its own validity:

export type InsightArtifact = {
  subjectId: string;
  periodStart: string;
  periodEnd: string;
  content: {
    headline: string;
    explanation: string;
  };
  sourceVersion: string;
  instructionVersion: string;
  modelVersion: string;
  generatedAt: string;
};

export function artifactKey(
  item: Pick<InsightArtifact, 'subjectId' | 'periodStart' | 'periodEnd'>,
) {
  return `${item.subjectId}:${item.periodStart}:${item.periodEnd}`;
}

The versions make invalidation explicit. A newer source snapshot, a changed instruction contract, or a different model can schedule regeneration without deleting the previous artifact before its replacement exists.

The period belongs in the identity because an explanation for one window is not interchangeable with an explanation for another, even when the subject is the same.

Generate after evidence is ready

This section starts inference from a background job rather than an incoming page request. The source pipeline should schedule generation only after the evidence snapshot is committed. Otherwise, the worker may read half-finished input and faithfully generate a polished explanation of incomplete data.

The job identity should include the subject, evidence window, and source version. Before generating, compare the requested source, instruction, and model versions with the current artifact. If all match, the job is already complete. If any differ, generate a replacement and publish it atomically.

Queue delivery is commonly at least once, so correctness cannot depend on a job arriving exactly once. Use an idempotent upsert or insert-once operation and keep the previous valid artifact readable until its replacement is complete. That prevents a regeneration failure from turning a stale result into no result.

Make stale and missing states explicit

This section keeps the read path fast without pretending that every artifact is current. The service should distinguish ready, stale, pending, and unavailable states rather than returning one generic success shape.

A stale artifact can still be useful when the interface labels its evidence window. Whether to show it depends on the consequence of staleness, but the service should expose enough status for that decision. Pending means no artifact exists for the expected version; unavailable means generation cannot currently produce one. Those states should not be collapsed.

The request path now performs authorization, a version lookup, and a read. Model timeouts no longer consume web-server capacity, and a provider outage delays new artifacts instead of taking down an otherwise readable page.

Observe lag instead of request latency

This section changes the operational question from “How long did this request wait for the model?” to “How far behind is generated content?” That metric better represents an asynchronous feature.

Measure source-to-artifact lag, queue age, generation duration, failure reason, and the distribution of active source, instruction, and model versions. Track them by artifact type and status rather than by subject identifier. A low request latency can otherwise hide a generator that has stopped producing new work.

Alert on the user-facing consequence: the share of reads that are stale or pending beyond policy, not every individual model timeout. This keeps operational thresholds aligned with the experience the architecture was designed to protect.

Synchronous inference is appropriate when user input truly requires an immediate, novel answer. For evidence-derived summaries and recurring insights, a versioned background artifact gives users faster reads and gives operators a failure boundary they can actually control.