Version the policy and evaluator separately

An AI feature changes when its model changes, but it also changes when display thresholds, ranking rules, or evaluation criteria move. Recording only a model name leaves most of that behavior impossible to reconstruct.

Policy and evaluation have different jobs. Policy decides what the product does now; the evaluator measures outcomes under a defined interpretation. Versioning them independently makes experiments comparable and prevents a changed measurement rule from rewriting the meaning of older results.

Give each decision input its own identity

This section defines immutable version records instead of one generic application version. Each output can point to the exact rules and measurement logic used at the time.

Separate version records make the distinction concrete:

export type PolicyVersion = {
  id: string;
  createdAt: string;
  minimumConfidence: number;
  maximumVisibleItems: number;
  allowedKinds: string[];
};

export type EvaluatorVersion = {
  id: string;
  createdAt: string;
  rubricVersion: string;
  datasetVersion: string;
  implementationHash: string;
};

export type DecisionRecord = {
  candidateId: string;
  policyVersionId: string;
  evaluatorVersionId: string;
  action: 'surface' | 'store-only' | 'discard';
  score: number;
  decidedAt: string;
};

A policy identifier should reference immutable configuration. Editing a row in place while keeping its identifier makes historical replay look precise when it is not.

The evaluator records its dataset and implementation separately because changing either can change a score. A rubric label alone does not capture a bug fix in the code that applies it.

Make policy evaluation a pure function

This section ensures that a stored policy, candidate, and bounded context are enough to reproduce the product decision. Time, random values, and mutable global configuration stay outside the decision function.

The policy maps supported candidates to explicit outcomes such as surface, store-only, or discard. Ordering and visibility limits are part of the same versioned decision. Persist the policy identifier with every outcome, not only in a deployment log. One rollout can process jobs created under different configurations, and timestamps alone are an unreliable join.

If policy depends on request context, persist the bounded context fields that affected the branch. Do not persist entire requests when a locale, plan type, or feature variant is sufficient.

Store evaluation as a snapshot

This section prevents later evaluator changes from silently mutating old scores. Re-evaluation creates a new immutable snapshot that points to the same candidate and a different evaluator version.

Store dimension scores such as evidence support, relevance, and clarity rather than only a weighted total. Weighting is itself policy, and retaining dimensions allows a team to compare a new weighting without rerunning every underlying judgment. The snapshot also records the dataset and implementation versions that produced those dimensions.

Replay changes before rollout

This section uses stored candidates to compare a proposed policy with the active one. Run both policies over the same representative, time-bounded sample and report movement between outcome categories.

The comparison should answer which candidates changed, in which direction, and whether the movement concentrates in one signal kind or confidence band. The purpose is not to guarantee production performance but to catch a policy that unexpectedly surfaces far more, far less, or a different type of content.

Evaluator changes need a parallel comparison. Score the same labeled sample with both evaluator versions and inspect disagreement before using the new scores to approve policy changes. Otherwise, the team may attribute movement to the product when only the yardstick changed.

Versioning is useful only when identifiers lead to immutable, recoverable definitions. Separate policy from evaluation so product behavior and the yardstick used to judge it can evolve without erasing each other’s history.