Groundedness is not one property
How to scope a hallucination grader across user-specific claims, general knowledge, and interpretation without making the judge permissive.
A sentence can be unsupported by retrieved context without being false. It can also be supported by fluent prose while making a claim the source never established. A groundedness score that collapses those cases may penalize useful explanation and miss the fabrication that matters.
Before allowing a groundedness metric to influence a release, define which claims require attribution, which require factuality, and which are not factual claims at all.
Inspect the definition behind the score
This section describes what a sentence-level grader normally measures. The score is only the aggregation layer; the labeling instructions contain the actual policy.
A typical grader splits an answer into statements and assigns labels such as supported, unsupported, contradictory, or not_applicable. Many retrieval-oriented graders require every factual statement to appear in the supplied context. That policy is appropriate for an assistant summarizing a closed document set.
It is less suitable for an assistant expected to combine private records with ordinary explanation. Consider an answer that says, “The report contains three failed jobs. A retry reruns a failed operation.” The first sentence must be supported by the user’s records. The second is general technical knowledge. Requiring both sentences to appear in the retrieved record confuses attribution with truth.
Define three claim classes
This section turns the distinction into a grading contract. Each sentence belongs to one of three classes before it receives a verdict.
A subject-specific claim describes the current user, account, document, or transaction. It must be entailed by supplied context. An external claim describes the world beyond that private subject. It must be true under an approved verification policy, but need not occur in the private context. An interpretation expresses a recommendation, opinion, or transition and is not graded for attribution unless it contains an embedded factual premise.
Represent the contract as data rather than burying it in one prompt. A compact representation might look like this:
from dataclasses import dataclass
from enum import StrEnum
class ClaimClass(StrEnum):
SUBJECT = "subject_specific"
EXTERNAL = "external"
INTERPRETATION = "interpretation"
class Verdict(StrEnum):
SUPPORTED = "supported"
UNSUPPORTED = "unsupported"
CONTRADICTORY = "contradictory"
NOT_APPLICABLE = "not_applicable"
@dataclass(frozen=True)
class StatementGrade:
text: str
claim_class: ClaimClass
verdict: Verdict
evidence: tuple[str, ...]
rationale: str
def valid_grade(grade: StatementGrade) -> bool:
if grade.claim_class is ClaimClass.SUBJECT:
return grade.verdict in {
Verdict.SUPPORTED,
Verdict.UNSUPPORTED,
Verdict.CONTRADICTORY,
}
if grade.claim_class is ClaimClass.INTERPRETATION:
return grade.verdict is Verdict.NOT_APPLICABLE
return grade.verdict in {
Verdict.SUPPORTED,
Verdict.UNSUPPORTED,
Verdict.CONTRADICTORY,
}The external class still needs a verification source. A judge model’s parametric knowledge may be acceptable for stable, low-consequence definitions. Current prices, security behavior, medical claims, and changing product capabilities require retrieval from an approved source. “External” is not a license to guess.
Keep classification and verification separate
This section prevents a permissive classifier from turning into a permissive grader. Use one pass to classify the claim and a second policy to decide what evidence it requires.
from dataclasses import dataclass
@dataclass(frozen=True)
class EvidencePolicy:
require_private_context: bool
require_external_source: bool
def evidence_policy(claim_class: ClaimClass) -> EvidencePolicy:
if claim_class is ClaimClass.SUBJECT:
return EvidencePolicy(True, False)
if claim_class is ClaimClass.EXTERNAL:
return EvidencePolicy(False, True)
return EvidencePolicy(False, False)An interpretation that includes a hidden subject claim must be split. “This looks healthy because your error rate fell by half” contains an opinion and a measurable assertion. Grade the error-rate statement against context before treating the remaining sentence as interpretation.
Ask the grader to return evidence spans or source identifiers with every supported verdict. A supported label without evidence is difficult to audit and easy for a judge to award based on plausibility.
Calibrate with adversarial pairs
This section checks whether a revised rubric improved classification or merely became lenient. A useful calibration set contains near-identical examples that should land on opposite sides of the rule.
Store calibration cases with synthetic context, answer, expected class, expected verdict, and a short human rationale. Pair a true general explanation with a false capability claim. Pair a supported subject number with a nearby fabricated number. Pair a harmless recommendation with a recommendation whose premise invents user state.
Calculate agreement by claim class and verdict, not only as one overall percentage. A single score can hide a grader that handles general explanations well while accepting fabricated subject-specific claims.
Report the denominator with every rate. Perfect agreement over a few obvious cases is weaker evidence than slightly lower agreement over a larger set of borderline examples. Keep the original calibration cases after changing the rule so regressions toward leniency remain visible.
Version the metric definition
This section handles the operational consequence of changing a grader. A score produced under a new claim policy is not comparable with a score produced under the old one.
Give the rubric a semantic version and write it into every result. When labeling scope changes, create a new baseline rather than comparing new scores with old scores. Install the same version in local evaluation, CI, calibration, and report generation; otherwise the calibration result describes a grader that the release process does not run.
If customization depends on a framework-owned prompt or other unstable surface, fail loudly when the expected hook is missing. A guarded adapter can disable the custom metric with an explicit error. It should not silently fall back to a different definition while retaining the same metric name.
Groundedness becomes useful when the word has an operational definition. Private claims need attribution, external claims need proportionate verification, and interpretation needs inspection for hidden premises. That separation keeps the grader strict where fabrication causes harm without treating every useful explanation as a hallucination.