How to find tests that assert nothing

Consider a response validator that checks a value only when the value exists:

def validate_response(response: dict) -> None:
    if response.get("type"):
        assert response["type"] in {"answer", "clarification"}

    if response.get("markup"):
        assert parse_markup(response["markup"]) is not None

The empty dictionary passes. Both checks reject malformed values, but neither states that a value must be present. If the realistic regression is a flow that stops producing output, the validator approves the failure.

Distinguish validity from presence

This section identifies the logical shape of the bug. “If X exists, X is valid” is true when X does not exist; it is an implication with a false premise.

The same pattern appears with optional events, empty metric collections, and callbacks whose signatures are checked only after arrival. Those checks answer whether observed data is valid. Separate tests must answer whether the event, metric, or callback was required and actually occurred.

Review each assertion with an empty-input mutation. Replace the real response with {}, [], None, or an omitted event. A test that remains green may be correct for an optional field, but that optionality must come from the case contract rather than from the validator’s convenience.

Declare a positive contract per case

This section moves presence requirements into test data because different flows legitimately produce different output. Keep each case beside its expected contract:

id: clarify_missing_range
message: "What about the previous period?"
expects:
  response_type: clarification
  markup: forbidden

A second case can use the same contract with markup: required. The point is not the serialization format; it is that every case states a positive expectation rather than inheriting optionality from the validator.

Use a closed vocabulary for optionality. A boolean can express required versus forbidden, but required, optional, and forbidden make all three states explicit when the product needs them.

The validator should fail closed when that positive contract is absent:

from dataclasses import dataclass
from enum import StrEnum
from typing import Callable


class Presence(StrEnum):
    REQUIRED = "required"
    OPTIONAL = "optional"
    FORBIDDEN = "forbidden"


@dataclass(frozen=True)
class ExpectedOutput:
    response_type: str
    markup: Presence


def assert_output_contract(
    response: dict,
    expected: ExpectedOutput,
    parse_markup: Callable[[str], object],
) -> None:
    assert response.get("type") == expected.response_type

    markup = response.get("markup")
    if expected.markup is Presence.REQUIRED:
        assert markup, "expected markup but received none"
    elif expected.markup is Presence.FORBIDDEN:
        assert not markup, "received markup where none was allowed"

    if markup:
        assert parse_markup(markup) is not None

Presence and validity are now separate statements. Required markup must exist; any emitted markup must parse.

Make the case loader fail closed

This section ensures the fix cannot reproduce the same silent-pass behavior one layer earlier. Unknown case IDs, keys, missing expectations, and vocabulary values should stop the run.

A warning is insufficient here. A misspelled key that gets ignored makes a case appear configured while removing its assertion.

Report expectation coverage

This section makes incomplete migration visible. A suite can support contracts while most of its cases still rely on negative validation. Report total cases, cases with declared expectations, and the identifiers still missing them.

Render the count and missing case IDs in every report. Decide separately whether incomplete coverage blocks CI. Visibility is required even during a gradual rollout.

Verify that cases reach the behavior they claim to test

This section addresses vacuous coverage at the execution-path level. A markup contract does not test rendering behavior if no case reaches the component that produces markup.

Capture stable lifecycle events during each synthetic run and let the case declare required events. A render case might require flow.selected, view.generated, and output.parsed. The assertion should fail when any required event is absent.

Do not assert private class or function names. Stable behavior-level events survive refactoring and make public examples independent of an internal architecture.

The parser itself is part of the test dependency graph. If it cannot load, mark the check unavailable and fail any run that relies on it. Treating an unavailable checker as a pass turns infrastructure failure into false evidence.

Positive contracts state what each case must produce, strict loaders protect those declarations, coverage reports expose missing expectations, and path assertions prove the intended behavior ran. Together they close the silent routes by which a test can stay green while the feature disappears.