An AI update changes useful answers: keep a regression set

Record the change that actually occurred

List the release time and every known change: model identifier, system instructions, generation settings, source content, indexing, retrieval filters, interface or tool permissions. Include changes made by a managed platform even when you did not edit the application.

Ask whether a model alias moved or a specific version was selected. Google’s model documentation distinguishes stable identifiers from latest aliases that can point to a newer release. That example shows why a friendly model label is insufficient for a change record; check the actual provider’s current behavior.

Do not promise identical output from a fixed identifier. Record repeatable conditions and assess meaning. Nondeterministic generation and external source changes can still affect the result.

Preserve cases that represent real obligations

Each regression case needs a question, relevant conversation state, expected facts, source version and failure boundary. Retain examples that previously exposed important weaknesses, including missing information, contradictory documents and requests beyond the assistant’s authority.

Use the fact-checking workflow to keep the answer key tied to approved evidence. A saved old answer can be a useful comparison artifact, but it is not automatically correct.

When a business policy legitimately changes, update the expected facts through an explicit review and keep the previous edition. Otherwise a correct new policy answer may be mislabeled as a regression simply because the test still expects the old deadline.

A fictional delivery assistant changes its advice

Imagine a fictional website whose approved delivery page says orders received before noon are considered for next-day dispatch, subject to availability. The old assistant includes the condition. After an update, it promises next-day delivery for every order.

The case’s expected facts distinguish dispatch from arrival and preserve the availability condition. That makes the failure clear even if both answers sound helpful. The owner saves both outputs and their citations rather than describing the new answer only as worse.

Investigation shows the source page is unchanged, but a retrieval setting now supplies a short marketing paragraph instead of the detailed policy passage. Reverting the model would not directly repair that demonstrated cause. The proposed correction is to restore the appropriate source selection and rerun the case.

This example is fictional. A different real investigation could implicate the prompt, model or source revision. The point is the evidence sequence, not an assumed universal cause.

Compare facts, support and boundaries

Use the same scoring dimensions before and after: useful answer, source support and correct limits. Anthropic’s evaluation guidance emphasizes task-specific criteria and consistency. Keep those criteria stable during the comparison.

Separate harmless style changes from material differences. A shorter introduction may be acceptable. A missing exception, changed amount, wrong source or false claim of a completed action is more consequential. Assign severity based on the actual website task.

Do not let improvements on easy questions cancel a serious new failure. A useful comparison report can show overall results while clearly listing cases that block the affected release or route.

Look behind a missing citation

For retrieval-based assistants, preserve the passages supplied to the answer where the platform exposes them. If the relevant source is absent, distinguish ingestion from retrieval selection. Our source-ingestion diagnostic covers the earlier discovery and indexing chain for that platform.

If the correct passage was supplied but its qualification was omitted, the answer-generation layer deserves closer inspection. If a citation points to a new document edition, review the expected answer before calling the behavior wrong.

Keep diagnostic records private and minimized. A regression investigation does not require copying entire customer conversations or hidden credentials into a shared test file. Use a synthetic reproduction when it preserves the important conditions.

Retain a safe release decision

Google’s evaluation overview includes inspecting individual responses as well as aggregate metrics. Apply that habit to the release record: identify which cases improved, which regressed and what remains unexplained.

If a supported rollback is available, retain the previous approved configuration until the affected behavior is resolved. If the previous model is unavailable, restrict the risky route or provide a human fallback instead of pretending the old system can be restored.

For actions beyond answering, preserve the human-review boundary. A model migration is not authorization to expand tool access or remove approval. Recheck retry and cancellation behavior when the update changes action execution.

Preserve the raw responses alongside the review labels. If a reviewer later challenges a score, the actual missing condition or changed claim should still be inspectable. A summary saying five cases passed cannot explain whether the same five cases passed under both configurations or whether different failures cancelled out in the aggregate.

Maintain the set without teaching only to the test

Add a new case when it covers a meaningful failure or new obligation. Keep some review examples separate from everyday prompt tuning, so the comparison still checks generalization. Avoid rewriting every question into the exact phrasing the assistant already handles well.

Record the accepted configuration, actual results, reviewer and unresolved limits. Your next step is a compact change manifest and a repeatable run of important cases. That makes a model update a reviewable release, rather than an unexplained change in the website’s promises.