Fix an AI answer that invents facts missing from your documents

A support assistant confidently describes a refund policy that is not in its source notes. Adding a more forceful instruction to be accurate does not supply the missing evidence.

Why this happens

The model may fill gaps with plausible language when retrieval returns irrelevant or incomplete information. A source citation can also be present without actually supporting the claim.

Step-by-step solution

1. Inspect the exact retrieved passages, not just the document titles. Check whether they contain the answer and apply to the current product version.

2. Require the answer to distinguish supported statements from missing information. Provide an explicit route to say that the available sources do not establish the answer.

3. Evaluate each material claim against its cited passage. Escalate unsupported policy decisions rather than presenting them as established company rules.

Worked example

If the notes describe delivery times but say nothing about refunds, the correct supported result is that the refund policy was not found. A delivery document is not evidence for a refund deadline.

Verify the fix

Test an answerable question, an unrelated question and a question whose answer was removed from the notes. The last two should not produce invented policy details.

Prevent a repeat

Track retrieval failures separately from generation failures. Keep source ownership and update dates visible so stale evidence can be replaced.

Reference

Official documentation or standards guidance

An original Gistmax troubleshooting guide. Scenarios and sample values are illustrative; configuration depends on your software version and environment. Cover diagram: Gistmax Tutorials.