A confident wrong answer costs more than no answer
A visitor who is told "I cannot confirm that, here is how to reach us" is mildly inconvenienced. A visitor who is told the return window is thirty days when it is fourteen, or that the curry is nut free when nobody checked, has been given a reason to never trust anything on your site again, and possibly a reason to call a lawyer. General-purpose assistants are trained to be helpful, and helpful, when the model does not know, means plausible. Plausible is the failure mode.
Grounded answers: only from what you gave it
The fix is structural, not a polite instruction in a prompt. A grounded agent answers from a specific body of content, your documents and pages, that is searched for every question. The model is shown the relevant passages and asked to answer from them. If the passages do not contain the answer, there is nothing to answer from, and the right output is a refusal.
The citation is how you know it worked. Every answer should link back to the passage it came from, so the visitor can check it and so you can see, when an answer is wrong, whether the content was wrong or the agent was.
The refusal is a product property, not a prompt
Anyone can write "only answer from the context" into a prompt. The question is what happens under pressure: a vague question, a question that is almost covered, a question in another language. How strictly the agent holds the line should be a setting you control, the rate at which it declines should be measured and visible to you, and the behavior should be pinned by tests so an update to the model cannot quietly loosen it. In our case every one of those is true, and the measurement is the one that matters: when we tuned the smaller models to answer more often, every lever that raised the answer rate eroded the refusals. We left the refusals alone.
What a good refusal looks like
It is short, it is specific about what it cannot confirm, and it does something useful next:
- It names the gap. "I do not have information about whether that dish contains sesame" beats "I am not sure about that."
- It offers a person. With lead capture on, the agent takes a name and a way to reach them, and the question lands in your inbox with the conversation attached.
- It records the question. A refusal nobody sees is a lost chance to write the page that would have answered it.
Turning refusals into content
This is where the refusal stops being a cost and starts paying. Every question the agent declined is grouped by topic with a trend, so the list of unanswered questions is also the list of what to write next. A clinic finds out that nobody can tell from its site whether it takes a particular insurer. A store finds out its size guide does not cover the one measurement people ask about. The content gets written, the next refusal does not happen, and the agent gets better without anyone touching it.
Questions to ask any vendor
Including us. If the answers are vague, the refusals will be too.
- Where exactly do answers come from, and can I see the source for each one?
- What does it say when my content does not cover the question? Show me, do not tell me.
- Can I see every question it declined, and how often it declines?
- Can I make it stricter, and is that behavior tested when the model changes?
- When it declines, what happens to the visitor? Is there a path to a person?