Improve response quality
Diagnose weak answers by testing sources, instructions, and models separately in Playground.
Response quality is easier to improve when you identify which layer failed. A weak answer may come from missing knowledge, ambiguous instructions, an unsuitable model, or a Playground candidate that was tested but never saved to the agent. Playground gives you controls for testing these causes without changing everything at once.
Start with an acceptance set
Write down the expected result before you test. A useful starter set contains:
| Test | What a passing response does |
|---|---|
| Known fact | Returns the exact policy, number, or step supported by a source |
| Paraphrase | Finds the same fact when the customer's wording differs |
| Ambiguous request | Asks for the information needed to answer safely |
| Unsupported request | Admits the knowledge is insufficient instead of inventing an answer |
| Boundary test | Refuses or redirects a request outside the agent's defined job |
| Multi-part question | Answers each supported part and separates any uncertainty |
Keep these prompts stable while you tune the agent. A moving test set makes before-and-after comparisons unreliable.
Test prompt changes without saving them
Open the agent's Playground. Edit System Prompt or choose another Model, then select Apply & Test. The chat is recreated with those Playground overrides so you can test the candidate configuration.
The Unsaved changes badge means the editor differs from the agent's stored configuration. Apply & Test does not persist those changes. Select Save to Agent only after the candidate passes your test set. Select Reset to return the editor to the last saved model and prompt.
This distinction is important when a live widget behaves differently from a successful Playground test: first confirm that the winning configuration was saved to the agent.
Diagnose the failure before changing the model
Use this order:
- Verify the source. Confirm processing completed, the relevant page is indexed, and its text contains the expected fact.
- Remove conflicts. Look for older pages or Q&A entries that give a different answer.
- Clarify the system prompt. State the boundary or response behavior directly, without burying it in repeated prose.
- Retest the exact prompt. Confirm the change fixed the original failure and did not break the other acceptance cases.
- Compare models. Escalate the model only when the knowledge and instructions are already sound.
A model cannot reliably recover a fact that never entered the knowledge base. Conversely, adding more source text will not fix an instruction that rewards confident guessing.
Compare candidate configurations
Select Compare Models in Playground to open synchronized panels. Comparison mode starts with multiple panels and supports up to four. Each active panel can have its own model, temperature, and system prompt. A shared input sends the same message to every active panel, and the header shows the combined credit cost before you send.
Every panel shares the agent's saved sources and retrieval settings, so comparison mode varies the model, prompt, and temperature—not the knowledge configuration. It also does not show the source chips available in normal Playground mode. Use normal mode when you need to inspect which sources supported an answer.
For a fair comparison:
- keep the system prompt and temperature equal when comparing models;
- keep the model equal when comparing prompt variants;
- use the same prompts in the same order;
- grade factual correctness before style;
- record which failures each candidate makes;
- repeat important tests because model output can vary.
When one panel wins across the set, use that panel's Save to Agent control. This saves its model and system prompt; temperature remains a comparison-only setting. Do not choose based on a single unusually good response.
Write instructions that can be evaluated
Prefer observable rules. “Use no more than five sentences unless the customer requests detail” can be checked. “Be excellent” cannot. Tell the agent how to handle missing knowledge, but do not script a false answer for every possible gap.
A practical system prompt usually covers role, audience, source discipline, response format, uncertainty, and escalation. Keep volatile product facts out of it. Those facts belong in sources so their owner can update them independently.
Verify the deployed experience
After saving a configuration, repeat the acceptance set on the actual channel. For the website widget, confirm that the agent is Public, the current domain is allowed, and the current embed code is installed. Use Activity to verify that the live conversation reached the expected agent.
If you need a starting prompt and deployment sequence, return to Create your first agent. For durable operating habits, see Agent best practices.