Agent best practices

The most dependable AgentStack agents start small and earn a broader scope through testing. If you add more instructions, sources, channels, and actions at once, failures become harder to diagnose. Build a reliable answer loop first. Then expand it in careful steps.

Define one job

Before you configure the agent, write the job in one sentence. For example, "Answer product and account questions for existing customers" is a testable job. "Help everyone with everything" is not testable.

Turn that job into a system prompt with explicit boundaries:

• who the agent serves
• which topics the agent answers
• the tone and level of detail
• when the agent asks a clarifying question
• how the agent phrases uncertainty when it lacks the knowledge to answer
• when the agent offers human help

Avoid long lists of overlapping rules. Put durable behavior in the system prompt and factual product information in sources. AgentStack's retrieval-confidence and handoff settings also change what happens when the agent lacks knowledge. Do not treat the system prompt as the only safeguard. This separation makes updates safer. To change a policy, edit the policy source. You do not need to search through the prompt's instructions.

Prefer authoritative, focused sources

Start with content that has a clear owner and a known update process. A concise policy page usually works better than a large site full of duplicated navigation, stale announcements, and unrelated marketing copy.

Before testing:

• Wait for the source to finish processing (website and page status show Completed, file processing finishes, or Notion sources show Synced).
• Check that the page is indexed.
• Remove outdated or conflicting content.
• If no suitable source exists, add a focused Q&A with one canonical answer.
• If the source workflow requires retraining, retrain the agent after you change the source content.

More content is not automatically better. Conflicting sources force the model to choose between competing facts. This makes the answer difficult to audit.

Build a repeatable test set

Do not judge agent quality only by an open-ended conversation that "feels good." Keep a small list of prompts with expected outcomes. Include common questions, paraphrases, ambiguous inputs, unsupported questions, and prohibited requests.

When a test fails, classify it before you change anything:

• Missing or incorrect fact: check the source.
• Wrong tone, structure, or boundary: check the system prompt.
• Retrieves the right evidence but reasons poorly: compare models.
• Works in Playground but not on the site: check the saved configuration, the Public setting, domain protection, and the embed installation.

Change one layer. Then rerun the same prompts. This produces evidence that the change helped. It does not hide the original problem.

Choose models with real examples

Use the default model as a baseline. In Playground, Compare Models can send the same message to multiple panels. Each panel can use its own model, temperature, and system prompt. The interface shows the total credit cost for the synced message.

Compare on your hardest representative questions, not on greetings. If a panel's model and system prompt perform better across the test set, save them to the agent. Do not save them because one answer sounds more polished. Temperature is a comparison-only control. The agent does not save it. The model guide explains the quality, latency, and credit trade-offs.

Deploy gradually

Begin with one channel and a limited audience. Check the real customer experience in a private browser session. Then watch Activity and Analytics for language and failure modes that your test set missed.

After the answer-only agent is stable, add actions. You can review and correct an answer. An action can change another system. Keep action scopes narrow. State confirmation requirements clearly. Test failure behavior as carefully as you test the happy path.

To set up an agent step-by-step, follow Create your first agent. To grade responses, continue to Improve response quality.