Agent best practices

The most dependable AgentStack agents start small and earn a broader scope through testing. Adding more instructions, sources, channels, and actions at once makes failures harder to diagnose. Establish a reliable answer loop first, then expand it deliberately.

Define one job

Write a one-sentence job before configuring the agent: “Answer product and account questions for existing customers” is testable; “Help everyone with everything” is not.

Turn that job into a system prompt with explicit boundaries:

• who the agent serves;
• which topics it should answer;
• the tone and level of detail;
• when it should ask a clarifying question;
• how it should phrase uncertainty when the available knowledge is insufficient;
• when it should offer human help.

Avoid long lists of overlapping rules. Put durable behavior in the system prompt and factual product information in sources. AgentStack's retrieval-confidence and handoff configuration also influence what happens when knowledge is missing, so do not treat the custom prompt as the only safeguard. This separation makes updates safer: a policy change should require changing the policy source, not hunting through behavioral instructions.

Prefer authoritative, focused sources

Start with content that has a clear owner and a known update process. A concise policy page is usually better than crawling a large site full of duplicated navigation, stale announcements, and unrelated marketing copy.

Before testing:

• wait for website and page processing to become Completed, file processing to finish, or Notion sources to become Synced, depending on the source type;
• confirm the relevant page was actually indexed;
• remove obsolete or contradictory material;
• use a focused Q&A for a single canonical answer when no suitable source exists;
• retrain after changing source content when the source workflow requires it.

More content is not automatically better. Conflicting sources force the model to choose between competing facts and make the resulting answer difficult to audit.

Build a repeatable test set

Do not evaluate quality by having an open-ended conversation and deciding that it “feels good.” Keep a small list of prompts with expected outcomes. Include common questions, paraphrases, ambiguous inputs, unsupported questions, and prohibited requests.

When a test fails, classify it before editing anything:

• Missing or incorrect fact: inspect the source.
• Wrong tone, structure, or boundary: inspect the system prompt.
• Retrieves the right evidence but reasons poorly: compare models.
• Works in Playground but not on the site: inspect the saved configuration, Public setting, domain protection, and embed installation.

Change one layer and rerun the same prompts. This produces evidence that the change helped instead of hiding the original problem.

Choose models with real examples

Use the default model as a baseline. In Playground, Compare Models can send the same message to multiple panels. Each panel can use its own model, temperature, and system prompt, and the interface displays the total credit cost for the synced message.

Compare on your hardest representative questions, not on greetings. Save a panel's model and system prompt to the agent only after they perform better across the test set, not because one answer sounded more polished. Temperature is a comparison-only control and is not saved to the agent. The model guide explains the quality, latency, and credit trade-offs.

Deploy gradually

Begin with one channel and a limited audience. Verify the real customer surface in a private browser session, then watch Activity and Analytics for language and failure modes that your test set missed.

Add actions only after the answer-only agent is stable. An answer can be reviewed and corrected; an action can change another system. Keep action scopes narrow, make confirmation requirements clear, and test failure behavior as carefully as the happy path.

For a concrete setup sequence, follow Create your first agent. For a grading workflow, continue to Improve response quality.