Crawl a website
Add a public website to an agent and verify that its pages are ready for retrieval.
In Sources → Content, select the Website tab. The Crawl Website card discovers pages beginning from one public URL. It is the right choice when a documentation area, help center, or policy site has several related pages. It is not a promise that every reachable page is appropriate knowledge; keep the crawl focused.
Start a crawl
Enter a URL and choose Max Pages, then select Fetch. AgentStack adds the HTTPS scheme when you enter a host without one. The default maximum is 10 pages. The value must be at least one and is also bounded by the workspace’s plan: Free 50, Hobby 200, Standard 500, and Pro or Enterprise 1,000 pages per crawl.
Open Advanced Options when the starting URL is broader than the material you want. Include Patterns and Exclude Patterns accept one pattern per line and support the asterisk as a wildcard. Includes first restrict the candidate set; excludes then remove matching pages. Prefer an include pattern for the documentation area instead of trying to exclude every unrelated page.
Read the result
The Websites table shows a status, page count, and any failure message. Crawling means discovery is underway; Processing can show page-by-page progress; Completed is terminal; Failed exposes the recorded error. A completed crawl may legitimately have zero pages, so open the Pages tab and verify the target content before testing the agent.
Submitting the same normalized website URL again restarts or replaces that crawl instead of creating a duplicate source. Use this intentionally when the site structure changes. Use the delete control to remove a website that is no longer authoritative.
For refreshes after a source changes, read Retrain knowledge. For one known URL, use Crawl a single page instead.