Crawl a website
Add a public website to an agent and check that its pages are ready for retrieval.
In Sources → Content, select the Website tab. The Crawl Website card finds pages. It starts from one public URL. Use it for a documentation area, help center, or policy site with several related pages. Not every reachable page is useful knowledge. Keep the crawl focused on the pages you need.
Start a crawl
Enter a URL. Choose Max Pages. Then select Fetch. If you enter a host name without a scheme, AgentStack adds HTTPS. The default maximum is 10 pages. The value must be at least 1. The workspace's plan also sets a maximum: 50 pages for Free, 200 for Hobby, 500 for Standard, and 1,000 for Pro or Enterprise.
If the starting URL is broader than the material you want, open Advanced Options. Include Patterns and Exclude Patterns accept one pattern per line. Both support the asterisk as a wildcard. Include patterns restrict the candidate set first. Exclude patterns then remove matching pages. Use an include pattern for the documentation area. Do not write an exclude pattern for each unrelated page.
Read the result
The Websites table shows a status, a page count, and any failure message.
- Crawling: the crawl finds pages.
- Processing: the table can show progress for each page.
- Completed: the crawl is finished.
- Failed: the table shows the recorded error.
A completed crawl can have zero pages. Open the Pages tab and check the target content before you test the agent.
If you submit the same normalized website URL again, AgentStack restarts or replaces that crawl. This does not create a duplicate source. When the site structure changes, submit the URL again on purpose. If a website is no longer authoritative, use the delete control to remove it.
After a source changes, read Retrain knowledge to refresh it. If you have one known URL, use Crawl a single page instead.