Crawl a website
Add a public website to an agent and check that its pages are ready for retrieval.
In Sources → Content, select the Website tab. The Crawl Website card finds pages. It starts from one public URL. Use it for a documentation area, help center, or policy site with several related pages. Not every reachable page is useful knowledge. Keep the crawl focused on the pages you need.
Start a crawl
Enter a URL. Choose Max Pages. Then select Fetch. If you enter a host name without a scheme, AgentStack adds HTTPS. The default maximum is 10 pages. The value must be at least 1. The workspace's plan also sets a maximum: 50 pages for Free, 200 for Hobby, 500 for Standard, and 1,000 for Pro or Enterprise.
Know the credit cost
A website crawl spends credits. One page costs 0.25 credits, so 4 pages cost 1 credit. AgentStack charges the pages that the crawl collected, after the crawl stops. The Free and Sandbox plans do not pay for crawls.
Before the crawl starts, AgentStack compares the balance against the Max Pages value, because that is the largest amount the crawl can cost. A 500-page limit needs 125 credits of headroom, even when the site has 40 pages. If the balance is lower, the crawl does not start and the table shows the reason. Select a Max Pages value that matches the material you need. See Manage billing and credits for balances and top-ups.
If the starting URL is broader than the material you want, open Advanced Options. Include Patterns and Exclude Patterns accept one pattern per line. Both support the asterisk as a wildcard. Include patterns restrict the candidate set first. Exclude patterns then remove matching pages. Use an include pattern for the documentation area. Do not write an exclude pattern for each unrelated page.
Read the result
The Websites table shows a status, a page count, an approximate token count, and any failure message. The token count tells you how much content the crawl indexed. A completed crawl with a page count but no token count did not index usable text.
- Crawling: the crawl finds pages.
- Processing: the table can show progress for each page.
- Completed: the crawl is finished.
- Failed: the table shows the recorded error.
A completed crawl can have zero pages. Open the Pages tab and check the target content before you test the agent.
If you submit the same normalized website URL again, AgentStack restarts or replaces that crawl. This does not create a duplicate source. When the site structure changes, submit the URL again on purpose. If a website is no longer authoritative, use the delete control to remove it.
After a source changes, read Retrain knowledge to refresh it. If you have one known URL, use Crawl a single page instead.