Crawl a website

In Sources → Content, select the Website tab. The Crawl Website card finds pages. It starts from one public URL. Use it for a documentation area, help center, or policy site with several related pages. Not every reachable page is useful knowledge. Keep the crawl focused on the pages you need.

Start a crawl

Enter a URL. Choose Max Pages. Then select Fetch. If you enter a host name without a scheme, AgentStack adds HTTPS. The default maximum is 10 pages. The value must be at least 1. The workspace's plan also sets a maximum: 50 pages for Free, 200 for Hobby, 500 for Standard, and 1,000 for Pro or Enterprise. If the URL is a sitemap, such as sitemap.xml, sitemap_index.xml, or sitemap.xml.gz, AgentStack reads the pages that the sitemap lists, including nested sitemaps, instead of following links. It keeps only pages on the same site that match your Include Patterns and Exclude Patterns, up to the Max Pages value.

Know the credit cost

A website crawl spends credits. One page costs 0.25 credits, so 4 pages cost 1 credit. AgentStack charges the pages that the crawl collected. The Free and Sandbox plans do not pay for crawls.

Before the crawl starts, AgentStack holds the largest amount the crawl can cost: the Max Pages value, or the page count of a sitemap. A 500-page limit holds 125 credits, even when the site has 40 pages. When the crawl stops, the unused part of the hold comes back, so that 40-page crawl costs 10 credits. If the balance cannot cover the hold, the crawl does not start and the table shows the reason. Two crawls cannot share one hold, so parallel crawls each need their own headroom. Select a Max Pages value that matches the material you need. See Manage billing and credits for balances and top-ups.

If the starting URL is broader than the material you want, open Advanced Options. Include Patterns and Exclude Patterns accept one pattern per line. A pattern is plain text that matches anywhere in the page URL, and the asterisk matches any run of characters. The form refuses regular expression and glob syntax, such as ^, $, .*, **, or a backslash. Write /docs/ or /docs/*/guide, not ^/docs/.*$ or /docs/**. Include patterns restrict the candidate set first. Exclude patterns then remove matching pages. Use an include pattern for the documentation area. Do not write an exclude pattern for each unrelated page.

Read the result

The Websites table shows a status, a page count, an approximate token count, and any failure message. The token count tells you how much content the crawl indexed. A completed crawl with a page count but no token count did not index usable text.

• Crawling: the crawl finds pages.
• Processing: the table can show progress for each page.
• Completed: the crawl is finished.
• Failed: the table shows the recorded error.

A completed crawl can have zero pages. Open the Pages tab and check the target content before you test the agent.

If you submit the same normalized website URL again, AgentStack restarts or replaces that crawl. This does not create a duplicate source. When the site structure changes, submit the URL again on purpose. If a website is no longer authoritative, use the delete control to remove it.

After a source changes, read Retrain knowledge to refresh it. If you have one known URL, use Crawl a single page instead.