REST API: sources and ingestion

Source automation follows the same lifecycle as the dashboard. It creates or finds a source. Then it waits for the source to finish processing. It retrains the agent when the new knowledge is ready.

Website crawling and page crawling are different operations. For files, first get an upload URL. Then request processing for the returned file identifier. Keep file bytes and API credentials on trusted infrastructure only.

Poll for status during asynchronous ingestion. Request retrain logs only as often as you need to check progress. Use bounded backoff. Do not restart a crawl or retrain job many times. A successful submission means the system accepted the work. It is not proof that every page, file, or external connection finished indexing.

List and delete operations are source-type-specific. Do not substitute a website identifier for a page, Q&A, file, or Notion source identifier. Before a destructive cleanup, check which source the agent currently uses. Record enough information so that a person can recover from an accidental removal. See the generated reference for accepted crawl and upload payloads.

Use POST /agents/{id}/sources/notion/{notionSourceId}/sync to trigger a manual re-sync of an already-enabled Notion source, for example after editing the page in Notion and needing the change reflected sooner than the next scheduled sync. This endpoint requires the source to already exist; use the Notion connection guide to find and enable a Notion page as a source in the first place.

Deleted pages

DELETE /agents/{id}/sources/pages/{pageId} removes one page of a crawled website and its content at once. The delete sticks. A later crawl, a manual retrain, an automatic retrain, and a website restore all skip that exact URL, even when the website still links to it. Other pages of the website are not affected, and the website's excludePatterns do not change. To index the page again, call POST /agents/{id}/sources/pages/{pageId}/restore with the same page identifier. The restore lifts the delete and queues a crawl of that one page, so the page returns to the list after the crawl finishes. A restore answers 409 CONFLICT when the page is not deleted, or when its website is deleted; restore the website first. Adding the same URL again through POST /agents/{id}/ingest/crawl-page also lifts the delete, and so does crawling the website again by its own URL when the deleted page has that URL. Both endpoints need the admin scope.

Crawl patterns

includePatterns and excludePatterns on POST /agents/{id}/ingest/crawl-website are plain text. A pattern matches when the text occurs anywhere in the page URL. The asterisk * matches any run of characters. There are no anchors and no regular expressions. Write /docs/ or /docs/*/guide. Do not write ^/docs/.*$ or /docs/**. The request is refused with 400 INVALID_INPUT when a pattern has a leading ^, a trailing $, .*, **, or a backslash. The error names the correct form. The refusal comes before the crawl starts, so it spends no crawl budget. A retrain reuses the patterns that are stored on the source and does not check them again.

Paging the source lists

GET /agents/{id}/sources/websites, .../sources/pages, .../sources/qnas, and .../sources/notion return a bare JSON array. Without limit, the array holds every item, exactly as before, and the response has no paging headers. With limit (1 to 100), the array holds one page, newest first, and the response carries two headers:

• X-Has-More: true when another page follows, false on the last page.
• X-Next-Cursor: the id of the last item in this page. It is sent only when another page follows. Pass it as after, with the same limit, to fetch that page.

Items created at the same moment are ordered by id, so a page boundary never skips or repeats one. An after whose item was deleted, or belongs to another agent, returns an empty array with X-Has-More: false. after is read only together with limit: without limit the list is always every item. GET /agents/{id}/sources/files is not paged: its limit field is the file quota, the most files one agent may hold, not a page size, and files is every file.

Changes

• 2026-10-17 — The websites, pages, QnAs, and Notion source lists accept limit and after. With limit, the response is one page plus the X-Has-More and X-Next-Cursor headers. Without limit, the response is unchanged: the whole array.

• 2026-10-11 — Deleting a page of a crawled website with DELETE /agents/{id}/sources/pages/{pageId} sticks: crawls and retrains skip that URL. Before this change, the next retrain found the URL again and indexed the page again. New endpoint POST /agents/{id}/sources/pages/{pageId}/restore (operation sources.pages.restore) undoes the delete and crawls the page again.

• 2026-09-24 — POST /agents/{id}/ingest/crawl-website refuses a crawl pattern that looks like a regular expression or a glob. Before this change, such a pattern was accepted, matched no pages, and spent the crawl budget. The dashboard crawl form uses the same check.

• 2026-09-24 — The MCP source tools get_source, list_sources, create_source, and restore_source return a source status of indexing, ready, or failed. list_sources also returns deleted. Before this change, a website or file source showed the raw pipeline word, for example completed, and never showed ready. Stop polling on ready or failed.