Crawl websites

Create a crawl job, wait for completion, then iterate pages. Scope with patterns and limit with depth/link_limit.

Note: Install the SDKs if you haven’t already: https://docs.supacrawler.com/api/install

Quickstart

curl https://api.supacrawler.com/api/v1/crawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/docs",
    "type": "crawl",
    "depth": 2,
    "link_limit": 20,
    "patterns": ["/docs/*"],
    "format": "markdown"
  }'

Tips:

Narrow the scope with patterns to avoid crawling marketing/blog
Raise depth gradually; start small to verify targets
Consider render_js for SPA docs if needed

Links: Jobs API