Scrape

The Scrape API allows you to extract clean, structured content from any webpage. Get markdown, HTML, or plain text content along with metadata like title and status codes. When using format=links, it transforms into a powerful link discovery tool for site mapping.

Quick example

Extract clean markdown content from any webpage with a simple GET request:

curl -G https://api.supacrawler.com/api/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d url="https://example.com" \
  -d format="markdown"

{
  "success": true,
  "url": "https://example.com",
  "content": "# Example Domain\n\nThis domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.\n\n[More information...](https://www.iana.org/domains/example)",
  "title": "Example Domain",
  "metadata": {
    "status_code": 200
  }
}

The scrape response model

The scrape response contains the extracted content and metadata about the webpage.

Properties

Scrape a webpage

curl -G https://api.supacrawler.com/api/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d url="https://example.com" \
  -d format="markdown" \
  -d render_js=false

{
  "success": true,
  "url": "https://example.com",
  "content": "# Example Domain\n\nThis domain is for use in illustrative examples in documents...",
  "title": "Example Domain",
  "metadata": {
    "status_code": 200
  }
}

Map website links

curl -G https://api.supacrawler.com/api/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d url="https://example.com" \
  -d format="links" \
  -d depth=2 \
  -d max_links=100

{
  "success": true,
  "url": "https://example.com",
  "links": [
    "https://example.com/about",
    "https://example.com/contact",
    "https://example.com/products",
    "https://example.com/blog"
  ],
  "discovered": 4,
  "metadata": {
    "status_code": 200,
    "depth": 2
  }
}

Content formats

The scrape API supports multiple output formats to suit different use cases:

Markdown (default)

Clean, structured markdown that preserves formatting while removing clutter. Perfect for content analysis and processing.

HTML

Raw HTML content with cleaning applied. Useful when you need to preserve specific HTML structure or styling.

Text

Plain text with all HTML markup removed. Ideal for text analysis, search indexing, or when you only need the textual content.

Links

Special format that returns all discovered links instead of page content. Great for site mapping and link analysis.

Error handling

The scrape API returns appropriate HTTP status codes and error messages:

{
  "success": false,
  "url": "https://invalid-url.com",
  "error": "Invalid query parameters",
  "metadata": {
    "status_code": 400
  }
}

Best practices

JavaScript-heavy sites

For websites that rely heavily on JavaScript (e.g., Single Page Applications built with React, Vue, or Svelte), you must enable the render_js parameter. This uses a full browser to execute JavaScript and get the final, accurate content. For simple, static sites, leaving render_js=false (the default) will be significantly faster.

curl -G https://api.supacrawler.com/api/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d url="https://spa-example.com" \
  -d format="markdown" \
  -d render_js=true \
  -d wait=3000

Mobile vs Desktop

Different devices can return different content. Use the device parameter to specify. This only has an effect when render_js=true.

Caching behavior

Supacrawler automatically caches scraped content to improve performance and reduce costs:

• Regular scrapes: Cached for 5 minutes
• Render scrapes: Cached for 15 minutes (due to higher computational cost)
• Fresh parameter: Use fresh=true to bypass cache and get the latest content

curl -G https://api.supacrawler.com/api/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d url="https://news-site.com/article" \
  -d format="markdown"

curl -G https://api.supacrawler.com/api/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d url="https://news-site.com/article" \
  -d format="markdown" \
  -d fresh=true

Rate limiting

• Free tier: 100 requests per hour
• Pro tier: 1,000 requests per hour
• Enterprise: Custom limits

Space out your requests appropriately to avoid rate limiting.