Watch

The Watch API lets you monitor websites for changes over time. Create watch jobs that automatically check for updates on your chosen schedule and notify you via email when content changes. Perfect for monitoring competitor websites, tracking product updates, or staying informed about important webpage modifications.

Quick example

Create a watch job to monitor a website for changes:

curl https://api.supacrawler.com/api/v1/watch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "frequency": "daily",
    "notify_email": "[email protected]",
    "notification_preference": "changes_only"
  }'

{
  "success": true,
  "watch_id": "550e8400-e29b-41d4-a716-446655440000",
  "message": "Watch job created successfully"
}

curl https://api.supacrawler.com/api/v1/watch/550e8400-e29b-41d4-a716-446655440000 \
  -H "Authorization: Bearer YOUR_API_KEY"

{
  "success": true,
  "watch": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "url": "https://example.com/pricing",
    "frequency": "daily",
    "notify_email": "[email protected]",
    "include_html": false,
    "include_image": false,
    "last_check": "2023-06-10T15:30:00Z",
    "last_notification": "2023-06-10T15:30:00Z",
    "status": "active",
    "created_at": "2023-06-01T12:00:00Z"
  },
  "results": [
    {
      "id": "650e8400-e29b-41d4-a716-446655440001",
      "executed_at": "2023-06-10T15:30:00Z",
      "has_changed": true,
      "change_type": "text",
      "content_hash": "a1b2c3d4e5f6..."
    },
    {
      "id": "750e8400-e29b-41d4-a716-446655440002",
      "executed_at": "2023-06-09T15:30:00Z",
      "has_changed": false,
      "change_type": null,
      "content_hash": "a1b2c3d4e5f6..."
    }
  ]
}

The watch response model

Watch responses contain information about the watch configuration and results.

Properties

Create a watch job

curl https://api.supacrawler.com/api/v1/watch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "frequency": "daily",
    "notification_preference": "changes_only",
    "notify_email": "[email protected]",
    "include_html": true
  }'

curl https://api.supacrawler.com/api/v1/watch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products",
    "frequency": "daily", 
    "notification_preference": "all_runs",
    "notify_email": "[email protected]",
    "include_image": true,
    "full_page": true,
    "quality": 90
  }'

{
  "success": true,
  "watch_id": "550e8400-e29b-41d4-a716-446655440000",
  "message": "Watch job created successfully"
}

Get watch job status

curl https://api.supacrawler.com/api/v1/watch/550e8400-e29b-41d4-a716-446655440000 \
  -H "Authorization: Bearer YOUR_API_KEY"

{
  "success": true,
  "watch_id": "550e8400-e29b-41d4-a716-446655440000",
  "url": "https://example.com/pricing",
  "status": "active",
  "frequency": "daily",
  "last_check": "2023-06-10T15:30:00Z",
  "has_changes": true,
  "results": [
    {
      "executed_at": "2023-06-10T15:30:00Z",
      "has_changed": true,
      "change_type": "text",
      "content_snippet": "Pricing updated with new enterprise tier...",
      "image_url": "https://storage.supacrawler.com/screenshots/watch-550e8400-e29b-41d4-a716-446655440000-2023-06-10.png"
    },
    {
      "executed_at": "2023-06-09T15:30:00Z",
      "has_changed": false,
      "change_type": "",
      "content_snippet": "No changes detected",
      "image_url": ""
    }
  ]
}

List all watch jobs

curl https://api.supacrawler.com/api/v1/watch \
  -H "Authorization: Bearer YOUR_API_KEY"

{
  "success": true,
  "total": 2,
  "watches": [
    {
      "id": "550e8400-e29b-41d4-a716-446655440000",
      "user_id": "user-123",
      "url": "https://example.com/pricing",
      "frequency": "daily",
      "notify_email": "[email protected]",
      "include_html": true,
      "include_image": false,
      "last_check": "2023-06-10T15:30:00Z",
      "status": "active",
      "created_at": "2023-06-01T12:00:00Z",
      "updated_at": "2023-06-10T15:30:00Z"
    },
    {
      "id": "660e8400-e29b-41d4-a716-446655440003",
      "user_id": "user-123",
      "url": "https://competitor.com/features",
      "frequency": "weekly",
      "selector": ".feature-list",
      "notify_email": "[email protected]",
      "include_html": false,
      "include_image": true,
      "last_check": "2023-06-07T12:00:00Z",
      "status": "active",
      "created_at": "2023-06-02T09:00:00Z",
      "updated_at": "2023-06-07T12:00:00Z"
    }
  ]
}

Delete a watch job

curl -X DELETE https://api.supacrawler.com/api/v1/watch/550e8400-e29b-41d4-a716-446655440000 \
  -H "Authorization: Bearer YOUR_API_KEY"

{
  "success": true,
  "message": "Watch job deleted successfully"
}

Frequency options

The watch API supports different frequency options to suit your monitoring needs:

Hourly

Check for changes every hour. Ideal for time-sensitive content that changes frequently.

Daily

Check for changes once per day (at approximately the same time the watch was created). This is the recommended option for most use cases.

Weekly

Check for changes once per week. Good for content that changes infrequently.

Monthly

Check for changes once per month. Suitable for very stable content with rare updates.

Custom

Specify your own schedule using a cron expression. This gives you maximum flexibility to define exactly when checks should happen.

# Every weekday at 9 AM
"0 9 * * MON-FRI"

# Every 6 hours
"0 */6 * * *"

# Every Monday and Thursday at 2 PM
"0 14 * * MON,THU"

Notification behavior

When a watch job detects changes, it will:

1. Record the change in your watch history
2. Send an email notification to the specified email address
3. Include relevant content based on your configuration:
   • Always: Text of what changed
   • If include_html: Full HTML content
   • If include_image: Screenshot of the page

Email notifications include:

• Subject line with the URL that changed
• Time of detection
• Type of change detected
• Link to view full details
• Content differences (highlighted where possible)

Best practices

Choosing the right frequency

• Start with less frequent checks (daily/weekly) and adjust as needed
• Consider the nature of the content you're monitoring
• Remember that higher frequency checks consume more credits

Using selectors effectively

For the most efficient monitoring, use CSS selectors to target specific elements:

curl https://api.supacrawler.com/api/v1/watch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "frequency": "daily",
    "notify_email": "[email protected]",
    "selector": "#pricing-table"
  }'

Common selector examples:

• .main-content - Target elements with class "main-content"
• #product-description - Target element with ID "product-description"
• table.pricing - Target pricing tables
• .blog-post p - Target paragraphs within blog posts

Reducing noise

To avoid notification fatigue:

• Monitor only specific elements that matter to you
• Choose appropriate check frequencies
• Consider using weekly monitoring for less critical content

Error handling

{
  "success": false,
  "error": "Invalid frequency specified. Must be one of: hourly, daily, weekly, monthly, custom"
}

Watch credits consumption

Watch jobs consume credits based on several factors. Understanding these costs will help you optimize your monitoring strategy.

Base cost calculation

For each check:

• Base cost: 10 credits
• JavaScript rendering: +5 credits (enabled by default)
• Include HTML: +2 credits
• Include screenshot (standard quality): +5 credits
• Include screenshot (high quality): +7 credits
• Full-page screenshot (standard quality): +8 credits
• Full-page screenshot (high quality): +10 credits

Frequency multipliers explained

The cost per month depends on how frequently your watch job runs:

• Hourly: 24 checks per day × 30 days = 720 checks per month
• Daily: 1 check per day × 30 days = 30 checks per month
• Weekly: 1 check per week × 4 weeks = 4 checks per month
• Monthly: 1 check per month = 1 check per month

Detailed examples with calculations

Common use cases and recommended configurations

Recommended Beta Parameters

For this beta release, we recommend focusing on these core parameters:

• Required: url, frequency, notify_email
• Recommended: selector (for targeted monitoring)
• Optional: include_html (for detailed content comparison)

Advanced screenshot options can be added as needed but may consume more credits.

Credit-saving tips

• Use selectors to monitor specific elements instead of entire pages
• Choose appropriate frequencies based on how often the content actually changes
• Disable JavaScript rendering for simple static pages (saves 5 credits per check)
• Only include screenshots when visual verification is necessary
• Use standard quality (80%) screenshots unless high fidelity is required
• Start with weekly monitoring and adjust based on change frequency