Firecrawl API SOP

Appearance

Testing Webhooks ​

Updated Feb 2026

Since webhooks require Firecrawl to reach your server from the internet, you need a strategy for testing during local development. This page covers tools for exposing your local server, debugging payloads, and troubleshooting common issues.

Local Development ​

Your local development server (e.g., localhost:3000) is not accessible from the internet. You need a tunnel service to create a public URL that forwards traffic to your local machine.

Cloudflare Tunnels provide a free, no-account-required way to expose your local server without opening firewall ports:

bash
# Install cloudflared (macOS)
brew install cloudflare/cloudflare/cloudflared

# Install cloudflared (Windows - via winget)
winget install Cloudflare.cloudflared

# Start a tunnel to your local server
cloudflared tunnel --url localhost:3000

You will get a public URL like https://abc123.trycloudflare.com. Use this in your webhook configuration:

json
{
  "webhook": {
    "url": "https://abc123.trycloudflare.com/webhook/firecrawl",
    "events": ["started", "page", "completed"]
  }
}

Tunnel URL Changes

Cloudflare quick tunnels generate a new random URL each time you restart cloudflared. For persistent URLs, set up a named tunnel with a Cloudflare account.

Using ngrok ​

ngrok is another popular option with a dashboard for inspecting payloads:

bash
# Install ngrok
npm install -g ngrok

# Start a tunnel
ngrok http 3000

ngrok provides a web inspection interface at http://localhost:4040 where you can see all incoming webhook requests, their headers, and bodies.

Using webhook.site ​

For quick payload inspection without running any local code, use webhook.site:

  1. Visit webhook.site -- you get a unique URL immediately
  2. Use that URL as your webhook endpoint
  3. All incoming requests are logged in the browser with full headers and body

This is ideal for verifying that Firecrawl is sending webhooks correctly before writing any handler code.

Debugging Payloads ​

Minimal Logging Server ​

Create a simple server that logs every webhook payload to the console:

javascript
import express from 'express';

const app = express();
app.use(express.json());

app.post('/webhook/firecrawl', (req, res) => {
  console.log('--- Webhook Received ---');
  console.log('Headers:', JSON.stringify(req.headers, null, 2));
  console.log('Type:', req.body.type);
  console.log('Success:', req.body.success);
  console.log('Job ID:', req.body.id);
  console.log('Data count:', req.body.data?.length || 0);
  console.log('Metadata:', JSON.stringify(req.body.metadata, null, 2));

  if (req.body.error) {
    console.error('Error:', req.body.error);
  }

  console.log('------------------------');
  res.status(200).send('ok');
});

app.listen(3000, () => {
  console.log('Webhook debug server running on port 3000');
});

Python Logging Server ​

python
from flask import Flask, request

app = Flask(__name__)

@app.route('/webhook/firecrawl', methods=['POST'])
def webhook():
    event = request.get_json()
    print(f"\n--- Webhook Received ---")
    print(f"Type: {event.get('type')}")
    print(f"Success: {event.get('success')}")
    print(f"Job ID: {event.get('id')}")
    print(f"Data items: {len(event.get('data', []))}")

    if event.get('error'):
        print(f"Error: {event['error']}")

    print(f"------------------------\n")
    return 'ok', 200

if __name__ == '__main__':
    app.run(port=3000, debug=True)

Retry Behavior ​

Understanding retry behavior is essential for testing. When your endpoint fails to respond:

ScenarioFirecrawl Behavior
Your server returns 200-299Delivery successful, no retry
Your server returns 4xx or 5xxRetry after 1 min, then 5 min, then 15 min
Connection timeout (>10 seconds)Retry after 1 min, then 5 min, then 15 min
DNS resolution failureRetry after 1 min, then 5 min, then 15 min
All 3 retries failWebhook marked as failed, no more attempts

Testing Retries

To test retry behavior, intentionally return a 500 status from your endpoint for the first few requests, then switch to 200. You should see the retried deliveries arrive at the expected intervals.

Testing Checklist ​

Use this checklist when setting up webhooks for the first time:

  • [ ] Endpoint accessible -- Verify your URL is reachable from the internet (try curl from an external machine)
  • [ ] Using HTTPS -- Webhook URLs must use HTTPS, not HTTP
  • [ ] Correct events filter -- Check the events array in your webhook config matches what you expect
  • [ ] Response time -- Ensure your endpoint responds within 10 seconds
  • [ ] Signature verification -- If implementing security, test with the correct raw body approach
  • [ ] Idempotency -- Handle the possibility of receiving the same event more than once (retries can cause duplicates)
  • [ ] Error handling -- Test with failed events (extract.failed, agent.failed) to ensure your handler processes errors gracefully

Troubleshooting ​

Webhooks Not Arriving ​

SymptomSolution
No requests at allVerify your server is publicly reachable and firewalls allow incoming HTTPS connections
Using http:// URLChange to https:// -- Firecrawl requires HTTPS
Events missingCheck the events filter in your webhook config. Remove the filter to receive all events.
Tunnel URL expiredRestart your tunnel service and update the webhook URL

Signature Verification Failing ​

The most common cause is using the parsed JSON body instead of the raw request body:

javascript
// WRONG -- parsed body may differ from raw bytes
const hash = crypto
  .createHmac('sha256', secret)
  .update(JSON.stringify(req.body))
  .digest('hex');

// CORRECT -- use raw body middleware
app.use('/webhook', express.raw({ type: 'application/json' }));
app.post('/webhook', (req, res) => {
  const hash = crypto
    .createHmac('sha256', secret)
    .update(req.body) // Raw buffer
    .digest('hex');
});

Other Common Issues ​

IssueSolution
Wrong secretVerify you are using the correct secret from your account settings
Timeout errorsEnsure your endpoint responds within 10 seconds. Return 200 immediately and process async.
Duplicate eventsImplement idempotency using the id field to deduplicate. Retries can cause the same event to arrive multiple times.