Batch Scrape

Updated Feb 2026

Scrape multiple URLs in a single operation. Batch scraping runs URLs concurrently, supports webhooks for real-time notifications, and returns results in the same formats as single-page scrape.

Overview

Batch scraping is ideal when you have a known list of URLs and want to process them efficiently. The operation runs asynchronously -- you start a job, optionally receive webhook notifications, and retrieve results when complete.

API Endpoints

Method	Path	Description
POST	`/v2/batch/scrape`	Start a batch scrape job
GET	`/v2/batch/scrape/{id}`	Get batch job status and results
DELETE	`/v2/batch/scrape/{id}`	Cancel a running batch job
GET	`/v2/batch/scrape/{id}/errors`	Get error details for failed URLs

Basic Usage

Python (Synchronous)

Blocks until all URLs are scraped:

python

from firecrawl import Firecrawl

firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")

# Synchronous — waits for completion
results = firecrawl.batch_scrape(
    [
        "https://example.com",
        "https://example.com/about",
        "https://example.com/pricing",
    ],
    formats=["markdown"],
    poll_interval=2,
    wait_timeout=120
)

for page in results['data']:
    print(page['metadata']['sourceURL'])
    print(page['markdown'][:200])
    print("---")

Python (Asynchronous)

Returns immediately with a job ID for polling:

python

# Start the batch job
job = firecrawl.start_batch_scrape(
    [
        "https://example.com",
        "https://example.com/about",
        "https://example.com/pricing",
    ],
    formats=["markdown", "html"]
)

print(f"Job ID: {job['id']}")

# Check status later
status = firecrawl.get_batch_scrape_status(job['id'])
print(f"Status: {status['status']}")
print(f"Completed: {status['completed']}/{status['total']}")

if status['status'] == 'completed':
    for page in status['data']:
        print(page['markdown'][:200])

Node.js (Synchronous)

javascript

import Firecrawl from '@mendable/firecrawl-js';

const firecrawl = new Firecrawl({ apiKey: "fc-YOUR-API-KEY" });

// Synchronous — waits for completion
const results = await firecrawl.batchScrape(
  [
    "https://example.com",
    "https://example.com/about",
    "https://example.com/pricing",
  ],
  { formats: ["markdown"] }
);

for (const page of results.data) {
  console.log(page.metadata.sourceURL);
  console.log(page.markdown.substring(0, 200));
}

Node.js (Asynchronous)

javascript

// Start the batch job
const job = await firecrawl.startBatchScrape(
  [
    "https://example.com",
    "https://example.com/about",
    "https://example.com/pricing",
  ],
  { formats: ["markdown", "html"] }
);

console.log(`Job ID: ${job.id}`);

// Check status later
const status = await firecrawl.getBatchScrapeStatus(job.id);
console.log(`Status: ${status.status}`);
console.log(`Completed: ${status.completed}/${status.total}`);

cURL

bash

# Start batch job
curl -X POST https://api.firecrawl.dev/v2/batch/scrape \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer fc-YOUR-API-KEY' \
  -d '{
    "urls": [
      "https://example.com",
      "https://example.com/about",
      "https://example.com/pricing"
    ],
    "formats": ["markdown"]
  }'

# Response: {"success": true, "id": "batch-abc123", "url": "https://api.firecrawl.dev/v2/batch/scrape/batch-abc123"}

# Check status
curl -X GET https://api.firecrawl.dev/v2/batch/scrape/batch-abc123 \
  -H 'Authorization: Bearer fc-YOUR-API-KEY'

# Cancel job
curl -X DELETE https://api.firecrawl.dev/v2/batch/scrape/batch-abc123 \
  -H 'Authorization: Bearer fc-YOUR-API-KEY'

Parameters

Parameter	Type	Default	Description
`urls`	string[]	Required	Array of URLs to scrape
`formats`	array	`["markdown"]`	Output formats (see Scrape formats)
`maxConcurrency`	number	Team limit	Max concurrent browsers for this job
`webhook`	string/object	--	Webhook URL or config for notifications
`pollInterval`	number	2	Seconds between status checks (SDK sync mode)
`timeout`	number	--	Max wait time in ms (SDK sync mode)
`only_main_content`	boolean	true	Strip navs, footers, sidebars
`proxy`	string	`"auto"`	Proxy mode: `basic`, `enhanced`, `auto`
`location`	object	--	Geographic targeting (`{country: "US"}`)

Concurrency Control

By default, a batch job uses your entire team's concurrent browser capacity. Use maxConcurrency to limit a single job's share:

python

results = firecrawl.batch_scrape(
    urls=url_list,
    formats=["markdown"],
    maxConcurrency=50  # Limit to 50 concurrent browsers
)

Large Batches

Setting maxConcurrency too low on large batches significantly slows processing. Find a balance between job isolation and throughput.

Structured Extraction (JSON Mode)

Extract structured data from every URL in the batch using a schema:

python

results = firecrawl.batch_scrape(
    [
        "https://example.com/product/1",
        "https://example.com/product/2",
        "https://example.com/product/3",
    ],
    formats=[{
        "type": "json",
        "prompt": "Extract the product name, price, and description",
        "schema": {
            "type": "object",
            "properties": {
                "product_name": {"type": "string"},
                "price": {"type": "string"},
                "description": {"type": "string"}
            },
            "required": ["product_name", "price"]
        }
    }]
)

for page in results['data']:
    print(page['json'])

Credit Cost

JSON mode adds +4 credits per page on top of the base 1 credit. A 100-URL batch with JSON extraction costs 500 credits. See Pricing.

Webhook Integration

Receive real-time notifications as your batch job progresses.

Configuration

python

# Simple webhook URL
results = firecrawl.batch_scrape(
    urls=url_list,
    formats=["markdown"],
    webhook="https://your-server.com/firecrawl-webhook"
)

# Webhook with specific events
results = firecrawl.batch_scrape(
    urls=url_list,
    formats=["markdown"],
    webhook={
        "url": "https://your-server.com/firecrawl-webhook",
        "events": ["batch_scrape.page", "batch_scrape.completed"]
    }
)

Webhook Events

Event	Fires When
`batch_scrape.started`	Job begins processing
`batch_scrape.page`	Each individual URL completes
`batch_scrape.completed`	All URLs finished successfully
`batch_scrape.failed`	Job fails

Webhook Payload

Each webhook request includes:

Body: JSON with event type, job ID, and page data (for .page events)
Header: X-Firecrawl-Signature -- HMAC-SHA256 signature for verification

Signature Verification

Retrieve your webhook secret from your account's Advanced settings, then verify:

python

import hmac
import hashlib

def verify_webhook(payload_body, signature, secret):
    expected = hmac.new(
        secret.encode(),
        payload_body,
        hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, signature)

javascript

import crypto from 'crypto';

function verifyWebhook(payloadBody, signature, secret) {
  const expected = crypto
    .createHmac('sha256', secret)
    .update(payloadBody)
    .digest('hex');
  return crypto.timingSafeEqual(
    Buffer.from(expected),
    Buffer.from(signature)
  );
}

Response Format

Async Start Response

json

{
  "success": true,
  "id": "batch-abc123",
  "url": "https://api.firecrawl.dev/v2/batch/scrape/batch-abc123"
}

Status/Results Response

json

{
  "status": "completed",
  "total": 3,
  "completed": 3,
  "creditsUsed": 3,
  "expiresAt": "2026-03-01T12:00:00.000Z",
  "data": [
    {
      "markdown": "# Example Page\n\nContent here...",
      "html": "<h1>Example Page</h1>...",
      "metadata": {
        "title": "Example Page",
        "description": "...",
        "sourceURL": "https://example.com",
        "statusCode": 200
      }
    }
  ],
  "next": "https://api.firecrawl.dev/v2/batch/scrape/batch-abc123?skip=10"
}

The next field provides a pagination cursor for large result sets. Results expire after 24 hours.

Cost

1 credit per URL (base cost)
+4 credits per URL if using JSON extraction mode
+4 credits per URL if using enhanced proxy

See Pricing for full cost details.

Scrape -- Single URL scraping with all format options
Crawl -- Recursive crawling for entire websites
Fast Scraping -- Cache-based speed optimization
Webhooks -- Webhook setup and event reference
Pricing -- Credit cost breakdown

Next: Document Parsing -->

Batch Scrape ​

Overview ​

API Endpoints ​

Basic Usage ​

Python (Synchronous) ​

Python (Asynchronous) ​

Node.js (Synchronous) ​

Node.js (Asynchronous) ​

cURL ​

Parameters ​

Concurrency Control ​

Structured Extraction (JSON Mode) ​

Webhook Integration ​

Configuration ​

Webhook Events ​

Webhook Payload ​

Signature Verification ​

Response Format ​

Async Start Response ​

Status/Results Response ​

Cost ​

Related Pages ​

Batch Scrape

Overview

API Endpoints

Basic Usage

Python (Synchronous)

Python (Asynchronous)

Node.js (Synchronous)

Node.js (Asynchronous)

cURL

Parameters

Concurrency Control

Structured Extraction (JSON Mode)

Webhook Integration

Configuration

Webhook Events

Webhook Payload

Signature Verification

Response Format

Async Start Response

Status/Results Response

Cost

Related Pages