Crawl

Core Feature

Recursively crawl entire websites and extract content from all pages.

Overview

Crawl automatically discovers and extracts content from all pages on a website. Scans sitemaps, follows links, handles JavaScript, respects robots.txt, and manages rate limits.

How Discovery Works

1. URL Analysis: Scans sitemap + crawls site to identify links
2. Traversal: Recursively follows links
3. Scraping: Extracts content from each page
4. Output: Clean markdown or structured format

Default behavior: Only crawls child paths of the starting URL. Use crawlEntireDomain for cross-parent paths, allowSubdomains for subdomains.

Sitemap: Default is sitemap: "include" — uses sitemap to discover URLs. Set sitemap: "skip" to only follow HTML links.

Default limit: 10,000 pages — always set a lower limit to control cost.

Basic Usage (Sync — Waits for Completion)

Python

from firecrawl import Firecrawl
from firecrawl.types import ScrapeOptions

firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")

crawl_status = firecrawl.crawl(
    "https://example.com",
    limit=100,
    scrape_options=ScrapeOptions(formats=["markdown", "html"]),
    poll_interval=30
)

for page in crawl_status.data:
    print(f"Title: {page.metadata['title']}")
    print(f"URL: {page.metadata['sourceURL']}")

Node.js

import Firecrawl from '@mendable/firecrawl-js';

const firecrawl = new Firecrawl({ apiKey: "fc-YOUR-API-KEY" });

const crawlResponse = await firecrawl.crawl("https://example.com", {
  limit: 100,
  scrapeOptions: {
    formats: ["markdown", { type: "json", schema: { /* ... */ } }],
    proxy: "auto",
    maxAge: 600000,
    onlyMainContent: true,
  },
});

Crawl Options

firecrawl.crawl(
    url="https://example.com",
    limit=100,                      # Max pages (default: 10,000)
    allowSubdomains=False,          # Include subdomains
    crawlEntireDomain=False,        # Crawl all domain paths (not just children)
    ignoreQueryParameters=False,    # Treat URLs with different params as same page
    sitemap="include",              # "include" | "skip" | "only"
    timeout=120000                  # Timeout in milliseconds
)

Async (Start & Poll)

For long-running crawls:

# Start crawl
job = firecrawl.start_crawl("https://example.com", limit=1000)
print(f"Crawl job: {job.id}")

# Check status later
status = firecrawl.get_crawl_status(job.id)
print(f"Status: {status.status}")
print(f"Completed: {status.completed}/{status.total}")

# Cancel if needed
firecrawl.cancel_crawl(job.id)

WebSocket Real-Time Watching

Watch crawl progress in real-time with async streaming:

import asyncio
from firecrawl import AsyncFirecrawl

async def main():
    firecrawl = AsyncFirecrawl(api_key="fc-YOUR-API-KEY")
    started = await firecrawl.start_crawl("https://firecrawl.dev", limit=5)

    async for snapshot in firecrawl.watcher(started.id, kind="crawl", poll_interval=2, timeout=120):
        print(snapshot.status, snapshot.completed, "/", snapshot.total)

asyncio.run(main())

// Node.js
const watcher = firecrawl.watcher(job.id, {
  kind: 'crawl', pollInterval: 2, timeout: 120
});
watcher.on('document', (doc) => console.log(doc.metadata.sourceURL));
watcher.on('done', (state) => console.log('Finished:', state.completed));
await watcher.start();

Pagination

For large crawls, paginate through results:

from firecrawl.v2.types import PaginationConfig

status = firecrawl.get_crawl_status(
    job.id,
    pagination_config=PaginationConfig(auto_paginate=False)
)

# Fetch next page
status = firecrawl.get_crawl_status_page(status.next)

PaginationConfig options: auto_paginate (bool), max_pages, max_results, max_wait_time

Webhooks

Get real-time notifications for crawl events:

result = firecrawl.crawl(
    url="https://example.com",
    limit=100,
    webhook={
        "url": "https://your-domain.com/webhook",
        "metadata": {"project": "competitor-monitor"},
        "events": ["started", "page", "completed"]
    }
)

Webhook Event Types

Event	Description
crawl.started	Crawl job has begun
crawl.page	Individual page scraped
crawl.completed	All pages done
crawl.failed	Crawl job failed

Webhook Signature Verification

Every webhook includes an X-Firecrawl-Signature header (HMAC-SHA256). Get your webhook secret from the Advanced tab in your account settings.

Error Handling

Get pages that failed during a crawl:

# Get failed pages (network errors, timeouts, robots.txt blocks)
curl -X GET "https://api.firecrawl.dev/v2/crawl/{id}/errors" \
  -H "Authorization: Bearer fc-YOUR-API-KEY"

Active Crawls

Check what crawls are currently running:

curl -X GET "https://api.firecrawl.dev/v2/crawl/active" \
  -H "Authorization: Bearer fc-YOUR-API-KEY"

API Endpoints

Method	Path	Description
POST	/v2/crawl	Start crawl
GET	/v2/crawl/{id}	Get crawl status
POST	/v2/crawl/params-preview	Preview crawl params
DELETE	/v2/crawl/{id}	Cancel crawl
GET	/v2/crawl/{id}/errors	Get failed pages
GET	/v2/crawl/active	Get active crawls

Cost

• Crawl: 1 credit per page
• Always set limit to control costs
• Example: Crawling 100 pages = 100 credits

---

Next: Browser Sandbox →