Firecrawl API SOP

Appearance

SOP 003: Website Crawling ​

Fresh 🌱

Recursively crawl entire websites and extract content from all pages.

Overview ​

Prerequisites ​

  • ✓ Basic scraping knowledge (SOP 001)
  • ✓ Understanding of credit costs
  • ✓ Patience for longer operations

Step-by-Step ​

Step 1: (Optional) Map Site First ​

Get URL count before crawling:

python
from firecrawl import Firecrawl

firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")

urls = firecrawl.map("https://example.com")
print(f"Found {len(urls)} URLs")

Step 2: Start Crawl ​

python
# Simple crawl
result = firecrawl.crawl(
    url="https://example.com",
    limit=100
)

Step 3: Handle Results ​

python
print(f"Status: {result.status}")
print(f"Pages crawled: {result.completed}")
print(f"Total pages: {result.total}")
print(f"Credits used: {result.creditsUsed}")

for page in result.data:
    print(f"- {page.metadata['title']}")
    print(f"  URL: {page.metadata['sourceURL']}")

Step 4: Save Results ​

python
import json

output = {
    "total_pages": result.completed,
    "credits_used": result.creditsUsed,
    "pages": [
        {
            "title": page.metadata.get('title'),
            "url": page.metadata.get('sourceURL'),
            "content_length": len(page.markdown)
        }
        for page in result.data
    ]
}

with open("crawl_results.json", "w") as f:
    json.dump(output, f, indent=2)

Advanced Options ​

Limit Pages ​

python
result = firecrawl.crawl(
    url="https://example.com",
    limit=50  # Only crawl first 50 pages
)

Include Subdomains ​

python
result = firecrawl.crawl(
    url="https://example.com",
    allowSubdomains=True
)

With Specific Formats ​

python
from firecrawl.types import ScrapeOptions

result = firecrawl.crawl(
    url="https://example.com",
    scrape_options={
        "formats": ["markdown", "links"],
        "onlyMainContent": True
    }
)

Start & Check Later ​

For long crawls:

python
# Start crawl
job = firecrawl.start_crawl("https://example.com", limit=1000)
job_id = job.id
print(f"Job ID: {job_id}")

# Check status later
status = firecrawl.get_crawl_status(job_id)
print(f"Completed: {status.completed}/{status.total}")

Cost Estimation ​

Site SizePagesCreditsCost Estimate
Small1010$0.01
Medium100100$0.10
Large10001000$1.00
XL1000010000$10.00

Verification Checklist ​

  • [ ] URL is correct
  • [ ] Limit is set appropriately
  • [ ] Have sufficient credits
  • [ ] Results saved successfully
  • [ ] No sensitive data included

Troubleshooting ​

Crawl takes too long ​

  • Reduce limit parameter
  • Check network connection
  • Website might be slow

Too many credits used ​

  • Set lower limit
  • Use onlyMainContent=True to reduce processing
  • Avoid JSON mode if not needed

Incomplete results ​

  • Some pages might be blocked
  • Try with different proxy settings
  • Check robots.txt compliance

Next Steps ​

Cost: 1 credit per page