Firecrawl API SOP

Appearance

Troubleshooting ​

Fresh 🌱

Common issues and solutions.

Authentication Issues ​

Error: "Invalid API key" ​

Cause: API key is incorrect or revoked

Solution:

  1. Verify API key starts with fc-
  2. Check it hasn't been revoked in dashboard
  3. Generate new key if needed
  4. Check for extra spaces/characters

Error: "Unauthorized" ​

Cause: Missing or malformed Authorization header

Solution:

bash
# Correct format
-H 'Authorization: Bearer fc-YOUR-API-KEY'

# Wrong (will fail)
-H 'Authorization: fc-YOUR-API-KEY'
-H 'API-Key: YOUR-KEY'

Scraping Issues ​

Error: "HTTP 404" ​

Cause: URL doesn't exist or is inaccessible

Solution:

  1. Verify URL is correct
  2. Test URL in browser
  3. Check if site is behind login
  4. Try different URL

Error: "Timeout" ​

Cause: Website is slow or unresponsive

Solution:

python
# Increase timeout
result = firecrawl.scrape(url, timeout=30000)  # 30 seconds

Empty Markdown ​

Cause: Page has no main content or complex structure

Solution:

python
# Try with full content
result = firecrawl.scrape(url, onlyMainContent=False)

JavaScript Content Not Loading ​

Cause: Page requires JavaScript rendering

Solution:

python
# Use enhanced mode or wait for JavaScript
result = firecrawl.scrape(url, proxy="enhanced")

Search Issues ​

No Results Found ​

Cause: Search query is too specific or has typos

Solution:

  • Try broader keywords
  • Fix spelling
  • Use different terms
  • Check query syntax

Low Quality Results ​

Cause: Search parameters not optimized

Solution:

python
# Use categories and filters
results = firecrawl.search(
    query="your query",
    categories=["github", "research"],
    tbs="qdr:w"  # Past week
)

Crawl Issues ​

Crawl Too Slow ​

Cause: Crawling too many pages or website is slow

Solution:

python
# Reduce limit
result = firecrawl.crawl(url, limit=50)

# Or start crawl and check later
job = firecrawl.start_crawl(url)
status = firecrawl.get_crawl_status(job.id)

High Credit Usage ​

Cause: Crawling too many pages or using expensive options

Solution:

  1. Set lower limit
  2. Avoid JSON mode if not needed
  3. Use basic proxy
  4. Avoid enhanced mode

Incomplete Crawl ​

Cause: Some pages blocked or timeout

Solution:

  1. Check robots.txt compliance
  2. Verify rate limits aren't exceeded
  3. Try with different proxy settings
  4. Check if site blocks automation

JSON Extraction Issues ​

Schema Mismatch ​

Cause: Schema doesn't match website structure

Solution:

  1. Review actual HTML structure
  2. Adjust schema fields
  3. Make fields optional
  4. Use prompt-based extraction

Validation Errors ​

Cause: Data types don't match schema

Solution:

python
# Make fields optional
from pydantic import BaseModel
from typing import Optional

class Data(BaseModel):
    required_field: str
    optional_field: Optional[str] = None

Credit Issues ​

Error: "Insufficient Credits" ​

Cause: Account is out of credits

Solution:

  1. Check credit balance: Dashboard → Logs
  2. Purchase credits
  3. Upgrade plan
  4. Set lower limit for crawls

Unexpected High Usage ​

Cause: Using expensive operations

Solution:

  • Review operation types used
  • Reduce JSON mode usage
  • Use basic proxy instead of enhanced
  • Set appropriate limit values

Webhook Issues ​

Webhook Not Receiving ​

Cause: URL misconfigured or network blocked

Solution:

  1. Verify webhook URL is public
  2. Check firewall/security settings
  3. Test with curl first
  4. Add debug logging

Invalid Signature ​

Cause: Signature verification failed

Solution:

python
# Verify signature correctly
import hmac
import hashlib

def verify_signature(payload, signature, secret):
    computed = hmac.new(
        secret.encode(),
        payload,
        hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(computed, signature)

Getting Help ​

Still stuck? Use the chat widget →