Firecrawl API SOP

Appearance

SOP 004: Search & Scrape ​

Fresh 🌱

Search the web and scrape content from results in one operation.

Overview ​

Prerequisites ​

  • ✓ Firecrawl API key
  • ✓ Search query defined
  • ✓ Understanding of scraping options

Step-by-Step ​

Step 1: Initialize Firecrawl ​

python
from firecrawl import Firecrawl

firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")
python
results = firecrawl.search(
    query="best web scraping tools",
    limit=5
)

for result in results.get('web', []):
    print(f"Title: {result['title']}")
    print(f"URL: {result['url']}")
    print()

Step 3: Search with Content Scraping ​

python
results = firecrawl.search(
    query="firecrawl features",
    limit=3,
    scrape_options={
        "formats": ["markdown", "links"]
    }
)

for result in results:
    print(f"Title: {result['title']}")
    print(f"URL: {result['url']}")
    print(f"Content preview: {result['markdown'][:200]}...")
    print()

Step 4: Process Results ​

python
search_data = []

for result in results:
    item = {
        "title": result['title'],
        "url": result['url'],
        "description": result.get('description', ''),
        "content": result.get('markdown', ''),
        "links": result.get('links', [])
    }
    search_data.append(item)

# Use results
for item in search_data:
    print(f"Processing: {item['title']}")

Search Types ​

python
results = firecrawl.search(
    query="machine learning frameworks 2025",
    limit=10,
    sources=["web"]
)
python
results = firecrawl.search(
    query="AI breakthroughs",
    sources=["news"],
    tbs="qdr:w"  # Past week
)
python
results = firecrawl.search(
    query="architecture photography",
    sources=["images"],
    limit=20
)

Search Categories ​

python
results = firecrawl.search(
    query="python web scraping",
    categories=["github"],
    limit=10
)

Academic Research ​

python
results = firecrawl.search(
    query="transformer models",
    categories=["research"],
    limit=10
)

Time-Based Filtering ​

Predefined ranges ​

python
# Past 24 hours
results = firecrawl.search(
    query="breaking news",
    tbs="qdr:d"
)

# Past week
results = firecrawl.search(
    query="tech updates",
    tbs="qdr:w"
)

# Past month
results = firecrawl.search(
    query="industry trends",
    tbs="qdr:m"
)

Custom date range ​

python
results = firecrawl.search(
    query="firecrawl updates",
    tbs="cdr:1,cd_min:12/1/2024,cd_max:12/31/2024"
)

Advanced Usage ​

With Location ​

python
results = firecrawl.search(
    query="web hosting",
    location="Germany",
    limit=5
)

Extract Specific Data ​

python
results = firecrawl.search(
    query="company pricing",
    limit=3,
    scrape_options={
        "formats": [{
            "type": "json",
            "schema": {
                "type": "object",
                "properties": {
                    "company": {"type": "string"},
                    "pricing_plans": {
                        "type": "array",
                        "items": {"type": "string"}
                    }
                }
            }
        }]
    }
)

Cost Calculation ​

  • Search: 2 credits per 10 results
  • Scraping: +1 credit per result

Example: 5 results with scraping = 1 credit (search) + 5 credits (scrape) = 6 credits

Verification Checklist ​

  • [ ] Search query is specific
  • [ ] Results are relevant
  • [ ] Content scraped successfully
  • [ ] No duplicate results
  • [ ] Data quality acceptable

Use Cases ​

1. Competitor Intelligence ​

python
competitors = ["competitor1", "competitor2", "competitor3"]

for comp in competitors:
    results = firecrawl.search(
        query=f"{comp} pricing features",
        limit=5,
        scrape_options={"formats": ["markdown"]}
    )
    # Analyze results

2. Market Research ​

python
results = firecrawl.search(
    query="market trends AI 2025",
    sources=["news", "web"],
    limit=10
)

3. Lead Research ​

python
results = firecrawl.search(
    query="B2B SaaS platforms",
    sources=["web"],
    limit=20,
    scrape_options={"formats": ["markdown"]}
)

Troubleshooting ​

No results found ​

  • Try different search terms
  • Use broader keywords
  • Check for typos

Scraping failing on results ​

  • Some sites might block scraping
  • Try without scraping first
  • Check onlyMainContent option

High credit usage ​

  • Reduce limit
  • Don't scrape if not needed
  • Use basic proxy only

Next Steps ​

Cost: 2 credits (per 10 results) + 1 credit per scraped result