Firecrawl API SOP

Appearance

Deep Research ​

Updated Feb 2026

Build agentic research tools that search, extract, synthesize, and cite information from across the web. Firecrawl powers deep research workflows that condense hours of manual investigation into minutes of automated, source-attributed analysis.

Why Firecrawl for Deep Research ​

Manual research is slow and incomplete. You search, open tabs, read, cross-reference, and still miss sources. Firecrawl automates the discovery and extraction phases so your research agents can follow citation chains, identify related sources, and synthesize findings with full attribution.

Key Advantages ​

  • Speed -- Automated discovery, extraction, and synthesis across multiple web sources
  • Completeness -- Follow citation chains and identify related sources that manual search misses
  • Attribution -- All extracted content maintains source URLs and timestamps for verification
  • Scale -- Process thousands of sources simultaneously with automatic infrastructure scaling
  • Depth -- Support longitudinal studies through scheduled crawls and temporal analysis

How It Works ​

Firecrawl Features Used ​

FeatureRole in Deep Research
SearchFind relevant sources across the web, news, and academic sites
ScrapeExtract full content from key sources in clean markdown
AgentNavigate complex multi-step research workflows autonomously
CrawlIngest entire reference sites or documentation
MapDiscover the full scope of content on research-relevant sites

Building a Research Agent ​

Step 1: Initial Source Discovery ​

python
from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

# Search for sources on the research topic
results = app.search(
    query="impact of retrieval-augmented generation on LLM accuracy 2025 2026",
    params={
        "limit": 15,
        "scrapeOptions": {
            "formats": ["markdown"]
        }
    }
)

# Collect initial sources
sources = []
for result in results["data"]:
    sources.append({
        "url": result["metadata"]["url"],
        "title": result["metadata"].get("title", "Untitled"),
        "content": result["markdown"],
        "relevance_score": result.get("score", 0)
    })

print(f"Found {len(sources)} initial sources")

Step 2: Deep Extraction from Key Sources ​

python
# Identify the most relevant sources and extract full content
top_sources = sorted(sources, key=lambda s: s["relevance_score"], reverse=True)[:5]

detailed_sources = []
for source in top_sources:
    full_content = app.scrape(
        url=source["url"],
        params={
            "formats": ["markdown", "links"]
        }
    )

    detailed_sources.append({
        "url": source["url"],
        "title": source["title"],
        "full_content": full_content["markdown"],
        "outbound_links": full_content.get("links", [])
    })

Step 3: Follow Citation Chains ​

python
# Extract links from key sources and follow citation chains
cited_urls = set()
for source in detailed_sources:
    for link in source["outbound_links"]:
        # Filter for likely citations (research papers, documentation, reports)
        if any(domain in link for domain in [
            "arxiv.org", "scholar.google", "doi.org",
            "nature.com", "ieee.org", "acm.org"
        ]):
            cited_urls.add(link)

print(f"Found {len(cited_urls)} cited sources to follow")

# Extract content from cited sources
for url in list(cited_urls)[:10]:
    try:
        cited_content = app.scrape(
            url=url,
            params={"formats": ["markdown"]}
        )
        detailed_sources.append({
            "url": url,
            "title": cited_content["metadata"].get("title", ""),
            "full_content": cited_content["markdown"],
            "source_type": "citation"
        })
    except Exception as e:
        print(f"Could not extract {url}: {e}")

Step 4: Synthesize with Attribution ​

python
import openai

# Build a structured research brief from all sources
source_material = "\n\n===\n\n".join([
    f"[Source: {s['url']}]\n{s['full_content'][:3000]}"
    for s in detailed_sources
])

synthesis_prompt = f"""You are a research analyst. Synthesize the following sources
into a comprehensive research report on "The Impact of RAG on LLM Accuracy."

REQUIREMENTS:
1. Cite every claim with [Source Title](URL)
2. Identify areas of consensus and disagreement
3. Note limitations and gaps in the research
4. Include a summary table of key findings
5. End with recommendations for practitioners

SOURCES:
{source_material}
"""

response = openai.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": synthesis_prompt}]
)

print(response.choices[0].message.content)

Research Domains ​

Academic Research ​

Firecrawl handles open-access research papers, academic websites, and publicly available scientific publications. It preserves formatting, equations, tables, and citations:

python
# Search for academic sources
academic = app.search(
    query="site:arxiv.org transformer architecture attention mechanism",
    params={
        "limit": 10,
        "scrapeOptions": {
            "formats": ["markdown"]
        }
    }
)

Market Research ​

Monitor industry trends, analyze competitor strategies, and gather market intelligence:

python
# Multi-source market research
queries = [
    "SaaS market trends 2026",
    "enterprise AI adoption statistics",
    "cloud infrastructure spending forecast",
]

all_research = []
for query in queries:
    results = app.search(
        query=query,
        params={"limit": 5, "scrapeOptions": {"formats": ["markdown"]}}
    )
    all_research.extend(results["data"])

Technical Documentation Research ​

Crawl entire documentation sites to build comprehensive technical references:

python
# Crawl documentation for in-depth technical research
docs = app.crawl(
    url="https://docs.example.com",
    params={
        "limit": 200,
        "scrapeOptions": {
            "formats": ["markdown"]
        }
    }
)

# Index by topic for quick reference
topic_index = {}
for page in docs["data"]:
    title = page["metadata"].get("title", "")
    url = page["metadata"]["url"]
    topic_index[title] = {
        "url": url,
        "content": page["markdown"]
    }

Longitudinal Research ​

Track how topics evolve over time using scheduled crawls:

python
# Set up periodic monitoring for research topics
import json
from datetime import datetime

def capture_snapshot(urls, topic_name):
    """Capture a point-in-time snapshot of research sources."""
    snapshot = {
        "topic": topic_name,
        "timestamp": datetime.now().isoformat(),
        "sources": []
    }

    for url in urls:
        result = app.scrape(
            url=url,
            params={"formats": ["markdown", "changeTracking"]}
        )
        snapshot["sources"].append({
            "url": url,
            "content_hash": hash(result["markdown"]),
            "changed": result.get("changeTracking", {}).get("changeStatus"),
            "word_count": len(result["markdown"].split())
        })

    return snapshot

Quick Start Templates ​

  • Fireplexity -- Blazing-fast AI search with real-time citations. A Perplexity-style research interface.
  • Firesearch -- Deep research agent built with LangGraph, featuring answer validation and multi-step reasoning.
  • Open Researcher -- Visual AI research assistant for comprehensive analysis with charts and visualizations.

Best Practices ​

  1. Start broad, then narrow -- Use Search for initial discovery, then Scrape for deep extraction on the most relevant sources
  2. Follow citation chains -- The best sources cite other good sources; follow those links for comprehensive coverage
  3. Track source quality -- Not all sources are equal; weight academic papers and primary sources over blog posts
  4. Include timestamps -- Record when each source was extracted for temporal context
  5. Use structured extraction -- Define schemas for consistent data extraction across different source types
  6. Handle failures gracefully -- Some URLs will be inaccessible; log failures and continue processing