Firecrawl API SOP

Appearance

AI Platforms & LLM Training ​

Updated Feb 2026

Build RAG chatbots, AI assistants, and knowledge bases powered by real-time web data. Firecrawl transforms any website into clean, structured, LLM-ready content -- eliminating hallucinations caused by stale training data.

Why Firecrawl for AI Platforms ​

Large language models are only as useful as the data they can access. Training data goes stale. Documentation changes. APIs evolve. Firecrawl bridges this gap by providing current, structured web content that your AI systems can consume on demand.

The core problem Firecrawl solves: Your AI assistant answers questions using outdated training data, producing hallucinations. With Firecrawl, you feed it live web knowledge so it responds with current, verified information.

Key Benefits ​

  • Real-time knowledge -- Extract current documentation, articles, and data instead of relying on training snapshots
  • Multiple output formats -- Get clean markdown for LLM context, structured JSON for databases, or raw HTML for custom parsing
  • Enterprise scale -- Process millions of pages with automatic infrastructure scaling
  • Framework integration -- Native support for LangChain, LlamaIndex, n8n, and custom RAG pipelines

How It Works ​

Firecrawl Features Used ​

FeatureRole in AI Platforms
ScrapeExtract single pages as clean markdown or structured JSON
CrawlIngest entire documentation sites or knowledge bases
SearchFind and extract relevant web pages on any topic
AgentNavigate complex multi-step workflows to gather data
MapDiscover all URLs on a site before targeted extraction

Building a RAG Knowledge Base ​

The most common AI platform use case is building a Retrieval-Augmented Generation (RAG) system. Here is the typical workflow:

Step 1: Crawl Documentation Sites ​

python
from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

# Crawl an entire documentation site
result = app.crawl(
    url="https://docs.example.com",
    params={
        "limit": 500,
        "scrapeOptions": {
            "formats": ["markdown"]
        }
    }
)

# Each page comes back as clean markdown
for page in result["data"]:
    print(f"URL: {page['metadata']['url']}")
    print(f"Content length: {len(page['markdown'])} chars")

Step 2: Chunk and Embed ​

python
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)

# Process each crawled page
for page in result["data"]:
    chunks = splitter.split_text(page["markdown"])
    embeddings = OpenAIEmbeddings().embed_documents(chunks)
    # Store in your vector database (Pinecone, Weaviate, Supabase, etc.)

Step 3: Query with Context ​

python
# When a user asks a question, retrieve relevant chunks
# and pass them as context to your LLM
query = "How do I configure authentication?"
relevant_chunks = vector_db.similarity_search(query, k=5)

context = "\n\n".join([chunk.page_content for chunk in relevant_chunks])
response = llm.invoke(f"Context:\n{context}\n\nQuestion: {query}")

For questions that need the latest information (not just your crawled knowledge base), combine with the Search feature:

python
# Search the web for current information
search_results = app.search(
    query="latest Python 3.13 release features",
    params={
        "limit": 5,
        "scrapeOptions": {
            "formats": ["markdown"]
        }
    }
)

# Feed results directly into your LLM
for result in search_results["data"]:
    print(result["markdown"][:500])

Keeping Knowledge Fresh ​

Use Change Tracking to detect when source documentation updates, then re-crawl only changed pages:

python
# Scrape with change tracking enabled
result = app.scrape(
    url="https://docs.example.com/api-reference",
    params={
        "formats": ["markdown", "changeTracking"]
    }
)

# Check if content changed since last scrape
if result.get("changeTracking", {}).get("changeStatus") == "changed":
    # Re-index this page in your vector database
    update_vector_db(result["markdown"])

Quick Start: Firestarter Template ​

The Firestarter template provides an instant AI chatbot with integrated web knowledge. Clone it and deploy in minutes:

  1. Clone the Firestarter template from GitHub
  2. Add your Firecrawl API key
  3. Point it at your documentation URLs
  4. Deploy -- your chatbot now has live web knowledge

Customer Stories ​

Replit ​

Replit uses Firecrawl to keep Replit Agent's knowledge base current with the latest API documentation. When APIs change, Firecrawl extracts the updated docs so Replit Agent generates code using current method signatures and parameters.

Stack AI ​

Stack AI leverages Firecrawl to feed agentic workflows with high-quality web data. Their platform builds AI agents that need reliable, structured information from across the web -- Firecrawl provides that data pipeline.

Output Formats for AI ​

Firecrawl delivers data in multiple formats optimized for different AI use cases:

FormatBest For
markdownDirect LLM context windows, RAG chunks
jsonStructured data extraction with schema enforcement
htmlCustom parsing pipelines, preserving document structure
summaryAI-generated page summaries for quick indexing
linksBuilding knowledge graphs and navigation maps
imagesMultimodal AI training and analysis
screenshotVisual AI, layout analysis, visual QA

Best Practices ​

  1. Start with Map, then Scrape -- Use Map to discover all URLs on a documentation site, then selectively scrape the pages you need
  2. Use markdown format for LLMs -- It preserves structure (headers, lists, code blocks) while being token-efficient
  3. Set up change tracking -- Do not re-crawl entire sites daily; track changes and only re-index what has updated
  4. Chunk intelligently -- Use header-based splitting to keep semantic context together
  5. Include metadata -- Store source URLs and timestamps with your embeddings for citation and freshness tracking