Firecrawl API SOP

Appearance

Self-Hosting Guide ​

Updated Feb 2026

Deploy Firecrawl on your own infrastructure using Docker Compose. Self-hosting gives you full control over data and the core scraping engine.

What You Get (and Don't Get) ​

Included ​

  • Full scrape, crawl, map, and search functionality
  • JSON extraction (with your own LLM key)
  • Batch scraping
  • Change tracking
  • All output formats (markdown, HTML, screenshot, etc.)
  • Local LLM support via Ollama

Not Included (Cloud Only) ​

  • Fire-engine (advanced anti-bot, IP blocking circumvention)
  • /agent endpoint (Spark-1 models)
  • /browser endpoint (remote browser sandbox)
  • Enhanced proxy infrastructure
  • Usage dashboard

Prerequisites ​

RequirementNotes
DockerInstall from docs.docker.com/get-docker
Docker ComposeIncluded with Docker Desktop; install separately on Linux
4GB+ RAMRecommended minimum for comfortable operation
LLM API KeyOpenAI or Ollama for JSON extraction features (optional)

Quick Start ​

1. Clone the Repository ​

bash
git clone https://github.com/mendableai/firecrawl.git
cd firecrawl

2. Create Environment File ​

Copy the example environment file and configure it:

bash
cp apps/api/.env.example .env

3. Build and Start ​

bash
docker compose build
docker compose up

The API will be accessible at http://localhost:3002.

4. Verify Installation ​

bash
curl -X POST http://localhost:3002/v2/crawl \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://docs.firecrawl.dev"}'

Full Environment Variables Reference ​

Create a .env file in the project root with these variables.

Required Variables ​

bash
# API Server
PORT=3002
HOST=0.0.0.0

# Authentication (set false for local dev)
USE_DB_AUTHENTICATION=false

# Bull Queue Management
BULL_AUTH_KEY=CHANGEME

LLM Configuration (Pick One) ​

bash
OPENAI_API_KEY=sk-your-openai-key
bash
OLLAMA_BASE_URL=http://localhost:11434/api
MODEL_NAME=deepseek-r1:7b
MODEL_EMBEDDING_NAME=nomic-embed-text
bash
OPENAI_BASE_URL=https://your-provider.com/v1
OPENAI_API_KEY=your-key
MODEL_NAME=your-model

LLM Required for These Features

Without an LLM provider configured, the following features will not work:

  • JSON format extraction on scrape
  • /extract API endpoint
  • Summary format
  • Branding format
  • Change tracking (JSON field compare mode)

Proxy Configuration (Optional) ​

bash
PROXY_SERVER=0.1.2.3:1234
PROXY_USERNAME=user
PROXY_PASSWORD=pass

Search API (Optional) ​

bash
# SearXNG instance for /search endpoint
SEARXNG_ENDPOINT=http://your.searxng.server

Resource Limits (Optional) ​

bash
# Fraction of system resources (0.0 - 1.0)
MAX_CPU=0.8
MAX_RAM=0.8

Additional Services (Optional) ​

bash
# LlamaParse for advanced PDF parsing
LLAMAPARSE_API_KEY=your-llamaparse-key

# Slack notifications
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/your/webhook/url

# Analytics
POSTHOG_API_KEY=your-posthog-key
POSTHOG_HOST=https://app.posthog.com

# Allow webhooks to localhost (dev only)
ALLOW_LOCAL_WEBHOOKS=true

Complete .env Template ​

bash
# ========== REQUIRED ==========
PORT=3002
HOST=0.0.0.0
USE_DB_AUTHENTICATION=false
BULL_AUTH_KEY=CHANGEME

# ========== LLM (choose one) ==========
# OpenAI
OPENAI_API_KEY=

# Ollama
# OLLAMA_BASE_URL=http://localhost:11434/api
# MODEL_NAME=deepseek-r1:7b
# MODEL_EMBEDDING_NAME=nomic-embed-text

# OpenAI-compatible
# OPENAI_BASE_URL=
# MODEL_NAME=

# ========== PROXY (optional) ==========
# PROXY_SERVER=
# PROXY_USERNAME=
# PROXY_PASSWORD=

# ========== SEARCH (optional) ==========
# SEARXNG_ENDPOINT=

# ========== RESOURCES (optional) ==========
# MAX_CPU=0.8
# MAX_RAM=0.8

# ========== SERVICES (optional) ==========
# LLAMAPARSE_API_KEY=
# SLACK_WEBHOOK_URL=
# POSTHOG_API_KEY=
# POSTHOG_HOST=
# ALLOW_LOCAL_WEBHOOKS=false

Services Architecture ​

Self-hosted Firecrawl runs these containers via Docker Compose:

ServicePurposeRequired
Firecrawl APIMain API server on port 3002Yes
RedisJob queue management and cachingYes
PlaywrightBrowser rendering for JavaScript-heavy pagesYes (included)
LLM ProviderPowers JSON extraction, summaries, brandingOptional
SearXNGSelf-hosted search engine for /search endpointOptional

Ollama Setup (Local LLM) ​

Run AI-powered features without external API calls using Ollama.

1. Install Ollama ​

bash
# Linux/macOS
curl -fsSL https://ollama.com/install.sh | sh

# Windows — download from https://ollama.com/download

2. Pull Models ​

bash
# Text generation model
ollama pull deepseek-r1:7b

# Embedding model
ollama pull nomic-embed-text

3. Configure .env ​

bash
OLLAMA_BASE_URL=http://host.docker.internal:11434/api
MODEL_NAME=deepseek-r1:7b
MODEL_EMBEDDING_NAME=nomic-embed-text

Docker Network Note

Use host.docker.internal instead of localhost when Ollama runs on the host machine and Firecrawl runs in Docker. On Linux, you may need --add-host=host.docker.internal:host-gateway in your Docker Compose config.

TypeScript Playwright Service ​

For improved browser rendering, switch to the TypeScript Playwright service:

1. Update docker-compose.yml ​

yaml
playwright-service:
  build: apps/playwright-service-ts
  # ... rest of config

2. Update .env ​

bash
PLAYWRIGHT_MICROSERVICE_URL=http://localhost:3000/scrape

3. Rebuild ​

bash
docker compose build playwright-service
docker compose up -d

Queue Manager ​

Monitor job queues through the built-in Bull Board dashboard:

http://localhost:3002/admin/{BULL_AUTH_KEY}/queues

Replace {BULL_AUTH_KEY} with the value from your .env file. The dashboard shows:

  • Active, waiting, completed, and failed jobs
  • Job details and error messages
  • Queue throughput metrics
  • Retry controls

Kubernetes Deployment ​

For production cluster deployments, refer to the Kubernetes installation guide in the repository:

examples/kubernetes-cluster-install/README.md

See: GitHub - Kubernetes Install

Troubleshooting ​

"Supabase client is not configured" ​

Expected in self-hosted setups. This warning appears because Supabase is used for authentication in the cloud version. With USE_DB_AUTHENTICATION=false, this message is harmless. Core scraping and crawling work normally.

"Bypassing authentication" ​

Expected in self-hosted setups. Similar to the Supabase warning, this reflects that cloud authentication is not configured. All API endpoints function normally without it.

Docker Container Failures ​

bash
# Check container logs
docker compose logs firecrawl-api
docker compose logs redis
docker compose logs playwright-service

# Restart specific service
docker compose restart firecrawl-api

# Full rebuild
docker compose down
docker compose build --no-cache
docker compose up

Redis Connection Errors ​

  1. Verify Redis container is running: docker compose ps
  2. Check Redis logs: docker compose logs redis
  3. Ensure no port conflicts on 6379
  4. Verify REDIS_URL and REDIS_RATE_LIMIT_URL are correct in .env
  5. Check firewall rules allow container-to-container communication

API Not Responding ​

  1. Confirm the API container is running: docker compose ps
  2. Verify PORT and HOST settings in .env
  3. Check for port conflicts: lsof -i :3002 (Linux/macOS) or netstat -ano | findstr 3002 (Windows)
  4. Review API logs: docker compose logs firecrawl-api
  5. Test connectivity: curl http://localhost:3002/v2/health

LLM Features Not Working ​

  1. Verify your LLM API key is set correctly in .env
  2. For Ollama: confirm it is running (ollama list) and models are pulled
  3. For Ollama in Docker: use host.docker.internal not localhost
  4. Test Ollama directly: curl http://localhost:11434/api/generate -d '{"model":"deepseek-r1:7b","prompt":"test"}'

Memory Issues ​

If containers crash due to OOM:

bash
# Increase Docker memory limit (Docker Desktop > Settings > Resources)
# Or set resource limits in .env
MAX_CPU=0.8
MAX_RAM=0.8

Updating ​

Pull the latest changes and rebuild:

bash
cd firecrawl
git pull origin main
docker compose down
docker compose build
docker compose up -d

Additional Resources ​

ResourceLink
Self-host docsSELF_HOST.md in repository root
Full documentationdocs.firecrawl.dev/llms.txt
Discord supportdiscord.gg/gSmWdAkdwd
GitHub Issuesgithub.com/mendableai/firecrawl/issues

Next: Contribution Guide -->