Firecrawl API SOP

Appearance

Firecrawl + LlamaIndex ​

Updated Feb 2026

Integrate Firecrawl with LlamaIndex to build AI applications with vector search and embeddings powered by web content. Crawl a site, create a vector index, and query it with natural language.

Source: docs.firecrawl.dev/developer-guides/llm-sdks-and-frameworks/llamaindex

Installation ​

bash
npm install llamaindex @llamaindex/openai @mendable/firecrawl-js

Create a .env file:

env
FIRECRAWL_API_KEY=your_firecrawl_key
OPENAI_API_KEY=your_openai_key

Node < 20

If using Node < 20, install dotenv and add import 'dotenv/config' to your code.

Crawl a website, create embeddings, build a vector index, and query the content using RAG (Retrieval-Augmented Generation).

js
import Firecrawl from '@mendable/firecrawl-js';
import { Document, VectorStoreIndex, Settings } from 'llamaindex';
import { OpenAI, OpenAIEmbedding } from '@llamaindex/openai';

// Configure LlamaIndex settings
Settings.llm = new OpenAI({ model: "gpt-4o" });
Settings.embedModel = new OpenAIEmbedding({ model: "text-embedding-3-small" });

// Crawl the website
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
const crawlResult = await firecrawl.crawl('https://firecrawl.dev', {
    limit: 10,
    scrapeOptions: { formats: ['markdown'] }
});
console.log(`Crawled ${crawlResult.data.length} pages`);

// Convert to LlamaIndex documents
const documents = crawlResult.data.map((page, i) =>
    new Document({
        text: page.markdown,
        id_: `page-${i}`,
        metadata: { url: page.metadata?.sourceURL }
    })
);

// Build vector index
const index = await VectorStoreIndex.fromDocuments(documents);
console.log('Vector index created with embeddings');

// Query the index
const queryEngine = index.asQueryEngine();
const response = await queryEngine.query({
    query: 'What is Firecrawl and how does it work?'
});

console.log('\nAnswer:', response.toString());

How It Works ​

  1. Crawl -- Firecrawl crawls the target website and returns markdown for each page
  2. Document creation -- Each page becomes a LlamaIndex Document with metadata
  3. Embedding -- LlamaIndex generates embeddings using OpenAI's text-embedding-3-small model
  4. Index -- Documents are stored in a VectorStoreIndex for similarity search
  5. Query -- The query engine retrieves relevant chunks and generates an answer using GPT

Tips ​

  • Use limit in the crawl options to control how many pages are indexed
  • Add onlyMainContent: true to scrapeOptions for cleaner document text
  • For production, persist the vector index to avoid re-crawling on every startup
  • Consider using firecrawl.search() instead of crawl() if you only need specific pages

Further Reading ​