Firecrawl + Dify AI Platform

Updated Feb 2026

Dify is an open-source LLM application development platform with an official Firecrawl plugin (44,000+ installations). The integration enables web scraping directly within AI chatflows, workflows, and agent applications.

Setup

Install the Plugin

1. Open your Dify instance and navigate to Plugins in the sidebar
2. Search for Firecrawl in the marketplace
3. Click Install on the official Firecrawl plugin
4. Wait for the installation to complete

Configure Credentials

1. Go to Plugin Settings for Firecrawl
2. Enter your Firecrawl API key from firecrawl.dev/app/api-keys
3. Click Save to authorize the connection
4. The plugin tools are now available in all Dify app types

Available Tools

Scrape

Converts a URL into structured data. Returns content as markdown, HTML, or JSON with schema validation.

Best for: Product data extraction, article content, structured data with JSON mode.

Parameters:

Parameter	Type	Description
url	string	The page URL to scrape
formats	array	Output formats: markdown, html, links, json
jsonOptions.schema	object	JSON schema for structured extraction
timeout	number	Request timeout in milliseconds

Crawl

Performs recursive site traversal for comprehensive content gathering across multiple pages.

Best for: Full site content extraction, documentation scraping, multi-page data collection.

Parameters:

Parameter	Type	Description
url	string	Starting URL for the crawl
limit	number	Maximum number of pages to crawl
includePaths	array	URL patterns to include
excludePaths	array	URL patterns to exclude

Map

Generates a complete URL inventory for a website without extracting content.

Best for: Site structure analysis, SEO auditing, URL discovery for batch scraping.

Parameters:

Parameter	Type	Description
url	string	The domain to map
limit	number	Maximum number of URLs to return

Crawl Job

Manages asynchronous crawl operations by Job ID. Use this to monitor long-running crawls, check progress, or cancel operations.

Parameters:

Parameter	Type	Description
jobId	string	The crawl job ID returned by the Crawl tool

Application Types

Chatflow Apps

Linear conversational pipelines where Firecrawl provides real-time web data to the LLM.

Pipeline: User Input --> Firecrawl Scrape --> LLM Processing --> Response

Example use case: A customer support chatbot that can look up current product pages, pricing, or documentation when asked.

User: "What are the current pricing plans on example.com?"
  --> Firecrawl scrapes example.com/pricing
  --> LLM summarizes the pricing tiers
  --> Response delivered to user

Workflow Apps

Scheduled or event-driven pipelines with data transformation and storage.

Pipeline: Trigger --> Firecrawl Crawl --> LLM Transform --> Database Write

Example use case: A nightly workflow that crawls competitor blogs, summarizes new posts, and stores them in a knowledge base.

Trigger: Schedule (daily at midnight)
Step 1: Firecrawl Crawl - competitor blog, limit 10 pages
Step 2: LLM - Summarize each page into 3 bullet points
Step 3: Database - Insert summaries with URL and timestamp

Agent Apps

Autonomous agents that decide when and how to use Firecrawl tools based on the conversation context.

Example use case: A research agent that independently decides to search the web, scrape pages, and synthesize findings.

The agent autonomously determines when web scraping is needed based on the user's query. No explicit tool-calling logic is required in the workflow -- the agent handles tool selection.

Building a Research Chatflow

Step-by-Step

1. Create a new Chatflow in Dify
2. Add a Start node with a text input variable for the user query
3. Add a Firecrawl Search node connected to the start node
   • Map the user query to the search input
   • Set result limit to 5
4. Add a Firecrawl Scrape node to get full content from the top result
   • Map the first search result URL to the scrape input
5. Add an LLM node (OpenAI, Anthropic, etc.)
   • System prompt: Analyze the following web content and answer the user's question with citations.
   • Map the scrape output to the LLM context
6. Add an End node connected to the LLM output
7. Publish the app and test

Dify vs. Other Automation Platforms

Feature	Dify	n8n / Make / Zapier
Primary focus	AI-native app development	Workflow automation
Self-hosting	Full self-host support	Varies by platform
Agent autonomy	Agents choose tools dynamically	Static workflow paths
LLM integration	First-class, multi-provider	Plugin/node based
Visual builder	Chatflow + Workflow canvases	Single workflow canvas
Best for	AI apps with web data needs	Data pipeline automation

Tips

• Use Scrape for single-page data in chatflows where speed matters
• Use Map before Crawl to preview the URL structure and set precise includePaths
• For long-running crawls, use the async Crawl tool and poll with Crawl Job
• In agent apps, provide clear tool descriptions so the agent knows when to use each Firecrawl tool

WARNING

Crawl operations can consume significant credits on large sites. Always set a limit parameter and use includePaths / excludePaths to control scope.

Resources

• Dify Documentation
• Firecrawl API Docs
• Dify GitHub
• Firecrawl Plugin on Dify Marketplace