SOP 007: Agent Research

Research Preview

Use Firecrawl's Agent API for autonomous deep research — no URLs needed. The Agent searches, navigates, and extracts data based on your natural language prompt.

When to Use Agent

Scenario	Use
Know the URL, need content	Scrape
Don't know where data lives	Agent
Need to research a topic	Agent
Compare products across sites	Agent
Extract structured data from unknown sources	Agent

Prerequisites

• Firecrawl API key
• pip install firecrawl-py

Procedure

Step 1: Simple Research Query

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR-API-KEY")

result = app.agent(
    prompt="Find the pricing plans for Notion and compare them with Confluence",
    model="spark-1-mini",
    maxCredits=100
)

print(result.data)

Step 2: Structured Output with Schema

Get typed, consistent results:

from pydantic import BaseModel, Field
from typing import List, Optional

class Competitor(BaseModel):
    name: str = Field(description="Company name")
    pricing: str = Field(description="Starting price")
    key_features: List[str] = Field(description="Top 3 features")
    target_audience: Optional[str] = Field(None, description="Primary target market")

class CompetitorAnalysis(BaseModel):
    competitors: List[Competitor]
    summary: str = Field(description="Overall market summary")

result = app.agent(
    prompt="Find the top 5 AI-powered web scraping tools, their pricing, key features, and target audience",
    schema=CompetitorAnalysis,
    model="spark-1-pro",
    maxCredits=200
)

for comp in result.data.competitors:
    print(f"{comp.name} — {comp.pricing}")
    for feature in comp.key_features:
        print(f"  - {feature}")

Step 3: Focus Agent on Specific URLs

When you know where to look but want AI-powered extraction:

result = app.agent(
    urls=["https://docs.firecrawl.dev", "https://firecrawl.dev/pricing"],
    prompt="Compare the features listed in the docs with the pricing page. What features are included in each plan?",
    model="spark-1-pro"
)

Step 4: Async for Long-Running Research

# Start the job
job = app.start_agent(
    prompt="Find all Y Combinator batch W25 companies in the AI space with their valuations and founding teams",
    model="spark-1-pro",
    maxCredits=500
)
print(f"Job ID: {job.id}")

# Poll for completion
import time
while True:
    status = app.get_agent_status(job.id)
    if status.status == "completed":
        print(status.data)
        break
    elif status.status == "failed":
        print("Job failed")
        break
    print(f"Processing... ({status.status})")
    time.sleep(5)

Use Case Examples

Lead Enrichment

class CompanyProfile(BaseModel):
    name: str
    website: str
    employee_count: Optional[str]
    funding_total: Optional[str]
    last_funding_round: Optional[str]
    tech_stack: List[str]
    headquarters: Optional[str]

result = app.agent(
    prompt="Find detailed company information for Vercel including their tech stack, funding history, and employee count",
    schema=CompanyProfile,
    model="spark-1-mini"
)

Competitive Pricing Monitor

class PricingPlan(BaseModel):
    plan_name: str
    monthly_price: str
    annual_price: Optional[str]
    features: List[str]

class PricingComparison(BaseModel):
    company_a: List[PricingPlan]
    company_b: List[PricingPlan]

result = app.agent(
    prompt="Compare the current pricing plans between Slack and Microsoft Teams, including all tiers",
    schema=PricingComparison,
    model="spark-1-pro"
)

Market Research

result = app.agent(
    prompt="What are the top 10 trending AI frameworks in 2025? Include GitHub stars, last commit date, and primary use case for each.",
    model="spark-1-pro",
    maxCredits=300
)

Content Research for SEO

class TopicResearch(BaseModel):
    topic: str
    top_ranking_pages: List[str]
    common_subtopics: List[str]
    content_gaps: List[str]
    recommended_word_count: str

result = app.agent(
    prompt="Research the topic 'local SEO for plumbers' — what are the top ranking pages, common subtopics covered, and content gaps I could fill?",
    schema=TopicResearch,
    model="spark-1-mini"
)

Choosing the Right Model

Model	When to Use
spark-1-mini (default)	Simple extraction, well-structured sites, high-volume, cost-sensitive
spark-1-pro	Complex analysis, deep reasoning, accuracy-critical, ambiguous data

Cost Management

• Set maxCredits to cap spending per job
• Most runs consume a few hundred credits
• Free: 5 daily runs for all users
• spark-1-mini is 60% cheaper than spark-1-pro

maxCredits

If the credit limit is hit, the job fails with no data — but used credits are still charged. Start with a reasonable cap and increase if needed.

Troubleshooting

Issue	Solution
Job failed, no data	Increase maxCredits — cap was likely hit
Results inaccurate	Switch to spark-1-pro for better accuracy
Job taking too long	Use start_agent + poll pattern for long research
Need more control	Provide urls parameter to focus the agent

---

← Back to SOPs