SOP 005: Browser Automation

New

Use Firecrawl Browser Sandbox for interactive web tasks that require clicking, typing, scrolling, and multi-step navigation.

When to Use Browser (vs Scrape)

Scenario	Use
Extract content from a known URL	Scrape
Fill forms, log in, navigate pagination	Browser
Multi-step workflows with clicks	Browser
Run Playwright code remotely	Browser
Scrape content behind authentication	Browser

Prerequisites

• Firecrawl API key
• pip install firecrawl-py or npm install @mendable/firecrawl-js

Procedure

Step 1: Launch a Browser Session

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR-API-KEY")

session = app.browser(ttl=300, activity_ttl=120)
print(f"Session: {session.id}")
print(f"Live View: {session.liveViewUrl}")

Step 2: Navigate and Interact

Option A — agent-browser CLI (simplest)

# Navigate
app.browser_execute(session.id, code='agent-browser open https://example.com', language="bash")

# Get page structure
result = app.browser_execute(session.id, code='agent-browser snapshot', language="bash")
print(result.result)  # Accessibility tree with @ref IDs

# Click an element
app.browser_execute(session.id, code='agent-browser click @e5', language="bash")

# Type into a field
app.browser_execute(session.id, code='agent-browser type @e3 "search query"', language="bash")

# Take screenshot
app.browser_execute(session.id, code='agent-browser screenshot /tmp/page.png', language="bash")

Option B — Playwright (full control)

result = app.browser_execute(
    session.id,
    code='''
await page.goto("https://example.com/login")
await page.fill('#email', 'user@example.com')
await page.fill('#password', 'secret123')
await page.click('button[type="submit"]')
await page.wait_for_load_state('networkidle')
title = await page.title()
print(title)
    ''',
    language="python",
)
print(result.result)

Step 3: Extract Data

result = app.browser_execute(
    session.id,
    code='''
await page.goto("https://example.com/dashboard")
data = await page.evaluate('''
    () => {
        const rows = document.querySelectorAll('table tbody tr');
        return Array.from(rows).map(row => ({
            name: row.cells[0].textContent,
            value: row.cells[1].textContent
        }));
    }
''')
import json
print(json.dumps(data))
    ''',
    language="python",
)

Step 4: Close Session

app.delete_browser(session.id)

Multi-Page Workflow Example

Scrape paginated search results:

from firecrawl import Firecrawl
import json

app = Firecrawl(api_key="fc-YOUR-API-KEY")
session = app.browser(ttl=600)

all_results = []

# Navigate to first page
app.browser_execute(session.id,
    code='agent-browser open https://example.com/search?q=firecrawl',
    language="bash")

for page_num in range(1, 6):
    # Get snapshot to find elements
    snapshot = app.browser_execute(session.id,
        code='agent-browser snapshot', language="bash")

    # Extract data with Playwright
    result = app.browser_execute(session.id,
        code='''
const items = await page.$$eval('.result-item', els =>
    els.map(el => ({
        title: el.querySelector('h3')?.textContent?.trim(),
        url: el.querySelector('a')?.href,
        desc: el.querySelector('p')?.textContent?.trim()
    }))
);
console.log(JSON.stringify(items));
        ''',
        language="node")

    items = json.loads(result.result)
    all_results.extend(items)

    # Click next page (if exists)
    app.browser_execute(session.id,
        code='agent-browser click @next-page', language="bash")

app.delete_browser(session.id)
print(f"Collected {len(all_results)} results across 5 pages")

Persistent Session Example

Save login state for reuse:

# First time — log in and save
session = app.browser(ttl=300, profile={"name": "my-app-login", "saveChanges": True})

app.browser_execute(session.id, code='''
await page.goto("https://app.example.com/login")
await page.fill('#email', 'user@example.com')
await page.fill('#password', 'secret')
await page.click('button[type="submit"]')
await page.wait_for_load_state('networkidle')
''', language="python")

app.delete_browser(session.id)  # Saves profile

# Later — reuse session (already logged in)
session = app.browser(ttl=300, profile={"name": "my-app-login", "saveChanges": False})
# Navigate directly to authenticated pages

Cost

• 2 credits per browser minute
• 5-minute session = 10 credits
• Free users get 5 hours of free usage

Troubleshooting

Issue	Solution
Session expired	Increase ttl parameter
Inactivity timeout	Increase activityTtl or send periodic commands
Profile conflict (409)	Only one write session per profile — use saveChanges: false for concurrent reads
Element not found	Use agent-browser snapshot to get current element refs

---

← Back to SOPs