Firecrawl API SOP

Appearance

SOP 006: Change Tracking ​

New

Monitor websites for content changes using Firecrawl's persistent snapshot comparison.

When to Use ​

  • Monitor competitor pricing pages
  • Track product availability changes
  • Detect content updates on key pages
  • Watch for new blog posts or announcements

Prerequisites ​

  • Firecrawl API key
  • pip install firecrawl-py

Procedure ​

Step 1: Baseline Scrape (First Run) ​

The first scrape creates the initial snapshot. Status will be "new":

python
from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR-API-KEY")

result = app.scrape(
    "https://competitor.com/pricing",
    formats=["markdown", "changeTracking"]
)

print(result.changeTracking.changeStatus)  # "new"
print("Baseline snapshot created")

Step 2: Subsequent Scrapes (Detect Changes) ​

Run the same scrape later to detect changes:

python
result = app.scrape(
    "https://competitor.com/pricing",
    formats=["markdown", "changeTracking"]
)

status = result.changeTracking.changeStatus
if status == "changed":
    print("CHANGE DETECTED!")
elif status == "same":
    print("No changes")
elif status == "removed":
    print("Page removed!")

Step 3: Get the Diff (Git-Diff Mode) ​

See exactly what changed:

python
result = app.scrape(
    "https://competitor.com/pricing",
    formats=["markdown", {"type": "changeTracking", "modes": ["git-diff"]}]
)

if result.changeTracking.changeStatus == "changed":
    print("=== DIFF ===")
    print(result.changeTracking.diff.text)

Step 4: Field-Level Comparison (JSON Mode) ​

Track specific data fields:

python
result = app.scrape(
    "https://competitor.com/pricing",
    formats=["markdown", {
        "type": "changeTracking",
        "modes": ["json"],
        "schema": {
            "type": "object",
            "properties": {
                "starter_price": {"type": "string"},
                "pro_price": {"type": "string"},
                "enterprise_price": {"type": "string"}
            }
        }
    }]
)

if result.changeTracking.changeStatus == "changed":
    for field, values in result.changeTracking.json.items():
        if values.get("previous") != values.get("current"):
            print(f"{field}: {values['previous']} → {values['current']}")

Full Monitoring Script ​

Monitor multiple competitor pages on a schedule:

python
from firecrawl import Firecrawl
import json
from datetime import datetime

app = Firecrawl(api_key="fc-YOUR-API-KEY")

WATCH_URLS = [
    "https://competitor-a.com/pricing",
    "https://competitor-b.com/pricing",
    "https://competitor-c.com/features",
]

def check_changes():
    changes = []
    for url in WATCH_URLS:
        result = app.scrape(
            url,
            formats=["markdown", {"type": "changeTracking", "modes": ["git-diff"]}]
        )
        status = result.changeTracking.changeStatus
        if status == "changed":
            changes.append({
                "url": url,
                "status": status,
                "diff": result.changeTracking.diff.text,
                "detected_at": datetime.now().isoformat()
            })
            print(f"CHANGED: {url}")
        else:
            print(f"{status}: {url}")

    if changes:
        with open("changes.json", "w") as f:
            json.dump(changes, f, indent=2)
        print(f"\n{len(changes)} changes detected — saved to changes.json")

    return changes

# Run this on a cron schedule
check_changes()

Site-Wide Monitoring with Crawl ​

Monitor an entire site for changes:

python
result = app.crawl(
    "https://competitor.com",
    limit=50,
    scrape_options={"formats": ["markdown", "changeTracking"]}
)

changed_pages = []
new_pages = []
removed_pages = []

for page in result.data:
    status = page.changeTracking.changeStatus
    url = page.metadata.url
    if status == "changed":
        changed_pages.append(url)
    elif status == "new":
        new_pages.append(url)
    elif status == "removed":
        removed_pages.append(url)

print(f"Changed: {len(changed_pages)}")
print(f"New pages: {len(new_pages)}")
print(f"Removed: {len(removed_pages)}")

Using Tags for Multiple Frequencies ​

Track the same URL at different intervals with separate histories:

python
# Hourly check (separate history)
result = app.scrape(
    "https://competitor.com/pricing",
    formats=["markdown", {"type": "changeTracking", "tag": "hourly"}]
)

# Daily check (separate history)
result = app.scrape(
    "https://competitor.com/pricing",
    formats=["markdown", {"type": "changeTracking", "tag": "daily"}]
)

Cost ​

  • Basic + git-diff: No extra cost (just the normal 1 credit per scrape)
  • JSON mode: +5 credits per page
  • Snapshots are stored permanently — no expiry

Important Notes ​

  • markdown format must always be included alongside changeTracking
  • Cache is bypassed (maxAge is ignored)
  • First scrape always returns status "new"
  • Snapshots are scoped to your team
  • Resistant to whitespace and content order changes

← Back to SOPs