Firecrawl API SOP

Appearance

Cybersecurity ​

Updated Feb 2026

Detect vulnerabilities, monitor attack surfaces, audit security posture, and track exposure across web properties. Firecrawl gives security teams the extraction capabilities to continuously assess and monitor web-facing assets at scale.

Note

Firecrawl does not currently have a dedicated cybersecurity page in their official docs. This guide covers practical security use cases based on Firecrawl's core extraction, crawling, and change tracking capabilities.

Why Firecrawl for Cybersecurity ​

Security teams need continuous visibility into their web-facing assets. Misconfigurations, exposed endpoints, stale pages, and forgotten subdomains create attack surfaces that adversaries exploit. Firecrawl automates the discovery and monitoring of these surfaces so your team can remediate issues before they become incidents.

Security Use Cases ​

Use CaseDescription
Attack Surface DiscoveryFind all pages, endpoints, and subdomains across your web properties
Configuration MonitoringDetect changes to security headers, SSL settings, and access controls
Exposure DetectionIdentify accidentally published sensitive information
Compliance AuditingVerify privacy policies, terms, and security disclosures are current
Incident DetectionMonitor for defacement, unauthorized changes, and injected content
Third-Party RiskAssess vendor and partner web security posture

How It Works ​

Firecrawl Features Used ​

FeatureRole in Cybersecurity
MapDiscover all URLs, endpoints, and pages across web properties
CrawlDeep-scan entire sites for exposed content and misconfigurations
ScrapeExtract security headers, SSL details, and page content from specific URLs
Change TrackingDetect unauthorized changes, defacement, and injected content
BrowserRender JavaScript-heavy applications to inspect actual client-side behavior

Attack Surface Discovery ​

Map All Web Assets ​

python
from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

# Discover all pages on your domain
site_map = app.map(url="https://yourcompany.com")

print(f"Total URLs discovered: {len(site_map['links'])}")

# Categorize by type
api_endpoints = []
admin_pages = []
static_files = []
standard_pages = []

for url in site_map["links"]:
    lower = url.lower()
    if "/api/" in lower or "/v1/" in lower or "/v2/" in lower:
        api_endpoints.append(url)
    elif "/admin" in lower or "/dashboard" in lower or "/login" in lower:
        admin_pages.append(url)
    elif any(ext in lower for ext in [".json", ".xml", ".txt", ".env", ".yml"]):
        static_files.append(url)
    else:
        standard_pages.append(url)

print(f"API endpoints: {len(api_endpoints)}")
print(f"Admin/auth pages: {len(admin_pages)}")
print(f"Static/config files: {len(static_files)}")
print(f"Standard pages: {len(standard_pages)}")

# Flag potential exposures
if static_files:
    print("\nPOTENTIAL EXPOSURES:")
    for f in static_files:
        print(f"  {f}")

Track New and Removed Pages ​

python
import json
from datetime import datetime

# Load previous scan
try:
    with open("surface_scan.json", "r") as f:
        previous = json.load(f)
        previous_urls = set(previous["urls"])
except FileNotFoundError:
    previous_urls = set()

current_urls = set(site_map["links"])

# Detect changes
new_urls = current_urls - previous_urls
removed_urls = previous_urls - current_urls

if new_urls:
    print(f"NEW PAGES DETECTED ({len(new_urls)}):")
    for url in sorted(new_urls):
        # Flag suspicious additions
        risk = "HIGH" if any(w in url.lower() for w in
            ["admin", "debug", "test", "staging", ".env", "backup"]) else "LOW"
        print(f"  [{risk}] {url}")

if removed_urls:
    print(f"PAGES REMOVED ({len(removed_urls)}):")
    for url in sorted(removed_urls):
        print(f"  {url}")

# Save current scan
with open("surface_scan.json", "w") as f:
    json.dump({
        "timestamp": datetime.now().isoformat(),
        "urls": list(current_urls)
    }, f)

Security Header Monitoring ​

Check Security Headers on Critical Pages ​

python
# Extract and validate security-relevant metadata
critical_pages = [
    "https://yourcompany.com",
    "https://yourcompany.com/login",
    "https://yourcompany.com/api/v1/status",
    "https://app.yourcompany.com",
]

for url in critical_pages:
    result = app.scrape(
        url=url,
        params={
            "formats": ["json"],
            "jsonOptions": {
                "schema": {
                    "type": "object",
                    "properties": {
                        "has_https": {"type": "boolean"},
                        "page_title": {"type": "string"},
                        "has_login_form": {"type": "boolean"},
                        "external_scripts": {
                            "type": "array",
                            "items": {"type": "string"}
                        },
                        "form_actions": {
                            "type": "array",
                            "items": {"type": "string"}
                        },
                        "visible_error_messages": {
                            "type": "array",
                            "items": {"type": "string"}
                        }
                    }
                }
            }
        }
    )

    data = result["json"]
    print(f"\n{url}")
    if data.get("visible_error_messages"):
        print(f"  WARNING: Error messages visible: {data['visible_error_messages']}")
    if data.get("external_scripts"):
        print(f"  External scripts: {len(data['external_scripts'])}")
        for script in data["external_scripts"]:
            print(f"    - {script}")

Defacement and Tampering Detection ​

Monitor for Unauthorized Content Changes ​

python
# Monitor critical pages for unauthorized changes
monitored_pages = [
    {"url": "https://yourcompany.com", "name": "Homepage"},
    {"url": "https://yourcompany.com/login", "name": "Login"},
    {"url": "https://yourcompany.com/checkout", "name": "Checkout"},
]

for page in monitored_pages:
    result = app.scrape(
        url=page["url"],
        params={"formats": ["markdown", "changeTracking"]}
    )

    status = result.get("changeTracking", {}).get("changeStatus")
    if status == "changed":
        diff = result["changeTracking"]["diff"]

        # Check for suspicious indicators
        suspicious_patterns = [
            "<script",       # Injected scripts
            "<iframe",       # Injected iframes
            "eval(",         # Code injection
            "document.cookie", # Cookie theft
            "base64",        # Encoded payloads
            "crypto",        # Cryptominer injection
        ]

        is_suspicious = any(
            pattern.lower() in diff.lower()
            for pattern in suspicious_patterns
        )

        if is_suspicious:
            print(f"SECURITY ALERT: Suspicious change on {page['name']}")
            print(f"  URL: {page['url']}")
            print(f"  Diff excerpt: {diff[:500]}")
        else:
            print(f"Content change on {page['name']} (appears benign)")

Third-Party Risk Assessment ​

Assess Vendor Security Posture ​

python
# Evaluate a vendor's web security basics
def assess_vendor(vendor_url):
    """Quick security assessment of a vendor's web presence."""

    # Map the vendor's site
    vendor_map = app.map(url=vendor_url)

    # Extract security-relevant signals
    assessment = app.scrape(
        url=vendor_url,
        params={
            "formats": ["json"],
            "jsonOptions": {
                "schema": {
                    "type": "object",
                    "properties": {
                        "has_privacy_policy": {"type": "boolean"},
                        "has_terms_of_service": {"type": "boolean"},
                        "has_security_page": {"type": "boolean"},
                        "has_soc2_mention": {"type": "boolean"},
                        "has_gdpr_mention": {"type": "boolean"},
                        "has_contact_info": {"type": "boolean"},
                        "copyright_year": {"type": "string"}
                    }
                }
            }
        }
    )

    data = assessment["json"]

    # Check for security-relevant pages
    security_pages = [
        u for u in vendor_map["links"]
        if any(w in u.lower() for w in [
            "security", "privacy", "compliance",
            "trust", "gdpr", "soc2", "status"
        ])
    ]

    report = {
        "vendor": vendor_url,
        "total_pages": len(vendor_map["links"]),
        "security_pages": security_pages,
        "has_privacy_policy": data.get("has_privacy_policy"),
        "has_security_page": data.get("has_security_page"),
        "compliance_mentions": {
            "SOC 2": data.get("has_soc2_mention"),
            "GDPR": data.get("has_gdpr_mention"),
        },
        "copyright_current": data.get("copyright_year", "") == "2026"
    }

    return report

# Assess multiple vendors
vendors = [
    "https://vendor-a.com",
    "https://vendor-b.com",
]

for vendor in vendors:
    report = assess_vendor(vendor)
    print(f"\nVendor: {report['vendor']}")
    print(f"  Pages: {report['total_pages']}")
    print(f"  Security pages: {len(report['security_pages'])}")
    print(f"  Privacy policy: {report['has_privacy_policy']}")
    print(f"  Security page: {report['has_security_page']}")

Compliance Monitoring ​

Track that required legal and compliance pages stay current:

python
# Monitor compliance pages for changes
compliance_pages = [
    "https://yourcompany.com/privacy-policy",
    "https://yourcompany.com/terms-of-service",
    "https://yourcompany.com/security",
    "https://yourcompany.com/data-processing-agreement",
]

for url in compliance_pages:
    result = app.scrape(
        url=url,
        params={"formats": ["markdown", "changeTracking"]}
    )

    content = result.get("markdown", "")
    change = result.get("changeTracking", {}).get("changeStatus")

    # Check for staleness
    word_count = len(content.split())
    if word_count < 500:
        print(f"WARNING: {url} seems incomplete ({word_count} words)")

    if change == "changed":
        print(f"COMPLIANCE PAGE UPDATED: {url}")
        print(f"  Review the changes to ensure they are authorized")

Exposed Information Detection ​

Scan for accidentally published sensitive content:

python
# Crawl the site looking for potential data exposure
crawl_result = app.crawl(
    url="https://yourcompany.com",
    params={
        "limit": 1000,
        "scrapeOptions": {
            "formats": ["markdown"]
        }
    }
)

# Scan content for sensitive patterns
import re

sensitive_patterns = {
    "API Key": r"(?:api[_-]?key|apikey)\s*[:=]\s*['\"]?[\w-]{20,}",
    "Email List": r"[\w.+-]+@[\w-]+\.[\w.-]+",
    "AWS Key": r"AKIA[0-9A-Z]{16}",
    "Private IP": r"(?:10\.\d{1,3}|172\.(?:1[6-9]|2\d|3[01])|192\.168)\.\d{1,3}\.\d{1,3}",
    "Phone Number": r"\b\d{3}[-.]?\d{3}[-.]?\d{4}\b",
}

findings = []
for page in crawl_result["data"]:
    url = page["metadata"]["url"]
    content = page.get("markdown", "")

    for pattern_name, pattern in sensitive_patterns.items():
        matches = re.findall(pattern, content)
        if matches:
            findings.append({
                "url": url,
                "pattern": pattern_name,
                "count": len(matches),
                "samples": matches[:3]
            })

if findings:
    print(f"EXPOSURE FINDINGS ({len(findings)}):")
    for f in findings:
        print(f"  {f['url']}")
        print(f"    Pattern: {f['pattern']} ({f['count']} matches)")
        print(f"    Samples: {f['samples']}")

Monitoring Frequency Guide ​

TargetIntervalRationale
Homepage / LoginEvery 15 minutesDefacement detection
Payment / CheckoutEvery 15 minutesSkimmer injection detection
API endpointsEvery hourUnauthorized endpoint exposure
Compliance pagesDailyRegulatory requirement
Full site mapWeeklyAttack surface changes
Vendor sitesMonthlyThird-party risk review

Best Practices ​

  1. Baseline first -- Establish a known-good baseline before enabling alerts; otherwise every change triggers a false positive
  2. Focus on high-value targets -- Login, payment, and admin pages deserve more frequent monitoring than blog posts
  3. Automate alerting -- Connect change detection to your SIEM or incident response workflow via webhooks
  4. Scan for patterns, not just changes -- Look for known malicious patterns (script injection, iframes, encoded payloads) in content diffs
  5. Monitor third parties -- Your security is only as strong as your weakest vendor; assess their posture regularly
  6. Track surface area over time -- Growing attack surface (new subdomains, endpoints, pages) increases risk; trend it monthly
  7. Respect scope -- Only monitor domains and properties you own or have authorization to assess