Firecrawl API SOP

Appearance

Data Migration ​

Updated Feb 2026

Transfer web data seamlessly between platforms and systems. Firecrawl extracts content, metadata, media, and structure from existing websites so you can migrate to new platforms without losing data or breaking SEO.

Why Firecrawl for Data Migration ​

Platform migrations are painful. Whether you are moving from WordPress to a headless CMS, switching e-commerce platforms, or onboarding a new customer, you need to extract everything -- content, structure, metadata, images, and relationships -- from the old system. Firecrawl handles the extraction side so you can focus on the import.

What You Can Migrate ​

Data TypeExamples
ContentPages, posts, articles, media files, documentation
StructureHierarchies, categories, tags, taxonomies, navigation
MetadataSEO titles, descriptions, canonical URLs, Open Graph tags
E-commerceProducts, variants, inventory, pricing, descriptions
User contentPublicly visible reviews, comments, testimonials
ConfigurationCustom fields, form structures, widget settings

How It Works ​

Firecrawl Features Used ​

FeatureRole in Data Migration
CrawlExtract all pages from a source site recursively
ScrapePull structured data from specific page types
MapDiscover every URL on the source site for completeness
Change TrackingValidate post-migration that content matches
BrowserHandle JavaScript-rendered pages and SPAs

CMS Migration Workflow ​

Step 1: Discover All Content ​

python
from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

# Map the entire source site
site_map = app.map(url="https://old-website.com")

print(f"Total pages found: {len(site_map['links'])}")

# Categorize URLs by type
blog_posts = [u for u in site_map["links"] if "/blog/" in u]
pages = [u for u in site_map["links"] if "/blog/" not in u]
products = [u for u in site_map["links"] if "/product/" in u]

print(f"Blog posts: {len(blog_posts)}")
print(f"Pages: {len(pages)}")
print(f"Products: {len(products)}")

Step 2: Extract Content with Metadata ​

python
# Crawl the entire site with full metadata
result = app.crawl(
    url="https://old-website.com",
    params={
        "limit": 5000,
        "scrapeOptions": {
            "formats": ["markdown", "html", "links", "images"],
        }
    }
)

# Process each page
migration_data = []
for page in result["data"]:
    migration_data.append({
        "url": page["metadata"]["url"],
        "title": page["metadata"].get("title", ""),
        "description": page["metadata"].get("description", ""),
        "og_image": page["metadata"].get("ogImage", ""),
        "markdown": page["markdown"],
        "html": page["html"],
        "images": page.get("images", []),
        "links": page.get("links", []),
    })

Step 3: Extract Structured Data for Specific Page Types ​

python
# Extract structured blog post data
for url in blog_posts:
    result = app.scrape(
        url=url,
        params={
            "formats": ["json"],
            "jsonOptions": {
                "schema": {
                    "type": "object",
                    "properties": {
                        "title": {"type": "string"},
                        "author": {"type": "string"},
                        "publish_date": {"type": "string"},
                        "categories": {
                            "type": "array",
                            "items": {"type": "string"}
                        },
                        "tags": {
                            "type": "array",
                            "items": {"type": "string"}
                        },
                        "featured_image": {"type": "string"},
                        "body_content": {"type": "string"}
                    }
                }
            }
        }
    )
    # Store for import into new CMS

Step 4: Generate Redirect Map ​

python
# Build a redirect map from old URLs to new URL structure
redirect_map = []
for page in migration_data:
    old_path = page["url"].replace("https://old-website.com", "")
    new_path = transform_url(old_path)  # Your URL transformation logic

    if old_path != new_path:
        redirect_map.append({
            "from": old_path,
            "to": new_path,
            "status": 301
        })

# Export as CSV or import directly into your server config

E-Commerce Platform Migration ​

Moving between e-commerce platforms (Magento to Shopify, WooCommerce to BigCommerce, etc.) requires extracting product catalogs with all variants and metadata:

python
# Extract product catalog with full details
result = app.scrape(
    url="https://store.com/products/example-product",
    params={
        "formats": ["json"],
        "jsonOptions": {
            "schema": {
                "type": "object",
                "properties": {
                    "product_name": {"type": "string"},
                    "sku": {"type": "string"},
                    "price": {"type": "string"},
                    "sale_price": {"type": "string"},
                    "description": {"type": "string"},
                    "specifications": {
                        "type": "object",
                        "additionalProperties": {"type": "string"}
                    },
                    "variants": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "properties": {
                                "name": {"type": "string"},
                                "price": {"type": "string"},
                                "sku": {"type": "string"},
                                "in_stock": {"type": "boolean"}
                            }
                        }
                    },
                    "images": {
                        "type": "array",
                        "items": {"type": "string"}
                    },
                    "categories": {
                        "type": "array",
                        "items": {"type": "string"}
                    }
                }
            }
        }
    }
)

Handling Scale ​

Firecrawl handles large-scale migrations with automatic infrastructure scaling:

  • Batching -- Process pages in parallel batches to handle sites with thousands of pages
  • Incremental processing -- Resume interrupted migrations without re-extracting completed pages
  • Rate limiting -- Automatic throttling to avoid overwhelming source servers
python
# For large sites, use batch scrape with parallel processing
all_urls = site_map["links"]  # Could be thousands

# Process in batches
batch_result = app.batch_scrape(
    urls=all_urls,
    params={
        "formats": ["markdown", "links", "images"]
    }
)

# Track progress
print(f"Successfully extracted: {len(batch_result['data'])} pages")

Post-Migration Validation ​

After importing data into the new platform, validate that everything transferred correctly:

python
# Compare old and new pages
for page in migration_data:
    old_url = page["url"]
    new_url = old_url.replace("old-website.com", "new-website.com")

    old_content = app.scrape(
        url=old_url,
        params={"formats": ["markdown"]}
    )

    new_content = app.scrape(
        url=new_url,
        params={"formats": ["markdown"]}
    )

    # Compare content lengths as a quick check
    old_len = len(old_content["markdown"])
    new_len = len(new_content["markdown"])
    diff_pct = abs(old_len - new_len) / max(old_len, 1) * 100

    if diff_pct > 10:
        print(f"WARNING: {old_url} content differs by {diff_pct:.1f}%")

SEO Preservation Checklist ​

Migrations can destroy SEO if not handled carefully. Use this checklist:

  • [ ] Map all URLs before migration using Map
  • [ ] Extract all metadata -- titles, descriptions, canonical tags, structured data
  • [ ] Catalog all images and media files with alt text
  • [ ] Build 301 redirect map from old URLs to new URLs
  • [ ] Preserve internal linking structure
  • [ ] Validate post-migration using Change Tracking to compare old vs. new
  • [ ] Submit updated sitemap to search engines
  • [ ] Monitor 404s for the first 30 days

Quick Start: Firecrawl Migrator Template ​

The Firecrawl Migrator template on GitHub provides a ready-made pipeline for extracting and migrating platform data. Clone it, configure your source and destination, and run.

Best Practices ​

  1. Always map first -- Use Map to discover every URL before starting extraction; do not assume you know all the pages
  2. Preserve metadata -- SEO metadata (titles, descriptions, canonicals) is just as important as body content
  3. Extract images separately -- Catalog all media files and their alt text for proper re-hosting
  4. Test with a subset -- Run the full pipeline on 50-100 pages before committing to a full migration
  5. Keep the old site live -- Do not take down the source until you have verified the migration is complete
  6. Set up redirects immediately -- 301 redirects should go live at the exact moment the new site launches