Firecrawl API SOP

Appearance

SOP 002: JSON Data Extraction ​

Fresh 🌱

Extract structured data from websites using JSON schema.

Overview ​

Prerequisites ​

  • ✓ Basic scraping knowledge (SOP 001)
  • ✓ Understanding of JSON schemas
  • ✓ Python Pydantic or equivalent

Step-by-Step ​

Step 1: Define Data Schema ​

Python with Pydantic:

python
from pydantic import BaseModel
from typing import Optional

class ProductInfo(BaseModel):
    name: str
    price: float
    rating: Optional[float] = None
    description: str
    in_stock: bool
    sku: str

Step 2: Initialize Firecrawl ​

python
from firecrawl import Firecrawl

firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")

Step 3: Extract Data ​

python
result = firecrawl.scrape(
    url="https://example.com/product",
    formats=[{
        "type": "json",
        "schema": ProductInfo.model_json_schema()
    }]
)

Step 4: Parse Response ​

python
extracted_data = result['json']
print(f"Product: {extracted_data['name']}")
print(f"Price: ${extracted_data['price']}")
print(f"Rating: {extracted_data['rating']}/5")

Step 5: Validate Data ​

python
from pydantic import ValidationError

try:
    product = ProductInfo(**extracted_data)
    print(f"Validated: {product}")
except ValidationError as e:
    print(f"Validation error: {e}")

Common Schemas ​

Company Information ​

python
class CompanyInfo(BaseModel):
    name: str
    description: str
    founded_year: int
    headquarters: str
    employees_count: int
    industry: str

Article/Blog Post ​

python
class Article(BaseModel):
    title: str
    author: str
    published_date: str
    content: str
    tags: list[str]
    word_count: int

E-Commerce Product ​

python
class Product(BaseModel):
    name: str
    price: float
    original_price: Optional[float]
    rating: float
    reviews_count: int
    in_stock: bool
    sku: str

Advanced: Without Schema ​

Use prompt-based extraction:

python
result = firecrawl.scrape(
    url="https://example.com",
    formats=[{
        "type": "json",
        "prompt": "Extract the company mission, team size, and founded year"
    }]
)

Verification Checklist ​

  • [ ] Schema matches website structure
  • [ ] All required fields extracted
  • [ ] Data types are correct
  • [ ] Optional fields handled properly
  • [ ] Response validates against schema

Troubleshooting ​

Error: "Schema mismatch" ​

  • Review HTML structure of page
  • Adjust schema to match available data
  • Make fields optional if not always present

Missing fields ​

  • Field might not exist on page
  • Make field optional: Optional[str] = None
  • Use prompt-based extraction as fallback

Validation errors ​

  • Check data types (string vs number)
  • Verify required vs optional
  • Add type conversions if needed

Next Steps ​

Cost: 1 credit (base) + 4 credits (JSON mode) = 5 credits per scrape