Appearance
Scrape
Note: A new v2 version of this API is now available with improved features and performance.
OpenAPI
yaml
openapi: 3.0.0
info:
title: Firecrawl API
version: v1
description: >-
API for interacting with Firecrawl services to perform web scraping and
crawling tasks.
contact:
name: Firecrawl Support
url: https://firecrawl.dev/support
email: support@firecrawl.dev
servers:
- url: https://api.firecrawl.dev/v1
security:
- bearerAuth: []
paths:
/scrape:
post:
tags:
- Scraping
summary: Scrape a single URL and optionally extract information using an LLM
operationId: scrapeAndExtractFromUrl
requestBody:
required: true
content:
application/json:
schema:
allOf:
- type: object
properties:
url:
type: string
format: uri
description: The URL to scrape
required:
- url
- $ref: '#/components/schemas/ScrapeOptions'
- type: object
properties:
zeroDataRetention:
type: boolean
default: false
description: >-
If true, this will enable zero data retention for this
scrape. To enable this feature, please contact
help@firecrawl.dev
responses:
'200':
description: Successful response
content:
application/json:
schema:
$ref: '#/components/schemas/ScrapeResponse'
'402':
description: Payment required
content:
application/json:
schema:
type: object
properties:
error:
type: string
example: Payment required to access this resource.
'429':
description: Too many requests
content:
application/json:
schema:
type: object
properties:
error:
type: string
example: >-
Request rate limit exceeded. Please wait and try again
later.
'500':
description: Server error
content:
application/json:
schema:
type: object
properties:
error:
type: string
example: An unexpected error occurred on the server.
security:
- bearerAuth: []
components:
schemas:
ScrapeOptions:
allOf:
- $ref: '#/components/schemas/BaseScrapeOptions'
- type: object
properties:
formats:
type: array
items:
type: string
enum:
- markdown
- html
- rawHtml
- links
- screenshot
- screenshot@fullPage
- json
- changeTracking
description: Formats to include in the output.
default:
- markdown
changeTrackingOptions:
type: object
description: >-
Options for change tracking (Beta). Only applicable when
'changeTracking' is included in formats. The 'markdown' format
must also be specified when using change tracking.
properties:
modes:
type: array
items:
type: string
enum:
- git-diff
- json
description: >-
The mode to use for change tracking. 'git-diff' provides a
detailed diff, and 'json' compares extracted JSON data.
schema:
type: object
description: >-
Schema for JSON extraction when using 'json' mode. Defines
the structure of data to extract and compare. Must conform
to JSON Schema.
prompt:
type: string
description: >-
Prompt to use for change tracking when using 'json' mode. If
not provided, the default prompt will be used.
tag:
type: string
nullable: true
default: null
description: >-
Tag to use for change tracking. Tags can separate change
tracking history into separate "branches", where change
tracking with a specific tagwill only compare to scrapes
made in the same tag. If not provided, the default tag
(null) will be used.
ScrapeResponse:
type: object
properties:
success:
type: boolean
data:
type: object
properties:
markdown:
type: string
html:
type: string
nullable: true
description: >-
Cleaned HTML of the page if `html` is in `formats`. Removes
`<script>`, `<style>`, `<noscript>`, `<meta>`, and `<head>`
tags; converts relative URLs to absolute; resolves responsive
image `srcset` to the largest version. Respects
`onlyMainContent`, `includeTags`, and `excludeTags` filters.
rawHtml:
type: string
nullable: true
description: >-
The exact, unmodified HTML as received from the page if
`rawHtml` is in `formats`. No cleaning or filtering is applied.
screenshot:
type: string
nullable: true
description: >-
Screenshot of the page if `screenshot` is in `formats`.
Screenshots expire after 24 hours and can no longer be
downloaded.
links:
type: array
items:
type: string
description: List of links on the page if `links` is in `formats`
actions:
type: object
nullable: true
description: >-
Results of the actions specified in the `actions` parameter.
Only present if the `actions` parameter was provided in the
request
properties:
screenshots:
type: array
description: >-
Screenshot URLs, in the same order as the screenshot actions
provided. Screenshots expire after 24 hours and can no
longer be downloaded.
items:
type: string
format: url
scrapes:
type: array
description: >-
Scrape contents, in the same order as the scrape actions
provided.
items:
type: object
properties:
url:
type: string
html:
type: string
javascriptReturns:
type: array
description: >-
JavaScript return values, in the same order as the
executeJavascript actions provided.
items:
type: object
properties:
type:
type: string
value: {}
pdfs:
type: array
description: >-
PDFs generated, in the same order as the pdf actions
provided.
items:
type: string
metadata:
type: object
properties:
title:
type: string
description:
type: string
language:
type: string
nullable: true
sourceURL:
type: string
format: uri
keywords:
oneOf:
- type: string
- type: array
items:
type: string
description: >-
Keywords extracted from the page, can be a string or array
of strings
ogLocaleAlternate:
type: array
items:
type: string
description: Alternative locales for the page
'<any other metadata> ':
type: string
statusCode:
type: integer
description: The status code of the page
error:
type: string
nullable: true
description: The error message of the page
llm_extraction:
type: object
description: >-
Displayed when using LLM Extraction. Extracted data from the
page following the schema defined.
nullable: true
warning:
type: string
nullable: true
description: >-
Can be displayed when using LLM Extraction. Warning message will
let you know any issues with the extraction.
changeTracking:
type: object
nullable: true
description: >-
Change tracking information if `changeTracking` is in `formats`.
Only present when the `changeTracking` format is requested.
properties:
previousScrapeAt:
type: string
format: date-time
nullable: true
description: >-
The timestamp of the previous scrape that the current page
is being compared against. Null if no previous scrape
exists.
changeStatus:
type: string
enum:
- new
- same
- changed
- removed
description: >-
The result of the comparison between the two page versions.
'new' means this page did not exist before, 'same' means
content has not changed, 'changed' means content has
changed, 'removed' means the page was removed.
visibility:
type: string
enum:
- visible
- hidden
description: >-
The visibility of the current page/URL. 'visible' means the
URL was discovered through an organic route (links or
sitemap), 'hidden' means the URL was discovered through
memory from previous crawls.
diff:
type: string
nullable: true
description: >-
Git-style diff of changes when using 'git-diff' mode. Only
present when the mode is set to 'git-diff'.
json:
type: object
nullable: true
description: >-
JSON comparison results when using 'json' mode. Only present
when the mode is set to 'json'. This will emit a list of all
the keys and their values from the `previous` and `current`
scrapes based on the type defined in the `schema`. Example
[here](/features/change-tracking)
BaseScrapeOptions:
type: object
properties:
onlyMainContent:
type: boolean
description: >-
Only return the main content of the page excluding headers, navs,
footers, etc.
default: true
includeTags:
type: array
items:
type: string
description: Tags to include in the output.
excludeTags:
type: array
items:
type: string
description: Tags to exclude from the output.
maxAge:
type: integer
description: >-
Returns a cached version of the page if it is younger than this age
in milliseconds. If a cached version of the page is older than this
value, the page will be scraped. If you do not need extremely fresh
data, enabling this can speed up your scrapes by 500%. Defaults to
0, which disables caching.
default: 0
headers:
type: object
description: >-
Headers to send with the request. Can be used to send cookies,
user-agent, etc.
waitFor:
type: integer
description: >-
Specify a delay in milliseconds before fetching the content,
allowing the page sufficient time to load.
default: 0
mobile:
type: boolean
description: >-
Set to true if you want to emulate scraping from a mobile device.
Useful for testing responsive pages and taking mobile screenshots.
default: false
skipTlsVerification:
type: boolean
description: Skip TLS certificate verification when making requests
default: false
timeout:
type: integer
description: Timeout in milliseconds for the request
default: 30000
parsePDF:
type: boolean
description: >-
Controls how PDF files are processed during scraping. When true, the
PDF content is extracted and converted to markdown format, with
billing based on the number of pages (1 credit per page). When
false, the PDF file is returned in base64 encoding with a flat rate
of 1 credit total.
default: true
jsonOptions:
type: object
description: JSON options object
properties:
schema:
type: object
description: >-
The schema to use for the extraction (Optional). Must conform to
JSON Schema.
systemPrompt:
type: string
description: The system prompt to use for the extraction (Optional)
prompt:
type: string
description: The prompt to use for the extraction without a schema (Optional)
actions:
type: array
description: Actions to perform on the page before grabbing the content
items:
oneOf:
- type: object
title: Wait
properties:
type:
type: string
enum:
- wait
description: Wait for a specified amount of milliseconds
milliseconds:
type: integer
minimum: 1
description: Number of milliseconds to wait
selector:
type: string
description: Query selector to find the element by
example: '#my-element'
required:
- type
- type: object
title: Screenshot
properties:
type:
type: string
enum:
- screenshot
description: >-
Take a screenshot. The links will be in the response's
`actions.screenshots` array.
fullPage:
type: boolean
description: >-
Whether to capture a full-page screenshot or limit to the
current viewport.
default: false
quality:
type: integer
description: >-
The quality of the screenshot, from 1 to 100. 100 is the
highest quality.
required:
- type
- type: object
title: Click
properties:
type:
type: string
enum:
- click
description: Click on an element
selector:
type: string
description: Query selector to find the element by
example: '#load-more-button'
all:
type: boolean
description: >-
Clicks all elements matched by the selector, not just the
first one. Does not throw an error if no elements match
the selector.
default: false
required:
- type
- selector
- type: object
title: Write text
properties:
type:
type: string
enum:
- write
description: >-
Write text into an input field, text area, or
contenteditable element. Note: You must first focus the
element using a 'click' action before writing. The text
will be typed character by character to simulate keyboard
input.
text:
type: string
description: Text to type
example: Hello, world!
required:
- type
- text
- type: object
title: Press a key
description: >-
Press a key on the page. See
https://asawicki.info/nosense/doc/devices/keyboard/key_codes.html
for key codes.
properties:
type:
type: string
enum:
- press
description: Press a key on the page
key:
type: string
description: Key to press
example: Enter
required:
- type
- key
- type: object
title: Scroll
properties:
type:
type: string
enum:
- scroll
description: Scroll the page or a specific element
direction:
type: string
enum:
- up
- down
description: Direction to scroll
default: down
selector:
type: string
description: Query selector for the element to scroll
example: '#my-element'
required:
- type
- type: object
title: Scrape
properties:
type:
type: string
enum:
- scrape
description: >-
Scrape the current page content, returns the url and the
html.
required:
- type
- type: object
title: Execute JavaScript
properties:
type:
type: string
enum:
- executeJavascript
description: Execute JavaScript code on the page
script:
type: string
description: JavaScript code to execute
example: document.querySelector('.button').click();
required:
- type
- script
- type: object
title: Generate PDF
properties:
type:
type: string
enum:
- pdf
description: >-
Generate a PDF of the current page. The PDF will be
returned in the `actions.pdfs` array of the response.
format:
type: string
enum:
- A0
- A1
- A2
- A3
- A4
- A5
- A6
- Letter
- Legal
- Tabloid
- Ledger
description: The page size of the resulting PDF
default: Letter
landscape:
type: boolean
description: Whether to generate the PDF in landscape orientation
default: false
scale:
type: number
description: The scale multiplier of the resulting PDF
default: 1
required:
- type
location:
type: object
description: >-
Location settings for the request. When specified, this will use an
appropriate proxy if available and emulate the corresponding
language and timezone settings. Defaults to 'US' if not specified.
properties:
country:
type: string
description: ISO 3166-1 alpha-2 country code (e.g., 'US', 'AU', 'DE', 'JP')
pattern: ^[A-Z]{2}$
default: US
languages:
type: array
description: >-
Preferred languages and locales for the request in order of
priority. Defaults to the language of the specified location.
See
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Language
items:
type: string
example: en-US
removeBase64Images:
type: boolean
description: >-
Removes all base 64 images from the output, which may be
overwhelmingly long. The image's alt text remains in the output, but
the URL is replaced with a placeholder.
default: true
blockAds:
type: boolean
description: Enables ad-blocking and cookie popup blocking.
default: true
proxy:
type: string
enum:
- basic
- enhanced
- auto
description: |-
Specifies the type of proxy to use.
- **basic**: Proxies for scraping sites with none to basic anti-bot solutions. Fast and usually works.
- **enhanced**: Enhanced proxies for scraping sites with advanced anti-bot solutions. Slower, but more reliable on certain sites. Costs up to 5 credits per request.
- **auto**: Firecrawl will automatically retry scraping with enhanced proxies if the basic proxy fails. If the retry with enhanced is successful, 5 credits will be billed for the scrape. If the first attempt with basic is successful, only the regular cost will be billed.
If you do not specify a proxy, Firecrawl will default to basic.
storeInCache:
type: boolean
description: >-
If true, the page will be stored in the Firecrawl index and cache.
Setting this to false is useful if your scraping activity may have
data protection concerns. Using some parameters associated with
sensitive scraping (actions, headers) will force this parameter to
be false.
default: true
securitySchemes:
bearerAuth:
type: http
scheme: bearer