firecrawl.dev

•

API Key

•

Ready to use

About Firecrawl

Firecrawl is an API service that scrapes, crawls, and maps websites, converting dynamic and JavaScript-heavy pages into structured markdown or JSON optimized for large language models. It is designed for AI developers building retrieval-augmented generation (RAG) pipelines, AI agents, and data extraction workflows. Features include full-site crawling, AI-powered structured data extraction, and a credit-based pricing model with a free tier.

AI Agent Use Cases

- Scrape individual URLs into LLM-ready markdown with a single API call

- Full-site crawl to extract content from all discoverable pages

- Map endpoint to retrieve all URLs on a site without scraping content

- AI-powered Extract for pulling structured JSON data from web pages

- Search with full page content returned (2 credits per 10 results)

- Handles JavaScript rendering, CAPTCHAs, and dynamic content

Available Actions

These are the specific actions that AI agents can perform with this tool

Batch Scrape

18 inputs

This endpoint allows you to scrape multiple web pages in a single batch operation.

Inputs

urls

A list of URLs to scrape.

webhook

Specifies webhook URL and configurations for events related to the scraping process.

ignoreInvalidURLs

Whether to ignore invalid URLs and proceed with the batch using valid URLs only.

formats

Formats to include in the output such as markdown, html, etc.

onlyMainContent

Whether to return only the main content of the page.

includeTags

Tags to include in the output.

excludeTags

Tags to exclude from the output.

headers

Headers to send with the request, such as cookies or user-agent.

waitFor

Delay in milliseconds before fetching content, allowing page load.

mobile

Set true to emulate scraping from a mobile device.

skipTlsVerification

Whether to skip TLS certificate verification.

timeout

Request timeout in milliseconds.

jsonOptions

Options for JSON extraction, including schema and prompts.

actions

List of actions to perform on the page before capturing the content.

location

Location settings including country and language preferences.

removeBase64Images

Whether to remove base64 encoded images from the output.

blockAds

Whether to enable ad-blocking and cookie popup blocking.

proxy

Type of proxy to use such as basic or stealth.

Batch Scrape Multiple Webpages

22 inputs

Scrape multiple web pages in one request, with customizable options, and receive batch status and results as JSON.

Inputs

urls

The URLs to scrape.

webhook

Webhook callback configuration for updates; requires at least a URL. Can include headers, metadata, and event types (e.g. 'completed', 'page', 'started', 'failed').

ignoreInvalidURLs

If true, ignores invalid URLs and proceeds using only valid URLs.

formats

Output formats to include in scrape results.

onlyMainContent

If true, only the main content (not header/footer/sidebar) is returned.

includeTags

Tags to always include in the output.

excludeTags

Tags to exclude from the output.

maxAge

Max cache age allowed in ms; set to 0 for no cache.

headers

Custom HTTP headers to use for the scrape requests (e.g. cookies, user-agent).

waitFor

Delay before scraping (ms), gives time for page elements to load.

mobile

If true, scrapes as a mobile device.

skipTlsVerification

Skip SSL/TLS certificate validation.

timeout

Request timeout in ms.

parsePDF

Whether to parse PDFs as markdown (true) or return as base64 (false).

jsonOptions

Options for structured JSON extraction.

actions

Actions to perform on page before scraping, such as waiting for elements.

location

Location context for scraping, including proxy geography and browser language.

removeBase64Images

If true, all base64 images will be removed from the output.

blockAds

If true, applies ad/cookie-blocking to scrapes.

proxy

Proxy mode for anti-bot evasion.

changeTrackingOptions

Change tracking settings, only used if 'changeTracking' format is specified.

storeInCache

If results should be indexed/cached by Firecrawl.

Crawl URL

21 inputs

Initiates a crawl operation on a specified URL using various options.

Inputs

url

The base URL to start crawling from.

crawlerOptionsIncludes

URL patterns to include.

crawlerOptionsExcludes

URL patterns to exclude.

crawlerOptionsGenerateImgAltText

Generate alt text for images using LLMs (must have a paid plan). Default is false.

crawlerOptionsReturnOnlyUrls

If true, returns only the URLs as a list. Default is false.

crawlerOptionsMaxDepth

Maximum depth to crawl relative to the entered URL. Default is 123.

crawlerOptionsMode

The crawling mode to use. Options include 'default' and 'fast'.

crawlerOptionsIgnoreSitemap

Ignore the website sitemap when crawling. Default is false.

crawlerOptionsLimit

Maximum number of pages to crawl. Default is 10000.

crawlerOptionsAllowBackwardCrawling

Enable the crawler to navigate to previously linked pages. Default is false.

crawlerOptionsAllowExternalContentLinks

Allow the crawler to follow links to external websites. Default is false.

pageOptionsHeaders

Headers to send with the request, e.g., cookies, user-agent.

pageOptionsIncludeHtml

Include the HTML version of the page content. Default is false.

pageOptionsIncludeRawHtml

Include the raw HTML content of the page. Default is false.

pageOptionsOnlyIncludeTags

Only include specified tags, classes, IDs from the page.

pageOptionsOnlyMainContent

Only return the main content of the page. Default is false.

pageOptionsRemoveTags

Tags, classes, and IDs to remove from the page.

pageOptionsReplaceAllPathsWithAbsolutePaths

Replace all relative paths with absolute paths.

pageOptionsScreenshot

Include a screenshot of the top of the page. Default is false.

pageOptionsFullPageScreenshot

Include a full page screenshot. Default is false.

pageOptionsWaitFor

Wait for specified milliseconds for the page to load. Default is 0.

Extract Endpoint

4 inputs

Simplifies collecting structured data from URLs or domains.

Inputs

urls

The URLs from which to extract data. By using wildcards, you can target broader sets of pages.

prompt

Describes the data you want or how it's structured.

schema

Defines how the extracted data should be structured, used if no prompt is provided.

enableWebSearch

Allows extraction to follow links outside the specified domain for broader context.

Map Website And Return All Links

7 inputs

I want to crawl a website starting at a base URL, searching for specific content, and retrieve a list of all links found, with options to customize sitemap usage, include subdomains, set limits, and configure the request timeout.

Inputs

url

The base URL to start crawling from

Search query to use for mapping; limits may apply to smart search results

ignoreSitemap

Whether to ignore the website sitemap during crawling

sitemapOnly

If true, only return links found in the sitemap

includeSubdomains

Whether to include subdomains when searching for links

limit

Maximum number of links to return; must be less than or equal to 30000

timeout

Timeout for the request in milliseconds; if omitted, uses the default API timeout

Scrape Web Page

17 inputs

Extracts and returns specified content from a given webpage.

Inputs

url

The URL to scrape

formats

Formats to include in the output. Available options: 'markdown', 'html', 'rawHtml', 'links', 'screenshot', 'screenshot@fullPage', 'json'

onlyMainContent

Only return the main content of the page excluding headers, navs, footers, etc.

includeTags

Tags to include in the output.

excludeTags

Tags to exclude from the output.

headers

Headers to send with the request, such as cookies, user-agent, etc.

waitFor

Delay in milliseconds before fetching the content.

mobile

Set to true for emulating a mobile device.

skipTlsVerification

Skip TLS certificate verification.

timeout

Timeout in milliseconds for the request.

jsonOptions

Options for JSON extraction including schema, system prompt, and prompt.

extractOptions

Options for extract format. When using extract format, this specifies extraction schema, mode, and other settings for structured data extraction.

actions

Actions to perform on the page before fetching, such as wait, screenshot, click, etc.

location

Location settings for the request, including country and languages.

removeBase64Images

Remove base64 images from the output.

blockAds

Enable ad-blocking and cookie popup blocking.

proxy

Type of proxy to use, options are 'basic' or 'stealth'.

Scrape Web Page Content

20 inputs

Scrape the content of a web page using Firecrawl's API.

Inputs

url

The URL of the web page to scrape

formats

The formats to include in the output

onlyMainContent

Whether to return only the main content of the page

includeTags

Tags to include in the output

excludeTags

Tags to exclude from the output

maxAge

Accept a cached version if not older than this age (ms). 0 disables cache.

headers

Additional HTTP headers to send with the request

waitFor

Delay in ms before fetching the content

mobile

Emulate scraping from a mobile device

skipTlsVerification

Skip TLS certificate verification

timeout

Request timeout in milliseconds

parsePDF

Controls PDF processing during scrape

jsonOptions

Options for structured JSON extraction

actions

Actions to perform on the page before scraping

location

Location and language preference for the scrape

removeBase64Images

Removes all base64 images from the output

blockAds

Enable ad and cookie popup blocking

proxy

Configure proxy use for scraping

changeTrackingOptions

Options for change tracking (applies if 'changeTracking' is in formats)

storeInCache

Whether to store the page in Firecrawl index and cache

Web Search with Full Page Scraping

8 inputs

The endpoint combines web search results with Firecrawl's scraping capabilities to return full page content for any query.

Inputs

Query

The search query

Limit

Maximum number of results to return (1 <= x <= 10, default: 5)

Tbs

Time-based search parameter

Lang

Language code for search results (default: en)

Country

Country code for search results (default: us)

Location

Location parameter for search results

Timeout

Timeout in milliseconds (default: 60000)

ScrapeOptions

Options for scraping search results (e.g., formats: ['markdown'])