Web Scrape API | Unified Agent APIs

Overview

Your agent extracts content from any URL. We automatically escalate from lightweight HTTP fetch to JS rendering to stealth browser with CAPTCHA solving. The agent never needs to know which method was used. Content is returned as clean Markdown by default.

Modes

The mode parameter is a preset that configures optimal provider routing for common use cases. You can also set individual parameters directly for fine-grained control.

Fast

1 credit

Raw HTTP fetch, no JS rendering. Under 1s.

No JavaScript rendering or proxy. Works for static HTML pages, blogs, and docs. Will fail on SPAs and JS-heavy sites.

Standard

2 credits

Full JS rendering via Firecrawl. Works for 90%+ of sites.

Handles SPAs, dynamic content, and lazy loading. Response time: 2-5s. The recommended default for most use cases.

Thorough

5 credits

Stealth browser with proxy and CAPTCHA solving.

Firecrawl with Anchor stealth browser fallback, residential proxy, and CAPTCHA solving. For anti-bot protected sites, login-gated pages, and Cloudflare-defended sites. Response time: 10-30s.

Parameters

url

Required

string

The URL to scrape. Must be a valid HTTP/HTTPS URL. Redirects are followed automatically.

output

select (default: markdown)

Content format for the scraped page. Markdown is optimized for LLM consumption (67% fewer tokens than HTML).

Markdown

Clean Markdown, optimized for LLMs

HTML

Cleaned HTML without head/scripts

Raw HTML

Full HTML including head section

wait_for

string

CSS selector to wait for before extracting content. Useful for SPAs that load data dynamically. Only used in standard and thorough modes.

timeout

number (default: 30)

Maximum seconds to wait for the page. Fast mode default: 10s. Standard: 30s. Thorough: 60s.

Example Response

{
  "success": true,
  "data": {
    "content": "# Pricing\n\n## Starter Plan\n$29/mo\n- 1,000 API calls...",
    "title": "Pricing - Example Corp",
    "description": "Simple, transparent pricing for teams of all sizes.",
    "links": ["https://example.com/signup", "https://example.com/docs"],
    "screenshot_url": null
  },
  "metadata": {
    "provider_used": "firecrawl",
    "providers_tried": ["firecrawl"],
    "mode_used": "standard",
    "response_time_ms": 3100,
    "request_id": "req_def456"
  },
  "credits_used": 2
}

Get Started

Use this API through the O-mega platform. Create an API key in your dashboard, then call the endpoint with your key in the Authorization header.

Try Web Scrape

Test Web Scrape in the interactive playground. No setup required.

Open Playground