The webclaw server lets you extract, crawl, and analyze web content for AI agents, RAG pipelines, and developer workflows. Here's what you can do:

• Scrape – Extract content from a single URL as markdown, LLM-optimized text, plain text, HTML, or JSON; supports CSS selector filtering, main-content extraction, and auto-fallback to cloud API when bot protection or JS rendering is detected.

• Crawl – Breadth-first crawl of a website from a seed URL, following links up to a configurable depth and page limit, with optional sitemap seeding and concurrent requests.

• Batch – Scrape multiple URLs concurrently and return extracted content for all of them at once.

• Map – Discover all URLs from a website's sitemaps (via robots.txt + sitemap.xml) without fully extracting every page.

• Extract – Extract structured data from a web page using an LLM, guided by a JSON schema or a natural language prompt.

• Summarize – Generate a concise LLM-powered summary of a web page, with configurable sentence count.

• Diff – Compare a URL's current content against a previously saved extraction snapshot to highlight what has changed.

• Brand – Extract brand identity assets (colors, fonts, logo, favicon) from a website's HTML and CSS.

• Search – Search the web for a query and return structured results (requires API key).

• Research – Run a deep, multi-source research investigation on a topic or question, with an optional deep mode for more thorough results (requires API key).

Integrates with local Ollama instances for private, AI-powered structured data extraction and content summarization.

Utilizes OpenAI's API to perform schema-enforced extraction and content summarization on scraped data.

Provides specialized extraction of structured metadata and content details from YouTube video pages.

webclaw https://example.com --format markdown

# Example Domain

This domain is for use in illustrative examples in documents.

You may use this domain in literature without prior coordination or asking for permission.

npx create-webclaw

brew tap 0xMassi/webclaw
brew install webclaw

docker run --rm ghcr.io/0xmassi/webclaw https://example.com

cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli
cargo install --git https://github.com/0xMassi/webclaw.git webclaw-mcp

webclaw https://stripe.com --format markdown

webclaw https://docs.anthropic.com --format llm

webclaw https://example.com/blog/post --only-main-content

webclaw https://example.com \
 --include "article, main, .content" \
 --exclude "nav, footer, .sidebar, .ad"

webclaw https://docs.rust-lang.org --crawl --depth 2 --max-pages 50

webclaw https://github.com --brand

webclaw https://example.com/pricing --format json > pricing-old.json
webclaw https://example.com/pricing --diff-with pricing-old.json

npx create-webclaw

{
 "mcpServers": {
 "webclaw": {
 "command": "~/.webclaw/webclaw-mcp"
 }
 }
}

Scrape these competitor pricing pages and summarize the differences.

Crawl this documentation site and prepare clean context for a RAG index.

Extract the brand colors, fonts, and logos from this company website.

npm install @webclaw/sdk
pip install webclaw
go get github.com/0xMassi/webclaw-go

import { Webclaw } from "@webclaw/sdk";

const client = new Webclaw({ apiKey: process.env.WEBCLAW_API_KEY! });

const page = await client.scrape({
 url: "https://example.com",
 formats: ["markdown"],
 only_main_content: true,
});

console.log(page.markdown);

from webclaw import Webclaw

client = Webclaw(api_key="wc_your_key")

page = client.scrape(
 "https://example.com",
 formats=["markdown"],
 only_main_content=True,
)

print(page.markdown)

curl -X POST https://api.webclaw.io/v1/scrape \
 -H "Authorization: Bearer $WEBCLAW_API_KEY" \
 -H "Content-Type: application/json" \
 -d '{
 "url": "https://example.com",
 "formats": ["markdown"],
 "only_main_content": true
 }'

export WEBCLAW_API_KEY=wc_your_key

webclaw https://example.com --cloud

webclaw/
 crates/
 webclaw-core HTML to markdown, text, JSON, and LLM-ready output
 webclaw-fetch Fetching, crawling, batching, and mapping
 webclaw-llm Local and hosted LLM provider support
 webclaw-pdf PDF text extraction
 webclaw-mcp MCP server for AI agents
 webclaw-cli Command-line interface

URL:
Command or API request:
Expected output:
Actual output:
Format used: markdown / llm / text / json / html
CLI, MCP, SDK, or API: