Website Content Crawler — AI & RAG Ready

Pricing

Rating

Developer

Actor stats

Categories

Share

Extract clean Markdown and plain text from any website, optimized for AI ingestion, RAG pipelines, and LLM context windows. Readability-style main content extraction strips navigation, footers, sidebars, and ads so your AI gets only the content that matters.

Why This Actor?

• AI-optimized output — Markdown + plain text per page, with content type detection
• Main content extraction — Readability-style selectors remove noise (nav, footer, ads, sidebars)
• Flexible crawl modes — Fetch a list of URLs directly (depth=0) or crawl entire sites (depth=1-5)
• Concurrent processing — Up to 20 parallel workers for high-throughput extraction
• Pay-per-page pricing — Only pay for pages successfully crawled

Use Cases

• Build RAG knowledge bases from company documentation sites
• Feed LLMs with up-to-date content from blog posts and news articles
• Extract article text for AI summarization pipelines
• Crawl competitor sites for content analysis
• Bulk-convert web pages to Markdown for offline use

Input

Output

One record per crawled page:

Example Input

Fetch a list of documentation pages (no crawling):

{
"startUrls":[
"https://docs.example.com/api/overview",
"https://docs.example.com/api/authentication"
],
"maxDepth":0,
"extractMainContent":true
}

Crawl an entire blog up to 2 levels deep:

{
"startUrls":["https://blog.example.com"],
"maxDepth":2,
"maxPagesPerCrawl":200,
"maxConcurrency":10,
"extractMainContent":true
}

Pricing

MCP Integration

Use this actor directly from Claude or any MCP-compatible AI tool:

Use apify/santamaria-automations/website-content-crawler to crawl https://docs.example.com with maxDepth=1 and extractMainContent=true, then summarize the documentation

Actor URL: apify/santamaria-automations/website-content-crawler

Notes

• Challenge pages (Cloudflare, DataDome, PerimeterX) are detected and skipped automatically
• Deduplication prevents the same URL from being crawled twice in the same run
• Content type detection identifies articles, blog posts, and documentation pages
• Main content extraction uses CSS selector priority: article-specific classes → semantic tags → body fallback