Crawl4AI MCP Wrapper

Custom Model Context Protocol (MCP) server that wraps the Crawl4AI Docker API with reliable stdio transport for Claude Code integration.

Overview

This MCP wrapper provides 7 tools for web scraping, crawling, and content extraction:

1. scrape_markdown - Extract clean markdown from webpages

2. extract_html - Get preprocessed HTML structures

3. capture_screenshot - Take full-page PNG screenshots

4. generate_pdf - Create PDF documents from webpages

5. execute_javascript - Run JavaScript in browser context

6. crawl_urls - Crawl multiple URLs with configuration options

7. ask_crawl4ai - Query Crawl4AI documentation and examples

Related MCP server: Enhanced Fetch MCP

Prerequisites

• Python 3.8 or higher

• Crawl4AI Docker container running on port 11235

• Claude Code installed

Installation

1. Start Crawl4AI Docker Container

docker run -d -p 11235:11235 --name crawl4ai --shm-size=2g unclecode/crawl4ai:latest

Verify it's running:

curl http://localhost:11235/health

2. Set Up Virtual Environment

cd /Volumes/4TB/Users/josephmcmyne/myProjects/mcp/crawl4ai-wrapper
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

3. Test the Wrapper

python test_wrapper.py

You should see all tests pass:

✅ All tests passed! MCP wrapper is ready to use.

Configuration

For Claude Code (Project-Level)

Add to /Volumes/4TB/Users/josephmcmyne/general/.mcp.json:

{
 "mcpServers": {
 "iterm-mcp": {
 "args": ["-y", "iterm-mcp"],
 "command": "npx"
 },
 "crawl4ai-custom": {
 "command": "/Volumes/4TB/Users/josephmcmyne/myProjects/mcp/crawl4ai-wrapper/venv/bin/python",
 "args": ["/Volumes/4TB/Users/josephmcmyne/myProjects/mcp/crawl4ai-wrapper/crawl4ai_mcp.py"],
 "env": {
 "CRAWL4AI_BASE_URL": "http://localhost:11235"
 }
 }
 }
}

For Claude Code (Global)

Add to ~/.claude/claude_desktop_config.json (if this file exists):

{
 "mcpServers": {
 "crawl4ai-custom": {
 "command": "/Volumes/4TB/Users/josephmcmyne/myProjects/mcp/crawl4ai-wrapper/venv/bin/python",
 "args": ["/Volumes/4TB/Users/josephmcmyne/myProjects/mcp/crawl4ai-wrapper/crawl4ai_mcp.py"],
 "env": {
 "CRAWL4AI_BASE_URL": "http://localhost:11235"
 }
 }
 }
}

Usage

Restart Claude Code

After configuration, restart Claude Code to load the new MCP server.

Verify Connection

claude mcp list

You should see:

crawl4ai-custom: ... - ✓ Connected

Use in Claude Code

Example prompts:

Scrape the content from https://example.com and summarize it.

Take a screenshot of https://github.com

Crawl these URLs and extract their main content: https://example.com, https://example.org

Tool Reference

scrape_markdown

Extract clean markdown from a webpage.

Parameters:

• url (required): URL to scrape

• filter_type (optional): 'fit', 'raw', 'bm25', or 'llm' (default: 'fit')

• query (optional): Query string for BM25/LLM filters

• cache_bust (optional): Cache-bust counter (default: '0')

Example:

result = await scrape_markdown("https://example.com")

extract_html

Get preprocessed HTML from a webpage.

Parameters:

• url (required): URL to extract HTML from

capture_screenshot

Capture a full-page PNG screenshot.

Parameters:

• url (required): URL to screenshot

• screenshot_wait_for (optional): Seconds to wait before capture (default: 2.0)

• output_path (optional): Path to save the screenshot file

generate_pdf

Generate a PDF document from a webpage.

Parameters:

• url (required): URL to convert to PDF

• output_path (optional): Path to save the PDF file

execute_javascript

Run JavaScript code on a webpage.

Parameters:

• url (required): URL to execute scripts on

• scripts (required): List of JavaScript snippets to execute

crawl_urls

Crawl multiple URLs with configuration.

Parameters:

• urls (required): List of 1-100 URLs to crawl

• browser_config (optional): Browser configuration overrides

• crawler_config (optional): Crawler configuration overrides

• hooks (optional): Custom hook functions

ask_crawl4ai

Query Crawl4AI documentation.

Parameters:

• query (required): Search query

• context_type (optional): 'code', 'doc', or 'all' (default: 'all')

• score_ratio (optional): Minimum score threshold (default: 0.5)

• max_results (optional): Maximum results (default: 20)

Troubleshooting

Container Not Running

If tests fail with connection errors:

docker ps | grep crawl4ai

If not running, start it:

docker start crawl4ai

MCP Server Not Connecting

Check Claude Code logs:

tail -f ~/Library/Logs/Claude/mcp*.log

Verify the Python path in your .mcp.json is correct:

which python
# Should match the venv path in your config

Port Conflicts

If port 11235 is in use:

1. Stop the Crawl4AI container: docker stop crawl4ai

2. Find the conflicting process: lsof -i :11235

3. Either stop that process or change the port in both Docker and this wrapper

Development

Adding New Tools

To add a new tool:

1. Add an async function decorated with @mcp.tool() in crawl4ai_mcp.py

2. Make an HTTP request to the appropriate Crawl4AI endpoint

3. Add error handling

4. Add a test in test_wrapper.py

5. Update this README

Running in Debug Mode

# Enable debug logging
export CRAWL4AI_MCP_LOG=DEBUG

# Run the server
python crawl4ai_mcp.py

Architecture

┌─────────────────┐
│ Claude Code │
└────────┬────────┘
 │ stdio (MCP protocol)
 ▼
┌─────────────────┐
│ FastMCP Server │ (this wrapper)
│ crawl4ai_mcp.py│
└────────┬────────┘
 │ HTTP/REST API
 ▼
┌─────────────────┐
│ Crawl4AI Docker │
│ Container │
│ (port 11235) │
└─────────────────┘

Benefits Over Official SSE Server

• Reliable stdio transport: No SSE connection issues

• Full control: Easy to extend and customize

• Better error handling: Graceful degradation

• Simple debugging: Standard Python stack traces

• No protocol mismatches: Direct HTTP to Crawl4AI API

License

MIT License - feel free to modify and distribute.

Credits

• Built with FastMCP

• Wraps Crawl4AI

• For use with Claude Code

Maintenance

Resources

• GitHub Repository
Need Help?Related Servers