Web Researcher MCP

@zoharbabin

Visit Server

23 days ago

# web-search

Your AI research assistant that cites real sources and stays honest. Search the entire web or narrow it down to just the sites you trust; medical journals, court databases, news outlets, academic papers. Analyze the full source, not just snippets. Links that work, citations you can trust, no made-up closed-garden pre-synthesized results.

Overview Tools Comments

Overview

web-researcher-mcp

A production-grade MCP server that gives AI assistants the power to search the web, extract content, and conduct multi-source research.

👁 CI
👁 Go Report Card
👁 Go Reference
👁 License: MIT
👁 Release
👁 Docker
👁 web-researcher-mcp MCP server
👁 GitHub Stars

Why Web Researcher MCP?

AI assistants are only as good as the information they can access. web-researcher-mcp bridges the gap between LLMs and the live internet through the Model Context Protocol standard:

Multiple specialized research tools in a single server (see Tools below)
Pluggable search backends with multi-provider routing and automatic fallback (Google, Brave, Serper, SearXNG, SearchAPI.io)
4-tier content extraction -- markdown negotiation, stealth HTTP, HTML parsing, headless browser (go-rod + stealth)
Search lenses for domain-focused research (programming, news, legal, medical, and more)
Single static binary with optional Chromium for JS rendering (auto-downloaded on first use)
Enterprise-ready with OAuth 2.1, multi-tenancy, rate limiting, and audit logging

Works with Claude Code, Claude Desktop, Cursor, and any MCP-compatible client.

Tools

Tool	Description
`web_search`	General web search with optional search lenses for domain-focused results
`scrape_page`	Extract content from any URL -- web pages, PDFs, DOCX, PPTX, YouTube transcripts (3-strategy fallback)
`search_and_scrape`	Combined search + extraction pipeline with quality scoring and deduplication
`image_search`	Search for images with size, type, color, and file format filters
`news_search`	Search news sources with freshness controls and source filtering
`academic_search`	Search peer-reviewed papers across arXiv, PubMed, IEEE, Nature, Springer, and other scholarly databases
`patent_search`	Search patent databases with CPC classification, strict office filtering (US/EP/WO/JP/CN/KR)
`sequential_search`	Multi-step research tracking with session state for iterative investigation

Quick Start

Option 1: Install with Go (Recommended)

go install github.com/zoharbabin/web-researcher-mcp/cmd/web-researcher-mcp@latest

The binary is now in your $GOPATH/bin. Add to Claude Code:

claude mcp add --scope user --transport stdio web-researcher -- web-researcher-mcp

Option 2: Download Binary

Download the latest release for your platform from Releases. Archives are named web-researcher-mcp_<version>_<os>_<arch>.tar.gz.

# Example: macOS Apple Silicon (replace VERSION with the latest from Releases)
curl -L https://github.com/zoharbabin/web-researcher-mcp/releases/download/v${VERSION}/web-researcher-mcp_${VERSION}_darwin_arm64.tar.gz | tar xz
chmod +x web-researcher-mcp

Option 3: Docker

docker run -e GOOGLE_CUSTOM_SEARCH_API_KEY=YOUR_KEY \
 -e GOOGLE_CUSTOM_SEARCH_ID=YOUR_CX \
 docker.io/zoharbabin/web-researcher-mcp:latest

Also available from GHCR: ghcr.io/zoharbabin/web-researcher-mcp:latest

Option 4: Build from Source

git clone https://github.com/zoharbabin/web-researcher-mcp.git
cd web-researcher-mcp
go build -o web-researcher-mcp ./cmd/web-researcher-mcp

Connect to Your AI Assistant

Add this to your MCP client configuration (example for Claude Code ~/.claude/settings.json):

{
 "mcpServers": {
 "web-researcher": {
 "command": "/path/to/web-researcher-mcp",
 "env": {
 "GOOGLE_CUSTOM_SEARCH_API_KEY": "YOUR_GOOGLE_API_KEY",
 "GOOGLE_CUSTOM_SEARCH_ID": "YOUR_SEARCH_ENGINE_ID"
 }
 }
 }
}

Or with Brave Search (no Google keys needed):

{
 "mcpServers": {
 "web-researcher": {
 "command": "/path/to/web-researcher-mcp",
 "env": {
 "SEARCH_ROUTING": "brave",
 "BRAVE_API_KEY": "YOUR_BRAVE_API_KEY"
 }
 }
 }
}

Done. Your AI assistant now has access to all research tools.

Configuration

Required (unless using `SEARCH_ROUTING`)

Variable	Description	How to Get
`GOOGLE_CUSTOM_SEARCH_API_KEY`	Google API key	Google Cloud Console
`GOOGLE_CUSTOM_SEARCH_ID`	Programmable Search Engine ID	PSE Console

Note: When SEARCH_ROUTING is set with non-Google providers (e.g., brave,serper), Google keys are not required. You only need credentials for the providers you configure.

Search Provider

Variable	Description	Default
`SEARCH_PROVIDER`	Backend: `google`, `brave`, `serper`, `searxng`, or `searchapi`	`google`
`BRAVE_API_KEY`	Brave Search API key
`SERPER_API_KEY`	Serper.dev API key
`SEARCHAPI_API_KEY`	SearchAPI.io API key
`SEARXNG_URL`	SearXNG instance URL
`SEARCH_ROUTING`	Multi-provider routing with automatic fallback (see Deployment docs)

HTTP Transport (Optional)

Variable	Description	Default
`PORT`	Enable HTTP/SSE mode	STDIO only
`OAUTH_ISSUER_URL`	JWT issuer URL for token validation
`OAUTH_AUDIENCE`	Expected JWT audience claim

Architecture

web-researcher-mcp/
├── cmd/web-researcher-mcp/ # Entry point (wiring only)
├── internal/
│ ├── config/ # Env-based strongly-typed configuration
│ ├── server/ # MCP server lifecycle + signal handling
│ ├── tools/ # Tool handlers (one file per tool)
│ ├── search/ # Pluggable search providers + router + lens routing
│ ├── scraper/ # 4-tier scraping pipeline (markdown → stealth → HTML → browser)
│ ├── documents/ # PDF, DOCX, PPTX parsing
│ ├── cache/ # Hybrid cache (memory + disk + optional Redis)
│ ├── auth/ # OAuth 2.1 middleware + JWKS
│ ├── audit/ # Structured audit logging
│ ├── session/ # Per-tenant session management
│ ├── content/ # Sanitize, dedup, truncate, quality score
│ ├── metrics/ # Prometheus metrics + per-tool stats
│ ├── ratelimit/ # Three-tier rate limiting
│ ├── circuit/ # Circuit breaker for external APIs
│ └── resources/ # MCP Resources + Prompts
├── lenses/ # Search lens JSON files
└── docs/ # Extended documentation

Design Principles

Zero global state -- all dependencies injected via constructors
Interface-driven -- every external dependency behind an interface for testing and swapping
Bounded concurrency -- explicit semaphores for external API calls
Defense in depth -- SSRF protection, rate limiting, content sanitization at every layer
Fail loud -- errors returned, never swallowed; validation at boundaries

Search Providers

All providers implement the same interface and support lenses via site: operator injection.

Provider	Whole-Web	Images	News	Notes
Google PSE	Yes	Yes	Yes	Default; free tier: 100 queries/day
Brave Search	Yes	Yes	Yes	Recommended for high-volume whole-web
Serper.dev	Yes	Yes	Yes	Google-identical results
SearXNG	Yes	Yes	Yes	Self-hosted, privacy-first, air-gapped deployments
SearchAPI.io	Yes	Yes	Yes	Unified API with multiple engine backends

Multi-Provider Routing

When SEARCH_ROUTING is set, the server uses multiple providers with automatic fallback:

# Priority-ordered list — requests go to the first healthy provider
export SEARCH_ROUTING=brave,google,serper
# Per-operation routing (JSON) — different priorities for different search types
export SEARCH_ROUTING='{"web":"brave,google","news":"brave,serper","images":"google,brave","default":"brave,google,searchapi"}'

Each provider gets an independent circuit breaker. If a provider fails (timeout, rate limit, 5xx), the next provider in the priority list is tried automatically. Lenses can override routing via the routing field in their JSON definition.

See docs/DEPLOYMENT.md for full routing documentation.

Single-Provider Routing

When SEARCH_ROUTING is not set, the server uses a single provider:

Request arrives
 |-- lens with dedicated CX? --> That specific Google PSE engine
 |-- lens (no CX)? --> Configured provider + site: operators
 |-- site: param set? --> Configured provider + site: operator
 `-- unrestricted? --> Configured SEARCH_PROVIDER

Search Lenses

Search lenses are curated domain lists that focus search results on high-quality sources for specific topics. They inject site: operators into queries and route through your configured search provider.

Built-in Lenses

Lens	Focus
`programming`	Code docs, tutorials, Q&A
`news`	Current events, journalism
`tech`	Technology industry
`legal`	Law, cases, statutes
`medical`	Health, medicine
`finance`	Markets, filings
`science`	Research, papers
`government`	Policy, regulations

Each lens is a JSON file in lenses/ containing the curated domain list. See Creating Custom Lenses below for the format.

Usage Example

{
 "tool": "web_search",
 "arguments": {
 "query": "golang context best practices",
 "lens": "programming"
 }
}

This searches only stackoverflow.com, github.com, go.dev, developer.mozilla.org, and other curated programming sites.

Security

For the full threat model and security architecture, see docs/SECURITY.md.

MCP Client Integration

Claude Code

Add to ~/.claude/settings.json:

{
 "mcpServers": {
 "web-researcher": {
 "command": "/path/to/web-researcher-mcp",
 "env": {
 "GOOGLE_CUSTOM_SEARCH_API_KEY": "AIza...",
 "GOOGLE_CUSTOM_SEARCH_ID": "017...",
 "SEARCH_PROVIDER": "brave",
 "BRAVE_API_KEY": "BSA..."
 }
 }
 }
}

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
 "mcpServers": {
 "web-researcher": {
 "command": "/path/to/web-researcher-mcp",
 "env": {
 "GOOGLE_CUSTOM_SEARCH_API_KEY": "AIza...",
 "GOOGLE_CUSTOM_SEARCH_ID": "017..."
 }
 }
 }
}

Cursor

Add to .cursor/mcp.json in your project root:

{
 "mcpServers": {
 "web-researcher": {
 "command": "/path/to/web-researcher-mcp",
 "env": {
 "GOOGLE_CUSTOM_SEARCH_API_KEY": "AIza...",
 "GOOGLE_CUSTOM_SEARCH_ID": "017..."
 }
 }
 }
}

HTTP/SSE Mode (Multi-Client, Teams)

For shared deployments serving multiple clients or web applications:

PORT=3000 \
OAUTH_ISSUER_URL=https://auth.example.com \
OAUTH_AUDIENCE=https://api.example.com \
./web-researcher-mcp

Connect any MCP client to http://localhost:3000/mcp/ (Streamable HTTP transport).

Performance

Search results are cached in-memory for sub-millisecond hits. The scraping pipeline tries the fastest tier first and falls back progressively — most pages resolve in under a second via stealth HTTP, with the headless browser reserved for JS-heavy sites. See ARCHITECTURE.md for detailed latency breakdowns.

Development

go build -o web-researcher-mcp ./cmd/web-researcher-mcp # Build
go test -race ./... # Test (with race detector)
golangci-lint run # Lint
govulncheck ./... # Security audit

See CONTRIBUTING.md for the full development workflow, code style guide, and PR process.

Troubleshooting

Contributing

Contributions are welcome. Please see CONTRIBUTING.md for code style guidelines, development workflow, and how to submit pull requests.

Documentation

Document	Description
ARCHITECTURE.md	Design decisions, technology stack, dependencies
CONTRIBUTING.md	Development setup, code style, PR workflow
docs/TOOLS.md	Tool specifications and parameter schemas
docs/EXAMPLES.md	Usage examples with JSON tool calls
docs/API_SETUP.md	Search provider API key setup for all providers
docs/SECURITY.md	Threat model, SSRF, auth, compliance (SOC2/GDPR/FedRAMP)
docs/DEPLOYMENT.md	Build, Docker, Kubernetes, client configs, scaling
docs/LESSONS_LEARNED.md	Node.js to Go migration story and lessons
docs/MIGRATION.md	Migrating from the deprecated google-researcher-mcp

License

MIT

Star History

👁 Star History Chart

Built with Go and the Model Context Protocol

If this project helps your workflow, consider giving it a ⭐

Try in Playground

Server Config

{
 "mcpServers": {
 "web-researcher": {
 "command": "web-researcher-mcp",
 "args": [],
 "env": {
 "GOOGLE_CUSTOM_SEARCH_API_KEY": "<YOUR_KEY>",
 "GOOGLE_CUSTOM_SEARCH_ID": "<YOUR_CX>",
 "SEARCH_PROVIDER": "google"
 }
 }
 }
}

Build with ShipAny.

URL: https://mcp.so/server/web-researcher-mcp/zoharbabin