free-search-mcp

👁 License
👁 Python
👁 MCP

A local-first, no-API-key Model Context Protocol server that gives any LLM (Claude, GPT, local Ollama, …) the ability to search the web, fetch and clean up pages, and read documents — without you signing up for a single search API.

It bundles together the best ideas from a handful of open-source MCPs into one Python package, and adds the LLM-ergonomics and reliability work they were each missing.

research("how does reciprocal rank fusion work", depth=3)
 ↓
# Research brief: how does reciprocal rank fusion work
_engines: duckduckgo, mojeek, googlenews · sources: 3 · ~3,400 tokens_

## Sources
- [1] Reciprocal rank fusion | Elasticsearch Reference — <https://…>
- [2] Hybrid Search Scoring (RRF) | Microsoft Learn — <https://…>
- [3] RRF explained in 4 mins — Medium — <https://…>

## Documents
…full Markdown bodies of each page, ready for the LLM to read…

One tool call. Three sources. No API key. No OPENAI_API_KEY-but-for-search shakedown.

🚀 One-click deploy

One command — keyless engines work immediately, no signup, no key:

curl -LsSf https://raw.githubusercontent.com/sweetcornna/free-search-mcp/main/scripts/install.sh | bash -s -- --client claude-code

It clones or updates the project under ~/.local/share/free-search-mcp, installs uv, syncs dependencies, installs Chromium for rendered engines, smoke-tests the server, and registers the search MCP server in Claude Code user scope.

Other client targets:

curl -LsSf https://raw.githubusercontent.com/sweetcornna/free-search-mcp/main/scripts/install.sh | bash -s -- --client claude-desktop
curl -LsSf https://raw.githubusercontent.com/sweetcornna/free-search-mcp/main/scripts/install.sh | bash -s -- --client codex
curl -LsSf https://raw.githubusercontent.com/sweetcornna/free-search-mcp/main/scripts/install.sh | bash -s -- --client generic
curl -LsSf https://raw.githubusercontent.com/sweetcornna/free-search-mcp/main/scripts/install.sh | bash -s -- --client add-mcp
curl -LsSf https://raw.githubusercontent.com/sweetcornna/free-search-mcp/main/scripts/install.sh | bash -s -- --client all

Codex, Cursor, Cline, Continue, Zed, and generic agent guidance: docs/AGENT_USAGE.md.

Optional extras, any time (the defaults already work without them):

uv run search-mcp-admin # browser UI to add API keys / a proxy (127.0.0.1:8765)
uv run search-mcp-login zhihu # one-time Zhihu login (persists cookies)

Prefer containers? docker compose run --rm search-mcp. Claude Desktop and other clients: see Install below.

Related MCP server: TOOL4LM

Why this exists

Existing search MCPs each do one thing well, but you usually want all of it:

"LLM-tuned" here means: Markdown-first output, token estimates, smart truncation at paragraph boundaries, "Best for / Not for / Returns / Common mistakes" docstrings the model uses to pick the right tool, actionable error hints, MCP prompts and resource templates, and a one-shot research() that collapses search→fetch→fetch→fetch into a single turn.

"Trafilatura" means we extract main content using trafilatura — winner of the Bevendorff 2023 ROUGE benchmark (~0.85 vs ~0.55 for naive boilerplate stripping). Each fetched page also returns author, published_date, and sitename for free.

"Filters" means search/research accept freshness, include_domains, exclude_domains, category (news/pdf/github/paper/forum/blog), include_text, exclude_text.

Anti-detection & resilience

• HTTP fast path uses curl_cffi with a real Chrome 131 JA3/JA4 + HTTP/2 fingerprint, fixing the DDG "anomaly 202" rate-limit response that vanilla httpx triggered.

• Playwright fallback uses launch_persistent_context (cookies survive restarts on disk), prefers a real installed Chrome (channel="chrome"), drops the --no-sandbox fingerprint marker on macOS, and randomizes the viewport per session.

• Result dedup is title-fuzzy + host-canonical (rapidfuzz token_set_ratio >= 92, host normalized for www./m./amp. and country-TLD collapse), catching bbc.co.uk vs bbc.com duplicates that URL-only dedup misses.

• search includes an honest extractive lead_snippet — picks the top-3 result whose snippet contains ≥2 query terms and is ≥80 chars; rendered as > **Lead:** According to {host}: …. No LLM call. Returns nothing if no snippet qualifies (no fake answer).

⚠️ We deliberately do not attempt to defeat proof-of-work captchas on Bing or Brave — that crosses the ToS line. When those engines challenge us, we fall back to other engines instead.

Tools (9)

Plus 4 MCP prompts (Research thoroughly, Fact-check claim, Compare sources, News brief) and 2 resource templates (cache://page/{url}, cache://search/{query_hash}).

Filters (search / research)

All tools default to format="markdown" — readable, ~40% fewer tokens than JSON, with provenance and a token-budget header. Pass format="json" for structured access.

Tool annotations

Every tool ships correct readOnlyHint, idempotentHint, and openWorldHint annotations so MCP clients can label them and gate elevated actions.

Engines

Default set (all-HTTP, no browser, ~2x faster than the old default that included Startpage): duckduckgo, mojeek, googlenews.

Opt-in:

• startpage — browser-rendered (~5-10s/query); good for hard-to-reach results that the HTTP defaults miss.

• brave, bing, baidu — intermittent challenges to headless clients.

• searx — meta-search proxy via public SearXNG instances; included for completeness but most public instances are slow/unreliable in 2026.

• google — keyless Google web SERP scrape (HTTP first, Playwright fallback when Google serves a JS/consent shell). serpsearch is a pure alias of google (all dedicated "SERP APIs" require a key, so the only keyless SERP is a direct scrape).

• anysearch — AnySearch unified-search REST API, anonymous (keyless) tier; one HTTP call returns fused, re-ranked results. IP rate-limited; 429/5xx degrade to empty.

• bilibili — keyless Bilibili (哔哩哔哩) video search via the public web-interface/search/all/v2 JSON API (synthetic buvid3 cookie, no login). Returns video results only.

• zhihu — best-effort keyless Zhihu (知乎) search. Zhihu's api/v4/search_v3 needs login cookies + x-zse-96 signing, so the only no-key path is browser-rendering the public search page. Zhihu hard-gates headless clients, so a login wall / empty result is common and honest — treat it like baidu/brave.

All five added engines are keyless (no API key, no account) and stay opt-in — they're not in the fast default pool, so the ~2x latency win of the all-HTTP defaults is preserved. Enable per call with engines=["google","bilibili", ...], or globally via SEARCH_MCP_DEFAULT_ENGINES.

API-key engines & the admin backend

Keyless is the default, but you can also plug in keyed providers for higher reliability/quality. These engines stay dormant (and return a clear "not configured" hint) until you add a key:

Simplest setup — the admin UI:

uv run search-mcp-admin # opens a local config page on 127.0.0.1:8765

It serves one page (bound to localhost only) with, per provider: a how-to-get-a-key guide + signup link, a masked key field, Save (applies live — no server restart), a Test button, and Clear. Keys are written to ~/.config/search-mcp/config.json (0600); they're never echoed back to the page. Prefer env vars? Set SEARCH_MCP_<PROVIDER>_API_KEY instead (these override the saved file). Full walkthrough for each provider: docs/API_KEYS.md.

search("…", engines=["brave_api"]) # once a key is saved
search("…", engines=["tavily", "serper"]) # mix keyed + keyless freely

When an engine is gated (proxy · fallback · login)

Some engines get blocked by the provider — Google/Bing serve a CAPTCHA to datacenter IPs, Zhihu needs a login. We don't defeat CAPTCHAs (ToS); instead:

• Proxy (the real fix for IP gating): set a proxy in the admin UI "Network / Proxy" card or SEARCH_MCP_PROXY (http/https/socks5, optional user:pass@). It routes the HTTP engines, the browser, and fetch through a non-blocked IP. Scope it with SEARCH_MCP_PROXY_ENGINES="google bing zhihu".

• SearXNG auto-fallback: when google/serpsearch/bing are CAPTCHA-gated, they transparently retry via the working searx meta-search — you still get results, honestly attributed to searx.

• Gate diagnostics: the response includes gated_engines + gated_hint telling you which engine was gated (captcha/consent/login) and how it was handled.

• Zhihu login: run uv run search-mcp-login zhihu (or the admin "Login" button) once — a browser opens, you log in, cookies persist, and zhihu search then works. Requires a desktop session.

Full guide: docs/PROXY_AND_GATES.md.

Brave/Bing/Baidu all gate headless browsers after a handful of calls (PoW CAPTCHAs, "something went wrong" pages, redirect wrappers). Pass engines=["brave"] etc. only when the defaults can't find what you need.

Sparse-result diagnostics

When filters drop results so aggressively that ≤3 are returned, the response includes filter_diagnostics so the LLM knows which knob to relax. Example for category="forum" + exclude_text="beginner":

⚠️ **Filter diagnostics** (results were sparse)
Raw results: 20 across 3 engines → 0 after filters.
Top drops: category_forum (20).
Hint: Filters dropped 20 of 20 raw results. Most were excluded by
category=forum. Try widening or removing one filter.

Install

One-click setup

curl -LsSf https://raw.githubusercontent.com/sweetcornna/free-search-mcp/main/scripts/install.sh | bash -s -- --client claude-code

The remote installer clones or updates ~/.local/share/free-search-mcp, installs uv if needed, syncs dependencies, installs Chromium, smoke-tests the MCP server, then registers it with the requested client:

--client claude-code # Claude Code user-scope config
--client claude-desktop # claude_desktop_config.json
--client codex # Codex config
--client generic # print portable MCP JSON for other agents
--client add-mcp # delegate to npx add-mcp
--client both # Claude Code + Claude Desktop
--client all # Claude Code + Claude Desktop + Codex
--client none # install only, no client config changes

Prefer a local checkout?

git clone https://github.com/sweetcornna/free-search-mcp.git
cd free-search-mcp
./scripts/install.sh --client none

Prefer to do it by hand?

uv sync
uv run playwright install chromium
cp .env.example .env # optional: customize engines/limits

Run as a stand-alone server (stdio transport):

uv run search-mcp

Docker (one-click, containerized)

docker compose build
docker compose run --rm search-mcp # attaches stdio for MCP

Config & env vars

All settings are env vars prefixed with SEARCH_MCP_. Copy .env.example → .env and edit — it documents every knob, including how to enable the new engines via SEARCH_MCP_DEFAULT_ENGINES. See the full table under Configuration and the usage guide.

Tests

uv run pytest -q # offline (default, no network)
SEARCH_MCP_TEST_NETWORK=1 uv run pytest -v # live tests, hit the real web

Wire into Claude Code

This repo ships a project-scoped .mcp.json, so running claude inside the project auto-detects the search server. To register it globally instead:

claude mcp add search -s user -- uv --directory /absolute/path/to/free-search-mcp run search-mcp

Wire into Codex

Register the server with the Codex CLI:

codex mcp add search -- uv --directory /absolute/path/to/free-search-mcp run search-mcp
codex mcp list

Wire into Claude Desktop

Add this to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or the equivalent on your platform:

{
 "mcpServers": {
 "search": {
 "command": "uv",
 "args": ["--directory", "/absolute/path/to/free-search-mcp", "run", "search-mcp"]
 }
 }
}

Restart Claude Desktop. The nine tools above will appear in the tool drawer.

Wire into other clients

The server speaks plain MCP over stdio. Anything that supports MCP works:

• Codex (codex mcp add search -- uv --directory /…/free-search-mcp run search-mcp)

• Claude Code (claude mcp add search -s user -- uv --directory /…/free-search-mcp run search-mcp)

• Cursor / Continue / Cline (use the JSON snippet above)

• Custom Python / TypeScript clients via the official MCP SDK

For agent-specific operating rules, tool-selection guidance, and a reusable system-prompt snippet, see docs/AGENT_USAGE.md.

Installer choice

Generic MCP installers are useful, but they do not replace this script yet: add-mcp can write config for many clients, Smithery is strongest for registry or remote MCP connections, and MCPB is the right future format for clickable desktop bundles. This project still needs a Python/uv checkout, Playwright Chromium, and a smoke test, so scripts/install.sh handles the full bootstrap and then performs client registration.

Configuration

All settings can be overridden by environment variables prefixed with SEARCH_MCP_:

Architecture

 ┌─────────────────────────────────────────────────────┐
 │ FastMCP server (stdio) │
 │ tools: search / research / fetch / fetch_batch / │
 │ read_doc / cache_search / engines │
 └────────────┬────────────────────────────────────────┘
 │
 ┌────────────▼────────────┐ ┌────────────────────────┐
 │ aggregator │ │ fetcher │
 │ - parallel engines │ │ - httpx fast path │
 │ - reciprocal rank │ │ - playwright fallback │
 │ fusion │ │ - markdownify │
 │ - search cache (FTS5) │ │ - page cache (FTS5) │
 └────┬────────────────────┘ └────────────┬───────────┘
 │ │
 ┌────▼─────────────────┐ ┌──────────────▼─────────────┐
 │ engines/ │ │ browser pool │
 │ duckduckgo.py │ │ - persistent context │
 │ mojeek.py │ │ - stealth init script │
 │ searx.py │ │ - shared cookies │
 │ startpage.py (opt) │ │ - semaphore-bounded pages│
 │ brave.py (opt) │ └────────────────────────────┘
 │ bing.py (opt) │
 │ baidu.py (opt) │
 │ google.py (opt) │
 │ serpsearch.py(opt) │
 │ anysearch.py (opt) │
 │ bilibili.py (opt) │
 │ zhihu.py (opt) │
 └──────────────────────┘

 ┌────────────────────────────┐ ┌──────────────────┐
 │ documents/ │ │ ratelimit │
 │ pypdf, python-docx, │ │ token bucket │
 │ markdownify │ │ per engine │
 └────────────────────────────┘ └──────────────────┘

 ┌────────────────────────────┐ ┌──────────────────┐
 │ formatting │ │ research │
 │ token estimate │ │ composed │
 │ smart truncation │ │ workflow │
 │ markdown renderers │ │ │
 └────────────────────────────┘ └──────────────────┘

Engine adapter pattern

Each engine in src/search_mcp/engines/ implements:

class Engine:
 name: str
 needs_browser: bool # Force Playwright?
 wait_selector: str | None # CSS to wait for in browser mode

 def build_url(self, query: str, max_results: int) -> str: ...
 def parse(self, html: str) -> list[SearchResult]: ...

The base class handles transport (httpx → Playwright fallback), rate limiting, and the case where HTTP returns a captcha shell instead of results (auto-retries via the browser).

Credits

This project stands on the shoulders of:

• mrkrsl/web-search-mcp — smart httpx-then-Playwright fetch strategy, multi-engine fallback chain

• Aas-ee/open-webSearch — multi-engine breadth (Bing/DDG/Baidu/Brave/Startpage)

• VincentKaufmann/noapi-google-search-mcp — anti-detection patterns (navigator.webdriver, UA, cookies), SQLite FTS5 cache idea, multi-format read_document

• nickclyde/duckduckgo-mcp-server — per-engine rate limiting, LLM-friendly content cleanup

• Mojeek — independent search index that doesn't gate on User-Agent

• Model Context Protocol and the official Python SDK

License

MIT — see LICENSE.

Maintenance

Resources

• GitHub Repository
Need Help?Related Servers

Tools

• cache_searchA
• compareA
• enginesA
• extract_structuredA
• fetchA
• fetch_batchA
• read_docA
• researchA