VOOZH about

URL: https://dev.to/simon_crawlforge_dev/how-to-give-chatgpt-web-scraping-with-mcp-connectors-2026-4ba6

⇱ How to Give ChatGPT Web Scraping with MCP Connectors (2026) - DEV Community


ChatGPT can now call your own tools through custom MCP connectors — including web scraping. But there is a catch the marketing pages skip: connectors must be remote servers, so a local tool like CrawlForge cannot be pasted in directly. This is the honest version: what is actually possible, why a wrapper is needed, and the exact bridge to build.


TL;DR: ChatGPT custom MCP connectors (renamed "apps" in Dec 2025) work on Plus, Pro, Business, Enterprise, and Edu via Developer mode — not Free/Go. Connectors must be remote HTTPS servers, so a local stdio server like CrawlForge can't be added directly. The fix: a ~30-line remote MCP wrapper that proxies CrawlForge's REST API.

Table of Contents

What ChatGPT Connectors Are

ChatGPT supports custom MCP connectors — renamed "apps" in December 2025, so the UI now says "Apps & Connectors." Through Developer mode, you connect an external MCP server and ChatGPT calls its tools mid-conversation, asking you to confirm before any write action. Same Model Context Protocol that powers web scraping in Claude — different client. Developer mode is explicitly a beta.

Which Plans Can Use Them

Per OpenAI's plan table, adding a custom MCP connector is available on Plus, Pro, Business, Enterprise, and Edu — not Free or Go. Full write-action support is rolling out most broadly to Business, Enterprise, and Edu. If you only need ChatGPT to read scraped data, the read-only path below is enough.

The Transport Catch: Remote Only

This trips people up. A ChatGPT connector must be a remote MCP server reachable over HTTPS (SSE or Streamable HTTP transport). You paste a URL; you do not point it at a command on your machine. That rules out local stdio servers — the kind you install with npx. To use one, host it publicly or tunnel a local server via ngrok or Cloudflare Tunnel.

There is also a naming rule: ChatGPT's deep research / company-knowledge paths require two read-only tools named search and fetch with a specific schema. Full Developer mode allows arbitrary tools, so that constraint applies only to the deep-research path.

Why CrawlForge Needs a Wrapper

CrawlForge ships as a local stdio MCP server (via npx) plus a REST API at https://www.crawlforge.dev/api/v1/tools/. Neither is a remote MCP URL, and its tools are named search_web, fetch_url, and extract_content — not the search/fetch pair deep research expects. So you cannot paste CrawlForge straight into ChatGPT today. The practical path is a thin remote MCP wrapper — about 30 lines.

Build the Bridge

FastMCP (Python) is the quickest way to stand up a remote MCP server exposing the search + fetch tools ChatGPT wants. Each calls CrawlForge's REST API with your cf_live_ key in the X-API-Key header:

Run it and expose it over HTTPS. For a quick test, tunnel your local port:

pip install fastmcp httpx
export CRAWLFORGE_API_KEY="cf_live_your_key_here"
python server.py
# in another terminal:
ngrok http 8000

For Developer mode you can skip the search/fetch naming and map tools one-to-one to CrawlForge — expose scrape_structured, stealth_mode, or deep_research directly. Same pattern.

Add the Connector in ChatGPT

  1. Settings → Apps & Connectors → Advanced → enable Developer mode.
  2. Apps & Connectors → Create.
  3. Paste your public HTTPS MCP URL (e.g. your ngrok URL plus /mcp), name it, choose an auth method.
  4. Confirm the "I trust this application" checkbox.

Your search and fetch tools appear. In a chat, select the connector and ask ChatGPT to research a topic — it calls search, then fetches the best results through CrawlForge.

Auth and Safety

Connectors authenticate with none (public) or OAuth — there is no API-key-header option in the UI, which is why the wrapper holds your CrawlForge key server-side. ChatGPT confirms before write actions, and you can inspect each call before approving.

Take OpenAI's warnings seriously: only connect servers you trust. Custom connectors increase risk, including prompt injection, and a model mistake on a write action could destroy or leak data. A read-only scraping bridge is low-risk; lock it down with OAuth before sharing.

A Simpler Alternative

If you would rather not host anything, use CrawlForge from code with the OpenAI Agents SDK or Responses API — no remote server required. See CrawlForge with the OpenAI Agents SDK.

Get a Free CrawlForge Key — 1,000 Credits