Touchpoint's MCP server gives LLM agents full read and control access to desktop UI elements across Linux, macOS, and Windows via native accessibility APIs (AT-SPI2, UIA, AX) and Chrome DevTools Protocol for browser/Electron web content.

Discovery & Inspection

• apps — List all applications visible in the accessibility tree

• windows — List all open windows with IDs, titles, sizes, and app names

• elements — Browse the accessibility tree with filtering by role, state, depth, and sort order

• find — Search for UI elements by name using 4-stage matching (exact → contains → word → fuzzy)

• get_element — Fetch a detailed snapshot of a single element by ID, including supported actions

Screenshots

• screenshot — Capture the full desktop, a specific app, window, element bounding box, or monitor

UI Actions

• click — Click an element by ID or screen coordinates (left, right, or double-click)

• set_value — Set text content of an editable field

• set_numeric_value — Set the value of a slider or spinbox

• focus — Move keyboard focus to an element

• action — Execute any raw accessibility action by name (e.g. "expand or collapse")

• activate_window — Bring a window to the foreground

Keyboard & Mouse Input

• type_text — Type text into the focused element (supports \n, \t, \b)

• press_key — Press a single key or combination (e.g. ["ctrl", "s"]), with repeat support

• mouse_move — Move the cursor to an element or screen coordinates

• scroll — Scroll up, down, left, or right at the cursor position

Waiting / Polling

• wait_for — Wait for UI elements to appear or disappear, with multi-query and any/all modes

• wait_for_app — Wait for an application to appear or disappear

• wait_for_window — Wait for a window (matched by title substring) to appear or disappear

Output is returned in flat, tree, or JSON formats with persistent element IDs for reliable cross-action references.

Enables interaction with Discord's desktop interface by reading its structured accessibility data and Electron-based web content.

Provides deep integration with Electron-based applications, allowing AI agents to read and interact with both native and web-based UI elements via CDP.

Enables discovery and interaction with Firefox UI elements, allowing agents to automate browser tasks like filling fields and clicking buttons.

Supports automated interaction with Google Chrome through native accessibility APIs and Chrome DevTools Protocol (CDP) for structured access to web content.

Allows programmatic control over the Slack desktop application by reading its accessibility tree to locate and interact with UI components like buttons and message fields.

import touchpoint as tp

elements = tp.find("Send", role=tp.Role.BUTTON, app="Slack")
tp.click(elements[0])

pip install touchpoint-py

import touchpoint as tp

# Discover
apps = tp.apps() # ["Firefox", "Slack", "Terminal", ...]
windows = tp.windows() # Window objects with title, position, size
all_els = tp.elements(app="Firefox", named_only=True) # only elements with text labels

# Find
results = tp.find("Search", role=tp.Role.TEXT_FIELD, app="Firefox")

# Act
tp.set_value(results[0], "touchpoint python", replace=True)
tp.press_key("enter")
tp.hotkey("ctrl", "s") # keyboard shortcuts

# Wait for UI changes
tp.wait_for("results", app="Firefox", timeout=10)

# Screenshot
img = tp.screenshot() # full desktop → PIL.Image
img = tp.screenshot(app="Firefox") # cropped to app window

results = tp.find("Send", max_results=1)
element_id = results[0].id # "atspi:1234:1:5.2"

# later...
tp.click(element_id) # works with just the string

tp.elements(app="Slack", format="flat") # one compact line per element (best for LLMs)
tp.elements(app="Slack", format="tree") # indented parent/child hierarchy
tp.elements(app="Slack", format="json") # full JSON with all fields

 ┌──────────┐
 ┌───▶│ ORIENT │ screenshot · apps · windows
 │ └────┬─────┘
 │ ▼
 │ ┌──────────┐
 │ │ LOCATE │ find · snapshot · get_element
 │ └────┬─────┘
 │ ▼
 │ ┌──────────┐
 │ │ ACT │ click · set_value · type_text · press_key
 │ └────┬─────┘
 │ ▼
 │ ┌──────────┐
 │ │ VERIFY │───▶ Done ✅
 │ └────┬─────┘
 │ │ not yet
 └─────────┘

{
 "mcpServers": {
 "touchpoint": {
 "command": "touchpoint-mcp"
 }
 }
}

{
 "servers": {
 "touchpoint": {
 "command": "touchpoint-mcp"
 }
 }
}

{
 "mcpServers": {
 "touchpoint": {
 "command": "touchpoint-mcp"
 }
 }
}

{
 "mcpServers": {
 "touchpoint": {
 "command": "touchpoint-mcp"
 }
 }
}

claude mcp add touchpoint -- touchpoint-mcp

{
 "mcpServers": {
 "touchpoint": {
 "command": "touchpoint-mcp"
 }
 }
}

# Linux
google-chrome --remote-debugging-port=9222 --user-data-dir=/tmp/tp-chrome

# macOS
open -na "Google Chrome" --args --remote-debugging-port=9222 --user-data-dir=/tmp/tp-chrome

# Windows
start chrome --remote-debugging-port=9222 --user-data-dir=%TEMP%\tp-chrome

import touchpoint as tp

tp.configure(cdp_discover=True) # auto-discover from running processes
# or
tp.configure(cdp_ports={"Google Chrome": 9222}) # explicit mapping

tp.elements(app="Google Chrome", source="full") # native chrome + web content (default)
tp.elements(app="Google Chrome", source="cdp_ax") # web content only (CDP accessibility tree)
tp.elements(app="Google Chrome", source="native") # native UI only (toolbar, tabs, menus)
tp.elements(app="Google Chrome", source="dom") # DOM walker (catches what AX misses)

┌───────────────────────────────────────────────────────┐
│ import touchpoint as tp │
│ tp.find() · tp.click() · tp.screenshot() · ... │
│ (Public API) │
├─────────────────────────┬─────────────────────────────┤
│ Backend (ABC) │ InputProvider (ABC) │
├─────────────────────────┼─────────────────────────────┤
│ AT-SPI2 (Linux) │ Xdotool (X11) │
│ UIA (Windows) │ SendInput (Win32) │
│ AX (macOS) │ CGEvent (macOS) │
│ CDP (browsers) │ │
├─────────────────────────┴─────────────────────────────┤
│ Utilities: formatter · matcher · screenshot · scale │
└───────────────────────────────────────────────────────┘

tp.configure(
 fuzzy_threshold=0.6, # minimum match score for find() (0.0–1.0)
 fallback_input=True, # use InputProvider when native actions fail
 type_chunk_size=40, # split long text into chunks for typing (0 = disable)
 max_elements=5000, # max elements per query
 max_depth=20, # default tree depth limit
 scale_factor=None, # display scale override (None = auto-detect)
 cdp_ports={"Chrome": 9222}, # explicit CDP port mapping
 cdp_discover=True, # auto-discover CDP ports from running processes
 cdp_refresh_interval=5.0, # seconds between CDP target scans
 ax_messaging_timeout=1.0, # max seconds to wait for a macOS AX app reply
)

git clone https://github.com/Touchpoint-Labs/touchpoint.git
cd touchpoint
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest