Touchscreens are typically found everywhere, aside from one place... your PC. It makes sense; it's designed to be operated with a mouse and keyboard, and sure, there are computers out there with touchscreens built in, and there are add-ons you can get to bring that compatibility over, but by and large, they're rather rare here. In an effort to add touchscreen support to my PC because, I don't know, vibes I guess, I turned an ESP32 into a tool to control my PC with.

I'm using the WT32-SC01 Plus, which is an ESP32-S3 development board with a 480x320 IPS display tacked on, while also packing 16MB of flash storage and 2MB of PSRAM. It's compact, relatively inexpensive (with even cheaper options out there, like the Cheap Yellow Display), and perfect for experimenting with touch inputs. Over a Wi-Fi connection, it can fetch what's on my PC's screen, display it, and forward touch inputs back.

To be frank, it's a bit of a small screen for this use case. However, there are plenty of applications that could work when full-screened, or you could simply take this concept and apply it to another, larger screen instead.

Building the connection between PC and ESP32

A simple server for two-way communication

The ESP32-S3 has built-in Wi-Fi, so it's perfect for connecting to my PC wirelessly. But what facilitates the connection? I modified an existing Python server and run it on my computer, connecting to the ESP32 and sending individual frames for rendering. The code works, but the device that developer used was too different from mine to port it over, so I re-wrote the code for the WT32-SC01 Plus instead.

Thankfully, there are many examples for the WT32-SC01 Plus that helped get me started. One particular project which features a clock, weather station, and online radio receiver saved me here, as previously, touch inputs failed to work correctly. Looking at the source code, I was able to tweak mine in order to fix it and get them working, too.

As for how it all works, I broke it down to a fairly simple two-way communication. On the PC side, the Python script continuously captures screenshots using mss, compresses them into JPEG images, and sends them over WebSocket. The ESP32 receives these chunks, stitches them back together, and renders the frame on its LCD panel.

Touch input works the opposite way: when you tap or swipe on the ESP32 screen, the coordinates are sent back to the Python server in a message form. The script reacts to those incoming messages, emulating mouse events using pynput and scaling them to fit your current resolution. It's a purely complementary addition that could prove useful in certain use cases, particularly so when paired with either a larger screen or an application showing larger buttons.

What can you actually use it for?

There are a few immediate use cases

Now, is this the most practical way to use a PC? As much as I wish it was the case, the answer is a resounding "absolutely not." With that said, imagine putting Spotify, a home automation dashboard, or OBS controls on it. You tap to pause your music, switch your lights, or swap scenes in a stream, all without reaching for your mouse.

This works because the ESP32 is just forwarding touch inputs, the computer interprets them as if you're clicking or dragging in those applications. The downside is that it moves your mouse pointer there, but the Python server could be easily modified to check what the previous location of the mouse was and then jump you back there after you tap.

On top of that, it can be a simple remote desktop companion. You can have an ESP32-display somewhere nowhere near your desk, and still be able to look at your screen and see if you received a notification or control your music. It's not going to blow you away in terms of quality, but it certainly gets the job done.

There's also a lot of room for improvement. Increasing the JPEG quality improves sharpness but uses a lot of bandwidth and requires more processing power. Lowering it makes it possible to send more frames, but at the cost of readability. With a more efficient codec or sending method, performance could actually improve significantly. It's even little things that can help here; the "switch" statement that I used will typically compile to a jump table, where the branch predictor can evaluate based on one conditional branch. In contrast, a chain of "if-else" conditionals will each require their own conditional branch, resulting in prediction misses. It's not a big deal when programming on your computer, but these little inefficiencies can add up when programming on a tiny, resource-constrained device like this.

Either way, this project isn't about replacing your mouse, nor replacing your monitor. It's about showing what you can do with some inexpensive hardware and some code. No matter what, at the end of the day, it's still a touch-capable extension for my PC, costing significantly less than what a "true" touchscreen would cost when hooked up instead.

If you want to try it yourself, you'll only need the WT32-SC01 Plus, a bit of tinkering with PlatformIO or Arduino IDE, and the Python server we linked earlier. With a little bit of modification, you can extend this setup into a streaming panel, remote desktop, or even a budget touchscreen monitor... though maybe try a larger display for that last one.