ESP32 microcontrollers come in all shapes and sizes, varying in capabilities and features from model to model. The ESP32-S3 is on the higher end of the spectrum when it comes to performance and capabilities, and I recently picked one up for playing with and testing. Thanks to its USB On The Go (OTG) support, I turned it into a computer mouse, and it's both amazing that it works and the most impractical way that I think I've ever used a computer.

I know one of the first questions you probably have: Why? The answer is simple: Because I can. The ESP32-S3 has native USB-OTG thanks to its second USB-C port, along with a couple of decent ADCs, and just enough power to impersonate a USB Human Interface Device (HID). That means you can plug it directly into a PC and pretend it's a keyboard, a gamepad, or, in my case, a mouse. I connected a cheap two-axis joystick to an ESP32-S3 N16R8, flashed a tiny Arduino sketch to it, and could control my computer with a level of fluidity and speed that typical users could only dream of.

It works. It clicks, it moves, and boy, is it the most impractical way I've ever used a computer.

The setup that shouldn't work this well

It's surprisingly consistent

I used a basic joystick module with three outputs: VRx on GPIO 8, VRy on GPIO 18, and SW (the push-to-click switch) on GPIO 17. The ESP32-S3 handles USB HID natively, so once it's picked up, Windows sees it as a standard mouse. No need for drivers or any of that nonsense, my creation is completely plug and play.

To be honest, while I knew on a technical level it would work, I couldn't help but still feel surprised. I guess it's because I was so preoccupied with whether or not I could do it, without stopping to think whether I should. Yet once my $5 microcontroller and what can only be described as a toy joystick were connected, Windows welcomed it with open arms in the same way it would welcome, well, an actual mouse.

I can move the mouse pointer. I can click. Everything works as it should, and because it's a "mouse" rather than a joystick (in Windows' eyes, anyway), it works in games that require a mouse and keyboard. I fired up Deadlock and could look around and shoot. With that said, there was no hope that this contraption would work for any level of competitive play. In fact, I'd argue that it would probably be more frustrating than fun.

But it works!

With that said, it makes you truly realize how much ergonomics, acceleration curves, polling rates, sensor fusion, and surface tracking matter when you replace a modern mouse with what essentially boils down to a potentiometer strapped to your thumb. The joystick's neutral position drifts over time, and the ADC's whispers of noise can translate into micro-jitters. Plus, any attempt at precision mousing turns into a tightrope act of flicking back and forth to get it just right.

To be clear, you can tame some of these issues with filtering, dead zones, and calibration. But you'll keep discovering new problems that make you question the point of this entire exercise. Trust me: I know from experience now. I did some work tuning it to make it usable, and to be fair, it was usable, but you'd want a proper controller housing for it.

Smoothing over the issues

Deadzones and more

So there are a number of issues with just connecting a joystick to an ESP32 and calling it a day as an input device. Modern joysticks require calibration, deadzones, and more, so I had to implement those manually. These are the constants I defined to cover up those immediate issues:

• constexpr int DEADZONE = 120;
  • This can be a 0 to 4095 range, and is used to ignore small stick noise.
• constexpr float MAX_SPEED_PX = 10.0f
  • This is the pixels per loop at full deflection
• constexpr float CURVE_EXPONENT = 1.35f;
  • This is used to define how fast it moves as we move the joystick to the edges.
• constexpr int SAMPLE_INTERVAL = 5;
  • This is the loop cadence and reporting rate. This works out to be 1000/5, or 200 cycles per second. Factoring in overheads, the reporting rate is likely a little bit lower.
• constexpr int CAL_SAMPLES = 200;
  • We take 200 samples at boot to calibrate the center value, so don't move it immediately after plugging in.
• int centerX = 2048, centerY = 2048;
  • We define a fixed calibration point, but this is overwritten when we gather our calibration samples.

So with these, we eliminate idle noise (because values below 120 are likely not from human inputs), we set a more realistic speed, and we build in a "curve" to the cursor speed, so that it becomes slightly faster the more we push the stick in a given direction. The curve speed calculation looks like this:

float curveSpeed(int delta) {
 int ad = abs(delta);
 if (ad 
 // normalize values beyond deadzone
 float norm = float(ad - DEADZONE) / float(4095 - DEADZONE);
 if (norm if (norm > 1) norm = 1;

 // curve + scale
 float spd = powf(norm, CURVE_EXPONENT) * MAX_SPEED_PX;
 return spd * (delta > 0 ? 1.0f : -1.0f);
}

The "delta" value that we pass is calculated by subtracting the analog value read from the pin from the center value on the corresponding axis, and so long as it doesn't fall in our deadzone, we then use our curve exponent and the max speed to calculate a movement speed and return a value to move the mouse by. If the value ends up being 0, the entire calculation resolves to 0, so the returned value is also 0, and the mouse doesn't move.

On top of that, we set a static calibration as a fallback, and we calculate a new calibration based on the current static value of the joystick. If you plug in the ESP32 while moving the joystick in any given direction, it will be "stuck" moving in the opposite direction, as the center value will have moved.

Finally, here's the main loop, and the button is a basic Mouse.press(MOUSE_LEFT) when activated.

void loop() {
 // Read joystick
 int rawX = analogRead(PIN_VRX);
 int rawY = analogRead(PIN_VRY);

 int dxRaw = centerX - rawX;
 int dyRaw = centerY - rawY;

 float fx = curveSpeed(dxRaw);
 float fy = curveSpeed(dyRaw);

 // Send integer deltas as HID wants int values
 int8_t mx = (int8_t)roundf(constrain(fx, -127, 127));
 int8_t my = (int8_t)roundf(constrain(fy, -127, 127));

 if (mx != 0 || my != 0) {
 Mouse.move(mx, my);
 }
}

That's it! By combining some safeguards with the joystick, it suddenly becomes somewhat usable.

Power, pins, and what's next

I could build a gamepad... or never touch it again

There are a few important things that I learned as a part of this project, so it wasn't as totally useless as I've joked throughout the article. Many joystick modules are just two 10k ohm potentiometers and a switch. Many happily run at 3.3 V, which keeps the analog outputs inside the ESP32-S3's comfort zone. If your specific board is labeled for 5V, double-check that the voltages heading into your ADC pins won’t exceed 3.3 V. The wiring I used was VRx to GPIO 8, VRy to GPIO 18, and the switch to GPIO 17 with the internal pull-up enabled, and while the joystick has a "5V" indicator, it didn't work unless it was connected to the 3V3 connector on the ESP32.

To be honest, by far the biggest and most annoying problem of this entire ordeal was the repeatability of movements. Returning to the exact same pixel twice in a row is hard on a thumbstick that doesn't spring to a mathematically perfect center and whose "center" drifts a little as your hand presses differently each time. However, you could theoretically build a full ESP32-based controller based on the gamepad example in the Arduino-ESP32 library for just a few dollars. You'd need an actual casing, but you could 3D print one or use an existing one.

Would I recommend using it as a mouse emulator of sorts, though? Absolutely not. However, as a weekend project that makes you appreciate how much engineering hides in a "simple" mouse or joystick? For sure! It taught me about USB HID, filtering signals, and the work that has to go into making a joystick "work" in the first place. For example, I had observed on cheaper controllers that plugging them in while pressing the stick in a certain way could exhibit similar behavior, and now I know a likely cause. I could have assumed it anyway, but it was cool to see the exact same behavior in something I built myself.

If I were to keep going, I'd add an on-device calibration mode, per-axis acceleration curves, and a few extra buttons mapped to middle-click and scroll. An IMU could even add "air mouse" gestures. Or, now that I've scratched this weird itch, I'll just go back to a real mouse and never touch this abomination again. Who knows.