Building a custom firewall running OPNsense is one of the greatest achievements and upgrades one can make to their local area network (LAN). A router supplied by an internet service provider (ISP) is good enough for standard home setups with a few wireless devices, but once you start adding more clients and plan out a network with advanced features such as VLANs, you're likely to encounter some issues. Then there's the whole privacy argument!

However, one area OPNsense seemed to lack was in download speeds on Point-to-Point Protocol over Ethernet (PPPoE) and Fibre to the Premises (FTTP) connections. This would essentially cut your download speeds by 50% and one solution was to virtually host OPNsense through a hypervisor like Proxmox. That's no bueno with my mindset, so I searched for a bare metal solution, and I managed to find it from Ben Tasker. All that was required was a few tunables and a cheeky reboot.

OPNsense, FTTP, and PPPoE

The issue lies with FreeBSD

According to various community discussions and online resources, I've come to realize that FTTP connections, PPPoE fiber, and FreeBSD do not play well together on bare metal. The underlying OS will only use a single core to handle incoming packets on the connection, effectively bottlenecking your network speed to how fast your cores can ramp up. Since I'm using a fanless firewall device with a low-power Intel CPU, this was severely hampering my bandwidth.

The device itself is more than adequate for OPNsense and a 1Gbps fibre connection, but it's difficult to notice this issue with the OPNsense dashboard unless you dive into the hardware to see what system resources are being used. After confirming that the firewall was indeed only using a single core during speed tests and file downloads, it was time to hunt down a solution to try and force OPNsense (and FreeBSD) to utilize the remaining three cores.

Depending on your hardware, your NIC may support Receive-Side Scaling (RSS), but OPNsense disables this by default. RSS allows your NIC to process incoming packets using multiple CPU cores. Think of your incoming connection like a highway with a lane per CPU core. Without RSS enabled, my fibre is running with three of the four lanes (quad-core CPU) blocked with road works. Enabling RSS clears the road works and provides more bandwidth.

Applying the quick fix

A few tunables and we're good to go

I was slightly disheartened to learn how deep this issue goes, as it's likely going to take much longer for it to be addressed, though I have heard pfSense+ has everything configured in such a way that FreeBSD will use multiple cores for the job, and these improvements have been submitted to FreeBSD. To fix OPNSense right now, I needed an immediate solution that didn't involve virtualizing my firewall.

And I viewed upgrading the device to one with a beefier CPU as a waste since it's not a hardware issue, but software, and this wouldn't solve the issue, but instead mask it. Thankfully, Ben Tasker (and a few others) came across the solution, which involved using a few tunables to alter the way OPNsense runs. It's nothing drastic, but it ensures OPNsense uses RSS with each core receiving packets, and drivers use the ISR queues, distributing load across all cores.

With these tunables configured, my download speed went from between 300-400 Mb/s to a whopping 900 Mb/s, which is precisely what my ISP plan provides. So, let's get to the fixes.

1. Log in to OPNSense.
2. Navigate to System > Settings > Tunables.
3. Click the + button to add a new tunable with the following to make OPNsense use all available cores for handling incoming packets. (Note that the name and value go into the Tunable and Value fields, respectively.)

   net.isr.maxthreads = -1
   net.isr.bindthreads = 1

4. Add another tunable. This time, we're allowing NIC drivers to use ISR queues.

   net.isr.dispatch = deferred

5. Next up is to add tunables enabling RSS. (Note that net.inet.rss.bits should be set to the square root of how many cores you have.)

   net.inet.rss.enabled = 1

   net.inet.rss.bits = 2

And that's all there was to it. Giving OPNsense a quick reboot unleashed the power of my 1Gbps FTTP connection using PPPoE, and it's glorious. I double checked my Bufferbloat rating just to make sure these settings didn't send my latency into the stratosphere, but it improved everything across the board. Alongside traffic shaping, I now had an almost perfect connection rating, making it a great link for online gaming and other latency-sensitive tasks.