While my home lab grows, I've been trying to build a more resilient network at the same time. Some of that is handled by VLAN design, some by adding uninterrupted battery backups to my network stack, and others by physically separating traffic as needed. I also wanted faster links for the backbone of the network, even if many of the devices on it are still limited to gigabit speeds.

When it came to slotting a new home server into this mix, I didn't want to be limited by the onboard networking ports. Or really, a port, because there was 10 GbE and 2.5 GbE onboard and I wanted everything in the rack to be connected at 10 GbE or above. So I picked up a PCIe network card from eBay to add more speedy ports, but as I got into designing further I realized that speed was the least impressive part of what I gained.

I confess I bought this network card for the 25 GbE port

But maybe not for the reasons you're thinking

If there is one networking lesson I've learned over and over again, it's that whatever specification you design for now might not be the one you want to use later on. I built a 2.5 GbE network and ended up with more 10 GbE client devices instead. Another time I bought a NAS with multiple gigabit ports thinking that would be a good way to expand the network, without considering the issues with the approach.

Sure, they're things that could have been avoided with more research, but you don't know what you don't know, and sometimes you only learn by failure. So when I wanted to add 10 GbE network links between some of the core devices, I made sure that the NICs I picked up could also do higher speeds again.

That led to picking up some ex-enterprise Mellanox NICs from eBay, which handle 10/25 GbE from SFP+ ports. The company lives on as Nvidia owns them now, but the older cards are still well-supported and perfect for home lab use. The ConnectX-4 models I picked up even have two ports, so I can daisy-chain them or use two links for redundancy, management, or dedicated storage links.

What I actually got was peace of mind

Having additional network ports gives you options

One of the reasons I don't like using onboard network ports is that for consumer and prosumer motherboards, they're often flaky. That could be driver or firmware issues, but also faster network ports get hot, and consumer boards don't do a good job of handling those thermals. Now, server motherboards are built differently, and many now have slot-in modules to swap out the networking portion if they fail, but that's not what I'm working with at home.

Adding more network ports via PCIe cards gives me redundancy in case of things breaking, which is common enough that I value having another port or two just in case. Having two connected as failovers makes the process automated, but that's not everything that having another network port gives me. It also allows for a dedicated management network, which keeps admin tasks separate from client devices and, more importantly, means I don't get locked out of devices by making changes to the main network.

Link aggregation is another potential use that's relevant to home server use, giving higher aggregate bandwidth for the number of devices connecting at once, even if it doesn't increase overall speed. Or I could dedicate that network port to virtual machines, or to storage traffic, or high availability needs, or even just to use as a learning platform for advanced networking concepts before applying them to the wider network.

Latency-sensitive storage I/O deserves its own network segment

iSCSI, NFS, or SMB all benefit from their own connection

I've moved all of my network storage away from the networked machines doing compute tasks, so I don't have to worry about destroying data if those machines need wiping. That's even more important as I consider high-availability clusters for crucial infrastructure, and that gives me a problem to solve.

See, for my workstations and laptops, having Quality of Service enabled on the network provides a more consistent user experience for browsing and other tasks that I'd notice latency issues. But I don't want that traffic management to affect my storage traffic, especially if it's from an iSCSI share where latency spikes hurt performance.

The fix is an easy one. Use that second network port as a dedicated storage path, so that latency-sensitive storage I/O isn't competing for resources with the usual LAN traffic. While I could handle this with VLANs, they still compete somewhat on a single NIC, and I'd rather do things with physical separation.

That means storage traffic to the server is always able to take advantage of the entire connection, regardless of how many users are on the network or accessing other services. It also means that storage traffic isn't slowing down other LAN traffic, as it would have been before adding the additional network port.

I might have bought a second network card for speed, but I gained much more

I don't particularly like using onboard motherboard network ports for critical infrastructure, as they're often flaky or have weird driver quirks, and that's to say nothing about if they fail on you. I'd rather use a dedicated network card with multiple ports, so I have redundancy in case of problems, and options for setting up a more resilient network. Whether that's as a second network segment, like bridging between an iOT network and the main network with Home Assistant, or providing a path for latency-sensitive operations like network storage shares.

This time around, I'm going to use that additional network port for a management VLAN, so it never gets used for any other task. That will keep me connected in case of misconfiguration of the rest of the server, and I can recover gracefully and with the minimum of cursing.