We've all been there at some point in our home lab journey. The decision to run a stack of Docker containers on our NAS, maybe because we didn't want to run experiments on our desktop PC, or perhaps because we thought that using the NAS as a mini server was a good idea. But the longer I had this setup, the more I got the nagging feeling that I shouldn't be continuing, and that a dedicated machine for container experiments was a better idea.

I didn't come to this decision quickly, and I'll be the first to say that it's your home lab, not mine, so run what you want, where you want. But for me, I'm splitting my home lab, keeping the NAS for storage only, while my Docker stack runs on its own Intel N150-based mini PC. It was a relatively low-cost switch, and I've noticed that both parts of my home lab run better as a result, so I'm not going back to the way I used to do things.

It's a single point of failure

I want my storage to be storage, not services

If there is one thing that my years of self-hosting and home lab experimentation have taught me, it's that the data center separates compute and storage for a reason. Running everything from my NAS was a necessary evil at many points along the way, partly because I didn't have another machine to spare, and I didn't want to run server tasks on my powerful (and power-hungry!) desktop PC.

But what that meant was a single point of failure for both my self-hosted services and my storage. That's not very resilient, or well planned, and I had too many other devices dependent on both parts of my tech stack for them to be combined in such a way. Moving my containers and VMs to an all-flash mini PC has meant my RAID array can go back to being backup and long-term storage, and a stepping stone before longer-term storage in Backblaze. It also means that my containers are no longer competing for precious compute cycles on the underwhelming NAS CPU, and I now have some level of redundancy in case things go wrong.

My NAS CPU is underpowered at the best of times

Let alone when it's running a full container stack

My home lab has evolved over time, and the current iteration has a Threadripper-based system for the main server, but this is the first time I've had a powerful CPU running things. Every other server or NAS was powered by chips that wouldn't have powered my smartphone, let alone a compute-heavy taskload. Running Docker containers alongside the everyday tasks the NAS needed to accomplish was often too much for the processor — even after a substantial RAM upgrade — slowing everything down until one task finished, reducing overall strain.

What was needed wasn't a powerful server to run everything on. It was a different type of containerization, moving compute tasks to another machine entirely, and allowing the NAS to do what it was good at, storing data and running backups from my other machines. It's still used for long-term data storage of some Docker containers, but it now does so over the network, which was recently upgraded and is no longer the bottleneck in my home lab configuration.

My NAS OS is a pain

Yes, Synology, I'm looking at you

When I got my first Synology NAS, I was unreasonably excited. There were only a few premium NAS companies back then, and Synology had a reputation amongst enthusiasts that rivaled QNAP or Asustor. Over the years since, the reputation has tarnished, and I no longer enjoy using Synology's DSM operating system. The desktop-like environment feels artificially restrictive, and that's before we even talk about the things that have been removed or cut down.

Synology used to have a very nice Docker implementation, until it replaced it with the substandard, proprietary Container Manager. This turns what should be simple Docker usage into a mess of GUI forms to fill, and it won't let you modify any settings post-installation, from ports, volumes, env settings or links, turning what should be a simple edit to a Docker Compose file into a drawn-out mess where you have to duplicate the container and do the changes during the duplication. It's a pain, and it's so much easier to use VS Code and push the compose files to my new mini PC setup.

I don't want filesystem degradation

Docker's storage driver adds substantial overhead

Docker uses its own virtualized storage drivers, adding an extra layer of performance overhead to I/O operations. That's fine on a powerful enough CPU, but the ones inside my NAS units are not that. The overlay2 filesystem Docker uses has copy-on-write penalties, and the more containers I have running, the worse it gets. That reduces performance across the board, especially on the mechanical hard drives in my NAS, which can't go into sleep mode anymore, increasing overall wear and reducing their lifespan.

At least my NAS is using btrfs now instead of EXT4 so that it has copy-on-write protections, but that's only part of the problem again. Backing up the databases used by my Docker containers and the Docker volumes used is a headache of file ownership and permissions that is much better served by doing full-device incremental backups from another source machine instead of doing them while they're running on the NAS.

It makes upgrades and maintenance more complicated

And in some cases, impossible

The kernel and Docker versions used in NAS devices are often behind the mainline ones, if they're updated at all. That becomes a problem pretty quickly when Docker images are updated to be compatible with the latest versions and the new system calls and features they enable. But a lack of ongoing support is only part of the issue.

Containers are meant to be ephemeral, but that doesn't mean they're always transferable. Upgrading my NAS to get kernel support could require a different CPU architecture, which would cause my Docker stack to fail. While you could make your Docker images multi-arch, not every image is built that way, and I'd have to rebuild and start from scratch. Or I could use QEMU or another virtualization tool to move the stack, but that introduces yet another layer of virtualization and file-system weirdness, and I'd rather not have to worry about it. And there's always an ongoing battle over permissions between my Docker stack and the NAS OS, so it's easier to deal with on a separate mini PC.

I prefer to keep my services away from my data storage

Containerized workloads like Docker are designed to be encapsulated so that services don't create an unstable OS environment. But that level of protection only goes so far, and it doesn't do anything when you overload the host device. And that's what I found with running Docker on my NAS. It worked, but it came with performance hits both to the self-hosted services and to the core services running on the NAS. There just weren't enough system resources to go around, including spinning up the hard drive array more than necessary to store the container's data. I got tired of hearing the array chugging around when I knew I wasn't writing active data to it via backups or other storage tasks, and now my Docker stack is efficient and quiet as it's on an NVMe-only mini PC.