I'm a sucker for a new distro to load up and play with, along with the dozens I've already tried. Then I discovered Proxmox, and my distro-hopping got to new heights. I mean, I can spin up dozens of VMs on the server I run Proxmox on, and then I'm not overwriting the same poor old laptop that was being used to fuel my habit.

Then I learned about immutable operating systems, and had to try them out. I mean, the promise of not being able to "break" your install to the point of failure is intoxicating, but it didn't quite live up to my expectations. I don't like flatpaks; I like being able to edit system files to fix issues or tweak things to my liking, and the whole thing was doomed.

But then I stumbled upon a nugget of wisdom that said I could make any of my Proxmox VMs immutable with a simple configuration variable, and the invasive thoughts started again. Would I like immutable systems if I could flick a switch, make the changes I wanted, and then make it immutable again? It turns out I do, but I found a bunch of other use cases I'd not considered, and now I can't live without them.

I didn't know you could make VMs immutable this way

But now I do it's pretty darn cool

Proxmox uses QEMU for virtualization, and only a few of the incredibly long list of configuration options are exposed to the user through the UI. That's by design: many of these flags can and will break your system if misused, so the company has wisely opted to let you use the safe options. That doesn't mean you can't edit some configuration files to use them.

When you create a VM on Proxmox, it creates a configuration file, which lives in the following directory:

/etc/pve/qemu-server/

Where VMID is the number of your Proxmox VM. In my case, it's 106, so substituting the number as a string and opening the file in nano gets me into the config.

This file contains QEMU details for the VM, including core count, memory allocation, attached virtual disks, and so on. But to make the install immutable, all we add is:

snapshot=1

That's it. Exit the file and save over the existing one, and you've got an immutable OS in that VM. When you boot it from now on, it will create a transparent overlay disk, meaning that any updates, downloads, and other changes are written to /tmp and discarded when the VM is turned off or rebooted.

If I want to keep that data, I use the commit command in the QEMU monitor, which flushes the /tmp file to the correct folders inside the VM, as if I'd made the changes to a 'normal' VM install. This makes Proxmox an incredibly versatile virtualization platform, able to adjust to my needs whenever they change.

Why would you want an immutable OS anyway?

It's all about not having to live with your mistakes

Linux is an unforgiving system in the wrong hands. I don't mean for general day-to-day stability, where it rivals macOS or Windows, but the wrong command in the terminal can wipe out system files, folders, or your entire install if you've entered the wrong folder tree.

The point I'm trying to make is that the guardrails are off, and you're solely responsible for the outcomes. Whether it's typing in a series of commands you found on a blog from ten years ago, or running a script via curl, nobody but you is in charge. Windows won't even let you delete some files, and macOS has a hidden recovery OS that can fix the operating system if everything goes wrong, but Linux will go "okay, I'll remove your system drive" and smile while doing it.

The Linux community still wants that control, but it also decided that immutable systems might be less intimidating to new users. After all, if you can't delete system files, your system is safer (from you, generally). Whether it's Fedora Silverblue, Bazzite, NixOS, or VanillaOS, there are plenty of options with read-only system files, which is good for some things but causes other issues to those used to normal Linux behavior.

Sure, failed updates are easier to fix. Still, dependency issues with packages you might want to install can be an absolute nightmare, and some command-line utilities can break even immutable installations.

It's different on Proxmox

Using QEMU's snapshot feature with my Proxmox VMs, it's only as immutable as I want it to be. What I mean is, when I need to install something, or update the system, I can go into the conf file and change the 1 to a 0, and then it's a normal VM installation again. When I've finished changing things, I can go back in and switch it to on, and it's immutable again.

This opens tons of options for setting up VMs for different experiments, tasks, or to avoid the weirdness of atomic updates, where the entire OS is replaced at once, like it's a Docker image. And I can use the usual package manager, without resorting to flatpaks, and install things from non-official repos with relative ease.

Plus, it's pretty neat for malware research

Because nothing you do to the VM is written to disk permanently unless you want it to, it's a perfect vehicle for investigating and analyzing malware. Create the VM, make it a snapshot, and let the malware run wild knowing that it's only making changes in a disk overlay and won't cause permanent harm.

You can even take snapshots of the VM's state while it's running, so you have a record of what the malware did, and then reboot the VM into a fresh OS.

Immutable VMs on Proxmox are now firmly part of my tool belt

I used to take multiple backups and snapshots of my VMs after every change I made, but it takes time away from actually doing tasks, and it's a chore managing the snapshots and storage. With immutable VMs, they're instantly refreshed to a known-good state, and I don't have to do anything. It's fantastic, and it reduces a ton of steps in my usual process, so I can spend more time on what interests me.