3 Immutable Operating Systems: Bottlerocket, Flatcar and Talos Linux

For those that don’t know, immutable operating systems have been increasing in popularity recently. An immutable operating system is one in which some, or all, of the operating system file systems, are read-only, and cannot be changed.

Immutable operating systems have a lot of advantages. They are inherently more secure, because many attacks and exploits depend on writing or changing files. Also, even if an exploit is found, bad actors cannot change the operating system on disk (which in itself will thwart attacks that depend on writing to the filesystem), so a reboot will clear any memory-resident malware and recover back to a non-exploited state.

Immutable systems are also easier to manage and update: the operating system images are not patched or updated but replaced atomically (in one operation that is guaranteed to fully complete or fully fail — no partial upgrades!

Immutable systems also can claim to be more stable than traditional operating systems, simply by virtue of eliminating many of the vectors that introduce instability into a system — most of which are human. No sysadmins can “just change this one setting to fix things” — with unforeseen impacts that aren’t found until hours later. (I’ve been that sysadmin.) No partially complete terraform or puppet runs that leave systems in odd states…

On the workstation side, there are approaches to immutable OSes such as rpm-ostree. This attempts to create immutability and image-based deployments in the operating system, but layers a flexible file system architecture on top, so that packages can still be managed and updated by RPM.

On the server side, there is a spectrum of immutability amongst container-specific operating systems. All support image-based OS updates, and no package manager at all. Some operating systems such as Flatcar Linux make /usr read-only, but allow common runtime modifications such as dynamically loading kernel modules, and overriding systemd configurations.

Bottlerocket, from Amazon Web Services, adds in the /root partition being read only, and disables SSH access by default (it can be run in a privileged container). Bottlerocket does allow kernel modules to be loaded in older releases, but has switched to only allowing modules that are signed with the Bottlerocket image key — those modules shipped with Bottlerocket.

Talos Linux, from Sidero Labs, takes the immutable concept even further. It is another OS with a fully immutable file system, and a full API for management, and unlike Bottlerocket, completely removes SSH and console access, and runs anywhere you run Kubernetes — all cloud providers, bare metal, virtualized systems, even within Docker and on SBCs like Raspberry Pis. Talos Linux also requires kernel modules to be signed with the same key used to build the kernel — and because this key is ephemeral, it makes the kernel completely static and immutable.

Of course, as in all things, the strengths that are inherent in immutable operating systems are also their weaknesses. Not all systems need to run the same workloads, firmware, and packages. How do you customize an immutable operating system to adapt to a variety of applications?

Flatcar Linux, having less immutable characteristics (the root filesystem is writable, and nodes support SSH, for example), addresses this issue with the Ignition project (often in conjunction with Afterburn), which runs on the first boot of a node, and can modify the file system, add files, configure users, and other operations. This is similar to a typical Linux workflow where tools such as Puppet, Chef, etc are used (except it can only run once at first boot).

Similarly, Bottlerocket uses the concept of Bootstrap containers to customize the way the node runs. These are containers that are executed by systemd after the node has booted, that have access to the root filesystem and host devices. They can execute on every boot, or just the first boot. The order of these containers is non-deterministic. Thus effectively the bootstrap containers are not constrained by the immutability of the file system — which somewhat reduces the effectiveness of the immutability for security purposes.

System Extensions

Talos Linux, being fully immutable, has to take a more stringent approach to modifying nodes. Talos Linux uses a concept called System Extensions. It can work while maintaining full immutability of the operating system because it requires that such extensions be set up during the node installation or update. In order to effect any change on the system extensions, Talos Linux has to be re-installed on the node.

This encourages (indeed, enforces!) the idea of `cattle-not-pets’ as in order to add or change a node via system extensions, the contents of the disk will be wiped and replaced. During the installation/re-installation, the system extension container images will be stored in the initramfs as squashfs images; during boot, they are overlaid on top of the Talos Linux squashfs root file system using read-only squashfs overlayfs mounts. All file systems (except /var for Kubernetes data) are immutable thereafter.

The systems extension concept preserves the immutability of the file systems while allowing nodes to load different capabilities, such as specific firmware, or gVisor for further isolation of containers.

These different approaches to allowing modification of immutable systems have different strengths, but allow the user to select what is appropriate for them: from managing systems in a familiar way (such as with rpm-ostree, or using SSH) to new paradigms such as that implemented by Talos Linux, where there is no SSH, all management is done by API, and security is paramount — with the requirement of learning a new way to manage systems.