Tutorial: Install Flatcar Container Linux on Remote Bare Metal Servers

This tutorial is the third part of a series on Flatcar Container Linux where we install the OS on a set of remote bare metal servers running the edge infrastructure. (Part 1), (Part 2).

Installing Flatcar Container Linux on a physical server is no different from other Linux distributions. You can download the ISO image and boot the machine with the live OS without actually installing it to the disk. Flatcar comes with a CLI called flatcar-install which can install the OS to a local disk. You can also use any Linux live CD to download and install Flatcar to a non-active device of the machine.

But, in most of the edge scenarios, we cannot access the physical machines to perform an interactive installation. Assuming we have access to another host that can run the PXE boot environment and an HTTP server, we can remotely deploy Flatcar Container Linux on any number of machines.

In this tutorial, I will walk you through all the steps required to configure and install Flatcar Container Linux on three nodes based on Seeed Studio’s Odyssey Mini PC. In the next part, we will extend this scenario to deploy a highly available K3s cluster.

The official documentation of Flatcar Container Linux doesn’t cover this scenario. It took me a lot of time and effort to learn the remote deployment process. Fortunately, the experience of dealing with the good old CoreOS Container Linux came in handy. This would help anyone looking at performing an unattended deployment of Flatcar Linux in remote locations at scale.

Deployment Topology

Before proceeding with the installation, let me share the network topology and the layout.

We have four hosts — a PXE boot server and three edge nodes. The PXE boot server can even be a virtual machine running in the same subnet as the edge nodes. This will enable the nodes to discover the PXE boot server.

👁 Image

Flatcar Container Linux heavily relies on Ignition files to perform the configuration. As soon as the OS boots, it downloads the Ignition file from an HTTP server and configures the essential elements such as the hostname, users, SSH keys, and other systemd unit files. The same PXE boot server doubles up as a simple HTTP server to serve the Ignition files.

You also need to know the MAC address or the IP address of each node to correctly deploy the associated configuration. The PXE boot server should have a static IP address or a DNS name for the nodes to discover and access the TFTP and HTTP endpoints.

👁 Image

I used an Ubuntu 18.04 machine as the gateway server with remote access. All the following steps assume you are running an Ubuntu or Debian machine as the PXE boot server.

Preparing the Gateway Machine

We will start by installing the dnsmasq and the tftp server on the gateway. Since I already have a DHCP server running, I am disabling it. You may want to include DHCP if you don’t have one running in the network.

systemctl disable systemd-resolved
systemctl mask systemd-resolved
systemctl stop systemd-resolved

apt-get install -y dnsmasq pxelinux syslinux-common
service dnsmasq stop
mkdir -p /var/lib/tftpboot
mkdir -p /var/lib/tftpboot/pxelinux.cfg

cp /usr/lib/PXELINUX/pxelinux.0 /var/lib/tftpboot/

cp /usr/lib/syslinux/modules/bios/{menu,ldlinux,libmenu,libutil}.c32 /var/lib/tftpboot/

mv /etc/dnsmasq.conf /etc/dnsmasq-conf.bak

cat > /etc/dnsmasq.conf << EOF port=0 log-dhcp dhcp-range=10.0.0.1,proxy dhcp-boot=pxelinux.0 pxe-service=x86PC,"Network Boot",pxelinux enable-tftp tftp-root=/var/lib/tftpboot EOF 
echo "DNSMASQ_EXCEPT=lo" >> /etc/default/dnsmasq 
service dnsmasq start

Next, we will download the Flatcar Container Linux PXE boot images into the TFTP directory and verify them.

cd /var/lib/tftpboot
wget https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_pxe.vmlinuz
wget https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_pxe.vmlinuz.sig
wget https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_pxe_image.cpio.gz
wget https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_pxe_image.cpio.gz.sig
gpg --verify flatcar_production_pxe.vmlinuz.sig
gpg --verify flatcar_production_pxe_image.cpio.gz.sig

Generating Ignition Files for PXE Boot

Flatcar Linux uses an Ignition file to configure itself which is a better alternative to cloud-init. An Ignition file contains everything from the user information, SSH keys, and even an entire definition of a systemd unit file.

We will now create two Ignition files — one used during the PXE boot process and the other to persist the configuration on the disk. The second one is the most critical element which will embed the SSH key and the hostname of the machine.

I am assuming that you have an SSH key pair handy with you. We will be embedding the public key in the Ignition file to enable remote access.

An Ignition file has two different forms — one is a human-readable YAML file while the other is a nested JSON file meant for the OS. We will first define the Ignition file in YAML and then transpile it into the JSON format. Refer to the documentation for more information on Ignition.

Firstly, download the Config Transpiler, the ct command-line tool, and add it to the path.

Let’s create the YAML file used for the installation process.

systemd:
 units:
 - name: installer.service
 enabled: true
 contents: |
 [Unit]
 Requires=network-online.target
 After=network-online.target
 [Service]
 Type=forking
 TimeoutStartSec=600
 RemainAfterExit=yes
 ExecStart=/usr/bin/sh -c "flatcar-install -d /dev/sda -i /opt/ignition.json && udevadm settle && systemctl reboot"
 [Install]
 WantedBy=multi-user.target
passwd:
 users:
 - name: core
 ssh_authorized_keys:
 - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGdByTgSVHq...
storage:
 files:
 - path: /opt/ignition.json
 filesystem: root
 mode: 777
 contents:
 remote:
 url: http://10.0.0.5:8080/node-1-ignite-boot.ign

This file has many sections that are critical for the successful installation of the OS.

The systemd section creates a unit called installer.service that’s run as soon as the PXE boot finishes. Inside the unit file, we invoke the flatcar-install CLI to install the OS on /dev/sda device. Please note that the entire disk will be formatted during the installation. As soon as the installation finishes, we perform a reboot by calling systemctl reboot. Make sure that the PXE boot service is turned off on the gateway server. Otherwise, the installation gets into a never-ending loop.

The passwd section adds users and their associated SSH keys to enable access to the machine. While this may not be necessary during the first boot, it makes it easy to debug any issues during the installation. Here, we added a user by name core.

In the next section, we write files to the disk that are persistent. Since Flatcar Linux looks for an Ignition file during the boot process, we will save a file to the disk that’s used each time the machine boots. This will contain essential elements such as users, SSH keys, hostnames, and other processes that we want to start automatically at the boot time. Note that instead of hardwiring that file, we are downloading from the gateway server over HTTP.

Let’s take a look at the YAML definition of the node-1-ignite-boot.ign file.

passwd:
 users:
 - name: core
 ssh_authorized_keys:
 - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGdByTgSVHq...
storage:
 files:
 - path: /etc/hostname
 filesystem: root
 mode: 0644 
 contents:
 inline: node-1 

This Ignition file essentially does two things — create a user and an associated SSH key, and setting the hostname by adding a line to the /etc/hostname file. This file will be effectively used to configure the OS on each boot.

Finally, we will run the YAML files through the transpiler to convert them into JSON format. We will also move them into a separate directory which will be used by the HTTP server.

mkdir -p /var/lib/tftpboot/ignition
mv node-1-ignite*.yaml /var/lib/tftpboot/ignition
cd /var/lib/tftpboot/ignition

ct < node-1-ignite.yaml > node-1-ignite.ign
ct < node-1-ignite-boot.yaml > node-1-ignite-boot.ign

It’s time for us to run an HTTP server to serve these files. If you have Apache or NGINX running anywhere in your network, you can host the files there. Otherwise, you can launch a simple HTTP server with Python.

 python3 -m http.server 8000&

If you have come this far, you know that we need to close the loop by creating the PXE boot files that bring everything together.

Under the /var/lib/tftpboot/pxelinux.cfg directory, create a file by name node-1 with the following content:

default flatcar
prompt 1
timeout 15

display boot.msg

label flatcar
 menu default
 kernel flatcar_production_pxe.vmlinuz
 initrd flatcar_production_pxe_image.cpio.gz
 append flatcar.first_boot=1 ignition.config.url=http://10.0.0.5:8000/node-1-ignite.ign

We now need to map the IP address of node-1 with the PXE boot file to ensure that the configuration is applied correctly to the target host. This is done by creating a link to the above file that has the hexadecimal form of the target IP address.

For example, the IP address of node-1 is 10.0.0.70 which becomes 0a.00.00.46 when translated to hex. So, we will create a link that matches the hex code of the IP address to the PXE boot file.

cd /var/lib/tftpboot/pxelinux.cfg
ln -s node-1 0A000046

Repeat these steps for node-2 and node-3 by modifying the YAML files, PXE boot files, and generating the links based on the IP addresses.

That’s all! We are all set to install Flatcar Container Linux remotely on a bare metal server through PXE boot.

Janakiram MSV’s Webinar series, “Machine Intelligence and Modern Infrastructure (MI2)” offers informative and insightful sessions covering cutting-edge technologies. Sign up for the upcoming MI2 webinar at http://mi2.live.

Feature image by S. Hermann & F. Richter de Pixabay.