Incus

Incus is a next generation system container and virtual machine manager. It offers a unified user experience around full Linux systems running inside containers or virtual machines. Incus is a fork from LXD.

Incus is image based and provides images for a wide number of Linux distributions. It provides flexibility and scalability for various use cases, with support for different storage backends and network types and the option to install on hardware ranging from an individual laptop or cloud instance to a full server rack.

When using Incus, you can manage your instances (containers and VMs) with a simple command line tool, directly through the REST API or by using third-party tools and integrations. Incus implements a single REST API for both local and remote access.

Installing

To install incus, issue:

Pay attention to kernel config requirement messages when installing Incus. Missing kernel features may mean the program won't function at all. Also pay attention to any optional features printed by emerge after successful installation.

In a stable system, currently Incus and some of its dependencies must be allowed via ACCEPT_KEYWORDS.

Configuring service files

Defaults should work just fine, but modifications for Incus's service can be made.

OpenRC

Edit /etc/conf.d/incus

systemd

Adding a user to correct groups

Incus can be run as a regular user, or as a root. To have a user capable of launching and operating containers, add the user to the incus group. For example, to add a user named larry into incus group, issue:

There's also incus-admin group for incus admin commands. It's recommended that only root is able to call these, and not regular users.

Configure idmaps

Incus uses idmaps to give non-root users root access inside containers. Easy way to get usable idmaps is to simply issue:

Start the daemons

OpenRC

With incus-user configured

When running incus as a regular user, the incus-user service needs to be started as well as main incus service.

To have the services automatically start at boot:

Without incus-user configured

If only using container through the root user, only incus main service needs to be started.

systemd

With incus-user configured

When running incus as a regular user, the incus-user service needs to be started as well as main incus service.

To have the services automatically start at boot:

Migrating from LXD

With the 'lxd-to-incus' tool

Incus ships with a tool called lxd-to-incus that can be used to convert an existing LXD system to Incus. If using this tool, Incus does not need to be initialized to get it running, but the main service needs to be started. The tool currently works with LXD-5.0 (LTS), LXD-5.18, LXD-5.19 and LXD-5.20.

Due to the way LXD and Incus works, it may be necessary to add users to the incus-admin group to see their LXD containers after migration.

Check that both Incus and LXD works before starting with

lxd-to-incus tool will do a dry-run before applying any changes. In case of errors, no data will be migrated.

After a successful migration, Incus is ready for use!

Manual migration

It's possible to add local lxd image storage as a "remote", with incus remote add then manually incus copy or incus move to a new incus storage pool. Note that the containers need to be stopped for this to work!

Local (single-host) system

In this example both incus and lxd are installed on a single-host system.

First both incus and lxd must be made to allow connections. This operation can be done either-way, so it's possible to add incus as remote and move from local lxd to remote incus, or by adding lxd as remote and moving from remote lxd to local incus.

In this example, LXD's image pool is added as a remote and then LXD container is copied under incus.

LXD's containers and virtual machines should now be visible for incus:

Copy your desired my-important-container from LXD to incus.

Use copy or move. Note that the source container must be stopped!

Disable TLS access when you're done:

Public system

Launching or copying from an already public image hosting system shouldn't change from how you normally operate. Use lxd publish to set your images visible.

Initializing Incus

This needs to be done once after Incus has been installed. Simply issue

to initialize Incus with default settings. This will fit most setups, and can be re-configured through incus admin and incus profile later.

To edit the defaults, issue:

Configuration

Profile

Incus has a top-level profile which the containers can inherit, and which is accessible through

command.

Individual containers can be configured likewise with

Custom profiles can be created and inherited for containers. Example shown in Launching containers.

Network

When initializing Incus with the incus admin init command, it should create working network properties on the fly. Network can be configured manually afterwards, too.

etc. See upstream documentation for more details.

Usage

Finding pre-built container images

Upstream offers variety of pre-built container and virtual machine images under the default images: remote repository. To list all images, please visit website or find the images through incus:

Find a specific distribution:

+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+------------+-------------------------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCHITECTURE | TYPE | SIZE | UPLOAD DATE |
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+------------+-------------------------------+
| gentoo/openrc (3 more) | 224df928674f | yes | Gentoo current amd64 (20231226_16:07) | x86_64 | VIRTUAL-MACHINE | 1056.96MiB | Dec 26, 2023 at 12:00am (UTC) |
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+------------+-------------------------------+
| gentoo/openrc (3 more) | 31864611e02c | yes | Gentoo current amd64 (20231226_16:07) | x86_64 | CONTAINER | 341.64MiB | Dec 26, 2023 at 12:00am (UTC) |
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+------------+-------------------------------+
| gentoo/openrc/arm64 (1 more) | 7bca8c9e9146 | yes | Gentoo current arm64 (20231226_16:07) | aarch64 | CONTAINER | 317.95MiB | Dec 26, 2023 at 12:00am (UTC) |
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+------------+-------------------------------+
| gentoo/systemd (3 more) | 5f36c387c427 | yes | Gentoo current amd64 (20231226_16:07) | x86_64 | CONTAINER | 360.51MiB | Dec 26, 2023 at 12:00am (UTC) |
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+------------+-------------------------------+
| gentoo/systemd (3 more) | 6a8e0e148c35 | yes | Gentoo current amd64 (20231226_16:07) | x86_64 | VIRTUAL-MACHINE | 1093.95MiB | Dec 26, 2023 at 12:00am (UTC) |
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+------------+-------------------------------+
| gentoo/systemd/arm64 (1 more) | 68463fabbf65 | yes | Gentoo current arm64 (20231226_16:07) | aarch64 | CONTAINER | 333.32MiB | Dec 26, 2023 at 12:00am (UTC) |
+-------------------------------+--------------+--------+---------------------------------------+--------------+-----------------+------------+-------------------------------+

A simple grep can also be paired with the listing. The TYPE field will show whether a virtual machine image is available.

Launching containers

To launch a container from upstream image server, check the ALIAS field an issue for example:

which will download, unpack, and start an OpenRC-based Gentoo container. Log in to the container with:

To edit container's config, first stop the container and then edit its config:

If the container can't be stopped, please see Troubleshooting#Containers or virtual machines (OpenRC) not shutting down properly

Find available containers:

Launch a container with a specific, unique profile:

Limiting container resources

Incus can allocate resources to containers in many different ways. Here are only few examples listed. Check upstream documentation for everything: CPU, memory, disk I/O, process etc limits.

CPU

Allow specific container to only use 8 cores.

Or allow it to only use 80 % of CPU capability:

And give it the lowest CPU priority:

Memory

Give the container 8 GB of memory:

Launching virtual machines

Incus can use QEMU to run virtual machines. Running and operating virtual machines requires QEMU to be installed, with the following USE-flags enabled: spice, usbredir, virtfs along with app-cdr/cdrtools for its mkisofs binary.

app-emulation/qemu spice usbredir virtfs

Note: You might also need to disable pin-upstream-blobs USE flag from Qemu to allow installation of newer sys-firmware/edk2-bin.

# incus vm support
app-cdr/cdrtools
app-emulation/qemu

The following kernel options are needed: CONFIG_MACVTAP, CONFIG_VHOST_VSOCK, CONFIG_KVM and either CONFIG_KVM_AMD or CONFIG_KVM_INTEL depending on the system's CPU. Please see QEMU#Kernel for more accurate config options. You'll also need to enable virtualization in your BIOS, otherwise you'll get an error message of "KVM: disabled by BIOS". Basically, make sure /dev/kvm exists before trying to launch a virtual machine. And setup QEMU properly so it works.

For graphical sessions to work in a virtual machine, i.e. logging to a desktop, either app-emulation/virt-viewer or net-misc/spice-gtk needs to be installed.

The default image server already hosts many virtual machine alternatives, even pre-configured desktop VM images. When listing upstream images through

as shown before, check the TYPE field that a virtual machine is available. Many pre-configured desktop images can be found with a "desktop" in their description.

Virtual machines can be launched in a similar manner to containers, just append the --vm flag to the command:

If everything is working properly, the VM will be started upon launching it. If not, check Troubleshooting.

Access the shell similarly with

A graphical virtual machine session can be opened with

Note that it requires a GUI-capable program to be installed. See options above.

Limiting virtual machine resources

Allocating CPU and memory resources for VMs works the same way as for containers. Please see Limiting container resources.

Growing default disk size

Simply the following should work when the VM is off:

These profile settings can be used to create VMs with bigger disk sizes. Make sure the profile is inherited by the VM in question.

config: {}
description: VM Profile for a bigger root disk
devices:
 root:
 path: /
 pool: default
 size: 50GB
 type: disk

...
profiles:
- default
- bigger-vm-disk
...

If it does not, the manual method below is a more sureway to get it done:

On the host resize image with qemu-img.

Now move on to the virtual machine.

Red Hat / CentOS variants

Due to RHEL/CentOS based distributions not shipping the 9p driver, workarounds are needed to launch their virtual machines.

A profile can be created with this config pre-defined, then vm's launched using that profile:

Making custom container images

Distrobuilder can be used to make a custom container or virtual machine images. Please see the wiki page for guidance.

Advanced usage

Container backups

TODO

Move/copy containers between projects

Let's say you later decided to start using Incus by utilizing the incus-user approach, but all your containers are migrated/created with incus-admin privileges. You can move/copy these containers under your user's profile. First make sure incus-user is initialized, and working properly on your user. Then identify the projects:

+-------------------+--------+----------+-----------------+-----------------+----------+---------------+---------------------------------------------+---------+
| NAME | IMAGES | PROFILES | STORAGE VOLUMES | STORAGE BUCKETS | NETWORKS | NETWORK ZONES | DESCRIPTION | USED BY |
+-------------------+--------+----------+-----------------+-----------------+----------+---------------+---------------------------------------------+---------+
| default (current) | YES | YES | YES | YES | YES | YES | Default Incus project | 10 |
+-------------------+--------+----------+-----------------+-----------------+----------+---------------+---------------------------------------------+---------+
| user-1000 | YES | YES | YES | YES | NO | YES | User restricted project for "larry" (1000) | 4 |
+-------------------+--------+----------+-----------------+-----------------+----------+---------------+---------------------------------------------+---------+

Check the container names:

Move or copy the desired containers under your user's project, therefore giving your user the full access to the moved/copied containers:

Fix any errors if there are some. After successful operation, the container should be visible in your user's project:

Multi-host setup

TODO

Pulseaudio in a container

TODO

Sharing files/directories between container and host

With incus-user your project should already have default idmaps solved. If it does not, add the following config either to your individual containers, or to the profile you're using.

config:
 raw.idmap: |-
 uid 1000 1000
 gid 1000 1000

Where the uid and gid values match your user's id -u and id -g values. These need to be present in hosts /etc/subuid and /etc/subgid files too:

root:1000000:1000000000
incus:1000000:1000000000
root:1000:1
incus:1000:1

root:1000000:1000000000
incus:1000000:1000000000
root:1000:1
incus:1000:1

Share directories with:

devices:
 containerscripts:
 path: /root/bin
 source: /home/larry/bin/containerscripts
 type: disk
 distfiles:
 path: /var/cache/distfiles/
 source: /var/cache/distfiles/
 type: disk

If you get a message saying "Disk source path not allowed", please check the Troubleshooting section for it.

systemd containers on OpenRC host

First openrc must be configured to be in hybrid or unified mode.

rc_cgroup_mode="hybrid"

rc_cgroup_controllers="yes"
rc_controller_cgroups="yes"

Reboot is a must if you made changes here.

Incus's default init.d file should already create the required /sys/fs mounts,

...
install-d/sys/fs/cgroup/systemd--groupincus-admin--ownerroot
mount-tcgroup-onone,name=systemdsystemd/sys/fs/cgroup/systemd
...

X and Wayland apps in a container

For X applications, edit the container config with:

config:
 environment.DISPLAY: :0.0
...
devices:
 X0:
 path: /tmp/.X11-unix
 source: /tmp/.X11-unix
 type: disk
 mygpu:
 gid: "1000"
 type: gpu
 uid: "1000"

Where environment.DISPLAY is the value of your hosts $DISPLAY env value. gid and uid should match your id -g and id -u respectively. With Xorg you'll also need to allow connections from container by using xhost +local:.

Wayland is a bit trickier. Many programs will disallow running wayland as root, therefore you'll need to create a new user to the container, and map that user's subgid and subid in your host.

Note: the id (1000) should match your user's id -g and id -u values.

Log in to the container to create the new user, then logout:

Edit your container, or profile with:

 mygpu:
 gid: "1000"
 type: gpu
 uid: "1000"
 xdgruntimedir:
 path: /home/gui/.xdg_runtime_dir
 source: /run/user/1000
 type: disk

Where the id, gid and /run/user/1000 needs to match your hosts user's id -g and id -u.

Start your container:

export XDG_RUNTIME_DIR=/home/gui/.xdg_runtime_dir
export WAYLAND_DISPLAY=wayland-0
export QT_QPA_PLATFORM=wayland
export DISPLAY=:0.0

Now inside your container the gui user can launch X or wayland applications. On Wayland host, Xwayland :0.0 needs to be launched before X applications can work from containers. See also Xwayland -help for more parameters, like -geometry. Alternatively, try Xephyr: Xephyr -br -ac -noreset -screen 1920x1080 :0.0

Troubleshooting

First things to do when facing problems is to check the containers log, incus's log and system log.

Attach immediately to systart:

By default incus's logs can be found from /var/log/incus/ - this can also be modified from conf.d and service files.

Upstream discussion forum is a great resource for finding answers to many different kind of issues.

Try also calling:

to find out any obvious missing kernel configuration bits. Installing sys-kernel/gentoo-kernel-bin and booting into that, to rule out any kernel-related errors could help.

Adding a disk device with incus-user fails to "Disk source path not allowed"

With incus-user by default the disk access is restricted. Check if this is the cause:

And if yes, unset or edit it:

Containers or virtual machines (OpenRC) not shutting down properly

If the container freezes during the stop command with

while using OpenRC, try to turn it off directly with poweroff:

If that works, edit the /etc/inittab file in the container adding following part:

pf:12345:powerwait:/sbin/halt

Shutdown the container with poweroff, and next time it is booted, the container should work like normal and incus stop should work as expected. Be careful when doing world updates to not blindly merge changes to /etc/inittab.

Fresh containers fail to start with a cryptic "newuidmap failed to write mapping" message

This is an unfortunate and annoying bug when using incus-user. The easiest and fastest way to "fix" it is to issue incus profile unset default raw.idmap and re-launching the container with a new name, since idmaps are issued when the container is spawned. Make sure to fix the profile you're using, or make sure the new container will inherit the default profile.

If raw.idmaps are desired, another way is to carefully observe the numerical idmaps in the error message, and allow them manually in /etc/lxc/default, /etc/subuid and /etc/subgid files. For example, these could look like:

lxc.idmap=u 0 100000 1000
lxc.idmap=g 0 100000 1000
lxc.idmap=u 1000 1000 1
lxc.idmap=g 1000 1000 1
lxc.idmap=u 1001 101001 0
lxc.idmap=g 1001 101001 0
lxc.idmap=u 1001 1001 1
lxc.idmap=g 1001 1001 1
lxc.idmap=u 1002 101002 0
lxc.idmap=g 1002 101002 0
lxc.idmap=u 1002 1002 1
lxc.idmap=g 1002 1002 1
lxc.idmap=u 1003 101003 0
lxc.idmap=g 1003 101003 0
lxc.idmap=u 1003 1003 1
lxc.idmap=g 1003 1003 1

root:1000:1
root:1001:1
root:1002:1
root:1003:1
root:100000:65536
incus:100000:65536

root:1000:1
root:1001:1
root:1002:1
root:1003:1
root:100000:65536
incus:100000:65536

Various dnsmasq related issues

Can't create new network bridge incusbr0

Most likely wrongly configured /etc/dnsmasq.conf - check the file's configuration, or create the network bridge manually:

incus-user

When opting to use incus-user, if the user can't use incus command it may be related to dnsmasq permissions. Add the user to dnsmasq group:

Relog/Reboot, just make sure the user is in the group!

... incus dnsmasq ...

IPv6

When primarily using ipv6, make sure net-dns/dnsmasq is built with the +ipv6 use flag.

Launching containers fails with "dnsmasq: bad command line options: bad dhcp-range"

Most likely wrongly configured /etc/dnsmasq.conf - check the file's configuration.

bind-interfaces
except-interface=incusbr0
dhcp-range=192.168.100.1,192.168.100.100,infinite

See also

• Incus/Gentoo Github pullrequest testing — easy and automated way for testing ebuild contributions via Gentoo's Github mirror that's based on
• LXD — a system container manager (original project, of which Incus is a fork)
• Virtualization — the concept and technique that permits running software in an environment separate from a computer operating system.

External resources

• First Steps with Incus.
• Incus's upstream discussion site, support forums etc.
• Youtube channel focusing on using and developing Incus.