System Containers

In my last few blog posts I’ve talked about my setup for hosting my blog, gitea and CI/CD servers (that last one’s still in the works, I’m still debating moving to gitlab… look it’s hard to make decisions sometimes, okay?) using Docker.

Recently, I’ve been getting into using full system containers (as opposed to Docker’s application containers) in my lab. In this post I will be talking about incus, its advantages and disadvantages.

Why Incus?

First, let’s talk about what Docker isn’t. Docker is great for running single applications in isolated, ephemeral containers. But sometimes, I just want a full Linux system. I want to apt install things. I want to run a VPN, a web server, and a GUI application like Xorg, all in separate environments that don’t interfere with each other. Docker Compose files are nice until they’re not — you start piling on services, overriding entrypoints, binding directories in weird ways, and suddenly your compose.yaml looks like a plate of spaghetti.

Incus fixes this by giving me system containers—essentially, lightweight VMs that share the host’s kernel but have their own separate userspace. Each container is a full Linux environment. You can run systemd, openSSH, a firewall, even a desktop environment if you’re feeling spicy. But unlike a VM, there’s little to no overhead. (a bare ubuntu 24.10 system container eats about 18 megabytes of RAM. Sure, still a lot more than docker containers, and potentially not-insignificant for microservice architectures built to run thousands of containers, but if you treat them as VMs instead of per-app containers then this can be quite nice) And system containers still carry most of the other benefits of docker containers - namely, reproducability. Better yet, if you do treat them like VMs, they can be snapshotted, cloned, created, published and destroyed very efficiently and very easily.

You can also use BTRFS subvolumes as your storage backend, which is awesome on its own (compared to docker’s overlayfs at least) - for all the reasons that BTRFS is generally awesome, and more. Storage and files can be efficiently de-duplicated - several system containers running from the same image will share the underlying storage for their system files. Since each container’s file system is simply a separate BTRFS subvolume, they can also be independantly copied/snapshotted even from outside the incus command-line tool. Compared to a full VM, the storage cost is very, very little.

As for why incus over LXD, well… there really isn’t much reason. They’re basically the same thing, LXD is canonical’s project, whereas incus is a continued fork of LXD. I personally tried both, and decided that I prefer incus.

Also, if you’d like to still have application containers alongside system containers, you can have that: incus (unlike lxd) can run OCI images. Although I should mention that I did not test them out very much - I do not know how incus compares to docker in this regard. Still, if you’d really like, there’s absolutely nothing stopping you from having both!

Additionally, if you’d like to run linux VMs from the command-line, incus (and lxd) provides a uniform command line interface for them. You can run an image (e.g. ubuntu/24.10) as a container with incus launch images:ubuntu/24.10 <container-name>, or you can add the --vm flag to run it as a virtual machine for stronger isolation: incus launch --vm images:ubuntu/24.10 <container-name>. Personally the uniform interface is what amazes me - I’ve found it to be much more of a hassle to use the cli to launch VM’s using virt-sh for example. incus’ image repository is also very nice - pretty much any linux distribution can be found, and installed within minutes (if you’re downloading the image for the first time - after that it’s pretty much instant)

Finally, the biggest reason for me to love incus was because it’s simply far too useful as a lab environment. Think about it, how many times did you come across some really nice looking piece of software, but you simply weren’t sure if you would actually like it, and you didn’t want to install all those dependancies only for it to end up… subpar? For things that are in your distribution’s package manager this may be okay, you can efficiently get rid of all unused dependancies, but what about things with custom installers? Who’s to say what those shell scripts are doing anyway, maybe they’re secretly stealing all your data! Docker generally helps with this as well - but not always. There are plenty of docker images that require you to use some custom docker compose file that mounts the host system’s docker socket into a container to run them, or many more that require you to mount some important directory, or simply require you to run them as priviliged containers…

Then there are issues where your favourite linux distro simply doesn’t have the tools necessary for certain things - maybe a package is missing, maybe your distro packages an older version of something important - and now you’d have to switch distros (unreasonable) or build from source (reasonable, but still annoying).

With incus, I can spin up a system container and test things out however I want. There is no personal information to be harvested in the container, and dependancies are cleanly handled by the container itself. If it turns out I don’t like the program - easy, just delete the container and it’s as if I never installed anything. If I like it, depending on the software, I may not even need to install on the host. Sometimes running in a container is perfectly adequate - self-hosted server software for example, or perhaps a pentesting container.

I tried this recently with coder, as the idea of a self-hosted cloud development environment piqued my interest. I tried it out in an up-to-date ubuntu container (running the shell script with relatively less fear compared to just running it as root on my host system) to see what it’s like. I tried it for a while, creating and deleting workspaces, trying to get used to the VS code browser version, etc. I finally decided that it wasn’t really my cup of tea, and simply deleted the container.

If I had decided to keep it, it would be trivial to - the system inside the container has its own systemd etc, so as long as the service is enabled it will start and stop along with the host.

It’s amazing to me that I could perform this type of experimentation even directly on a production server - each incus container is separate from each other, so if something goes wrong I can restart/rebuild that particular container/vm from scratch. I can update one container and restart it without affecting others. I can even create new containers on a whim and experiment however I’d like - even if I screw up, I can always start from a blank slate. This kind of freedom provides (even if purely psychological) the feeling of safety that I’ve really grown to like. The fact that these “system containers” act more like virtual machines rather than singular applications is, I believe, a strong part of this.

Basics of Incus

If you’ve used Docker, you’ll pick up Incus quickly. The main difference is that instead of thinking in terms of individual applications or services, you’re managing entire systems.

Effectively, you can break up your desired services into several different “hosts”, and manage them as you would VMs. (of course, they can absolutely be actual VMs)

For those looking for Infrastructure-as-Code cloud deployments, incus supplies its own terraform provider “incus” as well. For even heavier, “I need a million identical containers running on a thousand servers across the globe” kinds of needs, incus is probably not the option you need - I honestly can not provide an accurate view here.

Key incus commands

Here’s the cheat sheet:

# Launch an Ubuntu 22.04 container named "blog"  
incus launch images:ubuntu/22.04 blog  

# Launch the same system as above, but as a full Virtual Machine
incus launch --vm images:ubuntu/22.04 blog

# Shell into the container
incus exec blog -- bash  

# Forward port 80 from the host to the container’s port 80  
incus config device add blog myport80 proxy listen=tcp:0.0.0.0:80 connect=tcp:127.0.0.1:80  

# Take a snapshot (because you WILL break things, let's be honest here)
incus snapshot create blog pre-experiment  

# List all containers
incus list
incus ls

# Turn a container into a template (clone it forever)  
incus publish blog --alias blog-base  

It’s that simple. No Dockerfiles. No docker run –privileged hacks. Just… a Linux machine, living inside your machine.

Well, okay, it’s actually slightly different from a full virtual machine, it’s actually sharing the same kernel. Apart from that, however, it acts very convincingly as a lightweight virtual machine. That’s pretty elegant, I think, and very useful for all the reasons I already mentioned.

Conclusion

Incus (and LXC) have given me the best of both worlds: the isolation of VMs and the lightness of containers. Docker is still my go-to for app-centric workflows (shipping a Python script? Sure, Dockerize it). But for hosting services, experimenting, or just having a clean Linux environment to destroy? Incus feels so nice.

So if you’re interested, give Incus a try. Spin up a container, apt install something stupid, wreck your system a couple times and remember what it’s like to have fun!

Thanks for reading.