Running Docker containers with Proxmox VE (and LXC)

Mon, 07 Nov 2022 06:30:00 +0200

Introduction

I found several tutorials to install Docker (engine) inside an LXC container under Proxmox VE to then run Docker containers inside it. The problem is that you have to log into the LXC container before you can interact with your Docker containers.

You can also install Docker directly on Proxmox VE but it’s not really recommended and you don’t have the ease of use of LXC with Proxmox GUI.

And actually, it turns out you can run Docker images as the OS of an LXC container and thus have Docker controlled by Proxmox. So we won’t have 100% identical behavior to if you had done a docker run on your machine.

However, we’ll gain in exchange a container totally identical to your other LXC containers in your cluster (with all associated functions) but launched with a Docker image.

But first I suggest we go back a bit before showing you how…

Context

You know, I’ve been doing Proxmox VE for a while. Proxmox VE is a great turnkey and production-ready distribution for server virtualization (it’s even hyperconverged if you enable the Ceph part).

There are lots of little things I like about Proxmox VE, and one of them is that we can create Linux containers with LXC rather than QEMU virtual machines. I wrote an article to talk about it.

In use (except for specific kernel configuration), the LXC container behaves strictly identically to a VM in Proxmox, but for a fraction of the resources! We have an OS, we install applications on it, like a VM.

An nginx frontend for several applications. The complete LXC container only consumes a few MB

Admittedly it’s much less isolated than a real VM (because it’s a container, we’re isolated from other processes, but we run on the hypervisor’s kernel directly).

But with so few resource needs, I can host many small applications in different contexts with very small physical machines (Atom 4GB RAM at €6 per month at OneProviders) which would be impossible with a real VM.

It’s good, but not enough

However, LXC is not the most known/hyped containerization technology. For years on the forum, whenever people ask for Docker support in Proxmox VE, they get sent away. Not always very nicely either…

Yet, there are several reasons to want to run Docker containers on your Proxmox infrastructure:

First, Docker isn’t as much stateless and/or ephemeral workloads as we think. Many containerized applications would have their place on a virtualization cluster.
Then because many software vendors have become lazy. Some no longer manage installation processes (or only on one OS) and just provide a Docker image that they maintain up to date.

So we have to settle for VMs in Proxmox to run Docker containers (see my point at the beginning).

LXC supports OCI images

Plot twist. A few days ago, I discovered that LXC, Proxmox VE’s containerization engine, was compatible with the OCI format.

buzzwrd.me - Creating LXC containers from docker and OCI images

For those who don’t know OCI, it’s an organization aiming to create a standard to unify the way containers are stored.

The Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes. opencontainers.org/

And Docker respects this format: we can therefore theoretically run from LXC containers whose base image is a Docker image.

So, when do we start?

Disclaimer: what follows is tinkering. Nothing is supported and I don’t recommend it in production. There are also likely limitations (especially regarding storage).

First, some dependencies are missing for LXC’s OCI function to be usable under Proxmox VE:

sudo apt install skopeo umoci jq

Then, by default, LXC containers attach to a Linux bridge called lxcbr0. However this bridge probably doesn’t exist on your Proxmox VE installation (not installed by default).

In quick and dirty, we can just replace the default name, although ideally we should find the flag in the CLI to change the bridge. Depending on how you configured the network in your Proxmox VE server, you’ll probably use vmbr0 or vmbr1:

sed -i 's/lxcbr0/vmbr1/g' /etc/lxc/default.conf

Now that we have everything, we can run our LXC containers from docker images. Here I’m running the official alpine:latest image, but I could have taken any other:

lxc-create 500 -t oci -- --url docker://alpine:latest

I created a container called “500” (because VMs and LXC containers in proxmox VE are called with numbers). The image is downloaded then the container created:

We can now start it (with lxc-execute we run the container and open a prompt directly inside it, but we also have lxc-start and lxc-attach)

lxc-execute 500

As is, Proxmox VE doesn’t know our container exists and it won’t appear in the UI. But it works ;-).

Bonus

We can even cheat and make Proxmox believe it’s an LXC container it created itself by creating a 500.conf file, in the /etc/pve/lxc/ folder

The only mandatory fields are the following:

arch (amd64 unless you’re on ARM)
cores (the CPU limit to impose on the container)
hostname
memory (the RAM limit to impose on the container)
ostype
rootfs

Once the file is created, the container automatically appears in the host’s VM list. We can start it and it works :)

The container can be started, stopped, etc. All from the Proxmox VE GUI :)

Have fun!

QEMU on Zwindler's Reflection