LXC on Zwindler's Reflection

OCI Artifacts as LXC Machine Base on Proxmox 9.1, Finally?

Mon, 24 Nov 2025 18:00:00 +0200

Context

A few days ago, the Proxmox VE dev team announced a new version (9.1) whose major new feature is the addition of OCI artifacts support as a base for creating LXC containers.

A long-awaited feature, since in theory it finally opens the door to launching “Docker”-type containers in Proxmox, something the Proxmox devs had always refused.

That’s also why one of my blog articles gets the most traffic on Google (sniff, soon the end of glory):

This article where I explain how to launch Docker containers (via LXC) in Proxmox VE (normally not possible)

Create LXC containers from OCI images

Proxmox VE 9.1 integrates support for Open Container Initiative (OCI) images, a standard format for container distribution. Users can now download widely-adopted OCI images directly from registries or upload them manually to use as templates for LXC containers. Depending on the image, these containers are provisioned as full system containers or lean application containers.

Let’s Test? First the Prerequisites

So, I’m curious, how does it work?

First, you need to pre-download the image you’re interested in to your storage (one that can host “CT Templates”).

Fun thing, there’s a button called “Query tags”, which I imagine retrieves a list of tags. Well, it’s not starting well, it doesn’t work (at least not on docker.io…).

command 'skopeo list-tags docker://docker.io/zwindler/prime-calculator' failed: exit code 1 (500)

EDIT: since then, it works again, on all my nodes. Perhaps a temporary issue.

Note: skopeo is precisely one of the dependencies used in my “proxmox+docker hack” article that allows manipulating (mainly downloading) containers from remote registries.

Creating the LXC Container

Now that the template (the OCI image) is available on our storage, we can create an LXC container as usual. Click on “Create CT”, give a CT ID, a password:

For the template, we choose our OCI artifact:

And validate. Proxmox should create our container:

Even though the terminal tells us that application containers may not work well in the JS console, for all the containers I managed to start (see below), I always had a functional console.

So the bet seems rather successful.

Example with docker.io/python:3.11.14-trixie

When It Wasn’t Working (Yet)

I had 2 issues on a Proxmox VE 9.1 (that had been upgraded and tinkered with a bit, in short, a machine that had seen some use) starting an LXC container on OCI base with Proxmox VE 9.1 because I ran into 2 bugs.

I didn’t reproduce these bugs on a “fresh” install.

First Bug: “from scratch” Image

First, my prime-calculator image wasn’t even created.

TASK ERROR: unable to create CT 106 - Error while extracting OCI image: Unknown layer digest sha256:2bc4d17e1eb16f6b52acbafeb6d86c8f3eaf9588a5e2a7a245b6bb1d85e93396 found in rootfs.diff_ids: Unknown layer digest sha256:2bc4d17e1eb16f6b52acbafeb6d86c8f3eaf9588a5e2a7a245b6bb1d85e93396 found in rootfs.diff_ids

OK, fair enough. My image is a binary “from scratch”, with nothing but the binary, and maybe I made a mistake somewhere…

Still doesn’t work.

Second Bug: Unable to Start

However, even with known images, the container is properly created:

But impossible to start it:

problem with monitor socket, but continuing anyway: got timeout
TASK ERROR: unable to get PID for CT 105 (not running?)

Same error with command line:

pct start 105 --debug
problem with monitor socket, but continuing anyway: got timeout
lxc_start_main: 257 Container is already running
- Read uid map: type u nsid 0 hostid 100000 range 65536
INFO confile - ../src/lxc/confile.c:set_config_idmaps:2295 - Read uid map: type g nsid 0 hostid 100000 range 65536
ERROR lxc_start - ../src/lxc/tools/lxc_start.c:lxc_start_main:257 - Container is already running
unable to get PID for CT 105 (not running?)

After (laborious) investigation, it turns out it’s an AppArmor problem. I don’t know if I caused it with my tinkering or if it’s a more generalized bug for now (probably option 1, but 2 isn’t impossible either…)

Nov 23 10:37:26 node01 systemd[1]: Started pve-container-debug@105.service - PVE LXC Container: 151.
Nov 23 10:37:27 node01 kernel: audit: type=1400 audit(1764067047.008:175): apparmor="DENIED" operation="create" class="net"
Nov 23 10:37:27 node01 kernel: audit: type=1400 audit(1764067047.008:176): apparmor="DENIED" operation="create" class="net"
Nov 23 10:37:27 node01 systemd[1]: pve-container-debug@105.service: Deactivated successfully.
Nov 23 10:37:37 node01 pct[21369]: unable to get PID for CT 105 (not running?)

The temporary (and dirty) workaround is to disable the AppArmor profile for lxc-start, and remove any zombie files/processes.

apparmor_parser -R /etc/apparmor.d/usr.bin.lxc-start

ps aux | grep "lxc-start" | grep "105" | awk '{print $2}' | xargs -r kill -9
rm -f /var/lib/lxc/105/config.lock
rm -f /run/lxc/lock/var/lib/lxc/105

Note: You can also disable it permanently, but that’s obviously something you shouldn’t do.

From there, the container is accessible, sometimes in console (depends on the container)

Conclusion

Apparently, the problems I encountered don’t reproduce on a “fresh install”. So it’s probably an issue related to my “old” machines.

As a bonus, another article in English that made me persevere when I understood it worked for some:

Raymii.org blog - Finally, run Docker containers natively in Proxmox 9.1 (OCI images)

Note: this article helped me rediscover the pct enter CTID command that allows you to log into the container.

In the meantime, you still have my hack ;-)

Turning Proxmox VE into a Kubernetes Node with LXC and lxcri

Mon, 17 Nov 2025 18:00:00 +0200

He didn’t know it was completely stupid, so he did it

What a lie… Of course, I know perfectly well it’s dumb. Which is precisely why this idea is irresistible.

If you follow the blog, you know I use Proxmox VE extensively (the articles that attract the most traffic on the blog are actually about this virtualization technology).

Proxmox VE is pretty cool; you can run VMs (QEMU) with it, but also, if you don’t have VT-x or want “lightweight VMs” (a term we’ve overused), you can install complete OSes in containers with LXC.

But beyond these two solutions, the Proxmox VE project devs are quite rigid (and not just about that). That’s why I’ve posted a few “hacks” to work around Proxmox’s limitations, notably this article where I explain how to run Docker containers (via LXC) in Proxmox VE (normally not possible).

What if we went even further with the stupid hack?

What if we not only ran OCI containers in Proxmox VE, but ALSO transformed our Proxmox VE hosts into ✨️ Kubernetes Nodes ✨️ ???

How do we do this?

The goal is obviously not to install Kubernetes alongside Proxmox VE, but to reuse as much as possible. Typically, I’m going to reuse the virtualization technology.

With Proxmox VE supporting 2 different virtualization technologies (qemu and LXC), you have to choose. And since I like LXC and have already experimented with the Docker => LXC hack on Proxmox VE, the question was quickly answered.

The entire difficulty of this exercise is finding how to make LXC “CRI-compatible”. And we’re lucky, Linux Containers, the foundation that oversees LXC, Incus (ex LXD, “closed” by Canonical), wrote a CRI for LXC called lxcri a few years ago. And of course, as it’s a project of questionable utility, the thing has been unmaintained since 2021.

I had actually tried to use it, spent quite a bit of time on it in February 2024 (I faced bug after bug, went through forks, …) to fail miserably on a compatibility problem with my version of LXC (5+ on Proxmox 8, 6 on Proxmox 9), among other bugs.

lxcri is a wrapper around LXC which can be used as a drop-in container runtime replacement for use by CRI-O.

So we’re going to need 3 additional things on our Proxmox server for my dumb idea to work:

lxcri as the low-level container runtime
cri-o as the high-level container runtime
kubelet to control the runtime and communicate with the control plane

Let’s go

OK. So lxcri relies on cri_o, Redhat’s container runtime. Let’s start by installing cri-o:

https://github.com/cri-o/cri-o/blob/main/install.md

Note: my Proxmox VE 8 server back then was using Debian 12 as its base OS. So we should base ourselves on that for the $OS variable (as indicated in the docs). However, when I first tested, the CRI-O repositories were being migrated, with outdated documentation and missing releases. The repo download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o didn’t have the Debian_12 PATH, nor Kubernetes version 1.29… I was SO pissed.

We shouldn’t have any issues now:

export KUBERNETES_VERSION=v1.32
export CRIO_VERSION=v1.32

# create a directory for keyrings (it doesn't always exist on a fresh install)
sudo mkdir -p /usr/share/keyrings

# kubernetes repo
curl -fsSL https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/ /" |
 sudo tee /etc/apt/sources.list.d/kubernetes.list

# crio repo
curl -fsSL https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/deb/Release.key |
 sudo gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/deb/ /" |
 sudo tee /etc/apt/sources.list.d/cri-o.list

sudo apt update
sudo apt install cri-o

From here, we have cri-o. We could configure it, but if we do that, it will use runc and we wouldn’t be using LXC as a runtime.

That’s not the point of this hack!

So at this point, we switch back to the lxcri documentation:

github.com/lxc/lxcri

There’s no release

That’s right… we need to first build the lxcri binary and place it in the /usr/local of our Proxmox VE server. There’s no precompiled binary in the project, it’s an issue that was opened just before it was abandoned 😬😬.

And to make it even more fun, the build process goes through a Dockerfile, which is funny since we don’t have Docker on our proxmox node…

So I tried to build lxcri from a machine with docker, following the command shown on Github. Boom. (First, there’s a missing “.” at the end of the docker build command in the docs) it fails miserably at compilation…

Note: I no longer have that error, probably a dependency problem. It’s annoying because we’ll have to do lots of things manually… But don’t bother with git clone, we’re going to use a fork (of a fork).

Looking at the Dockerfile, we quickly realize we’re just running a script (install.sh)…

When we call docker with the buildarg installcmd=install_runtime, we run the install_runtime function, which calls install_runtime_noclean, which runs add_lxc then add_lxcri.

Me just want to compile lxc (for the bindings) and lxcri. So I’ll do it manually. For that, we need golang 1.16 (it’s old…).

There’s quite a bit of broken code everywhere and several open issues where maintainers recommend going with a fork:

https://github.com/drachenfels-de/lxcri

I spent quite a bit of time debugging it, and in the end I made my own fork (https://github.com/zwindler/lxcri), which requires golang 1.22+ and adds a bunch of fixes.

Prerequisites for building lxcri

Let’s install a bunch of stuff:

sudo apt update
sudo apt install curl git meson pkg-config cmake libdbus-1-dev docbook2x

Let’s install golang if we don’t have it (unlikely you have Golang on a Proxmox VE node):

wget https://go.dev/dl/go1.25.4.linux-amd64.tar.gz
sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.25.4.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
go version
 go version go1.25.4 linux/amd64

I’ll also need to fetch and compile lxc. For a while, I was stuck on version 4.0.12 (very precisely) because the 4.0.6 found on Debian 11 does not work with the lxcri code (I hit lots of bugs).

But the good news is that since I’m a try-harder from space, with all the fixes on my fork, my fork works with the latest version of lxc (6.x), which is convenient because that’s the version on my updated Proxmox VE 9.

git clone https://github.com/lxc/lxc
cd lxc

meson setup -Dprefix=/usr -Dsystemd-unitdir=PATH build
 Build targets in project: 30

 lxc 6.0.0

 User defined options
 prefix : /usr
 systemd-unitdir: PATH

 Found ninja-1.12.1 at /usr/bin/ninja

meson compile -C build
 INFO: autodetecting backend as ninja
 INFO: calculating backend command to run: /usr/bin/ninja -C /root/lxc/build
 ninja: Entering directory `/root/lxc/build'
 [544/544] Linking target src/lxc/tools/lxc-monitor

It built lots of things, that’s cool. But the files I’m interested in are here:

find . -name "lxc.pc"
./build/meson-private/lxc.pc

ls build/*lxc*
build/liblxc.so build/liblxc.so.1 build/liblxc.so.1.8.0 build/lxc.spec

build/liblxc.so.1.8.0.p:
liblxc.so.1.8.0.symbols

I put this in the right places on my Proxmox VE with:

sudo make install
sudo ldconfig

Build like never before

OK, time to play with my fork:

git clone https://github.com/zwindler/lxcri.git lxcri.zwindler
cd lxcri.zwindler

make build
 go build -ldflags '-X main.version=8805687-dirty -X github.com/lxc/lxcri.defaultLibexecDir=/usr/local/libexec/lxcri' -o lxcri ./cmd/lxcri
 cc -Werror -Wpedantic -o lxcri-start cmd/lxcri-start/lxcri-start.c $(pkg-config --libs --cflags lxc)
 CGO_ENABLED=0 go build -o lxcri-init ./cmd/lxcri-init
 # this is paranoia - but ensure it is statically compiled
 ! ldd lxcri-init 2>/dev/null
 go build -o lxcri-hook ./cmd/lxcri-hook
 go build -o lxcri-hook-builtin ./cmd/lxcri-hook-builtin

ls -alrt
 total 15288
 [...]
 -rwxr-xr-x 1 debian debian 7108288 Nov 16 20:06 lxcri
 -rwxr-xr-x 1 debian debian 17520 Nov 16 20:06 lxcri-start
 -rwxr-xr-x 1 debian debian 2942584 Nov 16 20:06 lxcri-init
 -rwxr-xr-x 1 debian debian 2834743 Nov 16 20:06 lxcri-hook
 -rwxr-xr-x 1 debian debian 2519097 Nov 16 20:06 lxcri-hook-builtin

If we were on a dev machine, we could send the binaries to the Proxmox VE server (lxcri in /usr/local/bin, the rest in /usr/local/libexec/lxcri)

In my case, I’m directly on the machine that builds and runs, so I do:

$ sudo make install
 mkdir -p /usr/local/bin
 cp -v lxcri /usr/local/bin
 'lxcri' -> '/usr/local/bin/lxcri'
 mkdir -p /usr/local/libexec/lxcri
 cp -v lxcri-start lxcri-init lxcri-hook lxcri-hook-builtin /usr/local/libexec/lxcri
 'lxcri-start' -> '/usr/local/libexec/lxcri/lxcri-start'
 'lxcri-init' -> '/usr/local/libexec/lxcri/lxcri-init'
 'lxcri-hook' -> '/usr/local/libexec/lxcri/lxcri-hook'
 'lxcri-hook-builtin' -> '/usr/local/libexec/lxcri/lxcri-hook-builtin'

and we can continue:

$ /usr/local/bin/lxcri help
NAME:
 lxcri - lxcri is a OCI compliant runtime wrapper for lxc

USAGE:
 lxcri [global options] command [command options] [arguments...]

VERSION:
 8805687
[...]

Back to crio

OK, we’ve built lxcri and have all the dependencies to use it. So we can go back to the official lxcri doc “setup.md” with the idea of configuring CRI-O, so it doesn’t use runc, only lxcri.

sudo tee /etc/crio/crio.conf.d/10-crio.conf > /dev/null <<'EOF'
[crio.image]
signature_policy = "/etc/crio/policy.json"

[crio.runtime]
default_runtime = "lxcri"

[crio.runtime.runtimes.lxcri]
runtime_path = "/usr/local/bin/lxcri"
runtime_type = "oci"
runtime_root = "/var/lib/lxc" #proxmox lxc folder
inherit_default_runtime = false
runtime_config_path = ""
container_min_memory = ""
monitor_path = "/usr/libexec/crio/conmon"
monitor_cgroup = "system.slice"
monitor_exec_cgroup = ""
privileged_without_host_devices = false
EOF

Note, the setup.md installation doc tells us to generate a clean config with the crio binary and the config command, but it doesn’t really work and we end up launching runc or crun unintentionally. I overwrite everything, it’s simpler.

And now, we can start crio:

sudo systemctl enable crio && sudo systemctl start crio

From here, we have a server with a presumably functional CRI.

systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
 Loaded: loaded (/lib/systemd/system/crio.service; enabled; vendor preset: enabled)
 Active: active (running) since Mon 2025-11-17 12:02:37 UTC; 12s ago

journalctl -u crio
Nov 17 20:29:43 instance-2025-11-16-15-31-55 systemd[1]: Starting Container Runtime Interface for OCI (CRI-O)...
[...]
Nov 17 20:29:43 instance-2025-11-16-15-31-55 crio[11906]: time="2025-11-17T20:29:43.61758425Z" level=info msg="Using runtime handler lxcri version 8805687"

Yeeees :D

So we can install kubelet, then add it to an existing Kubernetes cluster

Fast forward

I was too lazy to set up a proper cluster with kubeadm or other, and then enroll a node manually with the token+sha. I could have also replayed with my funny PoC of Clever cloud’s “linux” workers to create a control plane at Clever, then manually enroll with the node bootstrap token (I did things pretty well in that PoC).

So I replayed roughly with my demystifions-kubernetes project, which allows you to set up a single-node cluster manually by just running binaries.

Once the control plane is bootstrapped (etcd, api-server, controller-manager, scheduler), we stop BEFORE the containerd part (we’ve already configured cri-o) and manually start kubelet:

sudo bin/kubelet --kubeconfig admin.conf --container-runtime-endpoint --container-runtime-endpoint=unix:///var/run/crio/crio.sock --fail-swap-on=false --cgroup-driver="systemd"
[...]
I1117 13:41:58.741561 88434 kubelet_node_status.go:78] "Successfully registered node" node="instance-2025-11-16-15-31-55"

We’re making progress! Let’s try to see the cluster’s health and deploy a pod:

$ export KUBECONFIG=admin.conf
$ kubectl get nodes
 NAME STATUS ROLES AGE VERSION
 instance-2025-11-16-15-31-55 Ready <none> 13m v1.34.2

$ kubectl create deployment web --image=zwindler/vhelloworld
 deployment.apps/web created


$ kubectl get pods
 NAME READY STATUS RESTARTS AGE
 web-6c8cc48c68-cdbtj 1/1 Running 0 4m17s

$ kubectl logs web-6c8cc48c68-cdbtj
 [Vweb] Running app on http://localhost:8081/
 [Vweb] We have 3 workers

Great success!

Conclusion

From here, CRI-O functionally shares Proxmox VE’s LXC engine.

To convince yourself, you can run the lxc-ls command, which will display a mix of “real” LXC containers (created with proxmox, 151 and 200) and pod containers (the STOPPED ones are cilium init containers)

$ lxc-ls -f
NAME STATE AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED
151 STOPPED 0 - - - true
200 STOPPED 0 - - - true
3045d1f3abcc069b1009acc869a27b09cf9531d0701a3f9d5a760213d57c7b20 STOPPED 0 - - - false
78057100e5d0613944c2be859dfd09f790eeba88bceebea0e04970841c9bd950 RUNNING 0 - 10.0.0.90 - false
8e5506beea9a50214c46f31fa7fa6d04397963cb912aa397261359865d40a0cd RUNNING 0 - 10.0.0.167, 10.244.0.1, 203.0.113.42, 192.168.1.10 2001:db8::1 false
abdcd5be0a27417aa9e11cc2b27882bc86a63d94f547f909332ca7470a5e075a STOPPED 0 - - - false
aeaed9c2f56328bb6196ede4a78f06c4bb2d3edf5dca7912cf38407b9bf6610a STOPPED 0 - - - false
ed9aaf5a0fd4b11be121296290472d6d71d9016e4c6bb0d05fb2e4a7f4b7a85d RUNNING 0 - 10.0.0.167, 10.244.0.1, 203.0.113.42, 192.168.1.10 2001:db8::1 false
edb9d42e35a4029cfec5bed5597746bdf67fbe438f21446230252265ed1c849d STOPPED 0 - - - false
$ ps -ef |grep ed9aaf5a0fd4b11be121296290472d6d71d9016e4c6bb0d05fb2e4a7f4b7a85d
root 2674943 1 0 22:32 ? 00:00:00 /usr/libexec/crio/conmon -b /run/containers/storage/overlay-containers/ed9aaf5a0fd4b11be121296290472d6d71d9016e4c6bb0d05fb2e4a7f4b7a85d/userdata -c ed9aaf5a0fd4b11be121296290472d6d71d9016e4c6bb0d05fb2e4a7f4b7a85d --exit-dir /var/run/crio/exits -l /var/log/pods/kube-system_cilium-operator-56d6cd6767-dps78_6082ea2d-331e-4262-b8b3-d3f88eb4e446/cilium-operator/1.log --log-level info -n k8s_cilium-operator_cilium-operator-56d6cd6767-dps78_kube-system_6082ea2d-331e-4262-b8b3-d3f88eb4e446_1 -P /run/containers/storage/overlay-containers/ed9aaf5a0fd4b11be121296290472d6d71d9016e4c6bb0d05fb2e4a7f4b7a85d/userdata/conmon-pidfile -p /run/containers/storage/overlay-containers/ed9aaf5a0fd4b11be121296290472d6d71d9016e4c6bb0d05fb2e4a7f4b7a85d/userdata/pidfile --persist-dir /var/lib/containers/storage/overlay-containers/ed9aaf5a0fd4b11be121296290472d6d71d9016e4c6bb0d05fb2e4a7f4b7a85d/userdata -r /usr/local/bin/lxcri --runtime-arg --root=/var/lib/lxc --socket-dir-path /var/run/crio --syslog -u ed9aaf5a0fd4b11be121296290472d6d71d9016e4c6bb0d05fb2e4a7f4b7a85d -s
root 2674951 2674943 0 22:32 ? 00:00:00 /usr/local/libexec/lxcri/lxcri-start ed9aaf5a0fd4b11be121296290472d6d71d9016e4c6bb0d05fb2e4a7f4b7a85d /var/lib/lxc /var/lib/lxc/ed9aaf5a0fd4b11be121296290472d6d71d9016e4c6bb0d05fb2e4a7f4b7a85d/config

Nothing would prevent us from pushing it further, and making a small script that retrieves all container information and creates the associated /etc/pve/lxc/xxx.conf files to display them in the UI, as I did in the article:

Running Docker containers with Proxmox VE (and LXC)

In theory, if this step made sense, I could create releases myself on my fork to facilitate installation.

Finally, I could try to contribute my modifications to have everything merged into the main project, except the maintainers refuse PRs because the project is no longer maintained.

But I admit that after probably about fifteen hours of debugging, golang, C compilation, spread over 2 years, I’ve kind of reached the end of my patience for this “joke”.

OK it was dumb, now that I’ve succeeded, time to sleep X_X.

Running Docker containers with Proxmox VE (and LXC)

Mon, 07 Nov 2022 06:30:00 +0200

Introduction

I found several tutorials to install Docker (engine) inside an LXC container under Proxmox VE to then run Docker containers inside it. The problem is that you have to log into the LXC container before you can interact with your Docker containers.

You can also install Docker directly on Proxmox VE but it’s not really recommended and you don’t have the ease of use of LXC with Proxmox GUI.

And actually, it turns out you can run Docker images as the OS of an LXC container and thus have Docker controlled by Proxmox. So we won’t have 100% identical behavior to if you had done a docker run on your machine.

However, we’ll gain in exchange a container totally identical to your other LXC containers in your cluster (with all associated functions) but launched with a Docker image.

But first I suggest we go back a bit before showing you how…

Context

You know, I’ve been doing Proxmox VE for a while. Proxmox VE is a great turnkey and production-ready distribution for server virtualization (it’s even hyperconverged if you enable the Ceph part).

There are lots of little things I like about Proxmox VE, and one of them is that we can create Linux containers with LXC rather than QEMU virtual machines. I wrote an article to talk about it.

In use (except for specific kernel configuration), the LXC container behaves strictly identically to a VM in Proxmox, but for a fraction of the resources! We have an OS, we install applications on it, like a VM.

An nginx frontend for several applications. The complete LXC container only consumes a few MB

Admittedly it’s much less isolated than a real VM (because it’s a container, we’re isolated from other processes, but we run on the hypervisor’s kernel directly).

But with so few resource needs, I can host many small applications in different contexts with very small physical machines (Atom 4GB RAM at €6 per month at OneProviders) which would be impossible with a real VM.

It’s good, but not enough

However, LXC is not the most known/hyped containerization technology. For years on the forum, whenever people ask for Docker support in Proxmox VE, they get sent away. Not always very nicely either…

Yet, there are several reasons to want to run Docker containers on your Proxmox infrastructure:

First, Docker isn’t as much stateless and/or ephemeral workloads as we think. Many containerized applications would have their place on a virtualization cluster.
Then because many software vendors have become lazy. Some no longer manage installation processes (or only on one OS) and just provide a Docker image that they maintain up to date.

So we have to settle for VMs in Proxmox to run Docker containers (see my point at the beginning).

LXC supports OCI images

Plot twist. A few days ago, I discovered that LXC, Proxmox VE’s containerization engine, was compatible with the OCI format.

buzzwrd.me - Creating LXC containers from docker and OCI images

For those who don’t know OCI, it’s an organization aiming to create a standard to unify the way containers are stored.

The Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes. opencontainers.org/

And Docker respects this format: we can therefore theoretically run from LXC containers whose base image is a Docker image.

So, when do we start?

Disclaimer: what follows is tinkering. Nothing is supported and I don’t recommend it in production. There are also likely limitations (especially regarding storage).

First, some dependencies are missing for LXC’s OCI function to be usable under Proxmox VE:

sudo apt install skopeo umoci jq

Then, by default, LXC containers attach to a Linux bridge called lxcbr0. However this bridge probably doesn’t exist on your Proxmox VE installation (not installed by default).

In quick and dirty, we can just replace the default name, although ideally we should find the flag in the CLI to change the bridge. Depending on how you configured the network in your Proxmox VE server, you’ll probably use vmbr0 or vmbr1:

sed -i 's/lxcbr0/vmbr1/g' /etc/lxc/default.conf

Now that we have everything, we can run our LXC containers from docker images. Here I’m running the official alpine:latest image, but I could have taken any other:

lxc-create 500 -t oci -- --url docker://alpine:latest

I created a container called “500” (because VMs and LXC containers in proxmox VE are called with numbers). The image is downloaded then the container created:

We can now start it (with lxc-execute we run the container and open a prompt directly inside it, but we also have lxc-start and lxc-attach)

lxc-execute 500

As is, Proxmox VE doesn’t know our container exists and it won’t appear in the UI. But it works ;-).

Bonus

We can even cheat and make Proxmox believe it’s an LXC container it created itself by creating a 500.conf file, in the /etc/pve/lxc/ folder

The only mandatory fields are the following:

arch (amd64 unless you’re on ARM)
cores (the CPU limit to impose on the container)
hostname
memory (the RAM limit to impose on the container)
ostype
rootfs

Once the file is created, the container automatically appears in the host’s VM list. We can start it and it works :)

The container can be started, stopped, etc. All from the Proxmox VE GUI :)

Have fun!

Source

Should we have containers ?

Thu, 25 Aug 2016 16:00:21 +0000

A more complete answer to « Why do we have containers » by fntlnz

2 days ago, I stumbled apon fntlnz article Why do we have containers. (dead link, using Internet Archive) The article had been reposted by Docker official Twitter account and the first thing that struck me was the displayed image : a schema comparing full virtualization, para virtualization and linux containers.

And of course, I say this because I didn’t entirely agree with it ;-). But then I read the article and I found many good points.

So I decided to write « Should we have containers ? » as an answer to develop my remarks a little.

Full virtualization vs paravirtualization vs OS level virtualization

First, I’ll go with the schema. I know the point was to provide a really simple schema to help understand the differences between virtualization types but I don’t think it should be presented like that.

Hypervisor and host OS are both displayed in the full/para. I find this be misleading. Of course there is a Host OS and a virtualization engine on top of it. But most of the time, either the OS is really thin (ESXi fits in 8GB SD cards and runs nothing but virtualization) or the virt engine is a part of the kernel (KVM, Xen) just like LXC is a part of the Linux kernel.

Also, the bins/libs are only displayed on OS level virt. This is interesting to put in the « host level virtualization schema » but it should be added on full and para as well because this is one of the advantages host level virt. has over full/para.
Even if your VMs are all identicals, bins and libs have to be present (copied) in each VMs. On contrary, in OS Level virt, you CAN share identicals bins/libs between containers, thus reducing the storage footprint of the virtualized environment.

That’s how I would have presented it

You can see that OS level virtualization and Hypervisors are not so different on this simplified view. The differences appear more clearly when you add x86 Security Rings (0->3). The « overhead » of full virtualization comes from the calls that guest OSes do to hypervisor pilots and the translation of CPU instructions.

On a side note, I put Docker on the left to show that it’s basically an administration suite that helps you run LXC containers

The rest of the analysis

Security

When talking about containers, the security point is often raised.

VMs are really hermetics. Taking control of the hypervisor after hacking a VM should be MUCH harder than taking control of the host OS if you are inside a container with whom the kernel is shared.

But I agree with fntlnz: having to patch only one kernel for all your virtual environments instead of having to patch ALL kernels of your virtual machines is actually a good point.

I encourage you to read this (dead link, using Internet Archive) from Jérôme Petazzoni if you’d like a good presentation on the subject.

Performance

A point that’s not really developed in fntlnz article is performance, but I read it/ear it all the time.

General belief is that VMs are slow. That was true in the « early days » of x86 virtualization, because of the limitations of the x86 architectures which didn’t allow efficient OS virtualisation. Overhead for CPU instructions was known to be of about 15% (which is enormous). Thus, running processes directly on the OS are believed to be more efficent.

But nowadays, modern hypervisors on modern hardware nearly generates no overhead at all over CPU and RAM consumption. You could argue that guest OSes consume ressources while containers don’t. But let’s be honest: with the 32 GB RAM tomcat instances I regularly see, don’t say to me that a Linux guest OS CPU&RAM consumption is significative.

The real difference is the on the storage consumption. First, for N VMs, you save yourself N*xGB of guest OS files. If your Linux templates are good, you don’t consume that much (a few GB at most) but this can add up when we are talking about thousands of instances. [Edit jan. 2020]Not so true today if you host your own hardware and have so deduplicated storage[/Edit]

From there, you still have some margin for gains, mostly through Copy on write and union filesystems :

You make a modification to an official tomcat image to add you configuration files ? Only the delta will be stored.
You launch thousands of containers from this image ? You only consume the container image once.
The real data (user data, business data) is of course the same for both architectures

So, should I use containers ?

The problem here is that’s not the real question. Do you need containers ? Here’s fntlnz point of view.**
**

But if at some extent your motivation consist in decreasing costs related to full virtualization overheads while allowing developers to ship, develop and test code faster we already found two.

fntlnz

I won’t tell you not to use containers or Docker. Truth be told, we use it in dev and production. Docker is awesome.

Yes, Docker reduces overhead of full virtualization. Storage mostly, but it highly depends on your workload. For example if you have the need for millions of really tiny instances, then Guest OS overhead is significative in full virtualization.
Yes, Docker allows developers to ship, develop and test code faster. You still have to change your way to develop yours app but Docker and DevOps is the new thing. Better get on with it ;)

But you shouldn’t use Docker just because of that. Docker containers need you to change your philosophy toward applications and administration.

Shared kernel = same kernel

You need to understand that you’ll share a kernel. For a long time that ment a 100% Linux only which could be problematic in some cases. I just realized that Windows containers exists in Windows 2016 tech preview. And Solaris is coming.

But independantly of platforms, you still you share a kernel. If you want Linux containers, you’ll need Linux Docker nodes. If you need Windows containers, you’ll have to have Windows Docker nodes.

Non-persistant/cloud-ready instances

Docker containers are by design non-persistant and by best practice single-process. This isn’t mandatory of course, but Docker works better that way.

The whole idea is to help you make « cloud-ready » & atomic applications that don’t store anything in the container itself (persistant storage is presented to the container but outside it). This way, you can scale out really quickly: you just start up a few more instances if you need more firepower.

This also modifies the way you do systems/applications administration. A container/application is spinning out of control ? Don’t try to repair it.

Kill it
Launch a new one

The fact that persistant storage is handled outside the containers also helps you to deal with lifecycle management.

Update your application inside the docker image
Shutdown the old container
Start the new container with the new image, plugged to the persistant storage

All that requires of course applications that have been designed this way! This is a massive change in philosophy and old monolithic apps on a Docker platform won’t work as well as microservices will.

System administration

I don’t say solutions don’t exist because they do.

But starting up and shutting down containers on the fly will give headaches to your « old school Nagios » monitoring administrator. What could be done easily in the physical/VM world (installing an agent in the template), each of those having an IP adress to query for health checks is a lot easier than checking on a single host a highly volatile number of isolated processes.

You’ll probably have to adapt and use tools that are designed to handle such cases.

Backup administrator may find it confusing at first to make application consistant data backups when applications are running in an variable number of containers.

And so on…

Final thoughts

Yes, you should definitely look into containers. They are mature, stable, a great new way to speed up development and contribute to save money on hardware.

But be aware that VMs and containers don’t answer all usecases; and that you won’t migrate all your applications overnight!

LXC on Zwindler's Reflection

OCI Artifacts as LXC Machine Base on Proxmox 9.1, Finally?

Context

Let’s Test? First the Prerequisites

Creating the LXC Container

When It Wasn’t Working (Yet)

First Bug: “from scratch” Image

Second Bug: Unable to Start

Conclusion

Turning Proxmox VE into a Kubernetes Node with LXC and lxcri

He didn’t know it was completely stupid, so he did it

How do we do this?

Let’s go

There’s no release

Prerequisites for building lxcri

Build like never before

Back to crio

Fast forward

Conclusion

Running Docker containers with Proxmox VE (and LXC)

Introduction

Context

It’s good, but not enough

LXC supports OCI images

So, when do we start?

Bonus

Source

Should we have containers ?

A more complete answer to « Why do we have containers » by fntlnz

Full virtualization vs paravirtualization vs OS level virtualization

The rest of the analysis

Security

Performance

So, should I use containers ?

Shared kernel = same kernel~~ ~~

Non-persistant/cloud-ready instances

System administration

Final thoughts

Shared kernel = same kernel