OS on Zwindler's Reflection

Kubernetes UserNamespaces: the overhyped GA feature

Tue, 28 Apr 2026 10:00:00 +0200

The infographic that triggered me

Note: I stumbled upon this GenAI infographic and it was so bad I wrote a post about it. I didn’t generate this thing.

Over the past few days, LinkedIn has been flooded with the same kind of infographic. Kubernetes 1.36 is out, and one of the most talked-about features is the GA release of UserNamespaces.

It’s a topic I’ve been following since 2018 (talk The Route to rootless containers at KubeCon EU 2018), so I’m genuinely glad to see this long journey finally reach the finish line. That said, I’m appalled by the way it’s being marketed on LinkedIn, apparently by people who have no idea how it actually works — and frankly don’t care.

“Kubernetes just made root safer. Just add hostUsers: false to your Pod spec.”

The visual: an all-powerful king “inside the container” and a helpless beggar “outside on the host”. The promise: “No Host Access. No Privilege Escalation. No Lateral Movement. No Node Takeover.”

Catchy.

But presenting it this way is genuinely dangerous, because it obscures entire areas of application and operational security. Selling hostUsers: false as the universal fix for the “root in containers” problem is a dramatic oversimplification that will push teams to ignore the real security priorities.

What UserNamespaces actually do

The threat model: container escape

First, what are we actually talking about? A container escape is when an attacker manages to break out of their container and directly access the host’s kernel or filesystem — completely bypassing the normal isolation mechanisms.

This type of vulnerability is rare, but real-world examples exist:

CVE-2019-5736 (runc): write to /proc/self/exe of the host process from inside the container
CVE-2022-0492 (cgroups v1): escape via unshare in certain configurations
CVE-2024-21626 (runc, “Leaky Vessels”): file descriptor leak to the host working directory

If there’s a vulnerability of this type on your node AND a process is compromised AND it runs as root in the container AND it doesn’t use UserNamespaces, the attacker gets root on the host — game over. Full access to every file on the node, every secret mounted by other pods, ability to install a rootkit or exfiltrate data from all tenants running on that node.

It remains possible, but that’s a lot of “ifs”. Either way, this is exactly the scenario UserNamespaces address. They introduce UID mapping:

UID 0 inside the container is mapped to an unprivileged UID on the host (e.g. 100000, unique per pod)
If an attacker successfully escapes the container via a kernel exploit, they land as nobody on the node — the escape succeeds, but the post-escape impact is dramatically reduced

This is the “Breakouts Lose Impact” scenario from the infographic, and on that point, the infographic is right. That’s the real value of the feature.

Edge case: multi-tenancy even with non-root containers

Even without root containers, UserNamespaces provide something extra in a truly multi-tenant context (multiple customers on the same cluster). Without UserNamespaces, if two pods from different customers both run with runAsUser: 1000, they share the same UID 1000 on the node. If one escapes, the attacker can access files from the other pod with the same owner. UserNamespaces, by assigning a unique UID offset per pod, isolates UIDs between pods even when they use the same value inside the container.

For internal clusters where you control all workloads, this scenario is theoretical. For a multi-tenant SaaS platform or a public build service, it’s a real line of defense.

Technical requirements

There are a few prerequisites, but most up-to-date clusters should qualify.

Linux kernel ≥ 5.19
Compatible runtime (containerd ≥ 1.7, CRI-O ≥ 1.25)
Idmapped mounts support for persistent volumes (XFS, ext4 — not NFS in all cases)
Kubernetes ≥ 1.33 (Beta), ≥ 1.36 (GA)

What the infographic exaggerates (and leaves out)

The infographic is right about one specific thing: UserNamespaces reduces the impact of a successful container escape. That’s real. The problem is it sells the feature as a universal solution to “root in containers” — and that’s just wrong.

1. UID isolation is not application privilege isolation

The infographic promises “No Lateral Movement”. That’s false — completely false.

A root container with hostUsers: false can still read the ServiceAccount Token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. If that token has broad RBAC permissions (which happens — we may cover this in a future post), the attacker can call the API Server, enumerate cluster resources, and move laterally — all without ever touching the host node.

UID mapping protects the host. It does not protect the cluster.

2. A root container is still root inside the container

“Install Anything ✅” — it’s literally written in the infographic, presented as a feature 😖.

In a root container (even with UserNS), an attacker who gains control can:

Install nmap, curl, nc to scan the internal network
Modify application files, binaries, configurations
Read all files mounted as volumes
Persist in the container across restarts if the filesystem is writable

UserNamespaces removes none of these attack vectors. The ability to install software is a fast track to lateral movement.

3. It’s not that easy, especially for storage

Enabling hostUsers: false breaks existing storage in most cases.

Container UID 0 is mapped to UID 100000+ on the host (each container has its own offset). If a persistent volume (NFS, EBS, Ceph RBD) is owned by UID 1000, the root container can’t read or write it. The result: counterintuitive Permission Denied errors that are potentially hard to diagnose, since the application was likely never designed to be root yet have no access to its own files.

The technical solution exists (idmapped mounts), but it requires a recent kernel and a compatible filesystem. See the official idmapped mounts documentation for details.

4. Same story, but for networking

hostUsers: false is incompatible with hostNetwork: true. It’s a corner case, but it catches networking workloads (monitoring agents, CNI plugins, etc.).

Note: that said, running containers with hostNetwork is its own security problem, so…

Honest comparison: UserNS vs the real alternatives

Attack vector	UserNS (root inside)	Non-root (UID 1000)	Distroless / Scratch
Post-escape impact after successful container escape	✅ Nobody on host	⚠️ UID 1000 on host	⚠️ UID 1000 on host
UID isolation between pods (multi-tenant)	✅ Unique offset per pod	❌ Shared UID on node	❌ Shared UID on node
Malware installation inside the container	❌ Trivial	❌ Possible	✅ Near impossible
Write scope in ephemeral container FS	❌ Full filesystem	❌ App directory only	✅ Near impossible
Lateral movement via SA Token	❌ Possible	❌ Possible	⚠️ Potentially difficult
Operational complexity	❌ Sometimes high	✅ Often near zero	✅ Often low
Compatibility with existing storage	❌ Sometimes problematic	✅ Standard	✅ Standard

Reading the table reveals the true nature of UserNamespaces: it excels on exactly two rows:

post-escape impact
UID isolation in multi-tenant environments

On everything else, non-root + distroless does better, or just as well, without the operational complexity. And that “everything else” — write scope in ephemeral FS, malware installation, lateral movement via SA Token — represents the vast majority of real-world attack vectors, far more common than a container escape. We’ll come back to this in the Where to invest your security budget section.

Real use cases

It would be dishonest to dismiss the feature entirely. There are three scenarios where UserNamespaces aren’t a lazy option but a genuine technical necessity (with caveats).

1. Build-as-a-Service (Buildah, rootless Podman)

To build a Docker image, the build engine needs to perform chown, chmod and mknod. These operations require CAP_CHOWN and CAP_FOWNER. Before UserNamespaces, the solution was to run the pod as --privileged — an obvious open door to the host.

With hostUsers: false, the build engine believes it’s root for its own file manipulation, but it can’t touch the host. This is the only case where “root inside” is a technical constraint rather than technical debt.

Note: Kaniko, long the go-to for in-cluster builds, has been archived since October 2025 and no longer receives security updates. Buildah or rootless Podman are the active alternatives.

My take: it can be useful for shared CI/CD platforms (GitLab Runners, Tekton) that refuse privileged pods. But if isolation is critical (public platform, aggressive multi-tenancy), microVMs (Kata Containers, Firecracker) offer far stronger guarantees for an overhead that has become quite manageable.

2. Hostile multi-tenancy (user code platforms)

If your business is running code provided by strangers (PaaS, online code editors, public CI/CD), you know upfront that users will try to escalate their privileges. In this context, UserNS is an extra barrier against kernel 0-days.

My take: honestly, if the environment is truly hostile, UserNS alone isn’t enough. MicroVMs (Kata Containers, Firecracker) provide real hardware isolation and are the right choice here. UserNS can be a complement, not a substitute.

3. Hard-coded legacy (Postfix, Dovecot, BIND)

Some old UNIX daemons start as root to open a privileged port (< 1024) or read sensitive config files, then drop privileges via setuid(). This mechanism fails in a classic non-root container.

UserNamespaces let these processes believe they can make their identity management syscalls, because they are root inside their namespace.

Here’s a concrete example written by a colleague (thanks Louis 😘):

apiVersion: v1
kind: Pod
metadata:
 name: postfix
spec:
 hostUsers: false # UID mapping: root in container → nobody on host
 securityContext:
 runAsNonRoot: false # allowed under PSS Restricted *only* because of hostUsers: false
 fsGroup: 103 # postfix GID
 containers:
 - name: postfix
 image: postfix:latest
 securityContext:
 runAsNonRoot: false # same — cf. https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/#integration-with-pod-security-admission-checks
 allowPrivilegeEscalation: false
 readOnlyRootFilesystem: true
 seccompProfile:
 type: RuntimeDefault
 capabilities:
 drop: ["ALL"] # drop everything first
 add:
 - SETUID  # privilege drop via setuid() done by postfix itself at startup
 - SETGID  # same for groups
 - CHOWN  # chown on mail queues at startup
 - FOWNER  # file operations without being the owner
 - FSETID  # preserve setuid bit after write
 - DAC_OVERRIDE # MANDATORY: root in UserNS is not "real" root —
 # DAC checks are not automatically bypassed

This manifest illustrates several important things.

First, making a legacy application actually secure with UserNS is painful and requires compromises — especially around capabilities. This is a far cry from the “magic feature that secures root apps” the LinkedIn wanna-be influencers imply.

Then there are some interesting surprises. Normally, the Restricted Pod Security Standard forbids runAsNonRoot: false. Kubernetes makes an exception when hostUsers: false is present. This is documented here. Without UserNamespaces, this pod would be rejected by the admission controller.

There’s also the DAC_OVERRIDE capability, which is counterintuitive. Root in a UserNS is not real root from the kernel’s perspective for DAC (Discretionary Access Control) checks. When Postfix runs set-permissions to chown its queues, the kernel still verifies permissions — and denies them if DAC_OVERRIDE isn’t present. This is exactly the kind of operational surprise that stays invisible until the first production deployment.

Worth noting: we were still able to keep readOnlyRootFilesystem: true and allowPrivilegeEscalation: false — legacy doesn’t justify throwing everything overboard.

My take: this is the only use case where UserNS is genuinely acceptable. No untrusted third-party code, no hostile platform — just well-identified legacy with a migration plan. The other two cases are “acceptable under conditions”; the legacy case is the cleanest of the three.

Some counterarguments

I see you coming with objections, so let’s save everyone some time with a quick Q&A:

“It’s defense in depth.” True — but defense in depth assumes the foundational layers are already in place. If you haven’t migrated your images to non-root yet, investing energy in UserNS is putting the cart before the horse. And once you’re non-root, the marginal gain of UserNS is negligible compared to the complexity it introduces.

“We don’t control third-party images.” Somewhat weak, in my opinion: if a proprietary vendor’s black-box image is hardcoded to run as root, there’s a good chance it either genuinely needs it (as is the case for some proprietary security tooling) or it will break with UID mapping (see the storage problem above). UserNS is not a magic wand that makes any third-party image compatible and secure.

“It’s a centralized safeguard against human error.” It’s just as easy to forget hostUsers: false as it is to forget runAsNonRoot: true. The real centralized solution is Pod Security Standards or an Admission Controller (Kyverno, OPA) that outright rejects root pods. Simpler, more reliable, and it doesn’t break storage.

“We need it for SOC2/PCI-DSS/… compliance.” If your compliance requires strict tenant isolation, UserNS will likely be deemed insufficient by your auditors. VMs or microVMs remain the gold standard. Using UserNS for compliance means choosing the most complex tool to maintain for a result that remains debatable.

Where to invest your security budget

Setting aside the marketing, here’s where effort actually pays off — from highest impact to most niche:

Priority 1 — Non-root images + nobody (UID 65534) Move images to non-root, ideally using the nobody user (the least privileged on the system). If an application is compromised under nobody, the attacker can do almost nothing, even on the container filesystem. Combine with readOnlyRootFilesystem: true and capabilities: drop: ["ALL"].

spec:
 securityContext:
 runAsNonRoot: true
 runAsUser: 65534 # nobody
 seccompProfile:
 type: RuntimeDefault
 containers:
 - name: app
 image: my-app:distroless
 securityContext:
 allowPrivilegeEscalation: false
 capabilities:
 drop: ["ALL"]
 readOnlyRootFilesystem: true

Priority 2 — Pod Security Standards (PSS) at Baseline or Restricted Block root and privileges without breaking anything at the infra level. It requires having done Priority 1 first, but it’s free, standard, and applies cluster-wide via a namespace label (with per-namespace overrides when needed). No more risk of forgetting. Already enabled by default on several Kubernetes distributions (Talos being one, but not the only one).

Priority 3 — MicroVMs (Kata Containers, Firecracker) For truly untrusted workloads. Real hardware isolation, overhead now quite reasonable on recent generations.

Priority 4 — UserNamespaces When all else fails. Only for the legitimate cases identified above (builds, legacy, hostile multi-tenancy). This is genuinely the last thing to do.

Conclusion

Kubernetes 1.36 UserNamespaces are the result of a project that officially took five years (KEP-127 dates back to 2021) and has been discussed since nearly the dawn of Kubernetes. For shared build platforms and multi-tenant SaaS running user-provided code, it’s a potentially useful building block — particularly to prevent one customer’s app from reading another’s in the event of a container escape without privilege escalation.

For everything else — that is to say, 99% of production clusters — that’s not where container security starts. And that’s precisely the problem with this kind of infographic.

LinkedIn infographics selling effortless security are dangerous: “keep your 800MB root image full of tools, just add hostUsers: false, and you’re protected.” That’s exactly the wrong approach. Real container security is built in the Dockerfile, not in the PodSpec.

If you’re enabling UserNamespaces to secure an application whose source code you own, you’ve probably missed a step in your secure software development lifecycle.

References

KEP-127: Support for User Namespaces
Kubernetes docs — User Namespaces
Pod Security Standards
CVE-2019-5736 — runc: write to host’s /proc/self/exe from inside the container
CVE-2022-0492 — cgroups v1: escape via unshare, UserNS helps but runAsNonRoot does too
CVE-2024-21626 — runc “Leaky Vessels”: file descriptor leak to host working directory
Distroless containers — GoogleContainerTools

Installing Alpine Linux in Headless Mode on Raspberry Pi

Tue, 23 Dec 2025 08:00:00 +0200

Introduction

For an article that might be published in print (fingers crossed), I needed to test different OS types to install on a Raspberry Pi Zero 2 W.

Since I was unfamiliar with both Alpine and the specifics of the Pi Zero 2 W, I came across quite a few things I didn’t know.

Even though the print article ultimately doesn’t cover Alpine Linux installation (or even use it), I thought it would be a shame not to publish this Alpine + RPi Zero experience, so here it is.

Why Alpine Linux?

Mainly for its lightness and low resource consumption. On a nano-computer with only 512 MB of RAM, every megabyte counts. Alpine Linux is known for being minimalist and also somewhat controversial (especially in the Docker world) for replacing glibc with musl (which sometimes causes surprises).

The goal of this article is to have a headless installation (without screen or keyboard) accessible via SSH through WiFi, with maximum RAM optimization.

We’ll see lots of interesting technical details, especially about GPU RAM management and swap configuration with zswap!

Downloading the Alpine Linux Image for Raspberry Pi

The first thing to do is download the image before copying it to a microSD card that will be inserted into the RPi before boot.

Fortunately, Alpine Linux offers specific images for Raspberry Pi, so we won’t need to manually add the RPi firmware.

We’ll retrieve the latest available version from the official website. Example:

wget https://dl-cdn.alpinelinux.org/alpine/v3.23/releases/aarch64/alpine-rpi-3.23.2-aarch64.tar.gz

This image is specially compiled for the ARM64 (aarch64) architecture of recent Raspberry Pis (so my RPi Zero 2 W).

Headless Configuration with Overlay File

Once the image is flashed onto your microSD card (with Raspberry Pi Imager for example), we’ll prepare the system for headless access.

Alpine Linux supports what are called “overlay files” that allow you to preconfigure the system before first boot. This is documented in the official documentation on headless installation.

Mount the contents of your microSD card on your machine and place the overlay file at the root of this microSD card:

wget https://github.com/macmpi/alpine-linux-headless-bootstrap/raw/469ee440e7d394cca0976c78f357e7a0e1c82cc4/headless.apkovl.tar.gz

Note: it’s quite scary to mount such a sensitive file from a third-party GitHub repo (github.com/macmpi), but Alpine Linux is reassuring by telling us:

Note: The author of above repo macmpi also maintains a number of packages for Alpine Linux.

I’m not much more reassured… but well, it’s “official”

This headless.apkovl.tar.gz file configures Alpine to:

Boot in DHCP mode
Enable SSH automatically
Allow root login without password (be careful not to leave it like that for too long!)

For the Raspberry Pi to automatically connect to your WiFi network, create a wpa_supplicant.conf file at the root of the microSD:

cat > wpa_supplicant.conf << EOF
country=FR

network={
 key_mgmt=WPA-PSK
 ssid="mySSID"
 psk="myPassPhrase"
}
EOF

Obviously replace mySSID and myPassPhrase with your WiFi information.

Note: The alpine-linux-headless-bootstrap project offers other very useful configuration options, such as setting a root password, a static IP address, etc. More info in their README.

First Boot and Permanent Installation

Once the first boot is complete, connect via SSH:

ssh root@<raspberry_ip>

Warning: By default, the root password is empty! But we don’t need to change that if we perform the following installation immediately.

At this stage, Alpine runs entirely in RAM (“live” image). To install it permanently, we’ll use the setup-alpine utility whose options are described in wiki.alpinelinux.org/wiki/Installation.

setup-alpine

Here’s a small excerpt of the requested information (all quite standard for an OS install):

Follow the instructions, then reboot:

reboot

GPU RAM Optimization

By default on a Raspberry Pi, a (large) part of the RAM is allocated to the GPU for graphics processing. On a headless server, this is absolutely useless.

Let’s first check the available RAM before optimization:

free -m
 total used free shared buff/cache available
Mem: 417 37 325 0 55 369
Swap: 0 0 0

Only 417 MB available out of the theoretical 512 MB. Let’s see how to recover those precious megabytes.

Ideally, we want to set the gpu_mem=16 option in /boot/config.txt to allocate only 16 MB to the GPU (we can’t set less, and it must be multiples of 16).

But be careful, there’s a trap!

(a french mème, sorry english speakers)

When you set gpu_mem to 16 MB, the Raspberry Pi bootloader attempts to load lightweight firmware files:

start_cd.elf
fixup_cd.dat

Except these files are not present on your boot partition. If we just put gpu_mem=16 in /boot/config.txt and reboot, the Pi will refuse to start (ACT LED usually blinking 4 times).

Solutions to Minimize GPU RAM

You have two options:

Option 1: Set gpu_mem=32 in /boot/config.txt

This is the easy solution (who said lazy?), there’s nothing to do. You still recover 32 MB of RAM.

Option 2: Download the lightweight firmware files (the one I prefer)

We’re not short on a few MB on the microSD card, which is not the case for RAM, where every MB counts.

If you really want to go down to 16 MB, retrieve the missing files:

cd /boot
wget https://raw.githubusercontent.com/raspberrypi/firmware/master/boot/start_cd.elf
wget https://raw.githubusercontent.com/raspberrypi/firmware/master/boot/fixup_cd.dat

Then modify /boot/config.txt:

[...]
gpu_mem=16
start_file=start_cd.elf 
fixup_file=fixup_cd.dat

After reboot, check the gain:

free -m
 total used free shared buff/cache available
Mem: 465 37 393 0 35 416
Swap: 0 0 0

We go from 417 MB to 465 MB available! That’s almost 50 MB recovered, representing about 10% additional RAM. On such a constrained system, this is significant.

Swap Configuration with zswap

On a system with only 512 MB of RAM, it’s highly advisable to configure swap. However, Alpine Linux doesn’t configure it by default. I’ll configure one here with zswap, which compresses memory pages before writing them to disk (which also limits the size of accesses to the microSD card).

Creating the Swapfile

# Create a 2 GB swapfile
fallocate -l 2G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile

Persistent Configuration

To make the swapfile and zswap persistent across reboots, create /etc/fstab:

echo "/swapfile none swap sw 0 0" >> /etc/fstab

Create a startup script for Alpine:

cat > /etc/local.d/swap.start << 'EOF'
#!/bin/sh
swapon /swapfile
EOF

chmod +x /etc/local.d/swap.start
rc-update add local default

Finally, edit /boot/cmdline.txt and add at the end (on a single line):

zswap.enabled=1 zswap.compressor=lz4 zswap.max_pool_percent=20 zswap.accept_threshold_percent=85 zswap.shrinker_enabled=Y

Some explanations on these parameters:

compressor=lz4: uses lz4, fast and efficient for compression
max_pool_percent=20: zswap won’t use more than 20% of RAM (~93 MB) as compression cache before writing to the disk swapfile
accept_threshold_percent=85: only accepts compression if the expected compression rate is at least 85%
shrinker_enabled=Y: allows automatic freeing of compressed memory if necessary

The principle: zswap acts as a fast cache in RAM (compressed) in front of the 2 GB swapfile on disk. Pages are first compressed in zswap, then evacuated to disk if necessary.

Conclusion

Not really a conclusion to this article, actually.

I hit quite a few “walls”, not knowing how to use Alpine (beyond the uses I had made of it with Docker), and I discovered some subtleties related to the Raspberry Pi Zero firmware that I’ve been tinkering with for a few weeks now.

Hence this article ;-)

Happy hacking! 🚀

Proxmox VE 8 Tutorial - SDN in VXLAN Mode with Machines on the Internet

Sun, 30 Mar 2025 12:30:00 +0200

Preamble

This tutorial is a kind of special edition in my series of articles on Proxmox VE 8.

In the third part which came out a few days ago, we stopped at a cluster of remote machines on the Internet.

They’re not in the same LAN, and to be honest, they’re even about 100ms apart (and it works very well).

At the end of the article, it works, we have a single network on our two machines, with a functional IPAM and DHCP on both sides. But still, virtual machines on different hosts can’t talk to each other directly and it’s still a bit unfortunate not to have gone all the way.

Rather than rewriting that part, I decided to make a small aside where I finish the work and test the VXLAN function of the Proxmox VE SDN, because YES, it can work all the way.

Prerequisites

I’m assuming you already have two (or more) machines within the same Proxmox VE 8 cluster.

If not, you’re in for reading my series of articles (currently being written) on the subject (have fun!):

Deploying a Proxmox VE 8 Cluster on Dedicated Servers (1/4)

These machines are not in the same LAN (otherwise no need to bother with VXLAN, the Simple SDN can be enough, as we did in part 3)

Problem

What we want to do is create an SDN that goes through the Internet, to the point that virtual machines believe they’re on the same LAN.

We can’t do this with Simple SDN, VMs on one host can’t contact VMs on another distant host on the Internet, even being in the same Simple SDN and with the same addressing plan.

But if we read a bit further in the PVE documentation on SDN, we see that this is precisely the use case highlighted for using VXLAN type SDNs:

The VXLAN plugin establishes a tunnel (overlay) on top of an existing network (underlay). This encapsulates layer 2 Ethernet frames within layer 4 UDP datagrams… […] You can, for example, create a VXLAN overlay network on top of public internet, appearing to the VMs as if they share the same local Layer 2 network

Small issue however, in reality, you’d better not do it, because frames are not encrypted, and on the Internet, it’s really not the best idea in the world…

Warning VXLAN on its own does does not provide any encryption. When joining multiple sites via VXLAN, make sure to establish a secure connection between the site, for example by using a site-to-site VPN.

As an alternative, we could have gone directly with the most complete/complex version of the Proxmox VE SDN: EVPN Zones

The EVPN zone creates a routable Layer 3 network, capable of spanning across multiple clusters. This is achieved by establishing a VPN and utilizing BGP as the routing protocol.

But I don’t want to mess with BGP here and I want to keep things simple. However, it’s probably the “cleanest” option for a production setup.

VXLAN Then, But with a VPN

I won’t overthink it, we’ll set up a small VPN with wireguard manually. There are tons of VPN tools on the market and tons of tools to make life easier. If you want more info on Wireguard, I’m sure you’ll find plenty, I’m just putting the bare minimum here so it “just works”.

On both servers, install wireguard, which should be in the default debian repositories:

apt update && apt install -y wireguard

Generate a key pair on each machine:

wg genkey | tee privatekey | wg pubkey > publickey

A little mental gymnastics, we’re going to write a configuration file with for each server, the private key of the server (file privatekey) we’re on, but the public key (file publickey) of the remote server.

(When you think about it, it’s obvious, but I prefer to insist)

And you obviously need to change addresses and network interfaces depending on which server you’re on so as not to have conflicts.

vi /etc/wireguard/wg0.conf
[Interface]
Address = 10.10.30.1/24
ListenPort = 51820
PrivateKey = <local server private key>
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT
PostUp = iptables -t nat -A POSTROUTING -o <network interface name> -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT
PostDown = iptables -t nat -D POSTROUTING -o <network interface name> -j MASQUERADE
[Peer]
PublicKey = <remote server public key>
Endpoint = <remote server public IP>:51820
AllowedIPs = 10.10.30.2/32

Note: you can also add more nodes to the VPN, just add more “Peer” sections.

Finish by starting the VPN and verifying that nodes talk to each other:

systemctl enable wg-quick@wg0 --now

ping 10.10.30.1
ping 10.10.30.2

SDN Configuration

Normally, if you followed the previous articles, you already have the prerequisites, but for those who haven’t, I’ll repeat that in the /etc/network/interfaces file, it’s necessary to add at the end of the file the following line (followed by a small systemctl restart networking)

source /etc/network/interfaces.d/*

Then, in the Datacenter menu, open the SDN submenu.

In the SDN/Zones menu, add a “VXLan” type zone. Two important points here:

we need to fill in the list of all “peers”, here the IPs of our VPNs so 10.10.30.1 and 10.10.30.2
we’ll need to adjust the MTU. By default, a frame is 1500. Except that wireguard for its encapsulation has reduced the MTU to 1420 and VXLAN encapsulation takes 50. So we need to set 1370 to best avoid fragmentation, at worst, dropped packets

If you just read the previous article, you’ll notice that here, there’s no “checkbox” to enable DHCP on the zone. Unfortunately, that’s because it’s not available yet.

Currently only Simple Zones have support for automatic DHCP

Source: pve.proxmox.com/pve-docs/chapter-pvesdn.html#pvesdn_config_dhcp

Once you’ve validated, go to the SDN/VNets submenu, click the “Create” button to create a virtual network, then select it once created to create a subnet.

Once again, don’t forget to check the DHCP section, to declare the range for our future DHCP.

Once we’ve validated everything, we see that next to all these new things we’ve just created, there’s an icon with two yellow arrows and a “New” status.

To deploy these changes, we need to go back to the SDN menu from the beginning, and click Apply. Two “vxlan1” lines (one for each server, actually) should appear.

Is That All?

Yes, that’s all, honestly, I really should have pushed a bit more before posting my previous article…

To convince yourself, just pop two LXC containers (without DHCP 🥲), one on each server, and see if they ping each other.

It works!

Well… all that’s left is to migrate all my existing machines to the VXLAN vnet to finally have a cluster where VMs can see each other!!

Deploying a Proxmox VE 8 Cluster on Dedicated Servers (3/4)

Tue, 25 Mar 2025 12:30:00 +0200

Note - this article is part of a series:

Picking Up Where We Left Off: First Steps in the UI

In the previous tutorial (which I invite you to read, if you haven’t already, otherwise this won’t be clear), we configured storage, networking, and deployed our first virtual machine to make sure everything works. But there’s still quite a bit to explore, including clustering, the Proxmox VE SDN, the built-in firewall, backups, monitoring…

So we’re picking up where we left off, and we’re going to install another server (hard to make a cluster with a single machine).

This server was reinstalled with the same procedures as the server I’m talking about in parts 1 and 2, except that we don’t need to create the bridge (vmbr0) because we’re going to use the SDN!!

Ideally, all machines in the cluster should have the same storage pools since I remind you, this configuration is supposed to be common to all servers, as it’s managed at the “Datacenter” level of our cluster (even if you can configure exceptions).

Creating the Cluster

Clustering Proxmox VE machines is a topic I’ve covered extensively on the blog, to the point that some of you know me mainly for these articles ;-):

So I won’t spend too much time on it. Just know that:

it’s easier to do since version 6 because there’s no longer a need to set up a VPN between servers (corosync v3)
it can be done from the command line
in any case, you need to log in as root with password when clustering

Very briefly then, we go to the first server, in the Server View at the top left, we select our Datacenter, then in the Cluster menu.

Click on the Create Cluster button.

Note: here, it’s simple, I only have one network card, so there’s no doubt about which interface to use. However, in the case of a server with more interfaces and more bridges (or VPNs as we had to do until PVE 6), you need to choose carefully which interface is used for clustering.

We validate the form:

From this point on, the server is no longer in “standalone” mode and has additional systemd services.

We notably have a /etc/pve folder that is synchronized between all nodes (the famous Proxmox Cluster File System) with the configuration of each node and the configuration files of virtual machines. We also have the corosync software, which will regularly contact all cluster nodes to ensure everyone is alive/reachable.

Joining the Cluster

If we go back to the “Cluster” menu, we now have information displayed about our cluster, namely its number of nodes (here just one, normal). Click on the Join information button.

A popup opens with several pieces of information to copy-paste that will be requested in the menu of the other server, the one that will join the cluster:

So we connect to the graphical interface of our second Proxmox VE server (my Atom in the example), we go to the same menu, but instead of creating a new cluster, we click on the Join Cluster button:

Normally a little scare, once the node joins the cluster, you’ll lose control of the web interface and get a “401” error message. This is “normal”, when the server enters the cluster, services are restarted and we switch from standalone mode to clustered mode…

Don’t panic, an F5 should fix the issue (or connect to the “first” server of the cluster).

We now have a nice cluster :)

Split Brain

Note however that in my example, we only have 2 nodes, which is more than discouraged when making a Linux cluster (actually, a cluster in general). If the network link ever goes down between our two machines, they’ll have no way of knowing if it’s a network problem (also known as split brain) or a node failure.

To avoid breaking everything by restarting VMs on both sides (the most catastrophic case in a split brain), when quorum is no longer reached, Proxmox VE puts nodes without a majority in (almost) “read-only” mode.

And in case it’s not clear, quorum is half the nodes +1, so here: 2/2+1 = 2… so as soon as we lose one of the two nodes, we lose quorum.

So if we ever cut a node or the communication between the two, your still active VMs will continue to work (I debated this on LinkedIn a few days ago).

However, it will be impossible to modify the cluster configuration, start or create VMs, etc.

The only thing we can possibly do that isn’t “read only” is stop VMs, because that doesn’t induce any risk for data consistency. But you won’t be able to turn them back on, you’ve been warned…

For the record, the LinkedIn debate was “does ProxmoxVE cut all VMs in case of split brain” and the answer is obviously no. However, most actions are indeed blocked.

In any case, to avoid this, the best thing is to have an odd number of nodes so that split brain can never happen, or possibly add a complementary vote using an instance serving only as an arbiter as explained in my old 2019 article or in the official documentation:

SDN Configuration

OK, cool, we have a two-machine cluster. But in the current configuration, the VM we created in the previous part can’t easily migrate from one server to another. Indeed, we attached it to a Linux bridge (vmbr0) that has its own network addressing plan.

In case of migration, we would potentially need to start the machine, change the IP address, review iptables/firewall rules. That’s what I was doing until now, and in the case of a DR plan for a lab infra and a personal blog, it’s “OK”.

A big new feature in version 8.1 is the activation of an “SDN” (Software Defined Network) module that has been in testing by developers and some early adopters since version 6.X, which will allow us to manage this centrally from the interface. If you want to read the official documentation, it’s here:

pve.proxmox.com/pve-docs/chapter-pvesdn.html

The only prerequisite we’re missing is dnsmasq for DHCP, and the documentation tells us to run the following commands on our machines:

apt update
apt install dnsmasq
# disable default instance
systemctl disable --now dnsmasq

And in the /etc/network/interfaces file, it’s also necessary to add at the end of the file the line (followed by a small systemctl restart networking)

source /etc/network/interfaces.d/*

In the Datacenter menu, open the SDN submenu. Note that SDNs are already declared, but these are actually the default networks we declared in previous articles:

In the SDN/Zones menu, add a “Simple” type zone. There are more complex zones that will be useful in production contexts with non-trivial LANs (VLAN, VXLan) but in our case, Simple is sufficient.

Note: don’t forget to check the advanced box to enable DHCP. YES we’re finally going to have DHCP for our virtual machines and containers.

Once you’ve validated, go to the SDN/VNets submenu, click the “Create” button to create a virtual network, then select it once created to create a subnet.

Once again, don’t forget to check the DHCP section, to declare the range for our future DHCP.

Once we’ve validated everything, we see that next to all these new things we’ve just created, there’s an icon with two yellow arrows and a “New” status.

To deploy these changes, we need to go back to the SDN menu from the beginning, and click Apply. Two “sdn1” lines (one for each server, actually) should appear.

Fun fact, the Proxmox VE SDN will add iptables rules that should look familiar if you’ve followed the previous articles:

#version:5
auto vnet1
iface vnet1
address 10.10.20.1/24
post-up iptables -t nat -A POSTROUTING -s '10.10.20.0/24' -o vmbr0 -j SNAT --to-source 203.0.113.1
post-down iptables -t nat -D POSTROUTING -s '10.10.20.0/24' -o vmbr0 -j SNAT --to-source 203.0.113.1
post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
bridge_ports none
bridge_stp off
bridge_fd 0
ip-forward on

And if now, I recreate another virtual machine, but this time instead of vmbr0, I choose vnet1, I don’t need to enter the IP address (since I have the SDN serving DHCP) and it has Internet access without additional configuration.

With this configuration however (SDN in Simple Zone mode), it still won’t be possible to have virtual machines communicate with each other. That will be possible with more complex configurations:

You can, for example, create a VXLAN overlay network on top of public internet, appearing to the VMs as if they share the same local Layer 2 network

Note: I’ve explored VXLANs since then, and I wrote a complementary article here.

Conclusion

I still haven’t finished this series, because I still have things to say about SDN replication and Proxmox VE firewalling, backups, monitoring, but once again, I’m already exceeding 12,000 characters. So it’s time to publish the article and start writing the next one.

And in the meantime, have fun :)

Deploying a Proxmox VE 8 Cluster on Dedicated Servers (2/3)

Mon, 17 Feb 2025 10:30:00 +0200

Note - this article is part of a series:

Picking Up Where We Left Off: First Steps in the UI

In the previous tutorial (which I invite you to read, if you haven’t already, otherwise this won’t be clear), we chose a dedicated physical server from a provider like OVHCloud or Hetzner. We then installed Proxmox VE in the latest version (8.x), configured some basic security options, added users and groups.

Now we can try connecting to the UI. The first thing that should strike you (💥 ouch!) is that the administration page is in HTTPS with a self-signed certificate. We’ll start by fixing that.

Something quite frustrating with Proxmox VE (but it’s probably for “good reasons” that I don’t know) is that some operations can’t be done in the UI if you connect with the admin account zwindler@pve that we created in the previous blog post.

This is typically true for updates, clustering, and configuration and… for certificate configuration (non-exhaustive list). So here, we’ll have to connect as root@pam for now and not as zwindler@pve…

Note that there are (for now) 2 “realms”, Linux PAM (for root) and Proxmox VE Auth (for admin).

In the right bar, we find our server, which is for now alone in its datacenter (we’ll come back to this later). There are several views in the Proxmox VE UI, some things are hard to find if you don’t have this in mind. Here, I’m in the “server view”, probably the most classic if you come from the VMware world. In this menu, you’ll find your hypervisors and we can interact with each of them, how they are configured.

Once our server is selected, we have the main menu displaying many menus. Select System / Certificates.

We’re in luck, Proxmox VE comes with a module to automatically order Let’s Encrypt certificates. If you have your own certificate that works too (button Upload Custom Certificate), of course, but I’ll assume you don’t have one, like me.

In the ACME section, click on Add ACME Account to register your email and accept the Let’s Encrypt Terms Of Service (TOS).

From there, add our server’s FQDN by clicking the Add button under ACME. The button was grayed out until we added an account.

Now we can finally order the certificate by clicking on Order Certificates Now. If everything goes well, the certificate will be correctly delivered by Let’s Encrypt, and the pve-proxy server (which serves the Proxmox VE UI frontend) should restart, and your web page refresh, with a proper certificate this time :).

Storage / ZFS

If you read my blog often and I tell you that I’m an unconditional fan of ZFS, you won’t be surprised.

And if you read the previous article carefully, you’ll remember that we reduced the default partition of the OVHcloud template in LVM to be able to create a nice ZFS partition. But if we go to the storage view, it’s unfortunately not visible.

Small UI subtlety, the menu to add storage is found at the Datacenter level and not in the server configuration itself. In theory, storage management must be identical on all servers in the same DC in Proxmox VE, even if we can override this principle and set up exceptions for many versions already.

In Datacenter / Storage, click Add, then ZFS in the dropdown menu:

On ZFS type storage, we can only store “Disk images” and “Containers”, which won’t help us if we also want to store ISOs, templates, or backups.

We can settle for storing them on the default “local” storage.

But we can also “cheat” by connecting to the server via SSH, creating a folder (/tank/data in my example), then creating a new Directory type storage that has no restriction on data types.

Let’s Talk About ZFS

I’ve banged my head countless times with people who persist in saying nonsense about ZFS, even after explaining it to them.

no, you don’t need ECC RAM to store things on ZFS. First, it’s a recommendation (probably a bit zealous) made by people who designed an extremely robust filesystem. Many production systems run with ZFS without ECC RAM. Second, it’s especially true if you do deduplication, because losing a block can have catastrophic consequences in such cases…
no, ZFS doesn’t require 1 GB of RAM per TB of disk, again, that’s only if you enable deduplication (to store the correspondence tables between hashes and blocks).

On the other hand, what is accurate is that by default, ZFS will try to use 50% of the available RAM on your server and use it as cache. In many administration software, including Proxmox VE, this value will be way too high for us, because RAM is a precious resource on a hypervisor.

That doesn’t mean this feature is useless. Having cache when you have unused RAM is always good. So, we’ll reduce this value to avoid resource conflicts between optimizing our storage performance and the number of VMs we can host.

# Restrict to 512MB
echo 536870912 | sudo tee -a /sys/module/zfs/parameters/zfs_arc_max

# Restrict to 4GB
echo 4294967296 | sudo tee -a /sys/module/zfs/parameters/zfs_arc_max

Note: this command will need to be persisted one way or another

Networking

Well… I tried to postpone as long as possible when we get to networking (because I hate it) but at some point, we’ll have to get to it…

By default, our Proxmox VE server is installed with a Linux bridge that will allow us to share network between our server’s physical interface and our virtual machines.

This setup works well on your local network with DHCP and your own local IPv4 LAN or if you have an IP range (whether v6 or v4) and something to assign them to virtual machines.

Unfortunately in my case, I only have one IPv4 (I could enable IPv6 but I’d have other issues and I prefer to keep it simple here). If I create VMs and assign them to the bridge, they won’t get an IP and won’t have Internet access.

There are several ways to connect virtual machines to the external network. If you have time, I recommend reading carefully the Proxmox VE Wiki which has a very good page listing the different possibilities.

pve.proxmox.com/wiki/Network_Configuration

In the previous articles on Proxmox VE 5 then Proxmox VE 6, we did complicated stuff with multiple bridges, DMZ, pfSense firewalling, and a hellish iptables script.

Just thinking about it, I’m already exhausted.

Fortunately, there has been a built-in firewall in Proxmox VE for several versions now, as well as an SDN that I’ve never taken the time to explore and I think this is the right tutorial to do it :).

So we’ll refer to the official documentation, and choose the “Masquerading (NAT) with iptables” mode, but with a simpler setup compared to before.

Danger Zone

Network interface configuration can theoretically be done directly from the UI rather than modifying configuration files (/etc/network/interfaces). In the server view, go to the System / Network menu:

Here, you can see I wasn’t making things up, and we do have a bridge with our physical interface (here enp1s0).

(We even have an IPv6 range, oh my 🙈 I have no excuse… ANYWAY, let’s pretend we didn’t see that.)

We’re now at the fun part where we can easily cut ourselves off. I’ve done it often enough and had to go into rescue mode…

fun fun fun fun

The idea here is to remove the interface from the bridge, configure the network directly on it, and give a local network for our VMs on the bridge, while adding iptables rules for masquerading.

As long as we don’t click the Apply Configuration button, we’re not “at risk”.

The problem with the UI is that we’ll only be able to modify the IPs, gateways, and interfaces on the bridge or not. But to do the masquerading and have our virtual machines access the Internet, we need to add the famous iptables rules I’m talking about just before and we can’t do it from the UI.

We’ll need to add the last five lines in the vmbr0 configuration…

cat /etc/network/interfaces
[...]
auto vmbr0
iface vmbr0 inet static
 address 10.10.10.1/24
 bridge-ports none
 bridge-stp off
 bridge-fd 0

 # add for masquerading to work
 post-up echo 1 > /proc/sys/net/ipv4/ip_forward
 post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o enp1s0 -j MASQUERADE
 post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o enp1s0 -j MASQUERADE
 post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

Warning: the most important point is to verify that the output interface (-o) has the correct name. Here, my physical interface connected to the Internet is enp1s0, but if yours has a different name, you need to adapt.

From my point of view, the UI is still not completely useless, because it allows us to make the most delicate change (IP swap) with the visual diff and the “apply config” button. The post-up scripts are unlikely to cut off our access.

First Test

We’re far from done (it won’t fit in this article anyway, so might as well rush through…). But I’m sure you’re eager to launch a VM and we should have something functional, though incomplete.

So we’ll do a quick test of our setup to verify that we have a minimally functional hypervisor (meaning without clustering, without SDN, without firewall, without monitoring, without backup… without anything, really).

I started playing around with Talos Linux as a Linux distribution for Kubernetes. To keep it simple, I’ll launch the install of a Talos Node. If it manages to enroll in the Omni interface, it means I have a functional hypervisor, network included.

Once the ISO is generated on my Omni site (the SaaS control plane from Sidero Labs, the publisher of Talos Linux), we can push it to our Proxmox VE.

To do this, from our server (so not at the Datacenter level), we select a storage pool that accepts ISO images (if you remember correctly, our ZFS pool doesn’t support it by default), choose ISO Images, and click Upload to upload the ISO.

Once uploaded, we can now click the blue button at the top right Create VM.

From there, the VM creation wizard should take over. It’s going to be a bit verbose, but all menus are useful when you start to know Proxmox VE well. I’ll give you the minimum though:

In the first menu, we won’t choose which node to install the VM on since for now we only have one… However, we’ll have to give it a unique ID (100 by default) for the entire cluster. Ideally, we also give it a little name. I opted for talos02 (because I already have a talos01 on another server, see how original I am?).

Finally, I checked the “Start at boot” box, that’s the kind of annoying thing when you forget to set it and the hypervisor reboots (it happens)…

You can give tags to your VMs, they’ll have nice colored badges to distinguish them :)

In the second menu (OS), we need to indicate two things. First, that we want to use the ISO we just retrieved/uploaded to boot our machine (you need to choose the right storage pool, by default “local” is selected). We also need to give the OS type for driver compatibility reasons for virtual devices. It’s been a long time since we really needed that, “6.X - 2.6” works for all recent Linux (kernel 2.6 release was in 2003!).

I skip System, in Disks, we add a disk on the right pool (so not local, but the one on ZFS). For my 32 GB Talos VM, that’s more than enough… Note that there are many important options here for everything related to read/write “optimization”, but we’re well beyond the scope of this article.

I skip CPU and memory, the important thing is to give it at least 2 cores and 2 GB of RAM (at least). You can also optimize CPU performance by enabling the right flags, but again, we’re beyond the purpose of the article so I won’t insist.

In the network section, we just make sure we’re connected to the right bridge (normally if you followed the article, we only have one, vmbr0, so we can’t go wrong).

And we verify everything is OK. Protip, to save five seconds in your life, click “Start after created”, which will start the VM as soon as possible after creation.

It’s Alive?

At this point, the VM should automatically boot from the Talos CD and the OS should install on the disk. Once ready, the machine will try to enroll on my Omni control plane.

And that’s when disaster strikes…

Because yes, I haven’t configured DHCP or SDN. So we have to manually configure the virtual network card (which isn’t great), by pressing F3 in the console.

(And suffering a bit of QWERTY, but that’s how you recognize a good sysadmin normally: they can do QWERTY in consoles)

Once the network is configured, the machine should almost instantly start configuring itself…

… and enroll in Omni:

Victory!!

Well…

Well… yes, we have a functional VM. But I aspire to go a bit further before really declaring victory.

As I said above, we had to configure our VM’s network by hand, and we don’t yet have clustering, firewalling, SDN, monitoring, or backup… But we’re comfortably scratching 16,000 characters for this blog post, so it’s time to hang up and say “see you next time”.

And in the meantime, have fun :)

Deploying a Proxmox VE 8 Cluster on Dedicated Servers (1/2)

Mon, 27 Jan 2025 20:00:00 +0000

Note - this article is part of a series:

Introduction

Those who follow the “lore” of this blog know that I’m somewhat known in the Sunday tinkerer community for having written (or co-written) several tutorials on Proxmox VE. I’m thinking in particular of two article series on Proxmox VE 5, then Proxmox VE 6, to create a virtualization cluster from scratch with Proxmox VE (and pfSense at the time).

This is a new iteration of these tutorials, this time based on Proxmox VE 8, with quite a few things changing and new approaches.

The ultimate goal of this article series is to have a virtualization platform with Proxmox VE. This platform will be composed of several machines (ideally 3, but 2 works too), secured, monitored, highly available, and backed up.

You’ll need to follow several articles to achieve this.

Starting Assumptions

1- I’m assuming you don’t have machines to do this at home and don’t want or can’t afford to invest in a homelab (even though it can be a lot of fun, it requires investment). However, if you have machines at home, a large part of this article remains valid, once the machines are installed.

2- Since Proxmox VE is a virtualization platform, I’m also assuming you want dedicated (physical) machines and not virtual machines, precisely to be able to virtualize entire OSes without too many problems (even though we always have the “nested virt” option, it’s less ideal).

Note: if you don’t need virtualization (example: no Windows or special OS), Proxmox VE is a great containerization platform, particularly thanks to LXC, which allows you to create Linux containers with a full OS, that you can manage exactly like a VM but with much better performance (and some limitations due to kernel sharing, but it’s often sufficient). You can read my articles on LXC if you’re interested.

3- I’d like to draw your attention to the fact that many actions could have been scripted or managed via “infrastructure as code”. It’s more in tune with the times, because it’s more robust and reliable. However, the point of this article is to do it “by hand” to understand what we’re doing, step by step, not just provide you with a “turnkey” infrastructure that you won’t know how to manage at the first hiccup.

If you’re looking for more industrial methods to deploy virtualization clusters, you’ll surely find plenty on the Internet, from other bloggers (some French, some are even friends).

I’ve actually done the exercise myself several times with Ansible or Rudder in the past, for example here (note: this code is obsolete):

Now that the objective and context is clear for everyone, we can begin :)

Choosing the Machines

Since we said we’re not self-hosting, we need to find a host that offers physical machines. Ideally, something not too expensive would be nice to start with.

So we’re going to look for inexpensive machines BUT with support for VT-x / AMD-v instructions, which is pretty much all CPUs that still work today EXCEPT Atoms and ARM machines like Raspberry.

Basically, that leaves us with Kimsufi at OVHCloud, Dedibox at Scaleway, and machines at Hetzner.

At Hetzner for €45, you get an i5 gen13, 64 GB of RAM and 2 NVMe SSDs.

At OVHcloud, with a bit of luck, you can find machines around €15-20 per month with older hardware, but more than sufficient for what we’re going to do.

At Scaleway, the entry-level offer is centered on Atom Avoton or very old Xeons. The Avoton C2750 remains a valid option, as it’s an octa-core that supports VT-x (unlike the C2350, beware!). But well, it’s 2013 hardware 💀…

For the purposes of this article, I optimized costs and went with the OVHcloud server I present just above (KS-GAME-LE), unfortunately only available in Canada :(. But 8 threads and 16 GB of RAM for €12 including tax per month, that’s really not expensive.

Two downsides:

ridiculously small storage for virtualization (only 240 GB, a single disk)
latency - almost 100 ms from my place, compared to 11 ms for my other machine at Scaleway

64 bytes from proxmox.example.org (203.0.113.78): icmp_seq=1 ttl=53 time=98.0 ms
64 bytes from proxmox.example.org (203.0.113.78): icmp_seq=2 ttl=53 time=97.5 ms
64 bytes from proxmox.example.org (203.0.113.78): icmp_seq=3 ttl=53 time=97.6 ms
64 bytes from proxmox.example.org (203.0.113.78): icmp_seq=4 ttl=53 time=97.7 ms
64 bytes from proxmox.example.org (203.0.113.78): icmp_seq=5 ttl=53 time=97.7 ms

OS Installation

Once ordered, the server is relatively quickly available in your “OVH manager”. We have a fairly intuitive menu to directly install Proxmox VE 8, almost up to date (the provided images are rebuilt very regularly with all updates).

To start the installation, click on the “…” in the Operating System (OS) section, then select install from an OVH template

Scroll through the list of OS types, select Virtualization, then Proxmox VE 8 (while we’re at it). I also checked the “Customize partition configuration” box, because by default, OVH’s installation uses “LVM thick” type storage for most of the disk space, while we’ll use advanced ZFS features later:

Before validating the installation, don’t forget to change its name (Custom hostname) to avoid having to do it post-install, then add a public SSH key:

First Post-Installation Tasks: Upgrade and Reboot

There’s a whole series of things to do when you’ve just installed a machine, especially on Proxmox VE, and we’ll try to resist the urge to “quickly” connect to the graphical interface.

Anyway, we won’t be able to connect to the Proxmox VE graphical interface…

Really???

Yes, because while we can normally connect with the root user of our Linux, we won’t be able to in the case of OVHcloud, simply because we DON’T have the root password. Good thing we added an SSH key ;)

We’ll come back to it later, we have more urgent things to do.

The first thing I did was immediately add a type A DNS record so that my machine’s IP corresponds to an FQDN. For this article, I’ll refer to it as follows:

hostname: myPVEhost
FQDN: proxmox.example.org
IP: 203.0.113.159

So we connect to our freshly installed machine’s IP via SSH (root@IPmachine) and we’ll take the opportunity to do some cleanup.

The first thing to do is update the server, ideally followed by a reboot:

apt update && apt upgrade -y && reboot

Once the server is back to life, we reconnect and enable automatic updates in one way or another. The simplest is to use unattended-upgrades if it’s not installed by default (it may already be there) but there are more sophisticated tools.

apt install unattended-upgrades

It’s for Security

Once that’s done, I recommend installing CrowdSec right away. Here, I simply based myself on the official crowdsec documentation, to add the detection engine, the remediation component (here via iptables):

CrowdSec.net - Linux Installation

curl -s https://install.crowdsec.net | sudo sh
apt install crowdsec

# verify that the crowdsec service is working
systemctl status crowdsec
● crowdsec.service - Crowdsec agent
 Loaded: loaded (/lib/systemd/system/crowdsec.service; enabled; preset: enabled)
 Active: active (running) since Sun 2025-01-26 21:04:26 UTC; 1min 56s ago
 Process: 73830 ExecStartPre=/usr/bin/crowdsec -c /etc/crowdsec/config.yaml -t -error (code=exited, status=0/SUCCESS)
 Main PID: 73862 (crowdsec)
 Tasks: 13 (limit: 19059)
 Memory: 33.1M
 CPU: 2.571s
 CGroup: /system.slice/crowdsec.service
 ├─73862 /usr/bin/crowdsec -c /etc/crowdsec/config.yaml
 └─73877 journalctl --follow -n 0 _SYSTEMD_UNIT=ssh.service

Jan 26 21:04:23 myPVEhost systemd[1]: Starting crowdsec.service - Crowdsec agent...
Jan 26 21:04:26 myPVEhost systemd[1]: Started crowdsec.service - Crowdsec agent.

# install the remediation engine
apt install crowdsec-firewall-bouncer

Once Crowdsec is operational, we can add the “fulljackz/proxmox” collection, which we’ll need to modify (thanks to Julien Louis’s great article on his slash-root.fr blog), which will listen to the webUI logs to detect any brute force and ban those who try:

cscli collections install fulljackz/proxmox
[...]
INFO Enabled fulljackz/proxmox
INFO Run 'sudo systemctl reload crowdsec' for the new configuration to be effective.

The pattern listened to by the proxmox parser is no longer correct with new versions

Jan 26 21:23:14 myPVEhost pvedaemon[1250]: authentication failure; rhost=::ffff:203.0.113.159.78 user=coucou@pve msg=no such user ('coucou@pve')

We see here that Julien added the “::ffff:” prefix in the PVE_AUTH_FAIL variable of the /etc/crowdsec/parsers/s01-parse/proxmox-logs.yaml file, but when I look at the test cases provided in the collection, it seems like it’s not necessary…

In any case, we’ll create a /etc/crowdsec/acquis.d/proxmox.yaml file so that CrowdSec can start analyzing our logs:

mkdir -p /etc/crowdsec/acquis.d/
cat > /etc/crowdsec/acquis.d/proxmox.yaml <<EOF
journalctl_filter:
 - _SYSTEMD_UNIT=pvedaemon.service
labels:
 type: syslog
EOF

We finish by reloading crowdsec and watching ~~the world burn~~ script kiddies get blocked.

systemctl reload crowdsec

cscli alerts list
╭────┬────────────────────┬───────────────────────────┬─────────┬──────────────────────────────────────────────────────────────┬───────────┬─────────────────────────────────────────╮
│ ID │ value │ reason │ country │ as │ decisions │ created_at │
├────┼────────────────────┼───────────────────────────┼─────────┼──────────────────────────────────────────────────────────────┼───────────┼─────────────────────────────────────────┤
│ 2 │ Ip:203.0.113.222 │ crowdsecurity/ssh-slow-bf │ IR │ 202468 Gloubi Bo ulgua Co. ( Private Joint Stock) │ ban:1 │ 2025-01-26 22:31:15.366187524 +0000 UTC │
╰────┴────────────────────┴───────────────────────────┴─────────┴──────────────────────────────────────────────────────────────┴───────────┴─────────────────────────────────────────╯

Among the little annoying things shipped by default with Proxmox VE, there used to be the fact that Proxmox VE had enterprise apt repositories on all installations, and it was necessary to manually disable them before being able to do updates.

It was super annoying and the Proxmox VE developers heard (a little) their community and removed this requirement… to replace it with a popup.

Now, a popup, I’m fine with that! It’s important to reward developers’ work and support them. But is it really necessary to do it at every login AND every time we refresh the list of packages to update?

This is obviously a rhetorical question.

Obviously, many people have found several methods to disable the responsible JS code. Big up to fabio for his version that works well.

sed -Ezi.bak "s/(function\(orig_cmd\) \{)/\1\n\torig_cmd\(\);\n\treturn;/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service

Note: if like me you’ve already connected once, you’ll also need to clear the cache otherwise the popup will remain…

Creating Groups and Users for the Graphical Interface

Ok ok, we’ve done a lot of stuff, can we connect now???

Yes yes, we can. Anyway, I’m already at almost 15,000 characters, this article is way too long, we need to wrap up.

Last step in this article, we’ll create an administrator to connect to the UI and a user that we’ll use later for monitoring.

# you're not **really** obligated to call the admin "zwindler" you know?
pveum group add admin -comment "System Administrators"
pveum acl modify / -group admin -role Administrator
pveum useradd zwindler@pve
pveum usermod zwindler@pve -group admin
pveum passwd zwindler@pve

# our monitoring user
pveum groupadd monitoring -comment 'Monitoring group'
pveum aclmod / -group monitoring -role PVEAuditor
pveum useradd pve_exporter@pve
pveum usermod pve_exporter@pve -group monitoring
pveum passwd pve_exporter@pve

Alright, you’ve waited long enough, you can connect to your Proxmox VE! Go to:

https://@IPmachine:8006 or
https://proxmox.example.org:8006

Note: for now, the certificate is self-signed and the browser will show an error. We can accept to ignore the problem for now.

Don’t forget to change the “realm” from PAM (unix users, which we’re not using here) to “Proxmox” to be able to use the administrator we just created.

See you soon for the next part. And in the meantime, have fun!

Migrating Cilium routing from iptables to eBPF... hot!

Fri, 20 Oct 2023 06:00:00 +0200

Context

It could be that I configured production Kubernetes clusters with cilium in iptables mode and not eBPF mode. But you have no proof…

However, in the highly unlikely hypothesis that I could have made such a blunder, here’s how I would have gone about solving the problem.

#trollface

So let’s imagine that when checking the Cilium configuration, here’s what you found:

kubectl -n cilium exec -it cilium-aaaaa -- cilium status
[...]
Host Routing: Legacy
Masquerading: IPTables [IPv4: Enabled, IPv6: Disabled]
[...]

Damn!

Looking more closely, the cilium-agent containers in your cilium pods are using a lot of CPU and RAM and are starting to clog up your workers…

This is a disaster.

Why?

The first implementations of the Kubernetes imaginary network (the famous CNI plugins) used iptables. However, we’ve known for a very long time, especially in the Kubernetes use case, that iptables is pretty bad at handling large quantities of rules.

In the particular case of Kubernetes, the number of rules tends to increase exponentially with cluster size, and the CPU consumed to route network packets with it…

At some point, traversing the rules list takes all the CPU of a given node and makes it unresponsive.

Hello boss? We’re in trouble.

This is one of the reasons why I really like cilium as a CNI plugin - the developers were among the first to bet on eBPF as a replacement for iptables (although there are other implementations / other technologies that solve this problem).

How did we get here?

To be honest, I just read the docs and trusted them…

We introduced eBPF-based host-routing in Cilium 1.9 to fully bypass iptables and the upper host stack, and to achieve a faster network namespace switch compared to regular veth device operation. This option is automatically enabled if your kernel supports it. To validate whether your installation is running with eBPF host-routing, run cilium status in any of the Cilium pods and look for the line reporting the status for “Host Routing” which should state “BPF”.

Basically, according to the official docs, if you have the right kernel and the right modules, the cilium installation is supposed to automatically enable eBPF mode… Except that we can clearly see above that this isn’t the case, despite a recent kernel (6.2).

Digging a little deeper into the logs, here’s what you can find:

kubectl -n cilium logs cilium-7b5cp
[...]
level=info msg="BPF host routing requires enable-bpf-masquerade. Falling back to legacy host routing (enable-host-legacy-routing=true)." subsys=daemon

Well… apparently, an option is missing in the Helm chart.

[...]
 kubeProxyReplacement: strict
+ bpf:
+ masquerade: true
[...]

Problem solved, end of article?

Problem

Yes, of course, we can modify the chart like a bull in a china shop and end of story.

But let’s say we don’t want to cut production traffic… How do we do it?

The cleanest solution that comes to mind is the following:

we drain a node
we change its configuration
we uncordon it to put some traffic back on it
we check that the new configuration works AND
we check that nodes can talk to each other between those with iptables and those with eBPF

Because yeah, it would be a shame to have half the cluster unable to communicate with the other half…

Checking connectivity

What’s cool about cilium (did I tell you I love cilium?) is that we already have tooling to test everything.

The cilium CLI has a cilium connectivity test subcommand that launches dozens of internal/external tests to check that everything is OK.

➜ ~ cilium -n cilium connectivity test
ℹ️ Monitor aggregation detected, will skip some flow validation steps
✨ [node] Deploying echo-same-node service...
✨ [node] Deploying DNS test server configmap...
✨ [node] Deploying same-node deployment...
✨ [node] Deploying client deployment...
[...]
✅ All 32 tests (265 actions) successful, 2 tests skipped, 0 scenarios skipped.

There’s also a cilium-health command, embedded in the cilium-agent container, which allows you to get periodic latency statistics between all the cluster nodes. Useful!

kubectl -n cilium exec -it cilium-ccccc -c cilium-agent -- cilium-health status
Probe time: 2023-09-12T14:47:03Z
Nodes:
 node-02 (localhost):
 Host connectivity to 172.31.0.152:
 ICMP to stack: OK, RTT=860.424µs
 HTTP to agent: OK, RTT=110.142µs
 Endpoint connectivity to 10.0.2.56:
 ICMP to stack: OK, RTT=783.861µs
 HTTP to agent: OK, RTT=256.419µs
 node-01:
 Host connectivity to 172.31.0.151:
 ICMP to stack: OK, RTT=813.324µs
 HTTP to agent: OK, RTT=553.445µs
 Endpoint connectivity to 10.0.1.53:
 ICMP to stack: OK, RTT=865.976µs
 HTTP to agent: OK, RTT=3.440655ms
[...]

And finally, we can just look at the cilium status command which tells us who is “reachable” (again, in the cilium-agent container)

➜ kubectl -n cilium exec -ti cilium-ddddd -- cilium status
[...]
Host Routing: Legacy
Masquerading: IPTables [IPv4: Enabled, IPv6: Disabled]
[...]
Cluster health: 5/5 reachable (2023-09-14T12:01:51Z)

Changing a node’s configuration

We’re in luck, because since cilium version 1.13 (the latest to date is 1.14), it’s possible to apply different configurations to a subset of nodes (official documentation for Per-node configuration).

Note: before that, it was still possible, temporarily, by manually editing the cilium ConfigMap, then restarting the concerned pods for it to take effect.

There’s now a CRD to do this, called CiliumNodeConfig. The only things we have to do are add a label to a node (io.cilium.enable-ebpf: “true”) and add the following manifest to our cluster:

cat > cilium-fix.yaml << EOF
apiVersion: cilium.io/v2alpha1
kind: CiliumNodeConfig
metadata:
 namespace: cilium
 name: cilium-switch-from-iptables-ebpf
spec:
 nodeSelector:
 matchLabels:
 io.cilium.enable-ebpf: "true"
 defaults:
 enable-bpf-masquerade: "true"
EOF

kubectl apply -f cilium-fix.yaml

kubectl label node node-05 --overwrite 'io.cilium.enable-ebpf=true'

If you’re in production, the cleanest approach is to kubectl drain the Node beforehand.

Out of curiosity, I still tried to do it “hot”, for fun.

I deployed a daemonset containing a V(lang) app that simply returns the container name in a web page:

cat > vhelloworld-daemonset.yaml << EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
 name: vhelloworld-daemonset
spec:
 selector:
 matchLabels:
 app: vhelloworld
 template:
 metadata:
 labels:
 app: vhelloworld
 spec:
 containers:
 - name: vhelloworld
 image: zwindler/vhelloworld:latest
 ports:
 - containerPort: 8081
 imagePullPolicy: Always
EOF

kubectl apply -f vhelloworld-daemonset.yaml

kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
vhelloworld-daemonset-q4h44 1/1 Running 0 3m31s 10.0.4.87 nodekube-05 <none> <none>
vhelloworld-daemonset-w97pv 1/1 Running 0 3m31s 10.0.3.85 nodekube-04 <none> <none>

Then I created containers with a shell and curl to periodically check from multiple nodes that the containers were accessible:

kubectl run -it --image curlimages/curl:latest curler -- /bin/sh
If you don't see a command prompt, try pressing enter.
~ $ curl http://10.0.4.87:8081
hello from vhelloworld-daemonset-g2zdw
~ $ curl http://10.0.3.85:8081
hello from vhelloworld-daemonset-65k2n

~ $ while true; do
 date
 curl http://10.0.4.87:8081
 curl http://10.0.3.85:8081
 echo
 sleep 1
done

Once the label was applied to a node and its cilium pod killed (via kubectl delete), the pods remained accessible:

Fri Sep 15 14:05:34 UTC 2023
hello from vhelloworld-daemonset-g2zdw
hello from vhelloworld-daemonset-65k2n

Fri Sep 15 14:05:35 UTC 2023
hello from vhelloworld-daemonset-g2zdw
hello from vhelloworld-daemonset-65k2n

Fri Sep 15 14:05:36 UTC 2023
hello from vhelloworld-daemonset-g2zdw
hello from vhelloworld-daemonset-65k2n

At some point, cilium restarted, noticed the presence of existing pods, and took over with eBPF!

evel=info msg="Rewrote endpoint BPF program" containerID=8b7be1b032 datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=1018 identity=4773 ipv4=10.0.3.85 ipv6= k8sPodName=default/vhelloworld-daemonset-w97pv subsys=endpoint
level=info msg="Restored endpoint" endpointID=1018 ipAddr="[10.0.3.85 ]" subsys=endpoint

Does it work, but is it better in terms of resource consumption?

That was the good news. I was expecting gains because the cluster was really close to the dreaded death by iptables.

The cilium containers were using 10% of my nodes’ CPU, and several GB of RAM in iptables mode. It could quickly have gone much higher if I had grown the cluster beyond 50 nodes / 2000 pods.

Switching to eBPF mode allowed us to return to totally painless levels (1% CPU per node, 1-2% RAM) compared to my machines’ configurations.

Not bad, huh?

Bonus

A really hairy tutorial to fully migrate CNI hot (from flannel to cilium)

Running Docker containers with Proxmox VE (and LXC)

Mon, 07 Nov 2022 06:30:00 +0200

Introduction

I found several tutorials to install Docker (engine) inside an LXC container under Proxmox VE to then run Docker containers inside it. The problem is that you have to log into the LXC container before you can interact with your Docker containers.

You can also install Docker directly on Proxmox VE but it’s not really recommended and you don’t have the ease of use of LXC with Proxmox GUI.

And actually, it turns out you can run Docker images as the OS of an LXC container and thus have Docker controlled by Proxmox. So we won’t have 100% identical behavior to if you had done a docker run on your machine.

However, we’ll gain in exchange a container totally identical to your other LXC containers in your cluster (with all associated functions) but launched with a Docker image.

But first I suggest we go back a bit before showing you how…

Context

You know, I’ve been doing Proxmox VE for a while. Proxmox VE is a great turnkey and production-ready distribution for server virtualization (it’s even hyperconverged if you enable the Ceph part).

There are lots of little things I like about Proxmox VE, and one of them is that we can create Linux containers with LXC rather than QEMU virtual machines. I wrote an article to talk about it.

In use (except for specific kernel configuration), the LXC container behaves strictly identically to a VM in Proxmox, but for a fraction of the resources! We have an OS, we install applications on it, like a VM.

An nginx frontend for several applications. The complete LXC container only consumes a few MB

Admittedly it’s much less isolated than a real VM (because it’s a container, we’re isolated from other processes, but we run on the hypervisor’s kernel directly).

But with so few resource needs, I can host many small applications in different contexts with very small physical machines (Atom 4GB RAM at €6 per month at OneProviders) which would be impossible with a real VM.

It’s good, but not enough

However, LXC is not the most known/hyped containerization technology. For years on the forum, whenever people ask for Docker support in Proxmox VE, they get sent away. Not always very nicely either…

Yet, there are several reasons to want to run Docker containers on your Proxmox infrastructure:

First, Docker isn’t as much stateless and/or ephemeral workloads as we think. Many containerized applications would have their place on a virtualization cluster.
Then because many software vendors have become lazy. Some no longer manage installation processes (or only on one OS) and just provide a Docker image that they maintain up to date.

So we have to settle for VMs in Proxmox to run Docker containers (see my point at the beginning).

LXC supports OCI images

Plot twist. A few days ago, I discovered that LXC, Proxmox VE’s containerization engine, was compatible with the OCI format.

buzzwrd.me - Creating LXC containers from docker and OCI images

For those who don’t know OCI, it’s an organization aiming to create a standard to unify the way containers are stored.

The Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes. opencontainers.org/

And Docker respects this format: we can therefore theoretically run from LXC containers whose base image is a Docker image.

So, when do we start?

Disclaimer: what follows is tinkering. Nothing is supported and I don’t recommend it in production. There are also likely limitations (especially regarding storage).

First, some dependencies are missing for LXC’s OCI function to be usable under Proxmox VE:

sudo apt install skopeo umoci jq

Then, by default, LXC containers attach to a Linux bridge called lxcbr0. However this bridge probably doesn’t exist on your Proxmox VE installation (not installed by default).

In quick and dirty, we can just replace the default name, although ideally we should find the flag in the CLI to change the bridge. Depending on how you configured the network in your Proxmox VE server, you’ll probably use vmbr0 or vmbr1:

sed -i 's/lxcbr0/vmbr1/g' /etc/lxc/default.conf

Now that we have everything, we can run our LXC containers from docker images. Here I’m running the official alpine:latest image, but I could have taken any other:

lxc-create 500 -t oci -- --url docker://alpine:latest

I created a container called “500” (because VMs and LXC containers in proxmox VE are called with numbers). The image is downloaded then the container created:

We can now start it (with lxc-execute we run the container and open a prompt directly inside it, but we also have lxc-start and lxc-attach)

lxc-execute 500

As is, Proxmox VE doesn’t know our container exists and it won’t appear in the UI. But it works ;-).

Bonus

We can even cheat and make Proxmox believe it’s an LXC container it created itself by creating a 500.conf file, in the /etc/pve/lxc/ folder

The only mandatory fields are the following:

arch (amd64 unless you’re on ARM)
cores (the CPU limit to impose on the container)
hostname
memory (the RAM limit to impose on the container)
ostype
rootfs

Once the file is created, the container automatically appears in the host’s VM list. We can start it and it works :)

The container can be started, stopped, etc. All from the Proxmox VE GUI :)

Have fun!

OS on Zwindler's Reflection

Kubernetes UserNamespaces: the overhyped GA feature

The infographic that triggered me

What UserNamespaces actually do

What the infographic exaggerates (and leaves out)

Honest comparison: UserNS vs the real alternatives

Real use cases

1. Build-as-a-Service (Buildah, rootless Podman)

2. Hostile multi-tenancy (user code platforms)

3. Hard-coded legacy (Postfix, Dovecot, BIND)

Some counterarguments

Where to invest your security budget

Conclusion

References

Installing Alpine Linux in Headless Mode on Raspberry Pi

Introduction

Why Alpine Linux?

Downloading the Alpine Linux Image for Raspberry Pi

Headless Configuration with Overlay File

First Boot and Permanent Installation

GPU RAM Optimization

Solutions to Minimize GPU RAM

Swap Configuration with zswap

Creating the Swapfile

Persistent Configuration

Conclusion

Proxmox VE 8 Tutorial - SDN in VXLAN Mode with Machines on the Internet

Preamble

Prerequisites

Problem

VXLAN Then, But with a VPN

SDN Configuration

Is That All?

Deploying a Proxmox VE 8 Cluster on Dedicated Servers (3/4)

Picking Up Where We Left Off: First Steps in the UI

Creating the Cluster

Joining the Cluster

Split Brain

SDN Configuration

Conclusion

Deploying a Proxmox VE 8 Cluster on Dedicated Servers (2/3)

Picking Up Where We Left Off: First Steps in the UI

Storage / ZFS

Let’s Talk About ZFS

Networking

Danger Zone

First Test

It’s Alive?

Well…

Deploying a Proxmox VE 8 Cluster on Dedicated Servers (1/2)

Introduction

Starting Assumptions

Choosing the Machines

OS Installation

First Post-Installation Tasks: Upgrade and Reboot

It’s for Security

Nagging Popup

Creating Groups and Users for the Graphical Interface

Migrating Cilium routing from iptables to eBPF... hot!

Context

Why?

How did we get here?

Problem

Checking connectivity

Changing a node’s configuration

Does it work, but is it better in terms of resource consumption?

Bonus

Running Docker containers with Proxmox VE (and LXC)

Introduction

Context

It’s good, but not enough

LXC supports OCI images

So, when do we start?

Bonus

Source