Virtualisation on Zwindler's Reflection

Proxmox VE 8 Tutorial - SDN in VXLAN Mode with Machines on the Internet

Sun, 30 Mar 2025 12:30:00 +0200

Preamble

This tutorial is a kind of special edition in my series of articles on Proxmox VE 8.

In the third part which came out a few days ago, we stopped at a cluster of remote machines on the Internet.

They’re not in the same LAN, and to be honest, they’re even about 100ms apart (and it works very well).

At the end of the article, it works, we have a single network on our two machines, with a functional IPAM and DHCP on both sides. But still, virtual machines on different hosts can’t talk to each other directly and it’s still a bit unfortunate not to have gone all the way.

Rather than rewriting that part, I decided to make a small aside where I finish the work and test the VXLAN function of the Proxmox VE SDN, because YES, it can work all the way.

Prerequisites

I’m assuming you already have two (or more) machines within the same Proxmox VE 8 cluster.

If not, you’re in for reading my series of articles (currently being written) on the subject (have fun!):

Deploying a Proxmox VE 8 Cluster on Dedicated Servers (1/4)

These machines are not in the same LAN (otherwise no need to bother with VXLAN, the Simple SDN can be enough, as we did in part 3)

Problem

What we want to do is create an SDN that goes through the Internet, to the point that virtual machines believe they’re on the same LAN.

We can’t do this with Simple SDN, VMs on one host can’t contact VMs on another distant host on the Internet, even being in the same Simple SDN and with the same addressing plan.

But if we read a bit further in the PVE documentation on SDN, we see that this is precisely the use case highlighted for using VXLAN type SDNs:

The VXLAN plugin establishes a tunnel (overlay) on top of an existing network (underlay). This encapsulates layer 2 Ethernet frames within layer 4 UDP datagrams… […] You can, for example, create a VXLAN overlay network on top of public internet, appearing to the VMs as if they share the same local Layer 2 network

Small issue however, in reality, you’d better not do it, because frames are not encrypted, and on the Internet, it’s really not the best idea in the world…

Warning VXLAN on its own does does not provide any encryption. When joining multiple sites via VXLAN, make sure to establish a secure connection between the site, for example by using a site-to-site VPN.

As an alternative, we could have gone directly with the most complete/complex version of the Proxmox VE SDN: EVPN Zones

The EVPN zone creates a routable Layer 3 network, capable of spanning across multiple clusters. This is achieved by establishing a VPN and utilizing BGP as the routing protocol.

But I don’t want to mess with BGP here and I want to keep things simple. However, it’s probably the “cleanest” option for a production setup.

VXLAN Then, But with a VPN

I won’t overthink it, we’ll set up a small VPN with wireguard manually. There are tons of VPN tools on the market and tons of tools to make life easier. If you want more info on Wireguard, I’m sure you’ll find plenty, I’m just putting the bare minimum here so it “just works”.

On both servers, install wireguard, which should be in the default debian repositories:

apt update && apt install -y wireguard

Generate a key pair on each machine:

wg genkey | tee privatekey | wg pubkey > publickey

A little mental gymnastics, we’re going to write a configuration file with for each server, the private key of the server (file privatekey) we’re on, but the public key (file publickey) of the remote server.

(When you think about it, it’s obvious, but I prefer to insist)

And you obviously need to change addresses and network interfaces depending on which server you’re on so as not to have conflicts.

vi /etc/wireguard/wg0.conf
[Interface]
Address = 10.10.30.1/24
ListenPort = 51820
PrivateKey = <local server private key>
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT
PostUp = iptables -t nat -A POSTROUTING -o <network interface name> -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT
PostDown = iptables -t nat -D POSTROUTING -o <network interface name> -j MASQUERADE
[Peer]
PublicKey = <remote server public key>
Endpoint = <remote server public IP>:51820
AllowedIPs = 10.10.30.2/32

Note: you can also add more nodes to the VPN, just add more “Peer” sections.

Finish by starting the VPN and verifying that nodes talk to each other:

systemctl enable wg-quick@wg0 --now

ping 10.10.30.1
ping 10.10.30.2

SDN Configuration

Normally, if you followed the previous articles, you already have the prerequisites, but for those who haven’t, I’ll repeat that in the /etc/network/interfaces file, it’s necessary to add at the end of the file the following line (followed by a small systemctl restart networking)

source /etc/network/interfaces.d/*

Then, in the Datacenter menu, open the SDN submenu.

In the SDN/Zones menu, add a “VXLan” type zone. Two important points here:

we need to fill in the list of all “peers”, here the IPs of our VPNs so 10.10.30.1 and 10.10.30.2
we’ll need to adjust the MTU. By default, a frame is 1500. Except that wireguard for its encapsulation has reduced the MTU to 1420 and VXLAN encapsulation takes 50. So we need to set 1370 to best avoid fragmentation, at worst, dropped packets

If you just read the previous article, you’ll notice that here, there’s no “checkbox” to enable DHCP on the zone. Unfortunately, that’s because it’s not available yet.

Currently only Simple Zones have support for automatic DHCP

Source: pve.proxmox.com/pve-docs/chapter-pvesdn.html#pvesdn_config_dhcp

Once you’ve validated, go to the SDN/VNets submenu, click the “Create” button to create a virtual network, then select it once created to create a subnet.

Once again, don’t forget to check the DHCP section, to declare the range for our future DHCP.

Once we’ve validated everything, we see that next to all these new things we’ve just created, there’s an icon with two yellow arrows and a “New” status.

To deploy these changes, we need to go back to the SDN menu from the beginning, and click Apply. Two “vxlan1” lines (one for each server, actually) should appear.

Is That All?

Yes, that’s all, honestly, I really should have pushed a bit more before posting my previous article…

To convince yourself, just pop two LXC containers (without DHCP 🥲), one on each server, and see if they ping each other.

It works!

Well… all that’s left is to migrate all my existing machines to the VXLAN vnet to finally have a cluster where VMs can see each other!!

Deploying a Proxmox VE 8 Cluster on Dedicated Servers (3/4)

Tue, 25 Mar 2025 12:30:00 +0200

Note - this article is part of a series:

Picking Up Where We Left Off: First Steps in the UI

In the previous tutorial (which I invite you to read, if you haven’t already, otherwise this won’t be clear), we configured storage, networking, and deployed our first virtual machine to make sure everything works. But there’s still quite a bit to explore, including clustering, the Proxmox VE SDN, the built-in firewall, backups, monitoring…

So we’re picking up where we left off, and we’re going to install another server (hard to make a cluster with a single machine).

This server was reinstalled with the same procedures as the server I’m talking about in parts 1 and 2, except that we don’t need to create the bridge (vmbr0) because we’re going to use the SDN!!

Ideally, all machines in the cluster should have the same storage pools since I remind you, this configuration is supposed to be common to all servers, as it’s managed at the “Datacenter” level of our cluster (even if you can configure exceptions).

Creating the Cluster

Clustering Proxmox VE machines is a topic I’ve covered extensively on the blog, to the point that some of you know me mainly for these articles ;-):

So I won’t spend too much time on it. Just know that:

it’s easier to do since version 6 because there’s no longer a need to set up a VPN between servers (corosync v3)
it can be done from the command line
in any case, you need to log in as root with password when clustering

Very briefly then, we go to the first server, in the Server View at the top left, we select our Datacenter, then in the Cluster menu.

Click on the Create Cluster button.

Note: here, it’s simple, I only have one network card, so there’s no doubt about which interface to use. However, in the case of a server with more interfaces and more bridges (or VPNs as we had to do until PVE 6), you need to choose carefully which interface is used for clustering.

We validate the form:

From this point on, the server is no longer in “standalone” mode and has additional systemd services.

We notably have a /etc/pve folder that is synchronized between all nodes (the famous Proxmox Cluster File System) with the configuration of each node and the configuration files of virtual machines. We also have the corosync software, which will regularly contact all cluster nodes to ensure everyone is alive/reachable.

Joining the Cluster

If we go back to the “Cluster” menu, we now have information displayed about our cluster, namely its number of nodes (here just one, normal). Click on the Join information button.

A popup opens with several pieces of information to copy-paste that will be requested in the menu of the other server, the one that will join the cluster:

So we connect to the graphical interface of our second Proxmox VE server (my Atom in the example), we go to the same menu, but instead of creating a new cluster, we click on the Join Cluster button:

Normally a little scare, once the node joins the cluster, you’ll lose control of the web interface and get a “401” error message. This is “normal”, when the server enters the cluster, services are restarted and we switch from standalone mode to clustered mode…

Don’t panic, an F5 should fix the issue (or connect to the “first” server of the cluster).

We now have a nice cluster :)

Split Brain

Note however that in my example, we only have 2 nodes, which is more than discouraged when making a Linux cluster (actually, a cluster in general). If the network link ever goes down between our two machines, they’ll have no way of knowing if it’s a network problem (also known as split brain) or a node failure.

To avoid breaking everything by restarting VMs on both sides (the most catastrophic case in a split brain), when quorum is no longer reached, Proxmox VE puts nodes without a majority in (almost) “read-only” mode.

And in case it’s not clear, quorum is half the nodes +1, so here: 2/2+1 = 2… so as soon as we lose one of the two nodes, we lose quorum.

So if we ever cut a node or the communication between the two, your still active VMs will continue to work (I debated this on LinkedIn a few days ago).

However, it will be impossible to modify the cluster configuration, start or create VMs, etc.

The only thing we can possibly do that isn’t “read only” is stop VMs, because that doesn’t induce any risk for data consistency. But you won’t be able to turn them back on, you’ve been warned…

For the record, the LinkedIn debate was “does ProxmoxVE cut all VMs in case of split brain” and the answer is obviously no. However, most actions are indeed blocked.

In any case, to avoid this, the best thing is to have an odd number of nodes so that split brain can never happen, or possibly add a complementary vote using an instance serving only as an arbiter as explained in my old 2019 article or in the official documentation:

SDN Configuration

OK, cool, we have a two-machine cluster. But in the current configuration, the VM we created in the previous part can’t easily migrate from one server to another. Indeed, we attached it to a Linux bridge (vmbr0) that has its own network addressing plan.

In case of migration, we would potentially need to start the machine, change the IP address, review iptables/firewall rules. That’s what I was doing until now, and in the case of a DR plan for a lab infra and a personal blog, it’s “OK”.

A big new feature in version 8.1 is the activation of an “SDN” (Software Defined Network) module that has been in testing by developers and some early adopters since version 6.X, which will allow us to manage this centrally from the interface. If you want to read the official documentation, it’s here:

pve.proxmox.com/pve-docs/chapter-pvesdn.html

The only prerequisite we’re missing is dnsmasq for DHCP, and the documentation tells us to run the following commands on our machines:

apt update
apt install dnsmasq
# disable default instance
systemctl disable --now dnsmasq

And in the /etc/network/interfaces file, it’s also necessary to add at the end of the file the line (followed by a small systemctl restart networking)

source /etc/network/interfaces.d/*

In the Datacenter menu, open the SDN submenu. Note that SDNs are already declared, but these are actually the default networks we declared in previous articles:

In the SDN/Zones menu, add a “Simple” type zone. There are more complex zones that will be useful in production contexts with non-trivial LANs (VLAN, VXLan) but in our case, Simple is sufficient.

Note: don’t forget to check the advanced box to enable DHCP. YES we’re finally going to have DHCP for our virtual machines and containers.

Once you’ve validated, go to the SDN/VNets submenu, click the “Create” button to create a virtual network, then select it once created to create a subnet.

Once again, don’t forget to check the DHCP section, to declare the range for our future DHCP.

Once we’ve validated everything, we see that next to all these new things we’ve just created, there’s an icon with two yellow arrows and a “New” status.

To deploy these changes, we need to go back to the SDN menu from the beginning, and click Apply. Two “sdn1” lines (one for each server, actually) should appear.

Fun fact, the Proxmox VE SDN will add iptables rules that should look familiar if you’ve followed the previous articles:

#version:5
auto vnet1
iface vnet1
address 10.10.20.1/24
post-up iptables -t nat -A POSTROUTING -s '10.10.20.0/24' -o vmbr0 -j SNAT --to-source 203.0.113.1
post-down iptables -t nat -D POSTROUTING -s '10.10.20.0/24' -o vmbr0 -j SNAT --to-source 203.0.113.1
post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
bridge_ports none
bridge_stp off
bridge_fd 0
ip-forward on

And if now, I recreate another virtual machine, but this time instead of vmbr0, I choose vnet1, I don’t need to enter the IP address (since I have the SDN serving DHCP) and it has Internet access without additional configuration.

With this configuration however (SDN in Simple Zone mode), it still won’t be possible to have virtual machines communicate with each other. That will be possible with more complex configurations:

You can, for example, create a VXLAN overlay network on top of public internet, appearing to the VMs as if they share the same local Layer 2 network

Note: I’ve explored VXLANs since then, and I wrote a complementary article here.

Conclusion

I still haven’t finished this series, because I still have things to say about SDN replication and Proxmox VE firewalling, backups, monitoring, but once again, I’m already exceeding 12,000 characters. So it’s time to publish the article and start writing the next one.

And in the meantime, have fun :)

Deploying a Proxmox VE 8 Cluster on Dedicated Servers (2/3)

Mon, 17 Feb 2025 10:30:00 +0200

Note - this article is part of a series:

Picking Up Where We Left Off: First Steps in the UI

In the previous tutorial (which I invite you to read, if you haven’t already, otherwise this won’t be clear), we chose a dedicated physical server from a provider like OVHCloud or Hetzner. We then installed Proxmox VE in the latest version (8.x), configured some basic security options, added users and groups.

Now we can try connecting to the UI. The first thing that should strike you (💥 ouch!) is that the administration page is in HTTPS with a self-signed certificate. We’ll start by fixing that.

Something quite frustrating with Proxmox VE (but it’s probably for “good reasons” that I don’t know) is that some operations can’t be done in the UI if you connect with the admin account zwindler@pve that we created in the previous blog post.

This is typically true for updates, clustering, and configuration and… for certificate configuration (non-exhaustive list). So here, we’ll have to connect as root@pam for now and not as zwindler@pve…

Note that there are (for now) 2 “realms”, Linux PAM (for root) and Proxmox VE Auth (for admin).

In the right bar, we find our server, which is for now alone in its datacenter (we’ll come back to this later). There are several views in the Proxmox VE UI, some things are hard to find if you don’t have this in mind. Here, I’m in the “server view”, probably the most classic if you come from the VMware world. In this menu, you’ll find your hypervisors and we can interact with each of them, how they are configured.

Once our server is selected, we have the main menu displaying many menus. Select System / Certificates.

We’re in luck, Proxmox VE comes with a module to automatically order Let’s Encrypt certificates. If you have your own certificate that works too (button Upload Custom Certificate), of course, but I’ll assume you don’t have one, like me.

In the ACME section, click on Add ACME Account to register your email and accept the Let’s Encrypt Terms Of Service (TOS).

From there, add our server’s FQDN by clicking the Add button under ACME. The button was grayed out until we added an account.

Now we can finally order the certificate by clicking on Order Certificates Now. If everything goes well, the certificate will be correctly delivered by Let’s Encrypt, and the pve-proxy server (which serves the Proxmox VE UI frontend) should restart, and your web page refresh, with a proper certificate this time :).

Storage / ZFS

If you read my blog often and I tell you that I’m an unconditional fan of ZFS, you won’t be surprised.

And if you read the previous article carefully, you’ll remember that we reduced the default partition of the OVHcloud template in LVM to be able to create a nice ZFS partition. But if we go to the storage view, it’s unfortunately not visible.

Small UI subtlety, the menu to add storage is found at the Datacenter level and not in the server configuration itself. In theory, storage management must be identical on all servers in the same DC in Proxmox VE, even if we can override this principle and set up exceptions for many versions already.

In Datacenter / Storage, click Add, then ZFS in the dropdown menu:

On ZFS type storage, we can only store “Disk images” and “Containers”, which won’t help us if we also want to store ISOs, templates, or backups.

We can settle for storing them on the default “local” storage.

But we can also “cheat” by connecting to the server via SSH, creating a folder (/tank/data in my example), then creating a new Directory type storage that has no restriction on data types.

Let’s Talk About ZFS

I’ve banged my head countless times with people who persist in saying nonsense about ZFS, even after explaining it to them.

no, you don’t need ECC RAM to store things on ZFS. First, it’s a recommendation (probably a bit zealous) made by people who designed an extremely robust filesystem. Many production systems run with ZFS without ECC RAM. Second, it’s especially true if you do deduplication, because losing a block can have catastrophic consequences in such cases…
no, ZFS doesn’t require 1 GB of RAM per TB of disk, again, that’s only if you enable deduplication (to store the correspondence tables between hashes and blocks).

On the other hand, what is accurate is that by default, ZFS will try to use 50% of the available RAM on your server and use it as cache. In many administration software, including Proxmox VE, this value will be way too high for us, because RAM is a precious resource on a hypervisor.

That doesn’t mean this feature is useless. Having cache when you have unused RAM is always good. So, we’ll reduce this value to avoid resource conflicts between optimizing our storage performance and the number of VMs we can host.

# Restrict to 512MB
echo 536870912 | sudo tee -a /sys/module/zfs/parameters/zfs_arc_max

# Restrict to 4GB
echo 4294967296 | sudo tee -a /sys/module/zfs/parameters/zfs_arc_max

Note: this command will need to be persisted one way or another

Networking

Well… I tried to postpone as long as possible when we get to networking (because I hate it) but at some point, we’ll have to get to it…

By default, our Proxmox VE server is installed with a Linux bridge that will allow us to share network between our server’s physical interface and our virtual machines.

This setup works well on your local network with DHCP and your own local IPv4 LAN or if you have an IP range (whether v6 or v4) and something to assign them to virtual machines.

Unfortunately in my case, I only have one IPv4 (I could enable IPv6 but I’d have other issues and I prefer to keep it simple here). If I create VMs and assign them to the bridge, they won’t get an IP and won’t have Internet access.

There are several ways to connect virtual machines to the external network. If you have time, I recommend reading carefully the Proxmox VE Wiki which has a very good page listing the different possibilities.

pve.proxmox.com/wiki/Network_Configuration

In the previous articles on Proxmox VE 5 then Proxmox VE 6, we did complicated stuff with multiple bridges, DMZ, pfSense firewalling, and a hellish iptables script.

Just thinking about it, I’m already exhausted.

Fortunately, there has been a built-in firewall in Proxmox VE for several versions now, as well as an SDN that I’ve never taken the time to explore and I think this is the right tutorial to do it :).

So we’ll refer to the official documentation, and choose the “Masquerading (NAT) with iptables” mode, but with a simpler setup compared to before.

Danger Zone

Network interface configuration can theoretically be done directly from the UI rather than modifying configuration files (/etc/network/interfaces). In the server view, go to the System / Network menu:

Here, you can see I wasn’t making things up, and we do have a bridge with our physical interface (here enp1s0).

(We even have an IPv6 range, oh my 🙈 I have no excuse… ANYWAY, let’s pretend we didn’t see that.)

We’re now at the fun part where we can easily cut ourselves off. I’ve done it often enough and had to go into rescue mode…

fun fun fun fun

The idea here is to remove the interface from the bridge, configure the network directly on it, and give a local network for our VMs on the bridge, while adding iptables rules for masquerading.

As long as we don’t click the Apply Configuration button, we’re not “at risk”.

The problem with the UI is that we’ll only be able to modify the IPs, gateways, and interfaces on the bridge or not. But to do the masquerading and have our virtual machines access the Internet, we need to add the famous iptables rules I’m talking about just before and we can’t do it from the UI.

We’ll need to add the last five lines in the vmbr0 configuration…

cat /etc/network/interfaces
[...]
auto vmbr0
iface vmbr0 inet static
 address 10.10.10.1/24
 bridge-ports none
 bridge-stp off
 bridge-fd 0

 # add for masquerading to work
 post-up echo 1 > /proc/sys/net/ipv4/ip_forward
 post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o enp1s0 -j MASQUERADE
 post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o enp1s0 -j MASQUERADE
 post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

Warning: the most important point is to verify that the output interface (-o) has the correct name. Here, my physical interface connected to the Internet is enp1s0, but if yours has a different name, you need to adapt.

From my point of view, the UI is still not completely useless, because it allows us to make the most delicate change (IP swap) with the visual diff and the “apply config” button. The post-up scripts are unlikely to cut off our access.

First Test

We’re far from done (it won’t fit in this article anyway, so might as well rush through…). But I’m sure you’re eager to launch a VM and we should have something functional, though incomplete.

So we’ll do a quick test of our setup to verify that we have a minimally functional hypervisor (meaning without clustering, without SDN, without firewall, without monitoring, without backup… without anything, really).

I started playing around with Talos Linux as a Linux distribution for Kubernetes. To keep it simple, I’ll launch the install of a Talos Node. If it manages to enroll in the Omni interface, it means I have a functional hypervisor, network included.

Once the ISO is generated on my Omni site (the SaaS control plane from Sidero Labs, the publisher of Talos Linux), we can push it to our Proxmox VE.

To do this, from our server (so not at the Datacenter level), we select a storage pool that accepts ISO images (if you remember correctly, our ZFS pool doesn’t support it by default), choose ISO Images, and click Upload to upload the ISO.

Once uploaded, we can now click the blue button at the top right Create VM.

From there, the VM creation wizard should take over. It’s going to be a bit verbose, but all menus are useful when you start to know Proxmox VE well. I’ll give you the minimum though:

In the first menu, we won’t choose which node to install the VM on since for now we only have one… However, we’ll have to give it a unique ID (100 by default) for the entire cluster. Ideally, we also give it a little name. I opted for talos02 (because I already have a talos01 on another server, see how original I am?).

Finally, I checked the “Start at boot” box, that’s the kind of annoying thing when you forget to set it and the hypervisor reboots (it happens)…

You can give tags to your VMs, they’ll have nice colored badges to distinguish them :)

In the second menu (OS), we need to indicate two things. First, that we want to use the ISO we just retrieved/uploaded to boot our machine (you need to choose the right storage pool, by default “local” is selected). We also need to give the OS type for driver compatibility reasons for virtual devices. It’s been a long time since we really needed that, “6.X - 2.6” works for all recent Linux (kernel 2.6 release was in 2003!).

I skip System, in Disks, we add a disk on the right pool (so not local, but the one on ZFS). For my 32 GB Talos VM, that’s more than enough… Note that there are many important options here for everything related to read/write “optimization”, but we’re well beyond the scope of this article.

I skip CPU and memory, the important thing is to give it at least 2 cores and 2 GB of RAM (at least). You can also optimize CPU performance by enabling the right flags, but again, we’re beyond the purpose of the article so I won’t insist.

In the network section, we just make sure we’re connected to the right bridge (normally if you followed the article, we only have one, vmbr0, so we can’t go wrong).

And we verify everything is OK. Protip, to save five seconds in your life, click “Start after created”, which will start the VM as soon as possible after creation.

It’s Alive?

At this point, the VM should automatically boot from the Talos CD and the OS should install on the disk. Once ready, the machine will try to enroll on my Omni control plane.

And that’s when disaster strikes…

Because yes, I haven’t configured DHCP or SDN. So we have to manually configure the virtual network card (which isn’t great), by pressing F3 in the console.

(And suffering a bit of QWERTY, but that’s how you recognize a good sysadmin normally: they can do QWERTY in consoles)

Once the network is configured, the machine should almost instantly start configuring itself…

… and enroll in Omni:

Victory!!

Well…

Well… yes, we have a functional VM. But I aspire to go a bit further before really declaring victory.

As I said above, we had to configure our VM’s network by hand, and we don’t yet have clustering, firewalling, SDN, monitoring, or backup… But we’re comfortably scratching 16,000 characters for this blog post, so it’s time to hang up and say “see you next time”.

And in the meantime, have fun :)

Deploying a Proxmox VE 8 Cluster on Dedicated Servers (1/2)

Mon, 27 Jan 2025 20:00:00 +0000

Note - this article is part of a series:

Introduction

Those who follow the “lore” of this blog know that I’m somewhat known in the Sunday tinkerer community for having written (or co-written) several tutorials on Proxmox VE. I’m thinking in particular of two article series on Proxmox VE 5, then Proxmox VE 6, to create a virtualization cluster from scratch with Proxmox VE (and pfSense at the time).

This is a new iteration of these tutorials, this time based on Proxmox VE 8, with quite a few things changing and new approaches.

The ultimate goal of this article series is to have a virtualization platform with Proxmox VE. This platform will be composed of several machines (ideally 3, but 2 works too), secured, monitored, highly available, and backed up.

You’ll need to follow several articles to achieve this.

Starting Assumptions

1- I’m assuming you don’t have machines to do this at home and don’t want or can’t afford to invest in a homelab (even though it can be a lot of fun, it requires investment). However, if you have machines at home, a large part of this article remains valid, once the machines are installed.

2- Since Proxmox VE is a virtualization platform, I’m also assuming you want dedicated (physical) machines and not virtual machines, precisely to be able to virtualize entire OSes without too many problems (even though we always have the “nested virt” option, it’s less ideal).

Note: if you don’t need virtualization (example: no Windows or special OS), Proxmox VE is a great containerization platform, particularly thanks to LXC, which allows you to create Linux containers with a full OS, that you can manage exactly like a VM but with much better performance (and some limitations due to kernel sharing, but it’s often sufficient). You can read my articles on LXC if you’re interested.

3- I’d like to draw your attention to the fact that many actions could have been scripted or managed via “infrastructure as code”. It’s more in tune with the times, because it’s more robust and reliable. However, the point of this article is to do it “by hand” to understand what we’re doing, step by step, not just provide you with a “turnkey” infrastructure that you won’t know how to manage at the first hiccup.

If you’re looking for more industrial methods to deploy virtualization clusters, you’ll surely find plenty on the Internet, from other bloggers (some French, some are even friends).

I’ve actually done the exercise myself several times with Ansible or Rudder in the past, for example here (note: this code is obsolete):

Now that the objective and context is clear for everyone, we can begin :)

Choosing the Machines

Since we said we’re not self-hosting, we need to find a host that offers physical machines. Ideally, something not too expensive would be nice to start with.

So we’re going to look for inexpensive machines BUT with support for VT-x / AMD-v instructions, which is pretty much all CPUs that still work today EXCEPT Atoms and ARM machines like Raspberry.

Basically, that leaves us with Kimsufi at OVHCloud, Dedibox at Scaleway, and machines at Hetzner.

At Hetzner for €45, you get an i5 gen13, 64 GB of RAM and 2 NVMe SSDs.

At OVHcloud, with a bit of luck, you can find machines around €15-20 per month with older hardware, but more than sufficient for what we’re going to do.

At Scaleway, the entry-level offer is centered on Atom Avoton or very old Xeons. The Avoton C2750 remains a valid option, as it’s an octa-core that supports VT-x (unlike the C2350, beware!). But well, it’s 2013 hardware 💀…

For the purposes of this article, I optimized costs and went with the OVHcloud server I present just above (KS-GAME-LE), unfortunately only available in Canada :(. But 8 threads and 16 GB of RAM for €12 including tax per month, that’s really not expensive.

Two downsides:

ridiculously small storage for virtualization (only 240 GB, a single disk)
latency - almost 100 ms from my place, compared to 11 ms for my other machine at Scaleway

64 bytes from proxmox.example.org (203.0.113.78): icmp_seq=1 ttl=53 time=98.0 ms
64 bytes from proxmox.example.org (203.0.113.78): icmp_seq=2 ttl=53 time=97.5 ms
64 bytes from proxmox.example.org (203.0.113.78): icmp_seq=3 ttl=53 time=97.6 ms
64 bytes from proxmox.example.org (203.0.113.78): icmp_seq=4 ttl=53 time=97.7 ms
64 bytes from proxmox.example.org (203.0.113.78): icmp_seq=5 ttl=53 time=97.7 ms

OS Installation

Once ordered, the server is relatively quickly available in your “OVH manager”. We have a fairly intuitive menu to directly install Proxmox VE 8, almost up to date (the provided images are rebuilt very regularly with all updates).

To start the installation, click on the “…” in the Operating System (OS) section, then select install from an OVH template

Scroll through the list of OS types, select Virtualization, then Proxmox VE 8 (while we’re at it). I also checked the “Customize partition configuration” box, because by default, OVH’s installation uses “LVM thick” type storage for most of the disk space, while we’ll use advanced ZFS features later:

Before validating the installation, don’t forget to change its name (Custom hostname) to avoid having to do it post-install, then add a public SSH key:

First Post-Installation Tasks: Upgrade and Reboot

There’s a whole series of things to do when you’ve just installed a machine, especially on Proxmox VE, and we’ll try to resist the urge to “quickly” connect to the graphical interface.

Anyway, we won’t be able to connect to the Proxmox VE graphical interface…

Really???

Yes, because while we can normally connect with the root user of our Linux, we won’t be able to in the case of OVHcloud, simply because we DON’T have the root password. Good thing we added an SSH key ;)

We’ll come back to it later, we have more urgent things to do.

The first thing I did was immediately add a type A DNS record so that my machine’s IP corresponds to an FQDN. For this article, I’ll refer to it as follows:

hostname: myPVEhost
FQDN: proxmox.example.org
IP: 203.0.113.159

So we connect to our freshly installed machine’s IP via SSH (root@IPmachine) and we’ll take the opportunity to do some cleanup.

The first thing to do is update the server, ideally followed by a reboot:

apt update && apt upgrade -y && reboot

Once the server is back to life, we reconnect and enable automatic updates in one way or another. The simplest is to use unattended-upgrades if it’s not installed by default (it may already be there) but there are more sophisticated tools.

apt install unattended-upgrades

It’s for Security

Once that’s done, I recommend installing CrowdSec right away. Here, I simply based myself on the official crowdsec documentation, to add the detection engine, the remediation component (here via iptables):

CrowdSec.net - Linux Installation

curl -s https://install.crowdsec.net | sudo sh
apt install crowdsec

# verify that the crowdsec service is working
systemctl status crowdsec
● crowdsec.service - Crowdsec agent
 Loaded: loaded (/lib/systemd/system/crowdsec.service; enabled; preset: enabled)
 Active: active (running) since Sun 2025-01-26 21:04:26 UTC; 1min 56s ago
 Process: 73830 ExecStartPre=/usr/bin/crowdsec -c /etc/crowdsec/config.yaml -t -error (code=exited, status=0/SUCCESS)
 Main PID: 73862 (crowdsec)
 Tasks: 13 (limit: 19059)
 Memory: 33.1M
 CPU: 2.571s
 CGroup: /system.slice/crowdsec.service
 ├─73862 /usr/bin/crowdsec -c /etc/crowdsec/config.yaml
 └─73877 journalctl --follow -n 0 _SYSTEMD_UNIT=ssh.service

Jan 26 21:04:23 myPVEhost systemd[1]: Starting crowdsec.service - Crowdsec agent...
Jan 26 21:04:26 myPVEhost systemd[1]: Started crowdsec.service - Crowdsec agent.

# install the remediation engine
apt install crowdsec-firewall-bouncer

Once Crowdsec is operational, we can add the “fulljackz/proxmox” collection, which we’ll need to modify (thanks to Julien Louis’s great article on his slash-root.fr blog), which will listen to the webUI logs to detect any brute force and ban those who try:

cscli collections install fulljackz/proxmox
[...]
INFO Enabled fulljackz/proxmox
INFO Run 'sudo systemctl reload crowdsec' for the new configuration to be effective.

The pattern listened to by the proxmox parser is no longer correct with new versions

Jan 26 21:23:14 myPVEhost pvedaemon[1250]: authentication failure; rhost=::ffff:203.0.113.159.78 user=coucou@pve msg=no such user ('coucou@pve')

We see here that Julien added the “::ffff:” prefix in the PVE_AUTH_FAIL variable of the /etc/crowdsec/parsers/s01-parse/proxmox-logs.yaml file, but when I look at the test cases provided in the collection, it seems like it’s not necessary…

In any case, we’ll create a /etc/crowdsec/acquis.d/proxmox.yaml file so that CrowdSec can start analyzing our logs:

mkdir -p /etc/crowdsec/acquis.d/
cat > /etc/crowdsec/acquis.d/proxmox.yaml <<EOF
journalctl_filter:
 - _SYSTEMD_UNIT=pvedaemon.service
labels:
 type: syslog
EOF

We finish by reloading crowdsec and watching ~~the world burn~~ script kiddies get blocked.

systemctl reload crowdsec

cscli alerts list
╭────┬────────────────────┬───────────────────────────┬─────────┬──────────────────────────────────────────────────────────────┬───────────┬─────────────────────────────────────────╮
│ ID │ value │ reason │ country │ as │ decisions │ created_at │
├────┼────────────────────┼───────────────────────────┼─────────┼──────────────────────────────────────────────────────────────┼───────────┼─────────────────────────────────────────┤
│ 2 │ Ip:203.0.113.222 │ crowdsecurity/ssh-slow-bf │ IR │ 202468 Gloubi Bo ulgua Co. ( Private Joint Stock) │ ban:1 │ 2025-01-26 22:31:15.366187524 +0000 UTC │
╰────┴────────────────────┴───────────────────────────┴─────────┴──────────────────────────────────────────────────────────────┴───────────┴─────────────────────────────────────────╯

Among the little annoying things shipped by default with Proxmox VE, there used to be the fact that Proxmox VE had enterprise apt repositories on all installations, and it was necessary to manually disable them before being able to do updates.

It was super annoying and the Proxmox VE developers heard (a little) their community and removed this requirement… to replace it with a popup.

Now, a popup, I’m fine with that! It’s important to reward developers’ work and support them. But is it really necessary to do it at every login AND every time we refresh the list of packages to update?

This is obviously a rhetorical question.

Obviously, many people have found several methods to disable the responsible JS code. Big up to fabio for his version that works well.

sed -Ezi.bak "s/(function\(orig_cmd\) \{)/\1\n\torig_cmd\(\);\n\treturn;/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service

Note: if like me you’ve already connected once, you’ll also need to clear the cache otherwise the popup will remain…

Creating Groups and Users for the Graphical Interface

Ok ok, we’ve done a lot of stuff, can we connect now???

Yes yes, we can. Anyway, I’m already at almost 15,000 characters, this article is way too long, we need to wrap up.

Last step in this article, we’ll create an administrator to connect to the UI and a user that we’ll use later for monitoring.

# you're not **really** obligated to call the admin "zwindler" you know?
pveum group add admin -comment "System Administrators"
pveum acl modify / -group admin -role Administrator
pveum useradd zwindler@pve
pveum usermod zwindler@pve -group admin
pveum passwd zwindler@pve

# our monitoring user
pveum groupadd monitoring -comment 'Monitoring group'
pveum aclmod / -group monitoring -role PVEAuditor
pveum useradd pve_exporter@pve
pveum usermod pve_exporter@pve -group monitoring
pveum passwd pve_exporter@pve

Alright, you’ve waited long enough, you can connect to your Proxmox VE! Go to:

https://@IPmachine:8006 or
https://proxmox.example.org:8006

Note: for now, the certificate is self-signed and the browser will show an error. We can accept to ignore the problem for now.

Don’t forget to change the “realm” from PAM (unix users, which we’re not using here) to “Proxmox” to be able to use the administrator we just created.

See you soon for the next part. And in the meantime, have fun!

Should we have containers ?

Thu, 25 Aug 2016 16:00:21 +0000

A more complete answer to « Why do we have containers » by fntlnz

2 days ago, I stumbled apon fntlnz article Why do we have containers. (dead link, using Internet Archive) The article had been reposted by Docker official Twitter account and the first thing that struck me was the displayed image : a schema comparing full virtualization, para virtualization and linux containers.

And of course, I say this because I didn’t entirely agree with it ;-). But then I read the article and I found many good points.

So I decided to write « Should we have containers ? » as an answer to develop my remarks a little.

Full virtualization vs paravirtualization vs OS level virtualization

First, I’ll go with the schema. I know the point was to provide a really simple schema to help understand the differences between virtualization types but I don’t think it should be presented like that.

Hypervisor and host OS are both displayed in the full/para. I find this be misleading. Of course there is a Host OS and a virtualization engine on top of it. But most of the time, either the OS is really thin (ESXi fits in 8GB SD cards and runs nothing but virtualization) or the virt engine is a part of the kernel (KVM, Xen) just like LXC is a part of the Linux kernel.

Also, the bins/libs are only displayed on OS level virt. This is interesting to put in the « host level virtualization schema » but it should be added on full and para as well because this is one of the advantages host level virt. has over full/para.
Even if your VMs are all identicals, bins and libs have to be present (copied) in each VMs. On contrary, in OS Level virt, you CAN share identicals bins/libs between containers, thus reducing the storage footprint of the virtualized environment.

That’s how I would have presented it

You can see that OS level virtualization and Hypervisors are not so different on this simplified view. The differences appear more clearly when you add x86 Security Rings (0->3). The « overhead » of full virtualization comes from the calls that guest OSes do to hypervisor pilots and the translation of CPU instructions.

On a side note, I put Docker on the left to show that it’s basically an administration suite that helps you run LXC containers

The rest of the analysis

Security

When talking about containers, the security point is often raised.

VMs are really hermetics. Taking control of the hypervisor after hacking a VM should be MUCH harder than taking control of the host OS if you are inside a container with whom the kernel is shared.

But I agree with fntlnz: having to patch only one kernel for all your virtual environments instead of having to patch ALL kernels of your virtual machines is actually a good point.

I encourage you to read this (dead link, using Internet Archive) from Jérôme Petazzoni if you’d like a good presentation on the subject.

Performance

A point that’s not really developed in fntlnz article is performance, but I read it/ear it all the time.

General belief is that VMs are slow. That was true in the « early days » of x86 virtualization, because of the limitations of the x86 architectures which didn’t allow efficient OS virtualisation. Overhead for CPU instructions was known to be of about 15% (which is enormous). Thus, running processes directly on the OS are believed to be more efficent.

But nowadays, modern hypervisors on modern hardware nearly generates no overhead at all over CPU and RAM consumption. You could argue that guest OSes consume ressources while containers don’t. But let’s be honest: with the 32 GB RAM tomcat instances I regularly see, don’t say to me that a Linux guest OS CPU&RAM consumption is significative.

The real difference is the on the storage consumption. First, for N VMs, you save yourself N*xGB of guest OS files. If your Linux templates are good, you don’t consume that much (a few GB at most) but this can add up when we are talking about thousands of instances. [Edit jan. 2020]Not so true today if you host your own hardware and have so deduplicated storage[/Edit]

From there, you still have some margin for gains, mostly through Copy on write and union filesystems :

You make a modification to an official tomcat image to add you configuration files ? Only the delta will be stored.
You launch thousands of containers from this image ? You only consume the container image once.
The real data (user data, business data) is of course the same for both architectures

So, should I use containers ?

The problem here is that’s not the real question. Do you need containers ? Here’s fntlnz point of view.**
**

But if at some extent your motivation consist in decreasing costs related to full virtualization overheads while allowing developers to ship, develop and test code faster we already found two.

fntlnz

I won’t tell you not to use containers or Docker. Truth be told, we use it in dev and production. Docker is awesome.

Yes, Docker reduces overhead of full virtualization. Storage mostly, but it highly depends on your workload. For example if you have the need for millions of really tiny instances, then Guest OS overhead is significative in full virtualization.
Yes, Docker allows developers to ship, develop and test code faster. You still have to change your way to develop yours app but Docker and DevOps is the new thing. Better get on with it ;)

But you shouldn’t use Docker just because of that. Docker containers need you to change your philosophy toward applications and administration.

Shared kernel = same kernel

You need to understand that you’ll share a kernel. For a long time that ment a 100% Linux only which could be problematic in some cases. I just realized that Windows containers exists in Windows 2016 tech preview. And Solaris is coming.

But independantly of platforms, you still you share a kernel. If you want Linux containers, you’ll need Linux Docker nodes. If you need Windows containers, you’ll have to have Windows Docker nodes.

Non-persistant/cloud-ready instances

Docker containers are by design non-persistant and by best practice single-process. This isn’t mandatory of course, but Docker works better that way.

The whole idea is to help you make « cloud-ready » & atomic applications that don’t store anything in the container itself (persistant storage is presented to the container but outside it). This way, you can scale out really quickly: you just start up a few more instances if you need more firepower.

This also modifies the way you do systems/applications administration. A container/application is spinning out of control ? Don’t try to repair it.

Kill it
Launch a new one

The fact that persistant storage is handled outside the containers also helps you to deal with lifecycle management.

Update your application inside the docker image
Shutdown the old container
Start the new container with the new image, plugged to the persistant storage

All that requires of course applications that have been designed this way! This is a massive change in philosophy and old monolithic apps on a Docker platform won’t work as well as microservices will.

System administration

I don’t say solutions don’t exist because they do.

But starting up and shutting down containers on the fly will give headaches to your « old school Nagios » monitoring administrator. What could be done easily in the physical/VM world (installing an agent in the template), each of those having an IP adress to query for health checks is a lot easier than checking on a single host a highly volatile number of isolated processes.

You’ll probably have to adapt and use tools that are designed to handle such cases.

Backup administrator may find it confusing at first to make application consistant data backups when applications are running in an variable number of containers.

And so on…

Final thoughts

Yes, you should definitely look into containers. They are mature, stable, a great new way to speed up development and contribute to save money on hardware.

But be aware that VMs and containers don’t answer all usecases; and that you won’t migrate all your applications overnight!

Virtualisation on Zwindler's Reflection

Proxmox VE 8 Tutorial - SDN in VXLAN Mode with Machines on the Internet

Preamble

Prerequisites

Problem

VXLAN Then, But with a VPN

SDN Configuration

Is That All?

Deploying a Proxmox VE 8 Cluster on Dedicated Servers (3/4)

Picking Up Where We Left Off: First Steps in the UI

Creating the Cluster

Joining the Cluster

Split Brain

SDN Configuration

Conclusion

Deploying a Proxmox VE 8 Cluster on Dedicated Servers (2/3)

Picking Up Where We Left Off: First Steps in the UI

Storage / ZFS

Let’s Talk About ZFS

Networking

Danger Zone

First Test

It’s Alive?

Well…

Deploying a Proxmox VE 8 Cluster on Dedicated Servers (1/2)

Introduction

Starting Assumptions

Choosing the Machines

OS Installation

First Post-Installation Tasks: Upgrade and Reboot

It’s for Security

Nagging Popup

Creating Groups and Users for the Graphical Interface

Should we have containers ?

A more complete answer to « Why do we have containers » by fntlnz

Full virtualization vs paravirtualization vs OS level virtualization

The rest of the analysis

Security

Performance

So, should I use containers ?

Shared kernel = same kernel~~ ~~

Non-persistant/cloud-ready instances

System administration

Final thoughts

Shared kernel = same kernel