Docker on Zwindler's Reflection

Kubernetes UserNamespaces: the overhyped GA feature

Tue, 28 Apr 2026 10:00:00 +0200

The infographic that triggered me

Note: I stumbled upon this GenAI infographic and it was so bad I wrote a post about it. I didn’t generate this thing.

Over the past few days, LinkedIn has been flooded with the same kind of infographic. Kubernetes 1.36 is out, and one of the most talked-about features is the GA release of UserNamespaces.

It’s a topic I’ve been following since 2018 (talk The Route to rootless containers at KubeCon EU 2018), so I’m genuinely glad to see this long journey finally reach the finish line. That said, I’m appalled by the way it’s being marketed on LinkedIn, apparently by people who have no idea how it actually works — and frankly don’t care.

“Kubernetes just made root safer. Just add hostUsers: false to your Pod spec.”

The visual: an all-powerful king “inside the container” and a helpless beggar “outside on the host”. The promise: “No Host Access. No Privilege Escalation. No Lateral Movement. No Node Takeover.”

Catchy.

But presenting it this way is genuinely dangerous, because it obscures entire areas of application and operational security. Selling hostUsers: false as the universal fix for the “root in containers” problem is a dramatic oversimplification that will push teams to ignore the real security priorities.

What UserNamespaces actually do

The threat model: container escape

First, what are we actually talking about? A container escape is when an attacker manages to break out of their container and directly access the host’s kernel or filesystem — completely bypassing the normal isolation mechanisms.

This type of vulnerability is rare, but real-world examples exist:

CVE-2019-5736 (runc): write to /proc/self/exe of the host process from inside the container
CVE-2022-0492 (cgroups v1): escape via unshare in certain configurations
CVE-2024-21626 (runc, “Leaky Vessels”): file descriptor leak to the host working directory

If there’s a vulnerability of this type on your node AND a process is compromised AND it runs as root in the container AND it doesn’t use UserNamespaces, the attacker gets root on the host — game over. Full access to every file on the node, every secret mounted by other pods, ability to install a rootkit or exfiltrate data from all tenants running on that node.

It remains possible, but that’s a lot of “ifs”. Either way, this is exactly the scenario UserNamespaces address. They introduce UID mapping:

UID 0 inside the container is mapped to an unprivileged UID on the host (e.g. 100000, unique per pod)
If an attacker successfully escapes the container via a kernel exploit, they land as nobody on the node — the escape succeeds, but the post-escape impact is dramatically reduced

This is the “Breakouts Lose Impact” scenario from the infographic, and on that point, the infographic is right. That’s the real value of the feature.

Edge case: multi-tenancy even with non-root containers

Even without root containers, UserNamespaces provide something extra in a truly multi-tenant context (multiple customers on the same cluster). Without UserNamespaces, if two pods from different customers both run with runAsUser: 1000, they share the same UID 1000 on the node. If one escapes, the attacker can access files from the other pod with the same owner. UserNamespaces, by assigning a unique UID offset per pod, isolates UIDs between pods even when they use the same value inside the container.

For internal clusters where you control all workloads, this scenario is theoretical. For a multi-tenant SaaS platform or a public build service, it’s a real line of defense.

Technical requirements

There are a few prerequisites, but most up-to-date clusters should qualify.

Linux kernel ≥ 5.19
Compatible runtime (containerd ≥ 1.7, CRI-O ≥ 1.25)
Idmapped mounts support for persistent volumes (XFS, ext4 — not NFS in all cases)
Kubernetes ≥ 1.33 (Beta), ≥ 1.36 (GA)

What the infographic exaggerates (and leaves out)

The infographic is right about one specific thing: UserNamespaces reduces the impact of a successful container escape. That’s real. The problem is it sells the feature as a universal solution to “root in containers” — and that’s just wrong.

1. UID isolation is not application privilege isolation

The infographic promises “No Lateral Movement”. That’s false — completely false.

A root container with hostUsers: false can still read the ServiceAccount Token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. If that token has broad RBAC permissions (which happens — we may cover this in a future post), the attacker can call the API Server, enumerate cluster resources, and move laterally — all without ever touching the host node.

UID mapping protects the host. It does not protect the cluster.

2. A root container is still root inside the container

“Install Anything ✅” — it’s literally written in the infographic, presented as a feature 😖.

In a root container (even with UserNS), an attacker who gains control can:

Install nmap, curl, nc to scan the internal network
Modify application files, binaries, configurations
Read all files mounted as volumes
Persist in the container across restarts if the filesystem is writable

UserNamespaces removes none of these attack vectors. The ability to install software is a fast track to lateral movement.

3. It’s not that easy, especially for storage

Enabling hostUsers: false breaks existing storage in most cases.

Container UID 0 is mapped to UID 100000+ on the host (each container has its own offset). If a persistent volume (NFS, EBS, Ceph RBD) is owned by UID 1000, the root container can’t read or write it. The result: counterintuitive Permission Denied errors that are potentially hard to diagnose, since the application was likely never designed to be root yet have no access to its own files.

The technical solution exists (idmapped mounts), but it requires a recent kernel and a compatible filesystem. See the official idmapped mounts documentation for details.

4. Same story, but for networking

hostUsers: false is incompatible with hostNetwork: true. It’s a corner case, but it catches networking workloads (monitoring agents, CNI plugins, etc.).

Note: that said, running containers with hostNetwork is its own security problem, so…

Honest comparison: UserNS vs the real alternatives

Attack vector	UserNS (root inside)	Non-root (UID 1000)	Distroless / Scratch
Post-escape impact after successful container escape	✅ Nobody on host	⚠️ UID 1000 on host	⚠️ UID 1000 on host
UID isolation between pods (multi-tenant)	✅ Unique offset per pod	❌ Shared UID on node	❌ Shared UID on node
Malware installation inside the container	❌ Trivial	❌ Possible	✅ Near impossible
Write scope in ephemeral container FS	❌ Full filesystem	❌ App directory only	✅ Near impossible
Lateral movement via SA Token	❌ Possible	❌ Possible	⚠️ Potentially difficult
Operational complexity	❌ Sometimes high	✅ Often near zero	✅ Often low
Compatibility with existing storage	❌ Sometimes problematic	✅ Standard	✅ Standard

Reading the table reveals the true nature of UserNamespaces: it excels on exactly two rows:

post-escape impact
UID isolation in multi-tenant environments

On everything else, non-root + distroless does better, or just as well, without the operational complexity. And that “everything else” — write scope in ephemeral FS, malware installation, lateral movement via SA Token — represents the vast majority of real-world attack vectors, far more common than a container escape. We’ll come back to this in the Where to invest your security budget section.

Real use cases

It would be dishonest to dismiss the feature entirely. There are three scenarios where UserNamespaces aren’t a lazy option but a genuine technical necessity (with caveats).

1. Build-as-a-Service (Buildah, rootless Podman)

To build a Docker image, the build engine needs to perform chown, chmod and mknod. These operations require CAP_CHOWN and CAP_FOWNER. Before UserNamespaces, the solution was to run the pod as --privileged — an obvious open door to the host.

With hostUsers: false, the build engine believes it’s root for its own file manipulation, but it can’t touch the host. This is the only case where “root inside” is a technical constraint rather than technical debt.

Note: Kaniko, long the go-to for in-cluster builds, has been archived since October 2025 and no longer receives security updates. Buildah or rootless Podman are the active alternatives.

My take: it can be useful for shared CI/CD platforms (GitLab Runners, Tekton) that refuse privileged pods. But if isolation is critical (public platform, aggressive multi-tenancy), microVMs (Kata Containers, Firecracker) offer far stronger guarantees for an overhead that has become quite manageable.

2. Hostile multi-tenancy (user code platforms)

If your business is running code provided by strangers (PaaS, online code editors, public CI/CD), you know upfront that users will try to escalate their privileges. In this context, UserNS is an extra barrier against kernel 0-days.

My take: honestly, if the environment is truly hostile, UserNS alone isn’t enough. MicroVMs (Kata Containers, Firecracker) provide real hardware isolation and are the right choice here. UserNS can be a complement, not a substitute.

3. Hard-coded legacy (Postfix, Dovecot, BIND)

Some old UNIX daemons start as root to open a privileged port (< 1024) or read sensitive config files, then drop privileges via setuid(). This mechanism fails in a classic non-root container.

UserNamespaces let these processes believe they can make their identity management syscalls, because they are root inside their namespace.

Here’s a concrete example written by a colleague (thanks Louis 😘):

apiVersion: v1
kind: Pod
metadata:
 name: postfix
spec:
 hostUsers: false # UID mapping: root in container → nobody on host
 securityContext:
 runAsNonRoot: false # allowed under PSS Restricted *only* because of hostUsers: false
 fsGroup: 103 # postfix GID
 containers:
 - name: postfix
 image: postfix:latest
 securityContext:
 runAsNonRoot: false # same — cf. https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/#integration-with-pod-security-admission-checks
 allowPrivilegeEscalation: false
 readOnlyRootFilesystem: true
 seccompProfile:
 type: RuntimeDefault
 capabilities:
 drop: ["ALL"] # drop everything first
 add:
 - SETUID  # privilege drop via setuid() done by postfix itself at startup
 - SETGID  # same for groups
 - CHOWN  # chown on mail queues at startup
 - FOWNER  # file operations without being the owner
 - FSETID  # preserve setuid bit after write
 - DAC_OVERRIDE # MANDATORY: root in UserNS is not "real" root —
 # DAC checks are not automatically bypassed

This manifest illustrates several important things.

First, making a legacy application actually secure with UserNS is painful and requires compromises — especially around capabilities. This is a far cry from the “magic feature that secures root apps” the LinkedIn wanna-be influencers imply.

Then there are some interesting surprises. Normally, the Restricted Pod Security Standard forbids runAsNonRoot: false. Kubernetes makes an exception when hostUsers: false is present. This is documented here. Without UserNamespaces, this pod would be rejected by the admission controller.

There’s also the DAC_OVERRIDE capability, which is counterintuitive. Root in a UserNS is not real root from the kernel’s perspective for DAC (Discretionary Access Control) checks. When Postfix runs set-permissions to chown its queues, the kernel still verifies permissions — and denies them if DAC_OVERRIDE isn’t present. This is exactly the kind of operational surprise that stays invisible until the first production deployment.

Worth noting: we were still able to keep readOnlyRootFilesystem: true and allowPrivilegeEscalation: false — legacy doesn’t justify throwing everything overboard.

My take: this is the only use case where UserNS is genuinely acceptable. No untrusted third-party code, no hostile platform — just well-identified legacy with a migration plan. The other two cases are “acceptable under conditions”; the legacy case is the cleanest of the three.

Some counterarguments

I see you coming with objections, so let’s save everyone some time with a quick Q&A:

“It’s defense in depth.” True — but defense in depth assumes the foundational layers are already in place. If you haven’t migrated your images to non-root yet, investing energy in UserNS is putting the cart before the horse. And once you’re non-root, the marginal gain of UserNS is negligible compared to the complexity it introduces.

“We don’t control third-party images.” Somewhat weak, in my opinion: if a proprietary vendor’s black-box image is hardcoded to run as root, there’s a good chance it either genuinely needs it (as is the case for some proprietary security tooling) or it will break with UID mapping (see the storage problem above). UserNS is not a magic wand that makes any third-party image compatible and secure.

“It’s a centralized safeguard against human error.” It’s just as easy to forget hostUsers: false as it is to forget runAsNonRoot: true. The real centralized solution is Pod Security Standards or an Admission Controller (Kyverno, OPA) that outright rejects root pods. Simpler, more reliable, and it doesn’t break storage.

“We need it for SOC2/PCI-DSS/… compliance.” If your compliance requires strict tenant isolation, UserNS will likely be deemed insufficient by your auditors. VMs or microVMs remain the gold standard. Using UserNS for compliance means choosing the most complex tool to maintain for a result that remains debatable.

Where to invest your security budget

Setting aside the marketing, here’s where effort actually pays off — from highest impact to most niche:

Priority 1 — Non-root images + nobody (UID 65534) Move images to non-root, ideally using the nobody user (the least privileged on the system). If an application is compromised under nobody, the attacker can do almost nothing, even on the container filesystem. Combine with readOnlyRootFilesystem: true and capabilities: drop: ["ALL"].

spec:
 securityContext:
 runAsNonRoot: true
 runAsUser: 65534 # nobody
 seccompProfile:
 type: RuntimeDefault
 containers:
 - name: app
 image: my-app:distroless
 securityContext:
 allowPrivilegeEscalation: false
 capabilities:
 drop: ["ALL"]
 readOnlyRootFilesystem: true

Priority 2 — Pod Security Standards (PSS) at Baseline or Restricted Block root and privileges without breaking anything at the infra level. It requires having done Priority 1 first, but it’s free, standard, and applies cluster-wide via a namespace label (with per-namespace overrides when needed). No more risk of forgetting. Already enabled by default on several Kubernetes distributions (Talos being one, but not the only one).

Priority 3 — MicroVMs (Kata Containers, Firecracker) For truly untrusted workloads. Real hardware isolation, overhead now quite reasonable on recent generations.

Priority 4 — UserNamespaces When all else fails. Only for the legitimate cases identified above (builds, legacy, hostile multi-tenancy). This is genuinely the last thing to do.

Conclusion

Kubernetes 1.36 UserNamespaces are the result of a project that officially took five years (KEP-127 dates back to 2021) and has been discussed since nearly the dawn of Kubernetes. For shared build platforms and multi-tenant SaaS running user-provided code, it’s a potentially useful building block — particularly to prevent one customer’s app from reading another’s in the event of a container escape without privilege escalation.

For everything else — that is to say, 99% of production clusters — that’s not where container security starts. And that’s precisely the problem with this kind of infographic.

LinkedIn infographics selling effortless security are dangerous: “keep your 800MB root image full of tools, just add hostUsers: false, and you’re protected.” That’s exactly the wrong approach. Real container security is built in the Dockerfile, not in the PodSpec.

If you’re enabling UserNamespaces to secure an application whose source code you own, you’ve probably missed a step in your secure software development lifecycle.

References

KEP-127: Support for User Namespaces
Kubernetes docs — User Namespaces
Pod Security Standards
CVE-2019-5736 — runc: write to host’s /proc/self/exe from inside the container
CVE-2022-0492 — cgroups v1: escape via unshare, UserNS helps but runAsNonRoot does too
CVE-2024-21626 — runc “Leaky Vessels”: file descriptor leak to host working directory
Distroless containers — GoogleContainerTools

Running Docker containers with Proxmox VE (and LXC)

Mon, 07 Nov 2022 06:30:00 +0200

Introduction

I found several tutorials to install Docker (engine) inside an LXC container under Proxmox VE to then run Docker containers inside it. The problem is that you have to log into the LXC container before you can interact with your Docker containers.

You can also install Docker directly on Proxmox VE but it’s not really recommended and you don’t have the ease of use of LXC with Proxmox GUI.

And actually, it turns out you can run Docker images as the OS of an LXC container and thus have Docker controlled by Proxmox. So we won’t have 100% identical behavior to if you had done a docker run on your machine.

However, we’ll gain in exchange a container totally identical to your other LXC containers in your cluster (with all associated functions) but launched with a Docker image.

But first I suggest we go back a bit before showing you how…

Context

You know, I’ve been doing Proxmox VE for a while. Proxmox VE is a great turnkey and production-ready distribution for server virtualization (it’s even hyperconverged if you enable the Ceph part).

There are lots of little things I like about Proxmox VE, and one of them is that we can create Linux containers with LXC rather than QEMU virtual machines. I wrote an article to talk about it.

In use (except for specific kernel configuration), the LXC container behaves strictly identically to a VM in Proxmox, but for a fraction of the resources! We have an OS, we install applications on it, like a VM.

An nginx frontend for several applications. The complete LXC container only consumes a few MB

Admittedly it’s much less isolated than a real VM (because it’s a container, we’re isolated from other processes, but we run on the hypervisor’s kernel directly).

But with so few resource needs, I can host many small applications in different contexts with very small physical machines (Atom 4GB RAM at €6 per month at OneProviders) which would be impossible with a real VM.

It’s good, but not enough

However, LXC is not the most known/hyped containerization technology. For years on the forum, whenever people ask for Docker support in Proxmox VE, they get sent away. Not always very nicely either…

Yet, there are several reasons to want to run Docker containers on your Proxmox infrastructure:

First, Docker isn’t as much stateless and/or ephemeral workloads as we think. Many containerized applications would have their place on a virtualization cluster.
Then because many software vendors have become lazy. Some no longer manage installation processes (or only on one OS) and just provide a Docker image that they maintain up to date.

So we have to settle for VMs in Proxmox to run Docker containers (see my point at the beginning).

LXC supports OCI images

Plot twist. A few days ago, I discovered that LXC, Proxmox VE’s containerization engine, was compatible with the OCI format.

buzzwrd.me - Creating LXC containers from docker and OCI images

For those who don’t know OCI, it’s an organization aiming to create a standard to unify the way containers are stored.

The Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes. opencontainers.org/

And Docker respects this format: we can therefore theoretically run from LXC containers whose base image is a Docker image.

So, when do we start?

Disclaimer: what follows is tinkering. Nothing is supported and I don’t recommend it in production. There are also likely limitations (especially regarding storage).

First, some dependencies are missing for LXC’s OCI function to be usable under Proxmox VE:

sudo apt install skopeo umoci jq

Then, by default, LXC containers attach to a Linux bridge called lxcbr0. However this bridge probably doesn’t exist on your Proxmox VE installation (not installed by default).

In quick and dirty, we can just replace the default name, although ideally we should find the flag in the CLI to change the bridge. Depending on how you configured the network in your Proxmox VE server, you’ll probably use vmbr0 or vmbr1:

sed -i 's/lxcbr0/vmbr1/g' /etc/lxc/default.conf

Now that we have everything, we can run our LXC containers from docker images. Here I’m running the official alpine:latest image, but I could have taken any other:

lxc-create 500 -t oci -- --url docker://alpine:latest

I created a container called “500” (because VMs and LXC containers in proxmox VE are called with numbers). The image is downloaded then the container created:

We can now start it (with lxc-execute we run the container and open a prompt directly inside it, but we also have lxc-start and lxc-attach)

lxc-execute 500

As is, Proxmox VE doesn’t know our container exists and it won’t appear in the UI. But it works ;-).

Bonus

We can even cheat and make Proxmox believe it’s an LXC container it created itself by creating a 500.conf file, in the /etc/pve/lxc/ folder

The only mandatory fields are the following:

arch (amd64 unless you’re on ARM)
cores (the CPU limit to impose on the container)
hostname
memory (the RAM limit to impose on the container)
ostype
rootfs

Once the file is created, the container automatically appears in the host’s VM list. We can start it and it works :)

The container can be started, stopped, etc. All from the Proxmox VE GUI :)

Have fun!

Source

Kubernetes Compliance Policies with Kyverno - part 2

Mon, 05 Sep 2022 06:00:00 +0200

Part 1 recap

In the previous article, I introduced Kyverno, a “cloud native” tool for managing compliance policies on Kubernetes.

To illustrate the tool’s capabilities, I used an example: preventing two Ingresses from using the same entry URL. I had done the same exercise with OPA in an earlier article (Open Policy Agent (OPA) and its Kubernetes counterpart, Gatekeeper) and compared the two syntaxes (Kyverno vs rego) at the end.

I have a clear preference for Kyverno’s syntax, even though OPA has been around longer and isn’t limited to Kubernetes (Kyverno is Kube-only).

Going a bit further

Beyond my example, I mentioned that Kyverno provides a very large number of pre-written policies (191 at the time of writing).

kyverno.io/policies

By default, most available policies are set to “audit” mode initially. Don’t forget to switch them to “enforce” mode once your cluster is mature enough.

Obviously, when you’re just getting started, you don’t really know where to begin. The simplest approach is to deploy the kyverno/kyverno-policies Helm chart, which contains the Kyverno equivalent of the “Pod Security Standards” (PSPs being officially deprecated since 1.21 I think, and no longer usable in 1.25).

helm install kyverno-policies kyverno/kyverno-policies -n kyverno
[...]
Thank you for installing kyverno-policies v2.5.2 😀

We have installed the "baseline" profile of Pod Security Standards and set them in audit mode.

Visit https://kyverno.io/policies/ to find more sample policies.

Here’s what was deployed (the last one was already there from the previous article):

kubectl get clusterpolicy
NAME BACKGROUND ACTION READY
disallow-capabilities true audit true
disallow-host-namespaces true audit true
disallow-host-path true audit true
disallow-host-ports true audit true
disallow-host-process true audit true
disallow-privileged-containers true audit true
disallow-proc-mount true audit true
disallow-selinux true audit true
restrict-apparmor-profiles true audit true
restrict-seccomp true audit true
restrict-sysctls true audit true
unique-ingress-host false enforce true

With that, we already have plenty to work with to educate our users (developers) on producing secure Kubernetes deployments.

User experience

One of the annoying things about both tools is that out of the box, they’re not very user friendly. Personally, running a kubectl get events command (or kubectl get events --field-selector=reason=PolicyViolation in Kyverno’s case) to figure out why a deployment failed doesn’t bother me much (I almost enjoy it).

But for some people, it’s a bit off-putting, and I can understand that.

My goal is to make infrastructure simpler for people whose job it isn’t, and imposing my way of working isn’t necessarily the best approach.

Often, to help adopt a new tool or practice, the simplest thing is to provide a graphical interface.

And it turns out Kyverno has a tool for that.

Kyverno policy reporter

In reality, the tool wasn’t originally designed for that exact purpose. But you’ll see it’s actually even better.

Historically, it’s called policy reporter because the developers needed a tool to easily and quickly export the PolicyReports I mentioned in the previous article:

kubectl describe policyreport polr-ns-default
Name: polr-ns-default
Namespace: default
Labels: managed-by=kyverno
Annotations: <none>
API Version: wgpolicyk8s.io/v1alpha2
Kind: PolicyReport
[...]
Results:
 Category: Sample
 Message: The Ingress host name must be unique.
 Policy: unique-ingress-host
 Resources:
 API Version: networking.k8s.io/v1
 Kind: Ingress
 Name: ingress2
[...]
Summary:
 Error: 0
 Fail: 2
 Pass: 0
 Skip: 2
 Warn: 0

The policy reporter’s role is to export PolicyReports to several external sources for easier processing:

Grafana Loki
Elasticsearch
Slack
Discord
MS Teams
Policy Reporter UI
S3

I’ll cover 2 of them: Slack and Policy Reporter UI

Policy Reporter UI

I’ll start with the UI since it’s the simplest to set up. You just need to deploy the Policy Reporter Helm chart, making sure to specify that you want the UI:

helm repo add policy-reporter https://kyverno.github.io/policy-reporter
helm repo update
helm install -n kyverno policy-reporter policy-reporter/policy-reporter --set kyvernoPlugin.enabled=true --set ui.enabled=true --set ui.plugins.kyverno=true

kubectl -n kyverno get pods
NAME READY STATUS RESTARTS AGE
kyverno-5f8bfd6fc5-xp6bm 1/1 Running 1 (59m ago) 33d
policy-reporter-74cc9bf4b9-9ggwh 1/1 Running 0 18s
policy-reporter-kyverno-plugin-6b48c4bc5f-5rwfb 1/1 Running 0 18s
policy-reporter-ui-5c9449d58-f9fwf 1/1 Running 0 18s

We could obviously add an Ingress, but if you’re lazy like me during this demo, a simple port-forward will do.

kubectl port-forward service/policy-reporter-ui 8082:8080 -n kyverno

The UI itself is fairly intuitive. It includes notably:

a Dashboard with a big global indicator showing how many policy violations you have
a page with your “policy reports”, filterable by namespace or policy
the list of your policies
…

Slack

You can immediately see the value of alerting administrators in Slack (or Teams…) when a policy violation occurs.

The configuration is fairly trivial — you just need to add a webhook in Slack and a small piece of configuration in Kyverno:

kyvernoPlugin:
 enabled: true
ui:
 enabled: true
 plugins:
 kyverno: true
target:
 slack:
 webhook: "https://hooks.slack.com/services/T0xxxxxxxx"
 minimumPriority: "medium"
 skipExistingOnStartup: true
 sources:
 - kyverno

And we re-apply all this to our Helm release:

helm upgrade -n kyverno policy-reporter policy-reporter/policy-reporter -f kyverno-values.yaml

Note: If you don’t know how to do this, you need to be admin of the Slack workspace, create an application at https://api.slack.com/apps, enable Incoming Webhooks, and create one with permissions on a channel.

How about we trigger a policy violation, just to see?

To keep it simple, I went back to the example from the previous article. I switch my unique-ingress-host policy back to audit mode and recreate the problematic ingress:

kubectl edit clusterpolicy unique-ingress-host
kubectl apply -f ingress2.yaml

Instantly, 2 failures show up in the Kyverno UI, along with messages in Slack:

Victory :)

Additional Resources

Kubernetes Compliance Policies with Kyverno

Mon, 01 Aug 2022 06:00:00 +0000

Introduction

If the title of this article sounds familiar, it’s probably because I already used it 2 years ago when talking about another compliance policy management tool: Open Policy Agent (OPA) and its Kubernetes counterpart, Gatekeeper.

And I won’t lie, I haven’t done much OPA since that 2020 article…

The first reason is that I changed companies right at that time and had other priorities at my new employer.

The second reason, probably more accurate but perhaps less flattering, is that OPA is not exactly user friendly. Especially because of rego, the compliance policy language.

Since then, I discovered Kyverno, a newcomer in the “cloud native” game (recently promoted to “CNCF sandbox” level) that offers compliance policies for Kubernetes only (unlike OPA + rego which are general-purpose), written in YAML.

With Kyverno, policies are managed as Kubernetes resources and no new language is required to write policies.

Take that, rego ;-)

Note: This article will remain relatively simple, it’s an introduction. We’ll go further in subsequent articles (writing your own policy, the UI, etc).

I’m not going to innovate much compared to the OPA article and will start from the same use case: preventing two developers from using the same URL for their web app deployed on Kubernetes (because we’ve seen it’s pretty bad when that happens).

And you’ll see that the initial learning curve is MUUUCH more accessible with Kyverno than with OPA (even though it might annoy half of my Twitter followers that I’m once again talking about YAML).

Prerequisites

Same setup as for OPA / Gatekeeper, you need Kube 1.14 to use Kyverno. Version 1.14 wasn’t the latest back in 2020, it’s practically prehistoric by now ;-).

Your Kubernetes cluster version must be above v1.14 which adds webhook timeouts.

Obviously, you also need admin access to the cluster, because we’ll have to create Validating Admission Webhooks and CRDs. Again, I’ll refer you to the previous article and the official documentation if this doesn’t ring a bell.

In short, we’re going to insert ourselves into Kubernetes’ deployment process to prevent people from doing things that don’t follow your best practices (with webhooks) and we’ll add extra APIs to extend Kubernetes (the CRDs), so we can describe what’s not allowed.

Deployment

Installation-wise, we’re keeping it simple and efficient.

Kyverno offers several Helm Charts:

one for Kyverno itself
one for Policy Reporter
one containing a “pack” of default rules, configured as “non-blocking” (basically the equivalent of what you could do with Pod Security Policies)

For this article, I’m only interested in Kyverno itself. But as I mentioned above, we’ll discuss the other two in a future post.

Let’s start by adding Kyverno itself:

$ helm repo add kyverno https://kyverno.github.io/kyverno/
$ helm repo update
$ helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace
 NAME: kyverno
 LAST DEPLOYED: Mon Jul 25 21:55:11 2022
 NAMESPACE: kyverno
 STATUS: deployed
 REVISION: 1
 NOTES:
 Chart version: v2.5.2
 Kyverno version: v1.7.2

 Thank you for installing kyverno! Your release is named kyverno.
 ⚠️ WARNING: Setting replicas count below 3 means Kyverno is not running in high availability mode.

 💡 Note: There is a trade-off when deciding which approach to take regarding Namespace exclusions. Please see the documentation at https://kyverno.io/docs/installation/#security-vs-operability to understand the risks.

Note: you’ll notice the WARNING during install. By default, Kyverno is deployed with a single replica. If it crashes, you can no longer deploy or modify anything on your cluster (it happened to me). The solution to recover is to delete Kyverno’s webhooks to unblock the situation. And of course, give it more room (higher limits, especially for RAM) and definitely 3 replicas (or more)…

Note 2: It’s probably best to exclude certain namespaces (kube-system, kyverno, …) from Kyverno’s scope to avoid nasty surprises. Thibault Lengagne’s blogpost at Padok also discusses this (link at the bottom of the article).

What did we deploy?

At first glance, not much. If you run kubectl -n kyverno get all, you’ll find just a Pod (+ReplicaSet +Deployment) and two Services.

But behind the scenes, we’ve deployed much more than that ;)

Looking for CRDs, you’ll see we have quite a few new objects at our disposal.

$ kubectl api-resources --api-group=kyverno.io
 NAME SHORTNAMES APIVERSION NAMESPACED KIND
 clusterpolicies cpol kyverno.io/v1 false ClusterPolicy
 clusterreportchangerequests crcr kyverno.io/v1alpha2 false ClusterReportChangeRequest
 generaterequests gr kyverno.io/v1 true GenerateRequest
 policies pol kyverno.io/v1 true Policy
 reportchangerequests rcr kyverno.io/v1alpha2 true ReportChangeRequest
 updaterequests ur kyverno.io/v1beta1 true UpdateRequest

Similarly, we also have new webhooks:

$ kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io
 NAME WEBHOOKS AGE
 kyverno-policy-validating-webhook-cfg 1 8m48s
 kyverno-resource-validating-webhook-cfg 2 8m48s

$ kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io
 NAME WEBHOOKS AGE
 kyverno-policy-mutating-webhook-cfg 1 11m
 kyverno-resource-mutating-webhook-cfg 2 11m
 kyverno-verify-mutating-webhook-cfg 1 11m

What’s it for? How does it work?

To simplify, we’ll describe what we don’t want to allow on our cluster using the ClusterPolicy CRD.

Based on the rules we write, the webhooks will insert themselves into Kubernetes’ normal resource deployment process to either allow or reject them.

Kubernetes Blog - A Guide to Kubernetes Admission Controllers

Ingress with the same URL example

Enough theory, the easiest way is to show you. As I mentioned at the beginning of the article, to compare the two products, I’m going to start from the same use case.

In my Kubernetes clusters, I want to prevent two applications from creating an Ingress with the same URL, because in my case, that’s not allowed. It’s a sign of a copy/paste error from one Helm Chart to another (or any other human error).

The consequence of this error is a conflict in routing incoming HTTP requests, and you end up with some kind of janky round robin where half the requests hit the wrong backend. In production, that’s a bad look, and you risk ending up with customers complaining on Twitter that the application is down ;-).

Just as OPA provides a community policy to address this problem, Kyverno has the same.

We’ll look in more detail in the next article at how these rules are built. For now, we’ll just deploy this rule in “audit” mode and see if it works.

spec: validationFailureAction: audit

$ kubectl apply -f https://raw.githubusercontent.com/kyverno/policies/main/other/unique-ingress-paths/unique-ingress-paths.yaml
 clusterpolicy.kyverno.io/unique-ingress-host created

If you have a Kubernetes cluster at hand, I suggest you apply these two Ingresses, configured to receive traffic from the same URL (toto.example.org):

$ for counter in 1 2; do
cat > ingress${counter}.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: ingress${counter}
spec:
 rules:
 - host: toto.example.org
 http:
 paths:
 - path: /
 pathType: Prefix
 backend:
 service:
 name: service${counter}
 port:
 number: 4200
EOF
done

$ kubectl apply -f ingress1.yaml
 ingress.networking.k8s.io/ingress1 created
$ kubectl apply -f ingress2.yaml
 ingress.networking.k8s.io/ingress2 created

Normally, nothing happens. Kyverno doesn’t prevent you from creating these two Ingresses since we’re in “validationFailureAction: audit” mode (even if the backends don’t actually exist behind them, that doesn’t matter).

$ kubectl get ingress
 NAME CLASS HOSTS ADDRESS PORTS AGE
 ingress1 <none> toto.example.org 80 90s
 ingress2 <none> toto.example.org 80 87s

However, if you check the Events, you’ll find PolicyViolation entries:

$ kubectl get events --field-selector=reason=PolicyViolation
LAST SEEN TYPE REASON OBJECT MESSAGE
118s Warning PolicyViolation ingress/ingress1 policy unique-ingress-host/check-single-host fail: The Ingress host name must be unique.
118s Warning PolicyViolation ingress/ingress1 policy unique-ingress-host/check-single-host fail: The Ingress host name must be unique.
118s Warning PolicyViolation clusterpolicy/unique-ingress-host Ingress default/ingress1: [check-single-host] fail
115s Warning PolicyViolation clusterpolicy/unique-ingress-host Ingress default/ingress2: [check-single-host] fail

The fact that we have 2 Ingresses pointing to the same URL is properly detected and shows up in Kubernetes Events. Nice :)

In a much more verbose format, you can also retrieve PolicyReports, which say the same thing in an even less readable way.

kubectl describe policyreport polr-ns-default
Name: polr-ns-default
Namespace: default
Labels: managed-by=kyverno
Annotations: <none>
API Version: wgpolicyk8s.io/v1alpha2
Kind: PolicyReport
[...]
Results:
 Category: Sample
 Message: The Ingress host name must be unique.
 Policy: unique-ingress-host
 Resources:
 API Version: networking.k8s.io/v1
 Kind: Ingress
 Name: ingress2
[...]
Summary:
 Error: 0
 Fail: 2
 Pass: 0
 Skip: 2
 Warn: 0

Let’s clean up for what’s next:

$ kubectl delete ingress ingress1 ingress2

“The law is me”

Imagine you’ve now thoroughly tested your policies, your developers no longer generate manifests that don’t follow your best practices (e.g., no root containers, mandatory labels present, correct limits/requests, etc). Your cluster is clean!

You can switch your compliance policies to “enforcing” mode. Unlike “audit” mode, this mode is blocking, and manifests that don’t comply with the policies will be rejected by Kubernetes.

$ kubectl patch clusterpolicies.kyverno.io unique-ingress-host --patch '{"spec":{"validationFailureAction":"enforce"}}' --type merge

Yes, I’m using “patch” instead of “edit” to show off ;-)

And let’s test again:

$ kubectl apply -f ingress1.yaml
 ingress.networking.k8s.io/ingress1 created

So far, so good

$ kubectl apply -f ingress2.yaml
Error from server: error when creating "ingress2.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Ingress/default/ingress2.yaml was blocked due to the following policies

unique-ingress-host:
 check-single-host: The Ingress host name must be unique.

AND BAM!

Conclusion

In this basic (but genuinely useful) example, we can just as easily prevent users from creating Ingresses with the same URL as we could with OPA.

So one might ask “why prefer Kyverno?”. Beyond some other nice features we’ll explore in future posts, my answer fits in a single image.

My (very personal) take is that rego is unreadable and verbose, and therefore unmaintainable. Again, this is personal, but I’ve never managed to get into it.

Without going as far as idealizing Kyverno’s language*, I find it is clearly much simpler to write, but more importantly, to read. If you ever need to modify/write/maintain your own policies, this will be very handy.

(*Yes, there is a language, especially for pattern matching… and as an annoying point, it’s not jsonpath like kubectl, but JMESPath, similar but not identical…)

Additional Resources

Kubernetes Compliance Policies with OPA and Gatekeeper

Mon, 20 Jul 2020 06:35:00 +0000

Introduction

One of the problems I encountered while managing Kubernetes clusters shared across multiple teams was that the manifests being deployed were not always perfectly controlled.

In some cases, manifests were copy/pasted from other teams, sometimes causing small surprises (no label indicating the team responsible for the app, incorrect Ingress information…). One of the solutions we found was to maintain an up-to-date, as-generic-as-possible Helm Chart, with only values for developers to fill in. This Chart is jointly maintained by Ops and tech leads.

However, maintaining this template is complex, and some applications diverge too much from the general framework to comply, even by forking and modifying parts of it. And some issues like verifying images or registries running on the platform were not addressed by this solution.

Open Policy Agent / Gatekeeper

https://www.thingiverse.com/thing:1236133

Rather than trying at all costs to unify the way applications are deployed in the cluster (and having to manage exceptions), another approach is to add a tool that would verify, at deployment time, that what’s being deployed follows the company’s best practices in terms of security and configuration.

As you may have guessed, the tool for this is Open Policy Agent (or OPA). It’s an open-source generic policy engine (CNCF project at the Incubating stage).

Since it’s a generic engine, there’s also another project, Gatekeeper, which handles the interaction between OPA and Kubernetes.

Prerequisites

To use Gatekeeper, you should have a minimum Kubernetes version of 1.14, which adds webhook timeouts.

If you’re on an earlier version, things get complicated. There’s a bug, fixed in 1.14, that could crash your OPA/Gatekeeper setup. For this reason, if you’re not on 1.14 (and can’t update your platform), you’ll need to skip Gatekeeper (using K8s Policy Controller for instance). That’s a bit trickier and outside the scope of this tutorial.

The “V2”, without Gatekeeper, with Kube-Mgmt and OPA

Of course, you also need admin access on the cluster. Gatekeeper offers a simple way to test your permissions by trying to grant yourself admin rights.

wait for it

kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin \
--user [your user]

Quick tip: if you don’t know the name of the access account you’re currently using, you can run this command:

kubectl config view --template='{{ range .contexts }}{{ if eq .name "'$(kubectl config current-context)'" }}Current user: {{ .context.user }}{{ end }}{{ end }}'
Current user: clusterUser_zwindlerk8s_rg_zwindlerk8s

Installation

I set up a small AKS cluster in version 1.14 for the occasion and deployed the prepackaged version of Gatekeeper.

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml

Tada! And just like that, Gatekeeper is installed on your cluster. Note: The official documentation provides steps to build your own image.

What’s in my cluster?

Taking a closer look at what the manifest does, here’s what you deploy when running the above command:

a Namespace gatekeeper-system
a ServiceAccount gatekeeper-admin, associated with a Role and a ClusterRole gatekeeper-manager-role via (respectively) a RoleBinding and ClusterRoleBinding gatekeeper-manager-rolebinding
a Deployment gatekeeper-controller-manager and gatekeeper-audit
a Service gatekeeper-webhook-service
a Secret gatekeeper-webhook-server-cert containing a certificate

Nothing exceptional so far. Where it gets interesting is:

two CRDs (Custom Resource Definitions): configs.config.gatekeeper.sh and constrainttemplates.templates.gatekeeper.sh
a validating webhook configuration gatekeeper-validating-webhook-configuration

Validating Webhook

The validation webhook first. The first time I heard about validating webhooks was during the talk 101 Ways to “Break and Recover” Kubernetes Cluster at Kubecon 2018.

The problems these Oath (formerly Yahoo) employees were facing were identical to mine. Some teams, when pulling (a.k.a. copy/pasting) manifests from other teams, forgot to change the Ingress URLs. This resulted in user requests being distributed randomly between 2 completely different services.

The solution proposed at that conference was to use Kubernetes’ Validating/Mutating Admission Webhooks (roughly stable since 1.11).

Back in 2018, when digging into the topic, I found very little documentation about it beyond the official docs (I’ve found more since). Another issue: Admission webhooks require developing your own Controller, which wasn’t very accessible (see links at the end of the article).

Fortunately, Gatekeeper and OPA will handle that for us.

Custom Resource Definition

For those unfamiliar with CRDs, they are simply extensions of the Kubernetes API. The big advantage of CRDs is that they provide a way for third-party vendors to add their own logic inside Kubernetes.

I already talked about CRDs in my article on Rook which allows automated management of Ceph clusters (and more!) via the CephCluster CRD. All common administration tasks are integrated into Kubernetes (even though Kube has no native knowledge of them), managed by a Controller included in Rook, and configured via CRDs.

Gatekeeper will add 3 components to our Kubernetes:

a Config object
a ConstraintTemplate object
a Constraint object

Config

The principle of Gatekeeper, as shown in the following diagram, is that it hooks into your Kubernetes API server and “synchronizes” events for new object creation, looking for objects that don’t comply with your compliance policies.

Source: kubernetes.io/blog

By default, if you don’t configure it, Gatekeeper doesn’t listen to events on any Kubernetes objects… so it will do… absolutely nothing!

Let’s fix that right away by telling Gatekeeper to listen to all events on Ingresses:

cat gatekeeper/demo/basic/sync.yaml
apiVersion: config.gatekeeper.sh/v1alpha1
kind: Config
metadata:
name: config
namespace: "gatekeeper-system"
spec:
sync:
syncOnly:
- group: "extensions"
version: "v1beta1"
kind: "Ingress"
- group: "networking.k8s.io"
version: "v1beta1"
kind: "Ingress"

ConstraintTemplate

Before you can define a constraint, you must first define a ConstraintTemplate, which describes both the Rego that enforces the constraint and the schema of the constraint.

The idea here is that before you start creating constraints on the cluster, you need to create constraint templates. Think of these templates as the object that links the function (the OPA rego code) on one side with parameters on the other.

This extra step allows us to potentially reuse the same template for several different constraints.

Your first template

For the first time, we can use the sample templates provided on the OPA/Gatekeeper GitHub, which enforces the presence of a label on a given Kube object type, since it’s simpler (k8srequiredlabels_template.yaml).

However, since the beginning of this article, I’ve been talking about preventing teams from using the same FQDN in Ingresses for different services. Because if that were to happen, as a reminder, we’d end up with some kind of janky load balancer redirecting half of user requests to one application and the other half to another for the same URL.

And it just so happens that this template will help us ensure that!

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
name: k8suniqueingresshost
spec:
crd:
spec:
names:
kind: K8sUniqueIngressHost
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8suniqueingresshost
identical(obj, review) {
obj.metadata.namespace == review.object.metadata.namespace
obj.metadata.name == review.object.metadata.name
}
violation[{"msg": msg}] {
input.review.kind.kind == "Ingress"
re_match("^(extensions|networking.k8s.io)$", input.review.kind.group)
host := input.review.object.spec.rules[_].host
other := data.inventory.namespace[ns][otherapiversion]["Ingress"][name]
re_match("^(extensions|networking.k8s.io)/.+$", otherapiversion)
other.spec.rules[_].host == host
not identical(other, input.review)
msg := sprintf("ingress host conflicts with an existing ingress <%v>", [host])
}

At first glance, it’s pretty dense. But then again, I didn’t pick the simplest example either, and if you re-watch the video Kubecon 2019 | Intro: Open Policy Agent - Rita Zhang, Microsoft & Max Smythe, Google, it’s actually fairly easy to understand.

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/agilebank/dryrun/k8suniqueingresshost_template.yaml
constrainttemplate.templates.gatekeeper.sh/k8suniqueingresshost created
kubectl get constrainttemplate
NAME AGE
k8suniqueingresshost 23m

And a Constraint

Now that we have our template, we can simply instantiate it with the variables we’re interested in.

Sticking with the demo example, we ensure that across the entire cluster, no URL is used twice for two different Ingresses by instantiating this constraint:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sUniqueIngressHost
metadata:
name: unique-ingress-host
spec:
enforcementAction: dryrun
match:
kinds:
- apiGroups: ["extensions", "networking.k8s.io"]
kinds: ["Ingress"]

I’m going to slightly modify this Constraint because, as you can see, it defines an enforcementAction: dryrun, which has no immediate effect (it only logs an error for future audit). It’s a great feature for getting up to speed with OPA, but for my demo it’s less fun…

So I apply the file and change the enforcementAction:

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/agilebank/dryrun/unique-ingress-host.yaml
kubectl patch K8sUniqueIngressHost.constraints.gatekeeper.sh unique-ingress-host -p '{"spec":{"enforcementAction":"deny"}}' --type=merge

Let’s try it out

At this point, we should have everything. Gatekeeper listens to the API server for all events on Ingress objects (via the Config). We have a ConstraintTemplate with rego code that checks we don’t already have an identical URL in existing Ingresses, and a Constraint that defines what to do (dryrun) and on which objects (Ingresses).

Let’s run the demo :D

# Create the namespace that will contain the demo objects
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/agilebank/bad_resources/namespace.yaml
namespace/production created
# Create an Ingress with a URL that doesn't exist yet
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/agilebank/dryrun/existing_resources/example.yaml
ingress.extensions/ingress-host created
# Create a second Ingress with the SAME URL
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/agilebank/dryrun/bad_resource/duplicate_ing.yaml

And bam!

[denied by unique-ingress-host] ingress host conflicts with an existing ingress <example-host.example.org>

Conclusion

As I explained in the introduction, adding compliance policies and constraints to a Kubernetes cluster was not trivial.

Without going as far as saying it’s become easy, OPA and Gatekeeper make things simpler (no need to develop and then host your own microservice for each Admission/Validation Webhook). You still need to learn a new language (rego) to start doing cool things, but even with the default templates, there’s already plenty to work with.

Another really interesting point with OPA is the ability to log and audit all compliance errors using dryrun mode instead of deny, without blocking.

kubectl get K8sUniqueIngressHost.constraints.gatekeeper.sh unique-ingress-host -o yaml
[...]
- enforcementAction: dryrun
kind: Ingress
message: ingress host conflicts with an existing ingress <example-host.example.org>
name: ingress-host2
namespace: default

Resources

Should we have containers ?

Thu, 25 Aug 2016 16:00:21 +0000

A more complete answer to « Why do we have containers » by fntlnz

2 days ago, I stumbled apon fntlnz article Why do we have containers. (dead link, using Internet Archive) The article had been reposted by Docker official Twitter account and the first thing that struck me was the displayed image : a schema comparing full virtualization, para virtualization and linux containers.

And of course, I say this because I didn’t entirely agree with it ;-). But then I read the article and I found many good points.

So I decided to write « Should we have containers ? » as an answer to develop my remarks a little.

Full virtualization vs paravirtualization vs OS level virtualization

First, I’ll go with the schema. I know the point was to provide a really simple schema to help understand the differences between virtualization types but I don’t think it should be presented like that.

Hypervisor and host OS are both displayed in the full/para. I find this be misleading. Of course there is a Host OS and a virtualization engine on top of it. But most of the time, either the OS is really thin (ESXi fits in 8GB SD cards and runs nothing but virtualization) or the virt engine is a part of the kernel (KVM, Xen) just like LXC is a part of the Linux kernel.

Also, the bins/libs are only displayed on OS level virt. This is interesting to put in the « host level virtualization schema » but it should be added on full and para as well because this is one of the advantages host level virt. has over full/para.
Even if your VMs are all identicals, bins and libs have to be present (copied) in each VMs. On contrary, in OS Level virt, you CAN share identicals bins/libs between containers, thus reducing the storage footprint of the virtualized environment.

That’s how I would have presented it

You can see that OS level virtualization and Hypervisors are not so different on this simplified view. The differences appear more clearly when you add x86 Security Rings (0->3). The « overhead » of full virtualization comes from the calls that guest OSes do to hypervisor pilots and the translation of CPU instructions.

On a side note, I put Docker on the left to show that it’s basically an administration suite that helps you run LXC containers

The rest of the analysis

Security

When talking about containers, the security point is often raised.

VMs are really hermetics. Taking control of the hypervisor after hacking a VM should be MUCH harder than taking control of the host OS if you are inside a container with whom the kernel is shared.

But I agree with fntlnz: having to patch only one kernel for all your virtual environments instead of having to patch ALL kernels of your virtual machines is actually a good point.

I encourage you to read this (dead link, using Internet Archive) from Jérôme Petazzoni if you’d like a good presentation on the subject.

Performance

A point that’s not really developed in fntlnz article is performance, but I read it/ear it all the time.

General belief is that VMs are slow. That was true in the « early days » of x86 virtualization, because of the limitations of the x86 architectures which didn’t allow efficient OS virtualisation. Overhead for CPU instructions was known to be of about 15% (which is enormous). Thus, running processes directly on the OS are believed to be more efficent.

But nowadays, modern hypervisors on modern hardware nearly generates no overhead at all over CPU and RAM consumption. You could argue that guest OSes consume ressources while containers don’t. But let’s be honest: with the 32 GB RAM tomcat instances I regularly see, don’t say to me that a Linux guest OS CPU&RAM consumption is significative.

The real difference is the on the storage consumption. First, for N VMs, you save yourself N*xGB of guest OS files. If your Linux templates are good, you don’t consume that much (a few GB at most) but this can add up when we are talking about thousands of instances. [Edit jan. 2020]Not so true today if you host your own hardware and have so deduplicated storage[/Edit]

From there, you still have some margin for gains, mostly through Copy on write and union filesystems :

You make a modification to an official tomcat image to add you configuration files ? Only the delta will be stored.
You launch thousands of containers from this image ? You only consume the container image once.
The real data (user data, business data) is of course the same for both architectures

So, should I use containers ?

The problem here is that’s not the real question. Do you need containers ? Here’s fntlnz point of view.**
**

But if at some extent your motivation consist in decreasing costs related to full virtualization overheads while allowing developers to ship, develop and test code faster we already found two.

fntlnz

I won’t tell you not to use containers or Docker. Truth be told, we use it in dev and production. Docker is awesome.

Yes, Docker reduces overhead of full virtualization. Storage mostly, but it highly depends on your workload. For example if you have the need for millions of really tiny instances, then Guest OS overhead is significative in full virtualization.
Yes, Docker allows developers to ship, develop and test code faster. You still have to change your way to develop yours app but Docker and DevOps is the new thing. Better get on with it ;)

But you shouldn’t use Docker just because of that. Docker containers need you to change your philosophy toward applications and administration.

Shared kernel = same kernel

You need to understand that you’ll share a kernel. For a long time that ment a 100% Linux only which could be problematic in some cases. I just realized that Windows containers exists in Windows 2016 tech preview. And Solaris is coming.

But independantly of platforms, you still you share a kernel. If you want Linux containers, you’ll need Linux Docker nodes. If you need Windows containers, you’ll have to have Windows Docker nodes.

Non-persistant/cloud-ready instances

Docker containers are by design non-persistant and by best practice single-process. This isn’t mandatory of course, but Docker works better that way.

The whole idea is to help you make « cloud-ready » & atomic applications that don’t store anything in the container itself (persistant storage is presented to the container but outside it). This way, you can scale out really quickly: you just start up a few more instances if you need more firepower.

This also modifies the way you do systems/applications administration. A container/application is spinning out of control ? Don’t try to repair it.

Kill it
Launch a new one

The fact that persistant storage is handled outside the containers also helps you to deal with lifecycle management.

Update your application inside the docker image
Shutdown the old container
Start the new container with the new image, plugged to the persistant storage

All that requires of course applications that have been designed this way! This is a massive change in philosophy and old monolithic apps on a Docker platform won’t work as well as microservices will.

System administration

I don’t say solutions don’t exist because they do.

But starting up and shutting down containers on the fly will give headaches to your « old school Nagios » monitoring administrator. What could be done easily in the physical/VM world (installing an agent in the template), each of those having an IP adress to query for health checks is a lot easier than checking on a single host a highly volatile number of isolated processes.

You’ll probably have to adapt and use tools that are designed to handle such cases.

Backup administrator may find it confusing at first to make application consistant data backups when applications are running in an variable number of containers.

And so on…

Final thoughts

Yes, you should definitely look into containers. They are mature, stable, a great new way to speed up development and contribute to save money on hardware.

But be aware that VMs and containers don’t answer all usecases; and that you won’t migrate all your applications overnight!

Docker on Zwindler's Reflection

Kubernetes UserNamespaces: the overhyped GA feature

The infographic that triggered me

What UserNamespaces actually do

What the infographic exaggerates (and leaves out)

Honest comparison: UserNS vs the real alternatives

Real use cases

1. Build-as-a-Service (Buildah, rootless Podman)

2. Hostile multi-tenancy (user code platforms)

3. Hard-coded legacy (Postfix, Dovecot, BIND)

Some counterarguments

Where to invest your security budget

Conclusion

References

Running Docker containers with Proxmox VE (and LXC)

Introduction

Context

It’s good, but not enough

LXC supports OCI images

So, when do we start?

Bonus

Source

Kubernetes Compliance Policies with Kyverno - part 2

Part 1 recap

Going a bit further

User experience

Kyverno policy reporter

Policy Reporter UI

Slack

How about we trigger a policy violation, just to see?

Additional Resources

Kubernetes Compliance Policies with Kyverno

Introduction

Prerequisites

Deployment

What did we deploy?

Ingress with the same URL example

“The law is me”

Conclusion

Additional Resources

Kubernetes Compliance Policies with OPA and Gatekeeper

Introduction

Open Policy Agent / Gatekeeper

Prerequisites

Installation

What’s in my cluster?

Validating Webhook

Custom Resource Definition

Config

ConstraintTemplate

Your first template

And a Constraint

Let’s try it out

Conclusion

Resources

Should we have containers ?

A more complete answer to « Why do we have containers » by fntlnz

Full virtualization vs paravirtualization vs OS level virtualization

The rest of the analysis

Security

Performance

So, should I use containers ?

Shared kernel = same kernel~~ ~~

Non-persistant/cloud-ready instances

System administration

Final thoughts

Shared kernel = same kernel