Kubernetes on Zwindler's Reflection

zeropod v0.12.0: One Year Later, Does Scale-to-Zero Deliver?

Sat, 30 May 2026 12:00:00 +0200

What is zeropod?

A little less than a year ago, I published a first article, reviewing a tool called zeropod.

Zeropod is a Kubernetes runtime (more specifically a containerd shim) that automatically checkpoints containers to disk after a certain amount of time of the last TCP connection.

While in scaled down state, it will listen on the same port the application inside the container was listening on and will restore the container on the first incoming connection.

At the time, the stable version was v0.6.x, and I tested it on a k3s cluster. The results were mixed: it mostly worked, but with deal-breaking limitations for serious use (probes impossible, flaky behavior under load, checkpointing times a bit high for my taste).

Since then, the project has evolved quite a bit (now v0.12.0), with promises of fixes and improvements. So I decided to give it another shot to see where things stand.

One other change: I abandoned k3s for this test series, for reasons I’ll detail throughout the article.

Prerequisites

This time, I used a freshly provisioned server at my favorite hosting provider:

An Ubuntu 24.04.3 LTS (Noble) server, kernel 6.17.0-35 HWE, 7.7 Gi RAM, 100G disk
A Kubernetes cluster set up with kubeadm, flannel as CNI
Vanilla containerd
local-path-provisioner (Rancher) for default local storage
No cert-manager or Ingress, keeping it simple this time

Installation

I’ve installed kubeadm many times and you probably have too, so I won’t insult you with yet another tutorial.

# Install kubeadm, kubelet, kubectl + containerd
sudo kubeadm init --pod-network-cidr=10.42.0.0/16
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

Once the cluster is running, installing zeropod is trivial:

# Apply the generic kustomize
kubectl apply -k https://github.com/ctrox/zeropod/config/generic

# Add a label to our single node
kubectl label node <your-node> zeropod.ctrox.dev/node=true

# Verify the pod is running
kubectl -n zeropod-system wait --for=condition=Ready pod -l app.kubernetes.io/name=zeropod-node

That’s it. No special flags, no extra configuration. On kubeadm, kubelet is a native binary (/usr/bin/kubelet) detected automatically by zeropod.

Note about k3s: on k3s, the zeropod documentation differs since the config to use is config/k3s. The kustomize adds a -probe-binary-name=k3s flag on the initContainer so the shim knows the kubelet is embedded in the k3s binary. During my tests, even with this flag, the behavior wasn’t as expected (the socket tracker didn’t filter probes correctly). With the default config, the flag is on the initContainer but not on the manager. I suspected there was another component to patch, but I didn’t investigate further.

The DaemonSet deploys the following images:

ghcr.io/ctrox/zeropod-manager:v0.12.0
ghcr.io/ctrox/zeropod-installer:v0.12.0
ghcr.io/ctrox/zeropod-criu:v4.2 (CRIU has been updated since v0.6.x)

Let’s verify the zeropod runtimeClass is available:

kubectl get runtimeclass
NAME HANDLER AGE
zeropod zeropod 30m

First test: nginx

Like last time, let’s start with a simple nginx test. Deploy a basic pod:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx
spec:
 replicas: 1
 selector:
 matchLabels:
 app: nginx
 template:
 metadata:
 annotations:
 zeropod.ctrox.dev/scaledown-duration: 10s
 labels:
 app: nginx
 spec:
 runtimeClassName: zeropod
 containers:
 - image: nginx
 name: nginx
 ports:
 - containerPort: 80

The important parts are the annotation zeropod.ctrox.dev/scaledown-duration: 10s and runtimeClassName: zeropod.

Shortly after deployment, zeropod detects the absence of traffic. The container is checkpointed. Looking at the manager logs:

{"time":"2026-05-29T14:26:37.269453861Z","level":"INFO","msg":"status event","container":"nginx","pod":"nginx-bench-7c65c8874-6pts7","phase":"RUNNING","duration":"0s"}

Then 10 seconds later:

{"time":"2026-05-29T14:26:47.465510937Z","level":"INFO","msg":"status event","container":"nginx","pod":"nginx-bench-7c65c8874-6pts7","phase":"SCALED_DOWN","duration":"191.259383ms"}

The duration field in the SCALED_DOWN log is the checkpoint time: 191ms. That’s already significantly better than the ~400ms from v0.6.x on k3s (likely thanks to an update, perhaps CRIU 4.2?).

Restore

When we send an HTTP request, the container wakes up:

time curl http://<POD_IP>/ -s -o /dev/null -w "%{http_code} (%{time_total}s)"
HTTP 200 (0.101s)

real 0m0.101s
user 0m0.003s
sys 0m0.003s

101ms to restore nginx and serve a page. That’s comparable to the ~92ms from the original article.

Nice new feature: under v0.6.x, kubectl top pods returned an error for checkpointed pods:

# v0.6.x
kubectl top pods
error: Metrics not available for pod default/php-xxx

This bug was fixed in v0.9.0. Now pods in SCALED_DOWN state return 0m 0Mi:

kubectl top pods
NAME CPU(cores) MEMORY(bytes)
nginx 0m 0Mi

Much cleaner.

Testing liveness probes

This was THE major limitation of v0.6.x: zeropod was incompatible with Kubernetes probes. If you added an httpGet liveness probe to your container, the probe would reset the scale-down timer, and your container would never go SCALED_DOWN. Result: probes unusable, making zeropod unusable in production.

What changed

Two fixes were made between v0.6.x and v0.12.0:

The activator (the eBPF component that listens for traffic during SCALED_DOWN): it intercepts probes and replies 200 directly, without waking the container. That’s the “post scale-down” behavior.
The socket tracker (the component that ignores connections during RUNNING state): since PR #72 (merged August 2025), zeropod can detect connections coming from kubelet and not count them as “real” traffic. That’s the “pre scale-down” behavior.

Real-world test

I deployed nginx with an aggressive liveness probe (periodSeconds: 5) and scaledown-duration: 10s:

spec:
 runtimeClassName: zeropod
 containers:
 - image: nginx
 name: nginx
 livenessProbe:
 httpGet:
 path: /
 port: 80
 periodSeconds: 5

On k3s, this test gave me trouble: the socket tracker couldn’t filter probe connections, which kept resetting the scale-down timer indefinitely. The activator (which filters probes once scale-down has happened) worked fine though.

On kubeadm, the result is immediate:

kubectl get pod -l app=nginx-probe -o json | jq -r '.items[0].metadata.labels["status.zeropod.ctrox.dev/nginx"]'
SCALED_DOWN

The socket tracker correctly filters connections from the native kubelet. The periodSeconds > scaledown-duration rule I had to use on k3s is no longer necessary.

A word on changes between v0.6.x and v0.12.0

Here’s a summary of improvements between the two versions:

Point	v0.6.x	v0.12.0
`kubectl top pods` while scaled-down	❌ Error	✅ `0m 0Mi` (fix v0.9.0)
Checkpoint (nginx)	~400ms	~185ms (-54%)
Restore (nginx)	~92ms	~99ms (stable)
CRIU	v3.x	v4.2
Checkpoint failure handling	basic	metrics + events (v0.11.0)
Configurable proxy timeouts	❌	✅ (v0.11.0)
Inter-node migration	basic	improved + timeouts (v0.10.0)
Probes	❌ Incompatible	✅ Activator + socket tracker

Interesting technical note about CRIU flags: zeropod removed the --tcp-established option in September 2025. Previously, active TCP connections were saved and restored with the container. Now, zeropod uses --tcp-skip-in-flight (when runc >= 1.3 supports it). Practical consequence: if your container has outgoing TCP connections at checkpoint time, they will be lost. You’ll need to re-establish them after restore.

Deploying WordPress (the “““realistic””” use case)

Alright, nginx is fine but not very representative. Let’s deploy a real app with PHP, Apache, and a database.

Let’s reuse the WordPress manifest from the original article, without zeropod on MySQL:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: wordpress
spec:
 replicas: 1
 selector:
 matchLabels:
 app: wordpress
 template:
 metadata:
 annotations:
 zeropod.ctrox.dev/scaledown-duration: 10s
 zeropod.ctrox.dev/container-names: wordpress
 zeropod.ctrox.dev/ports-map: wordpress=80
 labels:
 app: wordpress
 spec:
 runtimeClassName: zeropod
 initContainers:
 - name: wait-for-mysql
 image: mysql:8
 command:
 - sh
 - -c
 - |
 until mysql -h mysql -u root -p${MYSQL_ROOT_PASSWORD} -e "SELECT 1"; do
 echo "Waiting for MySQL..."
 sleep 3
 done
 mysql -h mysql -u root -p${MYSQL_ROOT_PASSWORD} -e "CREATE DATABASE IF NOT EXISTS wordpress;"
 env:
 - name: MYSQL_ROOT_PASSWORD
 value: verySecurePassword
 containers:
 - image: wordpress:latest
 name: wordpress
 ports:
 - containerPort: 80
 env:
 - name: WORDPRESS_DB_HOST
 value: mysql
 - name: WORDPRESS_DB_USER
 value: root
 - name: WORDPRESS_DB_PASSWORD
 value: verySecurePassword
 - name: WORDPRESS_DB_NAME
 value: wordpress

MySQL (without zeropod):

apiVersion: v1
kind: Service
metadata:
 name: mysql
spec:
 ports:
 - port: 3306
 selector:
 app: mysql
 clusterIP: None
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: mysql
spec:
 serviceName: mysql
 replicas: 1
 selector:
 matchLabels:
 app: mysql
 template:
 metadata:
 labels:
 app: mysql
 spec:
 containers:
 - image: mysql:8
 name: mysql
 ports:
 - containerPort: 3306
 env:
 - name: MYSQL_ROOT_PASSWORD
 value: verySecurePassword
 volumeMounts:
 - mountPath: /var/lib/mysql
 name: data
 volumeClaimTemplates:
 - metadata:
 name: data
 spec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 5Gi

WordPress (Apache + PHP) checkpoint:

{"time":"2026-05-29T14:33:35.915988635Z","phase":"SCALED_DOWN","duration":"313.286787ms"}

Checkpoints take around 300ms. Restore is around 200ms:

{"time":"2026-05-29T14:34:31.56806589Z","phase":"RUNNING","duration":"206.33005ms"}

The first curl request confirms it:

time curl http://<POD_IP>/ -s -o /dev/null -w "%{http_code} (%{time_total}s)"
HTTP 302 (0.212s)

212ms, about twice faster than the 454ms from the original article.

The killer test: cascading WordPress + MySQL

In the original article, I attempted the ultimate test: putting both pods (WordPress AND MySQL) with runtimeClassName: zeropod, then sending an HTTP request to WordPress while both are checkpointed.

The scenario goes like this:

After a few seconds, WordPress goes SCALED_DOWN, MySQL goes SCALED_DOWN too
A client sends an HTTP request to WordPress
The WordPress activator intercepts the request and restores the WordPress container
WordPress (Apache+PHP) starts up, executes PHP code requiring a MySQL connection
WordPress connects to MySQL on port 3306
The MySQL activator intercepts the connection and restores MySQL
MySQL responds, WordPress generates the page, Apache sends back the HTTP response

True scale to zero that scales the entire app, not just the frontend/backend.

Important note: I know that you’d probably never want to scale down your database in production, but testing this shows that this approach (eBPF + CRIU) works beyond just scaling webservers to zero, which other tools on the market already do very well.

On k3s: no luck, on kubeadm: victory

On k3s, I still haven’t managed to make this scenario work: WordPress would restore but wouldn’t respond on port 80. I spent quite some time trying to figure out why, without success.

Same test, same zeropod version, but on kubeadm:

curl http://<POD_IP>/ -s -o /dev/null -w "%{http_code} (%{time_total}s)"
HTTP 302 (0.224s)

224ms. Both pods went from SCALED_DOWN to RUNNING, the WordPress page loads. I repeated the test 5 times in a row:

Cycle	Time (curl)	HTTP
1	229ms	302
2	192ms	302
3	227ms	302
4	230ms	302
5	211ms	302

The zeropod logs confirm both containers waking up:

WordPress: SCALED_DOWN → RUNNING
MySQL: SCALED_DOWN → RUNNING

What really changed since v0.6.x

Probes (finally!)

This was my main grievance in the first article. Today, it’s resolved:

Before: impossible to use Kubernetes probes → zeropod unusable in any somewhat serious use case
After: the activator intercepts probes in SCALED_DOWN (replies 200 without restore), and the socket tracker filters kubelet connections in RUNNING state

Stability

The flaky behavior (pod loss under simultaneous load) is gone. Where I had failures in the original article, sequential and simultaneous load tests all pass now.

Performance

Checkpoint gained ~50% speed (185ms vs 400ms). Restore too (200ms vs 454ms). Probably thanks to CRIU v4.2 and zeropod optimizations.

kubectl top pods

This small detail was an eyesore — kubectl top pods crashed on checkpointed pods. Fixed in v0.9.0.

Some remaining limitations

I’d be dishonest if I said everything is perfect. Here’s what’s still problematic:

Since September 2025, zeropod no longer uses --tcp-established for CRIU. This means if your container has outgoing TCP connections at checkpoint time, they will be lost. In practice, for a web server, this means reconnecting to the database after restore. With zeropod, the MySQL connection is re-established automatically (the current PHP request fails, the next one succeeds). This is a detail that may matter depending on your use case.
I only tested WordPress (Apache + mod_php) and MySQL. Applications using websockets, streaming, or long-lived connections might behave differently.
I had difficulties getting it to work properly on k3s.
I observed a glitch (Apache segfault) on the first restore of a freshly created WordPress pod. It’s not reproducible after a normal checkpoint/restore cycle, but if you frequently redeploy your pods, you might encounter it.

Verdict

One year later, zeropod seems to deliver on more of its promises:

Even though it wasn’t a dealbreaker, checkpoint/restore performance has improved significantly (~50% faster), which is always welcome
Kubernetes probe support has been added, lifting what I considered the main blocker
General stability is better
The cascade test (WordPress + MySQL) works

I’m still just as hyped about the idea of freezing a container and restoring it 10 seconds later like nothing happened. The magic of CRIU and eBPF combined is starting to materialize, after years of waiting.

Would I put this in production? Let’s say it’s less risky than a year ago. On my personal cluster for fun, why not. For a production database with long-lived connections, I think I’d still pass ;-).

But honestly, the project has evolved well and deserves a closer look.

nodes/proxy GET: One Kubernetes permission too many

Tue, 19 May 2026 12:00:00 +0200

TL;DR

This is a bit old now, but I still wanted to share a quick write-up on the topic.

Back in January, a cybersecurity researcher reported a Kubernetes flaw that generated quite a buzz. It had been a while since we had a Kube vulnerability that got people talking this much, at least from Denis’s memory (yes, I talk about myself in the third person).

Honestly, it’s pretty wild: the nodes/proxy GET RBAC permission allows any ServiceAccount to execute code inside any Pod in the cluster, without leaving a single trace in the audit logs. That’s unfortunate, especially when you have ServiceAccounts named rook-ceph-system that also happen to have read access to all Secrets in the cluster.

This article details the issue, how to check if you are vulnerable, the fixes to apply, and the preventive measures you can put in place if you can’t patch right away.

The problem: WebSocket + Kubelet = un-audited exec

The vulnerability was documented by Graham Helton in this article. Here is how it works.

The Kubernetes API exposes a nodes/proxy subresource that proxies HTTP requests to each node’s Kubelet. The Kubelet itself exposes an API on port 10250, specifically the /exec endpoint which allows executing commands inside a container.

The issue comes from how the Kubelet handles authorizations for WebSocket connections:

kubectl exec uses a WebSocket connection, whose handshake is an HTTP GET
The Kubelet maps this initial GET to the get RBAC verb
It checks nodes/proxy GET, then authorizes the operation
No secondary check is performed for the CREATE verb normally required for /exec

Result: any ServiceAccount with nodes/proxy GET can execute commands in any Pod in the cluster, including system Pods (etcd, kube-apiserver, etc.).

# Exploitation via websocat
websocat --insecure \
 --header "Authorization: Bearer $TOKEN" \
 --protocol "v4.channel.k8s.io" \
 "wss://$NODE_IP:10250/exec/default/nginx/nginx?output=1&error=1&command=id"

And that’s not all. Commands executed via this method do not generate any Kubernetes audit logs (well, assuming you even collect them 🙈). The access goes directly through the Kubelet, which does not report events back to the API server.

The official Kubernetes status on this: Won’t Fix. It is a “design behavior” (note the quotes), addressed via a feature gate (KEP-2862, see below).

Ouch.

The Audit: Vulnerable ServiceAccounts on our clusters

In January 2026, following the publication of Graham Helton’s article, a lot of people had to urgently audit their clusters. You can either manually audit all your Roles / ClusterRoles, or use a detection script provided by the researcher.

As an example, here are three relatively common components that make great candidates for a juicy privilege escalation:

Component	ClusterRole	ServiceAccounts
OpenTelemetry Collector	`otel-otelcol-k8sobjects`	`opentelemetry-collector-daemonset-collector`, `opentelemetry-collector-deployment-collector`
OpenTelemetry Operator	`otel-operator-resources` / `opentelemetry-operator-manager`	`opentelemetry-operator`
Rook-Ceph	`rook-ceph-global`, `rook-ceph-mgr-cluster`	`rook-ceph-system`, `rook-ceph-mgr`

Note: there are many more. Graham Helton added an “Appendix: Affected Helm Charts” section at the end of his article, referencing AT LEAST 69 affected Helm charts according to him.

The critical case: rook-ceph-system

In the official chart, the rook-ceph-system ServiceAccount combined two particularly dangerous permissions:

nodes/proxy GET - the RCE
secrets GET/LIST/WATCH across the entire cluster

Accessible secrets can include LUKS keys for volume encryption, Ceph admin keyrings, dashboard passwords… this kind of access makes it a prime target for an attacker.

The attack scenario: a compromise of the rook-ceph-operator Pod (via CVE, supply chain, or a malicious image) would allow reading all Ceph secrets, and then executing code in any Pod (including etcd), leading to a full compromise of the cluster and encrypted data.

To manually check if a ServiceAccount is vulnerable:

kubectl auth can-i get nodes --subresource=proxy \
 --as=system:serviceaccount:<namespace>:<serviceaccount>

Examples of fixes to apply

Rook-Ceph: Upstream fix

For Rook-Ceph, the fix came from upstream: PR rook/rook#16979 removed nodes/proxy from ClusterRoles. This fix is included in Rook v1.19.1, so updating the affected clusters was enough.

After updating, checking across all clusters:

kubectl get clusterroles rook-ceph-global -o yaml | grep -A3 nodes/proxy
# -> nothing

OTel / OTel operator

For OpenTelemetry, the situation is potentially more complex. If you are using otel-operator and OtelCollector Custom Resources, you likely have to manage your own RBAC manifests yourself.

Having had to do it myself, it’s quite painful. Depending on your collector type and the receivers you enabled, you need to cross-reference multiple documents on the official OTel and otel-operator websites.

Upstream merged a conditional approach in open-telemetry/opentelemetry-helm-charts#2083 based on the Kubernetes version:

# Upstream approach (opentelemetry-helm-charts#2083)
{{- if semverCompare ">=1.33-0" .Capabilities.KubeVersion.Version }}
 - nodes/pods
{{- else }}
 - nodes/proxy
{{- end }}

Here again, if all your clusters are up to date, you can simply replace nodes/proxy with nodes/pods directly (without the condition).

otel-collector-crb.yaml:

# Before
rules:
 - apiGroups: [""]
 resources:
 - nodes
 - nodes/proxy  # <- RCE risk
 - nodes/spec
 - nodes/stats
 verbs:
 - get

# After
rules:
 - apiGroups: [""]
 resources:
 - nodes
 # nodes/pods replaces nodes/proxy (RCE risk, see [https://grahamhelton.com/blog/nodes-proxy-rce](https://grahamhelton.com/blog/nodes-proxy-rce))
 # Requires K8s >= 1.33 (KEP-2862 fine-grained kubelet authz)
 - nodes/pods
 - nodes/spec
 - nodes/stats
 verbs:
 - get

otel-operator-rbac.yaml:

# Before
 - apiGroups: [""]
 resources:
 - nodes/proxy  # <- RCE risk
 verbs:
 - get

# After
 # nodes/pods replaces nodes/proxy (RCE risk, see [https://grahamhelton.com/blog/nodes-proxy-rce](https://grahamhelton.com/blog/nodes-proxy-rce))
 # Requires K8s >= 1.33 (KEP-2862 fine-grained kubelet authz)
 - apiGroups: [""]
 resources:
 - nodes/pods
 verbs:
 - get

Preventive Measures

KEP-2862: Fine-Grained Kubelet API Authorization

As teased earlier, the real long-term solution is KEP-2862 (Fine-Grained Kubelet API Authorization). It introduces granular subresources (nodes/pods, nodes/metrics, nodes/stats, nodes/log, etc.) allowing precise access without using nodes/proxy.

K8s Version	KEP-2862 Status
1.32	Alpha
1.33	Beta, enabled by default - `nodes/proxy GET` no longer grants access to `/exec`
1.36	GA (locked to enabled)

But this requires going through EVERY chart in use and checking all deployed manifests now and in the future.

CiliumNetworkPolicy: Blocking the Kubelet port

While waiting for the K8s upgrade, or as defense-in-depth, you can block access to port 10250 from the affected pods using NetworkPolicies (or CiliumNetworkPolicy if you use Cilium as your CNI plugin).

Warning: This only applies to components that do not need to access the Kubelet. The OTel collector potentially needs it to gather Kubelet metrics. In that case, you have no choice but to fix the RBAC.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
 name: deny-kubelet-api-access
 namespace: <namespace>
spec:
 endpointSelector:
 matchLabels:
 <app-label>: <value>
 egressDeny:
 - toEntities:
 - host
 - remote-node
 toPorts:
 - ports:
 - port: "10250"
 protocol: TCP

Kyverno: Blocking the creation of new Roles with nodes/proxy

To prevent any regression (remember, we need to protect ourselves in the future), we can add a Kyverno ClusterPolicy that rejects the creation or modification of a ClusterRole or Role containing nodes/proxy.

Luckily, there are ready-to-use examples on the official Kyverno website:

Note : there probably is a hole in the official Kyverno policy, I opened an issue

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: restrict-nodes-proxy
spec:
 validationFailureAction: Audit  # switch to Enforce after validation
 background: true
 rules:
 - name: deny-nodes-proxy-in-clusterroles
 match:
 any:
 - resources:
 kinds:
 - ClusterRole
 - Role
 exclude:
 any:
 - resources:
 names:
 - "system:kubelet-api-admin" # built-in K8s, unmodifiable
 validate:
 message: >
 nodes/proxy grants RCE capability via Kubelet WebSocket exec.
 Use nodes/pods (requires K8s >= 1.33, KEP-2862) instead.
 deny:
 conditions:
 any:
 - key: "nodes/proxy"
 operator: AnyIn
 value: "{{ request.object.rules[].resources[] }}"

Deployment is done in two stages: first in Audit mode to ensure there are no remaining vulnerable manifests (which you should fix before blocking), then in Enforce mode to actually block them.

Monitoring the Audit Log

Even if commands executed via the Kubelet leave no trace, we can monitor SubjectAccessReviews to detect attempts at enumerating nodes/proxy permissions.

The configuration in the Kubernetes audit policy:

# audit-policy.yaml
- level: Request
 verbs: ["create"]
 resources:
 - group: "authorization.k8s.io"
 resources: ["subjectaccessreviews"]

Then a Prometheus/Alertmanager alert on SARs related to nodes/proxy:

# Detect SARs targeting nodes/proxy
increase(
 apiserver_audit_event_total{
 verb="create",
 resource="subjectaccessreviews"
 }[5m]
) > 0

References

Kubernetes RCE via nodes/proxy GET - Graham Helton
Detection Script
KEP-2862 Fine-Grained Kubelet API Authorization
Kubernetes RBAC Good Practices - nodes/proxy
rook/rook#16979 - Fix upstream Rook-Ceph
open-telemetry/opentelemetry-helm-charts#2083 - Fix upstream OTel

Kubernetes UserNamespaces: the overhyped GA feature

Tue, 28 Apr 2026 10:00:00 +0200

The infographic that triggered me

Note: I stumbled upon this GenAI infographic and it was so bad I wrote a post about it. I didn’t generate this thing.

Over the past few days, LinkedIn has been flooded with the same kind of infographic. Kubernetes 1.36 is out, and one of the most talked-about features is the GA release of UserNamespaces.

It’s a topic I’ve been following since 2018 (talk The Route to rootless containers at KubeCon EU 2018), so I’m genuinely glad to see this long journey finally reach the finish line. That said, I’m appalled by the way it’s being marketed on LinkedIn, apparently by people who have no idea how it actually works — and frankly don’t care.

“Kubernetes just made root safer. Just add hostUsers: false to your Pod spec.”

The visual: an all-powerful king “inside the container” and a helpless beggar “outside on the host”. The promise: “No Host Access. No Privilege Escalation. No Lateral Movement. No Node Takeover.”

Catchy.

But presenting it this way is genuinely dangerous, because it obscures entire areas of application and operational security. Selling hostUsers: false as the universal fix for the “root in containers” problem is a dramatic oversimplification that will push teams to ignore the real security priorities.

What UserNamespaces actually do

The threat model: container escape

First, what are we actually talking about? A container escape is when an attacker manages to break out of their container and directly access the host’s kernel or filesystem — completely bypassing the normal isolation mechanisms.

This type of vulnerability is rare, but real-world examples exist:

CVE-2019-5736 (runc): write to /proc/self/exe of the host process from inside the container
CVE-2022-0492 (cgroups v1): escape via unshare in certain configurations
CVE-2024-21626 (runc, “Leaky Vessels”): file descriptor leak to the host working directory

If there’s a vulnerability of this type on your node AND a process is compromised AND it runs as root in the container AND it doesn’t use UserNamespaces, the attacker gets root on the host — game over. Full access to every file on the node, every secret mounted by other pods, ability to install a rootkit or exfiltrate data from all tenants running on that node.

It remains possible, but that’s a lot of “ifs”. Either way, this is exactly the scenario UserNamespaces address. They introduce UID mapping:

UID 0 inside the container is mapped to an unprivileged UID on the host (e.g. 100000, unique per pod)
If an attacker successfully escapes the container via a kernel exploit, they land as nobody on the node — the escape succeeds, but the post-escape impact is dramatically reduced

This is the “Breakouts Lose Impact” scenario from the infographic, and on that point, the infographic is right. That’s the real value of the feature.

Edge case: multi-tenancy even with non-root containers

Even without root containers, UserNamespaces provide something extra in a truly multi-tenant context (multiple customers on the same cluster). Without UserNamespaces, if two pods from different customers both run with runAsUser: 1000, they share the same UID 1000 on the node. If one escapes, the attacker can access files from the other pod with the same owner. UserNamespaces, by assigning a unique UID offset per pod, isolates UIDs between pods even when they use the same value inside the container.

For internal clusters where you control all workloads, this scenario is theoretical. For a multi-tenant SaaS platform or a public build service, it’s a real line of defense.

Technical requirements

There are a few prerequisites, but most up-to-date clusters should qualify.

Linux kernel ≥ 5.19
Compatible runtime (containerd ≥ 1.7, CRI-O ≥ 1.25)
Idmapped mounts support for persistent volumes (XFS, ext4 — not NFS in all cases)
Kubernetes ≥ 1.33 (Beta), ≥ 1.36 (GA)

What the infographic exaggerates (and leaves out)

The infographic is right about one specific thing: UserNamespaces reduces the impact of a successful container escape. That’s real. The problem is it sells the feature as a universal solution to “root in containers” — and that’s just wrong.

1. UID isolation is not application privilege isolation

The infographic promises “No Lateral Movement”. That’s false — completely false.

A root container with hostUsers: false can still read the ServiceAccount Token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. If that token has broad RBAC permissions (which happens — we may cover this in a future post), the attacker can call the API Server, enumerate cluster resources, and move laterally — all without ever touching the host node.

UID mapping protects the host. It does not protect the cluster.

2. A root container is still root inside the container

“Install Anything ✅” — it’s literally written in the infographic, presented as a feature 😖.

In a root container (even with UserNS), an attacker who gains control can:

Install nmap, curl, nc to scan the internal network
Modify application files, binaries, configurations
Read all files mounted as volumes
Persist in the container across restarts if the filesystem is writable

UserNamespaces removes none of these attack vectors. The ability to install software is a fast track to lateral movement.

3. It’s not that easy, especially for storage

Enabling hostUsers: false breaks existing storage in most cases.

Container UID 0 is mapped to UID 100000+ on the host (each container has its own offset). If a persistent volume (NFS, EBS, Ceph RBD) is owned by UID 1000, the root container can’t read or write it. The result: counterintuitive Permission Denied errors that are potentially hard to diagnose, since the application was likely never designed to be root yet have no access to its own files.

The technical solution exists (idmapped mounts), but it requires a recent kernel and a compatible filesystem. See the official idmapped mounts documentation for details.

4. Same story, but for networking

hostUsers: false is incompatible with hostNetwork: true. It’s a corner case, but it catches networking workloads (monitoring agents, CNI plugins, etc.).

Note: that said, running containers with hostNetwork is its own security problem, so…

Honest comparison: UserNS vs the real alternatives

Attack vector	UserNS (root inside)	Non-root (UID 1000)	Distroless / Scratch
Post-escape impact after successful container escape	✅ Nobody on host	⚠️ UID 1000 on host	⚠️ UID 1000 on host
UID isolation between pods (multi-tenant)	✅ Unique offset per pod	❌ Shared UID on node	❌ Shared UID on node
Malware installation inside the container	❌ Trivial	❌ Possible	✅ Near impossible
Write scope in ephemeral container FS	❌ Full filesystem	❌ App directory only	✅ Near impossible
Lateral movement via SA Token	❌ Possible	❌ Possible	⚠️ Potentially difficult
Operational complexity	❌ Sometimes high	✅ Often near zero	✅ Often low
Compatibility with existing storage	❌ Sometimes problematic	✅ Standard	✅ Standard

Reading the table reveals the true nature of UserNamespaces: it excels on exactly two rows:

post-escape impact
UID isolation in multi-tenant environments

On everything else, non-root + distroless does better, or just as well, without the operational complexity. And that “everything else” — write scope in ephemeral FS, malware installation, lateral movement via SA Token — represents the vast majority of real-world attack vectors, far more common than a container escape. We’ll come back to this in the Where to invest your security budget section.

Real use cases

It would be dishonest to dismiss the feature entirely. There are three scenarios where UserNamespaces aren’t a lazy option but a genuine technical necessity (with caveats).

1. Build-as-a-Service (Buildah, rootless Podman)

To build a Docker image, the build engine needs to perform chown, chmod and mknod. These operations require CAP_CHOWN and CAP_FOWNER. Before UserNamespaces, the solution was to run the pod as --privileged — an obvious open door to the host.

With hostUsers: false, the build engine believes it’s root for its own file manipulation, but it can’t touch the host. This is the only case where “root inside” is a technical constraint rather than technical debt.

Note: Kaniko, long the go-to for in-cluster builds, has been archived since October 2025 and no longer receives security updates. Buildah or rootless Podman are the active alternatives.

My take: it can be useful for shared CI/CD platforms (GitLab Runners, Tekton) that refuse privileged pods. But if isolation is critical (public platform, aggressive multi-tenancy), microVMs (Kata Containers, Firecracker) offer far stronger guarantees for an overhead that has become quite manageable.

2. Hostile multi-tenancy (user code platforms)

If your business is running code provided by strangers (PaaS, online code editors, public CI/CD), you know upfront that users will try to escalate their privileges. In this context, UserNS is an extra barrier against kernel 0-days.

My take: honestly, if the environment is truly hostile, UserNS alone isn’t enough. MicroVMs (Kata Containers, Firecracker) provide real hardware isolation and are the right choice here. UserNS can be a complement, not a substitute.

3. Hard-coded legacy (Postfix, Dovecot, BIND)

Some old UNIX daemons start as root to open a privileged port (< 1024) or read sensitive config files, then drop privileges via setuid(). This mechanism fails in a classic non-root container.

UserNamespaces let these processes believe they can make their identity management syscalls, because they are root inside their namespace.

Here’s a concrete example written by a colleague (thanks Louis 😘):

apiVersion: v1
kind: Pod
metadata:
 name: postfix
spec:
 hostUsers: false # UID mapping: root in container → nobody on host
 securityContext:
 runAsNonRoot: false # allowed under PSS Restricted *only* because of hostUsers: false
 fsGroup: 103 # postfix GID
 containers:
 - name: postfix
 image: postfix:latest
 securityContext:
 runAsNonRoot: false # same — cf. https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/#integration-with-pod-security-admission-checks
 allowPrivilegeEscalation: false
 readOnlyRootFilesystem: true
 seccompProfile:
 type: RuntimeDefault
 capabilities:
 drop: ["ALL"] # drop everything first
 add:
 - SETUID  # privilege drop via setuid() done by postfix itself at startup
 - SETGID  # same for groups
 - CHOWN  # chown on mail queues at startup
 - FOWNER  # file operations without being the owner
 - FSETID  # preserve setuid bit after write
 - DAC_OVERRIDE # MANDATORY: root in UserNS is not "real" root —
 # DAC checks are not automatically bypassed

This manifest illustrates several important things.

First, making a legacy application actually secure with UserNS is painful and requires compromises — especially around capabilities. This is a far cry from the “magic feature that secures root apps” the LinkedIn wanna-be influencers imply.

Then there are some interesting surprises. Normally, the Restricted Pod Security Standard forbids runAsNonRoot: false. Kubernetes makes an exception when hostUsers: false is present. This is documented here. Without UserNamespaces, this pod would be rejected by the admission controller.

There’s also the DAC_OVERRIDE capability, which is counterintuitive. Root in a UserNS is not real root from the kernel’s perspective for DAC (Discretionary Access Control) checks. When Postfix runs set-permissions to chown its queues, the kernel still verifies permissions — and denies them if DAC_OVERRIDE isn’t present. This is exactly the kind of operational surprise that stays invisible until the first production deployment.

Worth noting: we were still able to keep readOnlyRootFilesystem: true and allowPrivilegeEscalation: false — legacy doesn’t justify throwing everything overboard.

My take: this is the only use case where UserNS is genuinely acceptable. No untrusted third-party code, no hostile platform — just well-identified legacy with a migration plan. The other two cases are “acceptable under conditions”; the legacy case is the cleanest of the three.

Some counterarguments

I see you coming with objections, so let’s save everyone some time with a quick Q&A:

“It’s defense in depth.” True — but defense in depth assumes the foundational layers are already in place. If you haven’t migrated your images to non-root yet, investing energy in UserNS is putting the cart before the horse. And once you’re non-root, the marginal gain of UserNS is negligible compared to the complexity it introduces.

“We don’t control third-party images.” Somewhat weak, in my opinion: if a proprietary vendor’s black-box image is hardcoded to run as root, there’s a good chance it either genuinely needs it (as is the case for some proprietary security tooling) or it will break with UID mapping (see the storage problem above). UserNS is not a magic wand that makes any third-party image compatible and secure.

“It’s a centralized safeguard against human error.” It’s just as easy to forget hostUsers: false as it is to forget runAsNonRoot: true. The real centralized solution is Pod Security Standards or an Admission Controller (Kyverno, OPA) that outright rejects root pods. Simpler, more reliable, and it doesn’t break storage.

“We need it for SOC2/PCI-DSS/… compliance.” If your compliance requires strict tenant isolation, UserNS will likely be deemed insufficient by your auditors. VMs or microVMs remain the gold standard. Using UserNS for compliance means choosing the most complex tool to maintain for a result that remains debatable.

Where to invest your security budget

Setting aside the marketing, here’s where effort actually pays off — from highest impact to most niche:

Priority 1 — Non-root images + nobody (UID 65534) Move images to non-root, ideally using the nobody user (the least privileged on the system). If an application is compromised under nobody, the attacker can do almost nothing, even on the container filesystem. Combine with readOnlyRootFilesystem: true and capabilities: drop: ["ALL"].

spec:
 securityContext:
 runAsNonRoot: true
 runAsUser: 65534 # nobody
 seccompProfile:
 type: RuntimeDefault
 containers:
 - name: app
 image: my-app:distroless
 securityContext:
 allowPrivilegeEscalation: false
 capabilities:
 drop: ["ALL"]
 readOnlyRootFilesystem: true

Priority 2 — Pod Security Standards (PSS) at Baseline or Restricted Block root and privileges without breaking anything at the infra level. It requires having done Priority 1 first, but it’s free, standard, and applies cluster-wide via a namespace label (with per-namespace overrides when needed). No more risk of forgetting. Already enabled by default on several Kubernetes distributions (Talos being one, but not the only one).

Priority 3 — MicroVMs (Kata Containers, Firecracker) For truly untrusted workloads. Real hardware isolation, overhead now quite reasonable on recent generations.

Priority 4 — UserNamespaces When all else fails. Only for the legitimate cases identified above (builds, legacy, hostile multi-tenancy). This is genuinely the last thing to do.

Conclusion

Kubernetes 1.36 UserNamespaces are the result of a project that officially took five years (KEP-127 dates back to 2021) and has been discussed since nearly the dawn of Kubernetes. For shared build platforms and multi-tenant SaaS running user-provided code, it’s a potentially useful building block — particularly to prevent one customer’s app from reading another’s in the event of a container escape without privilege escalation.

For everything else — that is to say, 99% of production clusters — that’s not where container security starts. And that’s precisely the problem with this kind of infographic.

LinkedIn infographics selling effortless security are dangerous: “keep your 800MB root image full of tools, just add hostUsers: false, and you’re protected.” That’s exactly the wrong approach. Real container security is built in the Dockerfile, not in the PodSpec.

If you’re enabling UserNamespaces to secure an application whose source code you own, you’ve probably missed a step in your secure software development lifecycle.

References

KEP-127: Support for User Namespaces
Kubernetes docs — User Namespaces
Pod Security Standards
CVE-2019-5736 — runc: write to host’s /proc/self/exe from inside the container
CVE-2022-0492 — cgroups v1: escape via unshare, UserNS helps but runAsNonRoot does too
CVE-2024-21626 — runc “Leaky Vessels”: file descriptor leak to host working directory
Distroless containers — GoogleContainerTools

GenAI and software development, episode 2: kubectl-debug-pvc, from idea to krew in 2x30 minutes

Tue, 17 Mar 2026 18:00:00 +0100

Previously, on “GenAI and dev”

In my previous article, I talked about my experience with PodSweeper, a project developed with OpenCode and Claude Opus. The outcome was mixed: impressive raw speed, but race conditions, lax error handling, and a constant need for human supervision (among other disappointments).

Today, I’m talking about a second project, much simpler, born from a real production need. And the takeaway is quite different.

The incident that started it all

You may know this situation: a pod in production, running fine, using a PVC in ReadWriteOnce. You need to look at the contents of that volume. Production best practice: the pod in question has no shell (we reduce the attack surface). No /bin/sh, no /bin/bash, nothing.

No big deal, we have kubectl debug for that, right?

Ok, let’s create an ephemeral container in the pod and… oh wait. kubectl debug doesn’t allow mounting the pod’s volumes in the ephemeral container. It simply doesn’t expose the volumeMounts option for ephemeral containers.

We could kill the pod, which would allow mounting the RWO PVC in another pod with a shell, but we don’t want to — it’s production.

My colleague Maxime (him again!) suggested a workaround. The manual procedure is:

Create an ephemeral container on the pod with kubectl debug
Manually build a JSON patch to add volumeMounts to the ephemeral container we just created
In another terminal, connect directly to the kube API with kubectl proxy
Apply the patch via a curl to the Kubernetes API
Wait for the container to be ready, attach to the container
Wonder if there’s an easier career than patching containers with curl

For those who think this is doable, “check out this patch” as the kids say:

curl http://localhost:8001/api/v1/namespaces/<namespace>/pods/<pod>/ephemeralcontainers \
 -X PATCH \
 -H 'Content-Type: application/strategic-merge-patch+json' \
 -d '{
 "spec": {
 "ephemeralContainers": [
 {
 "name": "debugger",
 "image": "ubuntu",
 "command": ["/bin/sh"],
 "targetContainerName": "<target-container>",
 "stdin": true,
 "tty": true,
 "volumeMounts": [
 {
 "name": "<volume-name>",
 "mountPath": "/debug/<volume-name>"
 }
 ]
 }
 ]
 }
 }'

If you think this is error prone, you’re right. And if you think it’s painful to do under pressure during a production incident, you’re even more right.

Side note

The more seasoned among you with recent Kubernetes versions may have heard about KEP-2590, which introduces a (relatively) new --subresource flag for kubectl. Indeed, ephemeral containers created by kubectl debug are not resources (a pod, a deployment, …) but subresources.

Until version 1.33, it was literally impossible to perform operations on subresources with kubectl, only resources.

Kubernetes v1.33: Octarine brings support for the –subresource flag, but only for 3 subresources for now: status, scale and resize

No solution on that front either…

The meeting prompt

I had this idea in mind since the incident. Monday morning, at the start of a meeting (while people were still making jokes), I launched OpenCode with Claude Opus and typed a prompt describing:

The problem: kubectl debug doesn’t mount PVC volumes if they’re RWO
The manual procedure I was doing by hand (the painful steps described above)
What I wanted: a tool that automates all of that

OpenCode + Opus asked me 2 questions (and I added one instruction):

“Which language?” → Go (I insist)
“CLI only or TUI?” → Both (non-interactive mode for scripting, TUI for daily use)
And I asked it to always run the golangci-lint linter every time it considers being done (trauma from my previous test with OpenCode)

It went on its own with the idea of a TUI using Bubble Tea. Then I stopped watching and followed my meeting.

~30 minutes later, end of the meeting, I looked at the result. The POC was functional.

What Opus produced in 30 minutes

Without any intervention on my part, the LLM scaffolded a complete Go project:

Clean structure: cmd/ for the Cobra CLI, pkg/k8s/ for all Kubernetes interaction, pkg/tui/ for the Bubble Tea TUI
The core of the matter: a direct call to the Kubernetes API to patch the ephemeralcontainers subresource of the pod with a strategic merge patch including the volumeMounts
Non-interactive mode: -n namespace -p pod -v volume:/mount/path, ready for scripting
Complete TUI: vim-style navigation (j/k), fuzzy filtering (/), multi-selection of volumes, loading spinner…
Makefile with a whole bunch of targets (vet, lint, test, build, install)

What I asked it to add:

Smart discovery: instead of scanning all pods in all namespaces (slow on one of my large clusters), the tool first lists PVCs cluster-wide (a single API call) to identify relevant namespaces, THEN the pods in the selected namespace. This drastically reduces the number of calls to the kube API server and the TUI only shows namespaces and pods that have PVC volumes
SecurityContext inheritance: the ephemeral container copies the securityContext from the target container, allowing it to pass PodSecurity policies (restricted, baseline…). This was a case I hadn’t anticipated in my initial prompt and it immediately failed on a properly configured cluster.

Without these small improvements the tool was already usable (except when you have SecurityContext configured), but it’s much more comfortable in practice (because yes, I do use it).

The code for the core mechanism (the API patch) is a few hundred lines of clean Go. It retrieves the pod, builds the JSON patch with the ephemeral container and its volumeMounts, applies it via Patch() on the subresource, waits for the container to be Running, then launches kubectl attach. Nothing fancy, it’s just what’s needed, done right the first time.

 ephemeralContainer := corev1.EphemeralContainer{
 EphemeralContainerCommon: corev1.EphemeralContainerCommon{
 Name: containerName,
 Image: opts.Image,
 Command: []string{"/bin/sh"},
 Stdin: true,
 TTY: true,
 VolumeMounts: mounts,
 SecurityContext: targetSecurityContext,
 },
 TargetContainerName: targetContainer,
 }
...
 _, err = c.Clientset.CoreV1().Pods(opts.Namespace).Patch(
 ctx,
 opts.PodName,
 types.StrategicMergePatchType,
 patchBytes,
 metav1.PatchOptions{},
 "ephemeralcontainers",
 )

The following iterations (~30 minutes)

Once the POC was validated, I chained a few targeted prompts. I asked it what could be improved. It suggested (and implemented):

A few minor code fixes it had gotten wrong (but non-blocking)
Warnings: if the inherited SecurityContext has readOnlyRootFilesystem, runAsNonRoot, or other security measures, the tool warns the user before attach
Added a complete CI based on goreleaser and GitHub Actions
Added documentation

The cherry on top: publishing on Krew

Once I had a clean version, I asked Opus how to make the plugin easier to install. It suggested brew, or Krew, the kubectl plugin manager.

I asked if the plugin acceptance process was complex. It said no, I said “I dare you!”.

It completed the entire process:

Created a fork of krew-index
Wrote the Krew manifest (the .yaml file describing the plugin)
Took into account all the best practices requested by krew-index maintainers (download URL format, short descriptions, SHA256 checksums, etc.)
Prepared the PR, directly on krew-index

I just watched. The PR was merged 12 hours later by the maintainers.

Result: kubectl krew install debug-pvc works.

Minor failure — delegating the demo too

Emboldened by this success with the krew-index PR, I wondered if we could make a visual demo (a GIF to add to the project’s README.md showing how it works) with the LLM.

I asked if it could do it with asciinema and the LLM answered “yes” (yes, I chat with the LLM like I do with my colleagues). Deal, we tried.

The result was almost good, I could have settled for it if I weren’t such a nitpicker: it was a bit slow. I felt the LLM’s responsiveness in its actions was uneven, which made the viewing experience a bit unpleasant. I could have iterated until I got something correct, but I ultimately recorded a video myself, converted to GIF with ffmpeg. It was smoother and faster than iterating.

The difference with PodSweeper

The difference in experience between this project and PodSweeper is striking. Where PodSweeper was a constant fight (race conditions, LLM amnesia, out-of-spec features), kubectl-debug-pvc went smoothly without any notable hiccups.

Why? I think it comes down to the nature of the project:

One single thing to do, clearly defined: patch a Kubernetes subresource with volumeMounts
No concurrent logic: we make an API request, we wait, we attach
A well-documented domain: the Kubernetes API, Cobra, Bubble Tea. The LLM knows all of this by heart
No complex shared state: no goroutines stepping on each other, no graceful shutdown to manage, no multiple microservices
Limited scope: the entire project fits in about fifteen files, CI and documentation included

This is probably the ideal playground for an LLM. A well-defined problem, a well-charted technical domain, a linear solution.

What do I think about it?

Objectively, this kind of “simple” project (one thing to do, easy to understand) works really well with OpenCode + Opus. I think it’s because of this kind of small project that the hype is so strong around agentic development. You have an idea you were too lazy to dev, you test it, it works. “WOW”.

But I don’t want to downplay the work done either. The fact that a functional, clean tool, published on Krew and usable by anyone could emerge from a prompt launched at the beginning of a meeting is still pretty wild.

A bit scary too, thinking that an insane number of micro tools are going to appear in the coming months, with probably uneven quality.

The project

The code is available here:

https://github.com/zwindler/kubectl-debug-pvc

Installation:

# Via Krew (recommended)
kubectl krew install debug-pvc

# Interactive usage (TUI)
kubectl debug-pvc

# Non-interactive usage
kubectl debug-pvc -n my-namespace -p my-pod-0 -v data:/debug/data

If you’ve ever struggled to inspect a PVC on a pod without a shell, give it a try. And if you find bugs, you can blame the LLM 😄.

Flannel and NetworkPolicies: how to add support with Cilium in CNI chaining mode

Sun, 15 Mar 2026 18:00:00 +0200

Flannel, flannel, flannel…

Flannel is a simple and popular CNI 😢.

It’s the default CNI in k3s, the one half the kubeadm tutorials use, and you’ll find it in quite a few managed offerings too.

OK, it’s simple, it routes packets between pods, it supports VXLAN and WireGuard, it takes 2 minutes to set up. What more could you ask for?

Well, exactly. There’s one thing flannel does not do: NetworkPolicies. (And that’s a big deal).

Flannel is focused on networking. For network policy, other projects such as Calico can be used.

[Edit] Contrary to what I wrote here, flannel actually does support NetworkPolicies and has for at least 2 years, via the reference implementation kube-network-policies from the Kubernetes project. The option isn’t prominently featured in the README and is probably not more recommended for production, but it does exist. See the official documentation.

And the trap is that if you haven’t read that one little line in the README.md, nothing tells you explicitly.

You can perfectly well create NetworkPolicy objects in your cluster, kubectl apply won’t complain, kubectl get netpol will happily list them. Except… they’re not enforced. Traffic still flows. Your deny-all doesn’t deny anything at all.

Let’s verify to be sure

Before solving anything, let’s verify the problem exists. Starting from the prerequisite that we have a Kubernetes cluster with flannel as the CNI and everything is working, we’ll deploy two pods in two different namespaces: a client (curl) and a server (nginx).

Let’s check that the client can reach the server:

kubectl exec -n netpol-test-a client -- \
 curl -s --max-time 5 http://server.netpol-test-b.svc.cluster.local

We get the nginx welcome page. So far, so normal. Now, let’s apply a deny-all NetworkPolicy on the server’s namespace:

# 01-deny-all-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: deny-all-ingress
 namespace: netpol-test-b
spec:
 podSelector: {}
 policyTypes:
 - Ingress

kubectl apply -f 01-deny-all-ingress.yaml

And let’s test again:

kubectl exec -n netpol-test-a client -- \
 curl -s --max-time 5 http://server.netpol-test-b.svc.cluster.local

Result: the nginx page still shows up. The NetworkPolicy was created (kubectl get netpol -n netpol-test-b shows it), but it’s not enforced. Traffic flows as if nothing happened.

This is the expected behavior with flannel (by default). Flannel only does L3 routing (VXLAN or WireGuard overlay). It doesn’t implement a NetworkPolicy controller. The objects exist in etcd, but nobody translates them into filtering rules.

Let’s clean up the policy before moving on:

kubectl delete -f 01-deny-all-ingress.yaml

Alternatives for adding support

For a long time I thought this was just the way it was. I still think it’s not a good choice for any production.

BUT recently, I discovered that it was possible to chain CNIs within the same cluster, and thus have one CNI handling most tasks (here Flannel) and another one taking care of other tasks, such as enforcing NetworkPolicies.

This is actually the principle behind Canal, which I knew by name but had never explored. In fact, it’s nothing more than a manifest that deploys Flannel as the main CNI with Calico for NetworkPolicy enforcement!

Canal was the name of Tigera and CoreOS’s project to integrate Calico and flannel.

Note: the GitHub project was archived in October 2025 but in theory it should still work, if you can find the correct documentation (I couldn’t, the links are broken, but I didn’t really look that hard either).

So you get the idea: we’re going to add a component that will watch NetworkPolicy objects and translate them into effective filtering rules (iptables, eBPF, nftables…), without touching the existing flannel setup. This is called “CNI chaining” or “policy-only” mode.

There are several solutions:

Calico (Canal); Historically, the flannel + Calico combination is called “Canal”, which I just mentioned. But the official Canal manifest bundles its own flannel in the same DaemonSet as calico-node. If your flannel is already installed and managed (by you, by an operator, by a provider…), you probably don’t want to replace it. And the Tigera operator (the “official” Helm method) doesn’t support policy-only deployment on an existing flannel either. In short, it’s doable but requires some effort. Can’t be bothered.

kube-router; kube-router can run in firewall-only mode (--run-firewall=true) and only needs iptables/ipset. This is actually what k3s uses by default for NetworkPolicies. It’s the lightest solution (supposedly ~50 MB of RAM per node). Make sure your kernel has the ip_set module, otherwise it won’t work.

Cilium in CNI chaining mode; This is the solution I chose and that we’ll detail here. Cilium attaches to the veth interfaces created by flannel and adds its eBPF programs for policy enforcement. No dependency on iptables or ipset, and as a bonus we get Hubble for network observability.

Installing Cilium in CNI chaining mode

Prerequisites

A working Kubernetes cluster with flannel
helm v3+
A kernel >= 4.19 (ideally >= 5.10 for all eBPF features)

Retrieve flannel’s CNI configuration

Cilium in chaining mode needs to know the existing CNI configuration. Let’s retrieve it from a node:

kubectl debug node/$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') \
 -it --image=busybox -- cat /host/etc/cni/net.d/10-flannel.conflist

On my cluster, this gives:

{
 "name": "cbr0",
 "cniVersion": "0.3.1",
 "plugins": [
 {
 "type": "flannel",
 "delegate": {
 "hairpinMode": true,
 "isDefaultGateway": true
 }
 },
 {
 "type": "portmap",
 "capabilities": {
 "portMappings": true
 }
 }
 ]
}

Note the name field (here cbr0). We’ll need it.

Create the chaining ConfigMap

We’ll create a ConfigMap that takes the flannel config and adds the cilium-cni plugin in chaining mode:

apiVersion: v1
kind: ConfigMap
metadata:
 name: cni-configuration
 namespace: kube-system
data:
 cni-config: |-
 {
 "name": "cbr0",
 "cniVersion": "0.3.1",
 "plugins": [
 {
 "type": "flannel",
 "delegate": {
 "hairpinMode": true,
 "isDefaultGateway": true
 }
 },
 {
 "type": "portmap",
 "capabilities": {
 "portMappings": true
 }
 },
 {
 "type": "cilium-cni",
 "chaining-mode": "generic-veth"
 }
 ]
 }

Warning: the name field must match your flannel conflist. If yours is called cni0 or something else, adjust accordingly.

Before applying, also verify that your CNI uses veth interfaces (this is the default with flannel, but better safe than sorry). From a node:

ip -d link | grep veth

You should see veth-type interfaces corresponding to your pods, for example:

103: lxcb3901b7f9c02@if102: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
veth addrgenmode eui64 numtxqueues 1 numrxqueues 1

If that’s the case, Cilium’s generic-veth mode will work. Let’s apply:

kubectl apply -f cilium-cni-configmap.yaml

Install Cilium via Helm

helm repo add cilium https://helm.cilium.io/
helm repo update

Here are the values for chaining mode:

# cilium-values.yaml
cni:
 chainingMode: generic-veth
 customConf: true
 configMap: cni-configuration
 install: true

routingMode: native
enableIPv4Masquerade: false
enableIPv6Masquerade: false

hubble:
 enabled: true
 relay:
 enabled: true
 ui:
 enabled: true

The important points:

cni.chainingMode: generic-veth, this is chaining mode, Cilium attaches to existing veth interfaces
cni.customConf: true + cni.configMap, we provide our own CNI config
routingMode: native, flannel handles routing, not Cilium
enableIPv4Masquerade: false, flannel handles masquerading
hubble.enabled: true, network observability, the big bonus of Cilium

helm install cilium cilium/cilium --version 1.19.1 \
 --namespace kube-system \
 -f cilium-values.yaml

Wait for everything to be ready:

kubectl rollout status daemonset/cilium -n kube-system --timeout=120s

Verification

kubectl exec -n kube-system ds/cilium -- cilium status

What we’re looking for:

Kubernetes: Ok 1.35 (v1.35.0) [linux/amd64]
CNI Chaining: generic-veth
Cilium: Ok 1.19.1
Hubble: Ok

The CNI Chaining: generic-veth line confirms that Cilium is running in chaining mode and isn’t replacing flannel.

Important: pods that existed before Cilium was installed are not automatically managed by Cilium. You need to restart them so that Cilium can attach its eBPF programs. Remember to kubectl rollout restart your test workloads (or recreate them).

Testing NetworkPolicies

This is the moment of truth. Let’s re-apply our deny-all:

kubectl apply -f 01-deny-all-ingress.yaml

kubectl exec -n netpol-test-a client -- \
 curl -s --max-time 5 http://server.netpol-test-b.svc.cluster.local

Result: timeout! This time, the NetworkPolicy is properly enforced. Traffic is blocked.

I followed up with the other classic NetworkPolicy scenarios, and everything works:

Selective ingress allow by namespace; by adding a policy that allows traffic from netpol-test-a only, curl works from that namespace but remains blocked from default. Namespace isolation works.
Deny-all egress; by blocking all outgoing traffic from the client, even DNS resolution is blocked (immediate timeout).
Selective egress allow; by allowing only DNS (port 53) and the server (port 80 in the netpol-test-b namespace), curl to the server works but curl http://example.com remains blocked.

In short, ingress, egress, selective by namespace, everything works as expected.

Bonus: Hubble, network observability

This is for me the real advantage of Cilium over the alternatives. Hubble lets you see network flows and policy verdicts in real time:

kubectl exec -n kube-system ds/cilium -- \
 hubble observe --namespace netpol-test-b --last 5

Mar 15 13:20:42.287: netpol-test-a/client:40066 (ID:9745) ->
netpol-test-b/server:80 (ID:22271)
policy-verdict:none ALLOWED (TCP Flags: SYN)

You can see the source pod, destination pod, port, Cilium identity, and policy verdict. When you’re debugging a NetworkPolicy that isn’t behaving as expected, this is incredibly useful.

How much does it cost in resources?

On my cluster (3 nodes), here’s what Cilium consumes right after installation:

Component	Per node	RAM
cilium agent	yes (DaemonSet)	~160 MB
cilium-envoy	yes (DaemonSet)	~22 MB
cilium-operator	no (2 replicas)	~42 MB
hubble-relay	no (1 replica)	~16 MB
hubble-ui	no (1 replica)	~21 MB

That’s roughly 180 MB per node for the agent + envoy. I don’t have a point of comparison with Calico or kube-router, but it seems acceptable to me, and being able to have full visibility into all network flows with Hubble more than justifies the overhead (in my opinion).

Conclusion

If you have no choice and have to work with flannel, and you want to plug the gaping security hole that is the lack of NetworkPolicy enforcement, know that it is possible to chain another CNI to handle it.

Short of having it as CNI for everything, Cilium in CNI chaining mode (generic-veth) is a pretty nice solution to fill this gap. It doesn’t touch the existing flannel setup, it grafts onto it. And as a bonus, you get Hubble for network observability, which is genuinely valuable.

GenAI and software development: lessons learned with PodSweeper

Sun, 08 Mar 2026 18:00:00 +0200

GenAI, is it fantastic?

Quite a few people have shared their takes on GenAI for development in a very short time, so I realize it’s more than time I post this draft I started over two weeks ago 🙃.

For work, I use AI assistants more and more to help me with my daily tasks. In 2024, it was mainly for automating tedious tasks (scripting stuff, making a painful list of repetitive tasks without coding it properly). Throughout 2025, I tested it several times for Ops work, and each time the results were mediocre, both for designing coherent, reliable and efficient infrastructure and for incident resolution assistance (see my article on the topic).

In 2026, on that last point, I feel we still haven’t progressed, though I’ll admit there are specialized tools I haven’t tested enough yet, open source or not. I can mention k8sgpt and HolmesGPT for open source, and Anyshift with its SRE agent for root cause detection and incident resolution.

Note: fun fact, in his post “My position on Generative Artificial Intelligence” posted just before mine, Alex cites a long list of professions that will benefit from GenAI, and ops folks get nothing more than yet another argument to switch to Kubernetes 😭.

On the other hand, I’ve started using GenAI to build several web projects from scratch (vibe coding?), which I’ve already told you about:

Converting a blog from Bloggrify to Hugo (french only)
Creating a “pretty” website (way beyond my skills) to host the research I’ve done on the different ways to deploy Kubernetes
Improving my website’s security. (french only) by automating modifications to the Hugo theme I use, to get the best score on Mozilla HTTP Observatory

In all 3 cases, the result is there. Or rather, it seems to be for the layman that I am.

I’ve never hidden that I have zero front-end skills, and maybe for a domain expert, what I did with AI is horrible (or not?). In any case, it can’t be worse than the UIs I made without it. Small example 😄:

or rather: GêneAI (sorry that a french play on word I can’t translate)?

This is the kind of feedback you see from people who are actually experts in a domain when they watch novices get excited. We have good examples with AI-generated videos: if you’re not paying attention, you think it’s incredible. But anyone with a slightly critical eye immediately sees BIG consistency problems. Same goes for image generation, or music.

It’s also true for code, and we have great examples of vibe coded projects that are horrific spaghetti messes and cybersecurity sieves. In short, you get it: even though I’m a daily user, I’m not “amazified” (as my kids would say) by LLMs, even the best ones.

Yet, I have several friends who swear by them, including people far more brilliant/intelligent (call it what you will) than me. So I thought “OK, maybe I’m doing it wrong. Let’s try with the best there is”: a “Claude Code”-type IDE and an Opus model.

The editor

After a quick market survey, you realize the options are plentiful: Claude Code, OpenCode, Amp Code, …

The problem with Claude Code is the entry price. I’m not yet ready to spend 100 or 200€ per month for the use I have today. I don’t know if the 20€ plan would be enough. Beyond the rare personal side projects I mentioned above, I don’t code much. Most of my geeky activities are infrastructure: installing Kubernetes clusters and virtualization OSes. Stuff that’s hard to automate with an LLM.

Pierre (mostly) recommended Amp Code for personal use, mainly because it has a free tier with a certain number of tokens and if you’re not too greedy, it’s quite effective. I preferred to do things my own way and test OpenCode instead, which has the advantage of being more flexible with LLM providers and token consumption. It’s even possible to use local models (free, therefore) or models included for free (at least for now) such as GLM 5 Free, which performs quite well among dev models (most importantly, it’s free…).

OK, we have the editor. Now, the code?

To get a sense of the relevance of code generated by my favorite LLM (Sonnet and Opus, for a while now), I need a language and a use case that I master, so I can tell it “no, that’s absolute nonsense!?”.

A use case I master… as you might guess, there will inevitably be some Kubernetes in there.

A language I master: I learned Go in 2022 thanks to a colleague at Deezer (thanks again Martial!) and I’ve published a few tools and made minor contributions. I wrote a large part of the GroROTI tool and also started (but never finished) developing an RPG in Go, heavily inspired by Castle of the Winds, a game from my childhood: gocastle. And I have professional-level proficiency (even if it’s not the core of my job) in my day-to-day work.

After the context, the project

OK, we have the technical context: Kube + Go. Now we need the idea to implement. The project needs to be large enough for the test to be meaningful, fun enough that I want to spend personal time on it, but still somewhat useful so I’d want to talk about it and show progress. And above all, a project whose business logic stays within my reach: to honestly evaluate the quality of AI-generated code, I need to be able to read and critique it.

And there, in my overflowing drawer of dumb ideas, I remembered this “pitch” from 2023-2024, which I’d left to rot for lack of time:

PodSweeper: the most complex, over-engineered and chaotic way to play Minesweeper.

Don’t leave, you’ll see it’s fun. Really!

Instead of a visual grid where you click on cells hoping not to hit a “mine”, we have a virtual “grid” where each cell is a Pod and the click is a kubectl delete.

Yes, it’s dumb. I like it a lot :).

Beyond the deliberate trolling (over-engineering from hell) of this game, there’s actually an educational objective behind it.

Really, there is.

I designed the game as an introduction to Kubernetes security, with difficulty levels to unlock, CTF-style.

At the beginning, you can do everything. You can of course play normally (delete a Pod, see if there are mines nearby) to get the hang of it. You can also automate actions (a script that “clicks a cell” at random, retrieves proximity hints for mines, clicks somewhere safe, …).

But you can also cheat.

And that’s the whole point of the game. In the first difficulty levels, the game’s Kubernetes manifests are deliberately vulnerable. So you can win effortlessly, if you know where to look.

However, levels quickly progress and restrictions come with them. Pretty soon, you reach production-grade best practices that prevent any “cheating”. And if you find unintended vulnerabilities in the game code itself… well, that’s bonus 😈.

The initial process

It’s probably a mistake, but I did the entire ideation phase with Gemini before launching OpenCode. The experience would probably have been better (at least more representative) if I’d started directly with OpenCode.

I dumped everything I had in mind, along with a big shapeless block of dozens of lines from my notes from when I had the idea in 2023, and asked it to produce a SPECIFICATIONS.md file with all the important information, put back in order.

Once the specs were written, I asked it to detail the different levels and the difficulties we would progressively add, in GAMEPLAY.md.

Finally, I asked it to break down the project into “issues” by priority, so I could get an MVP quickly: ISSUES_PRIORITY.md. My idea was to give OpenCode the very detailed, finely sliced battle plan and let it run.

First disappointment: Gemini 3 Pro is pretty bad at this. The tasks were credible(-ish) but ordered randomly (task dependencies not respected). So I launched OpenCode for the first time in my repo and told it “here’s the context, here’s what we came up with for tasks. What do you think?”

The good (open)code…

Among the pretty nice things: OpenCode + Claude Opus immediately saw that the plan created by Gemini was flawed, and asked me questions and proposed a breakdown that was still imperfect but already more coherent. This is probably where the main value of these tools lies. They embed knowledge to guide software development and above all (ABOVE ALL) bring structure.

Pretty quickly, we agree on a plan I like. OpenCode updates the documents and starts developing (scaffolding, creating pipelines, first empty binaries and Docker images). We have a project ready to develop in a snap, which is quite exciting, at first glance.

LLM assistants are very eager to jump into code, this remains true with OpenCode+Opus. Even though we hadn’t finished discussing the plan and possible options, it was spontaneously proposing to move to code. That said, it was the same (or worse) with Gemini (to whom I had specifically said we wouldn’t be writing any code at all).

Very quickly, file creations pile up. I struggle to keep up and at each pause (when the LLM has finished a task and asks if it can move to the next), I spend long minutes reading what the LLM produced. It’s both exhilarating and exhausting (re-reading all that is hard on my rusty brain).

The raw speed is impressive. In 3 sessions of 1 to 2 hours, I have a working MVP. I understand why people regularly talk about 10x engineering when discussing code generation with recent LLMs. If we’re not talking about raw code generation (which doesn’t really make sense), roughly speaking, functional code is generated 3x to 10x faster than what I could have done alone. Features that end to end would have taken me hours to implement (grid generation, reveal logic, game state management) land in minutes. You’ve read it elsewhere, I’m saying the same thing — the bottleneck is no longer the code I type: it’s the code I have to review.

From what I’ve read, the code is correct, particularly in the first iterations, a bit less so after a while. But when I say “correct”, I might be underselling it: the code produced is idiomatic Go. Good package structure, naming conventions respected, error handling in Go style (when it’s there…), appropriate use of interfaces. This isn’t just code that compiles: it’s code a Go reviewer wouldn’t reject outright. We’re aiming for an MVP so the business logic remains very simple, but the foundation is clean.

Everything is tested, very quickly the project has perfect coverage and 100+ tests while we’re not even doing anything yet. If this were human-written code, I wouldn’t know if that’s a good thing. We’re absolutely not doing TDD, a lot of code tests useless things. In the case of LLM-generated code, it’s probably for the best, because the LLM tends to introduce regressions fairly regularly (we’ll come back to that). So it’s interesting here, because:

it costs almost nothing in dev time (it’s so fast to generate)
it allows the LLM to realize its code introduced a regression, and fix it on its own, without waiting.

… and the bad (open)code

On the tool and model side first, a few irritants:

By default, OpenCode doesn’t encourage the LLM to run golangci-lint on every commit. You fix this with a pre-commit hook and/or the famous AGENTS.md / skills, but it SHOULD have been part of the basic kit for a golang project (tests are there…).
The LLM (Claude Opus on OpenCode, but this is a common bias even outside of it) loves software versions that existed at the time of its training. You have to constantly remind it that the versions it suggests are outdated. This is true for everything: Go code, dependencies, GitHub Actions versions.
You very often hit the 100k token limit. You waste time “compacting” and lose precision. Typically: the LLM asks if it can git commit changes, the context compacts before I answer, and it commits without my permission while re-unreeling the context. For code this low-stakes, it’s fine. But in prod, that would be a real problem.
The LLM is often amnesiac: it forgets it has access to a kind cluster for real testing (even though it did it just before the previous compaction, for example).

On the code quality side, it’s more concerning:

Lax error handling. Some errors that should have returned a FATAL (controller that can’t initialize properly!) are simply logged as ERROR. You end up shipping versions that fail silently. Again, this can probably be fine-tuned with an AGENTS.md.
Race conditions. I ran into race conditions fairly quickly, pretty silly ones. In my case, pods stuck in terminating impacted the next game (poor graceful shutdown handling). Opus went into a loop without understanding the root cause. I had to stop it and specify that it should never start a new game without making sure the previous one was finished.
Code generation outside the lines. Sometimes, without warning, the LLM adds a feature that wasn’t requested, is useless, or even contradicts the previously written SPECIFICATION. You have to slap it back into line.

Once put back on track, the LLM rolls again, but these episodes clearly show that for concurrent code or outside of ultra-strict user stories (which goes beyond OpenCode’s default workflow), human supervision remains essential.

People will tell me I’m a beginner with these tools, and I could have avoided some pitfalls by better configuring my environment (AGENTS.md, pre-commit hooks, etc.). That’s true.

But that’s exactly where the shoe pinches: one of the hyped promises of GenAI applied to development is to enable non-developers (or beginners) to produce quality software at high speed, so they can ship complete software. If getting the most out of it requires being an experienced developer who knows how to configure a complex environment (with the specifics of these AI tools), anticipate race conditions and review concurrent code… we haven’t democratized development, we’ve just given another tool to people who already knew how to code. My profile (ops who codes, not a pure dev) was precisely the target audience of this promise. And today, the math doesn’t check out.

Where am I at?

I’ve been talking about PodSweeper for a while now. But where is this project?

After 3 sessions of 1 to 2 hours max, not counting ideation, I have a working MVP. As I said earlier, the productivity gain is undeniable, for someone who isn’t a “developer” by trade.

The code is probably “overall” better quality than the Go code I could have written, but the LLM lets gaping holes through in the business logic (simply because it doesn’t think, while I do. Well, normally). Hard on trust… You really have to remain very wary of everything it produces.

The code is available at:

https://github.com/zwindler/PodSweeper/tree/main

You can go check out the code to form your own opinion. If you’re not afraid of spoilers, you can read the SPECIFICATION.md (spoilers), or even GAMEPLAY.md (I detail all the levels there so it’s mega spoilers).

You can (normally) play the first difficulty levels. It’s still fairly basic, if you’ve already used kubectl you should find the solution very quickly. My goal is to add levels over time, I’ve already imagined 10, some quite devious 😈 and others might come later.

The code is under MPL v2.0 because it’s a copyleft license I like, since it’s both not very restrictive, OSI compliant and still requires contributing changes back if you make them.

There you go :) if you also find it fun, feel free to test and/or give me feedback.

$ kubectl apply -k https://github.com/zwindler/podsweeper//deploy/base?ref=v0.1.4
namespace/podsweeper-game created
...

$ kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=podsweeper \
 -n podsweeper-game --timeout=60s
pod/gamemaster-54f4dddcc4-6m88z condition met

Have fun!

Kyverno killed my API Server. Again.

Thu, 26 Feb 2026 08:00:00 +0200

The revenge strikes back

You may have read my previous article about etcd 3 years ago. If you remember correctly, the crash was etcd, but the real culprit was Kyverno. I love Kyverno. It’s really a piece of software I’m very fond of. Mainly because it’s incredibly powerful. I even wrote an introductory article and a second one that goes deeper on the topic (both in french, though).

But the sheer number of incidents and weird side effects it causes. Mamamia… This isn’t the first incident of the year I’ve had with Kyverno (yes, we’re in February) but since this one is entertaining, I’m sharing it with you.

During a routine maintenance operation to upgrade a Kubernetes cluster to version 1.34 (from 1.32), we ended up facing the dreaded scenario for any kube admin: a completely unreachable API Server after restarting the Control Plane nodes.

What initially looked like a typical network error turned out to be a subtle deadlock between new native Kubernetes networking features and our dear Kyverno 😘.

Spoiler: it wasn’t a network issue. It’s never a network issue. Well, sometimes it is. But not this time.

The upgrade that starts well

Alright, a Kubernetes upgrade has become pretty routine at this point. We do it regularly, we have our procedures, we’re pros (I swear). We jump from 1.32 to 1.34 in a single commit, skipping the hop through 1.33.

YOLO.

In the technical context I’m talking about, everything is managed as code. From machine provisioning all the way to Talos deployment, including MachineConfigs (the CustomResources to modify… well, the machine).

For more details, see the Talos documentation on Machine Configs.

The first cluster we test has only one control plane (don’t ask me why, it probably wouldn’t have changed anything). Talos restarts the API Server with the new version and then… nothing.

The “weird” API Server logs (technical term) speak for themselves:

I0224 15:32:50.979280 1 default_servicecidr_controller.go:166] Creating default ServiceCIDR with CIDRs: [10.1.0.0/20]
W0224 15:32:50.984784 1 dispatcher.go:225] rejected by webhook "validate.kyverno.svc-fail":
admission webhook "validate.kyverno.svc-fail" denied the request:
Get "https://10.1.0.1:443/api": dial tcp 10.1.0.1:443: connect: operation not permitted
I0224 15:32:50.985342 1 event.go:389] "Event occurred" kind="ServiceCIDR"
apiVersion="networking.k8s.io/v1" type="Warning"
reason="KubernetesDefaultServiceCIDRError"
message="The default ServiceCIDR can not be created"

😬😬😬

The root cause: a magnificent vicious circle

After investigation, we discovered that the incident was the result of a collision between a Kubernetes core evolution and our Kyverno configuration. A textbook deadlock case.

Let’s break down the mechanism:

The new ServiceCIDR Kind

In recent versions (v1.33+), Kubernetes migrates service IP range management to dedicated objects named ServiceCIDR. On the first boot after the upgrade, the API Server automatically tries to create the default object (e.g., 10.1.0.0/20).

For the curious, KEP-1880 and the official ServiceCIDR documentation detail this evolution.

It’s new, it’s clean, it’s well designed. Except that…

Interception by the Kyverno Webhook

Kyverno, configured with failurePolicy: Fail (because we’re serious people who don’t let just anything through in prod), is set up to intercept resource creations to validate them, and fail the call if Kyverno doesn’t respond.

Including the ServiceCIDR freshly created by the API Server itself.

Deadlock

And this is where it gets beautiful:

The API Server pauses the ServiceCIDR creation waiting for Kyverno’s “OK”
To contact the Kyverno service, the API Server needs to route the request through the Kubernetes service IP (typically 10.1.0.1)
But the network layer (service routing) can’t initialize until the ServiceCIDR object is validated and created

It’s the chicken and the egg, “I locked my keys inside the car” edition.

PTSD. Yes, that actually happened to me. In the desert. With no cell service.

Profit.

The API Server times out or returns a connect: operation not permitted error when trying to reach the webhook, blocking its own initialization. CrashLoopBackOff on the API Server. :D

Breaking out of the deadlock

To escape this deadlock, you need to temporarily bypass the admission layer. Easy, right?

The “usual” workaround: useless

Deadlocks with Kyverno, we’re used to them at this point. Normally, since kube-system is ignored, you can simply connect with a break-glass kubeconfig (we normally use OIDC) that has the cluster-admin cluster role and delete the Kyverno validating webhooks:

kubectl delete validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg

Except here, the API Server won’t even start. My kubectl isn’t going to work, obviously!

The real workaround: disable webhooks at boot

The solution we chose was to modify the API Server configuration to temporarily disable validation webhooks at startup. My esteemed colleague Maxime hot-edited the machine config (using break-glass talosctl access) to add the following flag directly in the API server’s extraArgs:

--disable-admission-plugins=ValidatingAdmissionWebhook

For those unfamiliar with admission control in Kubernetes, just know that there’s a list of “default” plugins but everything can be toggled off. I might do a deep dive on Kubernetes admission control someday, it’s fascinating ;).

With this flag, the API Server can finally create its ServiceCIDR objects without asking anyone for permission (completely bypassing all validation mechanisms that Kyverno or similar tools enforce), the network initializes, Kyverno starts, and then you can remove the flag and restart cleanly.

The “funny” option we didn’t try

Personally, I thought it would be hilarious to go directly into the etcd database and delete the webhook key causing the issue (also through talosctl):

# Example via etcdctl
etcdctl del /registry/admissionregistration.k8s.io/validatingwebhookconfigurations/kyverno-resource-validating-webhook-cfg

My colleagues were less enthusiastic: “Yeah but you know, if we break etcd it’s gonna be painful”. We played it safe with the flag. I’m deeply disappointed we didn’t try 😂.

The permanent fix: MatchConditions

OK, now that the cluster is back up, how do we make sure this doesn’t happen again on the next upgrade?

The clean solution is to use matchConditions (introduced in Kubernetes 1.27) on the ValidatingWebhookConfiguration. This allows you to exclude critical network bootstrap resources before the request even attempts to leave the API Server toward the Kyverno pod.

See the official documentation on matchConditions.

We were already using this option to throttle the sometimes excessive Kyverno traffic (if you manage Kyverno, you know what I’m talking about) on a number of events (we’d overwhelm the API server or Kyverno, in CPU or RAM, depending on the case). We just had to add exclusions for the new types:

# Exclude network bootstrap resources to prevent the deadlock
matchConditions:
 - name: 'exclude-ServiceCIDR'
 expression: '!(request.kind.kind == "ServiceCIDR")'
 - name: 'exclude-IPAddress'
 expression: '!(request.kind.kind == "IPAddress")'

With this, when the API Server creates a ServiceCIDR at boot, the request no longer goes through the Kyverno webhook. No circular dependency, no deadlock, everyone’s happy.

Conclusion

As the current French president would say about something that was painfully predictable:

who could have predicted this?

OK fine, all we had to do was read the Kubernetes 1.33 release notes. That said, we have a staging cluster, that’s what it’s for. We broke staging, no big deal.

Context: this is the “Big deal” TV Game Mascot

Maybe we’ll actually read them next time?

101 ways to deploy Kubernetes: a brand new UI to explore 118+ solutions

Mon, 09 Feb 2026 18:00:00 +0200

From Google Sheet to a real web application

You might remember my previous posts about this project: first a simple Google Sheet with 93 methods, then a GitHub repository with over 100 entries.

Today, I can present the latest iteration of this project: a real web interface to explore all these solutions!

👉 101-ways-to-deploy-kubernetes

Why a UI?

The Markdown table on GitHub was already better than the Google Sheet for collaboration, but barely. Hard to parse, hard to add columns without it becoming a mess (it already was, haha), and above all, incredibly UGLY.

I therefore decided to turn all of this into a modern and intuitive interface, with the help of an LLM.

Tech stack: Astro + Tailwind

For this project, I chose a simple but effective stack:

Astro: a modern framework that generates ultra-fast static sites
Tailwind CSS: for hassle-free responsive design

The result? A lightweight, fairly fast site that works on both desktop and mobile (though the desktop experience remains more comfortable given the amount of data).

Features

Cards for each solution

No more spreadsheet-style table worthy of a backend dev (no, worse, a kube engineer…)! Each tool now has its own animated “card” with:

The project logo
The license type (OSS or proprietary)
The GitHub star count
Direct links to the project and third-party resources (independent blogs, experience reports, tutorials…)

Powerful filters

Looking only for open source solutions? Tools for local development? Management platforms?

Category filters make navigation easy:

Desktop (local development)
Managed (cloud offerings)
Self-hosted (on-premise automation)
Infra As Code
Kubernetes OS (specialized operating systems)
Management Platform
Kubernetes in Kubernetes

You can also filter by status (active, abandoned) or show only open source or production ready solutions.

A search bar

Know what you’re looking for? Just type the name in the search bar to find the solution instantly.

Tags for refinement

Beyond categories, tags help quickly identify underlying technologies (kubeadm, k3s, k0s…).

Did you know? At least 18 tools use kubeadm!

While compiling all this data, I discovered something fascinating: at least 18 tools use kubeadm as a backend to deploy Kubernetes! 🤯

And that’s not even counting the managed Kubernetes offerings from cloud providers!

This is exactly the kind of information you can now visualize instantly thanks to this new interface.

A collaborative project

The project remains 100% open source and collaborative. The data is still stored in the GitHub repository, and the UI is automatically generated from that data (I even added PR previews).

Is a tool or provider missing? Spotted a bug? Feel free to open an issue or submit a Pull Request!

The project now lists 118 solutions (and I know there are certainly more missing), each with:

Up-to-date links
Project status
External references (tutorials, experience reports…)

Head over to zwindler.github.io/101-ways-to-deploy-kubernetes to explore all these solutions!

OK, this is a shameless “call to action” like you see on every social network. Fair enough.

However, I can’t know if this is useful (or not) if you don’t tell me. I can leave it as is (it’s not a big deal, I have plenty of other projects waiting for my spare time) or keep it alive, if you like it / find it useful.

And if you do find this project useful, please:

Star the project on GitHub
Share it with your colleagues in the Cloud Native community
Contribute by adding missing tools or fixing errors

Thanks in advance! 🙏