Zwindler's Reflection

GenAI and software development, episode 2: kubectl-debug-pvc, from idea to krew in 2x30 minutes

Tue, 17 Mar 2026 18:00:00 +0100

Previously, on “GenAI and dev”

In my previous article, I talked about my experience with PodSweeper, a project developed with OpenCode and Claude Opus. The outcome was mixed: impressive raw speed, but race conditions, lax error handling, and a constant need for human supervision (among other disappointments).

Today, I’m talking about a second project, much simpler, born from a real production need. And the takeaway is quite different.

The incident that started it all

You may know this situation: a pod in production, running fine, using a PVC in ReadWriteOnce. You need to look at the contents of that volume. Production best practice: the pod in question has no shell (we reduce the attack surface). No /bin/sh, no /bin/bash, nothing.

No big deal, we have kubectl debug for that, right?

Ok, let’s create an ephemeral container in the pod and… oh wait. kubectl debug doesn’t allow mounting the pod’s volumes in the ephemeral container. It simply doesn’t expose the volumeMounts option for ephemeral containers.

We could kill the pod, which would allow mounting the RWO PVC in another pod with a shell, but we don’t want to — it’s production.

My colleague Maxime (him again!) suggested a workaround. The manual procedure is:

Create an ephemeral container on the pod with kubectl debug
Manually build a JSON patch to add volumeMounts to the ephemeral container we just created
In another terminal, connect directly to the kube API with kubectl proxy
Apply the patch via a curl to the Kubernetes API
Wait for the container to be ready, attach to the container
Wonder if there’s an easier career than patching containers with curl

For those who think this is doable, “check out this patch” as the kids say:

curl http://localhost:8001/api/v1/namespaces/<namespace>/pods/<pod>/ephemeralcontainers \
 -X PATCH \
 -H 'Content-Type: application/strategic-merge-patch+json' \
 -d '{
 "spec": {
 "ephemeralContainers": [
 {
 "name": "debugger",
 "image": "ubuntu",
 "command": ["/bin/sh"],
 "targetContainerName": "<target-container>",
 "stdin": true,
 "tty": true,
 "volumeMounts": [
 {
 "name": "<volume-name>",
 "mountPath": "/debug/<volume-name>"
 }
 ]
 }
 ]
 }
 }'

If you think this is error prone, you’re right. And if you think it’s painful to do under pressure during a production incident, you’re even more right.

Side note

The more seasoned among you with recent Kubernetes versions may have heard about KEP-2590, which introduces a (relatively) new --subresource flag for kubectl. Indeed, ephemeral containers created by kubectl debug are not resources (a pod, a deployment, …) but subresources.

Until version 1.33, it was literally impossible to perform operations on subresources with kubectl, only resources.

Kubernetes v1.33: Octarine brings support for the –subresource flag, but only for 3 subresources for now: status, scale and resize

No solution on that front either…

The meeting prompt

I had this idea in mind since the incident. Monday morning, at the start of a meeting (while people were still making jokes), I launched OpenCode with Claude Opus and typed a prompt describing:

The problem: kubectl debug doesn’t mount PVC volumes if they’re RWO
The manual procedure I was doing by hand (the painful steps described above)
What I wanted: a tool that automates all of that

OpenCode + Opus asked me 2 questions (and I added one instruction):

“Which language?” → Go (I insist)
“CLI only or TUI?” → Both (non-interactive mode for scripting, TUI for daily use)
And I asked it to always run the golangci-lint linter every time it considers being done (trauma from my previous test with OpenCode)

It went on its own with the idea of a TUI using Bubble Tea. Then I stopped watching and followed my meeting.

~30 minutes later, end of the meeting, I looked at the result. The POC was functional.

What Opus produced in 30 minutes

Without any intervention on my part, the LLM scaffolded a complete Go project:

Clean structure: cmd/ for the Cobra CLI, pkg/k8s/ for all Kubernetes interaction, pkg/tui/ for the Bubble Tea TUI
The core of the matter: a direct call to the Kubernetes API to patch the ephemeralcontainers subresource of the pod with a strategic merge patch including the volumeMounts
Non-interactive mode: -n namespace -p pod -v volume:/mount/path, ready for scripting
Complete TUI: vim-style navigation (j/k), fuzzy filtering (/), multi-selection of volumes, loading spinner…
Makefile with a whole bunch of targets (vet, lint, test, build, install)

What I asked it to add:

Smart discovery: instead of scanning all pods in all namespaces (slow on one of my large clusters), the tool first lists PVCs cluster-wide (a single API call) to identify relevant namespaces, THEN the pods in the selected namespace. This drastically reduces the number of calls to the kube API server and the TUI only shows namespaces and pods that have PVC volumes
SecurityContext inheritance: the ephemeral container copies the securityContext from the target container, allowing it to pass PodSecurity policies (restricted, baseline…). This was a case I hadn’t anticipated in my initial prompt and it immediately failed on a properly configured cluster.

Without these small improvements the tool was already usable (except when you have SecurityContext configured), but it’s much more comfortable in practice (because yes, I do use it).

The code for the core mechanism (the API patch) is a few hundred lines of clean Go. It retrieves the pod, builds the JSON patch with the ephemeral container and its volumeMounts, applies it via Patch() on the subresource, waits for the container to be Running, then launches kubectl attach. Nothing fancy, it’s just what’s needed, done right the first time.

 ephemeralContainer := corev1.EphemeralContainer{
 EphemeralContainerCommon: corev1.EphemeralContainerCommon{
 Name: containerName,
 Image: opts.Image,
 Command: []string{"/bin/sh"},
 Stdin: true,
 TTY: true,
 VolumeMounts: mounts,
 SecurityContext: targetSecurityContext,
 },
 TargetContainerName: targetContainer,
 }
...
 _, err = c.Clientset.CoreV1().Pods(opts.Namespace).Patch(
 ctx,
 opts.PodName,
 types.StrategicMergePatchType,
 patchBytes,
 metav1.PatchOptions{},
 "ephemeralcontainers",
 )

The following iterations (~30 minutes)

Once the POC was validated, I chained a few targeted prompts. I asked it what could be improved. It suggested (and implemented):

A few minor code fixes it had gotten wrong (but non-blocking)
Warnings: if the inherited SecurityContext has readOnlyRootFilesystem, runAsNonRoot, or other security measures, the tool warns the user before attach
Added a complete CI based on goreleaser and GitHub Actions
Added documentation

The cherry on top: publishing on Krew

Once I had a clean version, I asked Opus how to make the plugin easier to install. It suggested brew, or Krew, the kubectl plugin manager.

I asked if the plugin acceptance process was complex. It said no, I said “I dare you!”.

It completed the entire process:

Created a fork of krew-index
Wrote the Krew manifest (the .yaml file describing the plugin)
Took into account all the best practices requested by krew-index maintainers (download URL format, short descriptions, SHA256 checksums, etc.)
Prepared the PR, directly on krew-index

I just watched. The PR was merged 12 hours later by the maintainers.

Result: kubectl krew install debug-pvc works.

Minor failure — delegating the demo too

Emboldened by this success with the krew-index PR, I wondered if we could make a visual demo (a GIF to add to the project’s README.md showing how it works) with the LLM.

I asked if it could do it with asciinema and the LLM answered “yes” (yes, I chat with the LLM like I do with my colleagues). Deal, we tried.

The result was almost good, I could have settled for it if I weren’t such a nitpicker: it was a bit slow. I felt the LLM’s responsiveness in its actions was uneven, which made the viewing experience a bit unpleasant. I could have iterated until I got something correct, but I ultimately recorded a video myself, converted to GIF with ffmpeg. It was smoother and faster than iterating.

The difference with PodSweeper

The difference in experience between this project and PodSweeper is striking. Where PodSweeper was a constant fight (race conditions, LLM amnesia, out-of-spec features), kubectl-debug-pvc went smoothly without any notable hiccups.

Why? I think it comes down to the nature of the project:

One single thing to do, clearly defined: patch a Kubernetes subresource with volumeMounts
No concurrent logic: we make an API request, we wait, we attach
A well-documented domain: the Kubernetes API, Cobra, Bubble Tea. The LLM knows all of this by heart
No complex shared state: no goroutines stepping on each other, no graceful shutdown to manage, no multiple microservices
Limited scope: the entire project fits in about fifteen files, CI and documentation included

This is probably the ideal playground for an LLM. A well-defined problem, a well-charted technical domain, a linear solution.

What do I think about it?

Objectively, this kind of “simple” project (one thing to do, easy to understand) works really well with OpenCode + Opus. I think it’s because of this kind of small project that the hype is so strong around agentic development. You have an idea you were too lazy to dev, you test it, it works. “WOW”.

But I don’t want to downplay the work done either. The fact that a functional, clean tool, published on Krew and usable by anyone could emerge from a prompt launched at the beginning of a meeting is still pretty wild.

A bit scary too, thinking that an insane number of micro tools are going to appear in the coming months, with probably uneven quality.

The project

The code is available here:

https://github.com/zwindler/kubectl-debug-pvc

Installation:

# Via Krew (recommended)
kubectl krew install debug-pvc

# Interactive usage (TUI)
kubectl debug-pvc

# Non-interactive usage
kubectl debug-pvc -n my-namespace -p my-pod-0 -v data:/debug/data

If you’ve ever struggled to inspect a PVC on a pod without a shell, give it a try. And if you find bugs, you can blame the LLM 😄.

Flannel and NetworkPolicies: how to add support with Cilium in CNI chaining mode

Sun, 15 Mar 2026 18:00:00 +0200

Flannel, flannel, flannel…

Flannel is a simple and popular CNI 😢.

It’s the default CNI in k3s, the one half the kubeadm tutorials use, and you’ll find it in quite a few managed offerings too.

OK, it’s simple, it routes packets between pods, it supports VXLAN and WireGuard, it takes 2 minutes to set up. What more could you ask for?

Well, exactly. There’s one thing flannel does not do: NetworkPolicies. (And that’s a big deal).

Flannel is focused on networking. For network policy, other projects such as Calico can be used.

[Edit] Contrary to what I wrote here, flannel actually does support NetworkPolicies and has for at least 2 years, via the reference implementation kube-network-policies from the Kubernetes project. The option isn’t prominently featured in the README and is probably not more recommended for production, but it does exist. See the official documentation.

And the trap is that if you haven’t read that one little line in the README.md, nothing tells you explicitly.

You can perfectly well create NetworkPolicy objects in your cluster, kubectl apply won’t complain, kubectl get netpol will happily list them. Except… they’re not enforced. Traffic still flows. Your deny-all doesn’t deny anything at all.

Let’s verify to be sure

Before solving anything, let’s verify the problem exists. Starting from the prerequisite that we have a Kubernetes cluster with flannel as the CNI and everything is working, we’ll deploy two pods in two different namespaces: a client (curl) and a server (nginx).

Let’s check that the client can reach the server:

kubectl exec -n netpol-test-a client -- \
 curl -s --max-time 5 http://server.netpol-test-b.svc.cluster.local

We get the nginx welcome page. So far, so normal. Now, let’s apply a deny-all NetworkPolicy on the server’s namespace:

# 01-deny-all-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: deny-all-ingress
 namespace: netpol-test-b
spec:
 podSelector: {}
 policyTypes:
 - Ingress

kubectl apply -f 01-deny-all-ingress.yaml

And let’s test again:

kubectl exec -n netpol-test-a client -- \
 curl -s --max-time 5 http://server.netpol-test-b.svc.cluster.local

Result: the nginx page still shows up. The NetworkPolicy was created (kubectl get netpol -n netpol-test-b shows it), but it’s not enforced. Traffic flows as if nothing happened.

This is the expected behavior with flannel (by default). Flannel only does L3 routing (VXLAN or WireGuard overlay). It doesn’t implement a NetworkPolicy controller. The objects exist in etcd, but nobody translates them into filtering rules.

Let’s clean up the policy before moving on:

kubectl delete -f 01-deny-all-ingress.yaml

Alternatives for adding support

For a long time I thought this was just the way it was. I still think it’s not a good choice for any production.

BUT recently, I discovered that it was possible to chain CNIs within the same cluster, and thus have one CNI handling most tasks (here Flannel) and another one taking care of other tasks, such as enforcing NetworkPolicies.

This is actually the principle behind Canal, which I knew by name but had never explored. In fact, it’s nothing more than a manifest that deploys Flannel as the main CNI with Calico for NetworkPolicy enforcement!

Canal was the name of Tigera and CoreOS’s project to integrate Calico and flannel.

Note: the GitHub project was archived in October 2025 but in theory it should still work, if you can find the correct documentation (I couldn’t, the links are broken, but I didn’t really look that hard either).

So you get the idea: we’re going to add a component that will watch NetworkPolicy objects and translate them into effective filtering rules (iptables, eBPF, nftables…), without touching the existing flannel setup. This is called “CNI chaining” or “policy-only” mode.

There are several solutions:

Calico (Canal); Historically, the flannel + Calico combination is called “Canal”, which I just mentioned. But the official Canal manifest bundles its own flannel in the same DaemonSet as calico-node. If your flannel is already installed and managed (by you, by an operator, by a provider…), you probably don’t want to replace it. And the Tigera operator (the “official” Helm method) doesn’t support policy-only deployment on an existing flannel either. In short, it’s doable but requires some effort. Can’t be bothered.

kube-router; kube-router can run in firewall-only mode (--run-firewall=true) and only needs iptables/ipset. This is actually what k3s uses by default for NetworkPolicies. It’s the lightest solution (supposedly ~50 MB of RAM per node). Make sure your kernel has the ip_set module, otherwise it won’t work.

Cilium in CNI chaining mode; This is the solution I chose and that we’ll detail here. Cilium attaches to the veth interfaces created by flannel and adds its eBPF programs for policy enforcement. No dependency on iptables or ipset, and as a bonus we get Hubble for network observability.

Installing Cilium in CNI chaining mode

Prerequisites

A working Kubernetes cluster with flannel
helm v3+
A kernel >= 4.19 (ideally >= 5.10 for all eBPF features)

Retrieve flannel’s CNI configuration

Cilium in chaining mode needs to know the existing CNI configuration. Let’s retrieve it from a node:

kubectl debug node/$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') \
 -it --image=busybox -- cat /host/etc/cni/net.d/10-flannel.conflist

On my cluster, this gives:

{
 "name": "cbr0",
 "cniVersion": "0.3.1",
 "plugins": [
 {
 "type": "flannel",
 "delegate": {
 "hairpinMode": true,
 "isDefaultGateway": true
 }
 },
 {
 "type": "portmap",
 "capabilities": {
 "portMappings": true
 }
 }
 ]
}

Note the name field (here cbr0). We’ll need it.

Create the chaining ConfigMap

We’ll create a ConfigMap that takes the flannel config and adds the cilium-cni plugin in chaining mode:

apiVersion: v1
kind: ConfigMap
metadata:
 name: cni-configuration
 namespace: kube-system
data:
 cni-config: |-
 {
 "name": "cbr0",
 "cniVersion": "0.3.1",
 "plugins": [
 {
 "type": "flannel",
 "delegate": {
 "hairpinMode": true,
 "isDefaultGateway": true
 }
 },
 {
 "type": "portmap",
 "capabilities": {
 "portMappings": true
 }
 },
 {
 "type": "cilium-cni",
 "chaining-mode": "generic-veth"
 }
 ]
 }

Warning: the name field must match your flannel conflist. If yours is called cni0 or something else, adjust accordingly.

Before applying, also verify that your CNI uses veth interfaces (this is the default with flannel, but better safe than sorry). From a node:

ip -d link | grep veth

You should see veth-type interfaces corresponding to your pods, for example:

103: lxcb3901b7f9c02@if102: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
veth addrgenmode eui64 numtxqueues 1 numrxqueues 1

If that’s the case, Cilium’s generic-veth mode will work. Let’s apply:

kubectl apply -f cilium-cni-configmap.yaml

Install Cilium via Helm

helm repo add cilium https://helm.cilium.io/
helm repo update

Here are the values for chaining mode:

# cilium-values.yaml
cni:
 chainingMode: generic-veth
 customConf: true
 configMap: cni-configuration
 install: true

routingMode: native
enableIPv4Masquerade: false
enableIPv6Masquerade: false

hubble:
 enabled: true
 relay:
 enabled: true
 ui:
 enabled: true

The important points:

cni.chainingMode: generic-veth, this is chaining mode, Cilium attaches to existing veth interfaces
cni.customConf: true + cni.configMap, we provide our own CNI config
routingMode: native, flannel handles routing, not Cilium
enableIPv4Masquerade: false, flannel handles masquerading
hubble.enabled: true, network observability, the big bonus of Cilium

helm install cilium cilium/cilium --version 1.19.1 \
 --namespace kube-system \
 -f cilium-values.yaml

Wait for everything to be ready:

kubectl rollout status daemonset/cilium -n kube-system --timeout=120s

Verification

kubectl exec -n kube-system ds/cilium -- cilium status

What we’re looking for:

Kubernetes: Ok 1.35 (v1.35.0) [linux/amd64]
CNI Chaining: generic-veth
Cilium: Ok 1.19.1
Hubble: Ok

The CNI Chaining: generic-veth line confirms that Cilium is running in chaining mode and isn’t replacing flannel.

Important: pods that existed before Cilium was installed are not automatically managed by Cilium. You need to restart them so that Cilium can attach its eBPF programs. Remember to kubectl rollout restart your test workloads (or recreate them).

Testing NetworkPolicies

This is the moment of truth. Let’s re-apply our deny-all:

kubectl apply -f 01-deny-all-ingress.yaml

kubectl exec -n netpol-test-a client -- \
 curl -s --max-time 5 http://server.netpol-test-b.svc.cluster.local

Result: timeout! This time, the NetworkPolicy is properly enforced. Traffic is blocked.

I followed up with the other classic NetworkPolicy scenarios, and everything works:

Selective ingress allow by namespace; by adding a policy that allows traffic from netpol-test-a only, curl works from that namespace but remains blocked from default. Namespace isolation works.
Deny-all egress; by blocking all outgoing traffic from the client, even DNS resolution is blocked (immediate timeout).
Selective egress allow; by allowing only DNS (port 53) and the server (port 80 in the netpol-test-b namespace), curl to the server works but curl http://example.com remains blocked.

In short, ingress, egress, selective by namespace, everything works as expected.

Bonus: Hubble, network observability

This is for me the real advantage of Cilium over the alternatives. Hubble lets you see network flows and policy verdicts in real time:

kubectl exec -n kube-system ds/cilium -- \
 hubble observe --namespace netpol-test-b --last 5

Mar 15 13:20:42.287: netpol-test-a/client:40066 (ID:9745) ->
netpol-test-b/server:80 (ID:22271)
policy-verdict:none ALLOWED (TCP Flags: SYN)

You can see the source pod, destination pod, port, Cilium identity, and policy verdict. When you’re debugging a NetworkPolicy that isn’t behaving as expected, this is incredibly useful.

How much does it cost in resources?

On my cluster (3 nodes), here’s what Cilium consumes right after installation:

Component	Per node	RAM
cilium agent	yes (DaemonSet)	~160 MB
cilium-envoy	yes (DaemonSet)	~22 MB
cilium-operator	no (2 replicas)	~42 MB
hubble-relay	no (1 replica)	~16 MB
hubble-ui	no (1 replica)	~21 MB

That’s roughly 180 MB per node for the agent + envoy. I don’t have a point of comparison with Calico or kube-router, but it seems acceptable to me, and being able to have full visibility into all network flows with Hubble more than justifies the overhead (in my opinion).

Conclusion

If you have no choice and have to work with flannel, and you want to plug the gaping security hole that is the lack of NetworkPolicy enforcement, know that it is possible to chain another CNI to handle it.

Short of having it as CNI for everything, Cilium in CNI chaining mode (generic-veth) is a pretty nice solution to fill this gap. It doesn’t touch the existing flannel setup, it grafts onto it. And as a bonus, you get Hubble for network observability, which is genuinely valuable.

Podcasts & Lives

Fri, 13 Mar 2026 00:00:00 +0000

Publications

Fri, 13 Mar 2026 00:00:00 +0000

GenAI and software development: lessons learned with PodSweeper

Sun, 08 Mar 2026 18:00:00 +0200

GenAI, is it fantastic?

Quite a few people have shared their takes on GenAI for development in a very short time, so I realize it’s more than time I post this draft I started over two weeks ago 🙃.

For work, I use AI assistants more and more to help me with my daily tasks. In 2024, it was mainly for automating tedious tasks (scripting stuff, making a painful list of repetitive tasks without coding it properly). Throughout 2025, I tested it several times for Ops work, and each time the results were mediocre, both for designing coherent, reliable and efficient infrastructure and for incident resolution assistance (see my article on the topic).

In 2026, on that last point, I feel we still haven’t progressed, though I’ll admit there are specialized tools I haven’t tested enough yet, open source or not. I can mention k8sgpt and HolmesGPT for open source, and Anyshift with its SRE agent for root cause detection and incident resolution.

Note: fun fact, in his post “My position on Generative Artificial Intelligence” posted just before mine, Alex cites a long list of professions that will benefit from GenAI, and ops folks get nothing more than yet another argument to switch to Kubernetes 😭.

On the other hand, I’ve started using GenAI to build several web projects from scratch (vibe coding?), which I’ve already told you about:

Converting a blog from Bloggrify to Hugo (french only)
Creating a “pretty” website (way beyond my skills) to host the research I’ve done on the different ways to deploy Kubernetes
Improving my website’s security. (french only) by automating modifications to the Hugo theme I use, to get the best score on Mozilla HTTP Observatory

In all 3 cases, the result is there. Or rather, it seems to be for the layman that I am.

I’ve never hidden that I have zero front-end skills, and maybe for a domain expert, what I did with AI is horrible (or not?). In any case, it can’t be worse than the UIs I made without it. Small example 😄:

or rather: GêneAI (sorry that a french play on word I can’t translate)?

This is the kind of feedback you see from people who are actually experts in a domain when they watch novices get excited. We have good examples with AI-generated videos: if you’re not paying attention, you think it’s incredible. But anyone with a slightly critical eye immediately sees BIG consistency problems. Same goes for image generation, or music.

It’s also true for code, and we have great examples of vibe coded projects that are horrific spaghetti messes and cybersecurity sieves. In short, you get it: even though I’m a daily user, I’m not “amazified” (as my kids would say) by LLMs, even the best ones.

Yet, I have several friends who swear by them, including people far more brilliant/intelligent (call it what you will) than me. So I thought “OK, maybe I’m doing it wrong. Let’s try with the best there is”: a “Claude Code”-type IDE and an Opus model.

The editor

After a quick market survey, you realize the options are plentiful: Claude Code, OpenCode, Amp Code, …

The problem with Claude Code is the entry price. I’m not yet ready to spend 100 or 200€ per month for the use I have today. I don’t know if the 20€ plan would be enough. Beyond the rare personal side projects I mentioned above, I don’t code much. Most of my geeky activities are infrastructure: installing Kubernetes clusters and virtualization OSes. Stuff that’s hard to automate with an LLM.

Pierre (mostly) recommended Amp Code for personal use, mainly because it has a free tier with a certain number of tokens and if you’re not too greedy, it’s quite effective. I preferred to do things my own way and test OpenCode instead, which has the advantage of being more flexible with LLM providers and token consumption. It’s even possible to use local models (free, therefore) or models included for free (at least for now) such as GLM 5 Free, which performs quite well among dev models (most importantly, it’s free…).

OK, we have the editor. Now, the code?

To get a sense of the relevance of code generated by my favorite LLM (Sonnet and Opus, for a while now), I need a language and a use case that I master, so I can tell it “no, that’s absolute nonsense!?”.

A use case I master… as you might guess, there will inevitably be some Kubernetes in there.

A language I master: I learned Go in 2022 thanks to a colleague at Deezer (thanks again Martial!) and I’ve published a few tools and made minor contributions. I wrote a large part of the GroROTI tool and also started (but never finished) developing an RPG in Go, heavily inspired by Castle of the Winds, a game from my childhood: gocastle. And I have professional-level proficiency (even if it’s not the core of my job) in my day-to-day work.

After the context, the project

OK, we have the technical context: Kube + Go. Now we need the idea to implement. The project needs to be large enough for the test to be meaningful, fun enough that I want to spend personal time on it, but still somewhat useful so I’d want to talk about it and show progress. And above all, a project whose business logic stays within my reach: to honestly evaluate the quality of AI-generated code, I need to be able to read and critique it.

And there, in my overflowing drawer of dumb ideas, I remembered this “pitch” from 2023-2024, which I’d left to rot for lack of time:

PodSweeper: the most complex, over-engineered and chaotic way to play Minesweeper.

Don’t leave, you’ll see it’s fun. Really!

Instead of a visual grid where you click on cells hoping not to hit a “mine”, we have a virtual “grid” where each cell is a Pod and the click is a kubectl delete.

Yes, it’s dumb. I like it a lot :).

Beyond the deliberate trolling (over-engineering from hell) of this game, there’s actually an educational objective behind it.

Really, there is.

I designed the game as an introduction to Kubernetes security, with difficulty levels to unlock, CTF-style.

At the beginning, you can do everything. You can of course play normally (delete a Pod, see if there are mines nearby) to get the hang of it. You can also automate actions (a script that “clicks a cell” at random, retrieves proximity hints for mines, clicks somewhere safe, …).

But you can also cheat.

And that’s the whole point of the game. In the first difficulty levels, the game’s Kubernetes manifests are deliberately vulnerable. So you can win effortlessly, if you know where to look.

However, levels quickly progress and restrictions come with them. Pretty soon, you reach production-grade best practices that prevent any “cheating”. And if you find unintended vulnerabilities in the game code itself… well, that’s bonus 😈.

The initial process

It’s probably a mistake, but I did the entire ideation phase with Gemini before launching OpenCode. The experience would probably have been better (at least more representative) if I’d started directly with OpenCode.

I dumped everything I had in mind, along with a big shapeless block of dozens of lines from my notes from when I had the idea in 2023, and asked it to produce a SPECIFICATIONS.md file with all the important information, put back in order.

Once the specs were written, I asked it to detail the different levels and the difficulties we would progressively add, in GAMEPLAY.md.

Finally, I asked it to break down the project into “issues” by priority, so I could get an MVP quickly: ISSUES_PRIORITY.md. My idea was to give OpenCode the very detailed, finely sliced battle plan and let it run.

First disappointment: Gemini 3 Pro is pretty bad at this. The tasks were credible(-ish) but ordered randomly (task dependencies not respected). So I launched OpenCode for the first time in my repo and told it “here’s the context, here’s what we came up with for tasks. What do you think?”

The good (open)code…

Among the pretty nice things: OpenCode + Claude Opus immediately saw that the plan created by Gemini was flawed, and asked me questions and proposed a breakdown that was still imperfect but already more coherent. This is probably where the main value of these tools lies. They embed knowledge to guide software development and above all (ABOVE ALL) bring structure.

Pretty quickly, we agree on a plan I like. OpenCode updates the documents and starts developing (scaffolding, creating pipelines, first empty binaries and Docker images). We have a project ready to develop in a snap, which is quite exciting, at first glance.

LLM assistants are very eager to jump into code, this remains true with OpenCode+Opus. Even though we hadn’t finished discussing the plan and possible options, it was spontaneously proposing to move to code. That said, it was the same (or worse) with Gemini (to whom I had specifically said we wouldn’t be writing any code at all).

Very quickly, file creations pile up. I struggle to keep up and at each pause (when the LLM has finished a task and asks if it can move to the next), I spend long minutes reading what the LLM produced. It’s both exhilarating and exhausting (re-reading all that is hard on my rusty brain).

The raw speed is impressive. In 3 sessions of 1 to 2 hours, I have a working MVP. I understand why people regularly talk about 10x engineering when discussing code generation with recent LLMs. If we’re not talking about raw code generation (which doesn’t really make sense), roughly speaking, functional code is generated 3x to 10x faster than what I could have done alone. Features that end to end would have taken me hours to implement (grid generation, reveal logic, game state management) land in minutes. You’ve read it elsewhere, I’m saying the same thing — the bottleneck is no longer the code I type: it’s the code I have to review.

From what I’ve read, the code is correct, particularly in the first iterations, a bit less so after a while. But when I say “correct”, I might be underselling it: the code produced is idiomatic Go. Good package structure, naming conventions respected, error handling in Go style (when it’s there…), appropriate use of interfaces. This isn’t just code that compiles: it’s code a Go reviewer wouldn’t reject outright. We’re aiming for an MVP so the business logic remains very simple, but the foundation is clean.

Everything is tested, very quickly the project has perfect coverage and 100+ tests while we’re not even doing anything yet. If this were human-written code, I wouldn’t know if that’s a good thing. We’re absolutely not doing TDD, a lot of code tests useless things. In the case of LLM-generated code, it’s probably for the best, because the LLM tends to introduce regressions fairly regularly (we’ll come back to that). So it’s interesting here, because:

it costs almost nothing in dev time (it’s so fast to generate)
it allows the LLM to realize its code introduced a regression, and fix it on its own, without waiting.

… and the bad (open)code

On the tool and model side first, a few irritants:

By default, OpenCode doesn’t encourage the LLM to run golangci-lint on every commit. You fix this with a pre-commit hook and/or the famous AGENTS.md / skills, but it SHOULD have been part of the basic kit for a golang project (tests are there…).
The LLM (Claude Opus on OpenCode, but this is a common bias even outside of it) loves software versions that existed at the time of its training. You have to constantly remind it that the versions it suggests are outdated. This is true for everything: Go code, dependencies, GitHub Actions versions.
You very often hit the 100k token limit. You waste time “compacting” and lose precision. Typically: the LLM asks if it can git commit changes, the context compacts before I answer, and it commits without my permission while re-unreeling the context. For code this low-stakes, it’s fine. But in prod, that would be a real problem.
The LLM is often amnesiac: it forgets it has access to a kind cluster for real testing (even though it did it just before the previous compaction, for example).

On the code quality side, it’s more concerning:

Lax error handling. Some errors that should have returned a FATAL (controller that can’t initialize properly!) are simply logged as ERROR. You end up shipping versions that fail silently. Again, this can probably be fine-tuned with an AGENTS.md.
Race conditions. I ran into race conditions fairly quickly, pretty silly ones. In my case, pods stuck in terminating impacted the next game (poor graceful shutdown handling). Opus went into a loop without understanding the root cause. I had to stop it and specify that it should never start a new game without making sure the previous one was finished.
Code generation outside the lines. Sometimes, without warning, the LLM adds a feature that wasn’t requested, is useless, or even contradicts the previously written SPECIFICATION. You have to slap it back into line.

Once put back on track, the LLM rolls again, but these episodes clearly show that for concurrent code or outside of ultra-strict user stories (which goes beyond OpenCode’s default workflow), human supervision remains essential.

People will tell me I’m a beginner with these tools, and I could have avoided some pitfalls by better configuring my environment (AGENTS.md, pre-commit hooks, etc.). That’s true.

But that’s exactly where the shoe pinches: one of the hyped promises of GenAI applied to development is to enable non-developers (or beginners) to produce quality software at high speed, so they can ship complete software. If getting the most out of it requires being an experienced developer who knows how to configure a complex environment (with the specifics of these AI tools), anticipate race conditions and review concurrent code… we haven’t democratized development, we’ve just given another tool to people who already knew how to code. My profile (ops who codes, not a pure dev) was precisely the target audience of this promise. And today, the math doesn’t check out.

Where am I at?

I’ve been talking about PodSweeper for a while now. But where is this project?

After 3 sessions of 1 to 2 hours max, not counting ideation, I have a working MVP. As I said earlier, the productivity gain is undeniable, for someone who isn’t a “developer” by trade.

The code is probably “overall” better quality than the Go code I could have written, but the LLM lets gaping holes through in the business logic (simply because it doesn’t think, while I do. Well, normally). Hard on trust… You really have to remain very wary of everything it produces.

The code is available at:

https://github.com/zwindler/PodSweeper/tree/main

You can go check out the code to form your own opinion. If you’re not afraid of spoilers, you can read the SPECIFICATION.md (spoilers), or even GAMEPLAY.md (I detail all the levels there so it’s mega spoilers).

You can (normally) play the first difficulty levels. It’s still fairly basic, if you’ve already used kubectl you should find the solution very quickly. My goal is to add levels over time, I’ve already imagined 10, some quite devious 😈 and others might come later.

The code is under MPL v2.0 because it’s a copyleft license I like, since it’s both not very restrictive, OSI compliant and still requires contributing changes back if you make them.

There you go :) if you also find it fun, feel free to test and/or give me feedback.

$ kubectl apply -k https://github.com/zwindler/podsweeper//deploy/base?ref=v0.1.4
namespace/podsweeper-game created
...

$ kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=podsweeper \
 -n podsweeper-game --timeout=60s
pod/gamemaster-54f4dddcc4-6m88z condition met

Have fun!

Kyverno killed my API Server. Again.

Thu, 26 Feb 2026 08:00:00 +0200

The revenge strikes back

You may have read my previous article about etcd 3 years ago. If you remember correctly, the crash was etcd, but the real culprit was Kyverno. I love Kyverno. It’s really a piece of software I’m very fond of. Mainly because it’s incredibly powerful. I even wrote an introductory article and a second one that goes deeper on the topic (both in french, though).

But the sheer number of incidents and weird side effects it causes. Mamamia… This isn’t the first incident of the year I’ve had with Kyverno (yes, we’re in February) but since this one is entertaining, I’m sharing it with you.

During a routine maintenance operation to upgrade a Kubernetes cluster to version 1.34 (from 1.32), we ended up facing the dreaded scenario for any kube admin: a completely unreachable API Server after restarting the Control Plane nodes.

What initially looked like a typical network error turned out to be a subtle deadlock between new native Kubernetes networking features and our dear Kyverno 😘.

Spoiler: it wasn’t a network issue. It’s never a network issue. Well, sometimes it is. But not this time.

The upgrade that starts well

Alright, a Kubernetes upgrade has become pretty routine at this point. We do it regularly, we have our procedures, we’re pros (I swear). We jump from 1.32 to 1.34 in a single commit, skipping the hop through 1.33.

YOLO.

In the technical context I’m talking about, everything is managed as code. From machine provisioning all the way to Talos deployment, including MachineConfigs (the CustomResources to modify… well, the machine).

For more details, see the Talos documentation on Machine Configs.

The first cluster we test has only one control plane (don’t ask me why, it probably wouldn’t have changed anything). Talos restarts the API Server with the new version and then… nothing.

The “weird” API Server logs (technical term) speak for themselves:

I0224 15:32:50.979280 1 default_servicecidr_controller.go:166] Creating default ServiceCIDR with CIDRs: [10.1.0.0/20]
W0224 15:32:50.984784 1 dispatcher.go:225] rejected by webhook "validate.kyverno.svc-fail":
admission webhook "validate.kyverno.svc-fail" denied the request:
Get "https://10.1.0.1:443/api": dial tcp 10.1.0.1:443: connect: operation not permitted
I0224 15:32:50.985342 1 event.go:389] "Event occurred" kind="ServiceCIDR"
apiVersion="networking.k8s.io/v1" type="Warning"
reason="KubernetesDefaultServiceCIDRError"
message="The default ServiceCIDR can not be created"

😬😬😬

The root cause: a magnificent vicious circle

After investigation, we discovered that the incident was the result of a collision between a Kubernetes core evolution and our Kyverno configuration. A textbook deadlock case.

Let’s break down the mechanism:

The new ServiceCIDR Kind

In recent versions (v1.33+), Kubernetes migrates service IP range management to dedicated objects named ServiceCIDR. On the first boot after the upgrade, the API Server automatically tries to create the default object (e.g., 10.1.0.0/20).

For the curious, KEP-1880 and the official ServiceCIDR documentation detail this evolution.

It’s new, it’s clean, it’s well designed. Except that…

Interception by the Kyverno Webhook

Kyverno, configured with failurePolicy: Fail (because we’re serious people who don’t let just anything through in prod), is set up to intercept resource creations to validate them, and fail the call if Kyverno doesn’t respond.

Including the ServiceCIDR freshly created by the API Server itself.

Deadlock

And this is where it gets beautiful:

The API Server pauses the ServiceCIDR creation waiting for Kyverno’s “OK”
To contact the Kyverno service, the API Server needs to route the request through the Kubernetes service IP (typically 10.1.0.1)
But the network layer (service routing) can’t initialize until the ServiceCIDR object is validated and created

It’s the chicken and the egg, “I locked my keys inside the car” edition.

PTSD. Yes, that actually happened to me. In the desert. With no cell service.

Profit.

The API Server times out or returns a connect: operation not permitted error when trying to reach the webhook, blocking its own initialization. CrashLoopBackOff on the API Server. :D

Breaking out of the deadlock

To escape this deadlock, you need to temporarily bypass the admission layer. Easy, right?

The “usual” workaround: useless

Deadlocks with Kyverno, we’re used to them at this point. Normally, since kube-system is ignored, you can simply connect with a break-glass kubeconfig (we normally use OIDC) that has the cluster-admin cluster role and delete the Kyverno validating webhooks:

kubectl delete validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg

Except here, the API Server won’t even start. My kubectl isn’t going to work, obviously!

The real workaround: disable webhooks at boot

The solution we chose was to modify the API Server configuration to temporarily disable validation webhooks at startup. My esteemed colleague Maxime hot-edited the machine config (using break-glass talosctl access) to add the following flag directly in the API server’s extraArgs:

--disable-admission-plugins=ValidatingAdmissionWebhook

For those unfamiliar with admission control in Kubernetes, just know that there’s a list of “default” plugins but everything can be toggled off. I might do a deep dive on Kubernetes admission control someday, it’s fascinating ;).

With this flag, the API Server can finally create its ServiceCIDR objects without asking anyone for permission (completely bypassing all validation mechanisms that Kyverno or similar tools enforce), the network initializes, Kyverno starts, and then you can remove the flag and restart cleanly.

The “funny” option we didn’t try

Personally, I thought it would be hilarious to go directly into the etcd database and delete the webhook key causing the issue (also through talosctl):

# Example via etcdctl
etcdctl del /registry/admissionregistration.k8s.io/validatingwebhookconfigurations/kyverno-resource-validating-webhook-cfg

My colleagues were less enthusiastic: “Yeah but you know, if we break etcd it’s gonna be painful”. We played it safe with the flag. I’m deeply disappointed we didn’t try 😂.

The permanent fix: MatchConditions

OK, now that the cluster is back up, how do we make sure this doesn’t happen again on the next upgrade?

The clean solution is to use matchConditions (introduced in Kubernetes 1.27) on the ValidatingWebhookConfiguration. This allows you to exclude critical network bootstrap resources before the request even attempts to leave the API Server toward the Kyverno pod.

See the official documentation on matchConditions.

We were already using this option to throttle the sometimes excessive Kyverno traffic (if you manage Kyverno, you know what I’m talking about) on a number of events (we’d overwhelm the API server or Kyverno, in CPU or RAM, depending on the case). We just had to add exclusions for the new types:

# Exclude network bootstrap resources to prevent the deadlock
matchConditions:
 - name: 'exclude-ServiceCIDR'
 expression: '!(request.kind.kind == "ServiceCIDR")'
 - name: 'exclude-IPAddress'
 expression: '!(request.kind.kind == "IPAddress")'

With this, when the API Server creates a ServiceCIDR at boot, the request no longer goes through the Kyverno webhook. No circular dependency, no deadlock, everyone’s happy.

Conclusion

As the current French president would say about something that was painfully predictable:

who could have predicted this?

OK fine, all we had to do was read the Kubernetes 1.33 release notes. That said, we have a staging cluster, that’s what it’s for. We broke staging, no big deal.

Context: this is the “Big deal” TV Game Mascot

Maybe we’ll actually read them next time?

101 ways to deploy Kubernetes: a brand new UI to explore 118+ solutions

Mon, 09 Feb 2026 18:00:00 +0200

From Google Sheet to a real web application

You might remember my previous posts about this project: first a simple Google Sheet with 93 methods, then a GitHub repository with over 100 entries.

Today, I can present the latest iteration of this project: a real web interface to explore all these solutions!

👉 101-ways-to-deploy-kubernetes

Why a UI?

The Markdown table on GitHub was already better than the Google Sheet for collaboration, but barely. Hard to parse, hard to add columns without it becoming a mess (it already was, haha), and above all, incredibly UGLY.

I therefore decided to turn all of this into a modern and intuitive interface, with the help of an LLM.

Tech stack: Astro + Tailwind

For this project, I chose a simple but effective stack:

Astro: a modern framework that generates ultra-fast static sites
Tailwind CSS: for hassle-free responsive design

The result? A lightweight, fairly fast site that works on both desktop and mobile (though the desktop experience remains more comfortable given the amount of data).

Features

Cards for each solution

No more spreadsheet-style table worthy of a backend dev (no, worse, a kube engineer…)! Each tool now has its own animated “card” with:

The project logo
The license type (OSS or proprietary)
The GitHub star count
Direct links to the project and third-party resources (independent blogs, experience reports, tutorials…)

Powerful filters

Looking only for open source solutions? Tools for local development? Management platforms?

Category filters make navigation easy:

Desktop (local development)
Managed (cloud offerings)
Self-hosted (on-premise automation)
Infra As Code
Kubernetes OS (specialized operating systems)
Management Platform
Kubernetes in Kubernetes

You can also filter by status (active, abandoned) or show only open source or production ready solutions.

A search bar

Know what you’re looking for? Just type the name in the search bar to find the solution instantly.

Tags for refinement

Beyond categories, tags help quickly identify underlying technologies (kubeadm, k3s, k0s…).

Did you know? At least 18 tools use kubeadm!

While compiling all this data, I discovered something fascinating: at least 18 tools use kubeadm as a backend to deploy Kubernetes! 🤯

And that’s not even counting the managed Kubernetes offerings from cloud providers!

This is exactly the kind of information you can now visualize instantly thanks to this new interface.

A collaborative project

The project remains 100% open source and collaborative. The data is still stored in the GitHub repository, and the UI is automatically generated from that data (I even added PR previews).

Is a tool or provider missing? Spotted a bug? Feel free to open an issue or submit a Pull Request!

The project now lists 118 solutions (and I know there are certainly more missing), each with:

Up-to-date links
Project status
External references (tutorials, experience reports…)

Head over to zwindler.github.io/101-ways-to-deploy-kubernetes to explore all these solutions!

OK, this is a shameless “call to action” like you see on every social network. Fair enough.

However, I can’t know if this is useful (or not) if you don’t tell me. I can leave it as is (it’s not a big deal, I have plenty of other projects waiting for my spare time) or keep it alive, if you like it / find it useful.

And if you do find this project useful, please:

Star the project on GitHub
Share it with your colleagues in the Cloud Native community
Contribute by adding missing tools or fixing errors

Thanks in advance! 🙏

Installing a Cluster HAT with Raspberry Pi 5 and Pi Zero

Tue, 27 Jan 2026 18:00:00 +0200

Introduction, What Is a Cluster HAT?

What have you found for us this time???

You’ve seen me play with Raspberry Pis and install Kubernetes on them, or just simply Alpine Linux.

So what on earth (yes, “on earth,” let’s not mince words) are we talking about today???

While searching the depths of the Internet to see if it was possible to install a Kubernetes cluster on Raspberry Pi Zeros, I came across 3 different people who had failed at this task:

Setting up a Kubernetes cluster using Raspberry Pi Zero 2 W’s and the associated reddit post with a photo (important for the rest of the article)
github.com/abelperezok/kubernetes-raspberry-pi-cluster-hat
A tiny cluster based on 4x Raspberry Pi Zero 2 W

In all 3 cases, people tried to install a cluster of RPi zeros, mounted on some kind of additional board I didn’t know existed: the Cluster HAT

Official Cluster HAT website clusterhat.com

The Cluster HAT (Hardware Attached on Top) which interfaces a (Controller) Raspberry Pi A+/B+/2/3 with 4 Raspberry Pi Zeros configured to use USB Gadget mode is an ideal tool for teaching, testing or simulating small scale clusters.

The Cluster HAT can be used with any mix of Pi Zero 1.2, Pi Zero 1.3 and Pi Zero W.

USB Gadget Mode: Ethernet and Serial Console.

Onboard 4 port USB 2.0 hub.

Raspberry Pi Zeros powered via Controller Pi GPIO (USB optional).

Individual Raspberry Pi Zero power controlled via Controller Pi GPIO (I2C).

Connector for Controller Serial Console (FTDI Basic).

Controller Pi can be rebooted without interrupting power to Pi Zeros, network recovers on boot.

Problem(s)

Well, obviously, it’s absurd, so it’s yet another project for me!!

First problem: the Cluster HAT theoretically cannot support Pi Zero 2Ws. At least that’s what you read everywhere (on reseller websites, and what you can deduce from the introduction I included just above). And that’s really unfortunate because that’s the model I have (I have two) and I’d have been bummed to have to buy 4 in version 1.3 just to make sure it works.

Fortunately:

Several comments found on the net indicate that in reality, it works if the power supply (27w for the official one) of the Pi 5 is powerful enough; moreover, the FAQ says it actually works
Quentin kindly gave me 2 Pi Zero 1.3s, which completes my collection. I’ll have 2 of each, that should be fine power-wise.

Second problem, finding one (Cluster HAT)! The last version of this product was manufactured several years ago. The official site doesn’t ship to France, the product no longer exists on Kubii.fr. I found a few on The Pi Hut (in the United Kingdom and there are none left today), for about forty euros, shipping included.

Last point, I had to find a Pi 5 at a reasonable price and it hurt to pay €70 for a second-hand Pi 5 with only 4 GB.

So with accessories (SD cards) and the various boards, we’re looking at a side project of about €200.

That’s a bit expensive for an absurd project, but I’m funding it with an article (on another topic) that will be published in a magazine in a few weeks. Those who know, know ;-P.

Hardware Installation

Probably the most fun part when you like (like me) tinkering with computer components and electronic boards.

Insert the small screws to raise the “Hat”
Insert the Cluster HAT into the GPIO ports of the Raspberry Pi 5
Insert the 4 Pi Zeros into the USB ports of the HAT

Power Supply: You must use a sufficiently powerful power supply, ideally the official Pi 5 power supply (27W USB-C). The Cluster HAT draws its energy from the Pi 5 to power the 4 Zeros.

SD Cards: You need 5 MicroSD cards in total (1 for the Pi 5, 4 for the Zeros). The site mentions booting the Pi zeros without MicroSD cards (experimental feature). I might look into it?

OK, that was trivial. But it proves the product is well designed!

3D Case

Since I have a 3D printer, I modified an existing model for RPis with an “M.2 hat” (another additional board, this one for connecting an M.2 SSD) by making big HOLES everywhere so the Pi zeros could fit.

Raspberry Pi 5 AI/M.2 Hat+ Snap Case

Unfortunately, I cannot share the modified profile because it’s against the 3D file license (Standard Digital File License, which prohibits modifications).

Note: I spent more time looking for models, modifying them, and testing prototypes than actually playing with the cluster hat itself.

Preparing the Installation

The first thing to do to start enjoying this magnificent Raspberry cluster is to install and configure our Pi 5.

Even though it’s theoretically possible to do everything yourself, the cluster hat manufacturer recommends starting with the preconfigured images available on their site 8086.net. The latest version is available for download here:

dist1.8086.net/clusterctrl/bookworm/2025-11-24/

Looking through the list, we note that there are:

6 versions for the “master” RPi (Bookworm lite, normal and full, with CNAT or CBRIDGE mode for each)
preconfigured Rpi OS lite images for the Pi Zeros (called Px, x being the position on the cluster hat), also with CBRIDGE / CNAT variants, armhf (32-bit) and arm64

Out of laziness, I just plugged in the Pi 5, I don’t have an Ethernet cable within reach. In this case, if like me you connect your Pi via WiFi, you need to go with CNAT mode (NATed). On the contrary, CBRIDGE mode (bridged, which requires the Ethernet port) means you’ll see all 5 machines on your LAN directly.

Second piece of info (reminder), in my case, I have a mix of 32 and 64-bit machines, so I need to be careful to select the right OS for the right machines.

Installing the Raspberry Pi 5 (Our Controller)

For the Raspberry Pi 5, in my case the file is 2025-11-24-2-bookworm-ClusterCTRL-arm64-lite-CNAT.img.xz

wget https://dist1.8086.net/clusterctrl/bookworm/2025-11-24/2025-11-24-2-bookworm-ClusterCTRL-arm64-lite-CNAT.img.xz

The Headless Mode Problem

Even though it’s possible, I don’t want to connect the Pi to a monitor (having to get out a keyboard and a monitor… exhausting) so I’m going to do everything in headless mode. And unfortunately, the latest versions of Raspberry Pi Imager (2.0.0) don’t allow (anymore?) customization of third-party OSes.

In theory, once the image is flashed to the microSD, you’ll need to manually configure:

WiFi
a basic user
enable SSH

I tried, it failed, despite a wpa_supplicant.conf file, a properly formatted userconf.txt, and an empty ssh file.

The Solution: Raspberry Pi Imager 1.9.6

After getting quite frustrated, I simply went back to the previous version of the imager (1.9.6) which works very well with the image provided by 8086.net.

Note: however, for installing recent RPi OS versions, I had quite a few failures. So use sparingly.

Installing the 4 Raspberry Pi Zeros (The Nodes)

Here, it’s a bit simpler on the network side, because in CNAT mode, the Pi Zeros will communicate via the USB cable of the Cluster HAT. BUT you need to pay attention to the 32 vs 64-bit architecture issues, in my somewhat hybrid case:

P1 (a Pi Zero 2): ...arm64-lite-CNAT-p1.img.xz
P2 (a Pi Zero 1.3): ...armhf-lite-CNAT-p2.img.xz
P3 (a Pi Zero 1.3): ...armhf-lite-CNAT-p3.img.xz
P4 (a Pi Zero 2): ...arm64-lite-CNAT-p4.img.xz

For customization, we leave SSH and the user/password, but no need to set up WiFi. We could also modify the boot parameters to remove the Bluetooth and WiFi kernel modules to save a few MB of RAM.

Boot Up

Once everything is ready and the RPi 5 is working, you can log in via SSH and run the command:

sudo clusterctrl on

From there, the cluster hat LEDs should start blinking, and the RPi Zeros should gradually turn on. It looks a bit like Christmas lights but I like it.

If you only want to turn on some of them, it’s possible with these commands:

sudo clusterctrl on p1 # to turn on only p1
sudo clusterctrl off p2 p3 p4 # to turn off p2, p3 and p4

Network Configuration

In NAT mode, the IP addresses of the Pi Zeros are assigned as follows:

on the RPi 5 side (the controller) we have a bridge br0 with address 172.19.181.254/24
on the Pi Zero side, 172.19.181.x with x ranging from 1 to 4

It’s as simple as that. From the controller, we can try to log in via SSH:

Even though p1 is NATed, the NAT works since we have Internet access:

zwindler@p1:~ $ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=115 time=12.5 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=115 time=17.1 ms

Conclusion

I lost quite a bit of time flashing images until I found the right way to have both the cluster HAT custom ROM and the OS customizations to run headless.

But once I had that, it was relatively trivial to get this little cluster working.

Now, all that’s left is to find what I’m going to run on it ;) I’m opening the bets!

Zwindler's Reflection

GenAI and software development, episode 2: kubectl-debug-pvc, from idea to krew in 2x30 minutes

Previously, on “GenAI and dev”

The incident that started it all

Side note

The meeting prompt

What Opus produced in 30 minutes

The following iterations (~30 minutes)

The cherry on top: publishing on Krew

Minor failure — delegating the demo too

The difference with PodSweeper

What do I think about it?

The project

Flannel and NetworkPolicies: how to add support with Cilium in CNI chaining mode

Flannel, flannel, flannel…

Let’s verify to be sure

Alternatives for adding support

Installing Cilium in CNI chaining mode

Prerequisites

Retrieve flannel’s CNI configuration

Create the chaining ConfigMap

Install Cilium via Helm

Verification

Testing NetworkPolicies

Bonus: Hubble, network observability

How much does it cost in resources?

Conclusion

Podcasts & Lives

Publications

GenAI and software development: lessons learned with PodSweeper

GenAI, is it fantastic?

or rather: GêneAI (sorry that a french play on word I can’t translate)?

The editor

After the context, the project

Don’t leave, you’ll see it’s fun. Really!

The initial process

The good (open)code…

… and the bad (open)code

Where am I at?

Kyverno killed my API Server. Again.

The revenge strikes back

The upgrade that starts well

The root cause: a magnificent vicious circle

Breaking out of the deadlock

The permanent fix: MatchConditions

Conclusion

101 ways to deploy Kubernetes: a brand new UI to explore 118+ solutions

From Google Sheet to a real web application

Why a UI?

Tech stack: Astro + Tailwind

Features

Did you know? At least 18 tools use kubeadm!

A collaborative project

Try it, comment, share

Installing a Cluster HAT with Raspberry Pi 5 and Pi Zero

Introduction, What Is a Cluster HAT?

Problem(s)

Hardware Installation

3D Case

Preparing the Installation

Installing the Raspberry Pi 5 (Our Controller)

The Headless Mode Problem

Installing the 4 Raspberry Pi Zeros (The Nodes)

Boot Up

Network Configuration

Conclusion

Sources