Kubernetes on Zwindler's Reflection

nodes/proxy GET: One Kubernetes permission too many

Tue, 19 May 2026 12:00:00 +0200

TL;DR

This is a bit old now, but I still wanted to share a quick write-up on the topic.

Back in January, a cybersecurity researcher reported a Kubernetes flaw that generated quite a buzz. It had been a while since we had a Kube vulnerability that got people talking this much, at least from Denis’s memory (yes, I talk about myself in the third person).

Honestly, it’s pretty wild: the nodes/proxy GET RBAC permission allows any ServiceAccount to execute code inside any Pod in the cluster, without leaving a single trace in the audit logs. That’s unfortunate, especially when you have ServiceAccounts named rook-ceph-system that also happen to have read access to all Secrets in the cluster.

This article details the issue, how to check if you are vulnerable, the fixes to apply, and the preventive measures you can put in place if you can’t patch right away.

The problem: WebSocket + Kubelet = un-audited exec

The vulnerability was documented by Graham Helton in this article. Here is how it works.

The Kubernetes API exposes a nodes/proxy subresource that proxies HTTP requests to each node’s Kubelet. The Kubelet itself exposes an API on port 10250, specifically the /exec endpoint which allows executing commands inside a container.

The issue comes from how the Kubelet handles authorizations for WebSocket connections:

kubectl exec uses a WebSocket connection, whose handshake is an HTTP GET
The Kubelet maps this initial GET to the get RBAC verb
It checks nodes/proxy GET, then authorizes the operation
No secondary check is performed for the CREATE verb normally required for /exec

Result: any ServiceAccount with nodes/proxy GET can execute commands in any Pod in the cluster, including system Pods (etcd, kube-apiserver, etc.).

# Exploitation via websocat
websocat --insecure \
 --header "Authorization: Bearer $TOKEN" \
 --protocol "v4.channel.k8s.io" \
 "wss://$NODE_IP:10250/exec/default/nginx/nginx?output=1&error=1&command=id"

And that’s not all. Commands executed via this method do not generate any Kubernetes audit logs (well, assuming you even collect them 🙈). The access goes directly through the Kubelet, which does not report events back to the API server.

The official Kubernetes status on this: Won’t Fix. It is a “design behavior” (note the quotes), addressed via a feature gate (KEP-2862, see below).

Ouch.

The Audit: Vulnerable ServiceAccounts on our clusters

In January 2026, following the publication of Graham Helton’s article, a lot of people had to urgently audit their clusters. You can either manually audit all your Roles / ClusterRoles, or use a detection script provided by the researcher.

As an example, here are three relatively common components that make great candidates for a juicy privilege escalation:

Component	ClusterRole	ServiceAccounts
OpenTelemetry Collector	`otel-otelcol-k8sobjects`	`opentelemetry-collector-daemonset-collector`, `opentelemetry-collector-deployment-collector`
OpenTelemetry Operator	`otel-operator-resources` / `opentelemetry-operator-manager`	`opentelemetry-operator`
Rook-Ceph	`rook-ceph-global`, `rook-ceph-mgr-cluster`	`rook-ceph-system`, `rook-ceph-mgr`

Note: there are many more. Graham Helton added an “Appendix: Affected Helm Charts” section at the end of his article, referencing AT LEAST 69 affected Helm charts according to him.

The critical case: rook-ceph-system

In the official chart, the rook-ceph-system ServiceAccount combined two particularly dangerous permissions:

nodes/proxy GET - the RCE
secrets GET/LIST/WATCH across the entire cluster

Accessible secrets can include LUKS keys for volume encryption, Ceph admin keyrings, dashboard passwords… this kind of access makes it a prime target for an attacker.

The attack scenario: a compromise of the rook-ceph-operator Pod (via CVE, supply chain, or a malicious image) would allow reading all Ceph secrets, and then executing code in any Pod (including etcd), leading to a full compromise of the cluster and encrypted data.

To manually check if a ServiceAccount is vulnerable:

kubectl auth can-i get nodes --subresource=proxy \
 --as=system:serviceaccount:<namespace>:<serviceaccount>

Examples of fixes to apply

Rook-Ceph: Upstream fix

For Rook-Ceph, the fix came from upstream: PR rook/rook#16979 removed nodes/proxy from ClusterRoles. This fix is included in Rook v1.19.1, so updating the affected clusters was enough.

After updating, checking across all clusters:

kubectl get clusterroles rook-ceph-global -o yaml | grep -A3 nodes/proxy
# -> nothing

OTel / OTel operator

For OpenTelemetry, the situation is potentially more complex. If you are using otel-operator and OtelCollector Custom Resources, you likely have to manage your own RBAC manifests yourself.

Having had to do it myself, it’s quite painful. Depending on your collector type and the receivers you enabled, you need to cross-reference multiple documents on the official OTel and otel-operator websites.

Upstream merged a conditional approach in open-telemetry/opentelemetry-helm-charts#2083 based on the Kubernetes version:

# Upstream approach (opentelemetry-helm-charts#2083)
{{- if semverCompare ">=1.33-0" .Capabilities.KubeVersion.Version }}
 - nodes/pods
{{- else }}
 - nodes/proxy
{{- end }}

Here again, if all your clusters are up to date, you can simply replace nodes/proxy with nodes/pods directly (without the condition).

otel-collector-crb.yaml:

# Before
rules:
 - apiGroups: [""]
 resources:
 - nodes
 - nodes/proxy  # <- RCE risk
 - nodes/spec
 - nodes/stats
 verbs:
 - get

# After
rules:
 - apiGroups: [""]
 resources:
 - nodes
 # nodes/pods replaces nodes/proxy (RCE risk, see [https://grahamhelton.com/blog/nodes-proxy-rce](https://grahamhelton.com/blog/nodes-proxy-rce))
 # Requires K8s >= 1.33 (KEP-2862 fine-grained kubelet authz)
 - nodes/pods
 - nodes/spec
 - nodes/stats
 verbs:
 - get

otel-operator-rbac.yaml:

# Before
 - apiGroups: [""]
 resources:
 - nodes/proxy  # <- RCE risk
 verbs:
 - get

# After
 # nodes/pods replaces nodes/proxy (RCE risk, see [https://grahamhelton.com/blog/nodes-proxy-rce](https://grahamhelton.com/blog/nodes-proxy-rce))
 # Requires K8s >= 1.33 (KEP-2862 fine-grained kubelet authz)
 - apiGroups: [""]
 resources:
 - nodes/pods
 verbs:
 - get

Preventive Measures

KEP-2862: Fine-Grained Kubelet API Authorization

As teased earlier, the real long-term solution is KEP-2862 (Fine-Grained Kubelet API Authorization). It introduces granular subresources (nodes/pods, nodes/metrics, nodes/stats, nodes/log, etc.) allowing precise access without using nodes/proxy.

K8s Version	KEP-2862 Status
1.32	Alpha
1.33	Beta, enabled by default - `nodes/proxy GET` no longer grants access to `/exec`
1.36	GA (locked to enabled)

But this requires going through EVERY chart in use and checking all deployed manifests now and in the future.

CiliumNetworkPolicy: Blocking the Kubelet port

While waiting for the K8s upgrade, or as defense-in-depth, you can block access to port 10250 from the affected pods using NetworkPolicies (or CiliumNetworkPolicy if you use Cilium as your CNI plugin).

Warning: This only applies to components that do not need to access the Kubelet. The OTel collector potentially needs it to gather Kubelet metrics. In that case, you have no choice but to fix the RBAC.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
 name: deny-kubelet-api-access
 namespace: <namespace>
spec:
 endpointSelector:
 matchLabels:
 <app-label>: <value>
 egressDeny:
 - toEntities:
 - host
 - remote-node
 toPorts:
 - ports:
 - port: "10250"
 protocol: TCP

Kyverno: Blocking the creation of new Roles with nodes/proxy

To prevent any regression (remember, we need to protect ourselves in the future), we can add a Kyverno ClusterPolicy that rejects the creation or modification of a ClusterRole or Role containing nodes/proxy.

Luckily, there are ready-to-use examples on the official Kyverno website:

Note : there probably is a hole in the official Kyverno policy, I opened an issue

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: restrict-nodes-proxy
spec:
 validationFailureAction: Audit  # switch to Enforce after validation
 background: true
 rules:
 - name: deny-nodes-proxy-in-clusterroles
 match:
 any:
 - resources:
 kinds:
 - ClusterRole
 - Role
 exclude:
 any:
 - resources:
 names:
 - "system:kubelet-api-admin" # built-in K8s, unmodifiable
 validate:
 message: >
 nodes/proxy grants RCE capability via Kubelet WebSocket exec.
 Use nodes/pods (requires K8s >= 1.33, KEP-2862) instead.
 deny:
 conditions:
 any:
 - key: "nodes/proxy"
 operator: AnyIn
 value: "{{ request.object.rules[].resources[] }}"

Deployment is done in two stages: first in Audit mode to ensure there are no remaining vulnerable manifests (which you should fix before blocking), then in Enforce mode to actually block them.

Monitoring the Audit Log

Even if commands executed via the Kubelet leave no trace, we can monitor SubjectAccessReviews to detect attempts at enumerating nodes/proxy permissions.

The configuration in the Kubernetes audit policy:

# audit-policy.yaml
- level: Request
 verbs: ["create"]
 resources:
 - group: "authorization.k8s.io"
 resources: ["subjectaccessreviews"]

Then a Prometheus/Alertmanager alert on SARs related to nodes/proxy:

# Detect SARs targeting nodes/proxy
increase(
 apiserver_audit_event_total{
 verb="create",
 resource="subjectaccessreviews"
 }[5m]
) > 0

References

Kubernetes RCE via nodes/proxy GET - Graham Helton
Detection Script
KEP-2862 Fine-Grained Kubelet API Authorization
Kubernetes RBAC Good Practices - nodes/proxy
rook/rook#16979 - Fix upstream Rook-Ceph
open-telemetry/opentelemetry-helm-charts#2083 - Fix upstream OTel

101 ways to deploy Kubernetes: a brand new UI to explore 118+ solutions

Mon, 09 Feb 2026 18:00:00 +0200

From Google Sheet to a real web application

You might remember my previous posts about this project: first a simple Google Sheet with 93 methods, then a GitHub repository with over 100 entries.

Today, I can present the latest iteration of this project: a real web interface to explore all these solutions!

👉 101-ways-to-deploy-kubernetes

Why a UI?

The Markdown table on GitHub was already better than the Google Sheet for collaboration, but barely. Hard to parse, hard to add columns without it becoming a mess (it already was, haha), and above all, incredibly UGLY.

I therefore decided to turn all of this into a modern and intuitive interface, with the help of an LLM.

Tech stack: Astro + Tailwind

For this project, I chose a simple but effective stack:

Astro: a modern framework that generates ultra-fast static sites
Tailwind CSS: for hassle-free responsive design

The result? A lightweight, fairly fast site that works on both desktop and mobile (though the desktop experience remains more comfortable given the amount of data).

Features

Cards for each solution

No more spreadsheet-style table worthy of a backend dev (no, worse, a kube engineer…)! Each tool now has its own animated “card” with:

The project logo
The license type (OSS or proprietary)
The GitHub star count
Direct links to the project and third-party resources (independent blogs, experience reports, tutorials…)

Powerful filters

Looking only for open source solutions? Tools for local development? Management platforms?

Category filters make navigation easy:

Desktop (local development)
Managed (cloud offerings)
Self-hosted (on-premise automation)
Infra As Code
Kubernetes OS (specialized operating systems)
Management Platform
Kubernetes in Kubernetes

You can also filter by status (active, abandoned) or show only open source or production ready solutions.

A search bar

Know what you’re looking for? Just type the name in the search bar to find the solution instantly.

Tags for refinement

Beyond categories, tags help quickly identify underlying technologies (kubeadm, k3s, k0s…).

Did you know? At least 18 tools use kubeadm!

While compiling all this data, I discovered something fascinating: at least 18 tools use kubeadm as a backend to deploy Kubernetes! 🤯

And that’s not even counting the managed Kubernetes offerings from cloud providers!

This is exactly the kind of information you can now visualize instantly thanks to this new interface.

A collaborative project

The project remains 100% open source and collaborative. The data is still stored in the GitHub repository, and the UI is automatically generated from that data (I even added PR previews).

Is a tool or provider missing? Spotted a bug? Feel free to open an issue or submit a Pull Request!

The project now lists 118 solutions (and I know there are certainly more missing), each with:

Up-to-date links
Project status
External references (tutorials, experience reports…)

Head over to zwindler.github.io/101-ways-to-deploy-kubernetes to explore all these solutions!

OK, this is a shameless “call to action” like you see on every social network. Fair enough.

However, I can’t know if this is useful (or not) if you don’t tell me. I can leave it as is (it’s not a big deal, I have plenty of other projects waiting for my spare time) or keep it alive, if you like it / find it useful.

And if you do find this project useful, please:

Star the project on GitHub
Share it with your colleagues in the Cloud Native community
Contribute by adding missing tools or fixing errors

Thanks in advance! 🙏

101 ways to deploy Kubernetes v2

Sun, 04 Jan 2026 10:00:00 +0200

A new version, a new home

A few months ago, I created a spreadsheet cataloging 93 different ways to deploy Kubernetes, hosted on Google Sheets.

Following a remark from a frenchie from the Cloud Native community (Ludovic Piot) who would have liked to contribute one or two missing tools, I took some (well, a lot, actually) time during these holidays to migrate everything to Github: 101-ways-to-deploy-kubernetes

Why on a git repository rather than on a Google Sheet like at the beginning? Because it allows me to easily make this project truly collaborative, especially through PRs (and more importantly their validation or not).

Google Sheets was fine to start with, when it was a personal tool to help me write my book, but for collaboration, it wasn’t viable.

What’s new in this v2?

Easier contributions

The project now has a CONTRIBUTING.md that explains how to participate.

Do you know a solution that’s not listed? Did you spot an error? A broken link? Just submit a Pull Request!

Many new methods (I haven’t counted but probably a dozen)

The project has grown from 93 to over 100 methods! I’ve continued my explorations and added a dozen new solutions, some of which you suggested to me.

License information

I’ve enriched the table with license details for each project. This is particularly important to quickly identify whether a product is truly open source or if it’s a proprietary or commercial license solution.

I had this debate with Ludovic actually, who wanted to understand why I hadn’t listed “MKE (Mirantis Kubernetes Engine)”. I had responded at the time that it was because I couldn’t access a trial version as a “random from the Internet” without going through a salesperson, but the argument wasn’t very solid.

To find a “unique” rule to indicate in the CONTRIBUTING.md, I therefore reintegrated some of the solutions I had “censored”.

While the heart of the cloud native ecosystem is open source, both open source and closed source (proprietary) tools are accepted in this list. What matters is whether the tool helps deploy Kubernetes clusters, not its licensing model.

Numerous external references

I did significant work on the “external references” section. The idea of the repo, beyond just providing a laundry list, is to reference third-party articles showing how to set up these solutions and benefit from the community’s feedback.

With the idea of making the project more international (it will be just as useful to English speakers as to the little frenchies in my community), I prioritized English links for as many solutions as possible on this list. There are still some without them, but it’s progressing.

Important note: beyond the official documentation (which exists very often for the vast majority of projects), I wanted to catalog:

Quality community tutorials
Detailed blog articles
Concrete feedback and experience reports

A project still under CC BY-SA 4.0

The content remains under CC BY-SA 4.0 (Attribution - Share Alike) license.

This means you are free to:

Share: copy and redistribute the content
Adapt: remix, transform, and build upon the material

Under the following conditions:

Attribution: you must credit the project
Share Alike: if you adapt the content, you must distribute it under the same license

How to contribute?

It’s simple:

Fork the 101-ways-to-deploy-kubernetes repository
Add or modify the content (following the table format)
Submit a Pull Request

Read the CONTRIBUTING.md for more details on the process and best practices.

All contributions are welcome: new entries, corrections, enhancements, suggestions…

What’s next?

The project will continue to evolve with new discoveries and your contributions.

I’m also continuing to explore new solutions and enrich existing information. And who knows, maybe we’ll exceed 150 methods by the end of the year? 😄

This is the moment where I play influencer and ask you not to hesitate to:

Star the project on Github if you find it useful ⭐⭐⭐
Contribute by adding your discoveries
Share around you if you find it useful

See you at github.com/zwindler/101-ways-to-deploy-kubernetes!

93 ways to deploy Kubernetes: I've cataloged (almost) all existing methods

Sun, 02 Nov 2025 18:00:00 +0200

A slightly crazy documentary project

When I started writing my book “Kubernetes: 50 solutions for development workstations and production clusters”, I quickly realized a problem: there are an infinite number of ways to deploy Kubernetes.

Well, not literally infinite, but still… many. Too many?

To structure my book and choose which solutions I would cover, I did what any good nerd would do: I created a spreadsheet.

A very large spreadsheet.

A Google Sheet that currently lists 93 different methods for deploying Kubernetes. Since I don’t like to “waste”, I’m sharing it with you today under CC BY-SA 4.0 (Attribution - Share Alike):

93 ways to deploy Kubernetes

What does this spreadsheet contain?

The spreadsheet is structured with several columns to help you navigate this jungle:

Product name (and publisher when interesting)
Product URL: I tried to restrict to open source products (or public managed services) although there are a few exceptions
Solution type (I’ll come back to this)
“Based on”: this is quite funny… many projects are layers on top of kubeadm, k3s or k0s. Not all of them say it openly, and I realized it by trying them or digging in the code

And maybe a bit less interesting for some of you (maybe it will disappear)

Do I talk about it in my book?
Do I talk about it on my blog?

The different tool categories

To structure first my thinking, and then my book, I tried to classify these methods into categories.

Some are quite obvious (a managed offering, you can immediately see what it’s about), others, a bit more personal (and therefore debatable).

Kubernetes on desktop (Local Development)

Tools for developing locally on your machine. We’re talking about Minikube, kind and other Docker Desktop

Infrastructure as Code (IaC)

Tools that allow you to describe Kubernetes deployment via code (opentofu, crossplane, pulumi…)

Kubernetes in Kubernetes

Because why make it simple when you can make it… recursive? 🤯. It’s currently limited to vCluster and k3k.

Specialized OSes

Operating systems designed specifically to run Kubernetes. I’m obviously thinking of Talos Linux, but not only ;-P.

Managed Kubernetes (turnkey cloud offerings)

No need to draw a picture, we immediately think of the EKS / AKS / GKE triplet, but also French solutions (OVHcloud Managed Kubernetes, soon Clever Cloud :smirk:)

Cluster management platforms

Here, it’s a somewhat separate category, which will allow us to manage many Kubernetes clusters and even generate new clusters managed by clusters… Often good Rube Goldberg machines like Gardener or worse Kubermatic Kubernetes Platform. We still have some slightly more fun things like Kamaji.

Automation tools for self-hosted

Solutions that automate deployment on your own machines, like kubeadm, k3s and k0s (the triplet, basis of about 50% of other market solutions).

The revelation: everyone copies from their neighbor

While filling in this spreadsheet, I discovered something funny: a large majority of tools don’t reinvent the wheel.

Many projects are actually layers or wrappers around three basic solutions I just mentioned (kubeadm, k3s and k0s). And sometimes, we also have layers on slightly more confidential solutions.

The information isn’t always available, I sometimes discovered it in a blog post, or even by digging into the solution’s internals.

This is typically the kind of interesting column to really realize that beyond the apparent diversity of these deployment solutions, there’s actually a big standard and a few variations.

I hope to manage to find more similar clues to fill this column even more :).

A living document

This spreadsheet is not static. The number of tools evolves regularly (I discover new ones almost every week).

If you know a method that’s not listed, don’t hesitate to comment on social networks.

Note: I remind you that I primarily target open source solutions or public managed services (I just removed Mirantis Kubernetes Engine for this reason).

So, what do I still have to test?

I’ve already teased on reputable social networks, in the list of things I haven’t tested yet but that could motivate me, there are: