Security on Zwindler's Reflection

nodes/proxy GET: One Kubernetes permission too many

Tue, 19 May 2026 12:00:00 +0200

TL;DR

This is a bit old now, but I still wanted to share a quick write-up on the topic.

Back in January, a cybersecurity researcher reported a Kubernetes flaw that generated quite a buzz. It had been a while since we had a Kube vulnerability that got people talking this much, at least from Denis’s memory (yes, I talk about myself in the third person).

Honestly, it’s pretty wild: the nodes/proxy GET RBAC permission allows any ServiceAccount to execute code inside any Pod in the cluster, without leaving a single trace in the audit logs. That’s unfortunate, especially when you have ServiceAccounts named rook-ceph-system that also happen to have read access to all Secrets in the cluster.

This article details the issue, how to check if you are vulnerable, the fixes to apply, and the preventive measures you can put in place if you can’t patch right away.

The problem: WebSocket + Kubelet = un-audited exec

The vulnerability was documented by Graham Helton in this article. Here is how it works.

The Kubernetes API exposes a nodes/proxy subresource that proxies HTTP requests to each node’s Kubelet. The Kubelet itself exposes an API on port 10250, specifically the /exec endpoint which allows executing commands inside a container.

The issue comes from how the Kubelet handles authorizations for WebSocket connections:

kubectl exec uses a WebSocket connection, whose handshake is an HTTP GET
The Kubelet maps this initial GET to the get RBAC verb
It checks nodes/proxy GET, then authorizes the operation
No secondary check is performed for the CREATE verb normally required for /exec

Result: any ServiceAccount with nodes/proxy GET can execute commands in any Pod in the cluster, including system Pods (etcd, kube-apiserver, etc.).

# Exploitation via websocat
websocat --insecure \
 --header "Authorization: Bearer $TOKEN" \
 --protocol "v4.channel.k8s.io" \
 "wss://$NODE_IP:10250/exec/default/nginx/nginx?output=1&error=1&command=id"

And that’s not all. Commands executed via this method do not generate any Kubernetes audit logs (well, assuming you even collect them 🙈). The access goes directly through the Kubelet, which does not report events back to the API server.

The official Kubernetes status on this: Won’t Fix. It is a “design behavior” (note the quotes), addressed via a feature gate (KEP-2862, see below).

Ouch.

The Audit: Vulnerable ServiceAccounts on our clusters

In January 2026, following the publication of Graham Helton’s article, a lot of people had to urgently audit their clusters. You can either manually audit all your Roles / ClusterRoles, or use a detection script provided by the researcher.

As an example, here are three relatively common components that make great candidates for a juicy privilege escalation:

Component	ClusterRole	ServiceAccounts
OpenTelemetry Collector	`otel-otelcol-k8sobjects`	`opentelemetry-collector-daemonset-collector`, `opentelemetry-collector-deployment-collector`
OpenTelemetry Operator	`otel-operator-resources` / `opentelemetry-operator-manager`	`opentelemetry-operator`
Rook-Ceph	`rook-ceph-global`, `rook-ceph-mgr-cluster`	`rook-ceph-system`, `rook-ceph-mgr`

Note: there are many more. Graham Helton added an “Appendix: Affected Helm Charts” section at the end of his article, referencing AT LEAST 69 affected Helm charts according to him.

The critical case: rook-ceph-system

In the official chart, the rook-ceph-system ServiceAccount combined two particularly dangerous permissions:

nodes/proxy GET - the RCE
secrets GET/LIST/WATCH across the entire cluster

Accessible secrets can include LUKS keys for volume encryption, Ceph admin keyrings, dashboard passwords… this kind of access makes it a prime target for an attacker.

The attack scenario: a compromise of the rook-ceph-operator Pod (via CVE, supply chain, or a malicious image) would allow reading all Ceph secrets, and then executing code in any Pod (including etcd), leading to a full compromise of the cluster and encrypted data.

To manually check if a ServiceAccount is vulnerable:

kubectl auth can-i get nodes --subresource=proxy \
 --as=system:serviceaccount:<namespace>:<serviceaccount>

Examples of fixes to apply

Rook-Ceph: Upstream fix

For Rook-Ceph, the fix came from upstream: PR rook/rook#16979 removed nodes/proxy from ClusterRoles. This fix is included in Rook v1.19.1, so updating the affected clusters was enough.

After updating, checking across all clusters:

kubectl get clusterroles rook-ceph-global -o yaml | grep -A3 nodes/proxy
# -> nothing

OTel / OTel operator

For OpenTelemetry, the situation is potentially more complex. If you are using otel-operator and OtelCollector Custom Resources, you likely have to manage your own RBAC manifests yourself.

Having had to do it myself, it’s quite painful. Depending on your collector type and the receivers you enabled, you need to cross-reference multiple documents on the official OTel and otel-operator websites.

Upstream merged a conditional approach in open-telemetry/opentelemetry-helm-charts#2083 based on the Kubernetes version:

# Upstream approach (opentelemetry-helm-charts#2083)
{{- if semverCompare ">=1.33-0" .Capabilities.KubeVersion.Version }}
 - nodes/pods
{{- else }}
 - nodes/proxy
{{- end }}

Here again, if all your clusters are up to date, you can simply replace nodes/proxy with nodes/pods directly (without the condition).

otel-collector-crb.yaml:

# Before
rules:
 - apiGroups: [""]
 resources:
 - nodes
 - nodes/proxy  # <- RCE risk
 - nodes/spec
 - nodes/stats
 verbs:
 - get

# After
rules:
 - apiGroups: [""]
 resources:
 - nodes
 # nodes/pods replaces nodes/proxy (RCE risk, see [https://grahamhelton.com/blog/nodes-proxy-rce](https://grahamhelton.com/blog/nodes-proxy-rce))
 # Requires K8s >= 1.33 (KEP-2862 fine-grained kubelet authz)
 - nodes/pods
 - nodes/spec
 - nodes/stats
 verbs:
 - get

otel-operator-rbac.yaml:

# Before
 - apiGroups: [""]
 resources:
 - nodes/proxy  # <- RCE risk
 verbs:
 - get

# After
 # nodes/pods replaces nodes/proxy (RCE risk, see [https://grahamhelton.com/blog/nodes-proxy-rce](https://grahamhelton.com/blog/nodes-proxy-rce))
 # Requires K8s >= 1.33 (KEP-2862 fine-grained kubelet authz)
 - apiGroups: [""]
 resources:
 - nodes/pods
 verbs:
 - get

Preventive Measures

KEP-2862: Fine-Grained Kubelet API Authorization

As teased earlier, the real long-term solution is KEP-2862 (Fine-Grained Kubelet API Authorization). It introduces granular subresources (nodes/pods, nodes/metrics, nodes/stats, nodes/log, etc.) allowing precise access without using nodes/proxy.

K8s Version	KEP-2862 Status
1.32	Alpha
1.33	Beta, enabled by default - `nodes/proxy GET` no longer grants access to `/exec`
1.36	GA (locked to enabled)

But this requires going through EVERY chart in use and checking all deployed manifests now and in the future.

CiliumNetworkPolicy: Blocking the Kubelet port

While waiting for the K8s upgrade, or as defense-in-depth, you can block access to port 10250 from the affected pods using NetworkPolicies (or CiliumNetworkPolicy if you use Cilium as your CNI plugin).

Warning: This only applies to components that do not need to access the Kubelet. The OTel collector potentially needs it to gather Kubelet metrics. In that case, you have no choice but to fix the RBAC.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
 name: deny-kubelet-api-access
 namespace: <namespace>
spec:
 endpointSelector:
 matchLabels:
 <app-label>: <value>
 egressDeny:
 - toEntities:
 - host
 - remote-node
 toPorts:
 - ports:
 - port: "10250"
 protocol: TCP

Kyverno: Blocking the creation of new Roles with nodes/proxy

To prevent any regression (remember, we need to protect ourselves in the future), we can add a Kyverno ClusterPolicy that rejects the creation or modification of a ClusterRole or Role containing nodes/proxy.

Luckily, there are ready-to-use examples on the official Kyverno website:

Note : there probably is a hole in the official Kyverno policy, I opened an issue

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: restrict-nodes-proxy
spec:
 validationFailureAction: Audit  # switch to Enforce after validation
 background: true
 rules:
 - name: deny-nodes-proxy-in-clusterroles
 match:
 any:
 - resources:
 kinds:
 - ClusterRole
 - Role
 exclude:
 any:
 - resources:
 names:
 - "system:kubelet-api-admin" # built-in K8s, unmodifiable
 validate:
 message: >
 nodes/proxy grants RCE capability via Kubelet WebSocket exec.
 Use nodes/pods (requires K8s >= 1.33, KEP-2862) instead.
 deny:
 conditions:
 any:
 - key: "nodes/proxy"
 operator: AnyIn
 value: "{{ request.object.rules[].resources[] }}"

Deployment is done in two stages: first in Audit mode to ensure there are no remaining vulnerable manifests (which you should fix before blocking), then in Enforce mode to actually block them.

Monitoring the Audit Log

Even if commands executed via the Kubelet leave no trace, we can monitor SubjectAccessReviews to detect attempts at enumerating nodes/proxy permissions.

The configuration in the Kubernetes audit policy:

# audit-policy.yaml
- level: Request
 verbs: ["create"]
 resources:
 - group: "authorization.k8s.io"
 resources: ["subjectaccessreviews"]

Then a Prometheus/Alertmanager alert on SARs related to nodes/proxy:

# Detect SARs targeting nodes/proxy
increase(
 apiserver_audit_event_total{
 verb="create",
 resource="subjectaccessreviews"
 }[5m]
) > 0

References

Kubernetes RCE via nodes/proxy GET - Graham Helton
Detection Script
KEP-2862 Fine-Grained Kubelet API Authorization
Kubernetes RBAC Good Practices - nodes/proxy
rook/rook#16979 - Fix upstream Rook-Ceph
open-telemetry/opentelemetry-helm-charts#2083 - Fix upstream OTel

Flannel and NetworkPolicies: how to add support with Cilium in CNI chaining mode

Sun, 15 Mar 2026 18:00:00 +0200

Flannel, flannel, flannel…

Flannel is a simple and popular CNI 😢.

It’s the default CNI in k3s, the one half the kubeadm tutorials use, and you’ll find it in quite a few managed offerings too.

OK, it’s simple, it routes packets between pods, it supports VXLAN and WireGuard, it takes 2 minutes to set up. What more could you ask for?

Well, exactly. There’s one thing flannel does not do: NetworkPolicies. (And that’s a big deal).

Flannel is focused on networking. For network policy, other projects such as Calico can be used.

[Edit] Contrary to what I wrote here, flannel actually does support NetworkPolicies and has for at least 2 years, via the reference implementation kube-network-policies from the Kubernetes project. The option isn’t prominently featured in the README and is probably not more recommended for production, but it does exist. See the official documentation.

And the trap is that if you haven’t read that one little line in the README.md, nothing tells you explicitly.

You can perfectly well create NetworkPolicy objects in your cluster, kubectl apply won’t complain, kubectl get netpol will happily list them. Except… they’re not enforced. Traffic still flows. Your deny-all doesn’t deny anything at all.

Let’s verify to be sure

Before solving anything, let’s verify the problem exists. Starting from the prerequisite that we have a Kubernetes cluster with flannel as the CNI and everything is working, we’ll deploy two pods in two different namespaces: a client (curl) and a server (nginx).

Let’s check that the client can reach the server:

kubectl exec -n netpol-test-a client -- \
 curl -s --max-time 5 http://server.netpol-test-b.svc.cluster.local

We get the nginx welcome page. So far, so normal. Now, let’s apply a deny-all NetworkPolicy on the server’s namespace:

# 01-deny-all-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: deny-all-ingress
 namespace: netpol-test-b
spec:
 podSelector: {}
 policyTypes:
 - Ingress

kubectl apply -f 01-deny-all-ingress.yaml

And let’s test again:

kubectl exec -n netpol-test-a client -- \
 curl -s --max-time 5 http://server.netpol-test-b.svc.cluster.local

Result: the nginx page still shows up. The NetworkPolicy was created (kubectl get netpol -n netpol-test-b shows it), but it’s not enforced. Traffic flows as if nothing happened.

This is the expected behavior with flannel (by default). Flannel only does L3 routing (VXLAN or WireGuard overlay). It doesn’t implement a NetworkPolicy controller. The objects exist in etcd, but nobody translates them into filtering rules.

Let’s clean up the policy before moving on:

kubectl delete -f 01-deny-all-ingress.yaml

Alternatives for adding support

For a long time I thought this was just the way it was. I still think it’s not a good choice for any production.

BUT recently, I discovered that it was possible to chain CNIs within the same cluster, and thus have one CNI handling most tasks (here Flannel) and another one taking care of other tasks, such as enforcing NetworkPolicies.

This is actually the principle behind Canal, which I knew by name but had never explored. In fact, it’s nothing more than a manifest that deploys Flannel as the main CNI with Calico for NetworkPolicy enforcement!

Canal was the name of Tigera and CoreOS’s project to integrate Calico and flannel.

Note: the GitHub project was archived in October 2025 but in theory it should still work, if you can find the correct documentation (I couldn’t, the links are broken, but I didn’t really look that hard either).

So you get the idea: we’re going to add a component that will watch NetworkPolicy objects and translate them into effective filtering rules (iptables, eBPF, nftables…), without touching the existing flannel setup. This is called “CNI chaining” or “policy-only” mode.

There are several solutions:

Calico (Canal); Historically, the flannel + Calico combination is called “Canal”, which I just mentioned. But the official Canal manifest bundles its own flannel in the same DaemonSet as calico-node. If your flannel is already installed and managed (by you, by an operator, by a provider…), you probably don’t want to replace it. And the Tigera operator (the “official” Helm method) doesn’t support policy-only deployment on an existing flannel either. In short, it’s doable but requires some effort. Can’t be bothered.

kube-router; kube-router can run in firewall-only mode (--run-firewall=true) and only needs iptables/ipset. This is actually what k3s uses by default for NetworkPolicies. It’s the lightest solution (supposedly ~50 MB of RAM per node). Make sure your kernel has the ip_set module, otherwise it won’t work.

Cilium in CNI chaining mode; This is the solution I chose and that we’ll detail here. Cilium attaches to the veth interfaces created by flannel and adds its eBPF programs for policy enforcement. No dependency on iptables or ipset, and as a bonus we get Hubble for network observability.

Installing Cilium in CNI chaining mode

Prerequisites

A working Kubernetes cluster with flannel
helm v3+
A kernel >= 4.19 (ideally >= 5.10 for all eBPF features)

Retrieve flannel’s CNI configuration

Cilium in chaining mode needs to know the existing CNI configuration. Let’s retrieve it from a node:

kubectl debug node/$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') \
 -it --image=busybox -- cat /host/etc/cni/net.d/10-flannel.conflist

On my cluster, this gives:

{
 "name": "cbr0",
 "cniVersion": "0.3.1",
 "plugins": [
 {
 "type": "flannel",
 "delegate": {
 "hairpinMode": true,
 "isDefaultGateway": true
 }
 },
 {
 "type": "portmap",
 "capabilities": {
 "portMappings": true
 }
 }
 ]
}

Note the name field (here cbr0). We’ll need it.

Create the chaining ConfigMap

We’ll create a ConfigMap that takes the flannel config and adds the cilium-cni plugin in chaining mode:

apiVersion: v1
kind: ConfigMap
metadata:
 name: cni-configuration
 namespace: kube-system
data:
 cni-config: |-
 {
 "name": "cbr0",
 "cniVersion": "0.3.1",
 "plugins": [
 {
 "type": "flannel",
 "delegate": {
 "hairpinMode": true,
 "isDefaultGateway": true
 }
 },
 {
 "type": "portmap",
 "capabilities": {
 "portMappings": true
 }
 },
 {
 "type": "cilium-cni",
 "chaining-mode": "generic-veth"
 }
 ]
 }

Warning: the name field must match your flannel conflist. If yours is called cni0 or something else, adjust accordingly.

Before applying, also verify that your CNI uses veth interfaces (this is the default with flannel, but better safe than sorry). From a node:

ip -d link | grep veth

You should see veth-type interfaces corresponding to your pods, for example:

103: lxcb3901b7f9c02@if102: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
veth addrgenmode eui64 numtxqueues 1 numrxqueues 1

If that’s the case, Cilium’s generic-veth mode will work. Let’s apply:

kubectl apply -f cilium-cni-configmap.yaml

Install Cilium via Helm

helm repo add cilium https://helm.cilium.io/
helm repo update

Here are the values for chaining mode:

# cilium-values.yaml
cni:
 chainingMode: generic-veth
 customConf: true
 configMap: cni-configuration
 install: true

routingMode: native
enableIPv4Masquerade: false
enableIPv6Masquerade: false

hubble:
 enabled: true
 relay:
 enabled: true
 ui:
 enabled: true

The important points:

cni.chainingMode: generic-veth, this is chaining mode, Cilium attaches to existing veth interfaces
cni.customConf: true + cni.configMap, we provide our own CNI config
routingMode: native, flannel handles routing, not Cilium
enableIPv4Masquerade: false, flannel handles masquerading
hubble.enabled: true, network observability, the big bonus of Cilium

helm install cilium cilium/cilium --version 1.19.1 \
 --namespace kube-system \
 -f cilium-values.yaml

Wait for everything to be ready:

kubectl rollout status daemonset/cilium -n kube-system --timeout=120s

Verification

kubectl exec -n kube-system ds/cilium -- cilium status

What we’re looking for:

Kubernetes: Ok 1.35 (v1.35.0) [linux/amd64]
CNI Chaining: generic-veth
Cilium: Ok 1.19.1
Hubble: Ok

The CNI Chaining: generic-veth line confirms that Cilium is running in chaining mode and isn’t replacing flannel.

Important: pods that existed before Cilium was installed are not automatically managed by Cilium. You need to restart them so that Cilium can attach its eBPF programs. Remember to kubectl rollout restart your test workloads (or recreate them).

Testing NetworkPolicies

This is the moment of truth. Let’s re-apply our deny-all:

kubectl apply -f 01-deny-all-ingress.yaml

kubectl exec -n netpol-test-a client -- \
 curl -s --max-time 5 http://server.netpol-test-b.svc.cluster.local

Result: timeout! This time, the NetworkPolicy is properly enforced. Traffic is blocked.

I followed up with the other classic NetworkPolicy scenarios, and everything works:

Selective ingress allow by namespace; by adding a policy that allows traffic from netpol-test-a only, curl works from that namespace but remains blocked from default. Namespace isolation works.
Deny-all egress; by blocking all outgoing traffic from the client, even DNS resolution is blocked (immediate timeout).
Selective egress allow; by allowing only DNS (port 53) and the server (port 80 in the netpol-test-b namespace), curl to the server works but curl http://example.com remains blocked.

In short, ingress, egress, selective by namespace, everything works as expected.

Bonus: Hubble, network observability

This is for me the real advantage of Cilium over the alternatives. Hubble lets you see network flows and policy verdicts in real time:

kubectl exec -n kube-system ds/cilium -- \
 hubble observe --namespace netpol-test-b --last 5

Mar 15 13:20:42.287: netpol-test-a/client:40066 (ID:9745) ->
netpol-test-b/server:80 (ID:22271)
policy-verdict:none ALLOWED (TCP Flags: SYN)

You can see the source pod, destination pod, port, Cilium identity, and policy verdict. When you’re debugging a NetworkPolicy that isn’t behaving as expected, this is incredibly useful.

How much does it cost in resources?

On my cluster (3 nodes), here’s what Cilium consumes right after installation:

Component	Per node	RAM
cilium agent	yes (DaemonSet)	~160 MB
cilium-envoy	yes (DaemonSet)	~22 MB
cilium-operator	no (2 replicas)	~42 MB
hubble-relay	no (1 replica)	~16 MB
hubble-ui	no (1 replica)	~21 MB

That’s roughly 180 MB per node for the agent + envoy. I don’t have a point of comparison with Calico or kube-router, but it seems acceptable to me, and being able to have full visibility into all network flows with Hubble more than justifies the overhead (in my opinion).

Conclusion

If you have no choice and have to work with flannel, and you want to plug the gaping security hole that is the lack of NetworkPolicy enforcement, know that it is possible to chain another CNI to handle it.

Short of having it as CNI for everything, Cilium in CNI chaining mode (generic-veth) is a pretty nice solution to fill this gap. It doesn’t touch the existing flannel setup, it grafts onto it. And as a bonus, you get Hubble for network observability, which is genuinely valuable.