Kyverno on Zwindler's Reflection

nodes/proxy GET: One Kubernetes permission too many

Tue, 19 May 2026 12:00:00 +0200

TL;DR

This is a bit old now, but I still wanted to share a quick write-up on the topic.

Back in January, a cybersecurity researcher reported a Kubernetes flaw that generated quite a buzz. It had been a while since we had a Kube vulnerability that got people talking this much, at least from Denis’s memory (yes, I talk about myself in the third person).

Honestly, it’s pretty wild: the nodes/proxy GET RBAC permission allows any ServiceAccount to execute code inside any Pod in the cluster, without leaving a single trace in the audit logs. That’s unfortunate, especially when you have ServiceAccounts named rook-ceph-system that also happen to have read access to all Secrets in the cluster.

This article details the issue, how to check if you are vulnerable, the fixes to apply, and the preventive measures you can put in place if you can’t patch right away.

The problem: WebSocket + Kubelet = un-audited exec

The vulnerability was documented by Graham Helton in this article. Here is how it works.

The Kubernetes API exposes a nodes/proxy subresource that proxies HTTP requests to each node’s Kubelet. The Kubelet itself exposes an API on port 10250, specifically the /exec endpoint which allows executing commands inside a container.

The issue comes from how the Kubelet handles authorizations for WebSocket connections:

kubectl exec uses a WebSocket connection, whose handshake is an HTTP GET
The Kubelet maps this initial GET to the get RBAC verb
It checks nodes/proxy GET, then authorizes the operation
No secondary check is performed for the CREATE verb normally required for /exec

Result: any ServiceAccount with nodes/proxy GET can execute commands in any Pod in the cluster, including system Pods (etcd, kube-apiserver, etc.).

# Exploitation via websocat
websocat --insecure \
 --header "Authorization: Bearer $TOKEN" \
 --protocol "v4.channel.k8s.io" \
 "wss://$NODE_IP:10250/exec/default/nginx/nginx?output=1&error=1&command=id"

And that’s not all. Commands executed via this method do not generate any Kubernetes audit logs (well, assuming you even collect them 🙈). The access goes directly through the Kubelet, which does not report events back to the API server.

The official Kubernetes status on this: Won’t Fix. It is a “design behavior” (note the quotes), addressed via a feature gate (KEP-2862, see below).

Ouch.

The Audit: Vulnerable ServiceAccounts on our clusters

In January 2026, following the publication of Graham Helton’s article, a lot of people had to urgently audit their clusters. You can either manually audit all your Roles / ClusterRoles, or use a detection script provided by the researcher.

As an example, here are three relatively common components that make great candidates for a juicy privilege escalation:

Component	ClusterRole	ServiceAccounts
OpenTelemetry Collector	`otel-otelcol-k8sobjects`	`opentelemetry-collector-daemonset-collector`, `opentelemetry-collector-deployment-collector`
OpenTelemetry Operator	`otel-operator-resources` / `opentelemetry-operator-manager`	`opentelemetry-operator`
Rook-Ceph	`rook-ceph-global`, `rook-ceph-mgr-cluster`	`rook-ceph-system`, `rook-ceph-mgr`

Note: there are many more. Graham Helton added an “Appendix: Affected Helm Charts” section at the end of his article, referencing AT LEAST 69 affected Helm charts according to him.

The critical case: rook-ceph-system

In the official chart, the rook-ceph-system ServiceAccount combined two particularly dangerous permissions:

nodes/proxy GET - the RCE
secrets GET/LIST/WATCH across the entire cluster

Accessible secrets can include LUKS keys for volume encryption, Ceph admin keyrings, dashboard passwords… this kind of access makes it a prime target for an attacker.

The attack scenario: a compromise of the rook-ceph-operator Pod (via CVE, supply chain, or a malicious image) would allow reading all Ceph secrets, and then executing code in any Pod (including etcd), leading to a full compromise of the cluster and encrypted data.

To manually check if a ServiceAccount is vulnerable:

kubectl auth can-i get nodes --subresource=proxy \
 --as=system:serviceaccount:<namespace>:<serviceaccount>

Examples of fixes to apply

Rook-Ceph: Upstream fix

For Rook-Ceph, the fix came from upstream: PR rook/rook#16979 removed nodes/proxy from ClusterRoles. This fix is included in Rook v1.19.1, so updating the affected clusters was enough.

After updating, checking across all clusters:

kubectl get clusterroles rook-ceph-global -o yaml | grep -A3 nodes/proxy
# -> nothing

OTel / OTel operator

For OpenTelemetry, the situation is potentially more complex. If you are using otel-operator and OtelCollector Custom Resources, you likely have to manage your own RBAC manifests yourself.

Having had to do it myself, it’s quite painful. Depending on your collector type and the receivers you enabled, you need to cross-reference multiple documents on the official OTel and otel-operator websites.

Upstream merged a conditional approach in open-telemetry/opentelemetry-helm-charts#2083 based on the Kubernetes version:

# Upstream approach (opentelemetry-helm-charts#2083)
{{- if semverCompare ">=1.33-0" .Capabilities.KubeVersion.Version }}
 - nodes/pods
{{- else }}
 - nodes/proxy
{{- end }}

Here again, if all your clusters are up to date, you can simply replace nodes/proxy with nodes/pods directly (without the condition).

otel-collector-crb.yaml:

# Before
rules:
 - apiGroups: [""]
 resources:
 - nodes
 - nodes/proxy  # <- RCE risk
 - nodes/spec
 - nodes/stats
 verbs:
 - get

# After
rules:
 - apiGroups: [""]
 resources:
 - nodes
 # nodes/pods replaces nodes/proxy (RCE risk, see [https://grahamhelton.com/blog/nodes-proxy-rce](https://grahamhelton.com/blog/nodes-proxy-rce))
 # Requires K8s >= 1.33 (KEP-2862 fine-grained kubelet authz)
 - nodes/pods
 - nodes/spec
 - nodes/stats
 verbs:
 - get

otel-operator-rbac.yaml:

# Before
 - apiGroups: [""]
 resources:
 - nodes/proxy  # <- RCE risk
 verbs:
 - get

# After
 # nodes/pods replaces nodes/proxy (RCE risk, see [https://grahamhelton.com/blog/nodes-proxy-rce](https://grahamhelton.com/blog/nodes-proxy-rce))
 # Requires K8s >= 1.33 (KEP-2862 fine-grained kubelet authz)
 - apiGroups: [""]
 resources:
 - nodes/pods
 verbs:
 - get

Preventive Measures

KEP-2862: Fine-Grained Kubelet API Authorization

As teased earlier, the real long-term solution is KEP-2862 (Fine-Grained Kubelet API Authorization). It introduces granular subresources (nodes/pods, nodes/metrics, nodes/stats, nodes/log, etc.) allowing precise access without using nodes/proxy.

K8s Version	KEP-2862 Status
1.32	Alpha
1.33	Beta, enabled by default - `nodes/proxy GET` no longer grants access to `/exec`
1.36	GA (locked to enabled)

But this requires going through EVERY chart in use and checking all deployed manifests now and in the future.

CiliumNetworkPolicy: Blocking the Kubelet port

While waiting for the K8s upgrade, or as defense-in-depth, you can block access to port 10250 from the affected pods using NetworkPolicies (or CiliumNetworkPolicy if you use Cilium as your CNI plugin).

Warning: This only applies to components that do not need to access the Kubelet. The OTel collector potentially needs it to gather Kubelet metrics. In that case, you have no choice but to fix the RBAC.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
 name: deny-kubelet-api-access
 namespace: <namespace>
spec:
 endpointSelector:
 matchLabels:
 <app-label>: <value>
 egressDeny:
 - toEntities:
 - host
 - remote-node
 toPorts:
 - ports:
 - port: "10250"
 protocol: TCP

Kyverno: Blocking the creation of new Roles with nodes/proxy

To prevent any regression (remember, we need to protect ourselves in the future), we can add a Kyverno ClusterPolicy that rejects the creation or modification of a ClusterRole or Role containing nodes/proxy.

Luckily, there are ready-to-use examples on the official Kyverno website:

Note : there probably is a hole in the official Kyverno policy, I opened an issue

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: restrict-nodes-proxy
spec:
 validationFailureAction: Audit  # switch to Enforce after validation
 background: true
 rules:
 - name: deny-nodes-proxy-in-clusterroles
 match:
 any:
 - resources:
 kinds:
 - ClusterRole
 - Role
 exclude:
 any:
 - resources:
 names:
 - "system:kubelet-api-admin" # built-in K8s, unmodifiable
 validate:
 message: >
 nodes/proxy grants RCE capability via Kubelet WebSocket exec.
 Use nodes/pods (requires K8s >= 1.33, KEP-2862) instead.
 deny:
 conditions:
 any:
 - key: "nodes/proxy"
 operator: AnyIn
 value: "{{ request.object.rules[].resources[] }}"

Deployment is done in two stages: first in Audit mode to ensure there are no remaining vulnerable manifests (which you should fix before blocking), then in Enforce mode to actually block them.

Monitoring the Audit Log

Even if commands executed via the Kubelet leave no trace, we can monitor SubjectAccessReviews to detect attempts at enumerating nodes/proxy permissions.

The configuration in the Kubernetes audit policy:

# audit-policy.yaml
- level: Request
 verbs: ["create"]
 resources:
 - group: "authorization.k8s.io"
 resources: ["subjectaccessreviews"]

Then a Prometheus/Alertmanager alert on SARs related to nodes/proxy:

# Detect SARs targeting nodes/proxy
increase(
 apiserver_audit_event_total{
 verb="create",
 resource="subjectaccessreviews"
 }[5m]
) > 0

References

Kubernetes RCE via nodes/proxy GET - Graham Helton
Detection Script
KEP-2862 Fine-Grained Kubelet API Authorization
Kubernetes RBAC Good Practices - nodes/proxy
rook/rook#16979 - Fix upstream Rook-Ceph
open-telemetry/opentelemetry-helm-charts#2083 - Fix upstream OTel

Kyverno killed my API Server. Again.

Thu, 26 Feb 2026 08:00:00 +0200

The revenge strikes back

You may have read my previous article about etcd 3 years ago. If you remember correctly, the crash was etcd, but the real culprit was Kyverno. I love Kyverno. It’s really a piece of software I’m very fond of. Mainly because it’s incredibly powerful. I even wrote an introductory article and a second one that goes deeper on the topic (both in french, though).

But the sheer number of incidents and weird side effects it causes. Mamamia… This isn’t the first incident of the year I’ve had with Kyverno (yes, we’re in February) but since this one is entertaining, I’m sharing it with you.

During a routine maintenance operation to upgrade a Kubernetes cluster to version 1.34 (from 1.32), we ended up facing the dreaded scenario for any kube admin: a completely unreachable API Server after restarting the Control Plane nodes.

What initially looked like a typical network error turned out to be a subtle deadlock between new native Kubernetes networking features and our dear Kyverno 😘.

Spoiler: it wasn’t a network issue. It’s never a network issue. Well, sometimes it is. But not this time.

The upgrade that starts well

Alright, a Kubernetes upgrade has become pretty routine at this point. We do it regularly, we have our procedures, we’re pros (I swear). We jump from 1.32 to 1.34 in a single commit, skipping the hop through 1.33.

YOLO.

In the technical context I’m talking about, everything is managed as code. From machine provisioning all the way to Talos deployment, including MachineConfigs (the CustomResources to modify… well, the machine).

For more details, see the Talos documentation on Machine Configs.

The first cluster we test has only one control plane (don’t ask me why, it probably wouldn’t have changed anything). Talos restarts the API Server with the new version and then… nothing.

The “weird” API Server logs (technical term) speak for themselves:

I0224 15:32:50.979280 1 default_servicecidr_controller.go:166] Creating default ServiceCIDR with CIDRs: [10.1.0.0/20]
W0224 15:32:50.984784 1 dispatcher.go:225] rejected by webhook "validate.kyverno.svc-fail":
admission webhook "validate.kyverno.svc-fail" denied the request:
Get "https://10.1.0.1:443/api": dial tcp 10.1.0.1:443: connect: operation not permitted
I0224 15:32:50.985342 1 event.go:389] "Event occurred" kind="ServiceCIDR"
apiVersion="networking.k8s.io/v1" type="Warning"
reason="KubernetesDefaultServiceCIDRError"
message="The default ServiceCIDR can not be created"

😬😬😬

The root cause: a magnificent vicious circle

After investigation, we discovered that the incident was the result of a collision between a Kubernetes core evolution and our Kyverno configuration. A textbook deadlock case.

Let’s break down the mechanism:

The new ServiceCIDR Kind

In recent versions (v1.33+), Kubernetes migrates service IP range management to dedicated objects named ServiceCIDR. On the first boot after the upgrade, the API Server automatically tries to create the default object (e.g., 10.1.0.0/20).

For the curious, KEP-1880 and the official ServiceCIDR documentation detail this evolution.

It’s new, it’s clean, it’s well designed. Except that…

Interception by the Kyverno Webhook

Kyverno, configured with failurePolicy: Fail (because we’re serious people who don’t let just anything through in prod), is set up to intercept resource creations to validate them, and fail the call if Kyverno doesn’t respond.

Including the ServiceCIDR freshly created by the API Server itself.

Deadlock

And this is where it gets beautiful:

The API Server pauses the ServiceCIDR creation waiting for Kyverno’s “OK”
To contact the Kyverno service, the API Server needs to route the request through the Kubernetes service IP (typically 10.1.0.1)
But the network layer (service routing) can’t initialize until the ServiceCIDR object is validated and created

It’s the chicken and the egg, “I locked my keys inside the car” edition.

PTSD. Yes, that actually happened to me. In the desert. With no cell service.

Profit.

The API Server times out or returns a connect: operation not permitted error when trying to reach the webhook, blocking its own initialization. CrashLoopBackOff on the API Server. :D

Breaking out of the deadlock

To escape this deadlock, you need to temporarily bypass the admission layer. Easy, right?

The “usual” workaround: useless

Deadlocks with Kyverno, we’re used to them at this point. Normally, since kube-system is ignored, you can simply connect with a break-glass kubeconfig (we normally use OIDC) that has the cluster-admin cluster role and delete the Kyverno validating webhooks:

kubectl delete validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg

Except here, the API Server won’t even start. My kubectl isn’t going to work, obviously!

The real workaround: disable webhooks at boot

The solution we chose was to modify the API Server configuration to temporarily disable validation webhooks at startup. My esteemed colleague Maxime hot-edited the machine config (using break-glass talosctl access) to add the following flag directly in the API server’s extraArgs:

--disable-admission-plugins=ValidatingAdmissionWebhook

For those unfamiliar with admission control in Kubernetes, just know that there’s a list of “default” plugins but everything can be toggled off. I might do a deep dive on Kubernetes admission control someday, it’s fascinating ;).

With this flag, the API Server can finally create its ServiceCIDR objects without asking anyone for permission (completely bypassing all validation mechanisms that Kyverno or similar tools enforce), the network initializes, Kyverno starts, and then you can remove the flag and restart cleanly.

The “funny” option we didn’t try

Personally, I thought it would be hilarious to go directly into the etcd database and delete the webhook key causing the issue (also through talosctl):

# Example via etcdctl
etcdctl del /registry/admissionregistration.k8s.io/validatingwebhookconfigurations/kyverno-resource-validating-webhook-cfg

My colleagues were less enthusiastic: “Yeah but you know, if we break etcd it’s gonna be painful”. We played it safe with the flag. I’m deeply disappointed we didn’t try 😂.

The permanent fix: MatchConditions

OK, now that the cluster is back up, how do we make sure this doesn’t happen again on the next upgrade?

The clean solution is to use matchConditions (introduced in Kubernetes 1.27) on the ValidatingWebhookConfiguration. This allows you to exclude critical network bootstrap resources before the request even attempts to leave the API Server toward the Kyverno pod.

See the official documentation on matchConditions.

We were already using this option to throttle the sometimes excessive Kyverno traffic (if you manage Kyverno, you know what I’m talking about) on a number of events (we’d overwhelm the API server or Kyverno, in CPU or RAM, depending on the case). We just had to add exclusions for the new types:

# Exclude network bootstrap resources to prevent the deadlock
matchConditions:
 - name: 'exclude-ServiceCIDR'
 expression: '!(request.kind.kind == "ServiceCIDR")'
 - name: 'exclude-IPAddress'
 expression: '!(request.kind.kind == "IPAddress")'

With this, when the API Server creates a ServiceCIDR at boot, the request no longer goes through the Kyverno webhook. No circular dependency, no deadlock, everyone’s happy.

Conclusion

As the current French president would say about something that was painfully predictable:

who could have predicted this?

OK fine, all we had to do was read the Kubernetes 1.33 release notes. That said, we have a staging cluster, that’s what it’s for. We broke staging, no big deal.

Context: this is the “Big deal” TV Game Mascot

Maybe we’ll actually read them next time?

Kubernetes Compliance Policies with Kyverno - part 2

Mon, 05 Sep 2022 06:00:00 +0200

Part 1 recap

In the previous article, I introduced Kyverno, a “cloud native” tool for managing compliance policies on Kubernetes.

To illustrate the tool’s capabilities, I used an example: preventing two Ingresses from using the same entry URL. I had done the same exercise with OPA in an earlier article (Open Policy Agent (OPA) and its Kubernetes counterpart, Gatekeeper) and compared the two syntaxes (Kyverno vs rego) at the end.

I have a clear preference for Kyverno’s syntax, even though OPA has been around longer and isn’t limited to Kubernetes (Kyverno is Kube-only).

Going a bit further

Beyond my example, I mentioned that Kyverno provides a very large number of pre-written policies (191 at the time of writing).

kyverno.io/policies

By default, most available policies are set to “audit” mode initially. Don’t forget to switch them to “enforce” mode once your cluster is mature enough.

Obviously, when you’re just getting started, you don’t really know where to begin. The simplest approach is to deploy the kyverno/kyverno-policies Helm chart, which contains the Kyverno equivalent of the “Pod Security Standards” (PSPs being officially deprecated since 1.21 I think, and no longer usable in 1.25).

helm install kyverno-policies kyverno/kyverno-policies -n kyverno
[...]
Thank you for installing kyverno-policies v2.5.2 😀

We have installed the "baseline" profile of Pod Security Standards and set them in audit mode.

Visit https://kyverno.io/policies/ to find more sample policies.

Here’s what was deployed (the last one was already there from the previous article):

kubectl get clusterpolicy
NAME BACKGROUND ACTION READY
disallow-capabilities true audit true
disallow-host-namespaces true audit true
disallow-host-path true audit true
disallow-host-ports true audit true
disallow-host-process true audit true
disallow-privileged-containers true audit true
disallow-proc-mount true audit true
disallow-selinux true audit true
restrict-apparmor-profiles true audit true
restrict-seccomp true audit true
restrict-sysctls true audit true
unique-ingress-host false enforce true

With that, we already have plenty to work with to educate our users (developers) on producing secure Kubernetes deployments.

User experience

One of the annoying things about both tools is that out of the box, they’re not very user friendly. Personally, running a kubectl get events command (or kubectl get events --field-selector=reason=PolicyViolation in Kyverno’s case) to figure out why a deployment failed doesn’t bother me much (I almost enjoy it).

But for some people, it’s a bit off-putting, and I can understand that.

My goal is to make infrastructure simpler for people whose job it isn’t, and imposing my way of working isn’t necessarily the best approach.

Often, to help adopt a new tool or practice, the simplest thing is to provide a graphical interface.

And it turns out Kyverno has a tool for that.

Kyverno policy reporter

In reality, the tool wasn’t originally designed for that exact purpose. But you’ll see it’s actually even better.

Historically, it’s called policy reporter because the developers needed a tool to easily and quickly export the PolicyReports I mentioned in the previous article:

kubectl describe policyreport polr-ns-default
Name: polr-ns-default
Namespace: default
Labels: managed-by=kyverno
Annotations: <none>
API Version: wgpolicyk8s.io/v1alpha2
Kind: PolicyReport
[...]
Results:
 Category: Sample
 Message: The Ingress host name must be unique.
 Policy: unique-ingress-host
 Resources:
 API Version: networking.k8s.io/v1
 Kind: Ingress
 Name: ingress2
[...]
Summary:
 Error: 0
 Fail: 2
 Pass: 0
 Skip: 2
 Warn: 0

The policy reporter’s role is to export PolicyReports to several external sources for easier processing:

Grafana Loki
Elasticsearch
Slack
Discord
MS Teams
Policy Reporter UI
S3

I’ll cover 2 of them: Slack and Policy Reporter UI

Policy Reporter UI

I’ll start with the UI since it’s the simplest to set up. You just need to deploy the Policy Reporter Helm chart, making sure to specify that you want the UI:

helm repo add policy-reporter https://kyverno.github.io/policy-reporter
helm repo update
helm install -n kyverno policy-reporter policy-reporter/policy-reporter --set kyvernoPlugin.enabled=true --set ui.enabled=true --set ui.plugins.kyverno=true

kubectl -n kyverno get pods
NAME READY STATUS RESTARTS AGE
kyverno-5f8bfd6fc5-xp6bm 1/1 Running 1 (59m ago) 33d
policy-reporter-74cc9bf4b9-9ggwh 1/1 Running 0 18s
policy-reporter-kyverno-plugin-6b48c4bc5f-5rwfb 1/1 Running 0 18s
policy-reporter-ui-5c9449d58-f9fwf 1/1 Running 0 18s

We could obviously add an Ingress, but if you’re lazy like me during this demo, a simple port-forward will do.

kubectl port-forward service/policy-reporter-ui 8082:8080 -n kyverno

The UI itself is fairly intuitive. It includes notably:

a Dashboard with a big global indicator showing how many policy violations you have
a page with your “policy reports”, filterable by namespace or policy
the list of your policies
…

Slack

You can immediately see the value of alerting administrators in Slack (or Teams…) when a policy violation occurs.

The configuration is fairly trivial — you just need to add a webhook in Slack and a small piece of configuration in Kyverno:

kyvernoPlugin:
 enabled: true
ui:
 enabled: true
 plugins:
 kyverno: true
target:
 slack:
 webhook: "https://hooks.slack.com/services/T0xxxxxxxx"
 minimumPriority: "medium"
 skipExistingOnStartup: true
 sources:
 - kyverno

And we re-apply all this to our Helm release:

helm upgrade -n kyverno policy-reporter policy-reporter/policy-reporter -f kyverno-values.yaml

Note: If you don’t know how to do this, you need to be admin of the Slack workspace, create an application at https://api.slack.com/apps, enable Incoming Webhooks, and create one with permissions on a channel.

How about we trigger a policy violation, just to see?

To keep it simple, I went back to the example from the previous article. I switch my unique-ingress-host policy back to audit mode and recreate the problematic ingress:

kubectl edit clusterpolicy unique-ingress-host
kubectl apply -f ingress2.yaml

Instantly, 2 failures show up in the Kyverno UI, along with messages in Slack:

Victory :)

Additional Resources

Kubernetes Compliance Policies with Kyverno

Mon, 01 Aug 2022 06:00:00 +0000

Introduction

If the title of this article sounds familiar, it’s probably because I already used it 2 years ago when talking about another compliance policy management tool: Open Policy Agent (OPA) and its Kubernetes counterpart, Gatekeeper.

And I won’t lie, I haven’t done much OPA since that 2020 article…

The first reason is that I changed companies right at that time and had other priorities at my new employer.

The second reason, probably more accurate but perhaps less flattering, is that OPA is not exactly user friendly. Especially because of rego, the compliance policy language.

Since then, I discovered Kyverno, a newcomer in the “cloud native” game (recently promoted to “CNCF sandbox” level) that offers compliance policies for Kubernetes only (unlike OPA + rego which are general-purpose), written in YAML.

With Kyverno, policies are managed as Kubernetes resources and no new language is required to write policies.

Take that, rego ;-)

Note: This article will remain relatively simple, it’s an introduction. We’ll go further in subsequent articles (writing your own policy, the UI, etc).

I’m not going to innovate much compared to the OPA article and will start from the same use case: preventing two developers from using the same URL for their web app deployed on Kubernetes (because we’ve seen it’s pretty bad when that happens).

And you’ll see that the initial learning curve is MUUUCH more accessible with Kyverno than with OPA (even though it might annoy half of my Twitter followers that I’m once again talking about YAML).

Prerequisites

Same setup as for OPA / Gatekeeper, you need Kube 1.14 to use Kyverno. Version 1.14 wasn’t the latest back in 2020, it’s practically prehistoric by now ;-).

Your Kubernetes cluster version must be above v1.14 which adds webhook timeouts.

Obviously, you also need admin access to the cluster, because we’ll have to create Validating Admission Webhooks and CRDs. Again, I’ll refer you to the previous article and the official documentation if this doesn’t ring a bell.

In short, we’re going to insert ourselves into Kubernetes’ deployment process to prevent people from doing things that don’t follow your best practices (with webhooks) and we’ll add extra APIs to extend Kubernetes (the CRDs), so we can describe what’s not allowed.

Deployment

Installation-wise, we’re keeping it simple and efficient.

Kyverno offers several Helm Charts:

one for Kyverno itself
one for Policy Reporter
one containing a “pack” of default rules, configured as “non-blocking” (basically the equivalent of what you could do with Pod Security Policies)

For this article, I’m only interested in Kyverno itself. But as I mentioned above, we’ll discuss the other two in a future post.

Let’s start by adding Kyverno itself:

$ helm repo add kyverno https://kyverno.github.io/kyverno/
$ helm repo update
$ helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace
 NAME: kyverno
 LAST DEPLOYED: Mon Jul 25 21:55:11 2022
 NAMESPACE: kyverno
 STATUS: deployed
 REVISION: 1
 NOTES:
 Chart version: v2.5.2
 Kyverno version: v1.7.2

 Thank you for installing kyverno! Your release is named kyverno.
 ⚠️ WARNING: Setting replicas count below 3 means Kyverno is not running in high availability mode.

 💡 Note: There is a trade-off when deciding which approach to take regarding Namespace exclusions. Please see the documentation at https://kyverno.io/docs/installation/#security-vs-operability to understand the risks.

Note: you’ll notice the WARNING during install. By default, Kyverno is deployed with a single replica. If it crashes, you can no longer deploy or modify anything on your cluster (it happened to me). The solution to recover is to delete Kyverno’s webhooks to unblock the situation. And of course, give it more room (higher limits, especially for RAM) and definitely 3 replicas (or more)…

Note 2: It’s probably best to exclude certain namespaces (kube-system, kyverno, …) from Kyverno’s scope to avoid nasty surprises. Thibault Lengagne’s blogpost at Padok also discusses this (link at the bottom of the article).

What did we deploy?

At first glance, not much. If you run kubectl -n kyverno get all, you’ll find just a Pod (+ReplicaSet +Deployment) and two Services.

But behind the scenes, we’ve deployed much more than that ;)

Looking for CRDs, you’ll see we have quite a few new objects at our disposal.

$ kubectl api-resources --api-group=kyverno.io
 NAME SHORTNAMES APIVERSION NAMESPACED KIND
 clusterpolicies cpol kyverno.io/v1 false ClusterPolicy
 clusterreportchangerequests crcr kyverno.io/v1alpha2 false ClusterReportChangeRequest
 generaterequests gr kyverno.io/v1 true GenerateRequest
 policies pol kyverno.io/v1 true Policy
 reportchangerequests rcr kyverno.io/v1alpha2 true ReportChangeRequest
 updaterequests ur kyverno.io/v1beta1 true UpdateRequest

Similarly, we also have new webhooks:

$ kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io
 NAME WEBHOOKS AGE
 kyverno-policy-validating-webhook-cfg 1 8m48s
 kyverno-resource-validating-webhook-cfg 2 8m48s

$ kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io
 NAME WEBHOOKS AGE
 kyverno-policy-mutating-webhook-cfg 1 11m
 kyverno-resource-mutating-webhook-cfg 2 11m
 kyverno-verify-mutating-webhook-cfg 1 11m

What’s it for? How does it work?

To simplify, we’ll describe what we don’t want to allow on our cluster using the ClusterPolicy CRD.

Based on the rules we write, the webhooks will insert themselves into Kubernetes’ normal resource deployment process to either allow or reject them.

Kubernetes Blog - A Guide to Kubernetes Admission Controllers

Ingress with the same URL example

Enough theory, the easiest way is to show you. As I mentioned at the beginning of the article, to compare the two products, I’m going to start from the same use case.

In my Kubernetes clusters, I want to prevent two applications from creating an Ingress with the same URL, because in my case, that’s not allowed. It’s a sign of a copy/paste error from one Helm Chart to another (or any other human error).

The consequence of this error is a conflict in routing incoming HTTP requests, and you end up with some kind of janky round robin where half the requests hit the wrong backend. In production, that’s a bad look, and you risk ending up with customers complaining on Twitter that the application is down ;-).

Just as OPA provides a community policy to address this problem, Kyverno has the same.

We’ll look in more detail in the next article at how these rules are built. For now, we’ll just deploy this rule in “audit” mode and see if it works.

spec: validationFailureAction: audit

$ kubectl apply -f https://raw.githubusercontent.com/kyverno/policies/main/other/unique-ingress-paths/unique-ingress-paths.yaml
 clusterpolicy.kyverno.io/unique-ingress-host created

If you have a Kubernetes cluster at hand, I suggest you apply these two Ingresses, configured to receive traffic from the same URL (toto.example.org):

$ for counter in 1 2; do
cat > ingress${counter}.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: ingress${counter}
spec:
 rules:
 - host: toto.example.org
 http:
 paths:
 - path: /
 pathType: Prefix
 backend:
 service:
 name: service${counter}
 port:
 number: 4200
EOF
done

$ kubectl apply -f ingress1.yaml
 ingress.networking.k8s.io/ingress1 created
$ kubectl apply -f ingress2.yaml
 ingress.networking.k8s.io/ingress2 created

Normally, nothing happens. Kyverno doesn’t prevent you from creating these two Ingresses since we’re in “validationFailureAction: audit” mode (even if the backends don’t actually exist behind them, that doesn’t matter).

$ kubectl get ingress
 NAME CLASS HOSTS ADDRESS PORTS AGE
 ingress1 <none> toto.example.org 80 90s
 ingress2 <none> toto.example.org 80 87s

However, if you check the Events, you’ll find PolicyViolation entries:

$ kubectl get events --field-selector=reason=PolicyViolation
LAST SEEN TYPE REASON OBJECT MESSAGE
118s Warning PolicyViolation ingress/ingress1 policy unique-ingress-host/check-single-host fail: The Ingress host name must be unique.
118s Warning PolicyViolation ingress/ingress1 policy unique-ingress-host/check-single-host fail: The Ingress host name must be unique.
118s Warning PolicyViolation clusterpolicy/unique-ingress-host Ingress default/ingress1: [check-single-host] fail
115s Warning PolicyViolation clusterpolicy/unique-ingress-host Ingress default/ingress2: [check-single-host] fail

The fact that we have 2 Ingresses pointing to the same URL is properly detected and shows up in Kubernetes Events. Nice :)

In a much more verbose format, you can also retrieve PolicyReports, which say the same thing in an even less readable way.

kubectl describe policyreport polr-ns-default
Name: polr-ns-default
Namespace: default
Labels: managed-by=kyverno
Annotations: <none>
API Version: wgpolicyk8s.io/v1alpha2
Kind: PolicyReport
[...]
Results:
 Category: Sample
 Message: The Ingress host name must be unique.
 Policy: unique-ingress-host
 Resources:
 API Version: networking.k8s.io/v1
 Kind: Ingress
 Name: ingress2
[...]
Summary:
 Error: 0
 Fail: 2
 Pass: 0
 Skip: 2
 Warn: 0

Let’s clean up for what’s next:

$ kubectl delete ingress ingress1 ingress2

“The law is me”

Imagine you’ve now thoroughly tested your policies, your developers no longer generate manifests that don’t follow your best practices (e.g., no root containers, mandatory labels present, correct limits/requests, etc). Your cluster is clean!

You can switch your compliance policies to “enforcing” mode. Unlike “audit” mode, this mode is blocking, and manifests that don’t comply with the policies will be rejected by Kubernetes.

$ kubectl patch clusterpolicies.kyverno.io unique-ingress-host --patch '{"spec":{"validationFailureAction":"enforce"}}' --type merge

Yes, I’m using “patch” instead of “edit” to show off ;-)

And let’s test again:

$ kubectl apply -f ingress1.yaml
 ingress.networking.k8s.io/ingress1 created

So far, so good

$ kubectl apply -f ingress2.yaml
Error from server: error when creating "ingress2.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Ingress/default/ingress2.yaml was blocked due to the following policies

unique-ingress-host:
 check-single-host: The Ingress host name must be unique.

AND BAM!

Conclusion

In this basic (but genuinely useful) example, we can just as easily prevent users from creating Ingresses with the same URL as we could with OPA.

So one might ask “why prefer Kyverno?”. Beyond some other nice features we’ll explore in future posts, my answer fits in a single image.

My (very personal) take is that rego is unreadable and verbose, and therefore unmaintainable. Again, this is personal, but I’ve never managed to get into it.

Without going as far as idealizing Kyverno’s language*, I find it is clearly much simpler to write, but more importantly, to read. If you ever need to modify/write/maintain your own policies, this will be very handy.

(*Yes, there is a language, especially for pattern matching… and as an annoying point, it’s not jsonpath like kubectl, but JMESPath, similar but not identical…)