Etcd on Zwindler's Reflection

Kyverno killed my API Server. Again.

Thu, 26 Feb 2026 08:00:00 +0200

The revenge strikes back

You may have read my previous article about etcd 3 years ago. If you remember correctly, the crash was etcd, but the real culprit was Kyverno. I love Kyverno. It’s really a piece of software I’m very fond of. Mainly because it’s incredibly powerful. I even wrote an introductory article and a second one that goes deeper on the topic (both in french, though).

But the sheer number of incidents and weird side effects it causes. Mamamia… This isn’t the first incident of the year I’ve had with Kyverno (yes, we’re in February) but since this one is entertaining, I’m sharing it with you.

During a routine maintenance operation to upgrade a Kubernetes cluster to version 1.34 (from 1.32), we ended up facing the dreaded scenario for any kube admin: a completely unreachable API Server after restarting the Control Plane nodes.

What initially looked like a typical network error turned out to be a subtle deadlock between new native Kubernetes networking features and our dear Kyverno 😘.

Spoiler: it wasn’t a network issue. It’s never a network issue. Well, sometimes it is. But not this time.

The upgrade that starts well

Alright, a Kubernetes upgrade has become pretty routine at this point. We do it regularly, we have our procedures, we’re pros (I swear). We jump from 1.32 to 1.34 in a single commit, skipping the hop through 1.33.

YOLO.

In the technical context I’m talking about, everything is managed as code. From machine provisioning all the way to Talos deployment, including MachineConfigs (the CustomResources to modify… well, the machine).

For more details, see the Talos documentation on Machine Configs.

The first cluster we test has only one control plane (don’t ask me why, it probably wouldn’t have changed anything). Talos restarts the API Server with the new version and then… nothing.

The “weird” API Server logs (technical term) speak for themselves:

I0224 15:32:50.979280 1 default_servicecidr_controller.go:166] Creating default ServiceCIDR with CIDRs: [10.1.0.0/20]
W0224 15:32:50.984784 1 dispatcher.go:225] rejected by webhook "validate.kyverno.svc-fail":
admission webhook "validate.kyverno.svc-fail" denied the request:
Get "https://10.1.0.1:443/api": dial tcp 10.1.0.1:443: connect: operation not permitted
I0224 15:32:50.985342 1 event.go:389] "Event occurred" kind="ServiceCIDR"
apiVersion="networking.k8s.io/v1" type="Warning"
reason="KubernetesDefaultServiceCIDRError"
message="The default ServiceCIDR can not be created"

😬😬😬

The root cause: a magnificent vicious circle

After investigation, we discovered that the incident was the result of a collision between a Kubernetes core evolution and our Kyverno configuration. A textbook deadlock case.

Let’s break down the mechanism:

The new ServiceCIDR Kind

In recent versions (v1.33+), Kubernetes migrates service IP range management to dedicated objects named ServiceCIDR. On the first boot after the upgrade, the API Server automatically tries to create the default object (e.g., 10.1.0.0/20).

For the curious, KEP-1880 and the official ServiceCIDR documentation detail this evolution.

It’s new, it’s clean, it’s well designed. Except that…

Interception by the Kyverno Webhook

Kyverno, configured with failurePolicy: Fail (because we’re serious people who don’t let just anything through in prod), is set up to intercept resource creations to validate them, and fail the call if Kyverno doesn’t respond.

Including the ServiceCIDR freshly created by the API Server itself.

Deadlock

And this is where it gets beautiful:

The API Server pauses the ServiceCIDR creation waiting for Kyverno’s “OK”
To contact the Kyverno service, the API Server needs to route the request through the Kubernetes service IP (typically 10.1.0.1)
But the network layer (service routing) can’t initialize until the ServiceCIDR object is validated and created

It’s the chicken and the egg, “I locked my keys inside the car” edition.

PTSD. Yes, that actually happened to me. In the desert. With no cell service.

Profit.

The API Server times out or returns a connect: operation not permitted error when trying to reach the webhook, blocking its own initialization. CrashLoopBackOff on the API Server. :D

Breaking out of the deadlock

To escape this deadlock, you need to temporarily bypass the admission layer. Easy, right?

The “usual” workaround: useless

Deadlocks with Kyverno, we’re used to them at this point. Normally, since kube-system is ignored, you can simply connect with a break-glass kubeconfig (we normally use OIDC) that has the cluster-admin cluster role and delete the Kyverno validating webhooks:

kubectl delete validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg

Except here, the API Server won’t even start. My kubectl isn’t going to work, obviously!

The real workaround: disable webhooks at boot

The solution we chose was to modify the API Server configuration to temporarily disable validation webhooks at startup. My esteemed colleague Maxime hot-edited the machine config (using break-glass talosctl access) to add the following flag directly in the API server’s extraArgs:

--disable-admission-plugins=ValidatingAdmissionWebhook

For those unfamiliar with admission control in Kubernetes, just know that there’s a list of “default” plugins but everything can be toggled off. I might do a deep dive on Kubernetes admission control someday, it’s fascinating ;).

With this flag, the API Server can finally create its ServiceCIDR objects without asking anyone for permission (completely bypassing all validation mechanisms that Kyverno or similar tools enforce), the network initializes, Kyverno starts, and then you can remove the flag and restart cleanly.

The “funny” option we didn’t try

Personally, I thought it would be hilarious to go directly into the etcd database and delete the webhook key causing the issue (also through talosctl):

# Example via etcdctl
etcdctl del /registry/admissionregistration.k8s.io/validatingwebhookconfigurations/kyverno-resource-validating-webhook-cfg

My colleagues were less enthusiastic: “Yeah but you know, if we break etcd it’s gonna be painful”. We played it safe with the flag. I’m deeply disappointed we didn’t try 😂.

The permanent fix: MatchConditions

OK, now that the cluster is back up, how do we make sure this doesn’t happen again on the next upgrade?

The clean solution is to use matchConditions (introduced in Kubernetes 1.27) on the ValidatingWebhookConfiguration. This allows you to exclude critical network bootstrap resources before the request even attempts to leave the API Server toward the Kyverno pod.

See the official documentation on matchConditions.

We were already using this option to throttle the sometimes excessive Kyverno traffic (if you manage Kyverno, you know what I’m talking about) on a number of events (we’d overwhelm the API server or Kyverno, in CPU or RAM, depending on the case). We just had to add exclusions for the new types:

# Exclude network bootstrap resources to prevent the deadlock
matchConditions:
 - name: 'exclude-ServiceCIDR'
 expression: '!(request.kind.kind == "ServiceCIDR")'
 - name: 'exclude-IPAddress'
 expression: '!(request.kind.kind == "IPAddress")'

With this, when the API Server creates a ServiceCIDR at boot, the request no longer goes through the Kyverno webhook. No circular dependency, no deadlock, everyone’s happy.

Conclusion

As the current French president would say about something that was painfully predictable:

who could have predicted this?

OK fine, all we had to do was read the Kubernetes 1.33 release notes. That said, we have a staging cluster, that’s what it’s for. We broke staging, no big deal.

Context: this is the “Big deal” TV Game Mascot

Maybe we’ll actually read them next time?

Kubernetes - Error from server: etcdserver: mvcc: database space exceeded

Thu, 30 Nov 2023 12:00:00 +0200

Little panic in prod!

It won’t surprise anyone that I manage Kubernetes clusters in production (and also for fun. Yes I’m weird). And sometimes, when you have slightly unusual workloads, you get some funny surprises.

To give you some context, I have an “on-prem” cluster of about sixty physical baremetal nodes, on which I run (among other things) very short ephemeral jobs.

This generates a lot of traffic on the etcd database (lots of pod creation/destruction events) and it has already caused us problems in the past (especially for metrics collection - we’ve crashed Prometheus even with lots of RAM).

But it was holding up.

And then, disaster strikes…

Recently, I also started adding lots of new “policies” on Kyverno (for those who don’t know, I wrote an article on the subject in french).

And I discovered somewhat to my detriment that Kyverno adds A LOT of pressure on etcd, by creating tons of events (policy evaluation for each Kubernetes object, and since I have many…). You can tweak that but that’s not the point of the article.

To such an extent that at some point, all write operations on the API server were returning the following error:

Error: etcdserver: mvcc: database space exceeded

Smells like a full DB 😬😬😬. Yet, no alerts on my disk spaces!? Weird.

Anyway, to make matters worse, since the etcd pods were no longer healthy because the database was supposedly full, the API server crashed after a while and I had no control over anything to debug.

PANIC!

Getting things back on track

Little “pro tip” when your Kubernetes control plane is in trouble - at some point nothing works anymore because the etcd/apiserver pods go into crashloop backoff, being alternately down and not restarted because of the “backoff”.

A good way to work around this, when you have control over the control plane (so, not on managed services, because in that case, it’s not your problem), is to restart the kubelet on all control plane nodes at the same time.

Why? Because restarting the kubelet resets all the counters so your pods aren’t in the “backoff” phase for too long, which avoids issues like “not enough etcd members are on, so I self-sabotage” and so on.

Diagram source: Natan Yellin’s Substack for robusta.dev

Cleaning up the etcd database

Now that’s done, I had enough living etcd pods (even if not operational) for a short time.

A quick look at my Grafana graphs shows that something indeed “happened” when I activated the new Kyverno policies.

And “BY SHEER COINCIDENCE” (or is it?), it stops working when the size of 2 out of 3 etcd databases reaches 2 GB

A quick Internet search confirms that my database is full, and that the solution to my problem is to compact then defragment my database.

Since everything is a bit broken, you can find bash commands to send to get the latest revision, compact then defragment the database:

# get latest revision
REVISION=$(ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9].*')

# compact
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key compact ${REVISION}

# defrag
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key defrag

# SUPER MEGA IMPORTANT
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key alarm disarm

I added a SUPER MEGA IMPORTANT comment on the last instruction because, since I’m a bit silly, I didn’t initially run it.

Well yeah. The idea of disarming an alarm seemed completely incongruous to me. The alarm, I want to keep it. Right?

But actually, this is the normal behavior. The etcd cluster is designed to block itself, and do nothing more, as long as the alarm hasn’t been disarmed, even if the space issue is resolved…

Ok, I’ll know for next time

What does “mvcc: database space exceeded” mean and how do I fix it? The multi-version concurrency control data model in etcd keeps an exact history of the keyspace. Without periodically compacting this history (e.g., by setting –auto-compaction), etcd will eventually exhaust its storage space. If etcd runs low on storage space, it raises a space quota alarm to protect the cluster from further writes. So long as the alarm is raised, etcd responds to write requests with the error mvcc: database space exceeded.

To recover from the low space quota alarm:

Compact etcd’s history.

Defragment every etcd endpoint. - Disarm the alarm.

memberID:11111111111111111111111 alarm:NOSPACE

Note: since etcd wasn’t coming back to life, I ran the compaction command 50-ish more times and it was returning an error that made me think there was another problem. But no, it’s just that the compaction was done correctly ;-).

error: code = OutOfRange desc = etcdserver: mvcc: required revision has been compacted"}
Error: etcdserver: mvcc: required revision has been compacted

Preventing it from happening again

OK, now that the DB is restored, how do we make sure this doesn’t happen again?

In reality, the default parameters of kubeadm’s (local) etcd database are not great.

First, we see that I hit the 2 GB limit, the default limit in etcd, even though I only have 60-70 nodes and a few thousand pods (ok, I create hundreds of thousands per day, but still).

Rancher’s documentation recommends raising this limit to 5 GB (Tuning etcd for Large Installations).

So we’re going to modify the quota-backend-bytes value.

Next, we need to enable automatic compaction for our etcd. Since I use kubeadm and a kubeadmcfg file, I need to add a section in my file to tune the parameters:

etcd:
 local:
 extraArgs:
 quota-backend-bytes: "5368709120"
 auto-compaction-retention: "1"
 auto-compaction-mode: periodic

Then regenerate the manifests on all my control plane nodes (unfortunately, this doesn’t happen automatically, cf official Kubernetes documentation - Reflecting ClusterConfiguration changes on control plane nodes)

kubeadm init phase etcd local --config kubeadmcfg.yaml

Note: be careful, the expected arguments are strings, so make sure to surround all numeric values with quotes:

json: cannot unmarshal number into Go struct field LocalEtcd.etcd.local.extraArgs of type string

Last point, while it’s possible to tell etcd to perform compaction all by itself, apparently defrag is not automatic.

Most of the time this might not be a problem, but since I’m extremely aggressive with pod creation/destruction + Kyverno policy reports, I also had to automate this “defrag” part.

Until I find something better, I’ve added a cron job:

00 */3 * * * /usr/bin/etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --endpoints https://@IP:2379 defrag --cluster

Well, it wasn’t the most fun operation of the year, but now it works again 😂.