Zeropod on Zwindler's Reflection

zeropod v0.12.0: One Year Later, Does Scale-to-Zero Deliver?

Sat, 30 May 2026 12:00:00 +0200

What is zeropod?

A little less than a year ago, I published a first article, reviewing a tool called zeropod.

Zeropod is a Kubernetes runtime (more specifically a containerd shim) that automatically checkpoints containers to disk after a certain amount of time of the last TCP connection.

While in scaled down state, it will listen on the same port the application inside the container was listening on and will restore the container on the first incoming connection.

At the time, the stable version was v0.6.x, and I tested it on a k3s cluster. The results were mixed: it mostly worked, but with deal-breaking limitations for serious use (probes impossible, flaky behavior under load, checkpointing times a bit high for my taste).

Since then, the project has evolved quite a bit (now v0.12.0), with promises of fixes and improvements. So I decided to give it another shot to see where things stand.

One other change: I abandoned k3s for this test series, for reasons I’ll detail throughout the article.

Prerequisites

This time, I used a freshly provisioned server at my favorite hosting provider:

An Ubuntu 24.04.3 LTS (Noble) server, kernel 6.17.0-35 HWE, 7.7 Gi RAM, 100G disk
A Kubernetes cluster set up with kubeadm, flannel as CNI
Vanilla containerd
local-path-provisioner (Rancher) for default local storage
No cert-manager or Ingress, keeping it simple this time

Installation

I’ve installed kubeadm many times and you probably have too, so I won’t insult you with yet another tutorial.

# Install kubeadm, kubelet, kubectl + containerd
sudo kubeadm init --pod-network-cidr=10.42.0.0/16
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

Once the cluster is running, installing zeropod is trivial:

# Apply the generic kustomize
kubectl apply -k https://github.com/ctrox/zeropod/config/generic

# Add a label to our single node
kubectl label node <your-node> zeropod.ctrox.dev/node=true

# Verify the pod is running
kubectl -n zeropod-system wait --for=condition=Ready pod -l app.kubernetes.io/name=zeropod-node

That’s it. No special flags, no extra configuration. On kubeadm, kubelet is a native binary (/usr/bin/kubelet) detected automatically by zeropod.

Note about k3s: on k3s, the zeropod documentation differs since the config to use is config/k3s. The kustomize adds a -probe-binary-name=k3s flag on the initContainer so the shim knows the kubelet is embedded in the k3s binary. During my tests, even with this flag, the behavior wasn’t as expected (the socket tracker didn’t filter probes correctly). With the default config, the flag is on the initContainer but not on the manager. I suspected there was another component to patch, but I didn’t investigate further.

The DaemonSet deploys the following images:

ghcr.io/ctrox/zeropod-manager:v0.12.0
ghcr.io/ctrox/zeropod-installer:v0.12.0
ghcr.io/ctrox/zeropod-criu:v4.2 (CRIU has been updated since v0.6.x)

Let’s verify the zeropod runtimeClass is available:

kubectl get runtimeclass
NAME HANDLER AGE
zeropod zeropod 30m

First test: nginx

Like last time, let’s start with a simple nginx test. Deploy a basic pod:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx
spec:
 replicas: 1
 selector:
 matchLabels:
 app: nginx
 template:
 metadata:
 annotations:
 zeropod.ctrox.dev/scaledown-duration: 10s
 labels:
 app: nginx
 spec:
 runtimeClassName: zeropod
 containers:
 - image: nginx
 name: nginx
 ports:
 - containerPort: 80

The important parts are the annotation zeropod.ctrox.dev/scaledown-duration: 10s and runtimeClassName: zeropod.

Shortly after deployment, zeropod detects the absence of traffic. The container is checkpointed. Looking at the manager logs:

{"time":"2026-05-29T14:26:37.269453861Z","level":"INFO","msg":"status event","container":"nginx","pod":"nginx-bench-7c65c8874-6pts7","phase":"RUNNING","duration":"0s"}

Then 10 seconds later:

{"time":"2026-05-29T14:26:47.465510937Z","level":"INFO","msg":"status event","container":"nginx","pod":"nginx-bench-7c65c8874-6pts7","phase":"SCALED_DOWN","duration":"191.259383ms"}

The duration field in the SCALED_DOWN log is the checkpoint time: 191ms. That’s already significantly better than the ~400ms from v0.6.x on k3s (likely thanks to an update, perhaps CRIU 4.2?).

Restore

When we send an HTTP request, the container wakes up:

time curl http://<POD_IP>/ -s -o /dev/null -w "%{http_code} (%{time_total}s)"
HTTP 200 (0.101s)

real 0m0.101s
user 0m0.003s
sys 0m0.003s

101ms to restore nginx and serve a page. That’s comparable to the ~92ms from the original article.

Nice new feature: under v0.6.x, kubectl top pods returned an error for checkpointed pods:

# v0.6.x
kubectl top pods
error: Metrics not available for pod default/php-xxx

This bug was fixed in v0.9.0. Now pods in SCALED_DOWN state return 0m 0Mi:

kubectl top pods
NAME CPU(cores) MEMORY(bytes)
nginx 0m 0Mi

Much cleaner.

Testing liveness probes

This was THE major limitation of v0.6.x: zeropod was incompatible with Kubernetes probes. If you added an httpGet liveness probe to your container, the probe would reset the scale-down timer, and your container would never go SCALED_DOWN. Result: probes unusable, making zeropod unusable in production.

What changed

Two fixes were made between v0.6.x and v0.12.0:

The activator (the eBPF component that listens for traffic during SCALED_DOWN): it intercepts probes and replies 200 directly, without waking the container. That’s the “post scale-down” behavior.
The socket tracker (the component that ignores connections during RUNNING state): since PR #72 (merged August 2025), zeropod can detect connections coming from kubelet and not count them as “real” traffic. That’s the “pre scale-down” behavior.

Real-world test

I deployed nginx with an aggressive liveness probe (periodSeconds: 5) and scaledown-duration: 10s:

spec:
 runtimeClassName: zeropod
 containers:
 - image: nginx
 name: nginx
 livenessProbe:
 httpGet:
 path: /
 port: 80
 periodSeconds: 5

On k3s, this test gave me trouble: the socket tracker couldn’t filter probe connections, which kept resetting the scale-down timer indefinitely. The activator (which filters probes once scale-down has happened) worked fine though.

On kubeadm, the result is immediate:

kubectl get pod -l app=nginx-probe -o json | jq -r '.items[0].metadata.labels["status.zeropod.ctrox.dev/nginx"]'
SCALED_DOWN

The socket tracker correctly filters connections from the native kubelet. The periodSeconds > scaledown-duration rule I had to use on k3s is no longer necessary.

A word on changes between v0.6.x and v0.12.0

Here’s a summary of improvements between the two versions:

Point	v0.6.x	v0.12.0
`kubectl top pods` while scaled-down	❌ Error	✅ `0m 0Mi` (fix v0.9.0)
Checkpoint (nginx)	~400ms	~185ms (-54%)
Restore (nginx)	~92ms	~99ms (stable)
CRIU	v3.x	v4.2
Checkpoint failure handling	basic	metrics + events (v0.11.0)
Configurable proxy timeouts	❌	✅ (v0.11.0)
Inter-node migration	basic	improved + timeouts (v0.10.0)
Probes	❌ Incompatible	✅ Activator + socket tracker

Interesting technical note about CRIU flags: zeropod removed the --tcp-established option in September 2025. Previously, active TCP connections were saved and restored with the container. Now, zeropod uses --tcp-skip-in-flight (when runc >= 1.3 supports it). Practical consequence: if your container has outgoing TCP connections at checkpoint time, they will be lost. You’ll need to re-establish them after restore.

Deploying WordPress (the “““realistic””” use case)

Alright, nginx is fine but not very representative. Let’s deploy a real app with PHP, Apache, and a database.

Let’s reuse the WordPress manifest from the original article, without zeropod on MySQL:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: wordpress
spec:
 replicas: 1
 selector:
 matchLabels:
 app: wordpress
 template:
 metadata:
 annotations:
 zeropod.ctrox.dev/scaledown-duration: 10s
 zeropod.ctrox.dev/container-names: wordpress
 zeropod.ctrox.dev/ports-map: wordpress=80
 labels:
 app: wordpress
 spec:
 runtimeClassName: zeropod
 initContainers:
 - name: wait-for-mysql
 image: mysql:8
 command:
 - sh
 - -c
 - |
 until mysql -h mysql -u root -p${MYSQL_ROOT_PASSWORD} -e "SELECT 1"; do
 echo "Waiting for MySQL..."
 sleep 3
 done
 mysql -h mysql -u root -p${MYSQL_ROOT_PASSWORD} -e "CREATE DATABASE IF NOT EXISTS wordpress;"
 env:
 - name: MYSQL_ROOT_PASSWORD
 value: verySecurePassword
 containers:
 - image: wordpress:latest
 name: wordpress
 ports:
 - containerPort: 80
 env:
 - name: WORDPRESS_DB_HOST
 value: mysql
 - name: WORDPRESS_DB_USER
 value: root
 - name: WORDPRESS_DB_PASSWORD
 value: verySecurePassword
 - name: WORDPRESS_DB_NAME
 value: wordpress

MySQL (without zeropod):

apiVersion: v1
kind: Service
metadata:
 name: mysql
spec:
 ports:
 - port: 3306
 selector:
 app: mysql
 clusterIP: None
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: mysql
spec:
 serviceName: mysql
 replicas: 1
 selector:
 matchLabels:
 app: mysql
 template:
 metadata:
 labels:
 app: mysql
 spec:
 containers:
 - image: mysql:8
 name: mysql
 ports:
 - containerPort: 3306
 env:
 - name: MYSQL_ROOT_PASSWORD
 value: verySecurePassword
 volumeMounts:
 - mountPath: /var/lib/mysql
 name: data
 volumeClaimTemplates:
 - metadata:
 name: data
 spec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 5Gi

WordPress (Apache + PHP) checkpoint:

{"time":"2026-05-29T14:33:35.915988635Z","phase":"SCALED_DOWN","duration":"313.286787ms"}

Checkpoints take around 300ms. Restore is around 200ms:

{"time":"2026-05-29T14:34:31.56806589Z","phase":"RUNNING","duration":"206.33005ms"}

The first curl request confirms it:

time curl http://<POD_IP>/ -s -o /dev/null -w "%{http_code} (%{time_total}s)"
HTTP 302 (0.212s)

212ms, about twice faster than the 454ms from the original article.

The killer test: cascading WordPress + MySQL

In the original article, I attempted the ultimate test: putting both pods (WordPress AND MySQL) with runtimeClassName: zeropod, then sending an HTTP request to WordPress while both are checkpointed.

The scenario goes like this:

After a few seconds, WordPress goes SCALED_DOWN, MySQL goes SCALED_DOWN too
A client sends an HTTP request to WordPress
The WordPress activator intercepts the request and restores the WordPress container
WordPress (Apache+PHP) starts up, executes PHP code requiring a MySQL connection
WordPress connects to MySQL on port 3306
The MySQL activator intercepts the connection and restores MySQL
MySQL responds, WordPress generates the page, Apache sends back the HTTP response

True scale to zero that scales the entire app, not just the frontend/backend.

Important note: I know that you’d probably never want to scale down your database in production, but testing this shows that this approach (eBPF + CRIU) works beyond just scaling webservers to zero, which other tools on the market already do very well.

On k3s: no luck, on kubeadm: victory

On k3s, I still haven’t managed to make this scenario work: WordPress would restore but wouldn’t respond on port 80. I spent quite some time trying to figure out why, without success.

Same test, same zeropod version, but on kubeadm:

curl http://<POD_IP>/ -s -o /dev/null -w "%{http_code} (%{time_total}s)"
HTTP 302 (0.224s)

224ms. Both pods went from SCALED_DOWN to RUNNING, the WordPress page loads. I repeated the test 5 times in a row:

Cycle	Time (curl)	HTTP
1	229ms	302
2	192ms	302
3	227ms	302
4	230ms	302
5	211ms	302

The zeropod logs confirm both containers waking up:

WordPress: SCALED_DOWN → RUNNING
MySQL: SCALED_DOWN → RUNNING

What really changed since v0.6.x

Probes (finally!)

This was my main grievance in the first article. Today, it’s resolved:

Before: impossible to use Kubernetes probes → zeropod unusable in any somewhat serious use case
After: the activator intercepts probes in SCALED_DOWN (replies 200 without restore), and the socket tracker filters kubelet connections in RUNNING state

Stability

The flaky behavior (pod loss under simultaneous load) is gone. Where I had failures in the original article, sequential and simultaneous load tests all pass now.

Performance

Checkpoint gained ~50% speed (185ms vs 400ms). Restore too (200ms vs 454ms). Probably thanks to CRIU v4.2 and zeropod optimizations.

kubectl top pods

This small detail was an eyesore — kubectl top pods crashed on checkpointed pods. Fixed in v0.9.0.

Some remaining limitations

I’d be dishonest if I said everything is perfect. Here’s what’s still problematic:

Since September 2025, zeropod no longer uses --tcp-established for CRIU. This means if your container has outgoing TCP connections at checkpoint time, they will be lost. In practice, for a web server, this means reconnecting to the database after restore. With zeropod, the MySQL connection is re-established automatically (the current PHP request fails, the next one succeeds). This is a detail that may matter depending on your use case.
I only tested WordPress (Apache + mod_php) and MySQL. Applications using websockets, streaming, or long-lived connections might behave differently.
I had difficulties getting it to work properly on k3s.
I observed a glitch (Apache segfault) on the first restore of a freshly created WordPress pod. It’s not reproducible after a normal checkpoint/restore cycle, but if you frequently redeploy your pods, you might encounter it.

Verdict

One year later, zeropod seems to deliver on more of its promises:

Even though it wasn’t a dealbreaker, checkpoint/restore performance has improved significantly (~50% faster), which is always welcome
Kubernetes probe support has been added, lifting what I considered the main blocker
General stability is better
The cascade test (WordPress + MySQL) works

I’m still just as hyped about the idea of freezing a container and restoring it 10 seconds later like nothing happened. The magic of CRIU and eBPF combined is starting to materialize, after years of waiting.

Would I put this in production? Let’s say it’s less risky than a year ago. On my personal cluster for fun, why not. For a production database with long-lived connections, I think I’d still pass ;-).

But honestly, the project has evolved well and deserves a closer look.

zeropod: scale-to-zero with container checkpointing

Fri, 20 Jun 2025 19:00:00 +0200

What is zeropod?

I kept the intro paragraph from the project documentation as is because I find it perfect. It says everything you need to know about the tool, neither too much nor too little.

Zeropod is a Kubernetes runtime (more specifically a containerd shim) that automatically checkpoints containers to disk after a certain amount of time of the last TCP connection.

While in scaled down state, it will listen on the same port the application inside the container was listening on and will restore the container on the first incoming connection.

Depending on the memory size of the checkpointed program this happens in tens to a few hundred milliseconds, virtually unnoticable to the user.

As all the memory contents are stored to disk during checkpointing, all state of the application is restored.

It adjusts resource requests in scaled down state in-place if the cluster supports it.

To prevent huge resource usage spikes when draining a node, scaled down pods can be migrated between nodes without needing to start up.

TL;DR: it will freeze your app if it doesn’t receive TCP calls, and restore it when a call arrives.

If you want to understand in more detail HOW it really works, in that case, I invite you to read the “How it works” section of the official project documentation, which has the merit of being quite clear:

https://github.com/ctrox/zeropod?tab=readme-ov-file#how-it-works

Prerequisites

Let’s not waste time, let’s dive into the experiment. As prerequisites, I needed:

an Ubuntu server with vanilla k3s (flannel + traefik, single node). If you don’t know how to install it, you can always check out my article on the subject.
cert-manager. Not necessarily required but I like having valid HTTPS certificates.

# Install cert-manager CRDs and namespace
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.0/cert-manager.yaml

# Wait for cert-manager to be ready
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=cert-manager -n cert-manager --timeout=60s
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=cainjector -n cert-manager --timeout=60s
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=webhook -n cert-manager --timeout=60s

ClusterIssuer configuration:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-prod
spec:
 acme:
 server: https://acme-v02.api.letsencrypt.org/directory
 email: your.email@example.org
 privateKeySecretRef:
 name: letsencrypt-prod
 solvers:
 - http01:
 ingress:
 class: traefik

Also optional, for easier access, I modified the traefik service ports to 30080 and 30443 since I don’t have LoadBalancer Service support on this cluster:

kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cert-manager cert-manager ClusterIP 10.43.212.125 <none> 9402/TCP 21h
cert-manager cert-manager-webhook ClusterIP 10.43.134.29 <none> 443/TCP 21h
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 21h
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 21h
kube-system metrics-server ClusterIP 10.43.83.225 <none> 443/TCP 21h
kube-system traefik LoadBalancer 10.43.208.56 192.168.1.242 80:30080/TCP,443:30443/TCP 21h

Installing zeropod

Once we have our functional cluster with all the prerequisites, we can install zeropod. The first step “just” involves applying the following kustomize manifest, which will create a customized DaemonSet with the right paths so it can hook into / patch containerd.

apiVersion: apps/v1
kind: DaemonSet
metadata:
 name: zeropod-node
 namespace: zeropod-system
spec:
 template:
 spec:
 volumes:
 - name: containerd-etc
 hostPath:
 path: /var/lib/rancher/k3s/agent/etc/containerd/
 - name: containerd-run
 hostPath:
 path: /run/k3s/containerd/
 - name: zeropod-opt
 hostPath:
 path: /var/lib/rancher/k3s/agent/containerd

kubectl apply -k https://github.com/ctrox/zeropod/config/k3s

Then we’ll label the Node and verify the controller pod is working:

kubectl label node zeropod zeropod.ctrox.dev/node=true
kubectl -n zeropod-system wait --for=condition=Ready pod -l app.kubernetes.io/name=zeropod-node

The documentation indicates that you need to restart the Node in the case of k3s because everything is packaged together in k3s (probably the same for k0s). This is probably not necessary for most “normal” distributions.

NAMESPACE NAME READY STATUS RESTARTS AGE
...
zeropod-system zeropod-node-qntzh 1/1 Running 1 (21h ago) 21h

Interesting point: zeropod will add its own runtimeClass (I’ll let you check out the Kubernetes documentation, it’s worth a look if you’re not familiar):

kubectl get runtimeclass
NAME HANDLER AGE
crun crun 21h
[...]
zeropod zeropod 21h

Deploying a WordPress application

What’s the best use case for Kubernetes?

Hosting a personal blog with WordPress and autoscaling, of course!! Everyone knows that.

Beyond the joke, the idea was to test a stateful application, preferably with a database, to see how far we can push the tool. Because one of the limitations of scale-to-zero tools in Kubernetes is precisely that they work great for stateless workloads (or FaaS), but it’s more complicated when you have state.

First thing to know: beyond the label we put on the node, it’s necessary to add 2 additional configuration points to our applications that we want to scale to zero.

The zeropod annotations; no need to explain what they do: we give it the port number, the container, and the duration after which, if I have no connection, I scale down:

annotations:
 zeropod.ctrox.dev/ports-map: "wordpress=80"
 zeropod.ctrox.dev/container-names: wordpress
 zeropod.ctrox.dev/scaledown-duration: 10s

The runtimeClass that must be defined:

runtimeClassName: zeropod

Application manifests

Here’s what it could roughly look like. We could do cleaner (helm charts) but I did quick and dirty, it’s enough for this PoC:

WordPress Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: php
spec:
 selector:
 matchLabels:
 app: php
 template:
 metadata:
 labels:
 app: php
 annotations:
 zeropod.ctrox.dev/ports-map: "wordpress=80"
 zeropod.ctrox.dev/container-names: wordpress
 zeropod.ctrox.dev/scaledown-duration: 10s
 spec:
 runtimeClassName: zeropod
 initContainers:
 - command:
 - sh
 - -c
 - |
 until mysql -h mysql -u root -p${MYSQL_ROOT_PASSWORD} -e "SELECT 1"; do
 echo "Waiting for MySQL to be ready..."
 sleep 5
 done
 echo "MySQL is ready!"
 mysql -h mysql -u root -p${MYSQL_ROOT_PASSWORD} -e "CREATE DATABASE IF NOT EXISTS wordpress;"
 mysql -h mysql -u root -p${MYSQL_ROOT_PASSWORD} -e "GRANT ALL PRIVILEGES ON wordpress.* TO 'root'@'%';"
 env:
 - name: MYSQL_ROOT_PASSWORD
 value: verySecurePassword
 image: mysql
 imagePullPolicy: IfNotPresent
 name: wait-for-mysql
 containers:
 - env:
 - name: WORDPRESS_DB_HOST
 value: mysql
 - name: WORDPRESS_DB_USER
 value: root
 - name: WORDPRESS_DB_PASSWORD
 value: verySecurePassword
 - name: WORDPRESS_DB_NAME
 value: wordpress
 image: wordpress:latest
 imagePullPolicy: Always
 name: wordpress
 ports:
 - containerPort: 80
 protocol: TCP

MySQL StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: mysql
spec:
 selector:
 matchLabels:
 app: mysql
 serviceName: "mysql"
 replicas: 1
 template:
 metadata:
 labels:
 app: mysql
 spec:
 containers:
 - image: mysql
 name: mysql
 ports:
 - containerPort: 3306
 env:
 - name: MYSQL_ROOT_PASSWORD
 value: verySecurePassword
 volumeMounts:
 - name: data
 mountPath: /var/lib/mysql
 volumeClaimTemplates:
 - metadata:
 name: data
 spec:
 accessModes: ["ReadWriteOnce"]
 resources:
 requests:
 storage: 5Gi

Services and Ingress:

apiVersion: v1
kind: Service
metadata:
 name: php
 labels:
 app: php
spec:
 ports:
 - port: 8080
 name: http
 targetPort: 80
 selector:
 app: php
---
apiVersion: v1
kind: Service
metadata:
 name: mysql
 labels:
 app: mysql
spec:
 ports:
 - port: 3306
 name: mysql
 clusterIP: None
 selector:
 app: mysql
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: zeropod-ingress
 namespace: default
 annotations:
 traefik.ingress.kubernetes.io/router.entrypoints: web,websecure
 cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
 rules:
 - host: zeropod.example.org
 http:
 paths:
 - path: /
 pathType: Prefix
 backend:
 service:
 name: php
 port:
 number: 8080
 tls:
 - hosts:
 - zeropod.example.org
 secretName: zeropod-tls

Observing the behavior

Once deployed, let’s quickly check the state of pods and services:

kubectl get pods
NAME READY STATUS RESTARTS AGE
mysql-0 1/1 Running 0 17h
php-dc7cb9cff-29hzb 1/1 Running 0 17h

kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
[...]
mysql ClusterIP None <none> 3306/TCP 21h
php ClusterIP 10.43.165.131 <none> 8080/TCP 21h

Detecting the absence of traffic

Shortly after deploying Apache PHP, zeropod notices that there hasn’t been a connection for a while and pauses the container.

The funny thing is that from Kubernetes’ point of view, nothing happened! The php-dc7cb9cff-29hzb pod still exists and is present in the Node’s Non terminated pods list:

kubectl get pods php-dc7cb9cff-29hzb
NAME READY STATUS RESTARTS AGE
php-dc7cb9cff-29hzb 1/1 Running 0 17h

kubectl describe nodes
[...]
Non-terminated Pods: (11 in total)
 Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
 --------- ---- ------------ ---------- --------------- ------------- ---
[...]
 default mysql-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 17h
 default php-dc7cb9cff-29hzb 0 (0%) 0 (0%) 0 (0%) 0 (0%) 17h
[...]

However, it no longer appears in the “top pods” metrics collected by metrics-server:

kubectl top pods
NAME CPU(cores) MEMORY(bytes)
mysql-0 4m 460Mi

And if we search for the apache2 process on the node, we won’t find it:

sudo ps -ef | grep apache2

Under the hood, zeropod logs

In the zeropod logs, we first notice the detection of a container eligible for scale to zero:

{"time":"2025-06-20T17:31:46.222502407Z","level":"INFO","msg":"subscribing to status events","sock":"/run/zeropod/s/5858a327ae5a70c2e12b5dad1e8320a4670ff11152476ad49605ebca5327f7d6.sock"}
{"time":"2025-06-20T17:31:47.572065537Z","level":"INFO","msg":"status event","component":"podlabeller","container":"wordpress","pod":"php-dc7cb9cff-29hzb","namespace":"default","phase":1}
{"time":"2025-06-20T17:31:47.5737556Z","level":"INFO","msg":"attaching redirector for sandbox","pid":64980,"links":["eth0","lo"]}

And after 10 seconds, since there hasn’t been a connection, zeropod shuts it down (“phase”:0):

{"time":"2025-06-20T17:31:57.932464147Z","level":"INFO","msg":"status event","component":"podlabeller","container":"wordpress","pod":"php-dc7cb9cff-29hzb","namespace":"default","phase":0}

Performance tests

When the process is checkpointed, we’ll see that curl still takes a bit of time to be served. Nothing dramatic, but still:

time curl https://zeropod.example.org:30443 -I
HTTP/2 200
content-type: text/html; charset=UTF-8
date: Fri, 20 Jun 2025 17:41:29 GMT
link: <https://zeropod.example.org:30443/wp-json/>; rel="https://api.w.org/"
server: Apache/2.4.62 (Debian)
x-powered-by: PHP/8.2.28

real 0m0.454s
user 0m0.052s
sys 0m0.010s

However, for all subsequent connections, the times are correct for an empty (and unoptimized) WordPress:

time curl https://zeropod.example.org:30443 -I
HTTP/2 200
content-type: text/html; charset=UTF-8
date: Fri, 20 Jun 2025 17:41:42 GMT
link: <https://zeropod.example.org:30443/wp-json/>; rel="https://api.w.org/"
server: Apache/2.4.62 (Debian)
x-powered-by: PHP/8.2.28

real 0m0.088s
user 0m0.053s
sys 0m0.008s

So, it works!

The first connection takes a bit longer, while the eBPF program that “listens” to traffic while the container is down turns it back on and hands over control (here +350-400ms on a small, cheap, and quite loaded VM).

Once the container is restored, we can actually see the apache2 processes reappear with a ps on the Node:

ps -ef |grep apache2
root 67038 64955 1 18:56 ? 00:00:00 apache2 -DFOREGROUND
www-data 67055 67038 0 18:56 ? 00:00:00 apache2 -DFOREGROUND
www-data 67056 67038 0 18:56 ? 00:00:00 apache2 -DFOREGROUND
www-data 67057 67038 0 18:56 ? 00:00:00 apache2 -DFOREGROUND
www-data 67058 67038 0 18:56 ? 00:00:00 apache2 -DFOREGROUND
www-data 67059 67038 0 18:56 ? 00:00:00 apache2 -DFOREGROUND
www-data 67060 67038 0 18:56 ? 00:00:00 apache2 -DFOREGROUND

Going further: testing with MySQL

Well, on the other hand, scaling an http server to 0 is fun, but it’s not revolutionary. There are already scale-to-zero solutions on the market in Kubernetes, especially for FaaS or stateless workloads. But what if we push the experiment all the way?

We rarely want to scale a database to 0 in real life. Stopping it and then restarting it can take time, and potentially cause errors in our apps if the scaling is done poorly.

However, with zeropod, the principle is a bit different, since we’re not really going to stop the process, just freeze it.

For science (don’t do this in prod), I therefore added zeropod to the MySQL database too!

As with wordpress, we add the annotations and runtimeClass:

template:
 metadata:
 labels:
 app: mysql
 annotations:
 zeropod.ctrox.dev/ports-map: "mysql=3306"
 zeropod.ctrox.dev/container-names: mysql
 zeropod.ctrox.dev/scaledown-duration: 10s
 spec:
 runtimeClassName: zeropod
 containers:
 # ...

After a few seconds, zeropod moves the mysql database to phase “0”:

{"time":"2025-06-20T18:06:14.92372877Z","level":"INFO","msg":"status event","component":"podlabeller","container":"mysql","pod":"mysql-0","namespace":"default","phase":1}
{"time":"2025-06-20T18:06:14.925316023Z","level":"INFO","msg":"attaching redirector for sandbox","pid":69570,"links":["eth0","lo"]}
{"time":"2025-06-20T18:06:25.766097339Z","level":"INFO","msg":"status event","component":"podlabeller","container":"mysql","pod":"mysql-0","namespace":"default","phase":0}

kubectl top pods no longer reports any pods (and an error…):

kubectl top pods
error: Metrics not available for pod default/php-dc7cb9cff-29hzb, age: 35m59.026573861s

But the pods remain visible and “Running” from Kubernetes’ point of view:

kubectl get pods
NAME READY STATUS RESTARTS AGE
mysql-0 1/1 Running 0 2m30s
php-dc7cb9cff-29hzb 1/1 Running 0 36m

Final test: cascading wake-up

The final test: will an HTTP call to php wake it up, which will trigger a connection to the mysql database which will in turn wake it up too?

Drum roll

time curl https://zeropod.example.org:30443 -I
HTTP/2 200
content-type: text/html; charset=UTF-8
date: Fri, 20 Jun 2025 18:09:24 GMT
link: <https://zeropod.example.org:30443/wp-json/>; rel="https://api.w.org/"
server: Apache/2.4.62 (Debian)
x-powered-by: PHP/8.2.28

real 0m0.978s
user 0m0.053s
sys 0m0.008s

Victory!!

Limitations

Beyond this somewhat silly example (who hasn’t wanted to host a wordpress on Kubernetes with scale to zero?), we realize that the technology “works” but remains a bit flaky.

I had several cases where scale to zero happened on the php pod, while I was running a loop with “while true; do curl” (maybe related to the ingress -> service -> pod chain?). And the checkpointing time is still visible on my test VM (400 ms per container, that’s not nothing).

One point that is not addressed in the project documentation is that it’s almost impossible to have proper liveness / readiness probes when you use zeropod.

If you put a liveness probe on a web service, you’ll trigger a call on the TCP port listened to by zeropod, and thus restart the app which will never be checkpointed. If you put a readiness probe, same thing.

And if you plan to get around this with a liveness / readiness probe that doesn’t trigger an HTTP call on the port monitored by zeropod, you’ll end up with an app seen by Kubernetes as (respectively) KO or “Not ready”, since the container won’t be available because it’s checkpointed.

an open issue on the subject - github.com/ctrox/zeropod/issues/34

Technically, zeropod is very fun and quite clever, I’m quite impressed.

But I don’t really see in what world we would want to have containers in prod without liveness / readiness probes, so I’m quite skeptical about using this technology as is, except for very non-critical examples. This limitation seems too big to me.