I Tested For You: k8s The Easier Way (k8s-tew)

Mon, 26 May 2025 20:00:00 +0000

Weren’t You Writing a Book on Kubernetes?

Yes! But for now, I can’t really talk about it :-p. But I promise as soon as I can, I’ll communicate about it.

However, I can still say that writing this book led me to test A LOT of tools for installing Kubernetes. And as I’ve done before on this blog, I’m going to present one of the tools I tested called: Kubernetes The Easier Way, or k8s-tew.

The most cultured among you will certainly have recognized a reference to the Kubernetes the hard way project by Kelsey Hightower. It’s intentional:

Kubernetes is a fairly complex project. For a newbie it is hard to understand and also to use. While Kelsey Hightower’s Kubernetes The Hard Way, on which this project is based, helps a lot to understand Kubernetes, it is optimized for the use with Google Cloud Platform.

k8s-tew is a CLI that will install all the prerequisites for us to get a working k8s in a few commands, as well as a whole bunch of third-party software.

The project supports several deployment modes:

on the local host (but requires ports 80 and 443 to be available)
on a single remote machine
on multiple remote machines
HA control plane or not

Prerequisites

The k8s-tew project was developed in golang. In theory, it’s possible to compile it for macOS, Windows or Linux ARM from source.

For fun, let’s do that.

However, if you’re on a Linux AMD64 machine, the precompiled binary is available on the k8s-tew github repository.

curl -s https://api.github.com/repos/darxkies/k8s-tew/releases/latest \
 | grep "browser_download_url" | cut -d : -f 2,3 | tr -d \" | sudo wget \
 -O /usr/local/bin/k8s-tew -qi -

sudo chmod a+x /usr/local/bin/k8s-tew

Compiling from Source

Thanks to Golang, there’s not much else to do than clone the repository:

git clone https://github.com/darxkies/k8s-tew.git

cd k8s-tew
make build-binaries
> CGO_ENABLED=0 go build -ldflags "-X github.com/darxkies/k8s-tew/pkg/version.Version=2.4.12-4-g25cfcb40 -s -w" -o k8s-tew github.com/darxkies/k8s-tew/cmd/k8s-tew

file ./k8s-tew
./k8s-tew: Mach-O 64-bit executable arm64

./k8s-tew
Version: 2.4.12-4-g25cfcb40
OS: /

Kubernetes - The Easier Way
[...]

Configuration

Once the CLI is installed (whether you compiled or downloaded it), we can tackle the real work.

I’m proposing the “simple multi-node” version (without HA for the control plane) using 3 virtual machines I created on a cloud provider.

Note: the workflow is different depending on whether you run the CLI in “local” mode or not. The workflow to follow is:

initialize –> configure –> generate –> deploy

The ./k8s-tew initialize command will generate a configuration file assets/etc/k8s-tew/config.yaml.

./k8s-tew initialize
INFO[0000] Saved config
INFO[0000] Done

This file contains all the configuration options for our cluster, whether for Kubernetes itself or for components that will be installed afterward.

Among the components chosen by the k8s-tew developer, we find calico (CNI plugin), metalLB (for LoadBalancer type Services), Ceph and Minio (storage), Velero (backup):

version: 2.4.0
cluster-id: 0df51ff0-3bae-4a33-b41f-66fb97bd3704
cluster-name: k8s-tew
email: k8s-tew@gmail.com
[...]
kube-state-metrics-count: 1
drain-grace-period-seconds: 0
versions:
 etcd: quay.io/coreos/etcd:v3.5.21
 kubernetes: v1.32.3
[...]
 containerd: 2.0.5
[...]
 velero: docker.io/velero/velero:v1.9.0
[...]
 minio-server: docker.io/minio/minio:RELEASE.2021-03-12T00-00-47Z
 minio-client: docker.io/minio/mc:RELEASE.2021-03-12T03-36-59Z
[...]
 metallb-controller: quay.io/metallb/controller:v0.9.5
 metallb-speaker: quay.io/metallb/speaker:v0.9.5
 ceph: quay.io/ceph/ceph:v19.2.2
[...]
nodes: {}

I only included a few lines, but if you open the file, you’ll notice that the list of pre-installed components is VERY (too) long, even including a wordpress+mysql instance.

The second thing we can note is that for now, the Nodes list is empty.

So we’ll add Nodes with the k8s-tew node-add command:

➜ ~ k8s-tew node-add -n tew1 -i 203.0.113.97 -l controller,node,storage,worker
 INFO[0000] Node added index=0 ip=203.0.113.97
 labels="[controller node storage]" name=tew1 storage-index=0
 INFO[0000] Saved config

➜ ~ k8s-tew node-add -n tew2 -i 203.0.113.49 -l controller,node,storage,worker
 [...]

➜ ~ k8s-tew node-add -n tew3 -i 203.0.113.43 -l controller,node,storage,worker
 [...]

Note: in this example, Nodes tew1, tew2 and tew3 all have all roles (control plane, worker and Ceph storage cluster member). In a production context, this configuration would probably not be recommended, a proper separation of roles would be preferable.

From there, the nodes: section of the configuration file should now contain this:

[...]
nodes:
 tew1:
 ip: 203.0.113.97
 index: 0
 storage-index: 0
 labels:
 - controller
 - node
 - storage
 - worker
 tew2:
 ip: 203.0.113.49
 index: 1
 storage-index: 1
 labels:
 - controller
 - node
 - storage
 - worker
 tew3:
 ip: 203.0.113.43
 index: 2
 storage-index: 2
 labels:
 - controller
 - node
 - storage
 - worker

Generate the files that will allow us to perform the installation (binaries for components, certificates, etc.):

k8s-tew generate
INFO[0000] Generated config entries
INFO[0000] Saved config
INFO[0000] Copied name=k8s-tew
INFO[0000] Downloading name=etcd-v3.5.21-linux-amd64.tar.gz
INFO[0010] Installed name=etcdctl
[...]
INFO[0027] Generated name=kube-scheduler-tew1.yaml
INFO[0027] Generated name=kube-scheduler-tew2.yaml
INFO[0027] Generated name=kube-scheduler-tew3.yaml
INFO[0027] Generated name=kube-proxy-tew1.yaml
INFO[0027] Generated name=kube-proxy-tew2.yaml
INFO[0027] Generated name=kube-proxy-tew3.yaml
INFO[0027] Done

Deployment

From now on, all the configuration files needed for k8s-tew are generated. So all that’s left is to deploy everything with the k8s-tew command.

And that’s when disaster strikes…

./k8s-tew deploy
INFO[0000] Executing remote command name=create-directories node=tew1
ERRO[0000] Failed deploying error="open /Users/zwindler/.ssh/id_rsa: no such file or directory"

Note 1: k8s-tew doesn’t use your ssh agent, but opens a key file present on your machine to connect to remote machines. Moreover, it only supports RSA type keys. Finally, using the root account is mandatory (no possibility of going through a regular user with sudo).

Note 2: if you compiled the binary like me for MacOS, you’re in for a bad joke. The binary is also used to install on remote servers (Linux/amd64 in my case) and the installation will fail miserably (and silently). You can cheat/work around the problem by downloading the binary for Linux and placing it at the path ./assets/opt/k8s-tew/bin/k8s-tew:

curl -s https://api.github.com/repos/darxkies/k8s-tew/releases/latest \
 | grep "browser_download_url" | cut -d : -f 2,3 | tr -d \" | sudo wget \
 -O ./assets/opt/k8s-tew/bin/k8s-tew -qi -

Once this configuration was modified on my virtual machines, I restarted the process:

./k8s-tew deploy
INFO[0000] Executing remote command name=create-directories node=tew1
INFO[0000] Executing remote command name=get-checksums node=tew1
INFO[0000] Executing remote command name=stop-service node=tew1
INFO[0001] Deploying name=k8s-tew.service node=tew1
[...]
NFO[0032] Configuring taint node=tew1
INFO[0040] Configuring taint node=tew2
INFO[0041] Configuring taint node=tew3
INFO[0045] Applying manifest name=kubelet-setup
INFO[0045] Applying manifest name=admin-user-setup
INFO[0045] Applying manifest name=calico-setup
INFO[0045] Applying manifest name=metallb-setup
INFO[0045] Applying manifest name=coredns-setup
[...]
INFO[0117] Applying manifest name=velero-setup
INFO[0124] Applying manifest name=wordpress-setup
INFO[0190] Done

At the end of the process (about 3 minutes), the complete Kubernetes cluster with many components should be operational. You can retrieve a series of variables that will facilitate the use of our cluster (including the KUBECONFIG variable, but also CONTAINER_RUNTIME_ENDPOINT and CONTAINERD_NAMESPACE if in local mode):

➜ ~ kubectl get nodes
NAME STATUS ROLES AGE VERSION
tew1 Ready controller,storage,worker 9m16s v1.32.3
tew2 Ready controller,storage,worker 9m7s v1.32.3
tew3 Ready controller,storage,worker 9m3s v1.32.3

Going Further

k8s-tew allows installing a Kubernetes cluster that already has a large number of third-party components, allowing you to immediately start using it.

All these configuration options are described in the official documentation (darxkies.github.io/k8s-tew/_build/html/usage.html#configuration).

However, k8s-tew is not very flexible for SSH access to perform the installation (not possible to separate internal IPs from external IPs, root account mandatory, no sudo), which limits its interest in many scenarios.

Pros and Cons

Pros	Cons
➕ a complete cluster with lots of pre-installed components (too many)	➖ root account + RSA key mandatory, no sudo, no ed25519
➕ open source go code	➖ golang client that could be multiplatform, but in reality linux amd64 compatible only (without workarounds)
	➖ too many pre-installed components, consumes too many resources (but it’s configurable)
	➖ impossible to distinguish external IP and internal IP in the case of deployments on a cloud provider with firewalling
	➖ no firewall rules management

Kubernetes the Easier Way on Zwindler's Reflection