Weren’t You Writing a Book on Kubernetes?

Yes! But for now, I can’t really talk about it :-p. But I promise as soon as I can, I’ll communicate about it.

However, I can still say that writing this book led me to test A LOT of tools for installing Kubernetes. And as I’ve done before on this blog, I’m going to present one of the tools I tested called: Kubernetes The Easier Way, or k8s-tew.

The most cultured among you will certainly have recognized a reference to the Kubernetes the hard way project by Kelsey Hightower. It’s intentional:

Kubernetes is a fairly complex project. For a newbie it is hard to understand and also to use. While Kelsey Hightower’s Kubernetes The Hard Way, on which this project is based, helps a lot to understand Kubernetes, it is optimized for the use with Google Cloud Platform.

k8s-tew is a CLI that will install all the prerequisites for us to get a working k8s in a few commands, as well as a whole bunch of third-party software.

The project supports several deployment modes:

• on the local host (but requires ports 80 and 443 to be available)
• on a single remote machine
• on multiple remote machines
• HA control plane or not

Prerequisites

The k8s-tew project was developed in golang. In theory, it’s possible to compile it for macOS, Windows or Linux ARM from source.

For fun, let’s do that.

However, if you’re on a Linux AMD64 machine, the precompiled binary is available on the k8s-tew github repository.

curl -s https://api.github.com/repos/darxkies/k8s-tew/releases/latest \
  | grep "browser_download_url" | cut -d : -f 2,3 | tr -d \" | sudo wget \
  -O /usr/local/bin/k8s-tew -qi -

sudo chmod a+x /usr/local/bin/k8s-tew

Compiling from Source

Thanks to Golang, there’s not much else to do than clone the repository:

git clone https://github.com/darxkies/k8s-tew.git

cd k8s-tew
make build-binaries
> CGO_ENABLED=0 go build -ldflags "-X github.com/darxkies/k8s-tew/pkg/version.Version=2.4.12-4-g25cfcb40 -s -w" -o k8s-tew github.com/darxkies/k8s-tew/cmd/k8s-tew 

file ./k8s-tew
./k8s-tew: Mach-O 64-bit executable arm64

./k8s-tew 
Version: 2.4.12-4-g25cfcb40
OS: /

Kubernetes - The Easier Way
[...]

Configuration

Once the CLI is installed (whether you compiled or downloaded it), we can tackle the real work.

I’m proposing the “simple multi-node” version (without HA for the control plane) using 3 virtual machines I created on a cloud provider.

Note: the workflow is different depending on whether you run the CLI in “local” mode or not. The workflow to follow is:

• initialize –> configure –> generate –> deploy

The ./k8s-tew initialize command will generate a configuration file assets/etc/k8s-tew/config.yaml.

./k8s-tew initialize
INFO[0000] Saved config                                 
INFO[0000] Done

This file contains all the configuration options for our cluster, whether for Kubernetes itself or for components that will be installed afterward.

Among the components chosen by the k8s-tew developer, we find calico (CNI plugin), metalLB (for LoadBalancer type Services), Ceph and Minio (storage), Velero (backup):

version: 2.4.0
cluster-id: 0df51ff0-3bae-4a33-b41f-66fb97bd3704
cluster-name: k8s-tew
email: k8s-tew@gmail.com
[...]
kube-state-metrics-count: 1
drain-grace-period-seconds: 0
versions:
  etcd: quay.io/coreos/etcd:v3.5.21
  kubernetes: v1.32.3
[...]
  containerd: 2.0.5
[...]
  velero: docker.io/velero/velero:v1.9.0
[...]
  minio-server: docker.io/minio/minio:RELEASE.2021-03-12T00-00-47Z
  minio-client: docker.io/minio/mc:RELEASE.2021-03-12T03-36-59Z
[...]
  metallb-controller: quay.io/metallb/controller:v0.9.5
  metallb-speaker: quay.io/metallb/speaker:v0.9.5
  ceph: quay.io/ceph/ceph:v19.2.2
[...]
nodes: {}

I only included a few lines, but if you open the file, you’ll notice that the list of pre-installed components is VERY (too) long, even including a wordpress+mysql instance.

The second thing we can note is that for now, the Nodes list is empty.

So we’ll add Nodes with the k8s-tew node-add command:

➜  ~ k8s-tew node-add -n tew1 -i 203.0.113.97 -l controller,node,storage,worker
  INFO[0000] Node added index=0 ip=203.0.113.97 
  labels="[controller node storage]" name=tew1 storage-index=0
  INFO[0000] Saved config

➜  ~ k8s-tew node-add -n tew2 -i 203.0.113.49 -l controller,node,storage,worker
  [...]

➜  ~ k8s-tew node-add -n tew3 -i 203.0.113.43 -l controller,node,storage,worker
  [...]

Note: in this example, Nodes tew1, tew2 and tew3 all have all roles (control plane, worker and Ceph storage cluster member). In a production context, this configuration would probably not be recommended, a proper separation of roles would be preferable.

From there, the nodes: section of the configuration file should now contain this:

[...]
nodes:
  tew1:
    ip: 203.0.113.97
    index: 0
    storage-index: 0
    labels:
    - controller
    - node
    - storage
    - worker
  tew2:
    ip: 203.0.113.49
    index: 1
    storage-index: 1
    labels:
    - controller
    - node
    - storage
    - worker
  tew3:
    ip: 203.0.113.43
    index: 2
    storage-index: 2
    labels:
    - controller
    - node
    - storage
    - worker

Generate the files that will allow us to perform the installation (binaries for components, certificates, etc.):

k8s-tew generate
INFO[0000] Generated config entries                     
INFO[0000] Saved config                                 
INFO[0000] Copied                                        name=k8s-tew
INFO[0000] Downloading                                   name=etcd-v3.5.21-linux-amd64.tar.gz
INFO[0010] Installed                                     name=etcdctl
[...]
INFO[0027] Generated                 name=kube-scheduler-tew1.yaml
INFO[0027] Generated                 name=kube-scheduler-tew2.yaml
INFO[0027] Generated                 name=kube-scheduler-tew3.yaml
INFO[0027] Generated                 name=kube-proxy-tew1.yaml
INFO[0027] Generated                 name=kube-proxy-tew2.yaml
INFO[0027] Generated                 name=kube-proxy-tew3.yaml
INFO[0027] Done 

Deployment

From now on, all the configuration files needed for k8s-tew are generated. So all that’s left is to deploy everything with the k8s-tew command.

And that’s when disaster strikes…

./k8s-tew deploy
INFO[0000] Executing remote command                      name=create-directories node=tew1
ERRO[0000] Failed deploying                              error="open /Users/zwindler/.ssh/id_rsa: no such file or directory"

Note 1: k8s-tew doesn’t use your ssh agent, but opens a key file present on your machine to connect to remote machines. Moreover, it only supports RSA type keys. Finally, using the root account is mandatory (no possibility of going through a regular user with sudo).

Note 2: if you compiled the binary like me for MacOS, you’re in for a bad joke. The binary is also used to install on remote servers (Linux/amd64 in my case) and the installation will fail miserably (and silently). You can cheat/work around the problem by downloading the binary for Linux and placing it at the path ./assets/opt/k8s-tew/bin/k8s-tew:

curl -s https://api.github.com/repos/darxkies/k8s-tew/releases/latest \
  | grep "browser_download_url" | cut -d : -f 2,3 | tr -d \" | sudo wget \
  -O ./assets/opt/k8s-tew/bin/k8s-tew -qi -

Once this configuration was modified on my virtual machines, I restarted the process:

./k8s-tew deploy
INFO[0000] Executing remote command  name=create-directories node=tew1
INFO[0000] Executing remote command  name=get-checksums node=tew1
INFO[0000] Executing remote command  name=stop-service node=tew1
INFO[0001] Deploying                 name=k8s-tew.service node=tew1
[...]
NFO[0032] Configuring taint                             node=tew1
INFO[0040] Configuring taint                             node=tew2
INFO[0041] Configuring taint                             node=tew3
INFO[0045] Applying manifest                             name=kubelet-setup
INFO[0045] Applying manifest                             name=admin-user-setup
INFO[0045] Applying manifest                             name=calico-setup
INFO[0045] Applying manifest                             name=metallb-setup
INFO[0045] Applying manifest                             name=coredns-setup
[...]
INFO[0117] Applying manifest                             name=velero-setup
INFO[0124] Applying manifest                             name=wordpress-setup
INFO[0190] Done  

At the end of the process (about 3 minutes), the complete Kubernetes cluster with many components should be operational. You can retrieve a series of variables that will facilitate the use of our cluster (including the KUBECONFIG variable, but also CONTAINER_RUNTIME_ENDPOINT and CONTAINERD_NAMESPACE if in local mode):

➜  ~ kubectl get nodes
NAME   STATUS   ROLES                       AGE     VERSION
tew1   Ready    controller,storage,worker   9m16s   v1.32.3
tew2   Ready    controller,storage,worker   9m7s    v1.32.3
tew3   Ready    controller,storage,worker   9m3s    v1.32.3

Going Further

k8s-tew allows installing a Kubernetes cluster that already has a large number of third-party components, allowing you to immediately start using it.

All these configuration options are described in the official documentation (darxkies.github.io/k8s-tew/_build/html/usage.html#configuration).

However, k8s-tew is not very flexible for SSH access to perform the installation (not possible to separate internal IPs from external IPs, root account mandatory, no sudo), which limits its interest in many scenarios.

Pros and Cons