ClusterConfiguration: introduction to kubeadm APIs for installing your clusters

Creating Kubernetes clusters with `kubeadm`, but without endless CLI arguments

One of the first articles I wrote on Kubernetes was about installing a cluster with kubeadm (in french). At the time, I had tested with the basic options and it already worked very very well.

But when you start having clusters a bit closer to production, you quickly find yourself adding lots of options to the CLI.

Worse, at a certain level of complexity, some options don’t work together!

Note: this happened to me when I tried to automate the kubeadm installation with Ansible. We can debate the point of automating a kubeadm install with Ansible when Kubespray exists, cf this other article I wrote in 2017 (in french as well) but that’s beyond the scope of this article…

To do things properly, stop passing the entire world as arguments

Clearly, after 5 or 6 options, it’s time to wonder if it wouldn’t be better to use a configuration file to manage our clusters.

And that’s convenient, because kubeadm has an API (several actually), so we’ll be able to write YAML before even having our Kubernetes cluster operational. YAY! (say the YAML engineers)

Concretely, how does it look?

Rather than passing all our arguments in our kubeadm command, we’ll fill in a kubeadmcfg.yaml file that will serve as our configuration. Actually, we’ll even have several, since the nodes don’t all have the same role in our cluster creation workflow.

The first file will be a file that contains the ClusterConfiguration (cf official docs)

Note, you can retrieve the default values file with the command:

$ kubeadm config print init-defaults
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
localAPIEndpoint:
  advertiseAddress: <your-control-plane-ip-address>
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: node
  taints: null
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
  timeoutForControlPlane: 4m0s
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kubernetesVersion: 1.28.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}

From there, we’ll be able to customize our control plane, via sections corresponding to each control plane component:

apiServer
controllerManager
scheduler
etcd

What could this be useful for?

Obviously the possibilities are numerous and I’m not going to list them all. But I’ll give you some examples.

The most obvious example is obviously adding arguments to the different components. Here, I’m adding OIDC authentication to the cluster and my feature gates to enable alpha APIs:

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
[...]
apiServer:
  extraArgs:
    feature-gates: <kube_feature_gates>
    oidc-issuer-url: <kube_oidc_issuer_url>
    oidc-client-id: <kube_oidc_client_id>
    oidc-username-claim: email
    oidc-groups-claim: groups
[...]

If you have multiple virtual IPs that allow you to connect to your Kubernetes cluster, it will probably be necessary to support multiple certSANs for the TLS part:

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
[...]
apiServer:
  certSANs:
  - <control_plane_endpoint_ip>
  - <control_plane_endpoint_ip_for_humans>
[...]

In case your machines have multiple IPs, it may be necessary to indicate the correct IPs for contacting the cluster. Small trap: you need to modify in another API that I’ve omitted so far: InitConfiguration!

---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
[...]
localAPIEndpoint:
  advertiseAddress: <app_ip>
  bindPort: 6443
[...]

If you followed my previous article on my little production issue with etcd being spammed by Kyverno, you’ll know that you need to tune the etcd database. Like this for example:

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
[...]
etcd:
  local:
    extraArgs:
      quota-backend-bytes: "5368709120"
      auto-compaction-retention: "1"
      auto-compaction-mode: periodic
[...]

And on my side, I need to change the domain name of Kubernetes services. I do it like this:

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
[...]
networking:
  dnsDomain: <kube_dns_domain>
[...]

Once it’s properly configured as we want:

/usr/bin/kubeadm init --config=/etc/kubernetes/kubeadmcfg.yaml --upload-certs --skip-certificate-key-print

And then?

ClusterConfiguration is cool, but there are other useful APIs with kubeadm, as I said earlier.

When we’re going to add nodes to the control plane, we won’t be able to reuse the ClusterConfiguration and the InitConfiguration, because they serve for the ENTIRE cluster and are unique. We already configured them when we did the init of the first control plane node.

So we’ll use the JoinConfiguration, which also allows us to define important things like:

localAPIEndpoint.advertiseAddress if we have multiple IPs on the machine (example above).
discovery.bootstrapToken.token (and caCertHashes) which will facilitate the automation of node joins

---
apiVersion: kubeadm.k8s.io/v1beta3
kind: JoinConfiguration
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
  bootstrapToken:
    apiServerEndpoint: <control_plane_endpoint>
    token: <kubeadm_token>
    caCertHashes: 
      - sha256:<kubeadm_cacert_hash>
    unsafeSkipCAVerification: false
nodeRegistration:
  name: <nodename>
controlPlane:
  certificateKey: <kubeadm_certificate_key>
  localAPIEndpoint: 
    advertiseAddress: <app_ip>
    bindPort: 6443

Once this file is created on the machine of the control plane that will join the first one, we can run the kubeadm join command which should go well ;-).

kubeadm join <control_plane_endpoint_ip> --config /etc/kubernetes/kubeadmcfg.yaml

Conclusion

Using kubeadm configuration files is not trivial because it’s poorly documented (I find, the docs are scattered across multiple pages and the examples aren’t very clear in my case).

I won’t lie, I pulled my hair out and asked for help several times on this.

Of course, you can go study the official docs Customizing components with the kubeadm API and kubeadm Configuration (v1beta3) but I still had trouble understanding HOW to use it, the first time.

I hope this article can help better understand how the different APIs work together and how to use them.