Politique on Zwindler's Reflection

Kubernetes Compliance Policies with Kyverno - part 2

Mon, 05 Sep 2022 06:00:00 +0200

Part 1 recap

In the previous article, I introduced Kyverno, a “cloud native” tool for managing compliance policies on Kubernetes.

To illustrate the tool’s capabilities, I used an example: preventing two Ingresses from using the same entry URL. I had done the same exercise with OPA in an earlier article (Open Policy Agent (OPA) and its Kubernetes counterpart, Gatekeeper) and compared the two syntaxes (Kyverno vs rego) at the end.

I have a clear preference for Kyverno’s syntax, even though OPA has been around longer and isn’t limited to Kubernetes (Kyverno is Kube-only).

Going a bit further

Beyond my example, I mentioned that Kyverno provides a very large number of pre-written policies (191 at the time of writing).

kyverno.io/policies

By default, most available policies are set to “audit” mode initially. Don’t forget to switch them to “enforce” mode once your cluster is mature enough.

Obviously, when you’re just getting started, you don’t really know where to begin. The simplest approach is to deploy the kyverno/kyverno-policies Helm chart, which contains the Kyverno equivalent of the “Pod Security Standards” (PSPs being officially deprecated since 1.21 I think, and no longer usable in 1.25).

helm install kyverno-policies kyverno/kyverno-policies -n kyverno
[...]
Thank you for installing kyverno-policies v2.5.2 😀

We have installed the "baseline" profile of Pod Security Standards and set them in audit mode.

Visit https://kyverno.io/policies/ to find more sample policies.

Here’s what was deployed (the last one was already there from the previous article):

kubectl get clusterpolicy
NAME BACKGROUND ACTION READY
disallow-capabilities true audit true
disallow-host-namespaces true audit true
disallow-host-path true audit true
disallow-host-ports true audit true
disallow-host-process true audit true
disallow-privileged-containers true audit true
disallow-proc-mount true audit true
disallow-selinux true audit true
restrict-apparmor-profiles true audit true
restrict-seccomp true audit true
restrict-sysctls true audit true
unique-ingress-host false enforce true

With that, we already have plenty to work with to educate our users (developers) on producing secure Kubernetes deployments.

User experience

One of the annoying things about both tools is that out of the box, they’re not very user friendly. Personally, running a kubectl get events command (or kubectl get events --field-selector=reason=PolicyViolation in Kyverno’s case) to figure out why a deployment failed doesn’t bother me much (I almost enjoy it).

But for some people, it’s a bit off-putting, and I can understand that.

My goal is to make infrastructure simpler for people whose job it isn’t, and imposing my way of working isn’t necessarily the best approach.

Often, to help adopt a new tool or practice, the simplest thing is to provide a graphical interface.

And it turns out Kyverno has a tool for that.

Kyverno policy reporter

In reality, the tool wasn’t originally designed for that exact purpose. But you’ll see it’s actually even better.

Historically, it’s called policy reporter because the developers needed a tool to easily and quickly export the PolicyReports I mentioned in the previous article:

kubectl describe policyreport polr-ns-default
Name: polr-ns-default
Namespace: default
Labels: managed-by=kyverno
Annotations: <none>
API Version: wgpolicyk8s.io/v1alpha2
Kind: PolicyReport
[...]
Results:
 Category: Sample
 Message: The Ingress host name must be unique.
 Policy: unique-ingress-host
 Resources:
 API Version: networking.k8s.io/v1
 Kind: Ingress
 Name: ingress2
[...]
Summary:
 Error: 0
 Fail: 2
 Pass: 0
 Skip: 2
 Warn: 0

The policy reporter’s role is to export PolicyReports to several external sources for easier processing:

Grafana Loki
Elasticsearch
Slack
Discord
MS Teams
Policy Reporter UI
S3

I’ll cover 2 of them: Slack and Policy Reporter UI

Policy Reporter UI

I’ll start with the UI since it’s the simplest to set up. You just need to deploy the Policy Reporter Helm chart, making sure to specify that you want the UI:

helm repo add policy-reporter https://kyverno.github.io/policy-reporter
helm repo update
helm install -n kyverno policy-reporter policy-reporter/policy-reporter --set kyvernoPlugin.enabled=true --set ui.enabled=true --set ui.plugins.kyverno=true

kubectl -n kyverno get pods
NAME READY STATUS RESTARTS AGE
kyverno-5f8bfd6fc5-xp6bm 1/1 Running 1 (59m ago) 33d
policy-reporter-74cc9bf4b9-9ggwh 1/1 Running 0 18s
policy-reporter-kyverno-plugin-6b48c4bc5f-5rwfb 1/1 Running 0 18s
policy-reporter-ui-5c9449d58-f9fwf 1/1 Running 0 18s

We could obviously add an Ingress, but if you’re lazy like me during this demo, a simple port-forward will do.

kubectl port-forward service/policy-reporter-ui 8082:8080 -n kyverno

The UI itself is fairly intuitive. It includes notably:

a Dashboard with a big global indicator showing how many policy violations you have
a page with your “policy reports”, filterable by namespace or policy
the list of your policies
…

Slack

You can immediately see the value of alerting administrators in Slack (or Teams…) when a policy violation occurs.

The configuration is fairly trivial — you just need to add a webhook in Slack and a small piece of configuration in Kyverno:

kyvernoPlugin:
 enabled: true
ui:
 enabled: true
 plugins:
 kyverno: true
target:
 slack:
 webhook: "https://hooks.slack.com/services/T0xxxxxxxx"
 minimumPriority: "medium"
 skipExistingOnStartup: true
 sources:
 - kyverno

And we re-apply all this to our Helm release:

helm upgrade -n kyverno policy-reporter policy-reporter/policy-reporter -f kyverno-values.yaml

Note: If you don’t know how to do this, you need to be admin of the Slack workspace, create an application at https://api.slack.com/apps, enable Incoming Webhooks, and create one with permissions on a channel.

How about we trigger a policy violation, just to see?

To keep it simple, I went back to the example from the previous article. I switch my unique-ingress-host policy back to audit mode and recreate the problematic ingress:

kubectl edit clusterpolicy unique-ingress-host
kubectl apply -f ingress2.yaml

Instantly, 2 failures show up in the Kyverno UI, along with messages in Slack:

Victory :)

Additional Resources

Kubernetes Compliance Policies with Kyverno

Mon, 01 Aug 2022 06:00:00 +0000

Introduction

If the title of this article sounds familiar, it’s probably because I already used it 2 years ago when talking about another compliance policy management tool: Open Policy Agent (OPA) and its Kubernetes counterpart, Gatekeeper.

And I won’t lie, I haven’t done much OPA since that 2020 article…

The first reason is that I changed companies right at that time and had other priorities at my new employer.

The second reason, probably more accurate but perhaps less flattering, is that OPA is not exactly user friendly. Especially because of rego, the compliance policy language.

Since then, I discovered Kyverno, a newcomer in the “cloud native” game (recently promoted to “CNCF sandbox” level) that offers compliance policies for Kubernetes only (unlike OPA + rego which are general-purpose), written in YAML.

With Kyverno, policies are managed as Kubernetes resources and no new language is required to write policies.

Take that, rego ;-)

Note: This article will remain relatively simple, it’s an introduction. We’ll go further in subsequent articles (writing your own policy, the UI, etc).

I’m not going to innovate much compared to the OPA article and will start from the same use case: preventing two developers from using the same URL for their web app deployed on Kubernetes (because we’ve seen it’s pretty bad when that happens).

And you’ll see that the initial learning curve is MUUUCH more accessible with Kyverno than with OPA (even though it might annoy half of my Twitter followers that I’m once again talking about YAML).

Prerequisites

Same setup as for OPA / Gatekeeper, you need Kube 1.14 to use Kyverno. Version 1.14 wasn’t the latest back in 2020, it’s practically prehistoric by now ;-).

Your Kubernetes cluster version must be above v1.14 which adds webhook timeouts.

Obviously, you also need admin access to the cluster, because we’ll have to create Validating Admission Webhooks and CRDs. Again, I’ll refer you to the previous article and the official documentation if this doesn’t ring a bell.

In short, we’re going to insert ourselves into Kubernetes’ deployment process to prevent people from doing things that don’t follow your best practices (with webhooks) and we’ll add extra APIs to extend Kubernetes (the CRDs), so we can describe what’s not allowed.

Deployment

Installation-wise, we’re keeping it simple and efficient.

Kyverno offers several Helm Charts:

one for Kyverno itself
one for Policy Reporter
one containing a “pack” of default rules, configured as “non-blocking” (basically the equivalent of what you could do with Pod Security Policies)

For this article, I’m only interested in Kyverno itself. But as I mentioned above, we’ll discuss the other two in a future post.

Let’s start by adding Kyverno itself:

$ helm repo add kyverno https://kyverno.github.io/kyverno/
$ helm repo update
$ helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace
 NAME: kyverno
 LAST DEPLOYED: Mon Jul 25 21:55:11 2022
 NAMESPACE: kyverno
 STATUS: deployed
 REVISION: 1
 NOTES:
 Chart version: v2.5.2
 Kyverno version: v1.7.2

 Thank you for installing kyverno! Your release is named kyverno.
 ⚠️ WARNING: Setting replicas count below 3 means Kyverno is not running in high availability mode.

 💡 Note: There is a trade-off when deciding which approach to take regarding Namespace exclusions. Please see the documentation at https://kyverno.io/docs/installation/#security-vs-operability to understand the risks.

Note: you’ll notice the WARNING during install. By default, Kyverno is deployed with a single replica. If it crashes, you can no longer deploy or modify anything on your cluster (it happened to me). The solution to recover is to delete Kyverno’s webhooks to unblock the situation. And of course, give it more room (higher limits, especially for RAM) and definitely 3 replicas (or more)…

Note 2: It’s probably best to exclude certain namespaces (kube-system, kyverno, …) from Kyverno’s scope to avoid nasty surprises. Thibault Lengagne’s blogpost at Padok also discusses this (link at the bottom of the article).

What did we deploy?

At first glance, not much. If you run kubectl -n kyverno get all, you’ll find just a Pod (+ReplicaSet +Deployment) and two Services.

But behind the scenes, we’ve deployed much more than that ;)

Looking for CRDs, you’ll see we have quite a few new objects at our disposal.

$ kubectl api-resources --api-group=kyverno.io
 NAME SHORTNAMES APIVERSION NAMESPACED KIND
 clusterpolicies cpol kyverno.io/v1 false ClusterPolicy
 clusterreportchangerequests crcr kyverno.io/v1alpha2 false ClusterReportChangeRequest
 generaterequests gr kyverno.io/v1 true GenerateRequest
 policies pol kyverno.io/v1 true Policy
 reportchangerequests rcr kyverno.io/v1alpha2 true ReportChangeRequest
 updaterequests ur kyverno.io/v1beta1 true UpdateRequest

Similarly, we also have new webhooks:

$ kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io
 NAME WEBHOOKS AGE
 kyverno-policy-validating-webhook-cfg 1 8m48s
 kyverno-resource-validating-webhook-cfg 2 8m48s

$ kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io
 NAME WEBHOOKS AGE
 kyverno-policy-mutating-webhook-cfg 1 11m
 kyverno-resource-mutating-webhook-cfg 2 11m
 kyverno-verify-mutating-webhook-cfg 1 11m

What’s it for? How does it work?

To simplify, we’ll describe what we don’t want to allow on our cluster using the ClusterPolicy CRD.

Based on the rules we write, the webhooks will insert themselves into Kubernetes’ normal resource deployment process to either allow or reject them.

Kubernetes Blog - A Guide to Kubernetes Admission Controllers

Ingress with the same URL example

Enough theory, the easiest way is to show you. As I mentioned at the beginning of the article, to compare the two products, I’m going to start from the same use case.

In my Kubernetes clusters, I want to prevent two applications from creating an Ingress with the same URL, because in my case, that’s not allowed. It’s a sign of a copy/paste error from one Helm Chart to another (or any other human error).

The consequence of this error is a conflict in routing incoming HTTP requests, and you end up with some kind of janky round robin where half the requests hit the wrong backend. In production, that’s a bad look, and you risk ending up with customers complaining on Twitter that the application is down ;-).

Just as OPA provides a community policy to address this problem, Kyverno has the same.

We’ll look in more detail in the next article at how these rules are built. For now, we’ll just deploy this rule in “audit” mode and see if it works.

spec: validationFailureAction: audit

$ kubectl apply -f https://raw.githubusercontent.com/kyverno/policies/main/other/unique-ingress-paths/unique-ingress-paths.yaml
 clusterpolicy.kyverno.io/unique-ingress-host created

If you have a Kubernetes cluster at hand, I suggest you apply these two Ingresses, configured to receive traffic from the same URL (toto.example.org):

$ for counter in 1 2; do
cat > ingress${counter}.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: ingress${counter}
spec:
 rules:
 - host: toto.example.org
 http:
 paths:
 - path: /
 pathType: Prefix
 backend:
 service:
 name: service${counter}
 port:
 number: 4200
EOF
done

$ kubectl apply -f ingress1.yaml
 ingress.networking.k8s.io/ingress1 created
$ kubectl apply -f ingress2.yaml
 ingress.networking.k8s.io/ingress2 created

Normally, nothing happens. Kyverno doesn’t prevent you from creating these two Ingresses since we’re in “validationFailureAction: audit” mode (even if the backends don’t actually exist behind them, that doesn’t matter).

$ kubectl get ingress
 NAME CLASS HOSTS ADDRESS PORTS AGE
 ingress1 <none> toto.example.org 80 90s
 ingress2 <none> toto.example.org 80 87s

However, if you check the Events, you’ll find PolicyViolation entries:

$ kubectl get events --field-selector=reason=PolicyViolation
LAST SEEN TYPE REASON OBJECT MESSAGE
118s Warning PolicyViolation ingress/ingress1 policy unique-ingress-host/check-single-host fail: The Ingress host name must be unique.
118s Warning PolicyViolation ingress/ingress1 policy unique-ingress-host/check-single-host fail: The Ingress host name must be unique.
118s Warning PolicyViolation clusterpolicy/unique-ingress-host Ingress default/ingress1: [check-single-host] fail
115s Warning PolicyViolation clusterpolicy/unique-ingress-host Ingress default/ingress2: [check-single-host] fail

The fact that we have 2 Ingresses pointing to the same URL is properly detected and shows up in Kubernetes Events. Nice :)

In a much more verbose format, you can also retrieve PolicyReports, which say the same thing in an even less readable way.

kubectl describe policyreport polr-ns-default
Name: polr-ns-default
Namespace: default
Labels: managed-by=kyverno
Annotations: <none>
API Version: wgpolicyk8s.io/v1alpha2
Kind: PolicyReport
[...]
Results:
 Category: Sample
 Message: The Ingress host name must be unique.
 Policy: unique-ingress-host
 Resources:
 API Version: networking.k8s.io/v1
 Kind: Ingress
 Name: ingress2
[...]
Summary:
 Error: 0
 Fail: 2
 Pass: 0
 Skip: 2
 Warn: 0

Let’s clean up for what’s next:

$ kubectl delete ingress ingress1 ingress2

“The law is me”

Imagine you’ve now thoroughly tested your policies, your developers no longer generate manifests that don’t follow your best practices (e.g., no root containers, mandatory labels present, correct limits/requests, etc). Your cluster is clean!

You can switch your compliance policies to “enforcing” mode. Unlike “audit” mode, this mode is blocking, and manifests that don’t comply with the policies will be rejected by Kubernetes.

$ kubectl patch clusterpolicies.kyverno.io unique-ingress-host --patch '{"spec":{"validationFailureAction":"enforce"}}' --type merge

Yes, I’m using “patch” instead of “edit” to show off ;-)

And let’s test again:

$ kubectl apply -f ingress1.yaml
 ingress.networking.k8s.io/ingress1 created

So far, so good

$ kubectl apply -f ingress2.yaml
Error from server: error when creating "ingress2.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Ingress/default/ingress2.yaml was blocked due to the following policies

unique-ingress-host:
 check-single-host: The Ingress host name must be unique.

AND BAM!

Conclusion

In this basic (but genuinely useful) example, we can just as easily prevent users from creating Ingresses with the same URL as we could with OPA.

So one might ask “why prefer Kyverno?”. Beyond some other nice features we’ll explore in future posts, my answer fits in a single image.

My (very personal) take is that rego is unreadable and verbose, and therefore unmaintainable. Again, this is personal, but I’ve never managed to get into it.

Without going as far as idealizing Kyverno’s language*, I find it is clearly much simpler to write, but more importantly, to read. If you ever need to modify/write/maintain your own policies, this will be very handy.

(*Yes, there is a language, especially for pattern matching… and as an annoying point, it’s not jsonpath like kubectl, but JMESPath, similar but not identical…)

Additional Resources

Kubernetes Compliance Policies with OPA and Gatekeeper

Mon, 20 Jul 2020 06:35:00 +0000

Introduction

One of the problems I encountered while managing Kubernetes clusters shared across multiple teams was that the manifests being deployed were not always perfectly controlled.

In some cases, manifests were copy/pasted from other teams, sometimes causing small surprises (no label indicating the team responsible for the app, incorrect Ingress information…). One of the solutions we found was to maintain an up-to-date, as-generic-as-possible Helm Chart, with only values for developers to fill in. This Chart is jointly maintained by Ops and tech leads.

However, maintaining this template is complex, and some applications diverge too much from the general framework to comply, even by forking and modifying parts of it. And some issues like verifying images or registries running on the platform were not addressed by this solution.

Open Policy Agent / Gatekeeper

https://www.thingiverse.com/thing:1236133

Rather than trying at all costs to unify the way applications are deployed in the cluster (and having to manage exceptions), another approach is to add a tool that would verify, at deployment time, that what’s being deployed follows the company’s best practices in terms of security and configuration.

As you may have guessed, the tool for this is Open Policy Agent (or OPA). It’s an open-source generic policy engine (CNCF project at the Incubating stage).

Since it’s a generic engine, there’s also another project, Gatekeeper, which handles the interaction between OPA and Kubernetes.

Prerequisites

To use Gatekeeper, you should have a minimum Kubernetes version of 1.14, which adds webhook timeouts.

If you’re on an earlier version, things get complicated. There’s a bug, fixed in 1.14, that could crash your OPA/Gatekeeper setup. For this reason, if you’re not on 1.14 (and can’t update your platform), you’ll need to skip Gatekeeper (using K8s Policy Controller for instance). That’s a bit trickier and outside the scope of this tutorial.

The “V2”, without Gatekeeper, with Kube-Mgmt and OPA

Of course, you also need admin access on the cluster. Gatekeeper offers a simple way to test your permissions by trying to grant yourself admin rights.

wait for it

kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin \
--user [your user]

Quick tip: if you don’t know the name of the access account you’re currently using, you can run this command:

kubectl config view --template='{{ range .contexts }}{{ if eq .name "'$(kubectl config current-context)'" }}Current user: {{ .context.user }}{{ end }}{{ end }}'
Current user: clusterUser_zwindlerk8s_rg_zwindlerk8s

Installation

I set up a small AKS cluster in version 1.14 for the occasion and deployed the prepackaged version of Gatekeeper.

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml

Tada! And just like that, Gatekeeper is installed on your cluster. Note: The official documentation provides steps to build your own image.

What’s in my cluster?

Taking a closer look at what the manifest does, here’s what you deploy when running the above command:

a Namespace gatekeeper-system
a ServiceAccount gatekeeper-admin, associated with a Role and a ClusterRole gatekeeper-manager-role via (respectively) a RoleBinding and ClusterRoleBinding gatekeeper-manager-rolebinding
a Deployment gatekeeper-controller-manager and gatekeeper-audit
a Service gatekeeper-webhook-service
a Secret gatekeeper-webhook-server-cert containing a certificate

Nothing exceptional so far. Where it gets interesting is:

two CRDs (Custom Resource Definitions): configs.config.gatekeeper.sh and constrainttemplates.templates.gatekeeper.sh
a validating webhook configuration gatekeeper-validating-webhook-configuration

Validating Webhook

The validation webhook first. The first time I heard about validating webhooks was during the talk 101 Ways to “Break and Recover” Kubernetes Cluster at Kubecon 2018.

The problems these Oath (formerly Yahoo) employees were facing were identical to mine. Some teams, when pulling (a.k.a. copy/pasting) manifests from other teams, forgot to change the Ingress URLs. This resulted in user requests being distributed randomly between 2 completely different services.

The solution proposed at that conference was to use Kubernetes’ Validating/Mutating Admission Webhooks (roughly stable since 1.11).

Back in 2018, when digging into the topic, I found very little documentation about it beyond the official docs (I’ve found more since). Another issue: Admission webhooks require developing your own Controller, which wasn’t very accessible (see links at the end of the article).

Fortunately, Gatekeeper and OPA will handle that for us.

Custom Resource Definition

For those unfamiliar with CRDs, they are simply extensions of the Kubernetes API. The big advantage of CRDs is that they provide a way for third-party vendors to add their own logic inside Kubernetes.

I already talked about CRDs in my article on Rook which allows automated management of Ceph clusters (and more!) via the CephCluster CRD. All common administration tasks are integrated into Kubernetes (even though Kube has no native knowledge of them), managed by a Controller included in Rook, and configured via CRDs.

Gatekeeper will add 3 components to our Kubernetes:

a Config object
a ConstraintTemplate object
a Constraint object

Config

The principle of Gatekeeper, as shown in the following diagram, is that it hooks into your Kubernetes API server and “synchronizes” events for new object creation, looking for objects that don’t comply with your compliance policies.

Source: kubernetes.io/blog

By default, if you don’t configure it, Gatekeeper doesn’t listen to events on any Kubernetes objects… so it will do… absolutely nothing!

Let’s fix that right away by telling Gatekeeper to listen to all events on Ingresses:

cat gatekeeper/demo/basic/sync.yaml
apiVersion: config.gatekeeper.sh/v1alpha1
kind: Config
metadata:
name: config
namespace: "gatekeeper-system"
spec:
sync:
syncOnly:
- group: "extensions"
version: "v1beta1"
kind: "Ingress"
- group: "networking.k8s.io"
version: "v1beta1"
kind: "Ingress"

ConstraintTemplate

Before you can define a constraint, you must first define a ConstraintTemplate, which describes both the Rego that enforces the constraint and the schema of the constraint.

The idea here is that before you start creating constraints on the cluster, you need to create constraint templates. Think of these templates as the object that links the function (the OPA rego code) on one side with parameters on the other.

This extra step allows us to potentially reuse the same template for several different constraints.

Your first template

For the first time, we can use the sample templates provided on the OPA/Gatekeeper GitHub, which enforces the presence of a label on a given Kube object type, since it’s simpler (k8srequiredlabels_template.yaml).

However, since the beginning of this article, I’ve been talking about preventing teams from using the same FQDN in Ingresses for different services. Because if that were to happen, as a reminder, we’d end up with some kind of janky load balancer redirecting half of user requests to one application and the other half to another for the same URL.

And it just so happens that this template will help us ensure that!

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
name: k8suniqueingresshost
spec:
crd:
spec:
names:
kind: K8sUniqueIngressHost
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8suniqueingresshost
identical(obj, review) {
obj.metadata.namespace == review.object.metadata.namespace
obj.metadata.name == review.object.metadata.name
}
violation[{"msg": msg}] {
input.review.kind.kind == "Ingress"
re_match("^(extensions|networking.k8s.io)$", input.review.kind.group)
host := input.review.object.spec.rules[_].host
other := data.inventory.namespace[ns][otherapiversion]["Ingress"][name]
re_match("^(extensions|networking.k8s.io)/.+$", otherapiversion)
other.spec.rules[_].host == host
not identical(other, input.review)
msg := sprintf("ingress host conflicts with an existing ingress <%v>", [host])
}

At first glance, it’s pretty dense. But then again, I didn’t pick the simplest example either, and if you re-watch the video Kubecon 2019 | Intro: Open Policy Agent - Rita Zhang, Microsoft & Max Smythe, Google, it’s actually fairly easy to understand.

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/agilebank/dryrun/k8suniqueingresshost_template.yaml
constrainttemplate.templates.gatekeeper.sh/k8suniqueingresshost created
kubectl get constrainttemplate
NAME AGE
k8suniqueingresshost 23m

And a Constraint

Now that we have our template, we can simply instantiate it with the variables we’re interested in.

Sticking with the demo example, we ensure that across the entire cluster, no URL is used twice for two different Ingresses by instantiating this constraint:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sUniqueIngressHost
metadata:
name: unique-ingress-host
spec:
enforcementAction: dryrun
match:
kinds:
- apiGroups: ["extensions", "networking.k8s.io"]
kinds: ["Ingress"]

I’m going to slightly modify this Constraint because, as you can see, it defines an enforcementAction: dryrun, which has no immediate effect (it only logs an error for future audit). It’s a great feature for getting up to speed with OPA, but for my demo it’s less fun…

So I apply the file and change the enforcementAction:

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/agilebank/dryrun/unique-ingress-host.yaml
kubectl patch K8sUniqueIngressHost.constraints.gatekeeper.sh unique-ingress-host -p '{"spec":{"enforcementAction":"deny"}}' --type=merge

Let’s try it out

At this point, we should have everything. Gatekeeper listens to the API server for all events on Ingress objects (via the Config). We have a ConstraintTemplate with rego code that checks we don’t already have an identical URL in existing Ingresses, and a Constraint that defines what to do (dryrun) and on which objects (Ingresses).

Let’s run the demo :D

# Create the namespace that will contain the demo objects
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/agilebank/bad_resources/namespace.yaml
namespace/production created
# Create an Ingress with a URL that doesn't exist yet
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/agilebank/dryrun/existing_resources/example.yaml
ingress.extensions/ingress-host created
# Create a second Ingress with the SAME URL
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/agilebank/dryrun/bad_resource/duplicate_ing.yaml

And bam!

[denied by unique-ingress-host] ingress host conflicts with an existing ingress <example-host.example.org>

Conclusion

As I explained in the introduction, adding compliance policies and constraints to a Kubernetes cluster was not trivial.

Without going as far as saying it’s become easy, OPA and Gatekeeper make things simpler (no need to develop and then host your own microservice for each Admission/Validation Webhook). You still need to learn a new language (rego) to start doing cool things, but even with the default templates, there’s already plenty to work with.

Another really interesting point with OPA is the ability to log and audit all compliance errors using dryrun mode instead of deny, without blocking.

kubectl get K8sUniqueIngressHost.constraints.gatekeeper.sh unique-ingress-host -o yaml
[...]
- enforcementAction: dryrun
kind: Ingress
message: ingress host conflicts with an existing ingress <example-host.example.org>
name: ingress-host2
namespace: default