monitoring on Zwindler's Reflection

Cilium’s new policy log field: our use case

Mon, 03 Nov 2025 12:00:00 +0200

TL;DR

Cilium 1.18 added a log field to CiliumNetworkPolicies to tag flows with custom labels. Great for filtering out expected blocked traffic from your monitoring dashboards!

But there’s a catch, unrelated to this feature, that made this irrelevant in our use case: you can’t use it with egressDeny + toFQDNs.

AND, there is a bug, that makes the “log” only visible on “allowed” traffic.

Here’s why we ran into this wall and what we learned.

The problem: monitoring all the things (but not too much)

Like any good ops team should, we monitor our Kubernetes cluster network flows using Hubble. We (mostly my colleague Nicolas Nativel) push all AUDIT and DROPPED flows to a dashboard so we can quickly spot when something’s blocked and decide:

Is this legitimate? → Open the flow
Is this suspicious? → Sound the alarm 🚨

This works pretty well… until you start explicitly blocking things that you know should be blocked.

In our case, we wanted to prevent a third-party application from phoning home with its “telemetry” (yeah, let’s call it that 😏). We’re talking about calls to external tracking domains.

The issue? If we just block these flows, they’ll show up as DROPPED in Hubble, trigger our monitoring, and we’ll end up with alerts for something we intentionally blocked.

That’s noise we don’t want.

Enter Cilium 1.18’s policy log field

Good news! Cilium 1.18 introduced exactly what we needed: the ability to add custom log fields to your network policies.

Check out the official announcement.

The idea is simple: you add a log field to your CiliumNetworkPolicy:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
 name: my-policy
spec:
 endpointSelector:
 matchLabels:
 app: my-app
 egress:
 - toFQDNs:
 - matchName: "example.com"
 log:
 value: "my-custom-log-tag"

Then, when you observe flows in Hubble, you can filter them out using CEL (Common Expression Language):

hubble observe \
 --verdict AUDIT \
 --not \
 --cel-expression "(_flow.policy_log.endsWith('my-custom-log-tag'))" \
 --print-raw-filters

Output:

allowlist:
- '{"verdict":["AUDIT"]}'
denylist:
- '{"experimental":{"cel_expression":["(_flow.policy_log.endsWith(''my-custom-log-tag''))"]}}'

Perfect! This is exactly what we need. We can now tag our “expected blocks” and exclude them from our monitoring.

The plan: block telemetry elegantly

Armed with this new feature, we crafted our strategy:

Use egressDeny to explicitly block telemetry domains
Add a custom log field: app-explicit-traffic-blocked
Configure Hubble to filter out flows with this tag
Profit! 🎉

Here’s what we tried:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
 name: app-external-block-policy
 namespace: my-namespace
spec:
 endpointSelector:
 matchLabels:
 app.kubernetes.io/name: my-app
 # note: egressDeny takes precedence over egress rules
 # https://docs.cilium.io/en/stable/security/policy/language/#deny-policies
 egressDeny:
 # Block all external traffic and log it with an arbitrary log field
 # This is used to prevent the app from sending telemetry data externally 
 # without triggering an AUDIT/DROPPED alert
 # feature added in cilium 1.18.0 https://github.com/cilium/cilium/pull/39902
 - toFQDNs:
 - matchPattern: "*.telemetry.example.com"
 toPorts:
 - ports:
 - port: "443"
 protocol: TCP
 - port: "80"
 protocol: TCP
 log:
 value: "app-explicit-traffic-blocked"

This should work, right? We’re using egressDeny (which takes precedence over other allow rules, for legitimate calls, which is good!), and we’re tagging it with our custom log.

Reality check: you can’t have nice things

And then… patatra (as we say in French 🇫🇷).

While reading the Cilium documentation on deny policies, we stumbled upon this little gem:

Deny policies do not support:

policy enforcement at L7, i.e., specifically denying an URL

toFQDNs, i.e., specifically denying traffic to a specific domain name.

Wait, what?

You cannot use toFQDNs with egressDeny. Our entire plan just collapsed 😱.

Why this is a problem

The issue is the precedence model in Cilium:

egressDeny rules take precedence over egress rules (by design, and that’s good!)
But if we use egressDeny without toFQDNs, we have to block by IP or CIDR
These telemetry services probably use dynamic IPs for their endpoints (good luck maintaining a list…)
If we block all 80/443 traffic in egressDeny, we can’t make exceptions for legitimate traffic in egress rules because… deny takes precedence to allow!

We’re stuck between a rock and a hard place:

Use egress with toFQDNs → works, but we can’t deny, only allow other traffic
Use egressDeny with IPs → we’ll be playing whack-a-mole with rotating IP ranges
Use egressDeny to block all 80/443 → we block everything, including legitimate traffic

Potential workarounds

While waiting for Cilium to support toFQDNs in egressDeny policies, here are some alternative approaches you might consider:

Find a way to disable telemetry in the app directly

That’s the best option but sadly not always on the table.

DNS-based blocking

Bend the DNS server to return NXDOMAIN for telemetry domains, like a personal pi-hole server would do with ads. The application will fail to resolve the domain and won’t send data.

Use IP-based egressDeny (with maintenance overhead)

Resolve the telemetry FQDNs to their current IP ranges and block them with egressDeny:

egressDeny:
 - toCIDRSet:
 - cidr: 203.0.113.0/24  # Example telemetry IP range
 toPorts:
 - ports:
 - port: "443"

If the list doesn’t evolve too often, this is a good option.

Ok, but let’s assume there is no legitimate traffic. Can we use the feature to add a log on dropped traffic?

Sadly no, not right now.

There is a bug in this new Cilium feature that only logs the policy_log field on “allowed” flows, not on audit/dropped flows.

Policy log does not work for DROPPED/AUDIT flow

When defining a CiliumNetworkPolicy with the spec.log field configured, I expect the relevant hubble flows to have the policy_log field. It works for allowed flow.

But for denied/audited flow resulting from the rule (implicit or explicit), policy_log is never available.

Note: I observe the same issue with --print-policy-names option of hubble, the k8s:io.cilium.k8s.policy.derived-from label is not set for denied flows (but correctly set for allowed flows).

and a related issue [Hubble CLI] –print-policy-names flag does not do anything opened by someone else.

Since 2 tickets are opened and maintainers have started to acknowledge the issue, we can hope this will be fixed, though.

Conclusion

In our use case, we finally didn’t use this new feature from Cilium, but adding details (and allowing filtering on them as well) is always nice.

A shout-out to my colleague Nicolas Nativel, who did most of the work around CiliumNetworkPolicies, including the dashboards, exploratory work on this feature, and took the time to create the issue on the Cilium repository.

References

Should we have containers ?

Thu, 25 Aug 2016 16:00:21 +0000

A more complete answer to « Why do we have containers » by fntlnz

2 days ago, I stumbled apon fntlnz article Why do we have containers. (dead link, using Internet Archive) The article had been reposted by Docker official Twitter account and the first thing that struck me was the displayed image : a schema comparing full virtualization, para virtualization and linux containers.

And of course, I say this because I didn’t entirely agree with it ;-). But then I read the article and I found many good points.

So I decided to write « Should we have containers ? » as an answer to develop my remarks a little.

Full virtualization vs paravirtualization vs OS level virtualization

First, I’ll go with the schema. I know the point was to provide a really simple schema to help understand the differences between virtualization types but I don’t think it should be presented like that.

Hypervisor and host OS are both displayed in the full/para. I find this be misleading. Of course there is a Host OS and a virtualization engine on top of it. But most of the time, either the OS is really thin (ESXi fits in 8GB SD cards and runs nothing but virtualization) or the virt engine is a part of the kernel (KVM, Xen) just like LXC is a part of the Linux kernel.

Also, the bins/libs are only displayed on OS level virt. This is interesting to put in the « host level virtualization schema » but it should be added on full and para as well because this is one of the advantages host level virt. has over full/para.
Even if your VMs are all identicals, bins and libs have to be present (copied) in each VMs. On contrary, in OS Level virt, you CAN share identicals bins/libs between containers, thus reducing the storage footprint of the virtualized environment.

That’s how I would have presented it

You can see that OS level virtualization and Hypervisors are not so different on this simplified view. The differences appear more clearly when you add x86 Security Rings (0->3). The « overhead » of full virtualization comes from the calls that guest OSes do to hypervisor pilots and the translation of CPU instructions.

On a side note, I put Docker on the left to show that it’s basically an administration suite that helps you run LXC containers

The rest of the analysis

Security

When talking about containers, the security point is often raised.

VMs are really hermetics. Taking control of the hypervisor after hacking a VM should be MUCH harder than taking control of the host OS if you are inside a container with whom the kernel is shared.

But I agree with fntlnz: having to patch only one kernel for all your virtual environments instead of having to patch ALL kernels of your virtual machines is actually a good point.

I encourage you to read this (dead link, using Internet Archive) from Jérôme Petazzoni if you’d like a good presentation on the subject.

Performance

A point that’s not really developed in fntlnz article is performance, but I read it/ear it all the time.

General belief is that VMs are slow. That was true in the « early days » of x86 virtualization, because of the limitations of the x86 architectures which didn’t allow efficient OS virtualisation. Overhead for CPU instructions was known to be of about 15% (which is enormous). Thus, running processes directly on the OS are believed to be more efficent.

But nowadays, modern hypervisors on modern hardware nearly generates no overhead at all over CPU and RAM consumption. You could argue that guest OSes consume ressources while containers don’t. But let’s be honest: with the 32 GB RAM tomcat instances I regularly see, don’t say to me that a Linux guest OS CPU&RAM consumption is significative.

The real difference is the on the storage consumption. First, for N VMs, you save yourself N*xGB of guest OS files. If your Linux templates are good, you don’t consume that much (a few GB at most) but this can add up when we are talking about thousands of instances. [Edit jan. 2020]Not so true today if you host your own hardware and have so deduplicated storage[/Edit]

From there, you still have some margin for gains, mostly through Copy on write and union filesystems :

You make a modification to an official tomcat image to add you configuration files ? Only the delta will be stored.
You launch thousands of containers from this image ? You only consume the container image once.
The real data (user data, business data) is of course the same for both architectures

So, should I use containers ?

The problem here is that’s not the real question. Do you need containers ? Here’s fntlnz point of view.**
**

But if at some extent your motivation consist in decreasing costs related to full virtualization overheads while allowing developers to ship, develop and test code faster we already found two.

fntlnz

I won’t tell you not to use containers or Docker. Truth be told, we use it in dev and production. Docker is awesome.

Yes, Docker reduces overhead of full virtualization. Storage mostly, but it highly depends on your workload. For example if you have the need for millions of really tiny instances, then Guest OS overhead is significative in full virtualization.
Yes, Docker allows developers to ship, develop and test code faster. You still have to change your way to develop yours app but Docker and DevOps is the new thing. Better get on with it ;)

But you shouldn’t use Docker just because of that. Docker containers need you to change your philosophy toward applications and administration.

Shared kernel = same kernel

You need to understand that you’ll share a kernel. For a long time that ment a 100% Linux only which could be problematic in some cases. I just realized that Windows containers exists in Windows 2016 tech preview. And Solaris is coming.

But independantly of platforms, you still you share a kernel. If you want Linux containers, you’ll need Linux Docker nodes. If you need Windows containers, you’ll have to have Windows Docker nodes.

Non-persistant/cloud-ready instances

Docker containers are by design non-persistant and by best practice single-process. This isn’t mandatory of course, but Docker works better that way.

The whole idea is to help you make « cloud-ready » & atomic applications that don’t store anything in the container itself (persistant storage is presented to the container but outside it). This way, you can scale out really quickly: you just start up a few more instances if you need more firepower.

This also modifies the way you do systems/applications administration. A container/application is spinning out of control ? Don’t try to repair it.

Kill it
Launch a new one

The fact that persistant storage is handled outside the containers also helps you to deal with lifecycle management.

Update your application inside the docker image
Shutdown the old container
Start the new container with the new image, plugged to the persistant storage

All that requires of course applications that have been designed this way! This is a massive change in philosophy and old monolithic apps on a Docker platform won’t work as well as microservices will.

System administration

I don’t say solutions don’t exist because they do.

But starting up and shutting down containers on the fly will give headaches to your « old school Nagios » monitoring administrator. What could be done easily in the physical/VM world (installing an agent in the template), each of those having an IP adress to query for health checks is a lot easier than checking on a single host a highly volatile number of isolated processes.

You’ll probably have to adapt and use tools that are designed to handle such cases.

Backup administrator may find it confusing at first to make application consistant data backups when applications are running in an variable number of containers.

And so on…

Final thoughts

Yes, you should definitely look into containers. They are mature, stable, a great new way to speed up development and contribute to save money on hardware.

But be aware that VMs and containers don’t answer all usecases; and that you won’t migrate all your applications overnight!

Create a plugin for Thruk using Thruk API, and work around permission issues

Tue, 19 Aug 2014 09:13:19 +0000

A friend of mine came to me with a problem. During one of his Shinken/Thruk deployment, he came with the idea of writing a plugin that could take advantage of Thruk statistics to graph the load of the monitoring engine.

Though you can get this information from Nagios itself through nagiostats, it’s a bit more difficult with the couple Thruk/Shinken. You could emulate Nagios by adding the Shinken old-cgi legacy module, or you could an authenticated wget the web page of Thruk and parse it, but let’s face it, these « solutions » aren’t pretty. I guess that the prettiest one should be to plug into Shinken, and maybe I’ll cover this later, because I think it’s the best solution, but this may need a little « dev effort ».

In order to avoid some useless effort, my friend asked Thruk developers how to do it the way Thruk does it. The main dev guided him to the « thruk » command line, which basically just generates the web page, but in a cleaner way as this is done internaly. You can get more information about Thruk command line with a simple « thruk –help » in your shell.

With this information, my friend developped a perl script (because he is a perl addict ;-) ) to parse this page and get the statistics he needed. But when he deployed it *shock*: he couldn’t see a thing INSIDE the Thruk console.

The main problem here isn’t the fact that the script doesn’t seem to work, but rather the lack of error output.

When first idea was to add the shinken user in apache group. This did help to make it work for the shinken user in shell, but for the web page of shinken/thruk it didn’t help.

To reproduce, I wrote a really simple script

#/bin/perl
#Print your page with thruk CLI, and display error output with 2>&1
print `/usr/bin/thruk -A thrukadmin extinfo.cgi?type=4 2>&1 `;

Though now the output was correct (lots of generated HTML jibber jabber) in the shell, the error is still displayed in Thruk

open file /etc/thruk/thruk_local.conf failed (id: uid=500(shinken) gid=500(shinken)
groups=500(shinken),0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023,
pwd: /var/run/shinken): at /usr/share/thruk/lib/Thruk/Utils/CLI.pm line 272.

AH-AH ! So, basically what I did was to add this at the first line of my script, to help me understand those groups/permissions issues.

#/bin/perl
print `id shinken`;
#Print your page with thruk CLI, and display error output with 2>&1
print `/usr/bin/thruk -A thrukadmin extinfo.cgi?type=4 2>&1 `;

Here is the output

uid=500(shinken) gid=500(shinken) groups=500(shinken),48(apache)
open file /etc/thruk/thruk_local.conf failed (id: uid=500(shinken) gid=500(shinken)
groups=500(shinken),0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023,
pwd: /var/run/shinken): at /usr/share/thruk/lib/Thruk/Utils/CLI.pm line 272.

This is what helped me understand what was really happening here. The groups were shinken and root, but there where no mention of apache, even though I had just added apache to shinken and restarted the whole lot.

In fact, the groups are set at another level : during the deamonization. By reading the files, I managed to find 3 places where shinken can set it’s own group :

/etc/default/shinken, but only used for run and var directories creation (first launch)
/etc/shinken/shinken.cfg, only the arbiter
/etc/daemons/*.ini, for the daemons

You need to have read access for these 3 files (for my setup anyway)

ll /etc/thruk/thruk*
-rw-r--r--. 1 apache apache 25933 17 mai 19:13 /etc/thruk/thruk.conf
-rw-rw----. 1 apache apache 671 18 août 15:36 /etc/thruk/thruk_local.conf
ll /var/lib/thruk/secret.key
-rw-r-----. 1 apache apache 32 13 août 13:13 /var/lib/thruk/secret.key

Opening theses file to « other » is obviously not an option, and I not a fan of changing the group of these conf files, even though this can/will work.

So, as of today, my favorite working solution (I’m still searching for a better one, and maybe I’ll switch to Shinken integration instead of Thruk) is to change shinken poller group in poller.ini to apache and restart the daemon. This way, the poller and the associated plugins can access those 3 files and your Thruk API will work in Thruk console as it does in the shell.