Exploring the Power of eBPF in Kubernetes: A Technical Perspective

Let's be blunt: Managing Kubernetes can be a headache. A large part of the reason why, is that Kubernetes is a complex system that includes many different components, each of which exposes logs and metrics in a different way. Consequently, there's no easy means of pulling out and analyzing all of the data you need to figure out what’s happening inside a Kubernetes cluster.

Or at least, there was traditionally no easy way to do that. But thanks to eBPF, a new approach to Kubernetes management has become possible. An eBPF Kubernetes strategy unlocks huge efficiencies that allow admins to tame Kubernetes observability, security and other challenges in ways that don't send them rushing for a bottle of aspirin. In fact, with eBPF, Kubernetes management changes from a headache-inducing endeavor to one that can be quite pleasant – provided you have the right tools and processes in place for taking full advantage of eBPF in Kubernetes.

That's what this article explains how to do. Read on for a deep dive into what eBPF brings to Kubernetes and how to leverage eBPF fully in your Kubernetes cluster.

What is eBPF, and Why Is It Gaining Traction in Kubernetes?

The answer to what is eBPF is this: eBPF is a framework built into modern versions of the Linux kernel (specifically, versions 4.16 and later) that makes it possible to run custom code in kernel space. You can do this without having to modify the kernel source code or resort to the messy business of loading kernel modules (which carry some security and performance risks) into the kernel.

To put this in somewhat less technical terms, eBPF lets you run programs of your choosing deep inside the Linux operating system in a simple way. Using those programs, you can collect information directly from the operating system. That information can unlock critical insights into workload performance, security risks, networking issues and anything else that is visible to the operating system.

Understanding What eBPF Monitoring is and Its Capabilities

Now, why should Kubernetes admins care? The reason is simple enough: Since the nodes that comprise Kubernetes typically run a Linux-based operating system, eBPF makes it possible to use the kernels of those nodes as the vector for collecting the data that admins need to make informed decisions.

In this way, eBPF brings consistency to what is otherwise a fraught process: Getting the data you need to figure out what's happening inside Kubernetes. Instead of using strategies like the sidecar pattern to monitor applications, while also pulling various log files that are scattered across different servers and directories on control plane nodes to track cluster status, you can turn to eBPF to collect the data you need in a centralized, consistent manner.

But wait, there's more! eBPF programs are also hyper-efficient because they run directly in the operating system – or, to use the more technical term, they run in kernel space, not user space. That means they consume very few resources, which in turn means that there is more CPU and memory available to your actual workloads.

The fact that eBPF programs must undergo a validation process before Linux allows them to execute is an advantage as well. It helps prevent risks that buggy eBPF code could crash a Kubernetes node and prevents the risks you'd face if you used unstable kernel modules. And because programs run in sandboxed environments, eBPF mitigates security concerns, too.

How eBPF Works in Kubernetes

The fact that eBPF allows you to run custom code inside the Linux kernel space of each Kubernetes node means that using eBPF in Kubernetes is fairly straightforward. It boils down to this:

• You determine which information you want to collect. As we'll discuss in more depth below, eBPF allows you to collect virtually any kind of data that is available to the kernel, so it can provide visibility into networking, CPU and memory usage, storage operations and more.
• You write an eBPF program that tells the operating system kernel to expose whichever type of data you want to view.
• You deploy the eBPF program on a node or nodes in your cluster that have access to the resources you want to manage. We discuss how to implement eBPF in Kubernetes in detail below. But suffice it to say for now that you can deploy the programs by SSHing into the nodes directly, or you can use a method like DaemonSets to run them on specific nodes.
• You collect the data that eBPF generates and feed it into your favorite analytics and/or visualization tools to gain a deeper understanding of what's happening in your clusters.

The ability to collect almost any type of information by deploying code directly on nodes is part of what makes eBPF so awesome in Kubernetes. As we mentioned above, eBPF saves you from having to deploy a bunch of Kubernetes monitoring agents and collect scattered log files across various nodes.

eBPF Applications for Kubernetes and Their Benefits

If you're thinking, "Wow, it sounds like there's almost no limit to the use cases that eBPF can enhance for Kubernetes!", you're right. The ability to collect data directly through the Linux kernel allows admins to apply eBPF to Kubernetes for a variety of different purposes. Here's a look at the most common.

‍

eBPF Performance Monitoring

By tracking metrics about workload performance on a process-by-process level, eBPF excels at helping Kubernetes admins monitor the performance not just of applications as a whole, but of individual containers and microservices. If you want to know how your app is performing or pinpoint performance bottlenecks, eBPF is your friend.

eBPF Cluster Observability

eBPF is a great way to observe your Kubernetes cluster as a whole. eBPF programs running on control plane nodes can observe control plane components and processes, like Etcd and the API server. You can also use eBPF to monitor worker node resources, like Kubelet, and correlate that data with control plane observability insights to gain granular visibility into your cluster.

Security Monitoring

The ability to monitor both processes running on nodes and network traffic flowing to and from them makes eBPF a great solution for security monitoring. Unusual process behavior and suspicious network activity – whether it originates from endpoints external to your cluster or involves packets flowing within the cluster – become easy to detect when you have eBPF on your side. 

Filtering Network Traffic

For generic traffic filtering and performance management, eBPF is a powerful tool. By tracking TCP connections and associating them with processes accessing the network, eBPF gives you granular visibility into which network resources your containers, Pods and so on are using.

eBPF Profiling and Tracing

If you want to trace how a request to a Kubernetes application flows through your stack – as you might if, for instance, you need to determine the source of a performance slowdown – eBPF tracing makes the process simple.

Administrative Convenience

Making Kubernetes administration more convenient may or may not count as a "use case" (we'll let you debate that one). But whatever you choose to call it, the fact is that eBPF makes it considerably simpler to administer Kubernetes because, once again, it eliminates the need to deploy a bunch of user space agents and collect logs and metrics from disparate sources. With eBPF, you get one framework and one set of tools to do it all.

Implementing eBPF In Your Kubernetes Cluster Structure

Now that you know what you can do with eBPF on a Kubernetes cluster, let's talk about how to implement eBPF in such an environment.

Approaches to Implementing eBPF in Kubernetes

‍

First, a little background about eBPF tooling. The eBPF framework is built into the Linux kernel, so as long as your nodes are running a supported kernel (and again, all kernel versions 4.16 and later support eBPF), they are already capable of running eBPF code.

However, Linux doesn't provide built-in tools for actually uploading and managing eBPF programs. To do that, you need to install a user space tool, like bpftrace, that allows you to interact with the operating system kernel's eBPF framework.

There are a few different ways to go about setting up the necessary tooling and deploying eBPF programs on Kubernetes.

Manual eBPF Implementation on Each Node

The simplest – but most time-consuming – approach is to log into each node, install a user space tool and then use it to run eBPF code.

You'll also need to make sure you move the data that eBPF produces to a location where you can process it, since you probably are not going to run an analytics program directly on each node.

Deploying eBPF with DaemonSets

To streamline the deployment of eBPF programs across clusters, you can create a DaemonSet and use it to run your eBPF code on whichever nodes need to host it. Those nodes could include every node in your cluster, or they could be a subset of nodes – such as only control plane nodes, or only nodes running a certain workload that you’re monitoring using eBPF.

Deploy an Tool with eBPF Built In

You can streamline eBPF deployment in Kubernetes even more by leveraging a tool, like groundcover, that runs eBPF under the hood to collect data. This way, you don't have to write and deploy eBPF programs yourself; you just deploy a user-friendly tool and let it use eBPF to help manage your Kubernetes cluster. We cover the process for monitoring Kubernetes using groundcover and eBPF below.

Challenges of Using eBPF in Kubernetes

While eBPF is a great tool, there are some potential pitfalls to be aware of if you choose to use it in Kubernetes.

Scaling the Clusters

Since eBPF runs on individual nodes, you'll need to ensure that you deploy it whenever you scale your cluster up by adding nodes. This isn't a big deal if you use an approach like DaemonSets or groundcover to run eBPF, because new nodes will be monitored with eBPF automatically in that case. But if you're deploying eBPF on each node manually, cluster scaling can become burdensome.

Orchestration

Likewise, orchestrating your eBPF programs by ensuring that the right ones are running on the right nodes can be challenging, especially if you deploy the programs manually. But here again, an automated approach to eBPF deployment will simplify things.

Centralized Monitoring

As we mentioned above, eBPF produces valuable data, but it's up to you to move it to a centralized monitoring location so you can put the data to use. Tools like groundcover help automate this process, but moving data can become a headache if you run eBPF on each node manually.

Best Practices for eBPF in Kubernetes

You can get more from eBPF in Kubernetes by adopting practices such as the following:

• Automate deployment: We said it above and we'll say it again: Deploying eBPF on each node manually is a ton of work, and it doesn't scale. Wherever possible, leverage tools that automate eBPF setup.
• Correlate data: eBPF can do many things and produce different types of data. To use the tool to maximum effect, correlate your data by, for example, comparing network performance data to application performance data so that you have the greatest possible context on workload performance.
• Require privileged mode: Be sure to configure each node to require privileged mode, which helps prevent abuse of eBPF by attackers who want to access the operating system kernel.

How to Use eBPF in Kubernetes to Enhance Monitoring

If you've read this far, you know that the best way to use eBPF in Kubernetes is to deploy a tool like groundcover, which combines the convenience of a production-ready, developer-friendly observability platform with the power of eBPF. In other words, groundcover lets you take advantage of eBPF without the headache of deploying eBPF code manually on each node, building pipelines to move the data or worrying about breaking observability processes when you scale your cluster.

The setup process is easy. First, install the groundcover CLI in your environment, making sure your cluster context is properly configured:

Then use the CLI to complete the setup process:

For a complete guide to setting up groundcover and alternative deployment methods, check out our installation documentation.

Once groundcover is up and running in your cluster, eBPF is, too – because again, groundcover uses eBPF under the hood to enable Kubernetes monitoring. eBPF is the killer feature that sets groundcover apart from most other Kubernetes monitoring and observability tools, which rely primarily on user space applications to track cluster health and performance.