eBPF Verifier: Why It Matters for Reliable Observability

They say that with great power comes great responsibility. Unfortunately, when it came to running custom code in the Linux kernel, there wasn’t traditionally a good way to be powerful and responsible at the same time.

In the past, engineers who wanted to deploy custom kernel code had to use solutions like Linux kernel modules – which offer the power of letting you run custom programs in kernel space, but which are not very responsible because buggy modules could cause a system to crash. Directly modifying kernel source code presented the same risk, with the added drawback of requiring a tremendous amount of work.

Thanks to the eBPF verifier, however, you can enjoy the power of kernel-space code while also being responsible. That’s because the verifier checks programs written for the extended Berkeley Packet Filter (eBPF) framework before they execute, making it possible to catch buggy or insecure code before it can cause problems.

Unfortunately, although the eBPF verifier is a powerful and important tool, it’s also one that is tricky to work with sometimes. It may reject programs for reasons that aren’t always immediately clear. It may also disallow code from running in cases where the code isn’t actually buggy, but due to code design and implementation practices, the verifier can’t confirm that the program is safe.

Thus, if you want to write and run eBPF programs while still being responsible, you need to know how to leverage the eBPF verifier effectively. Keep reading for guidance as we walk through what the eBPF verifier is, how it works, and how to get the most value out of it.

What is the eBPF verifier?

The eBPF verifier is a part of the eBPF framework that checks eBPF bytecode for safety risks before they run in the Linux kernel. Programs must pass the verifier’s tests before the kernel can execute them.

The types of safety checks that the verifier performs fall into two categories:

1. Checking whether eBPF code may introduce stability issues that could cause problems like an operating system kernel panic, leading to a server crash.
2. Checking for security issues, such as a situation where an eBPF program run by unprivileged users can access memory space owned by other users, creating a potential for the leakage of sensitive data through eBPF maps.

These are the types of validations eBPF developers have in mind when they talk about “safety” checks by the verifier.

Why the eBPF verifier matters for observability

Although the eBPF verifier is not itself a tool for collecting telemetry data or otherwise assisting in the observability process, it is a key component for executing eBPF programs, which are themselves a way of collecting observability data. That’s because the verifier is not an optional component of eBPF. eBPF code must pass the verifier because the kernel will execute it. There’s no getting around it.

Thus, if you want to leverage eBPF as an observability solution – which we strongly think you should, given that eBPF tracing offers a hyper-efficient way of monitoring and observing any process, application, or network traffic flow running on a Linux kernel – you need to use the verifier as part of the process. You can’t bypass it (unless you went to the trouble of modifying the kernel source code, which would be a lot of work and not a safe practice).

It’s worth noting, too, that the verifier is a part of what makes eBPF so great. Thanks to the verifier, admins can run observability code directly in kernel space without worrying about breaking their system. Without this component of eBPF, you’d have to run eBPF programs inside some kind of protected environment, like a virtual machine, which would be much less efficient. Or, you’d have to undertake the risk that bugs inside your eBPF code could lead to catastrophic problems, like a server failure or data breach.

The verifier also makes eBPF programs preferable to loading kernel modules, which are another way of inserting custom code into the kernel without modifying the Linux source code. Kernel modules don’t undergo a verification process, so they pose a much higher risk of crashing your system.

How does the eBPF verifier work?

The eBPF verifier works via a process known as static code analysis. This means it analyzes eBPF bytecode while the code is not running, looking for issues that could cause problems when the code executes.

To detect potential issues, the verifier runs through all possible permutations of a program’s behavior. Based on the results, it can discover whether there is a risk of the program accessing out-of-bounds memory or ending up in an infinite loop state. If the verifier detects a problem with the code, it registers an error message with more information. We’ll discuss common eBPF verifier errors and what they mean later in this article. If the verifier doesn’t detect any problems, it passes the program into the execution phase.

The verifier prints messages to the log buffer. If you want to be able to view the messages, your eBPF loader (the userspace tool that requests execution of an eBPF program) must provide a buffer for the verifier to write to, then collect and print its output in a place visible to the user.

Limitations of eBPF verifier

Importantly, the scope of the verifier’s functionality is limited to checking for safety issues that could destabilize a system or violate kernel-level access control restrictions. It’s not a general-purpose static code analyzer. It doesn’t perform other types of checks, such as making sure that your eBPF code actually collects the type of data you want it to collect. It’s on you to make sure your program works as you intend.

So, don’t think of the verifier as a general-purpose static code analyzer or debugger. Think of it as a specialized tool designed only to protect against issues that could make the Linux kernel behave in unexpected ways.

Benefits of the eBPF verifier for observability

Although the verifier supports a limited range of functionality, it offers a wide range of benefits for enhancing the observability process.

Ensuring safe and reliable eBPF execution

As noted above, the verifier protects against unsafe code that could cause stability or security problems. In this way, it ensures that the programs teams use to help optimize performance don’t paradoxically introduce performance issues.

Reducing performance overhead in monitoring

The design of the verifier keeps monitoring overhead low. This is because, by using the verifier to check code prior to execution, eBPF avoids having to perform checks or impose restrictions at runtime, which would be more costly in terms of CPU and memory usage.

The verifier model also eliminates the need to run code in an isolated environment, which would also add overhead, as a way of mitigating safety risks.

Enhancing security by detecting malicious code

The verifier plays an important role in protecting against situations where code inside an eBPF program could access data that shouldn’t be available to it. This primarily happens when a program attempts to read from out-of-bounds memory.

Facilitating faster debugging and troubleshooting

Because the verifier provides error messages indicating why a program failed to pass verification checks, it helps with the debugging process. The errors can be frustrating in some cases because it’s not always clear exactly which code triggered an error. Still, getting feedback via verifier error messages is better than having eBPF simply refuse to execute a program at all without providing clues about why.

Challenges in passing the eBPF verifier

eBPF developers should design their programs to pass the verifier. Unfortunately, this can prove tough for several reasons:

• Complexity of verification rules: The verifier checks programs from many angles, including register states, function calls, memory safety, and control flow. This means there is no single set of rules you can follow to ensure valid code; you need to understand multiple types of constraints.
• Recursion and loop constraints: Certain recursion and loop techniques can trigger automatic failures. For example, eBPF doesn’t support unbounded loops or recursion to ensure that programs always terminate within a limited number of instructions and recursive functions are rejected outright.
• Memory access and execution paths: In some cases, the verifier may reject programs based on the way they interact with memory, even if their behavior would not actually trigger safety problems. For instance, indirect or dynamic memory accesses are difficult to verify and may cause rejection unless the verifier can determine memory bounds definitively.
• Changes to kernel versions: Since eBPF is part of Linux, different Linux kernel versions can contain different versions of the verifier. A program that passes checks under one set of kernel versions may fail under other kernel versions.

Issues such as these may lead to false positives, meaning the verifier rejects code even though it will not pose safety risks. In that case, developers must find a way to rewrite their code in a way that will pass the checks; it’s not possible to tell the verifier to suppress or ignore errors as a workaround.

Common eBPF verifier error messages

The verifier can produce a wide range of error messages. The more common ones include the following.

unreachable insn

Error messages that mention unreachable insn indicate that a program contains unreachable instructions, meaning code that can’t be executed because no control flow path leads to it. Unreachable instructions are not allowed in eBPF under any circumstances. To fix the issue, you need to remove the unreachable code from your program.

invalid stack

An invalid stack error means that a program attempts to access a memory stack that is out of bounds. This is usually because the program is trying to read memory that has been allocated to a different user than the one attempting to execute the program.

!read_ok

Error messages that contain the string !read_ok are usually an indication that a value returned by a function has not been initialized. The solution is to initialize the value.

invalid indirect read from stack

The invalid indirect read from stack error means a program fails to initialize a stack before passing its address into a function. To fix it, initialize the stack.

Best practices for writing verifier-friendly eBPF programs

While you can modify an eBPF program to fix the issues that cause a verification failure, it’s preferable to avoid having failures in the first place. The following best practices can help in this regard by minimizing the chances that the verifier will reject your program:

• Strive for simple code: The simpler your eBPF code, the less likely you are to run into unexpected issues that trigger a verification failure. In general, it’s best to design each eBPF program to do one specific task, rather than trying to write a complex program that performs multiple tasks.
• Initialize stacks and registers: Many common eBPF verification errors occur because of uninitialized stacks and registers. Developers should pay careful attention to ensuring that they initialize these components properly.
• Minimize recursion: As noted above, the verifier will reject programs that use certain types of recursion techniques, like unbounded loops. As a best practice, try to avoid recursion where possible, and, when that’s not possible, keep recursion logic simple and well-defined.
• Avoid indirect and dynamic memory access: Attempting to access memory in this way may also trigger an error, as we mentioned above.
• Check your kernel version: As noted, different kernel versions can have different versions of the verifier. It’s important to know which version your code will run against and ensure that you conform to the requirements of that version.

Tools and techniques for debugging eBPF verification issues

If it’s not readily obvious based on the verifier error message what caused the rejection of a program, there are some additional tools and methods you can use to examine your program’s data structures and get to the root of the issue.

Debugging with bpftool

Bpftool is a command-line program for analyzing eBPF programs. In addition to allowing you to view verifier logs, it provides features like the ability to disassemble eBPF instructions and attach and trace maps. These capabilities may be useful if you need to figure out exactly which code caused a verification error.

Leveraging veristat for verification insights

Veristat is a tool for analyzing eBPF verifier log files and helping engineers understand messages inside them. It can do things like count the total number of verification passes; if this number is very high (such as more than 1000), it could be a sign of issues like an unbounded loop.

Debugging with bpf_printk()

bpf_printk() is a debugging function in eBPF that makes it possible to print messages to the Linux kernel trace buffer. It can’t directly help with verifier failures because you can only use bpf_printk() once a program executes, and it needs to pass verifications in order to execute. Still, the tool can help with debugging verifier-related issues and examining data structures before and after you load eBPF programs.

For example, if you want to confirm whether one of your function calls is causing a verification failure, you can replace the function with a call like bpf_printk("Function X goes here"); and then load your code again. If the code runs successfully, you know that the function you disabled was the cause of the verification issue.

How groundcover helps with eBPF observability

At groundcover, we can’t tell you how to debug your eBPF programs. But we can save you from the hassle of having to do so in the first place. That’s because groundcover is a user-friendly, end-to-end observability platform that uses eBPF “under the hood” to collect insights.

This means that, when you use groundcover, you benefit from the speed and safety of eBPF observability without having to implement it yourself. You can spend your days collecting and reacting to performance insights – which we think is a lot more fun than fighting with the eBPF verifier to get your code to run.