Kubernetes DaemonSet: Examples, Use Cases and Best Practices

“Kubernetes jargon is simple and intuitive” is not something anyone has ever said – and when you encounter terms like DaemonSet, it’s easy to see why. The word implies that it refers to a set of daemons – which it does, sometimes, although that’s not necessarily the case. And to be clear, we’re referring here to daemons in the sense of “processes on Linux-based operating systems,” not spiritual entities.

At any rate, if you’re wondering what DaemonSets actually are, how they work, and how to create and manage them in Kubernetes, you’ve come to the right place. This article explains all of the above and more.

What is a Kubernetes DaemonSet?

In Kubernetes, a DaemonSet is an application deployment method that configures specific pods to run on some or all nodes within the cluster. The main purpose of DaemonSets is to provide control over which nodes host a particular pod. DaemonSets are valuable for use cases like ensuring that a copy of a logging or monitoring agent runs on each node.

This is important because with most other application deployment techniques in Kubernetes, there is no (simple) way of saying “make sure that pod A runs on nodes B, C, and D.” On the contrary, Kubernetes by default will automatically decide where to place pods in a way that balances workloads across available nodes, and that mitigates the risk of downtime due to a node ceasing to operate.

In general, the ability to let Kubernetes figure out on its own where to place pods is good because it eliminates the need for admins to make those decisions manually. But sometimes, you do want to control which node or nodes host a given pod – and DaemonSets offer a way to do that.

DaemonSets are so-called because in Linux, a daemon is a continuously running background process. DaemonSets effectively let you run background processes in Kubernetes by deploying copies of pods on some or all cluster nodes. Technically speaking, a DaemonSet may or may not actually run a set of daemons, because not all applications running inside pods consist solely of daemons – although for common DaemonSet use cases like running monitoring agents, the software inside the pods is usually a daemon.

DaemonSet vs. Deployment

Although a DaemonSet is one way to deploy an application in Kubernetes, it’s distinct from a Deployment. A Deployment is typically used to deploy applications when it doesn’t matter which node or nodes they run on.

You can use pod template labels and node selectors in conjunction with Deployments to assign pods to specific nodes; this can be useful in situations where, for example, you need a pod to run on a specific node because the pod requires specialized hardware resources (like a GPU) that are available only on that node. But usually, admins would choose to use a Deployment when they simply want to run a pod and don’t care exactly which node happens to host it.

DaemonSet vs. StatefulSet

DaemonSets and StatefulSets are both ways of controlling pod deployment and behavior, but they serve distinct purposes. A Kubernetes StatefulSet lets you assign persistent identities to pods and control the order in which they start. This is useful if you need to associate pods with stateful storage resources (hence the term StatefulSet).

In contrast, the main purpose of a DaemonSet, as we said, is to deploy copies of a pod across multiple nodes.

How do Kubernetes DaemonSets work?

DaemonSets work based on the following steps:

1. Define a DaemonSet: Admins write YAML code that describes which pod the DaemonSet should manage (based on pod selectors defined within the DaemonSet) and which nodes the pod should run on. You can use a DaemonSet to run a copy of a pod on every node, or you can use node affinity or selector rules to run it on only certain nodes.
2. Apply the DaemonSet: After configuring a DaemonSet, you apply it using kubectl. At this point, the configuration takes effect.
3. Watch your DaemonSet in action: Once applied, a DaemonSet takes effect automatically. Kubernetes will run your selected pod on all of the requested nodes (assuming they are actually up and able to host workloads, of course).

Note that if new nodes join the cluster, or if existing nodes stop and restart, Kubernetes copies of the pod will run on them, too – so you don’t need to reapply a DaemonSet whenever the state of your cluster changes. Kubernetes tracks your nodes and applies the DaemonSet configuration to them automatically.

Key Kubernetes DaemonSet operations: Examples

To illustrate how DaemonSets work in practice, let’s take a look at some examples.

How to create a DaemonSet

Here’s an example of a simple DaemonSet with a pod selector that matches pods labeled “example-daemonset”:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: example-daemonset
  labels:
	app: example-daemonset
spec:
  selector:
	matchLabels:
  	name: example-daemonset
  template:
	metadata:
  	  labels:
    	    name: example-daemonset
	spec:
  	  containers:
  	     - name: example-container
    	     image: monitoring-agent

In this example, the pod would run on every node in the cluster.

To apply this DaemonSet, you’d save the code in a file (such as my-daemonset.yaml) and apply it with:

kubectl apply -f my-daemonset.yaml

How to create a DaemonSet restricted to specific nodes

Imagine that instead of running pods on all cluster nodes, you wanted to run them only on specific nodes. You could do that by including a nodeSelector value within your DaemonSet configuration. Here’s an example:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: example-daemonset
  labels:
	app: example-daemonset
spec:
  selector:
	matchLabels:
  	name: example-daemonset
  template:
	metadata:
  	  labels:
    	    name: example-daemonset
	spec:
  	nodeSelector:
    	   example: daemonset
  	  containers:
  	     - name: example-container
    	     image: monitoring-agent

This DaemonSet would run pods labeled “example-daemonset” only on nodes labeled “example=daemonset.”

By the way, you can apply labels to nodes using a command like:

kubectl label nodes node-name label=value

How to update a DaemonSet

The easiest way to update a DaemonSet is to modify its YAML configuration code, and then use kubectl to apply the new version of the DaemonSet configuration.

For instance, if you had initially deployed the preceding example DaemonSet without a nodeSelector value, but you then decided to add a nodeSelector, you’d simply update your code, then run the kubectl apply command again:

kubectl apply -f my-daemonset.yaml

Note that Kubernetes can handle DaemonSet updates in two different ways:

• OnDelete: The OnDelete update strategy means that if you have existing pods that are already running on nodes as part of a DaemonSet, Kubernetes won’t modify them to match a newer, updated version of the DaemonSet on those nodes unless you manually remove the pod from the nodes.
• RollingUpdate: The RollingUpdate strategy automatically updates nodes to match the most recent version of a DaemonSet.

RollingUpdate is the default behavior in Kubernetes versions 1.6 and later. If you want to use OnDelete instead, add a configuration like the following to your DaemonSet spec:

updateStrategy:
	type: OnDelete

How to use a DaemonSet for log collection

As we mentioned, one of the most common use cases for a DaemonSet is to support log collection. You can use a DaemonSet for log collection by following these steps:

1. Configure your logging agent to run as a pod.
2. Label the logging agent pod with a unique identifier, such as “logging-agent”
3. Create a DaemonSet with a pod selector that is the same as the pod label.

With this approach, every node in your cluster will automatically run a copy of the logging-agent (or, if you add a node selector, the agent would run on whichever nodes match the selected node label).

Kubernetes DaemonSet use cases

Log collection is just one of several popular use cases for DaemonSets in Kubernetes. Here’s a more complete list:

• Log collection and aggregation: DaemonSets are a simple way to run software agents (like those supported by OpenTelemetry) that collect and aggregate logs from each node.
• Kubernetes observability and metrics collection: DaemonSets can run observability and monitoring agents that collect performance metrics from each node.
• Security and intrusion detection: Security monitoring tools can be deployed using DaemonSets as a way of monitoring intrusions and vulnerabilities at the node level.
• Networking and traffic analysis: You can use DaemonSets to deploy agents that monitor and analyze network traffic as it flows to and from nodes.

How are DaemonSet pods scheduled?

The task of assigning pods to nodes using a DaemonSet falls to the DaemonSet controller. The DaemonSet controller works in the background, checking for DaemonSet configurations and updating the state of application deployments to match them.

As we’ve noted, the DaemonSet controller will schedule DaemonSet pods on every node in the cluster by default. If you want to schedule DaemonSet pods only on certain nodes, you have to add a node selector (or node affinity, a more flexible way of selecting nodes) to your DaemonSet configuration.

Note as well that the DaemonSet controller isn’t magic and can’t assign pods to nodes that don’t have enough resources to host the DaemonSet pods, that have crashed or that are not reachable via the network. So, don’t try to use DaemonSets as a workaround for node performance problems; you need your nodes to be healthy in order for DaemonSet pods to be scheduled on them.

Kubernetes DaemonSet best practices

To get the most value out of DaemonSets, consider the following best practices.

Use DaemonSets only when appropriate

In general, you should only use a DaemonSet when you need to run copies of a pod across all or several nodes within your cluster.

If your goal is instead to run a specific pod on a specific node or two, a better approach is to use a Deployment with a node selector. Although you could also support this use case using a DaemonSet, it wouldn’t be the simplest way to achieve your goal.

Likewise, if you want to run multiple copies of a pod but it doesn’t matter which specific nodes host them, use a ReplicaSet. ReplicaSets are similar to DaemonSets in that they can run multiple pod instances, but without scheduling them on particular nodes.

Avoid manual management of DaemonSet pods

As mentioned, you can configure an OnDelete update strategy within a DaemonSet as a way of requiring manual deletion of Daemon pods before updates take place. In general, this isn’t desirable. Not only is it more work for admins, but it also increases the risk that you’ll forget to update a pod instance, leaving an older version of your pod in place.

The exception to this rule is use cases where it’s important not to disrupt running Daemon pods by updating them. For instance, if you’re using a DaemonSet to stream real-time data from every node and can’t tolerate a disruption to the data stream, you may want to manage DaemonSet pods manually. This would allow you to take each node offline before updating its pod, ensuring that you don’t lose any data.

Track DaemonSet resource utilization

Because DaemonSets run copies of a pod across multiple nodes, and because DaemonSet pods (like all pods) consume resources, it can happen that the DaemonSet leads to an undesired spike in resource consumption. For this reason, it’s important to monitor pods and nodes following the creation of a DaemonSet and ensure DaemonSet pods are not starving your cluster of resources.

Track changes to DaemonSets over time

By automatically tracking DaemonSet code using a version control tool like Git, you can monitor how your DaemonSet configurations change over time. You can also easily revert to an earlier DaemonSet if desired.

Common DaemonSets challenges

DaemonSets are a powerful feature in Kubernetes, but they can pose some challenges:

• Resource overhead and performance impact: The more pods you run on each node, the more node resources your pods will consume. For this reason, DaemonSets can eat into total cluster resource availability and deprive other workloads of enough memory and CPU.
• Node affinity issues: Sometimes, accidental mislabeling of nodes within the pod template, or technical failures like intermittent lack of network connectivity, can cause the DaemonSet controller not to schedule a pod on all selected nodes.
• Node variations: A DaemonSet assumes that a copy of a given pod can run across all selected nodes. This is true in many clusters, but it can create problems if, for example, some nodes don’t have enough memory to support a pod, or if a pod requires specialized hardware that is not available on all nodes.

Update complexity: As explained above, updates to DaemonSets may or may not result in immediate changes to the pods running on each selected node, due to the differences between the OnDelete and RollingUpdate update strategies. This may lead to unexpected behavior when admins don’t pay close attention to the type of updates they’ve configured.

How groundcover optimizes Kubernetes observability with DaemonSets

Historically, DaemonSets were a cornerstone of Kubernetes observability and security. In most cases, admins would rely on traditional monitoring agents to collect performance and security data, and they’d deploy those agents using DaemonSets.

This worked, but it came with a steep drawback – the performance overhead posed by DaemonSets.

That’s why we built groundcover to work differently. We use eBPF to collect observability data from each node. Because eBPF code runs in kernel space, it’s much, much more efficient than traditional monitoring agents, which run in user space. It can also access data that’s not always available from user space, leading to deeper observability insights.

Now, we’re not anti-DaemonSet. On the contrary, DaemonSets are one option for deploying eBPF programs. But when you run eBPF via DaemonSets, you get all of the visibility (and more!) that you expect from traditional monitoring agents, without the performance drawbacks.

Plus, groundcover helps you keep track of what your various DaemonSets are doing – whether you’re using them to deploy eBPF code or for any other purpose. By continuously observing each pod and each node, groundcover clues you into situations like failure of the DaemonSet controller to schedule a pod on a given node, or excess CPU or memory consumption.

All set with DaemonSets

DaemonSets may not have the most intuitive-sounding name (although admittedly, a term like DeploymentThatRunsPodsOnSomeOrAllNodes has its own drawbacks). But they are a handy feature, especially for supporting use cases like observability and performance management – and especially when you combine DaemonSets with eBPF to unlock maximum efficiency and visibility.