Kubernetes Statefulset: What it is, Components & Best Practices

In general, Kubernetes treats Pods the same way you might treat someone forgettable whom you meet at a party: Once the party's over, you forget the person's name forever, and you move on with your life. In the same way, most Pods in Kubernetes – specifically, Pods that are launched as part of a Deployment – are gone forever from Kubernetes's mind once they shut down because Kubernetes does not identify them in a unique, persistent way.

But sometimes, you do need Kubernetes to treat each Pod in a unique way, and you want it to remember a Pod's identity even if the Pod shuts down and restarts. This is necessary if, for instance, you want to assign persistent volumes to Pods and need to ensure that those resources can persist even if Pods shut down and restart. Or, you may want to set up a specific networking configuration for one of your applications.

This is where StatefulSets come in. By running Pods using StatefulSets instead of a traditional Deployment, you tell Kubernetes that it needs to give each Pod a unique and persistent identity, which makes it easier to support certain use cases.

Keep reading for a breakdown of how StatefulSets work, when you should use them, and how to make the most of StatefulSets within your Kubernetes clusters.

What is a Kubernetes StatefulSet?

In Kubernetes, a StatefulSet is a set of Pods that receive unique, persistent identities. (The Kubernetes developers like to say that the Pods' identities are "sticky.") This means that if Kubernetes stops and restarts the Pods, their identities remain the same. In addition, StatefulSets make it possible to control the order in which Pod replicas start and stop.

Configuring a persistent identity using StatefulSets is beneficial in situations where Pods need to access stateful resources, such as persistent volumes – hence the term StatefulSet.

Having persistent identities is the key characteristic that distinguishes StatefulSets from other workload models in Kubernetes, including DaemonSets and Deployments.

ReplicaSet

StatefulSet vs. Deployment

A Kubernetes Deployment is a way of telling Kubernetes how to run and manage a set of Pods that don't require persistent identities or any other type of special configuration. (Notice here that we're referring to Deployments with a capital D; this is a specific type of object in Kubernetes, whereas "deploying" resources in general is a more generic term that folks don't always use when referring to Deployments specifically.) If you just want to run an application, and you have no need to associate it persistently with stateful storage, networking or other resources, a Deployment lets you do that.

Deployments are the simplest way to deploy an application in Kubernetes because they have the fewest configuration requirements. In addition, a Deployment leaves it up to Kubernetes to determine which nodes should host which Pods, and to reschedule Pods (which means stopping them and restarting them on a different node) whenever Kubernetes deems it necessary in order to balance performance or achieve stable resource availability. Since running applications in an automated way across a distributed cluster of servers is the main purpose of Kubernetes, Deployments are important for admins who want to say, "Hey K8s, here's my app. Please go and run it."

For these reasons, Deployments are the most common means of running applications in Kubernetes. If you're new to Kubernetes, you'll probably want to master how Deployments work before learning about StatefulSets, which are more complicated.

But when it comes time to deploy resources that require a persistent state, you'll need to learn about StatefulSets.

StatefulSet vs. DaemonSet

A DaemonSet tells Kubernetes that certain nodes in a cluster should always host a given Pod or Pods. DaemonSets are commonly used to address requirements like the need to run a Kubernetes monitoring agent on each node (which isn't necessary if you leverage a framework like eBPF, which is a much more efficient and effective way to achieve Kubernetes observability than conventional agent-based monitoring, but we digress).

In contrast, a StatefulSet's purpose is to ensure that Pods have persistent identities so that they can be reliably mapped to the same resources even if the Pods stop and restart.

When to use StatefulSets

Here's a look at the main reasons why you'd want to use StatefulSets, and which Kubernetes benefits they offer.

Configuring stable storage

The most common use case for StatefulSets is deploying applications that require access to persistent or stable storage.

StatefulSets are important for accessing persistent storage because having a unique, persistent identifier for Pods makes it possible to associate Pods with storage volumes. If the identities of the Pods that need to access storage changed whenever the Pods restarted, you'd have to update your storage volumes' configurations as well with the new Pods' identities. That would be a huge pain – not to mention less secure, because the more often you're changing storage resource configurations, the more likely it is that you'll make a mistake that allows access to a workload that shouldn't be able to view your data.

Assigning unique network identifiers

Similarly, StatefulSets are useful in situations where you want Pods to have unique, persistent network identities. For example, if you want to assign a certain hostname to a Pod and ensure that the hostname remains the same if the Pod restarts, you can do so using a StatefulSet.

Here again, the alternative solution would be to try to configure identities manually and update them whenever Pods restart, which would be a ton of work that no sane K8s admin wants to do.

Starting and stopping Pods in a certain order

StatefulSets allow you to control the order in which replicas of a Pod start. You can also require a set of Pod replicas to terminate in a certain order. This control is useful for workloads where you need to ensure that one Pod is running before another one starts.

For example, if you’re applying rolling updates to an application – meaning you want to update an application without any downtime – you can configure a StatefulSet such that there are multiple replicas for the Pods you want to update. Then, when you update the Pods, the StatefulSet controller will automatically ensure that the update to one Pod has successfully completed before it begins updating the next one.

StatefulSets Limitations

While StatefulSets are a powerful tool, they are subject to some limitations and drawbacks.

The biggest limitation of StatefulSets is that they make certain assumptions about how stateful resources are managed and configured. For example, they're designed only to work with stateful storage that is provisioned manually or via a PersistentVolume provisioner. If you operate persistent storage in Kubernetes using a different approach, you're likely to find it difficult to connect that storage to Pods using StatefulSets.

Similarly, you have to set up a headless Service if you want to manage persistent network identities for Pods using StatefulSets. If you don't have a Service configured for this purpose, it will be challenging to work with network identities based on StatefulSets.

Another limitation is that although StatefulSets allow you to control the order in which Pods are stopped in the event that they are rescheduled, this is not the case if you delete a StatefulSet entirely.

In short, while StatefulSets give you a level of control over Pods that you would lack when using a standard Deployment, their flexibility is limited. In particular, they can be frustrating to work with if you have storage or network resources that are configured in an unusual way.

Components of a StatefulSet

The components of StatefulSets vary depending on what a given StatefulSet is intended to do. But in general, they include:

• A spec that defines basic characteristics of the Pods that you want to run, such as which container images to use and how many replicas you want of each Pod.
• Configuration data that defines resources that the StatefulSet controller should associate with the Pod. For example, if you want to associate a persistent volume with a Pod, you'd define the configuration using a volumeClaimTemplate.

In the following section, we'll explore what these components look like in an actual StatefulSet example.

How to create a StatefulSet in Kubernetes

Now that you know all about how StatefulSets work, let's talk about how to create one.

As we mentioned in the preceding section, you basically need to define two things: How to run your Pods, and how to associate them with any stateful resources they should access.

To do this, you write YAML code, just as you would for a standard Kubernetes Deployment or Pod template. For example, here's code that creates a sample StatefulSet named example-statefulset and 

sample StatefulSet (borrowed from the Kubernetes documentation) that assigns a persistent volume to an NGINX instance:

To use this StatefulSet, you'd save the code to a file (like example-statefulset.yaml), then apply it using a command like:

How to Debug a StatefulSet

If a StatefulSet behaves in an unexpected way – if some or all of the Pods fail to start, for example – you can debug it using the following steps.

List StatefulSet Pods

First, you'll want to know which Pods are in the StatefulSet. If you labeled them when creating the StatefulSet, you can list the Pods using a command like the following:

If you didn't label the Pods, you'll have to look at your YAML code manually to determine which Pods are in the StatefulSet.

Debug Pods

Once you know which Pods are in the StatefulSet, debug them one-by-one using standard Pod and container troubleshooting techniques, including:

• Check Pod state to determine if any Pods failed to start. And remember that since StatefulSet Pods are ordered, an issue with one Pod may cause other Pods never to start.
• Look at Pod exit codes and error codes to determine why Pods may have failed.
• View operating system logs from nodes that host the Pods for additional context about failure events.
• Try starting the containers defined for the Pods manually to make sure you can successfully pull the images and run containers based on them.

Step-by-step Pod initialization

If your Pods appear to operate normally when you run them individually but you still have issues with the StatefulSet, consider starting Pods within the StatefulSet one-by-one. You can do this by adding the following annotation to your StatefulSet configuration:

Then, redeploy the StatefulSet. Because of the annotation, Kubernetes will start the first Pod in the set, then pause before starting others. This allows you to validate manually that the first Pod has started successfully. If it has, you can tell Kubernetes to proceed with launching the next Pod in the set by running:

Be sure to replace pod-name with the name of the actual Pod you want to launch.

Proceed in a similar fashion by verifying that the second Pod is operating normally, then move on to the following Pod, and so on until you've debugged all Pods in the set. With any luck, you'll figure out which Pod is holding up the others from starting, then dive deeper to determine why the problematic Pod is not operating normally.

Deleting a StatefulSet as part of Troubleshooting

In the course of troubleshooting a StatefulSet, you may decide that you want to delete it entirely, and then restart from scratch.

You can do this using a kubectl delete command like the following:

Alternatively, you can specify the name of the YAML file you used to start the StatefulSet when deleting it:

After deleting the StatefulSet, you may restart it with a kubectl apply command.

StatefulSet best practices

To get the most out of StatefulSets, consider the following best practices:

• Assign unique labels: Using unique, meaningful labels when creating a StatefulSet makes it easier to list the Pods associated with the StatefulSet and perform troubleshooting tasks.
• Configure unique resources for each Pod: Rather than trying to share a persistent volume or other stateful resources between Pods, configure a different one for each Pod.
• Use StatefulSets only when necessary: Avoid using StatefulSets when there is a simpler way to achieve your goals. For example, if you simply want to create replicas of a Pod, you can do so using a ReplicaSet, which is less complicated to configure and manage than a StatefulSet.
• Avoid unusual configurations: As we mentioned above, StatefulSets are designed with the assumption that you manage Kubernetes resources in a traditional way – using PersistentVolume provisioners or headless Services, for example. If you are dealing with edge cases or bespoke configurations, StatefulSets may not work well, and you should probably standardize your configurations to make them look more like "classic" Kubernetes resources before creating StatefulSets.

StatefulSet Troubleshooting with groundcover

If you run into a problem with StatefulSets that you just can't sort out, groundcover has you covered. By continuously monitoring and observing every log, metric, and trace available from your Kubernetes cluster, groundcover detects performance anomalies, and then provides the deep context necessary to troubleshoot them quickly.

But don't just take our word for it. See for yourself by requesting a demo.

Sometimes, each Pod is truly unique…which is why we love StatefulSets

In short, StatefulSets are your go-to solution in any scenario where you need to treat each Pod as its own special, persistent entity – in other words, when you want to assign a unique identity to the Pod and carefully control how Kubernetes manages it.

StatefulSets do create more complexity than standard Deployments, and you shouldn't use a StatefulSet unless you require the specific capabilities that it supports. Most stateless Pods are not special, and should not be part of a StatefulSet. But when your Pod needs some extra handholding, a StatefulSet is the way to give it the attention it requires.