Kubernetes Federation: How to Set It Up & Best Practices

If Kubernetes were a political system, most Kubernetes clusters would be akin to independent states – meaning countries that operate autonomously, without being part of any larger confederation or alliance.

But there's an exception: Clusters managed using Kubernetes federation. In Kubernetes, federation turns clusters into something akin to a federal state where a central authority (a host cluster) oversees quasi-independent entities – kind of like the United States or the European Union.

We won't take the analogy further than this because Kubernetes isn't actually a political system and it's hard to compare clusters to countries. What we will do in this article, however, is explain the benefits of Kubernetes federation as a means of managing multiple clusters at once. We also cover the potential downsides of Kubernetes cluster federation, and we explain how to get started setting up a federated collection of clusters.

What is Kubernetes federation (KubeFed)?

Kubernetes federation, or KubeFed, is a type of Kubernetes architecture that makes it possible to manage multiple Kubernetes clusters as a single entity. With Kubernetes federation, a single control plane connects to multiple clusters, tracking their state and allowing admins to enforce policies and deploy resources across them.

KubeFed is also the name of an open source project designed to enable Kubernetes cluster federation. That project has been mostly inactive for the past several years, but developers in the Kubernetes ecosystem are still working on various approaches to supporting federated clusters. So, when we talk about Kubernetes federation in this article, we're referring to federation as a general concept, not the KubeFed project specifically – although we'll explain later how to use KubeFed to set up federation.

Kubernetes federation architecture

Typically, a federated Kubernetes cluster includes the following architectural components:

• A host cluster, which stores the configurations that govern the other clusters within the federated environment.
• Clusters that are members are the federation. These are typically called member clusters or worker clusters.
• An API that allows the host cluster to communicate with the member clusters and apply configurations to them.

This approach effectively allows a central control plane – the one running in the host cluster – to manage multiple Kubernetes clusters.

That said, it's important to understand that more than one control plane exists under the Kubernetes federation. All of the clusters – the host cluster and the member or worker clusters – have independent control planes. However, the host cluster's control plane essentially tells the member clusters' control planes what to do using the centralized API.

This means that each of the member or worker clusters operates as a distinct entity, with clear boundaries separating it from the other member clusters. At the same time, however, all clusters remain under the oversight of the central control plane.

Again, it's kind of like how in a country like the United States, each state government is quasi-independent and is able to do things like collect its own taxes and establish various types of laws that apply within its borders – just as each worker cluster decides how to manage workloads within its cluster. But at the same time, the federal government exercises authority across all the states, similar to how the host cluster governs the worker clusters in federated Kubernetes.

Multiple Kubernetes clusters vs. federated Kubernetes clusters

It's also important to understand that Kubernetes federation is different from simply running multiple clusters. The term multi-cluster Kubernetes sometimes refers to any Kubernetes setup or strategy where an organization operates multiple clusters at once. However, it doesn't necessarily imply that the organization has a unified control plane to manage multiple clusters in a centralized way.

So, deploying multiple clusters is a step toward Kubernetes federation, and multiple clusters can be joined to create a federation, but not all multi-cluster environments are federated Kubernetes clusters.

Kubernetes federation pros and cons

Compared to single-cluster setups, as well as architectures where multiple clusters exist but operate independently, federated Kubernetes provides several pros – but it also presents a few key disadvantages.

Pros

The advantages of federated Kubernetes include:

• Enhanced scalability, since each cluster can scale up and down independently of others. At the same time, however, it's possible to move workloads across multiple Kubernetes clusters.
• Greater fault tolerance and disaster recovery capabilities because if one cluster fails, the others can take over and host affected workloads.
• The potential for better application performance because organizations can place workloads in a cluster located closer to different groups of end-users, which reduces network latency.

Cons

On the other hand, Kubernetes federation may present the following challenges:

• Less isolation between clusters, which could increase the risk that a security breach in one cluster may spread to impact others.
• Low maturity in the tools available for setting up and managing multiple Kubernetes clusters.
• Increased bandwidth and hosting costs due to the resources required to operate a host cluster on top of worker clusters.

How to set up multiple Kubernetes clusters using federation

As we mentioned, there are multiple tools available for setting up and managing multiple Kubernetes clusters using a federated approach. Some are more mature than others, and some work with any Kubernetes distribution whereas others are tied to certain ecosystems.

Setting up federation with KubeFed

KubeFed is an agnostic cluster federation tool that will work with most Kubernetes distributions and configurations. The drawback is that KubeFed is no longer actively developed, as we noted above, but it may still work with your Kubernetes environments.

To set up a cluster federation using KubeFed, follow these steps.

1. Install KubeFed

First, using Helm, install KubeFed in the cluster that will serve as your host cluster:

helm --namespace kube-federation-system upgrade -i kubefed kubefed-charts/kubefed
--create-namespace --kube-context hostcluster

2. Join clusters

Next, using the kubefedctl CLI tool, join worker clusters to your host to create a confederation of clusters:

kubefedctl join cluster1 --luster-context admin@hostcluster --host-cluster-context Admin@Cluster1
kubefedctl join workercluster1 --cluster-context admin@workercluster1 --host-cluster-context admin@hostcluster
kubefedctl join workercluster2 --cluster-context admin@workercluster2 --host-cluster-context admin@hostcluster

This joins two worker clusters – named workercluster1 and workercluster2 – to the federation.

3. Validate configuration

To check that you successfully created a federated collection of clusters, run:

kubectl -n kube-federation-system get kubefedclusters

The output should display a list of clusters joined to the federation – such as workercluster1 and workercluster2.

4. Set up a federated namespace

Optionally, you can now create a federated namespace using a manifest like the following:

apiVersion: v1
kind: Namespace
metadata:
  name: federated-namespace1
  labels:
	federated: “true”

This is useful because any resources that you deploy within this namespace (called federated-namespace1 in the example code above) will be applied across all of your worker clusters.

Note that you can still deploy resources to individual clusters using their local namespaces if you wish.

Other ways to set up federated Kubernetes clusters

Depending on your Kubernetes distribution, there may be other tools available for creating federated Kubernetes clusters or a similar setup.

For example, Rancher's Multi-Cluster Management (MCM) tool allows you to connect multiple clusters to a central control panel. It's not exactly the same as using the KubeFed approach, but the end result is similar. Likewise, Amazon supports a federated cluster architecture based on Elastic Kubernetes Service (EKS).

Kubernetes federation best practices

To make the most from Kubernetes federation, consider the following best practices:

Choose the right federation tool

In general, federation tools associated with your Kubernetes distribution or vendor are likely to work better than the KubeFed tool, due to the latter's lack of active development. That said, KubeFed may be your best bet if there are no other federation tools that are compatible with your distribution.

Minimize network latency

In cases where the host cluster is located at a geographically separate site from worker clusters (e.g., if they are in different data centers), network latency can pose challenges. If it takes a few seconds for data to move between clusters, the clusters won't be in perfect sync.

To mitigate this issue, strive to minimize network latency between clusters. Your ability to do this will depend on which network infrastructure and optimizations are available in each of the sites that host your clusters. But in general, solutions like network interconnects (if they are available from your data center provider) can help speed network performance and, by extension, improve federated cluster performance.

Avoid unnecessary cross-cluster resources

Due to the challenge of keeping distributed clusters in sync, it's a best practice to avoid deploying cross-cluster resources unless you actually need them. If a given workload can run in just one cluster, keep it there. Save cross-cluster deployments for situations where you actually need resources to exist within every cluster at once – for example, if you have a mission-critical app that needs to be available in every cluster.

Validate and test multi-cluster setups

Kubernetes health checks, configuration validation and testing are important in any context. But they're especially critical when you use cluster federation, due to the complexity of this type of setup and the fact that federated cluster support for Kubernetes is not yet fully mature. Before entrusting production workloads to a federated cluster, make doubly sure that they run as required.

Kubernetes cluster federation alternatives

As we mentioned, although creating federated clusters with KubeFed is the traditional approach to Kubernetes cluster federation, it's not the only way to achieve federated clusters or something resembling them. You can also use tools like Rancher MCM or Mirantis Container Cloud, which provide multi-cluster management features. Some Kubernetes distributions or services also let you connect and manage multiple clusters through a unified set of tools.

The downside of most of these alternative solutions is that they may not work with every Kubernetes distribution. They may also tie you into a particular vendor's ecosystem. If you're worried about these things, sticking with KubeFed, an agnostic, open source solution, may be preferable. Just remember, though, that KubeFed isn't fully mature and is not currently under active development, so you're taking a certain type of risk in using KubeFed, too.

Note, too, that Kubernetes developers maintain some open source projects, like Cluster Registry and Cluster Inventory API, that share similar goals to KubeFed; for example, they can help track the state of resources across clusters using a centralized API. Like KubeFed, most of these solutions are not actively developed at present. But they may be worth checking out if you are committed to a fully open source, vendor-agnostic approach to Kubernetes cluster federation. You can read more about these various solutions on GitHub. 

Kubernetes observability with groundcover

No matter how you choose to go about setting up federated clusters, groundcover's Kubernetes monitoring and Kubernetes observability capabilities can help ensure that they perform optimally. By providing deep visibility into the state of every Kubernetes resource – nodes, Pods, services and more – on each and every cluster you operate, groundcover alerts you to performance risks and problems. It also provides the actionable intelligence you need to get to the root of problems fast.

Observability capabilities like these are extra important when you're dealing with multi-cluster setups and need to determine whether a performance issue results from, say, a problem with a worker cluster's local configuration, a latency issue on the network that connects your host and worker clusters or a problem with the host cluster. With groundcover, you can tell the difference fast.

Indivisible clusters, with liberty and justice for all

There's nothing wrong with operating just a single Kubernetes cluster – which, indeed, is the approach that a majority of organizations are likely to take for the foreseeable future.

But sometimes, it makes more sense to deploy more than one cluster, while uniting them all under the flag of a single control plane. That's precisely what the Kubernetes federation lets you do. While this approach remains a bit rough around the edges due to the lack of fully mature, distribution-agnostic federation tools, it's still worth considering if goals like high fault tolerance or the distribution of multiple clusters across different sites are priorities for you.