Kubernetes DNS Troubleshooting: Causes & Best Practices

In Kubernetes, DNS errors are the equivalent of a thief in the night: They arrive without warning, and – because Kubernetes doesn’t do a great job of explicitly telling you when DNS issues are occurring – they often go unnoticed until significant damage has been done.

The good news, however, is that with the right techniques and tools, it’s possible to find and fix DNS problems in Kubernetes. Keep reading for details as we walk through the ins and outs of DNS troubleshooting in Kubernetes environments.

What is Kubernetes DNS?

Kubernetes DNS is a built-in feature of Kubernetes that generates human-readable names for pods and services using the Domain Name System (DNS). It’s called kube-dns and is based on CoreDNS, an open source DNS server.

DNS capabilities in Kubernetes mean that rather than having network identities in the form of IP addresses alone, Kubernetes resources can be identified on the network using names or words. For instance, a service could be reachable at the hostname my-service as well as via the specific IP address assigned to it.

In the context of Kubernetes (and the Internet in general), DNS offers two main benefits:

1. Simplicity: DNS queries make it easier for humans to remember and refer to pods and services because they can use hostnames instead of IP addresses, which are more tedious to work with.
2. Consistency: Using Kubernetes DNS queries, you can maintain a consistent hostname for a pod or service even if the IP address assigned to it changes. This is valuable because it eliminates the need to update network configurations every time an IP address assignment changes.

Note that the purpose of Kubernetes DNS is to resolve names for pods and services inside Kubernetes only. On its own, the built-in DNS server doesn’t resolve external hostnames; it won’t tell you the IP address for google.com, for example. It would need to connect to an external DNS server for that purpose.

How Kubernetes DNS works

Because Kubernetes offers a built-in DNS feature, you don’t have to deploy any external tools (like a standalone DNS server) to perform DNS queries or manage DNS records for pods and services. Instead, Kubernetes automatically manages network identities based on DNS using the following steps:

• Name assignment: When you create a pod or service, Kubernetes assigns it a DNS name. By default, the name is the name of the pod followed by the pod's namespace – for example, mypod.default. However, you can specify a different hostname using the hostname field when creating a pod or service.
• Query management: When a pod or service wants to interact with another pod or service in the cluster, it sends DNS queries to Kubernetes DNS (which, again, runs on CoreDNS). The server responds by looking up the pod or service’s DNS name and sending the corresponding IP address back to the original pod or service.
• Tracking DNS changes: When IP address assignments for a pod or service change (which they might when pods or services restart or move to a different node, for example), kube-dns tracks the change automatically. This ensures that DNS queries will always resolve to the correct IP address, regardless of which IP address assignment changes take place in the background.

Common Kubernetes DNS issues

Unfortunately, many things can go wrong during the processes described above. Common DNS problems that you may experience on Kubernetes include:

• Delayed DNS responses: kube-dns may take a long time to respond to queries. These delays, which often result from network congestion or a lack of sufficient CPU or memory for the server, may impede the ability of pods and services to connect to each other because they can’t do so until they receive responses to DNS queries.
• Incorrect DNS resolutions: DNS resolutions may resolve names to the wrong IP addresses. This mainly happens as a result of outdated DNS records, but it could also result from bugs in kube-dns or your DNS configuration.
• DNS record conflicts: Conflicts in IP addresses or names (meaning multiple pods or services share the same IP address or name) could cause traffic to route to the wrong host following DNS resolution. In this case, the lookup isn’t wrong, exactly, but it won’t result in the intended outcome.
• DNS lookup failures (SERVFAIL): In some cases, a DNS lookup may fail entirely, which usually results in a SERVFAIL error. This is typically because kube-dns is unreachable either because it crashed or is no longer connected to the network.
• NXDOMAIN errors: An NXDOMAIN (short for “nonexistent domain”) DNS error happens when the DNS server can’t find the appropriate domain name for DNS queries. The cause is typically outdated or incomplete DNS records.

External domain failures: Although Kubernetes DNS deals primarily with the DNS resolution of internal names, situations may arise (such as deployments that require Kubernetes to integrate with external cloud services) where it needs to resolve an external hostname. If the kube-dns can’t reach external DNS servers (which manage records for domains outside the Kubernetes cluster), you’ll experience external DNS resolution failures, even if the internal kube-DNS service is working normally.

How to troubleshoot Kubernetes DNS problems

Part of what makes it so challenging to troubleshoot DNS issues on Kubernetes is that Kubernetes doesn’t explicitly alert you to DNS problems. Depending on the nature of the issue, it may be recorded in a log, but even then, you have to find the appropriate log file. Plus, many DNS failures simply occur silently, with DNS traffic failing to reach its destination but with no specific indication that DNS is the issue.

So, the first step in Kubernetes troubleshooting when you suspect DNS issues is to confirm that a DNS failure of some type is indeed at fault. The simplest way to do this is to deploy a pod that provides some simple network diagnostic tools. Kubernetes provides a pod for this purpose that you can deploy using this command:

kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml

By default, the pod (which is named dnsutils) deploys to the default namespace. If you need to troubleshoot DNS within a different namespace, modify the pod’s YAML code to change to a different namespace. This is important because most Kubernetes DNS functionality operates on a per-namespace basis, so you need to deploy your tools within the appropriate namespace.

Once the dnsutils pod is running, you can use the kubectl exec command to run DNS tests. For example, to check whether Kubernetes can resolve the hostname mypod.default (which would correspond by a pod named mypod hosted in the default namespace), run:

kubectl exec -i -t dnsutils -- nslookup mypod.default

If DNS is working properly, the command should respond with the name and IP address of the pod, in a format similar to:

Server:	 1.2.3.4
Address 1: 1.2.3.4

Name: mypod.default
Address 1: 2.3.4.5

This tells you that mypod.default resolves to the IP address 2.3.4.5.

If the correct information isn’t returned, or you see an error message, you know something is wrong with DNS in your Kubernetes environment. The most common error scenarios that you can discover via this method include:

• Server connection failure: If DNS queries can’t reach the server at all, your server is likely down or unreachable because of a network configuration issue.
• Lookup failure: A message similar to can't resolve ‘mypod.default' indicates that the DNS record for mypod is not available, likely because of incomplete DNS records.
• Slow DNS responses: If you get accurate DNS responses but it takes a long time, try using the traceroute command to trace connections to kube-dns. This may identify which part of the network is causing a slowdown – or, if packets reach the server quickly but it still takes a while for a response, the issue is likely that the server is overloaded, rather than a networking problem.

Advanced DNS debugging techniques

If the method described above doesn’t provide all of the information you need about DNS errors in Kubernetes, you can run some more extensive checks for additional information.

Check DNS logs

First, check the logs for kube-dns. You can view the logs using the following command:

kubectl logs --namespace=kube-system -l k8s-app=kube-dns

Warnings or error messages in the logs may help you hone in on the source of your DNS problem, especially if the root of the issue lies with kube-dns itself as opposed to the network.

Check kube-dns resource utilization

Checking the CPU utilization of the kube-DNS service can help you determine whether the server is simply overloaded. To view real-time CPU metrics, run:

kubectl top pods -n kube-system -l k8s-app=kube-dns

Check server permissions

In some cases, insufficient permissions for kube-dns can cause issues because they prevent the server from being able to list the services and endpoints necessary to resolve names. Verify permission settings using the following command:

kubectl describe clusterrole system:coredns -n kube-system

The output should indicate that kube-dns has list and watch permissions for endpoints, namespaces, pods, services and endpointslices. If any of these values are missing, use the following command to add the appropriate permissions:

kubectl edit clusterrole system:coredns -n kube-system

Tools for Kubernetes DNS debugging

We’ve touched on some of the tools you can use for Kubernetes DNS error management above, but let’s describe them in more detail.

The most important tools include:

• nslookup: Looks up DNS records.
• dig: Another tool for looking up DNS records.
• host: Resolves a hostname to an IP address based on DNS records. Handy for a quick lookup, but provides less detail than nslookup and dig.
• ping: Sends simple requests to a server to check whether it responds and how long the responses take. Useful for determining whether a DNS server is up and tracking response latency.
• traceroute: Traces the flow of packets on the network. Helpful for identifying which segment of a network is associated with DNS traffic slowdowns.

These are all generic command-line tools; they’re not specific to Kubernetes. As noted above, however, you can run these commands inside any pod that has them installed using the kubectl exec command. Note that kubectl itself offers no built-in DNS-related tooling, so you’ll need to lean on external tools for troubleshooting purposes – but in most cases you’ll want to run them through kubectl rather than executing them directly on a node because otherwise, DNS requests will be routed to the external DNS server that the node uses, rather than internal Kubernetes DNS.

How to prevent Kubernetes DNS issues

To minimize your risk of experiencing DNS issues on Kubernetes in the first place, consider best practices like the following:

• Allocate sufficient resources: Failure to provide enough CPU and memory resources to your Kubernetes cluster could cause DNS issues because it may result in kube-dns not having enough resources to operate properly, especially when it receives large numbers of requests.
• Create pod replicas for kube-dns service: To increase the availability of the kube-dns service, create replicas of the pod that hosts it so that if one pod fails, the service will remain available. Many Kubernetes distributions do this by default.
• Avoid duplicate names: If you assign hostnames explicitly rather than using the defaults created by Kubernetes, make sure to avoid giving the same name to multiple pods or services.
• Minimize networking restrictions: Network policies that restrict DNS traffic flow may lead to issues where a pod or service can’t reach the DNS server. To reduce this risk, avoid networking restrictions except in cases where you have a specific reason to implement them.

Troubleshooting Kubernetes DNS with groundcover

As a comprehensive Kubernetes observability solution, groundcover delivers the visibility you need to detect and fix Kubernetes DNS issues. You can use groundcover to monitor the status of DNS pods and get ahead of problems like CPU or memory exhaustion. You can also track network health and performance to determine whether issues like latency are undercutting pod performance.