Kubernetes includes a DNS server, Kube-DNS, for use in service discovery. This DNS server utilizes the libraries from SkyDNS to serve DNS requests for Kubernetes pods and services. The author of SkyDNS2, Miek Gieben, has a new DNS server, CoreDNS, that is built with a more modular, extensible framework. Infoblox has been working with Miek to adapt this DNS server as an alternative to Kube-DNS.
CoreDNS utilizes a server framework developed as part of the web server Caddy. That framework has a very flexible, extensible model for passing requests through various middleware components. These middleware components offer different operations on the request - for example logging, redirecting, modifying or servicing it. Although it started as a web server, Caddy is not bound specifically to the HTTP protocol, and makes an ideal framework on which to base CoreDNS.
Adding support for Kubernetes within this flexible model amounts to creating a Kubernetes middleware. That middleware uses the Kubernetes API to fulfill DNS requests for specific Kubernetes pods or services. And because Kube-DNS runs as just another service in Kubernetes, there is no tight binding between kubelet and Kube-DNS. You just need to pass the DNS service IP address and domain into kubelet, and Kubernetes doesn’t really care who is actually servicing the requests at that IP.
Today, CoreDNS Kubernetes middleware supports serving A records with the cluster IP of the service. In the near future, we will add the following features of Kube-DNS:
Serve pod IPs for A records of services without a cluster IP
Serve SRV records for named ports that are part of a service (with or without a cluster IP)
Serve PTR records (reverse DNS lookups)
Additionally, there are a few features unique to the CoreDNS integration:
Flexible record templates
Label-based filtering of responses
And of course, you have access to all of the various other middleware in CoreDNS.
To configure CoreDNS to provide Kubernetes service discovery, you have to set up your Corefile - that is, your CoreDNS configuration file. Here is an example that runs CoreDNS on port 53 and serves the cluster.local domain out of Kubernetes.
Figure 1: Simple Corefile for Kubernetes Service Discovery
This Corefile enables two middlewares - the log middleware and the kubernetes middleware. The log middleware is configured to write logs to STDOUT. If you do not include this statement, you won’t see any logging unless there are errors. Any requests for cluster.local will be fulfilled using the data from the Kubernetes API, which is automatically accessed via a service account. By default, services can be looked up with the same format used by Kube-DNS; for example, if you have a service called nginx running in the default namespace, then you would look up its cluster IP with a request for an A record nginx.default.svc.cluster.local.
This basic Corefile will handle the base requirements, but it is missing a few things we would have in a standard Kube-DNS install:
There is no way to do health checking of the CoreDNS instance.
There is no caching (Kube-DNS handles this through a separate dnsmasq instance)
Requests for domains other than cluster.local are not handled.
Fortunately, CoreDNS has middleware to enable all of these functions and a lot more. For health checking, we have the health middleware. This provides an HTTP endpoint on a specified port (8080 by default) that will return “OK” if the instance is healthy. For caching we have the cache middleware. This allows the caching of both positive (i.e., the query returns a result) and negative (the query returns “no such domain”) responses, with separate cache sizes and TTLs. Finally, for handling other domains we have the proxy middleware. This can be configured with several upstream nameservers, and also can be used to defer lookups to the nameservers defined in /etc/resolv.conf. Configuring these, we end with a Corefile like this:
Figure 2: Complete Corefile for Kubernetes Service Discovery
This Corefile serves Kubernetes services out of cluster.local and sends all other queries to the nameservers defined in /etc/resolv.conf, caching all responses for thirty seconds (for Kubernetes and proxied requests).
There are many additional middleware components available in CoreDNS. These can enable a lot more functionality than can be achieved with Kube-DNS. For example, while CoreDNS does not currently offer dynamic service registration directly, you can achieve a similar result by enabling the etcd middleware for another domain. We will take a closer look at that in our next post.
So, how do you get CoreDNS up and running as the cluster DNS? The details will vary a bit depending on how you have configured your Kubernetes cluster, but the same two steps apply:
Run CoreDNS as a service in your cluster
Update kubelet parameters to include the IP of CoreDNS and the cluster domain
If you are already running Kube-DNS, then the kubelet parameters will already define the cluster domain and IP; you simply need to replace your Kube-DNS service with CoreDNS, and use the same cluster IP.
We have a Kubernetes cluster setup using CoreOS as described here, on the CoreOS site. In this configuration, the specifications for system-level services are kept in /srv/kubernetes/manifests. So, to update our cluster to run CoreDNS, we replace the existing kube-dns-rc.yaml and kube-dns-svc.yaml with the coredns-de.yaml and coredns-svc.yaml below. This deployment uses a CoreDNS container image built especially for this blog (infoblox/coredns:k8sblog), but once CoreDNS v003 is released, the standard CoreDNS image may be used.
Notice that because we need to configure CoreDNS with a file, we also create a ConfigMap to house that file. Other than that, this is quite similar to the previous Kube-DNS versions of the files (though we are using a deployment instead of a replication controller).
Manifests in /srv/kubernetes/manifests are not automatically reloaded, so we need to manually delete the Kube-DNS entries and create the CoreDNS entries (this is not the way to do this in production!):
If you were not already running Kube-DNS, or you used a different ClusterIP for CoreDNS, then you’ll need to update the kubelet configuration to set the cluster_dns and cluster_domain appropriately. On our cluster, this is done by modifying the /etc/systemd/system/kubelet.service file, then restarting the kubelet service.