Kubernetes external API calls slow and DNS load high: ndots:5 search path expansion

Our checkout service calls an external payment API for every order. Its own metrics showed p50 of 180 ms for those calls, but the payment provider's dashboard reported 110 ms server time, and network RTT to their region was around 15 ms. Something was adding 40 to 200 ms on our side. At the same time, CoreDNS pods were the top CPU consumers in the cluster, and we had scaled them to 8 replicas without a clear reason why.

Reproducing it

From inside a checkout pod:

$ cat /etc/resolv.conf
search shop.svc.cluster.local svc.cluster.local cluster.local eu-west-1.compute.internal
nameserver 10.100.0.10
options ndots:5

Capturing DNS traffic during one API call (with a debug sidecar running tcpdump -n port 53):

A?    api.payments.example.shop.svc.cluster.local.     NXDomain
AAAA? api.payments.example.shop.svc.cluster.local.     NXDomain
A?    api.payments.example.svc.cluster.local.          NXDomain
AAAA? api.payments.example.svc.cluster.local.          NXDomain
A?    api.payments.example.cluster.local.              NXDomain
AAAA? api.payments.example.cluster.local.              NXDomain
A?    api.payments.example.eu-west-1.compute.internal. NXDomain
AAAA? api.payments.example.eu-west-1.compute.internal. NXDomain
A?    api.payments.example.                            203.0.113.44
AAAA? api.payments.example.                            (no records)

Ten queries to resolve one external name, eight of them guaranteed to fail.

What we ruled out

• Upstream DNS slowness. The final, real query took under 2 ms from the cache.
• Network latency to the provider. TCP connect times were consistent with 15 ms RTT.
• Connection reuse. The HTTP client reused connections, but DNS was re-resolved on each new connection, and connections were recycled every 30 seconds by our client settings.

Root cause

Kubernetes sets ndots:5 in pod resolv.conf. Any name with fewer than five dots is treated as possibly relative, so the resolver tries each search domain first. api.payments.example has two dots, so every lookup walked four search domains, for both A and AAAA records, before trying the name as written. Each failed lookup was a round trip to CoreDNS, which forwarded the last search domain query (compute.internal) to the VPC resolver. Under load, the extra queries added latency to every new connection and multiplied CoreDNS load roughly fivefold for external names.

What is happening under the hood

The resolver rule: if a name contains fewer dots than ndots, try it with each search domain appended first, then as an absolute name. If it has at least ndots dots, try it as absolute first. A name ending in a dot is always absolute and never uses search domains.

Kubernetes uses ndots:5 so that names like my-service.my-namespace.svc (two dots) resolve through the search list. That is great for in-cluster names and expensive for external ones. Modern glibc and musl send A and AAAA queries in parallel or in sequence, so each search attempt is two queries.

CoreDNS caches negative answers too, which helps, but cache entries are per name and expire, and many pods per node still produce a lot of traffic.

Environment

Minimal reproduction

kubectl run dnstest --rm -it --image=nicolaka/netshoot -- \
  bash -c 'for i in 1 2 3; do dig +search +showsearch api.payments.example | grep -c ";; ANSWER"; done'

+showsearch prints each search domain attempt.

The fix

1. Use fully qualified names with a trailing dot for external hosts. The payment client base URL became:

https://api.payments.example./v2/

One A and one AAAA query, no search expansion. TLS still works because clients strip the trailing dot for SNI and certificate checks, but we tested this with our specific HTTP client before rolling out, since some older libraries handled it badly.

2. Lower ndots for pods that mostly talk to external services.

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"

With ndots:2, names with two or more dots are tried as absolute first. In-cluster calls must then use at least service.namespace (one dot, still searched) or the full service.namespace.svc.cluster.local. We audited service URLs in config first: all used service.namespace.svc.cluster.local or service, both of which still work.

3. NodeLocal DNSCache. A DNS cache on every node answers repeated queries locally, removes the conntrack race issues that affect UDP DNS through kube-proxy, and cuts CoreDNS load. We enabled it cluster-wide.

4. Longer DNS caching in the client. The JVM was configured with a 10-second positive DNS cache; we raised it to 60 seconds for this service, which matches the provider's record TTL.

Alternatives and trade-offs

• ndots:1. Even fewer queries, but service.namespace style names would be tried as absolute first, adding a failed external lookup for in-cluster calls.
• Remove the cloud search domain (eu-west-1.compute.internal) through kubelet resolv.conf settings. Saves one attempt per lookup; we did not need internal EC2 hostnames from pods.
• Only scale CoreDNS. Treats the symptom; latency per lookup stays.

How to test the fix

$ kubectl exec deploy/checkout -- bash -c 'time getent hosts api.payments.example.'
203.0.113.44   api.payments.example
real    0m0.004s

A capture during a load test showed two DNS queries per new connection instead of ten.

Confirming in production

• p50 latency of payment API calls fell from 180 ms to 128 ms, and p99 from 610 ms to 240 ms.
• CoreDNS queries per second dropped by 71 percent after the change plus NodeLocal DNSCache; we scaled CoreDNS back to 3 replicas.
• NXDOMAIN responses from CoreDNS, a good proxy for search expansion, dropped by about 85 percent.

Regression guard

• A dashboard of CoreDNS NXDOMAIN rate by query suffix; spikes for cluster.local suffixes of external names are visible immediately.
• Service templates set ndots: "2" by default, with a documented exception for workloads that rely on short in-cluster names.
• Lint check for external URLs in Helm values: warn when they lack a trailing dot in latency-sensitive services.

Security implications

Search domain expansion can leak internal query names to upstream resolvers, and in some environments it can resolve names you did not intend if someone controls a matching name under a search domain. Using FQDNs makes resolution explicit. NodeLocal DNSCache adds a component on every node; it runs with host networking and needs to be kept patched like any other node-level agent.

Prevention checklist

• Know that ndots:5 makes external names expensive.
• Use trailing-dot FQDNs for external dependencies in hot paths.
• Lower ndots per workload after auditing internal names.
• Run NodeLocal DNSCache on busy clusters.
• Monitor CoreDNS NXDOMAIN rate and query volume.

References

• Kubernetes DNS for Services and Pods, pod DNS config — https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config
• NodeLocal DNSCache — https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/
• resolv.conf(5), ndots — https://man7.org/linux/man-pages/man5/resolv.conf.5.html
• CoreDNS cache plugin — https://coredns.io/plugins/cache/