Incident Postmortem: Debugging a 3-Second DNS Delay in EKS

The Incident

At 14:00 UTC, PagerDuty triggered multiple alerts for elevated 5xx error rates across three critical microservices running in our production EKS cluster. The affected services were returning `502 Bad Gateway` errors to the ingress controller, causing a 12% drop in successful checkout requests.

The Investigation

Initial metrics in Grafana showed CPU and memory utilization were well within normal boundaries (sub-40% across all nodes). However, reviewing our distributed tracing (Jaeger) revealed a massive anomaly: inter-service HTTP requests were taking exactly 5,000ms to timeout, or succeeding after exactly 3,000ms.

This 3-second delay is a classic signature of DNS lookup timeouts in Alpine Linux environments due to `ndots:5` configurations and IPv6 AAAA record resolution failures.

The Root Cause

The microservices were built using Node.js Alpine images. Alpine uses `musl` libc, which processes DNS queries sequentially rather than in parallel (unlike `glibc`).

The Configuration Fix

To immediately mitigate the issue, we deployed NodeLocal DNSCache and patched our deployment manifests to adjust the `dnsConfig`.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  template:
    spec:
      dnsConfig:
        options:
          - name: ndots
            value: "2"
          - name: single-request-reopen

The Permanent Solution

While tuning `ndots` resolved the immediate timeouts, we fundamentally re-architected the cluster's DNS pipeline:

• Rolled out NodeLocal DNSCache to all nodes via DaemonSet, entirely bypassing `kube-dns` for local queries.
• Migrated critical high-throughput Node.js microservices off `alpine` base images to `debian-slim` to leverage `glibc` parallel DNS resolution.