DNS

The 30-second version

Computers on the internet find each other by IP address, the way phones call numbers. But nobody wants to type 142.250.74.46 to search the web. DNSDNSThe Domain Name System: the internet's phone book that turns a name like example.com into an IP address your computer can connect to. (the Domain Name System) is the lookup step in between: before your browser, your Go service or your Node script can send a single byte to api.example.com, something has to ask "what's the IP address for this name?"

One phone book? No — a tree of them

No single server could hold every name on the internet and answer billions of questions a second. So DNS is split into a tree, and each part of the tree is run by whoever owns it. Read a name right to left to walk down from the top:

• The invisible root at the top (written as a final ., which you almost never type) knows who runs each ending: com, org, dev, uk …
• The com servers know who runs each name registered under .com — example.com, google.com, millions more.
• example.com's own servers know the actual addresses: www.example.com, api.example.com, and so on.

Nobody has to know everything. Each level just knows who to ask next. Section §02 walks through a real lookup, hop by hop.

Why "it's always DNS"

There's a running joke among engineers: when something mysterious breaks, "it's always DNS." Three facts explain it:

A first taste

Here is the question your program asks, in three forms. digdigA command-line tool that sends a DNS query and prints the full reply. The standard way to see what DNS actually says (install the dnsutils or bind-tools package; macOS has it built in). is the command-line tool engineers use to ask DNS directly; we'll lean on it all through this topic.

$ dig +short example.com
192.0.2.10

The map

§02–§04 are the basics: how a lookup travels, and what DNS can store. §05–§07 are what bites developers: caching and TTLs, what goes over the wire, and how Go and Node actually resolve names. §08–§11 are the production stories: real failure modes, debugging, security, and DNS as a service-discovery and load-balancing tool. §12–§14 wrap up with pitfalls, a quiz and a cheat sheet.

What happens when you type example.com

The cast

Five players take part in a lookup. Two of them work for you; three are the tree from §01.

Step through a real lookup

Pick a scenario and step through it. Watch two things: the cache in each box filling up, and the latency bar at the bottom growing with each round trip across the internet.

What you just saw

Each server between the resolver and the answer replied with a referral: "I don't know, but these servers do." Handing off part of the tree like that is called DelegationDelegationHanding control of part of the DNS tree to other servers. The .com servers delegate example.com by publishing NS records that say “ask these servers about example.com”.. The resolver follows referrals until it reaches a server that does know — the authoritative one — and that server answers with the record.

There are two styles of asking, and both appeared in the walk-through:

• Your OS asks recursively: "find me the final answer, however many hops it takes." It sends one question and waits for one reply.
• The resolver asks iteratively: each server answers just one hop ("ask the .com servers next") and the resolver does the next hop itself.

A chicken-and-egg detail: glue

The .com servers said "example.com is handled by ns1.example.com". But to reach ns1.example.com, you'd need its IP… which only example.com's servers know. Round and round.

DNS breaks the loop with a Glue recordGlue recordAn IP address the parent zone publishes for a name server whose own name is inside the zone it serves (ns1.example.com for example.com). Without it you'd need the answer to find the server that has the answer.: alongside the referral, the .com servers also hand over ns1.example.com's IP address. That's why, in the cold scenario above, the .com referral arrived with an address attached. When your name servers live under a different domain — like ns1.example-dns.net, which the rest of this topic uses for example.com — no glue is needed: the resolver just looks that name up first, which can cost extra round trips.

Record types: what DNS can store

Anatomy of a record

Every piece of DNS data is a DNS recordDNS record (resource record)One line of DNS data: a name, a type (A, CNAME, MX …), a TTL and a value. Example: example.com. 300 IN A 192.0.2.10 means “example.com's IPv4 address is 192.0.2.10, cache it for 300 seconds”. with the same five fields. Here's one, exactly as dig prints it:

The TTL is the TTLTTL (time to live, DNS)How many seconds a DNS answer may be cached before it must be asked for again. Set by the domain owner on each record. A TTL of 300 means “you may reuse this answer for up to 5 minutes”.: how many seconds anyone may cache this answer before asking again. It gets a whole section (§05) because it causes more confusion than everything else here combined.

A name can have several records of the same type — two A records means "either of these two IPs works". The group of all records with one name and one type is answered together, as a set.

Reading dig output

Without options, dig prints the whole reply. It looks noisy, but only four parts matter (output trimmed — real dig prints a few more lines, including an OPT PSEUDOSECTION: that's the one ADDITIONAL record, an EDNS marker explained in §06):

$ dig example.com A

; <<>> DiG 9.18.28 <<>> example.com A
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31012     ← ① status
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:                                          ← ② what you asked
;example.com.                   IN      A

;; ANSWER SECTION:                                            ← ③ the records
example.com.            300     IN      A       192.0.2.10

;; Query time: 23 msec                                        ← ④ who answered, how fast
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; MSG SIZE  rcvd: 56

1. status: NOERROR means the name exists (the answer section may still be empty if it has no record of that type). NXDOMAIN means the name doesn't exist at all — that's the NXDOMAINNXDOMAINThe DNS answer “this name does not exist”. Different from “the name exists but has no record of the type you asked for” (called NODATA), and from a timeout (nobody answered). answer. SERVFAIL means the resolver couldn't get an answer, e.g. the domain's servers are broken.
2. flags: qr = this is a reply, rd = "recursion desired" (you asked the resolver to do the full walk), ra = "recursion available". An aa flag would mean the answer came straight from an authoritative server rather than a cache.
3. answer section: the records, in the five-field format above.
4. server and time: which resolver answered (1.1.1.1, on PortPortA number from 0 to 65535 that picks which program on a machine should get the traffic — like an apartment number after the street address. HTTPS servers usually listen on port 443. 53), and how long it took. Run the same command twice: the second time is usually a few milliseconds, because the resolver cached it — and the TTL will have gone down.

Most of the time you just want the records: dig +short example.com prints only the values, and dig +noall +answer example.com prints only the answer section.

Explore the record types

Each scenario is something real software does. Pick one to see the dig command, its output, and what the client does with the answer.

$ dig +noall +answer example.com A
$ dig +noall +answer example.com AAAA

example.com.		300	IN	A	192.0.2.10
example.com.		300	IN	AAAA	2001:db8::10

The whole family on one table

CNAMEs, the apex problem and ALIAS records

Why aliases exist

Say you host your site on a CDNCDNContent Delivery Network: servers spread around the world that keep copies of your files close to users, so they're served from a nearby city instead of your origin server.. The CDN runs thousands of servers and changes their IPs whenever it likes. If you copied their IPs into your own A records, you'd have to update them every time the CDN moved — and you'd never know when.

Instead, you publish one record that points at a name the CDN controls:

$ dig +noall +answer www.example.com
www.example.com.                3600  IN  CNAME  example-com.edge.cdn.example.net.
example-com.edge.cdn.example.net.  60  IN  A      203.0.113.7
example-com.edge.cdn.example.net.  60  IN  A      203.0.113.8

Read it top to bottom: www.example.com is an alias for example-com.edge.cdn.example.net, and that name has two IPs. The CDN changes its own A records whenever it wants (notice its short 60-second TTL), and your CNAME never has to change.

The rule that causes all the trouble

A CNAME means "everything about this name lives over there". So the DNS standard says: a name with a CNAME may have no other records at all — no A, no MX, no TXT. Otherwise a resolver asking for the MX wouldn't know whether to use the local MX or follow the alias.

That rule collides with the zone apex — the bare domain, example.com itself, also called the Zone apexZone apex (naked domain)The top name of a zone itself — example.com, without www. or any other prefix. It must carry SOA and NS records, which is why it can't be a CNAME.. Every zone's apex must carry an SOA record and NS records (§03). So the apex already has other records, and can't be a CNAME.

example.com.  3600  IN  SOA    ns1.example-dns.net. …   ← required at the apex
example.com.  3600  IN  NS     ns1.example-dns.net.     ← required at the apex
example.com.  3600  IN  CNAME  myapp.herokudns.com.     ← ✕ not allowed alongside them

In practice: www.example.com can be a CNAME to your hosting provider, but example.com can't. Many DNS dashboards simply refuse to save it.

The workarounds

TTLs and the caches hiding everywhere

"DNS propagation" is really cache expiry

People say "wait for DNS to propagate", as if the new record were travelling around the world. It isn't. The moment you save it, your authoritative servers start serving it. What you're actually waiting for is every cache that already holds the old answer to throw it away.

Worked example. Your record has a TTL of 3600 (one hour), and you change the IP at 12:00:

• A resolver that last fetched it at 11:10 keeps the old IP until 12:10 (11:10 + 1 hour).
• A resolver that fetched it at 11:59 keeps the old IP until 12:59.
• A resolver that hasn't asked since yesterday has nothing cached — it gets the new IP on its next question.

So the worst case for well-behaved caches is "change time + old TTL": 13:00 here. Note that it's the old TTL that matters — whatever TTL was attached to the answer at the moment each cache stored it.

The caches between your users and your servers

See it happen

The record below changes from the old IP to a new one at time 0. Each row is a client that last looked up the name at a different moment. Drag through time, change the TTL, and try the two ways of preparing for a migration.

The migration playbook

You're moving api.example.com to a new server. The trick is to make the TTL short before the move, and to keep the old server alive after it.

1. Days before (at least one old-TTL ahead — a full day if the TTL is 86400): lower the TTL to 60 seconds. Now wait for the old, long-TTL copies to expire everywhere. Lowering it 10 minutes before the move does almost nothing — the widget's "too late" option shows why.
2. Move: change the record to the new IP. Well-behaved caches pick it up within about a minute.
3. Keep the old server running and watch its traffic fall. Some clients ignore TTLs, and long-lived connections never re-resolve. Shut it down only when its traffic is near zero — often a day or more later.
4. After: raise the TTL again (an hour is a common choice) so resolvers cache it and your DNS provider isn't asked constantly.

Negative caching: "doesn't exist" is cached too

Resolvers also cache NXDOMAIN — the answer "this name doesn't exist". That's Negative cachingNegative cachingCaching the answer “this name doesn't exist” (NXDOMAIN), so repeated questions for a missing name don't hammer the servers. The cache time comes from the zone's SOA record., and it stops a typo storm from hammering the authoritative servers. How long it's cached comes from the zone's SOA record: the last number in the SOA (often called the "minimum"), or the SOA's own TTL if that's smaller.

The trap, which every team hits once:

$ dig +short new-api.example.com     # checking whether it works yet… (it doesn't exist yet)
$                                    # empty: NXDOMAIN — and the resolver caches that
# you create the record in the dashboard one minute later
$ dig +short new-api.example.com
$                                    # still empty, for up to the SOA's negative-cache time!
$ dig +short @ns1.example-dns.net new-api.example.com
192.0.2.30                           # the authoritative server already has it

The @server form asks that server directly, skipping your resolver's cache — the quickest way to check "is the record right, and I'm just looking at a stale cache?" (more in §09). Create records before anything looks them up, and keep the SOA negative-cache time modest (300–900 s is common).

On the wire: UDP, TCP and encrypted DNS

Why UDP?

There are two main ways to send data across the internet. TCPTCPTransmission Control Protocol: delivers a stream of bytes between two programs reliably and in order, resending anything lost. HTTP/1.1, HTTP/2 and most databases run on it. sets up a connection first — a back-and-forth HandshakeHandshakeA short exchange of messages at the start of a connection where both sides agree it's open and settle the details (versions, keys, starting numbers) before any real data flows. before any data moves — and then guarantees delivery. UDPUDPUser Datagram Protocol: sends individual packets with no connection, no ordering and no resending. Fast and simple; used by DNS, video calls, games and QUIC (HTTP/3). just fires off individual packetsPacketOne small chunk of data sent across a network on its own, with a header saying where it's from and where it's going. Big messages are split into many packets.: no setup, no guarantee. (TCP vs UDP covers the difference in depth.)

A DNS question and its answer each usually fit in a single packet of a hundred bytes or so. With TCP, you'd spend a full round trip on the handshake before even asking. With UDP, the question goes out immediately. For something in front of every request, that saved round trip matters.

$ sudo tcpdump -n -i eth0 port 53      # print DNS traffic on interface eth0 as it happens
10:42:01.120 IP 10.0.0.5.51234 > 1.1.1.1.53: 31012+ A? example.com. (29)
10:42:01.143 IP 1.1.1.1.53 > 10.0.0.5.51234: 31012 1/0/0 A 192.0.2.10 (45)
#                ↑ random source port   ↑ ID   ↑ 1 answer   ↑ the answer   ↑ size in bytes

Two lines, 23 milliseconds, done. 31012 is the query's ID number: the reply must carry the same one, so the client can match replies to questions. (It also matters for security — §10.)

The price of UDP: lost packets mean waiting

UDP doesn't resend lost packets. If the question or the answer is dropped, the client hears nothing — it can't tell "slow" from "lost". So it waits for a TimeoutTimeoutA limit on how long to wait for something (a connection, a response) before giving up with an error, so a slow or dead peer can't hang your program forever., then tries again.

On Linux that waiting is set in /etc/resolv.conf, and the defaults are harsh: 5 seconds per try, 2 tries per server.

# /etc/resolv.conf — the OS resolver's settings
nameserver 10.0.0.2            # ask this resolver first…
nameserver 10.0.0.3            # …then this one if the first doesn't answer
options timeout:2 attempts:3   # wait 2 s per try, up to 3 rounds (defaults: 5 s and 2)

When the answer is too big: truncation and TCP

Original DNS limited UDP answers to 512 bytes. A name with many records, long TXT values or security signatures (§10) can exceed that. Two mechanisms deal with it:

1. EDNSEDNS(0)An extension to DNS that lets a query say “I can accept UDP answers bigger than 512 bytes” (commonly 1232 bytes today) and carries extra flags, such as the one DNSSEC uses.: the query says "I can accept bigger UDP answers" — usually 1232 bytes today, a size chosen so the packet won't need to be split up on the way.
2. If the answer still doesn't fit, the server sends a cut-down reply with the TC flag set — that's TruncationTruncation (TC bit)A flag in a DNS reply over UDP that means “the full answer didn't fit; ask me again over TCP”. Firewalls that block DNS over TCP break these lookups.. The client then asks again over TCP, where size doesn't matter.

Privacy: plain DNS is a postcard

Plain DNS is not encrypted. Everyone on the path — the café Wi-Fi, your ISP, anyone on the network in between — can read every name you look up, and can tamper with the answers. Even when the page itself is HTTPSHTTPSHTTP sent through an encrypted TLS connection — the padlock in your browser. Same requests and responses as HTTP, but private and tamper-proof., the DNS lookup before it gave away which site you visited.

Two standards wrap DNS in encryption (TLSTLSTransport Layer Security: the encryption layer that makes HTTPS secure. It proves the server is who it claims to be and scrambles everything so nobody in between can read or change it., the same layer that puts the S in HTTPS): DoH / DoTDoH / DoTDNS over HTTPS (port 443) and DNS over TLS (port 853): DNS queries sent inside an encrypted connection so nobody on the network path can read or tamper with them..

DNS from Go and Node: two resolvers each

The two paths

The "ask the OS" path goes through getaddrinfogetaddrinfoThe standard C library function programs call to turn a hostname into IP addresses. It reads /etc/hosts and the OS's resolver settings, and it blocks the thread that calls it until it has an answer., a function in the system's libclibc (C library)The standard library that almost every program on Linux or macOS uses to talk to the operating system — file access, memory, and hostname lookups via getaddrinfo. On most Linux systems it's glibc; on Alpine it's musl. that every C program uses. It honours /etc/hosts, the OS's resolver settings and anything else the OS is configured to consult. But it blocks the ThreadThreadOne worker that runs your code line by line. A program can have several threads, and the operating system decides which one runs on which CPU core. that calls it until the answer arrives.

The "talk DNS directly" path builds DNS packets itself and sends them to the resolver listed in /etc/resolv.conf. It doesn't need to block a thread, but it only knows what DNS knows.

Go: a pure-Go resolver, and a fallback to the C library

Go ships its own DNS client written in Go. On Linux it's the default: it reads /etc/hosts and /etc/resolv.conf itself, sends the A and AAAA queries in parallel, and waits for answers the same cheap way Go waits for any network I/O — a parked GoroutineGoroutineA very lightweight task in Go, started with the go keyword. The Go runtime runs thousands of them on just a few OS threads., not a blocked thread (see Async / Await §09).

The alternative is the cgo resolver, which calls the system's getaddrinfo through cgocgoGo's mechanism for calling C code. Go's cgo DNS resolver uses it to call the system's getaddrinfo; the pure-Go resolver doesn't need C at all.. Go switches to it automatically when it sees settings it can't handle itself — for example, an /etc/nsswitch.conf that asks for name sources Go doesn't understand. On macOS and Windows, Go asks the OS by default. You can see and override the choice:

$ GODEBUG=netdns=1 ./server          # print which resolver Go picked
go package net: dynamic selection of DNS resolver
$ GODEBUG=netdns=go ./server         # force the pure-Go resolver
$ GODEBUG=netdns=cgo ./server        # force the C library (getaddrinfo)
$ CGO_ENABLED=0 go build             # no C at all → always pure Go

For everyday code, net.LookupHost (or just dialing a hostname) is fine. When you need control — a deadline, a specific DNS server, a forced resolver — build a net.Resolver:

r := &net.Resolver{
    PreferGo: true, // ← use Go's own DNS client, not getaddrinfo
    Dial: func(ctx context.Context, network, address string) (net.Conn, error) {
        d := net.Dialer{Timeout: 2 * time.Second}
        return d.DialContext(ctx, network, "1.1.1.1:53") // ← ignore resolv.conf's server, ask this one
    },
}

ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel() // ← always release the timer, even on success

ips, err := r.LookupHost(ctx, "api.example.com")
var dnsErr *net.DNSError
if errors.As(err, &dnsErr) && dnsErr.IsNotFound {
    // NXDOMAIN: the name really doesn't exist — retrying won't help
} else if err != nil {
    // timeout or server failure: maybe temporary, maybe retry
}
fmt.Println(ips) // [192.0.2.20]

The error text tells you a lot. From the pure-Go resolver: lookup nope.example.com on 8.8.8.8:53: no such host — the on 8.8.8.8:53 part names the resolver from /etc/resolv.conf (it still says that even with a custom Dial, so don't trust it blindly when you've overridden one).

Node: dns.lookup vs dns.resolve* — the same module, two different machines

import { lookup } from "node:dns/promises";

// → getaddrinfo, on a libuv thread-pool thread
// ✓ honours /etc/hosts and OS settings
// ✗ no TTL, no timeout option of its own
// ✗ only ~2 run at once by default (see below)
const { address } = await lookup("api.example.com");

Neither tab is "wrong". The first is the default: http.get, https.request, net.connect and fetch all resolve hostnames with dns.lookup. The second uses c-aresc-aresA C library that sends DNS queries over the network asynchronously, without blocking a thread. Node's dns.resolve*() functions use it., a library that does DNS on the network without blocking any thread.

Failures look different too: getaddrinfo ENOTFOUND nope.invalid comes from dns.lookup; queryA ENOTFOUND nope.example.com comes from resolve4. And getaddrinfo EAI_AGAIN means "temporary failure" — usually the resolver didn't answer at all, a classic sign of broken DNS inside a container.

The production trap: dns.lookup and the thread pool

Because getaddrinfo blocks, Node runs it on the libuv thread poollibuv thread poolA small pool of background threads (4 by default) inside every Node process. Node uses it for work the OS can't do asynchronously: file system calls, DNS lookups, some crypto and zlib compression. (4 threads by default — Threads & Processes §07). And libuv treats DNS as "slow work": to keep file reads from starving, it lets slow work use at most half the pool (half, rounded up). With the default 4 threads, that's 2 lookups at a time.

When DNS is healthy, lookups take a millisecond and nobody notices. When the resolver gets slow — say 2 s per answer — lookups line up behind each other:

So a slow resolver doesn't just add 2 s to each request — it turns into a queue, and every outgoing call to a new hostname waits its turn. Under load the queue never drains, and your service looks like it's hanging on "nothing".

Neither Go nor Node caches DNS

Go's net package and Node's dns module both send a fresh question every time a new connection needs a name. (Reused Keep-aliveKeep-alive (connection reuse)Leaving a connection open after a request finishes so the next request can reuse it, skipping the cost of a new handshake. connections skip the lookup entirely, because they're already connected.) On a server without an OS-level cache, every new connection is a round trip to the resolver.

That's often fine. When it isn't, there are three fixes, from least to most code: run a local caching resolver next to the app (systemd-resolved, dnsmasq, or NodeLocal DNSCache in KubernetesKubernetesA system that runs your containers across many machines: it decides where each one runs, restarts them when they crash, rolls out new versions, and gives each group a stable name other services can call.); in Node, raise UV_THREADPOOL_SIZE — an environment variable read when the pool first starts, so set it at launch (UV_THREADPOOL_SIZE=16 node server.js), not from inside your code; or give Node's HTTP client a custom lookup that caches:

import http from "node:http";
import { Resolver } from "node:dns/promises";

const resolver = new Resolver({ timeout: 2000, tries: 2 });
const cache = new Map(); // hostname → { ips, expires }

async function cachedResolve(hostname) {
  const hit = cache.get(hostname);
  if (hit && hit.expires > Date.now()) return hit.ips;           // ← fresh: no DNS query at all
  const recs = await resolver.resolve4(hostname, { ttl: true }); // ← network query, no thread pool
  const ttl = Math.min(...recs.map((r) => r.ttl), 60);           // honour the TTL, but cap it at 60 s
  const ips = recs.map((r) => r.address);
  cache.set(hostname, { ips, expires: Date.now() + ttl * 1000 });
  return ips;
}

// The callback shape Node's http/net modules expect from a custom `lookup`.
function lookup(hostname, options, callback) {
  cachedResolve(hostname).then(
    (ips) => options.all
      ? callback(null, ips.map((address) => ({ address, family: 4 })))
      : callback(null, ips[0], 4),
    callback, // ← on failure, pass the error straight through
  );
}

const agent = new http.Agent({ keepAlive: true, lookup }); // ← every request through this agent uses it
http.get("http://api.example.com/health", { agent }, (res) => res.resume());

“It's always DNS”: how it actually fails

Here are the failure modes behind most "it was DNS" postmortems, roughly in the order you'll meet them.

1. The failover that half the clients ignored

The primary database dies. Your automation repoints db.internal.example.com to the replica in seconds. Yet for the next hour, a third of your app servers keep trying the dead primary.

The record was right — the caches weren't. The TTL was 3600, so every cache that fetched the name in the last hour keeps the old answer for up to an hour (§05). Some app servers also held open connections to the old IP, and an open connection never re-resolves (§11). A DNS-based failover is only as fast as the longest TTL and the oldest connection.

2. The name that "didn't exist" for an hour

You checked a name before creating it, so the resolver cached NXDOMAIN — for as long as the zone's SOA allows (§05). The fix is to wait, or to flush that resolver's cache. Prevention: create records first, test second.

3. Search domains and ndots:5: five lookups for one name

Short names like wiki work at many companies because the resolver adds a suffix for you — those suffixes are the search domainsSearch domainsSuffixes your resolver may append to a short name. With search corp.example, looking up “wiki” also tries wiki.corp.example. Set by the search line in /etc/resolv.conf.. An option called ndotsndotsA resolver option: if a name has fewer than this many dots, try the search domains first, before the name as written. Kubernetes sets ndots:5, so even api.example.com (2 dots) tries the search list first. decides when to try them: if the name has fewer dots than ndots, the resolver tries every search domain before the name as written.

KubernetesKubernetesA system that runs your containers across many machines: it decides where each one runs, restarts them when they crash, rolls out new versions, and gives each group a stable name other services can call. sets this up for every PodPod (Kubernetes)The unit Kubernetes runs and restarts: one or a few containers that live and die together. Stopping a pod is how Kubernetes stops your process during a deploy., so that a short service name like payments finds payments.default.svc.cluster.local:

# /etc/resolv.conf inside a Kubernetes pod (namespace "default")
nameserver 10.96.0.10                                   # the cluster's DNS server
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5                                         # ← fewer than 5 dots? try search domains first

api.stripe.com has only 2 dots — fewer than 5 — so the resolver first tries api.stripe.com.default.svc.cluster.local, then two more made-up names, each getting NXDOMAIN, before it finally asks for the real one. And each attempt is two queries, A and AAAA. Try it:

search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The fixes, from simplest:

• Write the name with a trailing dot — api.stripe.com. — to mark it as a complete name, an FQDNFQDN (fully qualified domain name)A complete name all the way to the root, written with a trailing dot: api.example.com. — the dot tells the resolver “don't append any search domains”.. The resolver then skips the search list. It's ideal for things that only do DNS, like database or cache hosts. Careful with HTTP: the dot can end up in the request's Host headerHTTP headerA Name: value line sent before the body of an HTTP request or response, carrying metadata — Content-Type: application/json, Authorization: Bearer …, Cache-Control: max-age=60., and many servers and CDNs then answer 404 or 421, or present the wrong certificate. For external HTTP APIs, the next two fixes are usually safer.
• Lower ndots for pods that mostly call external names, in the pod spec:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"     # names with 2+ dots (api.stripe.com) are tried as written first

• Run a cache on every node (NodeLocal DNSCache), so the wasted lookups are answered locally in microseconds instead of crossing the network to the cluster DNS server.

4. Requests that are slow by exactly five seconds

When p99p99 (99th percentile)The value that 99% of measurements are below — i.e. how bad the slowest 1% is. Averages hide slow outliers; p99 shows them. LatencyLatencyHow long one request or job takes from the moment it arrives until it's finished — including any time spent waiting in a queue. shows a cluster of requests taking 5 s longer than normal, suspect a lost DNS packet and the resolver's 5-second retry timer (§06). A notorious example on Linux and Kubernetes: getaddrinfo sends the A and AAAA queries at the same instant from the same socket. On some kernels, the connection-tracking table used by NATNATNetwork Address Translation: a router rewrites private addresses (like 192.168.1.20) to its one public address on the way out, and back on the way in. Why your laptop isn't directly reachable from the internet. can race on those two packets and drop one — and the client waits 5 s for a reply that will never come.

# glibc option: send A and AAAA one after another, from separate sockets
options single-request-reopen

The broader fix is the same as above: a node-local DNS cache, which keeps the hop short and avoids that NAT path. And always set timeout/attempts deliberately rather than living with the 5 s default.

5. The cache that never expires

Some software looks a name up once and keeps the answer until it restarts:

• Older Java setups cached successful lookups forever. Modern JVMs default to 30 s, but security-manager configs and custom settings still bring back "forever". Check networkaddress.cache.ttl.
• nginx resolves hostnames in proxy_pass at startup and never again, unless you configure a resolver and use a variable for the upstream. When the upstream's IP changes, nginx keeps sending traffic to the old one.
• Your own code: a Map of hostname → IP with no expiry is the same bug.

6. Different answers for different people

Split-horizon DNSSplit-horizon DNSServing different answers for the same name depending on who asks — e.g. a private 10.x address to machines inside the company network and a public address to everyone else. means one name gives different answers depending on who asks: git.example.com is a private 10.x address inside the office and a public IP (or nothing) outside. It's useful — and it makes "works for me" bugs. The developer on the VPN gets one answer; the CI runner outside gets another; a browser using encrypted DNS (§06) skips the company resolver and gets the public one. When answers differ, always ask: which resolver did this machine use?

7. The resolver itself falls over

Every service in a cluster leans on a handful of DNS servers. Combine ndots:5 (8 queries per external name), no caching in Go/Node (a lookup per new connection), and a burst of new connections after a deploy, and the cluster DNS can be the first thing to saturate. Cloud resolvers also have per-machine packet limits (on AWS, roughly 1,024 DNS packets per second per network interface — check current docs). Past the limit, queries are dropped… and you're back to 5-second timeouts.

The debugging toolkit

Three questions, then one more

When a name "doesn't work", answer these in order. Each one uses a different tool, and each rules out one layer.

And the fourth: is the delegation right? dig +trace walks the tree from the root yourself, so you see whether .com actually points at the servers you think it does.

Here's the checklist as a picture. Each tool asks a different layer; the culprit is the layer where the answers start to differ:

dig, the tool you'll use most

dig api.example.com                 # A record, full output
dig +short api.example.com AAAA     # just the IPv6 addresses
dig +noall +answer example.com MX   # just the answer section
dig @8.8.8.8 api.example.com        # ask a specific resolver
dig @ns1.example-dns.net api.example.com   # ask the authoritative server (no caches)
dig -x 192.0.2.10                   # reverse lookup (PTR)
dig +tcp example.com                # force TCP (tests the firewall rule from §06)

Run dig @8.8.8.8 and dig @1.1.1.1 for the same name: if they disagree, one of them has an older cached copy. The TTL in each answer tells you how many seconds that copy has left.

dig +trace: do the resolver's job yourself

+trace ignores caches and walks the tree exactly like the walkthrough in §02 — root, then TLD, then authoritative — printing each referral. (Output trimmed.)

$ dig +trace api.example.com
.                    518400  IN  NS  a.root-servers.net.         ← ① the root servers
;; Received 239 bytes from 10.0.0.2#53(10.0.0.2) in 3 ms

com.                 172800  IN  NS  a.gtld-servers.net.         ← ② root's referral: ask .com
;; Received 1170 bytes from 198.41.0.4#53(a.root-servers.net) in 12 ms

example.com.         172800  IN  NS  ns1.example-dns.net.        ← ③ .com's referral: ask these
;; Received 268 bytes from 192.5.6.30#53(a.gtld-servers.net) in 21 ms

api.example.com.     300     IN  A   192.0.2.20                  ← ④ the authoritative answer
;; Received 60 bytes from 198.51.100.1#53(ns1.example-dns.net) in 34 ms

If step ③ lists servers you stopped using last year, your registrar still has the old name servers: moving your zone to a new provider isn't finished until the registrar's NS records point there.

The local files

$ cat /etc/hosts                     # checked BEFORE DNS by getaddrinfo
127.0.0.1   localhost
10.0.0.42   api.example.com          # ← a forgotten line like this overrides DNS for this machine

$ cat /etc/resolv.conf               # which resolver, which search domains, which timeouts
nameserver 127.0.0.53                # 127.0.0.53 = systemd-resolved, a local cache; ask it what it uses:
search corp.example.com
$ resolvectl status                  # Linux with systemd-resolved: the real upstream resolvers
$ scutil --dns                       # macOS equivalent

/etc/hosts is the oldest trick in the book — and the oldest trap. A line someone added for testing silently wins over DNS for every program that uses getaddrinfo, but not for dig, which only speaks DNS. "dig says one thing and my app does another" is very often /etc/hosts.

Flushing caches (when you must)

Flushing only fixes your machine. Your users' resolvers keep their copies until the TTL runs out — which is why the real fix is the migration playbook in §05, not a flush.

Security: poisoning, DNSSEC, rebinding and takeovers

Cache poisoning: forging the answer

A resolver sends a question over UDP and accepts the first reply that looks right: correct question, correct 16-bit Transaction IDTransaction IDA 16-bit number a resolver puts in each DNS query; the reply must carry the same number. Only 65,536 possible values — which is why it's not enough on its own to stop forged replies., arriving at the right PortPortA number from 0 to 65535 that picks which program on a machine should get the traffic — like an apartment number after the street address. HTTPS servers usually listen on port 443.. Nothing proves the reply came from the real server. If an attacker can get a forged reply in before the real one, the resolver caches the fake answer and serves it to all its users until the TTL runs out. That's Cache poisoningCache poisoning (DNS spoofing)Tricking a resolver into caching a forged answer, so everyone who uses that resolver is sent to the attacker's IP until the bad entry expires..

For years this looked hard: the attacker had to guess the ID (1 in 65,536) and could only try when the name wasn't already cached — once per TTL. In 2008, Dan Kaminsky showed how to get unlimited tries:

Because rnd1, rnd2, rnd3… are never cached, every one forces a new query and a new race. With enough races, a guessed ID eventually lands.

The industry-wide fix, shipped within weeks, was source port randomization: the resolver sends each query from a random UDP port, so the attacker must guess the port and the ID — roughly 65,536 × ~60,000 combinations instead of 65,536. Some resolvers add more randomness by MiXiNg the letter case of the name in the query and checking the reply copies it exactly. These make blind forgery impractical, not impossible.

DNSSEC: signatures, not secrecy

DNSSECDNSSECDNS Security Extensions: zones sign their records with a private key, and resolvers check the signatures, so forged answers can be detected. It proves an answer is authentic; it does not encrypt it. fixes the root cause: the zone owner signs each set of records with a private key, and the signature (an RRSIG record) travels with the answer. A validating resolver checks it with the zone's public key (a DNSKEY record), and checks that key against a fingerprint published by the parent zone (a DS record in .com), all the way up to the root's key, which resolvers ship with. A forged answer can't carry a valid signature, so it's rejected.

What DNSSEC does not do: it doesn't encrypt anything (that's DoH/DoT, §06), and it only helps if the resolver validates. Adoption is uneven — many large domains don't sign — and a mistake in signing (an expired signature, a key rollover gone wrong) makes the whole domain fail to resolve for validating resolvers, with a SERVFAIL status. Teams that enable it automate key rotation and monitor signature expiry.

DNS rebinding: turning your browser against localhost

Browsers protect you with the Same-origin policySame-origin policyThe browser rule that a page from one origin (scheme + host name + port) can send requests to other origins but can't read their responses unless they allow it (that's CORS). It compares host names, not IP addresses.: a script from evil.example can't read responses from other sites. But the check compares names, not IP addresses. DNS rebindingDNS rebindingAn attack where a malicious domain first resolves to the attacker's server, then — after a very short TTL — to an internal address like 127.0.0.1, so a script in your browser can reach services that were never meant to be public. exploits that gap:

1. You visit evil.example. Its DNS answer is the attacker's server, with a TTL of 1 second. The page loads a script.
2. A few seconds later, the script requests evil.example/api again. The cached answer has expired, so the browser looks the name up again — and this time the attacker's DNS answers 127.0.0.1.
3. To the browser, it's still the same origin, evil.example, so the script can read the response. But the request went to your own machine: a dev server, a database admin page, a router config page, a local AI tool — services that "only listen on localhost, so they're safe".

Defences, for anyone running a local or internal HTTP service:

• Check the Host header. A request that arrived via rebinding still says Host: evil.example. Reject anything that isn't the name you expect (localhost, your domain). Many dev servers now do this by default.
• Require authentication even on localhost and internal networks.
• Resolvers and routers with rebinding protection refuse public names that resolve to private addresses; browsers are also adding restrictions on public pages calling private addresses (check current browser docs).

Subdomain takeover: the CNAME you forgot

Your marketing team used promo.example.com → CNAME → example-promo.some-hosting.app. The campaign ended, the hosting account was deleted — but the CNAME stayed. Now anyone can create an account on that hosting service, claim example-promo, and serve their content at promo.example.com, under your domain, with a valid certificate and your cookies' domain.

DNS in production: discovery, balancing, failover

DNS as service discovery

In a system of microservicesMicroserviceA small, separately deployed program that owns one part of a bigger system and talks to the others over the network (HTTP, gRPC, a message queue)., instances start, crash and move all day. Service discoveryService discoveryHow one service finds the current addresses of another when instances come and go all the time. DNS is the simplest form: look up a name, get today's list of IPs. is how a caller finds today's addresses. DNS is the simplest version: every service gets a stable name, and the name's records track the live instances.

KubernetesKubernetesA system that runs your containers across many machines: it decides where each one runs, restarts them when they crash, rolls out new versions, and gives each group a stable name other services can call. gives you two flavours:

Kubernetes also publishes SRV records for named ports, so a client can learn the port as well as the address: _grpc._tcp.payments.default.svc.cluster.local. Outside Kubernetes, Consul offers the same idea: every registered service is reachable as name.service.consul (on port 8600 by default), and Consul only returns instances whose Health checkHealth checkA small automatic request (e.g. GET /healthz every 5 s) a load balancer or orchestrator sends to each server to decide whether it should still receive traffic. is passing.

DNS as a load balancer — and why it's a weak one

Give a name several A records and clients spread across them ("round-robin DNS"). It's free and needs no extra hop. But compared to a real Load balancerLoad balancerA component that spreads incoming requests or connections across several servers, so no single one is overloaded and a dead one can be skipped. (Load Balancing covers those):

• The client chooses, and you can't control how. Many clients just take the first address; others sort them by their own rules. The spread is rarely even.
• Caches multiply the skew. A resolver serving 50,000 users hands all of them the same cached answer in the same order for a whole TTL.
• It can't see load or health. Plain DNS keeps returning a dead server's IP until someone removes the record — and then every cache still has it until the TTL expires.
• It balances connections, not requests. A client that opens one Keep-aliveKeep-alive (connection reuse)Leaving a connection open after a request finishes so the next request can reuse it, skipping the cost of a new handshake. connection sends everything to one IP, possibly for hours.

So DNS usually balances between places — data centres, regions, load balancers — and a real load balancer balances within each place.

Sending users somewhere nearby: GeoDNS and anycast

Health-checked failover vs TTL reality

DNS providers such as Route 53 can run health checks and swap records automatically: "if the primary fails 3 checks, answer with the secondary". The DNS side of that is quick. The client side isn't:

Plan for "most traffic moves within a couple of minutes, the long tail takes much longer". For anything that needs faster failover, put a stable address in front — a load balancer or an anycast IP that never changes — and do the failover behind it.

Connections outlive DNS answers

This is the single most surprising production fact about DNS: a DNS lookup only happens when a new connection is opened. Connection poolsConnection poolA set of open database (or HTTP) connections that your program keeps and reuses instead of opening a new one per request. Pools usually have a size limit; when every connection is busy, the next caller waits for one to be returned., HTTP keep-alive, database pools and gRPC channels deliberately keep connections open for minutes or hours, so a DNS change doesn't reach them at all.

You can watch it happen. In Go, httptrace reports each lookup and whether a connection was reused:

package main

import (
    "fmt"
    "io"
    "log"
    "net/http"
    "net/http/httptest"
    "net/http/httptrace"
    "strings"
    "time"
)

func main() {
    // A local test server, so this runs anywhere. Use "localhost" so there's a name to resolve.
    srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprint(w, "ok")
    }))
    defer srv.Close()
    url := strings.Replace(srv.URL, "127.0.0.1", "localhost", 1)

    var dnsStart time.Time
    trace := &httptrace.ClientTrace{
        DNSStart: func(info httptrace.DNSStartInfo) { dnsStart = time.Now() },
        DNSDone: func(info httptrace.DNSDoneInfo) {
            log.Printf("dns took %v, addrs=%v, err=%v", time.Since(dnsStart), info.Addrs, info.Err)
        },
        GotConn: func(info httptrace.GotConnInfo) {
            log.Printf("conn to %s, reused=%v", info.Conn.RemoteAddr(), info.Reused) // ← reused=true: no DNS at all
        },
    }

    for range 2 { // send the same request twice
        req, err := http.NewRequest("GET", url, nil)
        if err != nil {
            log.Fatal(err)
        }
        req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            log.Fatal(err)
        }
        io.Copy(io.Discard, resp.Body) // ← read to the end AND close,
        resp.Body.Close()              //   or the connection can't be reused
    }
}

Output (timestamps trimmed; times will differ):

dns took 89µs, addrs=[{127.0.0.1 }], err=<nil>
conn to 127.0.0.1:35483, reused=false
conn to 127.0.0.1:35483, reused=true        ← second request: no DNS lookup at all

In Node, the socket emits a lookup event — and on a reused keep-alive socket, it doesn't fire. This prints only one line, for request 1:

import http from "node:http";

// A local test server, so this runs anywhere.
const server = http.createServer((req, res) => res.end("ok")).listen(0, async () => {
  const url = `http://localhost:${server.address().port}/`;
  const agent = new http.Agent({ keepAlive: true }); // ← keep connections open for reuse

  for (let i = 0; i < 2; i++) {
    await new Promise((done) => {
      const req = http.get(url, { agent }, (res) => {
        res.resume();          // read the body to the end…
        res.on("end", done);   // …so the socket goes back to the agent for reuse
      });
      req.on("socket", (socket) => {
        socket.on("lookup", (err, address, family, host) => {
          console.log(`request ${i + 1}: dns ${host} → ${address}`, err ?? ""); // only NEW connections
        });
      });
    });
  }
  agent.destroy();
  server.close();
});

Ways to make clients notice DNS changes:

• Give connections a maximum age. Database pools usually have a "max lifetime" setting; for HTTP servers, closing keep-alive connections after N minutes or N requests forces clients to reconnect (and re-resolve).
• gRPC: use a client that re-resolves and balances across all addresses (Go's gRPC dns:/// resolver with round-robin), and set a server-side maximum connection age so clients reconnect periodically. (gRPC has the details.)
• In Go, http.Transport has no max-lifetime setting; calling transport.CloseIdleConnections() on a timer is a common workaround.

What to monitor

Treat DNS like any other dependency you call over the network: give it a TimeoutTimeoutA limit on how long to wait for something (a connection, a response) before giving up with an error, so a slow or dead peer can't hang your program forever., measure it, alert on it.

Pitfalls gallery

// "DNS was flaky, so…"
const API = "http://203.0.113.7/v1";

const API = "https://api.payments.example/v1";
// …and run a local caching resolver if lookups are slow

addrs, err := net.LookupHost(host) // waits as long as the OS lets it

ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
defer cancel()
addrs, err := net.DefaultResolver.LookupHost(ctx, host)

const ipCache = new Map();
async function ipFor(host) {
  if (!ipCache.has(host)) ipCache.set(host, (await lookup(host)).address);
  return ipCache.get(host); // ← valid forever
}

// store { ips, expires } and check expires > Date.now()
// (the cached lookup in §07)

var dnsErr *net.DNSError
if errors.As(err, &dnsErr) {
    switch {
    case dnsErr.IsNotFound: // NXDOMAIN (or no records): reject the input
    case dnsErr.IsTimeout:  // resolver didn't answer: retry later, don't reject
    }
}

www  IN  CNAME  cdn.example.net     ; → cdn.example.net.example.com.

www  IN  CNAME  cdn.example.net.    ; the dot means "complete name"

Check yourself

Commit to an answer before you click. Each explanation points back to the section that covers it.

import dns from "node:dns";
const t0 = Date.now();
for (let i = 0; i < 8; i++) {
  dns.lookup(`host${i}.example.com`, () => {
    console.log(i, Date.now() - t0); // when does the LAST one print?
  });
}

Cheat sheet