DNS migration incident: half our traffic hit the old server for a day

This is the postmortem for a migration we had planned as a one-hour job. On Saturday at 09:00 we moved our invoicing app from a VPS at an old provider (198.51.100.10) to a new cloud setup (203.0.113.50), and at the same time moved DNS hosting for example.com from the old provider's nameservers to a managed DNS service. The new stack passed smoke tests, we updated the records, and we went home.

On Sunday morning, support had 37 tickets: invoices created on Saturday were "missing". The new database had them. The old server, which we had left running read-write "just in case", had received a different set. The new load balancer's request count was only about 55 percent of a normal Sunday, and the old server's Nginx access log was very much alive:

203.0.113.77 - - [15/Jun/2025:08:12:44 +0000] "POST /api/invoices HTTP/1.1" 201 512 "-" "Mozilla/5.0 ..."
198.51.100.200 - - [15/Jun/2025:08:12:45 +0000] "GET /dashboard HTTP/1.1" 200 18233 "-" "Mozilla/5.0 ..."

That was nearly 24 hours after the switch.

Reproducing it

Different resolvers gave different answers to the same question:

for r in 1.1.1.1 8.8.8.8 9.9.9.9 208.67.222.222; do
  printf '%-16s ' "$r"; dig +short app.example.com A @"$r"
done

1.1.1.1          203.0.113.50
8.8.8.8          198.51.100.10
9.9.9.9          198.51.100.10
208.67.222.222   203.0.113.50

Asking with full output showed the remaining TTL on the stale answer:

;; ANSWER SECTION:
app.example.com.   71240   IN  A   198.51.100.10

Seventy-one thousand seconds left, almost 20 hours, on a record we had changed a day earlier. That meant some resolvers had fetched the old value after our change.

What we ruled out

• Our new zone being wrong. Asking the new nameservers directly returned the new IP.
• Browser or OS caches. Those explained a handful of users, not half the traffic, and a stale answer from 8.8.8.8 is not a browser problem.
• A CDN or proxy caching the origin. The app was not behind a CDN.

Root cause

Three DNS caching layers combined:

1. Record TTL. The old zone had app.example.com with TTL 86400 (24 hours), the old provider's default. We never lowered it before the move, so any resolver that cached it shortly before 09:00 kept it for a day.
2. Delegation TTL. We changed the domain's nameservers at the registrar at the same time. The NS records for example.com in the .com zone carry a TTL of 172800 seconds (48 hours). Resolvers that had cached the old delegation kept asking the old provider's nameservers.
3. The old zone was still live. We had not deleted or updated the zone at the old provider, so those resolvers kept getting the old A record, freshly, with a brand-new 24-hour TTL.

The third point is why stale answers kept being renewed instead of expiring.

What is happening under the hood

A recursive resolver caches each answer for the record's TTL. It also caches the delegation: the NS records that tell it which servers are authoritative for a zone. The parent zone (.com) and the child zone (example.com) both publish NS records, and resolvers may use either set until it expires.

You can see the parent's answer and its TTL by asking a .com server directly:

dig +norecurse example.com NS @a.gtld-servers.net

;; AUTHORITY SECTION:
example.com.   172800  IN  NS  ns1.newdns.example.net.
example.com.   172800  IN  NS  ns2.newdns.example.net.

There is also negative caching, defined in RFC 2308. When a name does not exist, the resolver caches the NXDOMAIN for the smaller of the SOA record's TTL and the SOA MINIMUM field. We hit this too: a teammate tested api-v2.example.com before we created it, and the corporate resolver kept answering NXDOMAIN for an hour after the record existed, because the new zone's SOA minimum was 3600.

Environment

Minimal reproduction

The failure needs no code, just a timeline:

T-0 h   Zone at old provider: app A 198.51.100.10 TTL 86400
T+0 h   Registrar NS changed to new provider; new zone: app A 203.0.113.50 TTL 300
        Old zone left untouched at old provider
T+0..48 h  Resolvers with cached old NS ask old provider -> old IP, new 24 h TTL
T+0..24 h  Resolvers with cached old A record keep it until expiry

Who is affected

Anyone changing an IP, a hosting provider or a DNS provider. Long TTLs are common defaults (3600 to 86400) at many registrars and hosts. Changing nameservers and records in one step multiplies the risk, and leaving the old zone active turns a 24-hour tail into a 48-hour one.

The fix

During the incident, in this order:

1. Make the old server harmless. We turned off the app on the old VPS and put Nginx in front as a reverse proxy to the new stack, so stale clients reached the right database:

server {
    listen 443 ssl;
    server_name app.example.com;
    ssl_certificate     /etc/letsencrypt/live/app.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;

    location / {
        proxy_pass https://203.0.113.50;
        proxy_ssl_server_name on;
        proxy_ssl_name app.example.com;
        proxy_set_header Host app.example.com;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

1. Fix the old zone. We updated the A record at the old provider to 203.0.113.50 with TTL 300, so resolvers still following the old delegation got the right answer.
2. Flush large public resolvers. Google Public DNS and Cloudflare's 1.1.1.1 both offer a public cache flush page for a name; we used them for app.example.com.
3. Reconcile data. We exported the 214 invoices written to the old database on Saturday and replayed them into the new one with their original numbers, after checking for collisions.

Alternatives and trade-offs

• Wait it out. Correct answers arrive eventually, but split-brain writes kept piling up, so waiting was not an option for a write-heavy app.
• Shut down the old server immediately. Stops split-brain, but users on stale resolvers would get connection errors for up to two days. The proxy gave them a working app instead.
• Keep the old provider as a secondary. Some DNS services support zone transfers to secondaries, which keeps both nameserver sets serving identical data during a move. Our old provider did not.

How to test the fix

We checked every layer, not just our laptop:

# What the parent delegates to
dig +norecurse example.com NS @a.gtld-servers.net

# What each set of authoritative servers answers
dig +short app.example.com A @ns1.oldhost.example.net
dig +short app.example.com A @ns1.newdns.example.net

# Full resolution path from the root, bypassing caches
dig +trace app.example.com A

All four answers returned 203.0.113.50 after the old zone was fixed.

Confirming in production

• Old server: requests passing through the proxy fell from about 45 percent of total on Sunday morning to 6 percent by Monday morning and to zero by Tuesday 10:00, matching the 48-hour delegation TTL.
• New load balancer request count returned to normal weekday levels on Monday.
• Zero new invoices in the old database after the app on it was stopped at 10:20 Sunday.
• A resolver sweep script across 20 public resolvers returned the new IP from all of them on Tuesday.

Regression guard

We wrote a migration runbook that is now required for any DNS or hosting change:

T-72h  Lower TTL of records to be changed to 300. Wait at least the OLD TTL.
T-72h  If changing nameservers: make new zone identical, confirm with dig @new-ns.
T-0    Change records (not nameservers) first. Keep old server serving or proxying.
T+1h   Verify with resolver sweep. Watch old server access log.
T+48h  Change nameservers, keep the old zone identical until parent NS TTL has passed twice.
T+7d   Raise TTLs back, decommission old server and old zone.

The resolver sweep runs as a scheduled job during any migration window and posts mismatches to the on-call channel.

Security implications

Leaving an old server online and reachable after a move is a risk in its own right: it keeps old software, old certificates and an old copy of the data. A stale zone at a provider you no longer use can also enable subdomain takeover if records point at resources you have released, such as a deleted cloud bucket or an unclaimed hosting account. We deleted the old zone after the TTLs expired, revoked the old server's API keys, and wiped its disk instead of just powering it off.

Prevention checklist

• Lower TTLs to 300 seconds at least one old-TTL period before a cutover.
• Never change nameservers and record values in the same step.
• Keep the old zone serving correct data until the parent delegation TTL has expired.
• Make the old server read-only or a proxy, never a second writable primary.
• Verify with dig +trace and several public resolvers, not only your own machine.

References

• RFC 1034, Domain names concepts and facilities — https://www.rfc-editor.org/rfc/rfc1034
• RFC 2308, Negative caching of DNS queries — https://www.rfc-editor.org/rfc/rfc2308
• BIND 9 documentation: dig — https://bind9.readthedocs.io/en/latest/manpages.html#dig-dns-lookup-utility
• Google Public DNS: flush cache — https://developers.google.com/speed/public-dns/cache
• Cloudflare 1.1.1.1: purge cache — https://one.one.one.one/purge-cache/