Expired TLS certificate outage: certbot renewal had been failing silently for weeks

At 07:12 local time on a Tuesday our uptime checker paged the on-call engineer: the customer portal at portal.example.com was failing its HTTPS probe. Within ten minutes support had a queue of screenshots showing a full-page browser warning instead of the login form. Chrome showed:

Your connection is not private
Attackers might be trying to steal your information from portal.example.com
NET::ERR_CERT_DATE_INVALID

Our mobile app, which pins nothing but does validate the chain, failed every API call with a generic network error. The server itself was healthy: nginx was up, the application responded on the internal health port, and CPU and memory were flat.

Reproducing it

The quickest confirmation was to ask the server what certificate it was presenting and when it expired:

echo | openssl s_client -connect portal.example.com:443 -servername portal.example.com 2>/dev/null \
  | openssl x509 -noout -subject -enddate

subject=CN = portal.example.com
notAfter=Jan 14 07:11:52 2025 GMT

The certificate had expired one minute before the first page. Every client that checked the date, which is every client, would fail from that moment on.

What we ruled out

• A clock problem on the server or clients. timedatectl showed NTP synchronised, and the notAfter timestamp matched the moment the probes started failing. Nothing was wrong with time; the certificate really was old.
• nginx serving a stale file after a successful renewal. This is a classic: certbot renews, nobody reloads nginx, and the old certificate stays in memory until it expires. We checked the files on disk and the dates matched the served certificate, so the renewal itself had never succeeded.
• A Let's Encrypt outage. Their status page was green, and a manual dry-run against staging from a different host in the same VPC worked.

Root cause

certbot certificates showed the same expiry date as the live site. The renewal log told the real story:

sudo grep -E "Renewing|Failed|Challenge failed|Timeout" /var/log/letsencrypt/letsencrypt.log* | tail -n 20

Failed to renew certificate portal.example.com with error: Some challenges have failed.
...
Detail: 203.0.113.40: Timeout during connect (likely firewall problem)

The systemd timer had been running twice a day since December 15, when the certificate entered its 30-day renewal window. Every attempt failed the HTTP-01 challenge because Let's Encrypt could not reach port 80. Five weeks earlier a security hardening change had replaced our cloud security group with one that only allowed 443 from the internet. Port 80 had looked pointless, since nginx only used it to redirect to HTTPS. Nobody knew that certbot's webroot challenge depended on it. The failures were written to a log nobody read, the timer unit exited non-zero without alerting anyone, and the expiry emails went to a mailbox belonging to an engineer who had left in the autumn.

What is happening under the hood

Let's Encrypt issues certificates valid for 90 days. Certbot's packaged timer runs certbot renew twice a day, and by default a certificate becomes eligible for renewal 30 days before it expires. That gives roughly 60 renewal attempts before the deadline, which sounds generous, but it only helps if somebody notices the attempts failing.

With the HTTP-01 challenge, the ACME server asks for a token at http://<domain>/.well-known/acme-challenge/<token> over plain HTTP on port 80. With --webroot, certbot writes that token into a directory nginx serves. Three things must all be true for renewal to succeed:

The last row matters even when renewal works. nginx reads certificate files at startup and on reload. Certbot only reloads it if you used the nginx installer plugin or configured a deploy hook.

Environment

Minimal reproduction

The failure is easy to reproduce on a staging host. Block port 80 and run a dry-run renewal, which talks to the Let's Encrypt staging environment and does not touch your real certificate:

sudo ufw deny 80/tcp
sudo certbot renew --dry-run --cert-name portal.example.com

Certbot reports the challenge failure and exits non-zero, but when the same command runs from a timer nothing surfaces that exit code anywhere a human looks.

Who is affected

Anyone using HTTP-01 challenges (certbot webroot or standalone, many other ACME clients) who changes firewalls, load balancers or CDNs without testing renewal, and anyone whose only expiry warning is an email. Let's Encrypt announced on January 22, 2025 that it will stop sending expiration notification emails on June 4, 2025 and recommends a monitoring service instead, so relying on those emails stops being an option entirely this year. DNS-01 setups have their own version of this bug: an expired or rotated DNS API token fails the same way, just as quietly.

The fix

First we restored service, then we fixed the system that let it happen.

1. Restore the challenge path. We added an ingress rule for TCP 80 and made sure the port 80 server block answers ACME requests before redirecting everything else:

server {
    listen 80;
    listen [::]:80;
    server_name portal.example.com;

    location ^~ /.well-known/acme-challenge/ {
        root /var/www/acme;
        default_type text/plain;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

2. Renew and reload. We forced the renewal and checked the result before touching nginx:

sudo certbot renew --cert-name portal.example.com --force-renewal
sudo nginx -t && sudo systemctl reload nginx

The site was back at 07:41, 29 minutes after the first page.

3. Make reloads automatic. A deploy hook runs only after a successful renewal, which is exactly when nginx needs to pick up new files. We dropped an executable script into certbot's hook directory so it applies to every certificate on the host:

sudo tee /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh > /dev/null <<'EOF'
#!/bin/sh
nginx -t && systemctl reload nginx
EOF
sudo chmod 755 /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh

4. Make failures loud. We added a systemd drop-in so a failed renewal triggers a unit that posts to our alerting webhook:

# /etc/systemd/system/snap.certbot.renew.service.d/alert.conf
[Unit]
OnFailure=notify-oncall@%n.service

notify-oncall@.service is a small one-shot unit that calls our incident tool with the failing unit name. We wrote that before touching anything else, because the real bug was not the firewall. It was a month of failures that nobody saw.

Alternatives and trade-offs

• DNS-01 challenge. No inbound port needed and it supports wildcards, but it requires a DNS provider API token on the server. That token can usually edit every record in the zone, so scope it narrowly if your provider supports that.
• TLS-ALPN-01. Works over port 443 only, but certbot does not support it out of the box; clients such as Caddy and some Kubernetes controllers do.
• Managed certificates from your load balancer or CDN. Renewal becomes the provider's job, at the cost of terminating TLS on their edge.

We stayed with HTTP-01 for this host because it is the simplest moving part, but we wrote the port 80 dependency into the security group's description so the next person sees why it exists.

How to test the fix

The dry run exercises the full challenge path against the staging environment. Deploy hooks are skipped during a dry run unless you ask for them, so we pass --run-deploy-hooks to exercise the reload as well:

sudo certbot renew --dry-run --run-deploy-hooks
sudo systemctl list-timers 'snap.certbot*'
sudo systemctl status snap.certbot.renew.service --no-pager

To test the alert, we temporarily blocked port 80 on staging, started the renew service manually and confirmed a page arrived within a minute.

Confirming in production

• openssl s_client now reports a notAfter 90 days out, and the certificate serial changed.
• Our external monitor checks the served certificate (not the file on disk) from three regions every five minutes and warns at 21 days and pages at 10 days remaining. Its dashboard showed 89 days the morning after.
• Over the following two weeks the renewal service logged zero failures, and a deliberate firewall test on staging produced an alert within one timer run.

Regression guard

• A CI job runs a small script against every public hostname we own, parses notAfter and fails the pipeline if any certificate has fewer than 20 days left. It is cheap and catches hosts we forgot existed.
• The infrastructure-as-code module for edge security groups now has a test asserting TCP 80 stays open on hosts tagged acme-http01.
• Certificate expiry is a standard probe in our uptime tool, separate from the HTTP check, so "certificate is about to expire" and "site is down" are two different alerts.

Security implications

An expired certificate does not leak data by itself, but it trains users to click through warnings, and a few of our customers did exactly that via the "Advanced" link. That habit is what makes real interception attacks work, so we treated the outage as a security incident and sent a short note asking users not to bypass certificate warnings. Reopening port 80 adds no meaningful surface: it only serves static challenge files and a redirect. We also checked that HSTS was in place, which it was, so browsers refused to downgrade to plain HTTP while the certificate was invalid.

Prevention checklist

• Monitor the certificate the server actually serves, from outside, with thresholds well above zero days.
• Alert on failed renewal jobs, not just on expiry; a failing timer is the earliest signal you get.
• Use a deploy hook (or the nginx plugin) so a renewed certificate is always loaded.
• Run certbot renew --dry-run after any firewall, CDN, DNS or web server change.
• Never route expiry warnings to a personal mailbox.

References

• Certbot user guide: renewing certificates and hooks — https://eff-certbot.readthedocs.io/en/stable/using.html
• Let's Encrypt: Ending support for expiration notification emails — https://letsencrypt.org/2025/01/22/ending-expiration-emails/
• Let's Encrypt: Monitoring service options — https://letsencrypt.org/docs/monitoring-options/
• Let's Encrypt: Challenge types — https://letsencrypt.org/docs/challenge-types/
• RFC 8555, Automatic Certificate Management Environment (ACME) — https://www.rfc-editor.org/rfc/rfc8555