SSL works in Chrome but fails on Android and curl: incomplete certificate chain

On a Thursday morning our mobile team reported that the Android app could not log in for any user whose session had expired. Crash-free rate was fine, but the login screen showed a generic network error. Within the hour, a partner that pulls order data from our API every 15 minutes emailed that their sync was failing. Meanwhile, https://api.example.com/health loaded perfectly in desktop Chrome for everyone on the team.

The partner sent their Java stack trace:

javax.net.ssl.SSLHandshakeException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

The Android logs from OkHttp said:

javax.net.ssl.SSLHandshakeException: java.security.cert.CertPathValidatorException: Trust anchor for certification path not found.

And plain curl from a CI runner:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

Reproducing it

Every non-browser client failed, every time. The quickest check was openssl s_client, which shows exactly what the server sends:

openssl s_client -connect api.example.com:443 -servername api.example.com -showcerts </dev/null

depth=0 CN = api.example.com
verify error:num=20:unable to get local issuer certificate
verify error:num=21:unable to verify the first certificate
---
Certificate chain
 0 s:CN = api.example.com
   i:C = US, O = Let's Encrypt, CN = R11
---
Verify return code: 21 (unable to verify the first certificate)

Only one certificate in the chain, issued by Let's Encrypt's R11 intermediate, and the server did not send R11.

What we ruled out

• Outdated CA bundles on clients. The CI runner had a current ca-certificates package, and the Android devices were Android 12 to 15. The ISRG Root X1 root is trusted on all of them.
• Clock skew. Expired or not-yet-valid certificates give different errors (certificate has expired).
• SNI problems. We passed -servername, and the leaf had the right subject alternative name.

Root cause

The previous week we had moved the API from a managed load balancer, which handled TLS for us, to Nginx on our own instances with certbot. The Nginx config written during that migration used the wrong file:

ssl_certificate     /etc/letsencrypt/live/api.example.com/cert.pem;
ssl_certificate_key /etc/letsencrypt/live/api.example.com/privkey.pem;

cert.pem contains only the leaf certificate. fullchain.pem contains the leaf followed by the intermediate. It took a week to surface because the first certificate was installed manually from fullchain.pem; the config pointing at cert.pem was only picked up after certbot's renewal hook reloaded Nginx on Wednesday night.

What is happening under the hood

A TLS client validates a certificate by building a path from the server's leaf certificate up to a root it already trusts. Roots live in the client's trust store. Intermediates do not; the server is expected to send them during the handshake. RFC 8446 (TLS 1.3) and earlier TLS versions both say the server sends its certificate followed by the certificates needed to chain to a trust anchor.

When the intermediate is missing, the client must find it somewhere else, and clients differ:

So the desktop browser repaired our mistake silently and hid it from us.

Environment

Minimal reproduction

# Serve only the leaf certificate
ssl_certificate /etc/letsencrypt/live/api.example.com/cert.pem;

# Then from any machine
curl -sS https://api.example.com/health
# curl: (60) SSL certificate problem: unable to get local issuer certificate

Who is affected

Any server that sends a leaf without its intermediate. This shows up most with hand-installed certificates from commercial CAs (where the intermediate arrives as a separate file in the zip), configs that reference cert.pem instead of fullchain.pem, and Java keystores imported without the chain. Browser users are usually unaffected, which is why it reaches production.

The fix

Point Nginx at the full chain and reload:

-ssl_certificate     /etc/letsencrypt/live/api.example.com/cert.pem;
+ssl_certificate     /etc/letsencrypt/live/api.example.com/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/api.example.com/privkey.pem;

sudo nginx -t && sudo systemctl reload nginx

For certificates from a commercial CA, build the file yourself in the right order: leaf first, then each intermediate, and do not include the root:

cat api_example_com.crt intermediate_ca.crt > /etc/nginx/tls/api.example.com.chain.pem

Including the root is harmless for most clients but wastes bytes on every handshake, and the order matters: a chain file starting with the intermediate makes Nginx fail with a key mismatch error, because it pairs the first certificate with the private key.

Alternatives and trade-offs

• Pinning the intermediate in the Android app or partner trust store. It hides the server problem and breaks again when the CA rotates intermediates, which Let's Encrypt does.
• Let the load balancer terminate TLS again. Managed load balancers usually send the full chain automatically, but you trade that convenience for less control over TLS settings.
• Apache. Since Apache 2.4.8, SSLCertificateFile can contain the chain directly, which makes the separate SSLCertificateChainFile directive unnecessary.

How to test the fix

# Should list two certificates (depth 0 leaf, depth 1 R11) and return code 0
openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null \
  | grep -E '^ [0-9] s:|Verify return code'

# Verify the chain file itself offline, before deploying it
# (certbot keeps the intermediates alone in chain.pem)
openssl verify -untrusted /etc/letsencrypt/live/api.example.com/chain.pem \
  /etc/letsencrypt/live/api.example.com/cert.pem

curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://api.example.com/health

The curl line should print 200 0; a non-zero ssl_verify_result means verification failed.

Confirming in production

After the reload at 10:42:

• openssl s_client from three regions showed a two-certificate chain and Verify return code: 0 (ok).
• The Android login error rate in our analytics dropped from 31 percent of attempts to the usual baseline of about 0.4 percent within ten minutes.
• The partner confirmed the 10:45 sync succeeded and back-filled the missed window.
• An external TLS scanner rated the chain as complete with no "extra download" warning.

Regression guard

• A synthetic check in our monitoring runs curl (not a browser) against every public hostname every five minutes, so AIA fetching can never hide a broken chain again.
• A CI test lints Nginx configs and fails on any ssl_certificate path ending in /cert.pem.
• The certbot deploy hook now runs nginx -t and the openssl chain check before reloading, and aborts the reload if either fails.

#!/bin/sh
# /etc/letsencrypt/renewal-hooks/deploy/check-and-reload.sh
set -e
nginx -t
for host in api.example.com www.example.com; do
  openssl verify -untrusted "/etc/letsencrypt/live/$host/chain.pem" "/etc/letsencrypt/live/$host/cert.pem"
done
systemctl reload nginx

Security implications

The bug itself did not weaken TLS; clients that failed did exactly the right thing by refusing an unverifiable chain. The danger is in the workarounds people reach for under pressure: curl -k, verify=False in Python, a trust-all TrustManager on Android, or disabling certificate checks in the partner's Java client. Any of those turns a configuration bug into a man-in-the-middle vulnerability that outlives the fix. We explicitly asked the partner not to ship such a change and checked our own codebase for rejectUnauthorized: false and similar flags.

Prevention checklist

• Always serve fullchain.pem (leaf plus intermediates), never cert.pem alone.
• Test TLS with curl or openssl, not just a browser.
• Monitor public endpoints from a non-browser client.
• Validate the chain in the renewal hook before reloading.
• Never disable certificate verification to work around a chain problem.

References

• RFC 8446, TLS 1.3, Certificate message — https://www.rfc-editor.org/rfc/rfc8446#section-4.4.2
• Nginx: configuring HTTPS servers, SSL certificate chains — https://nginx.org/en/docs/http/configuring_https_servers.html
• Certbot documentation: where are my certificates — https://eff-certbot.readthedocs.io/en/stable/using.html#where-are-my-certificates
• Let's Encrypt: chains of trust — https://letsencrypt.org/certificates/
• curl: SSL certificate verification — https://curl.se/docs/sslcerts.html
• Android developers: security with network protocols — https://developer.android.com/privacy-and-security/security-ssl