cert-manager certificate stuck: HTTP-01 challenge wrong status code 404, expected 200

We added status.example.com to our cluster with a standard Ingress and a cert-manager annotation. An hour later the site still served the controller's default fake certificate. The Certificate was not ready:

$ kubectl -n status get certificate
NAME         READY   SECRET       AGE
status-tls   False   status-tls   68m

Following the chain down to the Challenge:

$ kubectl -n status describe challenge status-tls-1-2817364910-3051847721
...
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: wrong status code '404', expected '200'
  State:       pending

Reproducing it

cert-manager's self check fetches the challenge URL before asking Let's Encrypt to validate. We did the same from outside the cluster:

$ curl -si http://status.example.com/.well-known/acme-challenge/Wt7k1...  | head -3
HTTP/1.1 404 Not Found
Server: nginx
Content-Type: text/html

And looked at the temporary Ingress cert-manager had created for the solver:

$ kubectl -n status get ingress
NAME                        CLASS    HOSTS               ADDRESS        PORTS
cm-acme-http-solver-x7k2p   <none>   status.example.com                 80
status                      nginx    status.example.com  203.0.113.20   80, 443

The solver Ingress had no class and no address. Our ingress controller had not picked it up.

What we ruled out

• DNS. dig status.example.com returned the load balancer address.
• Let's Encrypt rate limits. The challenge never reached the validation stage; the self check failed first.
• Firewall on port 80. Requests reached nginx; nginx returned 404 because it had no route for the solver path.

Root cause

Our ClusterIssuer configured the HTTP-01 solver without an ingress class. The cluster runs ingress-nginx with --watch-ingress-without-class=false (the default) and an IngressClass named nginx that is not marked as default. So the solver's Ingress had no class, ingress-nginx ignored it, and the challenge path was never routed to the solver pod.

What is happening under the hood

HTTP-01 validation works like this:

1. cert-manager creates an Order and a Challenge for each domain.
2. For HTTP-01, it starts a small solver pod and a Service, and creates (or edits) an Ingress that routes /.well-known/acme-challenge/<token> to the solver.
3. It self checks the URL over HTTP from inside the cluster, expecting the key authorisation with status 200.
4. When the self check passes, it tells the ACME server to validate, and Let's Encrypt fetches the same URL from the internet.

Anything that stops step 3 or 4 leaves the challenge pending: an Ingress the controller ignores, an HTTP-to-HTTPS redirect that breaks the fetch, a WAF or auth middleware in front of /.well-known, or DNS pointing elsewhere.

Since Kubernetes moved to ingressClassName, ingress controllers usually only handle Ingresses with their class, unless the class is marked as the cluster default.

Environment

Minimal reproduction

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: {name: letsencrypt-prod}
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef: {name: letsencrypt-prod-account}
    solvers:
      - http01:
          ingress: {}            # no class: solver Ingress ignored by nginx

The fix

1. Set the ingress class on the solver.

solvers:
  - http01:
      ingress:
        ingressClassName: nginx

cert-manager uses it when creating the solver Ingress. Older configurations used the class field; on current versions, ingressClassName is the field that matches the Kubernetes API. After editing the issuer we deleted the stuck Challenge so cert-manager created a fresh one:

kubectl -n status delete challenge status-tls-1-2817364910-3051847721
kubectl -n status get certificate -w
# status-tls   True   status-tls   2m

2. Check the redirect path. Our main Ingress had nginx.ingress.kubernetes.io/force-ssl-redirect: "true". ingress-nginx excludes the ACME challenge path from SSL redirects by default, but we had also enabled a global HTTPS redirect at the cloud load balancer for another app. Let's Encrypt follows redirects to HTTPS on port 443, so that worked for us, but it is a common failure when the HTTPS side has no valid route yet. We removed the load balancer redirect and left redirects to the ingress controller.

3. Make the class explicit everywhere. Every Ingress in our charts now sets ingressClassName, and we decided not to mark any class as default, so mistakes show up as "no class" in review rather than as traffic going to the wrong controller.

Alternatives and trade-offs

• Mark nginx as the default IngressClass. Ingresses without a class would then be handled, including the solver's. Convenient with one controller; ambiguous with several.
• DNS-01 challenges. No dependency on inbound HTTP, and required for wildcard certificates. Needs DNS provider credentials in the cluster, which is a bigger blast radius.
• Gateway API HTTP-01 solver. If you are moving to Gateway API, cert-manager can create HTTPRoutes for the challenge instead of Ingresses.

How to test the fix

A staging issuer pointed at Let's Encrypt's staging endpoint, plus a throwaway hostname:

kubectl apply -f test-certificate-staging.yaml
kubectl get challenges -A -w
cmctl status certificate test-tls -n sandbox

We also ran the self check by hand during issuance from a pod in the cluster with curl, which matches what cert-manager does.

Confirming in production

• status-tls became ready two minutes after the fix and the site served a valid Let's Encrypt certificate.
• Renewal of other certificates issued through the same issuer continued to work; we forced one renewal with cmctl renew to confirm.

Regression guard

• An alert on certmanager_certificate_ready_status{condition="False"} for more than 30 minutes.
• An alert on certificates expiring within 14 days, independent of cert-manager's own status.
• A chart lint check that every Ingress has ingressClassName.

Security implications

The fake default certificate served during the incident caused browser warnings, which trains users to click through warnings, a bad habit. Monitoring readiness fixes that. When choosing DNS-01 as an alternative, keep DNS credentials scoped to the specific zones and records needed, since DNS control can be abused to issue certificates for your domains.

Prevention checklist

• Set ingressClassName on HTTP-01 solvers and on every Ingress.
• Follow Certificate to Order to Challenge when issuance stalls.
• Curl the challenge URL from outside to see what Let's Encrypt sees.
• Keep /.well-known/acme-challenge free of auth, WAF blocks and broken redirects.
• Alert on certificate readiness and expiry.

References

• cert-manager HTTP-01 solver configuration — https://cert-manager.io/docs/configuration/acme/http01/
• cert-manager troubleshooting ACME — https://cert-manager.io/docs/troubleshooting/acme/
• Let's Encrypt challenge types — https://letsencrypt.org/docs/challenge-types/
• Kubernetes IngressClass — https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-class