Kubernetes basics learned by fixing one broken deploy, from Pod to Ingress

New engineers on our team usually learn Kubernetes the same way: someone hands them a YAML file that does not work, and they fix it. That turns out to be a better way to learn the core objects than reading definitions, because each mistake shows what an object is responsible for.

This post walks through that exercise. The application is a small web service, notes-api, listening on port 8080 and reading its database URL from an environment variable. The goal is to reach it at https://notes.example.com. The manifests contain five mistakes. We fix them in the order you would meet them.

The manifests

apiVersion: v1
kind: ConfigMap
metadata:
  name: notes-config
data:
  DATABASE_HOST: postgres.data.svc.cluster.local
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: notes-api
  template:
    metadata:
      labels:
        app: notes-api
    spec:
      containers:
        - name: api
          image: registry.example.com/notes-api:1.4
          ports:
            - containerPort: 8080
          envFrom:
            - configMapRef:
                name: notes-configs
          resources:
            requests: { cpu: "100m", memory: "32Mi" }
            limits:   { memory: "64Mi" }
---
apiVersion: v1
kind: Service
metadata:
  name: notes-api
spec:
  selector:
    app: notes
  ports:
    - port: 80
      targetPort: 8000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: notes-api
spec:
  rules:
    - host: notes.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: notes-api
                port:
                  number: 80

kubectl apply -f notes.yaml reports four objects created. Nothing works.

Mistake 1: the Pod cannot start (Deployment and Pod)

What the objects do. A Pod is one or more containers scheduled together on a node, sharing a network address. You rarely create Pods directly. A Deployment declares how many identical Pods you want and what they look like (the template), and a controller keeps reality matching that declaration, replacing Pods that die and rolling out changes.

kubectl get pods -l app=notes-api

NAME                         READY   STATUS                       RESTARTS   AGE
notes-api-6d8f9c7b5d-4jx2p   0/1     CreateContainerConfigError   0          40s
notes-api-6d8f9c7b5d-q8wln   0/1     CreateContainerConfigError   0          40s

CreateContainerConfigError means Kubernetes could not build the container's configuration. describe says why:

kubectl describe pod notes-api-6d8f9c7b5d-4jx2p | tail -5

Events:
  Type     Reason  Age   From     Message
  Warning  Failed  12s   kubelet  Error: configmap "notes-configs" not found

The Deployment references notes-configs; the ConfigMap is called notes-config.

Fix: correct the name in configMapRef. A ConfigMap holds non-secret configuration as key-value pairs, which Pods consume as environment variables or files. Secrets work the same way for sensitive values. Note that environment variables from a ConfigMap are read when the container starts; changing the ConfigMap later does not affect running containers until they restart.

Mistake 2: the container keeps dying (resources)

After the fix:

NAME                         READY   STATUS      RESTARTS      AGE
notes-api-7b9d5f6c8-m2kvd    0/1     OOMKilled   3 (20s ago)   90s

kubectl describe pod notes-api-7b9d5f6c8-m2kvd | grep -A3 "Last State"

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

The memory limit is 64 Mi. Someone had lowered the request to 32 Mi and the limit to 64 Mi to "save resources". The app needs about 150 Mi once it has loaded its configuration and connection pool.

What they do. Requests are what the scheduler reserves on a node for the container; they decide placement. Limits are the maximum the container may use. Exceeding the memory limit gets the process killed by the kernel (OOMKilled, exit code 137). Exceeding a CPU limit only throttles it.

Fix: measure real usage (kubectl top pod once it runs) and set the request near typical use and the limit with headroom:

resources:
  requests: { cpu: "100m", memory: "192Mi" }
  limits:   { memory: "256Mi" }

Mistake 3: the Service has no endpoints (Service and selectors)

Pods are now Running and 1/1 READY. From another pod in the cluster:

kubectl run tmp --rm -it --image=busybox:1.36 -- wget -qO- http://notes-api/health

wget: can't connect to remote host (10.96.41.12): Connection refused

What a Service does. Pods come and go, and their IP addresses change. A Service gives a stable name and virtual IP, and forwards traffic to whichever Pods match its selector. The list of matching Pod addresses is kept in EndpointSlices.

kubectl get endpointslices -l kubernetes.io/service-name=notes-api

NAME              ADDRESSTYPE   PORTS     ENDPOINTS   AGE
notes-api-x7k2m   IPv4          <unset>   <unset>     6m

No endpoints. The Service selects app: notes; the Pods are labelled app: notes-api. Labels are the only connection between a Service and its Pods, so a typo breaks it silently.

Fix: selector: { app: notes-api }. The endpoints now list both Pod IPs.

Mistake 4: connection refused through the Service (ports)

wget: can't connect to remote host (10.96.41.12): Connection refused

Same error, different cause. The Service has endpoints now, but targetPort: 8000 sends traffic to port 8000 on the Pods. The app listens on 8080.

What the ports mean. port is the port the Service itself listens on (80 here, so clients use http://notes-api). targetPort is where traffic is sent on the Pod. It must match the container's listening port. Naming the container port and referring to it by name avoids this class of mistake:

# in the Deployment
ports:
  - name: http
    containerPort: 8080
# in the Service
ports:
  - port: 80
    targetPort: http

After the fix, wget -qO- http://notes-api/health returns ok.

Mistake 5: the Ingress does nothing (Ingress and IngressClass)

curl -i https://notes.example.com/health

curl: (7) Failed to connect to notes.example.com port 443: Connection refused

kubectl get ingress notes-api

NAME        CLASS    HOSTS               ADDRESS   PORTS   AGE
notes-api   <none>   notes.example.com             80      9m

No class and no address. What an Ingress does: it is a set of HTTP routing rules (host and path to Service). It does nothing on its own. An Ingress controller (ingress-nginx, a cloud load balancer controller, and others) watches Ingress objects and configures a real proxy. When a cluster has several controllers, or none is marked default, an Ingress without ingressClassName is ignored by all of them.

kubectl get ingressclass

NAME    CONTROLLER             PARAMETERS   AGE
nginx   k8s.io/ingress-nginx   <none>       120d

Fix: add ingressClassName: nginx, plus TLS:

spec:
  ingressClassName: nginx
  tls:
    - hosts: [notes.example.com]
      secretName: notes-tls

The certificate in notes-tls comes from cert-manager in our cluster. After a few seconds the Ingress shows an address, DNS points at it, and curl https://notes.example.com/health returns ok.

Kubernetes also offers the newer Gateway API, which splits routing into Gateway and HTTPRoute objects with clearer ownership between platform and application teams. The concepts are the same: rules on one side, a controller that implements them on the other.

The chain, in one picture

Every request travels the same chain, and each link is matched by name or label:

DNS notes.example.com
  -> Ingress controller (selected by ingressClassName)
    -> Ingress rule (host + path)
      -> Service notes-api, port 80 (by name)
        -> EndpointSlice (Pods selected by label app=notes-api)
          -> Pod port 8080 (targetPort)
            -> container, configured from ConfigMap notes-config (by name)

When something does not work, walk the chain from the inside out: is the Pod running and ready, does the Service have endpoints, does the Service reach the Pod port, does the Ingress have a class and an address.

A debugging cheat sheet

Summary

• A Deployment keeps a set of identical Pods running from a template; Pods are disposable.
• ConfigMaps and Secrets supply configuration by name; environment values are read at container start.
• Requests drive scheduling, limits cap usage; exceeding the memory limit means OOMKilled.
• A Service finds Pods only through labels, and targetPort must match the container port.
• An Ingress is only rules; a controller selected by ingressClassName makes them real.

References: Kubernetes Deployments, Kubernetes Services, Kubernetes Ingress, Resource management for pods and containers.