Docker Compose depends_on is not enough: app crashes with Connection refused to the database

Our booking API runs its integration tests in GitHub Actions with docker compose up, and new developers use the same compose file locally. For months we lived with a flaky first run: on about one in five CI runs, and on nearly every brand-new laptop, the API container exited immediately:

api-1  | > booking-api@3.8.0 start
api-1  | > node dist/server.js
api-1  |
api-1  | Error: connect ECONNREFUSED 172.19.0.2:5432
api-1  |     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1611:16) {
api-1  |   errno: -111,
api-1  |   code: 'ECONNREFUSED',
api-1  |   syscall: 'connect',
api-1  |   address: '172.19.0.2',
api-1  |   port: 5432
api-1  | }
api-1 exited with code 1
db-1   | 2024-10-29 08:12:44.913 UTC [1] LOG:  database system is ready to accept connections

Note the order: the API gave up before Postgres logged that it was ready. Re-running the job usually passed, so it was filed under "flaky" and ignored, until a new hire lost a morning to it.

Reproducing it

The failure is almost guaranteed on a cold start with an empty volume, because Postgres has to run initdb and our init scripts first:

docker compose down -v      # remove the data volume
docker compose up api

What we ruled out

• Wrong hostname or port. The API connects to db:5432, and the address in the error is the db container's IP. Later runs with the same config worked.
• Postgres crashing. The database logs showed a normal, if slow, first boot.
• Networking between containers. Both services were on the default project network; once Postgres was up, docker compose exec api nc -zv db 5432 succeeded.

Root cause

The compose file used the short form of depends_on:

services:
  api:
    build: .
    depends_on:
      - db
  db:
    image: postgres:16.4

That only makes Compose start the db container before the api container. It does not wait for Postgres inside it to accept connections. On a cold start, the Postgres entrypoint runs initdb, starts a temporary server for the init scripts, stops it, and only then starts the real server on TCP. That takes 3 to 8 seconds on CI runners. Our API connected once at boot, failed, and exited.

What is happening under the hood

Compose models startup order, not readiness. A container is "started" as soon as its main process is running. For a database, that is long before it listens. Compose can only know about readiness if you tell it how to check, which is what a healthcheck is for. With a healthcheck defined, the container's health moves from starting to healthy or unhealthy, and the long form of depends_on can wait for healthy.

There is a subtle trap with the official Postgres image. During initialisation it runs a temporary server that listens only on the Unix socket, not on TCP, so that clients cannot connect to a half-initialised database. A healthcheck such as pg_isready with no host connects through the socket, so it can report ready while the temporary server is up, then the server restarts and TCP clients still get refused. Checking over TCP with -h 127.0.0.1 checks what the app actually uses.

Environment

Minimal reproduction

services:
  app:
    image: postgres:16.4
    command: ["psql", "-h", "db", "-U", "postgres", "-c", "select 1"]
    environment:
      PGPASSWORD: example
    depends_on:
      - db
  db:
    image: postgres:16.4
    environment:
      POSTGRES_PASSWORD: example

docker compose down -v && docker compose up app
# psql: error: connection to server at "db" (172.20.0.2), port 5432 failed: Connection refused

Who is affected

Any Compose setup where one service needs another to be ready: apps and databases, workers and message brokers, migrations and Elasticsearch or OpenSearch. Version 3 of the old Compose file format removed condition from depends_on, so many tutorials and older projects still use the short form. The current Compose Specification supports conditions again.

The fix

Two parts: make Compose wait for real readiness, and make the app tolerate the database being briefly unavailable anyway.

1. Healthcheck and service_healthy.

services:
  api:
    build: .
    environment:
      DATABASE_URL: postgres://booking:example@db:5432/booking
    depends_on:
      db:
        condition: service_healthy
        restart: true
      migrate:
        condition: service_completed_successfully

  migrate:
    build: .
    command: ["npm", "run", "migrate"]
    environment:
      DATABASE_URL: postgres://booking:example@db:5432/booking
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:16.4
    environment:
      POSTGRES_USER: booking
      POSTGRES_PASSWORD: example
      POSTGRES_DB: booking
    volumes:
      - ./docker/initdb:/docker-entrypoint-initdb.d:ro
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -p 5432 -U booking -d booking"]
      interval: 10s
      timeout: 3s
      retries: 5
      start_period: 30s
      start_interval: 1s

volumes:
  pgdata:

What each part does:

• pg_isready -h 127.0.0.1 checks over TCP, so the temporary socket-only init server does not count.
• start_period gives Postgres 30 seconds where failures do not count against retries, and start_interval (Docker Engine 25 or later) probes every second during that window, so a fast boot is detected in about a second instead of waiting for the 10-second interval.
• restart: true restarts the API if the database container is restarted by Compose.
• Migrations run once in their own service, and the API waits for it to complete successfully.

2. Retry at startup in the app. Compose ordering does not exist in production, and databases restart. The API now retries the first connection with backoff instead of exiting:

import { Pool } from 'pg';

export async function connectWithRetry(pool: Pool, attempts = 10): Promise<void> {
  for (let i = 1; i <= attempts; i++) {
    try {
      const client = await pool.connect();
      client.release();
      return;
    } catch (err) {
      if (i === attempts) throw err;
      const delay = Math.min(500 * 2 ** (i - 1), 5000);
      console.warn(`db not ready (attempt ${i}/${attempts}), retrying in ${delay} ms`);
      await new Promise((r) => setTimeout(r, delay));
    }
  }
}

Alternatives and trade-offs

• Wait scripts such as wait-for-it.sh or dockerize. They wrap the command and poll the port. They work, but they duplicate what Compose healthchecks now do natively and only check that a port is open, not that the database is usable.
• docker compose up --wait. Waits until services are running and healthy before returning, which is useful in CI before starting tests from the host. It complements, not replaces, depends_on.
• restart: on-failure on the app. Hides the race by crash-looping until the database is up. It works eventually, but logs fill with errors and tests may start against a restarting app.

How to test the fix

Run a cold start in a loop and count failures:

for i in $(seq 1 20); do
  docker compose down -v >/dev/null 2>&1
  if docker compose up -d --wait api >/dev/null 2>&1; then echo "run $i ok"; else echo "run $i FAILED"; fi
done
docker compose ps --format "table {{.Service}}\t{{.Status}}"

All 20 runs succeed, and docker compose ps shows db as Up ... (healthy).

Confirming in production

• In CI, the integration job's "flaky first-run" retries dropped from 47 in the previous month to zero in the next 300 runs.
• Cold-start time to a healthy API went from "fail or about 9 s" to a consistent 6 to 7 s, because start_interval detects readiness quickly.
• In staging on ECS, where Compose does not exist, the retry loop logged two "db not ready" warnings during an RDS maintenance failover and recovered without a task restart.

Regression guard

• A CI job runs the cold-start loop five times nightly with down -v.
• A lint step in CI fails if any service in compose.yaml uses the short depends_on form for a service that has a healthcheck.
• The app's startup retry has a unit test with a fake pool that fails three times then succeeds.

Security implications

None beyond the bug. The healthcheck uses pg_isready, which does not need a password. Keep real credentials out of the compose file; ours uses placeholder values for local and CI only and reads production secrets from the platform's secret store. Do not publish the database port to the host in CI unless the tests need it.

Prevention checklist

• Give every stateful service in Compose a healthcheck that tests what clients use.
• Use the long depends_on form with service_healthy or service_completed_successfully.
• Run migrations as a separate one-shot service.
• Make apps retry their first connection with backoff.
• Test cold starts with an empty volume, not just warm restarts.

References

• Docker documentation: Control startup and shutdown order in Compose — https://docs.docker.com/compose/how-tos/startup-order/
• Compose file reference: depends_on — https://docs.docker.com/reference/compose-file/services/#depends_on
• Compose file reference: healthcheck — https://docs.docker.com/reference/compose-file/services/#healthcheck
• Postgres official image documentation (initialization scripts) — https://hub.docker.com/_/postgres
• PostgreSQL documentation: pg_isready — https://www.postgresql.org/docs/current/app-pg-isready.html