AWS Lambda times out on cold start and exhausts RDS connections

Our booking API runs on AWS Lambda behind API Gateway, with a PostgreSQL database on RDS in private subnets. For months it looked fine. Then the marketing team started sending push notifications at 10:00 sharp, and every morning we got a wave of 504s from API Gateway for about two minutes. The same thing happened, on a smaller scale, right after every deploy.

CloudWatch Logs for the function showed two different failures mixed together:

2025-05-27T03:00:04.118Z 9f1c2d0e-... Task timed out after 6.00 seconds
REPORT RequestId: 9f1c2d0e-... Duration: 6000.00 ms Billed Duration: 6000 ms Memory Size: 512 MB Max Memory Used: 141 MB Init Duration: 2873.41 ms

error: sorry, too many clients already
    at Parser.parseErrorMessage (/var/task/node_modules/pg-protocol/dist/parser.js:283:98)
    code: '53300'

The first is a plain Lambda timeout, with an unusually large Init Duration. The second is Postgres refusing a new connection because max_connections was reached.

Reproducing it

We reproduced it in staging with a load generator that jumps from zero to 300 concurrent requests, which is roughly what a push notification does to us:

# k6 script ramps 0 -> 300 virtual users in 10 seconds, each calling GET /availability
k6 run --vus 300 --duration 60s availability.js

Within 15 seconds, timeouts started. In the RDS console the DatabaseConnections metric climbed to the instance limit and stayed flat there, even after the load test stopped.

What we ruled out

• VPC networking adding cold start time. That was true years ago, but since Lambda moved to shared Hyperplane ENIs, attaching a function to a VPC does not add per-invocation ENI creation. Our Init Duration on a non-VPC copy of the function was nearly identical.
• Slow queries. Performance Insights showed the availability query at 4 ms average. The database was not busy, it was full of idle sessions.
• API Gateway's own timeout. The REST API integration timeout was 29 seconds; the function's 6-second timeout fired first.

Root cause

Two mistakes compounded. First, the handler did all its setup on every invocation: it fetched the database password from Secrets Manager, built a new pg.Client and connected, then ran a "schema version" check query before doing any work. That made a single cold invocation cost close to three seconds before the real query. Second, the client was never closed. When Lambda froze an execution environment after the response, its open connection stayed open on the Postgres side. During a burst, Lambda scaled to around 350 concurrent environments, each holding one or more connections, and a db.t4g.medium instance only allows about 410. New environments then waited on connection attempts until their 6-second timeout hit.

What is happening under the hood

A Lambda execution environment goes through an Init phase (start the runtime, run module-level code) and then any number of Invoke phases. Module-level code runs once per environment; code inside the handler runs on every request. Between invocations the environment is frozen, not destroyed, so anything created at module level, such as a connection pool or an SDK client, survives and is reused on the next warm invocation.

Concurrency in Lambda means environments. One environment handles one request at a time, so 300 concurrent requests means at least 300 environments, and each one opens its own database connections. Postgres creates a backend process per connection and enforces a hard max_connections ceiling. On RDS, the default value is derived from instance memory, which is why a medium instance runs out quickly.

There is also a 10-second ceiling on the Init phase for on-demand functions. If module-level code takes longer, Lambda retries initialization during the first invocation and counts it against the function timeout. Moving slow work to module level is good, but it still has to be fast.

Environment

Minimal reproduction

// handler.mjs (broken)
import pg from 'pg';
import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';

export const handler = async (event) => {
  const sm = new SecretsManagerClient({});
  const secret = await sm.send(new GetSecretValueCommand({ SecretId: process.env.DB_SECRET_ARN }));
  const { username, password } = JSON.parse(secret.SecretString);

  const client = new pg.Client({ host: process.env.DB_HOST, user: username, password, database: 'booking' });
  await client.connect();                       // new TCP + TLS + auth handshake per request
  await client.query('SELECT version FROM schema_meta'); // extra round trip
  const { rows } = await client.query('SELECT * FROM slots WHERE day = $1', [event.queryStringParameters.day]);
  return { statusCode: 200, body: JSON.stringify(rows) }; // client never closed
};

Who is affected

Any function that connects to a relational database with a fixed connection limit (Postgres, MySQL, SQL Server) and can scale beyond a few dozen concurrent environments. Functions that call DynamoDB or HTTP APIs do not have this exact failure, but still pay the cold-start cost of handler-level setup.

The fix

We changed three things: move setup to module scope, cap connections per environment, and put RDS Proxy between Lambda and the database.

1. Initialize once per environment. The secret, the pool and the schema check now live outside the handler. The pool has max: 1 because an environment never handles two requests at once.

// handler.mjs (fixed)
import pg from 'pg';
import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';

const sm = new SecretsManagerClient({});
const secret = await sm.send(new GetSecretValueCommand({ SecretId: process.env.DB_SECRET_ARN }));
const { username, password } = JSON.parse(secret.SecretString);

const pool = new pg.Pool({
  host: process.env.DB_PROXY_HOST,   // RDS Proxy endpoint, not the instance
  user: username,
  password,
  database: 'booking',
  ssl: { rejectUnauthorized: true },
  max: 1,                            // one request at a time per environment
  idleTimeoutMillis: 60_000,
  connectionTimeoutMillis: 2_000,    // fail fast instead of eating the whole timeout
});

export const handler = async (event) => {
  const day = event.queryStringParameters?.day;
  const { rows } = await pool.query('SELECT * FROM slots WHERE day = $1', [day]);
  return { statusCode: 200, body: JSON.stringify(rows) };
};

Top-level await works because the function is an ES module (.mjs), and the Node.js runtime waits for it to settle before the first invocation. The schema version check moved to the deploy pipeline, where it belongs.

2. Add RDS Proxy. The proxy keeps a pool of real database connections and multiplexes client sessions onto them. We set MaxConnectionsPercent to 60 so the proxy can never take the whole instance, leaving room for migrations and admin sessions.

aws rds modify-db-proxy-target-group \
  --db-proxy-name booking-proxy \
  --target-group-name default \
  --connection-pool-config MaxConnectionsPercent=60,MaxIdleConnectionsPercent=30

One caveat: RDS Proxy "pins" a client session to one database connection when the session uses features it cannot safely share, such as session-level SET statements or temporary tables. We checked the DatabaseConnectionsCurrentlySessionPinned metric after rollout and it stayed near zero because our queries are plain parameterized statements.

3. Cap and pre-warm concurrency. We set reserved concurrency to 400 on the function so a runaway burst cannot starve other functions in the account, and added provisioned concurrency of 40 on the live alias from 09:45 to 10:30 with Application Auto Scaling scheduled actions, so the 10:00 burst lands on initialized environments.

Alternatives and trade-offs

• Lower concurrency instead of a proxy. Reserved concurrency alone protects the database, but excess requests get throttled (429) instead of served.
• Larger instance. Raising max_connections by moving up instance classes costs money and only moves the ceiling.
• RDS Data API (Aurora only). HTTP-based, no connection management, but not available for our RDS for PostgreSQL instance and adds per-call latency.
• Close the connection at the end of each invocation. It stops the leak, but every request pays the full TCP, TLS and auth handshake again.

How to test the fix

We repeated the staging load test and watched both sides:

k6 run --vus 300 --duration 60s availability.js

aws cloudwatch get-metric-statistics --namespace AWS/RDS \
  --metric-name DatabaseConnections --dimensions Name=DBInstanceIdentifier,Value=booking-staging \
  --start-time 2025-06-02T08:00:00Z --end-time 2025-06-02T08:10:00Z --period 60 --statistics Maximum

On the database we confirmed where connections came from:

SELECT client_addr, state, count(*)
FROM pg_stat_activity
WHERE datname = 'booking'
GROUP BY 1, 2
ORDER BY 3 DESC;

Before the fix, hundreds of idle rows came from Lambda ENI addresses. After the fix, a few dozen rows came from the proxy only.

Confirming in production

The next three 10:00 pushes produced zero 504s. Numbers from the first week after rollout:

• p99 latency for GET /availability during the burst went from 6,000 ms (timeout) to 310 ms.
• Init Duration for on-demand cold starts went from about 2,900 ms to about 650 ms, and those cold starts now happen at module level instead of inside the timed request.
• DatabaseConnections on the instance peaked at 118, against the previous flat line at the limit.
• No sorry, too many clients already log lines in CloudWatch Logs Insights:

fields @timestamp, @message
| filter @message like /too many clients/
| stats count() by bin(1h)

Regression guard

• A CloudWatch alarm on DatabaseConnections above 70 percent of max_connections for five minutes.
• A Logs Insights saved query on Init Duration p95, reviewed in the weekly ops meeting.
• A lint rule in the repo that flags new pg.Client( and new pg.Pool( inside exported handler functions, with a message pointing to the shared db.mjs module.

Security implications

The fix improves the security posture slightly. Credentials are fetched once per environment, so there are fewer Secrets Manager calls to audit, and the database is now reachable only through the proxy's security group. The proxy can also enforce TLS and IAM authentication; we enabled RequireTLS on it. Keep in mind that module-level state persists across invocations, so never store per-user data (tokens, request bodies) in module variables, or it can leak into another user's request on the same warm environment.

Prevention checklist

• Create SDK clients and database pools at module level, never inside the handler.
• Use one connection per environment and size the database for maximum concurrency, not average.
• Put RDS Proxy (or another pooler) in front of any relational database used by Lambda.
• Set a connection timeout far below the function timeout so failures are fast and visible.
• Load test from zero to peak concurrency, because steady load hides cold start behaviour.

References

• AWS Lambda execution environment lifecycle — https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html
• Using Amazon RDS Proxy — https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.html
• Configuring provisioned concurrency — https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html
• Lambda reserved concurrency — https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html
• node-postgres pooling — https://node-postgres.com/features/pooling