Intermittent ECONNRESET and 502s: Node.js keepAliveTimeout vs load balancer idle timeout

Our pricing service is a Node.js API behind an AWS Application Load Balancer, called by the web frontend and by three internal Node services. For months it had a background error rate of about 0.05 percent: a few hundred failed requests per day out of roughly 600,000. Nobody could reproduce them, and every retry succeeded, so they lived in the "known noise" column of the dashboard until a partner integration started failing whole batch jobs on a single error.

The ALB access logs showed 502s with no target response time:

https 2025-06-30T14:02:11.913Z app/pricing-alb/... 203.0.113.24:51234 10.0.12.37:3000 0.000 -1 -1 502 - 412 272
"POST https://pricing.example.com:443/v3/quotes HTTP/1.1" ... "-" "-" 0 2025-06-30T14:02:11.912Z
"forward" "-" "-" "10.0.12.37:3000" "-" "-" "-" TID_...

The internal callers, which talk to the pricing service directly through Kubernetes service DNS, logged a different error for the same kind of failure:

Error: socket hang up
    at Socket.socketOnEnd (node:_http_client ...)
  code: 'ECONNRESET'

Reproducing it

The failures clustered on requests that arrived after a short pause on an existing connection. We reproduced it with a keep-alive client that waited exactly as long as the server's idle timeout before reusing the socket:

// repro.mjs
import http from 'node:http';

const agent = new http.Agent({ keepAlive: true, maxSockets: 1 });
const get = () => new Promise((resolve, reject) => {
  http.get('http://localhost:3000/health', { agent }, (res) => { res.resume(); res.on('end', resolve); })
    .on('error', reject);
});

await get();
await new Promise((r) => setTimeout(r, 5000)); // equal to the server keepAliveTimeout
await get().catch((e) => console.error(e.code, e.message)); // ECONNRESET socket hang up, often

Running it 100 times produced 17 failures. With a 4,000 ms pause it failed 0 times; with 6,000 ms it also failed 0 times, because the agent had already noticed the closed socket and opened a new one.

What we ruled out

• The application crashing. No restarts, no unhandled rejections, and memory was flat. The 502s showed -1 for target processing time, meaning the request never reached our handler.
• Slow requests hitting a timeout. The failed requests were not slow; most were quote lookups that normally take 30 ms.
• Pod terminations during deploys. Errors happened at the same rate on days without deploys, and our preStop hook already handled graceful shutdown.

Root cause

Node's HTTP server closes idle keep-alive connections after server.keepAliveTimeout, which defaults to 5 seconds. The ALB keeps idle connections to targets open for its idle timeout, 60 seconds by default, and reuses them freely. When the ALB sent a request on a connection at the same moment Node was closing it, the target reset the connection and the ALB returned a 502. Our internal callers had the same race with Node's own client agents, which reuse idle sockets and cannot know for certain that the server is about to close one.

What is happening under the hood

HTTP/1.1 keep-alive lets a client send many requests over one TCP connection. Either side may close an idle connection at any time, and there is no message that says "I am about to close". The client only learns about it when its next write fails or the server's FIN arrives.

That leaves a race window whenever the server's idle timeout is equal to or shorter than the client's:

AWS documents this directly: the target's keep-alive timeout should be longer than the load balancer's idle timeout, so that the load balancer always closes first. The rule generalises: in any chain of keep-alive hops, the downstream server must keep idle connections open longer than the upstream client will try to reuse them.

Node's server does advertise its limit with a Keep-Alive: timeout=5 response header, and both Node's http.Agent and undici use that hint to expire idle sockets. That narrows the window but does not close it: the two timers start at slightly different moments on each side, and a request already in flight when the server closes is lost.

Node has a second, related setting. server.headersTimeout limits how long the server waits for the complete request headers, and it must not be shorter than keepAliveTimeout, otherwise a slow request on a reused connection can be cut off by the headers timer instead.

Environment

Minimal reproduction

// server.mjs
import http from 'node:http';

const server = http.createServer((req, res) => res.end('ok'));
// Defaults: server.keepAliveTimeout === 5000
server.listen(3000);

Put this behind an ALB with default settings, send low but steady traffic so connections go idle for a few seconds at a time, and the 502s appear at a low background rate.

Who is affected

Any Node.js HTTP server with default settings behind a reverse proxy that reuses upstream connections: AWS ALB, Google Cloud load balancers, nginx with keepalive in an upstream block, Envoy and service meshes. Low-traffic services are hit hardest, because their connections spend more time idle. Busy services keep connections active and rarely cross the 5-second line.

The fix

Server side: outlive the load balancer. We set Node's keep-alive timeout above the ALB's 60 seconds and kept headersTimeout above that:

// server.mjs (Fastify)
import Fastify from 'fastify';

const app = Fastify({
  keepAliveTimeout: 65_000, // longer than ALB idle timeout (60 s)
  logger: true,
});

app.server.headersTimeout = 66_000; // must be greater than keepAliveTimeout
await app.listen({ port: 3000, host: '0.0.0.0' });

With a plain http server, the same two properties go on the server object:

const server = http.createServer(handler);
server.keepAliveTimeout = 65_000;
server.headersTimeout = 66_000;

Client side: expire idle sockets before the server does. For internal callers we did the reverse: make the client give up on idle sockets earlier than the server's timeout. Undici exposes this per dispatcher:

import { Agent, setGlobalDispatcher } from 'undici';

setGlobalDispatcher(new Agent({
  keepAliveTimeout: 30_000,     // client-side idle limit, below the server's 65 s
  keepAliveMaxTimeout: 60_000,
}));

For callers on Node's http.Agent, we set the timeout option on the agent so idle sockets are closed client-side well before 65 seconds.

Retry only what is safe. Even with correct timeouts, connections can drop for other reasons. We let the internal HTTP client retry once on ECONNRESET for idempotent methods (GET, PUT, DELETE) and for POSTs carrying our Idempotency-Key header. Plain POSTs are not retried automatically.

Alternatives and trade-offs

• Lower the ALB idle timeout instead. Setting it to 4 seconds would also make the ALB close first, but it affects every client connection and long-polling endpoints. We wanted the default.
• Disable keep-alive on the server. Removes the race at the cost of a TCP (and often TLS) handshake per request. Our p50 latency rose by 9 ms in a test, so no.
• Retries everywhere. They hide the symptom and can duplicate non-idempotent writes. Retries are a safety net, not the fix.

How to test the fix

We reran the reproduction script against the new server with pauses between 4 and 64 seconds, 100 times each:

for pause in 4000 5000 6000 30000 64000; do
  PAUSE=$pause node repro.mjs | sort | uniq -c
done

Zero failures at every pause. We also verified the effective values on the running process:

curl -s http://pricing.internal.example.com/debug/timeouts
# {"keepAliveTimeout":65000,"headersTimeout":66000}

The second command hits a debug endpoint we added that reports app.server.keepAliveTimeout and app.server.headersTimeout, because it is easy to set these on the wrong object.

Confirming in production

• ALB 502s with target processing time -1 dropped from about 300 per day to zero over the first week. We query them in Athena:

SELECT date(from_iso8601_timestamp(time)) AS day, count(*) AS bad_gateway
FROM alb_logs
WHERE elb_status_code = 502 AND target_processing_time = -1
  AND time > '2025-07-01'
GROUP BY 1 ORDER BY 1;

• ECONNRESET errors in internal callers fell from about 120 per day to 2 to 3 per day, all during node rotations, and all succeeded on retry.
• The partner's batch jobs ran for 30 days without a failed run.
• Open connections per pod rose from around 40 to 70, well within limits, as expected when idle connections live longer.

Regression guard

• A unit test starts the server and asserts server.keepAliveTimeout > 60000 and server.headersTimeout > server.keepAliveTimeout.
• Our service template sets both values from one constant, LB_IDLE_TIMEOUT_MS + 5000, so changing the load balancer setting in Terraform and the app together is one variable.
• An alert fires if ALB 502s with -1 target time exceed 10 in an hour.

Security implications

Longer keep-alive timeouts mean more idle connections held open, which slightly increases exposure to connection-exhaustion attacks. The ALB fronts all public traffic and only it connects to the pods, so clients on the internet cannot hold sockets open on Node directly. Keep headersTimeout and requestTimeout bounded (Node's defaults are 60 seconds and 300 seconds) to protect against slow header attacks, and do not set either to 0. Automatic retries must stay limited to idempotent requests to avoid duplicate charges or writes.

Prevention checklist

• In every keep-alive hop, make the server's idle timeout longer than the client's.
• Set keepAliveTimeout and headersTimeout explicitly in your Node service template.
• Give HTTP clients an idle socket timeout shorter than the server's.
• Retry ECONNRESET only for idempotent requests or requests with an idempotency key.
• Monitor 502s with no target response time as a distinct signal from application errors.

References

• Node.js docs: server.keepAliveTimeout — https://nodejs.org/docs/latest-v22.x/api/http.html#serverkeepalivetimeout
• Node.js docs: server.headersTimeout — https://nodejs.org/docs/latest-v22.x/api/http.html#serverheaderstimeout
• AWS docs: Application Load Balancer connection idle timeout — https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html#connection-idle-timeout
• AWS docs: Troubleshoot your Application Load Balancer (HTTP 502) — https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html
• Undici docs: Agent options — https://undici.nodejs.org/#/docs/api/Agent