Users randomly logged out: Redis maxmemory eviction removed session keys

Support tickets with the same theme started to pile up in late April: "I keep getting logged out", "my cart disappeared when I came back from lunch", "I had to sign in three times today". Our session lifetime is 7 days, so nobody should be logged out within hours. There was no error anywhere. The web app simply received requests with a valid, signed session cookie whose session id no longer existed in the store, and treated them as anonymous.

The Redis server that backs the session store told the story in two lines of INFO:

redis-cli -h cache-1.internal.example.com INFO stats | grep -E 'evicted_keys|expired_keys'
redis-cli -h cache-1.internal.example.com INFO memory | grep -E 'used_memory_human|maxmemory_human|maxmemory_policy'

evicted_keys:1843302
expired_keys:902117
used_memory_human:1.99G
maxmemory_human:2.00G
maxmemory_policy:allkeys-lru

Redis was full, and it had evicted 1.8 million keys since its last restart.

Reproducing it

We correlated the eviction rate with time of day. evicted_keys grew in bursts at five minutes past every hour, and the login page's traffic jumped about ten minutes later as logged-out users signed in again. Five past the hour is when a job warms the product recommendation cache for the whole catalogue, writing about 600 MB of new keys in a few minutes.

To prove it, we pointed a staging app at a Redis instance with maxmemory 64mb and allkeys-lru, logged in, left the session idle for a minute, ran a script that wrote 80 MB of cache keys, and reloaded the page. We were logged out every time.

What we ruled out

• Session TTL misconfiguration. TTL sess:<id> on fresh sessions returned about 604,800 seconds, seven days, as configured.
• Cookie problems. The session cookie was present and correctly signed in requests from logged-out users; the server could not find the matching key.
• Redis restarts or failovers. uptime_in_seconds showed 23 days with no restarts, and there was no replica promotion in that period.

Root cause

Three changes had accumulated over a year. Sessions and application cache were placed in the same Redis instance, using different key prefixes, to save money. Later, someone set maxmemory-policy allkeys-lru to stop write errors when Redis reached maxmemory. Finally, the new recommendation cache made the cache's working set grow past what the instance could hold. From that point, every write that needed memory evicted the least recently used keys across the whole instance, and an idle user's session is exactly that: a key nobody has touched for hours.

What is happening under the hood

When maxmemory is reached, Redis applies the configured maxmemory-policy before executing a command that needs more memory:

The LRU is approximate: Redis samples a few keys (maxmemory-samples, default 5) and evicts the best candidate among them, rather than tracking a perfect global order.

Two details matter for sessions. First, volatile-lru would not have protected them: session keys have a TTL, so they are eligible for eviction under every volatile-* policy too. Second, Redis logical databases (SELECT 0, SELECT 1) do not help either. All databases in an instance share one maxmemory limit and one eviction pool, so moving sessions to database 1 would have changed nothing.

The only reliable separation is a separate Redis instance with its own memory limit and its own policy.

Environment

Minimal reproduction

redis-server --port 6390 --maxmemory 16mb --maxmemory-policy allkeys-lru &
redis-cli -p 6390 SET sess:abc '{"user":42}' EX 604800
sleep 3   # let the session go idle
redis-benchmark -p 6390 -q -t set -n 400000 -r 1000000 -d 100   # cache-like write pressure
redis-cli -p 6390 EXISTS sess:abc
redis-cli -p 6390 INFO stats | grep evicted_keys

EXISTS returns 0: the idle session was evicted along with roughly 300,000 benchmark keys, even though its TTL had almost seven days left.

Who is affected

Any setup where data that must not be lost (sessions, rate-limit counters, queues, locks) shares a Redis instance with cache data under an eviction policy. Managed services are not immune: they choose a default policy, often one of the volatile-* policies, which still evicts sessions with TTLs.

The fix

We split Redis into two instances with different jobs.

Sessions instance: noeviction, sized with headroom, persistence on. If it ever fills up, writes fail loudly instead of silently deleting sessions, and an alert fires long before that.

# redis-sessions.conf
maxmemory 1gb
maxmemory-policy noeviction
appendonly yes
appendfsync everysec

Cache instance: allkeys-lfu, because recommendations accessed by many users should survive over one-off keys, and losing any cache key is harmless by design.

# redis-cache.conf
maxmemory 3gb
maxmemory-policy allkeys-lfu
appendonly no
save ""

In the app, the session store and the cache client now get different connection URLs:

-const redis = createClient({ url: process.env.REDIS_URL });
-app.use(session({ store: new RedisStore({ client: redis, prefix: 'sess:' }), ...sessionOpts }));
-const cache = new Cache(redis);
+const sessionRedis = createClient({ url: process.env.REDIS_SESSIONS_URL });
+const cacheRedis = createClient({ url: process.env.REDIS_CACHE_URL });
+await Promise.all([sessionRedis.connect(), cacheRedis.connect()]);
+app.use(session({ store: new RedisStore({ client: sessionRedis, prefix: 'sess:' }), ...sessionOpts }));
+const cache = new Cache(cacheRedis);

We migrated live sessions by copying sess:* keys with their TTLs from the old instance to the new one with a small script using SCAN, DUMP, PTTL and RESTORE, then switched the app over. Users who had been logged in stayed logged in.

Sizing the session instance

We sized from measurements rather than guesses:

redis-cli -h sessions-1.internal.example.com --memkeys --memkeys-samples 0 | tail -n 5
redis-cli -h sessions-1.internal.example.com MEMORY USAGE sess:0b1f2c...

310,000 sessions at about 1.2 KB plus per-key overhead came to roughly 450 MB. We set maxmemory to 1 GB to allow for twice the peak session count, and kept the VM at 2 GB so that AOF rewrites, which fork the process, have room for copy-on-write pages.

Alternatives and trade-offs

1. Keep one instance, size it so that it never evicts. Works until the next cache feature. It depends on everyone remembering a constraint that nothing enforces.
2. Sessions in the primary database. Durable and simple, at the cost of a database read on every request. Reasonable for low traffic.
3. Stateless sessions in signed cookies or tokens. No server-side store to evict, but revocation and size limits become your problem.
4. volatile-lru with cache keys that all have TTLs and sessions without TTL. Technically protects sessions, but sessions would never expire on their own, and one cache key written without a TTL changes the behaviour again. We rejected it as too fragile.

How to test the fix

An integration test starts both instances with small memory limits, creates a session, floods the cache instance and asserts the session survives:

test('cache pressure does not evict sessions', async () => {
  const agent = request.agent(app);
  await agent.post('/login').send({ email: 'qa@example.com', password: 'test-password' }).expect(302);

  await floodCache(cacheRedis, 50 * 1024 * 1024); // 50 MB into a 16 MB cache instance

  await agent.get('/account').expect(200);
  expect(await cacheRedis.info('stats')).toMatch(/evicted_keys:[1-9]/);
});

npx vitest run test/integration/sessions-eviction.test.js

Confirming in production

In the two weeks after the split, evicted_keys on the sessions instance stayed at 0 and its memory peaked at 520 MB. The cache instance evicts around 40,000 keys per hour, as intended. Hourly login spikes after the cache warm-up disappeared, sign-ins per active user dropped by 31%, and "logged out" tickets went from about 25 per week to 1, which turned out to be a user clearing cookies.

Regression guard

• An alert fires if evicted_keys on the sessions instance increases at all, and when its used_memory passes 70% of maxmemory.
• A startup check in the app reads CONFIG GET maxmemory-policy on the session connection and refuses to start if it is anything other than noeviction, where our managed setup permits the command.
• A dashboard tracks session count and average session size so that capacity planning sees growth.

Security implications

Random logouts are an availability and user-experience problem, but the shared instance also had a security angle: anyone able to write large amounts of cache data, for example by requesting many uncached search pages, could indirectly evict other users' sessions, and the same would apply to rate-limit counters stored in that instance, effectively resetting them. Separating stateful keys from cache removes that lever. With noeviction on sessions, a full instance makes logins fail instead, so the memory alert is part of the fix, not an extra.

Prevention checklist

• Keep data that must not disappear (sessions, locks, counters, queues) in a Redis instance with noeviction.
• Remember that volatile-* policies still evict keys with TTLs, and logical databases share memory.
• Alert on evicted_keys for every instance where eviction is not expected.
• Size Redis from measured key counts and sizes, with headroom for forks.
• Review memory impact when adding a new cache feature to a shared instance.

References

• Redis docs, Key eviction (maxmemory policies) — https://redis.io/docs/latest/develop/reference/eviction/
• Redis docs, INFO command — https://redis.io/docs/latest/commands/info/
• Redis docs, MEMORY USAGE — https://redis.io/docs/latest/commands/memory-usage/
• Redis docs, Persistence (AOF and RDB) — https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/
• express-session documentation — https://github.com/expressjs/session