One KEYS user:* command froze our production Redis for four seconds

Our API latency graph had a strange pattern: every few minutes during business hours, p99 latency for every endpoint jumped from about 40 ms to over 4 seconds for one data point, then went back to normal. Not one endpoint, all of them, at the same instant. Errors followed the same pattern, mostly timeouts reading from Redis:

redis.exceptions.TimeoutError: Timeout reading from socket

Every service uses the same Redis instance for sessions and caching, so a pause in Redis would explain a pause everywhere. The Redis host's CPU graph showed one core at 100 percent for a few seconds at exactly those moments.

SLOWLOG

Redis records commands that take longer than slowlog-log-slower-than microseconds (default 10,000, which is 10 ms). The slow log is the first thing to check when Redis stalls:

127.0.0.1:6379> SLOWLOG GET 3
1) 1) (integer) 88214
   2) (integer) 1792407112
   3) (integer) 4127733
   4) 1) "KEYS"
      2) "user:*:sessions"
   5) "10.0.4.21:51022"
   6) "admin-api"
2) 1) (integer) 88213
   2) (integer) 1792406933
   3) (integer) 4093120
   4) 1) "KEYS"
      2) "user:*:sessions"
   ...

KEYS user:*:sessions, taking 4.1 seconds (4,127,733 microseconds), from a client named admin-api. The LATENCY framework, enabled with latency-monitor-threshold, confirmed the same timing:

127.0.0.1:6379> LATENCY LATEST
1) 1) "command"
   2) (integer) 1792407112
   3) (integer) 4127
   4) (integer) 4310

The feature behind it

The admin tool had a support page that showed "active users right now". Its implementation:

def active_user_count() -> int:
    keys = redis.keys("user:*:sessions")
    return len(keys)

The dashboard auto-refreshed every two minutes. With a couple of support agents keeping the page open, that was a KEYS call every minute or so. The instance held about 40 million keys: sessions, cache entries, rate limit counters.

Redis executes commands one at a time

Redis processes commands on a single main thread. While one command runs, every other client's command waits. That is what makes Redis operations atomic and simple to reason about, and it is why a slow command is never just slow for its caller: it is slow for everyone.

KEYS pattern walks the entire keyspace and matches each key against the pattern. Its cost is O(N) in the total number of keys in the database, not in the number of matches. With 40 million keys, it touched 40 million entries to find about 180,000 matches. The Redis documentation for KEYS warns explicitly that it should be used in production only with extreme care, and recommends SCAN or sets instead.

The pattern user:*:sessions makes no difference. A prefix like user:* might look like it narrows the search, but Redis has no index by prefix; it still scans everything.

Why it looked like a network problem

From the application's point of view, Redis simply stopped answering for four seconds. Client libraries reported socket timeouts, connection pools filled with waiting requests, and some health checks failed. Without the slow log, it would have been easy to blame the network or the Redis host.

Environment

1. Stop scanning the keyspace for a count

The feature only needed a number. The right data structure is one that maintains the answer as data changes, instead of computing it by searching. We track active users in a sorted set, scored by last activity time:

def touch_session(user_id: int) -> None:
    redis.zadd("active_users", {user_id: time.time()})

def active_user_count(window_seconds: int = 900) -> int:
    cutoff = time.time() - window_seconds
    redis.zremrangebyscore("active_users", "-inf", cutoff)    # trim old entries
    return redis.zcard("active_users")

ZADD is O(log N), ZCARD is O(1), and ZREMRANGEBYSCORE is O(log N + M) where M is the number removed. The count now takes microseconds. A scheduled job trims the set every minute too, so the dashboard call rarely has much to remove.

For "list all sessions of user X", the other use of the same pattern, each user's session IDs now live in a set user:{id}:sessions, maintained when sessions are created and deleted.

2. Use SCAN when you must iterate

Some tasks really do need to visit many keys: a one-off migration, a cleanup of an old key prefix. For those, SCAN iterates the keyspace with a cursor, a small batch per call:

def iter_keys(pattern: str, batch: int = 1000):
    cursor = 0
    while True:
        cursor, keys = redis.scan(cursor=cursor, match=pattern, count=batch)
        yield from keys
        if cursor == 0:
            break

Each SCAN call does a bounded amount of work, so other clients' commands run in between. The trade-offs are documented: a key may be returned more than once, and keys added or removed during the iteration may or may not appear. COUNT is a hint about work per call, not a guarantee of how many keys are returned; with a selective pattern, many calls return nothing.

SCAN still visits every key overall. It spreads the cost, it does not reduce it, so it belongs in background jobs, never in a request path.

3. Make KEYS unavailable to applications

To ensure no other code can do this again, we removed the command for application users with ACLs, which are the recommended mechanism since Redis 6:

ACL SETUSER admin-api on >REDACTED ~* &* +@all -@dangerous
ACL SETUSER web-app   on >REDACTED ~* &* +@all -@dangerous

The @dangerous category includes KEYS, FLUSHALL, FLUSHDB, DEBUG and other commands that can stall or wipe the server. An application using a forbidden command now receives an immediate error:

(error) NOPERM User admin-api has no permissions to run the 'keys' command

rename-command in redis.conf is the older way to disable commands and still works, but it is deprecated in favour of ACLs, and it applies to all users, including operators who occasionally need KEYS on a local copy.

4. Separate workloads

Sessions, cache entries and rate limits all lived in one instance, so one expensive command hurt all of them. We moved the cache to its own Redis instance. A slow cache command now affects cache reads, not logins.

1. Reproduction and measurement. On a staging instance loaded with 40 million keys, we measured both approaches with redis-cli --latency running in parallel:

2. Production. After deploying the sorted set and the ACL change, the periodic latency spikes disappeared. SLOWLOG GET has not shown an entry above 10 ms from application clients since. We also lowered slowlog-log-slower-than to 5,000 microseconds and export the slow log length as a metric, alerting on growth.

3. Code check. A CI grep fails the build on .keys( in application code under our Redis client wrapper, with a message pointing to the sorted set and SCAN guidance.

4. Security implications. KEYS was also an information disclosure risk: the admin endpoint returned key names that included user IDs to anyone with admin dashboard access. The new count returns only a number. And ACLs, now in place for every application user, also restrict each service to its own key prefixes (~session:* for the auth service), which limits what a compromised service can read.

Takeaways

• Redis runs commands one at a time. An O(N) command over the keyspace stalls every client.
• KEYS is O(total keys) regardless of the pattern. Never use it in application code.
• Maintain answers in data structures (sets, sorted sets, counters) instead of searching for them.
• Use SCAN for background iteration, and block dangerous commands with ACLs.

References: Redis KEYS, Redis SCAN, Redis SLOWLOG, Redis ACL.