Kraken API EAPI:Invalid nonce when a second worker starts with the same key

Our order router sends orders and queries balances on Kraken through the private REST API. To handle more strategies, we scaled the router Deployment from one to two replicas. Immediately, roughly a third of private calls failed:

{"error": ["EAPI:Invalid nonce"], "result": {}}

Failures were random across both replicas, and retries sometimes failed too.

Reproducing it

Our nonce generator:

def nonce() -> str:
    return str(int(time.time() * 1000))     # milliseconds since epoch

Two processes calling private endpoints a few times per second each produced overlapping and interleaved nonces:

replica A: 1747212000101, 1747212000350, 1747212000352
replica B: 1747212000349, 1747212000351, 1747212000600

Kraken received ...350 from A and then ...349 from B, which was lower than the last nonce seen for the key, and rejected B's request.

What we ruled out

• Clock skew between pods. Both nodes were synchronised within a millisecond; the issue occurs even with perfect clocks.
• Signature errors. Those return EAPI:Invalid key or EAPI:Invalid signature.

Root cause

Kraken requires the nonce on private requests to be strictly increasing for each API key. With one process and millisecond timestamps, requests were serial and nonces increased naturally. With two processes sharing one key, requests arrived in an order that did not match their nonce order, and the exchange rejected any request whose nonce was not higher than the previous one it had processed. Network jitter made arrival order unpredictable, so failures looked random.

What is happening under the hood

A nonce prevents replay: an attacker who captures a signed request cannot send it again, because its nonce has been used. Exchanges that use incrementing nonces (Kraken, and others with similar schemes) track the highest nonce per key. Kraken's API key settings include a nonce window option, which allows some tolerance for out-of-order nonces; it reduces errors from small reorderings but does not make sharing one key across independent processes reliable.

Other exchanges use a timestamp plus a receive window instead (for example Binance), which tolerates concurrency but requires accurate clocks.

Environment

Minimal reproduction

Run two processes with the same key, each sending Balance requests in a loop with millisecond nonces. A share of requests will fail with EAPI:Invalid nonce.

The fix

1. One API key per worker. Each replica gets its own key, with the same permissions and IP allowlist. The router is a StatefulSet so each pod has a stable identity, and pod router-0 loads key 0, router-1 loads key 1, from the secret manager.

ordinal = int(os.environ["POD_NAME"].rsplit("-", 1)[1])
api_key, api_secret = load_exchange_key(f"kraken/router/{ordinal}")

2. Monotonic nonces within a process. Even with one process, concurrent asyncio tasks can produce equal timestamps. The nonce source guarantees strictly increasing values and uses microseconds:

import threading, time

class NonceSource:
    def __init__(self):
        self._last = 0
        self._lock = threading.Lock()

    def next(self) -> str:
        with self._lock:
            n = max(int(time.time() * 1_000_000), self._last + 1)
            self._last = n
            return str(n)

3. Serialise private calls per key. The HTTP client for private endpoints sends requests for one key through a single queue, so a request with a higher nonce cannot overtake one with a lower nonce inside our process. Throughput per key was not a constraint for us.

Alternatives and trade-offs

• A shared nonce service (for example Redis INCR) for one key used by many workers. Guarantees unique, increasing nonces at generation time, but requests can still arrive out of order over the network; combine with a nonce window.
• Increase the nonce window on the key. Helps with small reordering, does not fix the design.
• Single writer. Route all private calls through one process. Simple, a single point of failure.

How to test the fix

A load test with both replicas sending 10 private requests per second each for an hour produced zero nonce errors. A unit test runs 1,000 concurrent next() calls and asserts all nonces are unique and increasing in generation order.

Confirming in production

• EAPI:Invalid nonce errors dropped to zero after the change.
• Each key's activity is visible separately in the exchange's API key list, which made reconciliation easier.

Regression guard

• The router refuses to start if two pods resolve to the same key id (checked through a lease in our coordination store).
• Alert on any EAPI: error class above a small rate.

What we would do differently

Scaling a component that talks to an exchange should include a review of every per-key or per-account mechanism: nonces, rate limits, order count limits and stream subscriptions. Each of these can behave differently when two processes share one identity. We added a short "scaling checklist" to our runbook covering exactly these points, and the router's README now states that each replica needs its own key. Before scaling any exchange-facing service, someone reads that checklist and confirms each item, which takes ten minutes and would have prevented this incident.

Security implications

Nonces are a replay protection mechanism, so do not weaken them unnecessarily: keep the nonce window as small as your setup allows. One key per worker also helps containment: a compromised pod's key can be revoked without affecting others. All router keys have trading permissions only, no withdrawals, and are restricted to the cluster's egress IPs.

Prevention checklist

• Do not share an incrementing-nonce API key across processes.
• Use a monotonic nonce generator even within one process.
• Serialise private requests per key.
• Prefer one key per worker for isolation and clarity.
• Keep nonce tolerance settings as tight as possible.

References

• Kraken REST API authentication and nonce — https://docs.kraken.com/api/docs/guides/spot-rest-auth
• Kraken API documentation home — https://docs.kraken.com/api/
• Kraken support centre (API key settings, nonce window) — https://support.kraken.com/