EVM bot transactions fail with nonce too low and replacement transaction underpriced

Our settlement bot moves funds between our own contracts and wallets on an EVM chain, several transactions per minute from one hot address. To speed up a backlog, we ran four async workers. Errors followed immediately:

Error: nonce too low: next nonce 4812, tx nonce 4810
Error: replacement transaction underpriced

Worse, a few transactions sat pending for hours, and every later transaction from the address stayed pending behind them.

Reproducing it

Each worker did:

const nonce = await provider.getTransactionCount(wallet.address, "pending");
await wallet.sendTransaction({ ...tx, nonce });

Two workers read the same pending count at the same moment and built transactions with the same nonce. The node accepted the first; the second was treated as a replacement for it, which requires a sufficiently higher fee, so it failed with "replacement transaction underpriced". If one of them was already mined by the time the other arrived, the node reported "nonce too low".

The stuck transactions were sent with a fee that became too low when base fees rose; since nonces must be used in order, everything after them waited.

What we ruled out

• Node problems. Two different providers gave the same errors.
• Insufficient balance. Balance errors have different messages.

Root cause

Every transaction from an externally owned account carries a nonce that must equal the account's next nonce: unique, sequential and without gaps. Reading the pending count from the node in multiple concurrent workers is a race. Gaps or underpriced transactions block every later nonce from the same address. Our workers had no shared nonce source and no logic to replace stuck transactions.

What is happening under the hood

• The chain processes an account's transactions strictly in nonce order. Nonce 4811 cannot be mined before 4810.
• A node's "pending" count includes transactions in its own mempool, which differs between nodes and changes from moment to moment.
• Sending a new transaction with a nonce already in the mempool is a replacement. Nodes accept replacements only when fees increase by a minimum percentage (commonly 10 percent for both max fee and priority fee in geth-based clients).
• A transaction with a fee too low for current conditions can stay pending a long time and block everything behind it.

Environment

Minimal reproduction

Start two tasks that read getTransactionCount(address, "pending") and send at the same time. One fails with "replacement transaction underpriced" or "nonce too low".

The fix

1. One nonce allocator per address. All sends from an address go through a single in-process allocator backed by a durable store, initialised from the chain on startup:

class NonceAllocator {
  private next?: number;
  private queue = Promise.resolve();

  constructor(private provider: Provider, private address: string, private store: NonceStore) {}

  async allocate(): Promise<number> {
    const run = async () => {
      if (this.next === undefined) {
        const chain = await this.provider.getTransactionCount(this.address, "pending");
        this.next = Math.max(chain, await this.store.lastUsed(this.address) + 1);
      }
      const n = this.next++;
      await this.store.markUsed(this.address, n);
      return n;
    };
    const p = this.queue.then(run);
    this.queue = p.then(() => undefined, () => undefined);
    return p;
  }
}

Workers can prepare transactions in parallel, but nonces are assigned one at a time. Running multiple bot instances against the same address is forbidden; a lease in the coordination store enforces a single active sender.

2. Track every sent transaction and replace stuck ones. A monitor checks pending transactions every block. If a transaction has not been mined after N blocks, it is re-sent with the same nonce and fees increased by at least the replacement threshold:

function bumped(fee: { maxFeePerGas: bigint; maxPriorityFeePerGas: bigint }) {
  const bump = (x: bigint) => (x * 1125n) / 1000n;        // +12.5%, above the 10% rule
  return { maxFeePerGas: bump(fee.maxFeePerGas), maxPriorityFeePerGas: bump(fee.maxPriorityFeePerGas) };
}

If a transaction is no longer needed, it is replaced with a zero-value transfer to ourselves at the same nonce, which clears the slot.

3. Fees from current conditions, with maxFeePerGas set to a margin above the current base fee so a rising base fee does not strand transactions.

4. Handle "nonce too low" as information. It means the nonce was already used, possibly by our own earlier send. The allocator re-syncs from the chain and the transaction tracker checks whether the intended transaction was in fact mined.

Alternatives and trade-offs

• Multiple hot addresses, one per worker. Parallelism without nonce coordination; more keys to secure and fund.
• Smart contract wallets with batching. Many operations in one transaction; different security and gas trade-offs.
• ethers' built-in NonceManager. Fine for a single process and sequential use; we needed persistence and replacement logic.

How to test the fix

A local devnet test sends 1,000 transactions from four workers through the allocator: all nonces unique and contiguous, all mined. A second test pauses mining, raises base fee, and checks that stuck transactions are replaced with bumped fees and mined when mining resumes.

Confirming in production

• No "nonce too low" or "replacement underpriced" errors from normal operation over two months.
• Stuck transactions are replaced within 3 blocks; the longest wait since the fix was under 2 minutes.

Regression guard

• Alerts on pending transactions older than 10 blocks and on gaps between the allocator's next nonce and the chain's.
• The single-sender lease, checked at startup and renewed continuously.

Security implications

Hot wallet keys live in a hardware-backed signing service with policies on allowed destinations and amounts. Stuck transactions are a security concern too: a transaction you no longer want can still be mined later if not replaced, so cancellation by replacement is part of the runbook. Never broadcast the same signed transaction through untrusted relays you do not control.

Prevention checklist

• Allocate nonces from one place per address.
• Never let multiple processes send from the same address without coordination.
• Track sent transactions and replace stuck ones with sufficiently higher fees.
• Set max fees with headroom over the current base fee.
• Treat "nonce too low" as a signal to re-sync and check what was mined.

References

• Ethereum transactions and nonces — https://ethereum.org/en/developers/docs/transactions/
• EIP-1559 fee market — https://eips.ethereum.org/EIPS/eip-1559
• go-ethereum transaction pool (replacement rules, price bump) — https://geth.ethereum.org/docs/fundamentals/command-line-options
• ethers v6 documentation — https://docs.ethers.org/v6/