One malformed message blocked our order queue for four hours

On a Tuesday morning the warehouse team noticed that no new orders had arrived in their picking system since about 05:40. The web shop was taking orders normally. Payments were captured. The orders just were not reaching fulfilment.

Orders flow from the shop to fulfilment through an Amazon SQS FIFO queue, orders-to-fulfilment.fifo, consumed by a small worker service. The queue metrics showed the problem immediately:

ApproximateNumberOfMessagesVisible      8,412 (and rising)
ApproximateNumberOfMessagesNotVisible   1
ApproximateAgeOfOldestMessage           14,630 seconds
NumberOfMessagesDeleted (last 4h)       0

One message in flight, eight thousand waiting, the oldest four hours old, nothing deleted. The worker was running and healthy according to its health check, and its logs repeated the same lines every 30 seconds:

05:41:02 INFO  received message group=shop-eu id=6f1a... order=ORD-2026-0918822
05:41:02 ERROR failed to process order ORD-2026-0918822
TypeError: Cannot read properties of null (reading 'countryCode')
    at mapShippingAddress (src/mapper.ts:44:31)
    at handle (src/worker.ts:28:18)
05:41:32 INFO  received message group=shop-eu id=6f1a... order=ORD-2026-0918822
05:41:32 ERROR failed to process order ORD-2026-0918822
TypeError: Cannot read properties of null (reading 'countryCode')
...

The same order, every 30 seconds, for four hours.

The message

{
  "orderId": "ORD-2026-0918822",
  "items": [{ "sku": "GC-50", "qty": 1 }],
  "shippingAddress": null,
  "billingAddress": { "countryCode": "DE", "...": "..." }
}

A digital gift card order. A new product type launched the day before, with no physical shipping and therefore no shipping address. The shop sent null; the worker assumed every order had one.

What the queue did

SQS delivers a message and hides it from other consumers for the visibility timeout (30 seconds here). If the consumer deletes the message within that time, it is gone. If not, because the consumer crashed, threw, or just did not delete it, the message becomes visible again and is delivered again. That is the at-least-once guarantee working exactly as designed: SQS cannot know whether the failure was temporary.

Without a limit, it will retry forever. SQS offers that limit through a redrive policy: after maxReceiveCount receives, the message moves to a dead-letter queue. Our queue had no redrive policy. It had been created by hand during a migration, and the infrastructure template that would have added one was never applied to it.

Why one message blocked everything

On a standard SQS queue, a failing message would retry forever but other messages would keep flowing past it. Ours was a FIFO queue, with all orders from one shop in a single message group (shop-eu) so they are processed in order. FIFO queues guarantee that messages in the same group are delivered in order, and SQS will not deliver the next message in a group while an earlier one is still in flight or waiting to be retried. So the poison message did not just fail; it held up every order behind it in its group.

The health check lied

The worker's health check reported "up" because the process was alive and connected to SQS. It had no notion of whether messages were actually being processed. A process stuck in a loop of failures looked healthy.

Environment

1. Unblock immediately

We moved the poison message out by hand: received it, saved its body to an S3 bucket for analysis, and deleted it. The backlog began draining at once. Fulfilment caught up in about 40 minutes.

2. Add a dead-letter queue with a redrive policy

{
  "RedrivePolicy": {
    "deadLetterTargetArn": "arn:aws:sqs:eu-central-1:123456789012:orders-to-fulfilment-dlq.fifo",
    "maxReceiveCount": "5"
  }
}

The DLQ of a FIFO queue must also be FIFO. After five failed receives, SQS moves the message to the DLQ and the next message in the group is delivered. We chose five because our transient failures (a database failover, a downstream timeout) normally clear within one or two retries, and five receives at a 30 second timeout is under three minutes of delay for the rest of the group.

The policy now lives in Terraform with the queue, and a policy-as-code check fails any SQS queue in our accounts without a redrive policy.

3. Tell temporary failures from permanent ones

Not every error deserves retries. A malformed message will fail the same way five times. The worker now classifies errors:

async function handle(msg: Message): Promise<void> {
  let order: OrderEvent;
  try {
    order = parseOrderEvent(msg.Body);        // validates against the schema
  } catch (err) {
    await deadLetter(msg, "invalid_payload", err);   // permanent: send to DLQ now
    await deleteMessage(msg);
    return;
  }

  try {
    await fulfilment.createOrder(toFulfilment(order));
    await deleteMessage(msg);
  } catch (err) {
    if (isTransient(err)) throw err;          // let SQS retry via visibility timeout
    await deadLetter(msg, "processing_failed", err);
    await deleteMessage(msg);
  }
}

deadLetter writes the message, the error and context to the DLQ directly. Validation errors skip the five retries entirely. The redrive policy remains as a safety net for anything the worker does not classify, including crashes.

4. Fix the actual bug and the contract

The mapper now handles orders without a shipping address, sending digital orders to a separate fulfilment flow. More importantly, the shop and the worker now share a JSON Schema for order events, and the schema says shippingAddress is nullable. The shop's tests validate every event type against it, so a new product type that changes the shape is caught before release.

5. A redrive process

A DLQ that nobody looks at is a place where orders go to be forgotten. We added:

• An alarm when ApproximateNumberOfMessagesVisible on any DLQ is above zero.
• A runbook: inspect the message and error, fix the code or data, then redrive.
• Redrive with the SQS console or the StartMessageMoveTask API, which moves messages from the DLQ back to the source queue at a controlled rate once the fix is deployed.

1. Poison message test in staging. We published a malformed message followed by 100 valid ones:

aws sqs send-message --queue-url "$Q" --message-group-id shop-eu \
  --message-deduplication-id poison-1 --message-body '{"orderId":"X","shippingAddress":null,"items":"oops"}'

The malformed message went to the DLQ on its first receive (classified as invalid). All 100 valid messages were processed within seconds. With classification disabled, the message went to the DLQ after five receives, about two and a half minutes, and the rest followed.

2. Alarms. The DLQ alarm fired within a minute and paged the owning team. We also added an alarm on ApproximateAgeOfOldestMessage above five minutes on the main queue, which would have caught the original incident in five minutes instead of four hours.

3. Health check. The worker's health endpoint now reports unhealthy if it has received messages in the last five minutes but successfully processed none. That turns a silent failure loop into a visible problem.

4. Redrive tested. We fixed a deliberately broken mapper in staging, redrove the DLQ, and confirmed the messages were processed once and in order within their group.

5. Security and data implications. DLQ messages contain customer addresses, so the DLQ has the same encryption, retention (14 days) and access policy as the main queue. The S3 copies of poison messages expire after 30 days.

Takeaways

• At-least-once queues redeliver failed messages until something stops them. Always configure a DLQ.
• On FIFO queues, one failing message blocks its whole message group.
• Send obviously invalid messages to the DLQ immediately; retry only errors that can succeed later.
• Alarm on DLQ depth and on the age of the oldest message, and have a tested redrive procedure.

References: Amazon SQS dead-letter queues, SQS visibility timeout, FIFO queue message ordering, StartMessageMoveTask.