Our circuit breaker never opened, because timeouts did not count as failures

Our product page calls a recommendation service to show "customers also bought". It is a nice-to-have: if recommendations fail, the page renders without them. The call is wrapped in a Resilience4j circuit breaker, with a fallback that returns an empty list.

One afternoon the recommendation service's model server degraded and responses slowed from about 40 ms to 6 to 9 seconds. The product page's p95 latency went from 300 ms to over 8 seconds. The circuit breaker, whose entire purpose was to stop calling a failing dependency, stayed closed for the 50 minutes it took the recommendations team to roll back.

The breaker's metrics, exported through Micrometer, during the incident:

resilience4j_circuitbreaker_state{name="recommendations",state="closed"}            1
resilience4j_circuitbreaker_failure_rate{name="recommendations"}                    0.0
resilience4j_circuitbreaker_calls_seconds_count{kind="successful"}             246,125
resilience4j_circuitbreaker_calls_seconds_count{kind="failed"}                       0
resilience4j_circuitbreaker_calls_seconds_count{kind="ignored"}                      0
resilience4j_circuitbreaker_slow_call_rate{name="recommendations"}                -1.0

Zero failed calls. Every call, including the ones that timed out, was counted as successful. Failure rate 0 percent. Slow-call rate -1, meaning not enough data or not configured. From the breaker's point of view, nothing was wrong.

The configuration

resilience4j:
  circuitbreaker:
    instances:
      recommendations:
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 50
        failureRateThreshold: 50
        waitDurationInOpenState: 30s
        recordExceptions:
          - java.io.IOException
          - org.springframework.web.client.HttpServerErrorException
        ignoreExceptions:
          - org.springframework.web.client.HttpClientErrorException
  timelimiter:
    instances:
      recommendations:
        timeoutDuration: 2s

And the call site:

@CircuitBreaker(name = "recommendations", fallbackMethod = "noRecommendations")
@TimeLimiter(name = "recommendations")
public CompletableFuture<List<Product>> recommendationsFor(String productId) {
    return CompletableFuture.supplyAsync(() -> client.fetch(productId), executor);
}

Three things combined:

1. recordExceptions was a whitelist. When it is set, only those exceptions (and subclasses) count as failures. The Resilience4j documentation is explicit that all other exceptions then count as a success, unless they are listed in ignoreExceptions.
2. The timeout exception was java.util.concurrent.TimeoutException, thrown by the TimeLimiter. It is not an IOException and not an HttpServerErrorException, so it was not recorded.
3. Slow-call detection was off. slowCallRateThreshold defaults to 100 percent, so the breaker only opens on slow calls when every call is slow, and slowCallDurationThreshold defaults to 60 seconds, far above our 2 second timeout. Calls that took 1.9 seconds (just under the TimeLimiter) were counted as successful and fast.

Many calls finished in just under 2 seconds, were "successful", and still made every product page wait.

What a breaker is supposed to do

A circuit breaker watches the outcomes of calls to a dependency. When too many fail, it opens: further calls fail immediately, without touching the dependency, and the fallback runs. After a wait, it goes half-open, lets a few trial calls through, and closes again if they succeed. This protects callers from waiting on a sick dependency, and protects the dependency from load while it recovers.

It only works if the breaker's definition of "failure" matches what hurts you. For an optional dependency on a latency-sensitive page, a slow response is as bad as an error. Our breaker only knew about errors, and only some kinds.

Why timeouts slipped through

Exceptions from a timeout depend on where the timeout happens: SocketTimeoutException (an IOException) from the HTTP client's read timeout, TimeoutException from a CompletableFuture or the TimeLimiter, ResourceAccessException wrapping the socket timeout in Spring's RestTemplate. Our HTTP client had a read timeout of 10 seconds, longer than the TimeLimiter's 2 seconds, so the TimeLimiter always fired first, with an exception type nobody had added to the list.

The configuration had been copied from another service that called its dependency synchronously without a TimeLimiter, where timeouts surfaced as IOException. It looked correct and was tested only for HTTP 500s.

Order of decorators

With annotations, Resilience4j applies aspects in a configurable order. The TimeLimiter must sit inside the circuit breaker so the breaker sees the TimeoutException. We confirmed ours did, so the exception reached the breaker, which then classified it as not recorded.

Environment

1. Record every exception except the ones that are clearly the caller's fault

We inverted the logic: record all exceptions by default, and ignore only client errors that the dependency is right to return:

resilience4j:
  circuitbreaker:
    instances:
      recommendations:
        slidingWindowType: TIME_BASED
        slidingWindowSize: 20              # seconds
        minimumNumberOfCalls: 20
        failureRateThreshold: 50
        slowCallDurationThreshold: 500ms
        slowCallRateThreshold: 50
        waitDurationInOpenState: 15s
        permittedNumberOfCallsInHalfOpenState: 5
        automaticTransitionFromOpenToHalfOpenEnabled: true
        ignoreExceptions:
          - org.springframework.web.client.HttpClientErrorException
  timelimiter:
    instances:
      recommendations:
        timeoutDuration: 800ms
        cancelRunningFuture: true

Without recordExceptions, Resilience4j records every exception as a failure unless it matches ignoreExceptions. A TimeoutException, a connection reset, an unexpected NullPointerException in the client: all count.

2. Treat slow calls as failures

slowCallDurationThreshold: 500ms and slowCallRateThreshold: 50 mean the breaker opens when at least half the calls in the window take longer than 500 ms, even if they eventually succeed. For a 40 ms dependency, 500 ms is already badly degraded.

3. Make the timeouts consistent

The TimeLimiter is now 800 ms, the HTTP client's read timeout 1 second (slightly longer, so the TimeLimiter is the one that fires and the exception type is predictable), and cancelRunningFuture interrupts the work instead of leaving it running in the background.

4. Use a time-based window

A count-based window of 50 calls fills in a fraction of a second under heavy traffic and can flap open and closed. A 20 second time-based window with a minimum of 20 calls gives steadier decisions and still reacts within seconds.

1. A test for each failure type. We use WireMock to simulate the dependency and assert on the breaker's state, not just on the fallback's output:

@Test
void opensOnTimeouts() {
    wiremock.stubFor(get(urlPathMatching("/recommendations/.*"))
        .willReturn(ok().withFixedDelay(3_000)));

    for (int i = 0; i < 30; i++) {
        service.recommendationsFor("p-" + i).join();     // fallback returns empty list
    }

    assertThat(registry.circuitBreaker("recommendations").getState())
        .isEqualTo(CircuitBreaker.State.OPEN);
}

Separate tests cover slow-but-successful responses (600 ms delay, must open), HTTP 500s (must open), HTTP 404s (must stay closed) and recovery through half-open after the stub returns to normal. On the old configuration, the timeout and slow-response tests failed: the breaker stayed closed.

2. Fault injection in staging. We added 5 seconds of latency to the recommendation service with a proxy. The breaker opened about 3 seconds after the latency started. Product page p95 rose to 900 ms during those seconds, then returned to 310 ms with recommendations hidden. When the latency was removed, the breaker went half-open after 15 seconds, the five trial calls succeeded, and it closed.

3. Dashboards and alerts. The breaker's state, failure rate, slow-call rate and the count of not_permitted calls are on the product page dashboard. An alert fires when any breaker has been open for more than five minutes, so a dependency problem is visible even though users are shielded from it.

4. Library-wide defaults. Our internal starter now provides a default breaker configuration with slow-call detection on and no recordExceptions whitelist. Services can override it, and a review rule asks for a test of timeouts whenever someone does.

5. Security implications. A breaker that never opens also gives an attacker a lever: slowing one optional dependency (for example, by flooding it) slows every page that calls it. With slow-call detection, degrading the dependency now only removes a feature.

Takeaways

• A circuit breaker is only as good as its definition of failure. Make sure timeouts are recorded.
• Avoid recordExceptions whitelists; record everything and ignore only the caller's own errors.
• Turn on slow-call detection with a threshold that reflects what "too slow" means for the caller.
• Test the breaker's state transitions for timeouts, slow responses, errors and recovery.

References: Resilience4j CircuitBreaker, Resilience4j TimeLimiter, Resilience4j Spring Boot aspect order, WireMock simulating faults.