What is Retry pattern and how to use it correctly

The Retry Pattern is an architectural resilience mechanism that automatically re-executes a failed remote operation, assuming that the failure was caused by a transient, self-re...

🟢 Junior Level

The Retry Pattern is an architectural resilience mechanism that automatically re-executes a failed remote operation, assuming that the failure was caused by a transient, self-resolving glitch (e.g., momentary packet loss, cloud router rebalancing, or brief thread saturation):

Transient Network Blip (Transient Fault):
[Client] ──1. Request──► [Payment Service] ──(Temporary network glitch: Connection Timeout ❌)
[Client] ──(Pause 500ms Backoff)
[Client] ──2. Retry────► [Payment Service] ──(Network recovered: HTTP 200 OK ✅)

When Retry is Appropriate:

Temporary Network Drops: Packet loss, TCP socket timeouts, or transient DNS blips.
Rolling Deployments: A pod behind a load balancer is restarting; the subsequent retry is transparently rerouted to a healthy instance.
Transient Concurrency Collisions: Optimistic locking collisions in the database (OptimisticLockException).

The Double-Edged Sword:

Retry is a dangerous pattern if applied naively. If a downstream service is struggling under heavy load, thousands of clients retrying failed requests simultaneously will multiply inbound traffic $3\times$ or $4\times$, triggering a Retry Storm that drives the struggling service into total collapse.

🟡 Middle Level

1. The Golden Rule: Mandatory Idempotency

CRITICAL ARCHITECTURAL AXIOM: Never apply the Retry pattern to non-idempotent operations without an explicit uniqueness guarantee!

Safe to Retry (Idempotent): GET /api/v1/orders/10 or PUT /api/v1/users/42/status (read operations and state-replacement updates).
Extremely Dangerous to Retry (Non-Idempotent): POST /api/v1/payments/charge. If a socket timeout occurs, the client cannot know whether:
1. The packet was lost before reaching the bank (no charge occurred).
2. The bank processed the charge, but the response packet was lost on the return path.

If the client retries blindly without an Idempotency-Key, the customer’s credit card will be charged twice:

@Service
@RequiredArgsConstructor
public class PaymentClient {

    private final RestClient restClient;
    private final RetryRegistry retryRegistry;

    public PaymentResponse chargeWithRetry(PaymentRequest request) {
        // Unique idempotency key shared across all retry attempts for this business transaction
        String idempotencyKey = UUID.randomUUID().toString();

        return retryRegistry.retry("paymentRetry").executeSupplier(() ->
            restClient.post()
                .uri("/api/v1/payments/charge")
                .header("X-Idempotency-Key", idempotencyKey)
                .body(request)
                .retrieve()
                .body(PaymentResponse.class)
        );
    }
}

2. Error Classification: Retryable vs Non-Retryable

Error Category	Exceptions / HTTP Codes	Retry?	Technical Rationale
Network & Socket Drops	`ConnectException`, `SocketTimeoutException`	YES	Temporary routing blip or brief packet drop; likely to succeed on next try.
Server Overload	`HTTP 503 Service Unavailable`, `HTTP 429 Too Many Requests`	YES (with Backoff)	Downstream is temporarily saturated; retry after pause (respect `Retry-After`).
Client Syntax Errors	`HTTP 400 Bad Request`, `HTTP 404 Not Found`, `HTTP 422`	NO ❌	Deterministic error. A malformed payload will never succeed upon retry.
Security Failures	`HTTP 401 Unauthorized`, `HTTP 403 Forbidden`	NO ❌	Expired tokens or missing RBAC permissions cannot be healed by retrying.
Domain Business Errors	`InsufficientFundsException`, `ProductOutOfStockException`	NO ❌	Business rule violation. An empty bank account will not have money in 500 ms.

3. Production Configuration with Resilience4j

# application.yml: Resilience4j Retry Configuration
resilience4j:
  retry:
    instances:
      inventoryRetry:
        max-attempts: 3
        wait-duration: 500ms
        enable-exponential-backoff: true
        exponential-backoff-multiplier: 2
        enable-random-wait: true
        randomized-wait-factor: 0.5      # Full jitter: wait duration randomized between 50% and 150%
        retry-exceptions:
          - java.io.IOException
          - java.net.SocketTimeoutException
          - org.springframework.web.client.HttpServerErrorException.ServiceUnavailable
        ignore-exceptions:
          - org.springframework.web.client.HttpClientErrorException.BadRequest
          - org.springframework.web.client.HttpClientErrorException.NotFound

🔴 Senior Level

1. The Retry Storm Disaster and Retry Budgets

When a downstream database slows down, thousands of client requests fail simultaneously. If each client executes 3 immediate retries, traffic explodes:

Normal Operation: 1,000 RPS ──► [Order Service] (Healthy)
Outage Ingestion: 1,000 RPS ──(Initial Fail)──► [Each client executes 3 retries!]
Resulting Load:   4,000 RPS ──► [Order Service] (Total server collapse 💥)

Enterprise Defenses Against Retry Storms:

Exponential Backoff with Full Jitter: Instead of retrying on synchronized ticks ($1.0\text{s}, 2.0\text{s}$), randomize intervals ($t = \text{random}(0, \text{base} \times 2^{\text{attempt}})$) to disperse traffic peaks across a broad time window.
Retry Budgets (Envoy / Finagle): Client-side load balancers track request metrics across a sliding window. Retries are permitted only if they account for less than 10% of total outbound requests. If overall error rates exceed 10%, retries are disabled globally, converting the client into immediate Fail-Fast mode.
Circuit Breaker Coordination: When a downstream service enters an OPEN circuit breaker state, all calls fail fast in 0 ms, preventing the retry pipeline from ever executing.

2. End-to-End Deadline Propagation

A critical production trap is Useless Retries that outlive the client’s end-to-end deadline:

An API Gateway enforces an end-to-end client timeout of 3.0 seconds.
Microservice C fails on attempt #1 and schedules attempt #2 with an exponential backoff pause of 4.0 seconds.
While Microservice C is sleeping, the API Gateway terminates the client connection and returns HTTP 504 Gateway Timeout.
Microservice C eventually wakes up, retries, and executes heavy database mutations—wasting compute, network, and lock resources for an operation whose caller has already disconnected!

[Mobile Client] ──► [API Gateway: 3s Deadline] ──► [Service A] ──► [Service B] ──► [Service C]
                            │                                                              │
                     (3.0s Deadline Expires)                                         (Sleeps 4.0s for Retry)
                            ▼                                                              ▼
                  [Returns HTTP 504 to User]                                       [Executes useless retry 💥]

Solution: Deadline Propagation (gRPC Deadlines or HTTP X-Request-Deadline headers). Every downstream service calculates remaining deadline time: $\text{Remaining Time} = \text{Deadline} - \text{Current Time}$ If $\text{Remaining Time} \le \text{Backoff Duration}$, the retry is aborted immediately.

4 Tricky Questions

1. Why does Spring Retry’s `@Retryable` fail to work when invoked from another method within the same class (`this.method()`)?

Answer: Spring Retry relies on Spring AOP Dynamic Proxies. When another bean calls a method via dependency injection, it invokes the method on the generated CGLIB/JDK dynamic proxy, which intercepts the call and applies retry logic. If method methodA() calls methodB() within the same class instance (this.methodB()), the invocation bypasses the Spring proxy completely and executes directly on the underlying target instance. Consequently, @Retryable and @Recover annotations on methodB() are completely ignored. Solution: Move methodB() to a separate Spring @Service component, or inject self-referential beans via ApplicationContext.

2. What catastrophic bug occurs when `@Transactional` is placed outside `@Retryable`?

Answer: If @Transactional wraps @Retryable:

// CRITICAL BUG!
@Transactional
@Retryable(retryFor = TransientDataAccessException.class)
public void processOrder() { ... }

When the first attempt throws a database exception, Spring’s transaction manager marks the underlying Hibernate session / JDBC transaction as Rollback-Only, and the EntityManager enters a corrupted state. When the subsequent retry executes, it runs inside the same tainted transaction. When the method attempts to commit, Spring throws UnexpectedRollbackException: Transaction silently rolled back because it has been marked as rollback-only, causing the retry to fail even if the database issue was resolved! Rule: @Retryable must always be on the outside of @Transactional. Each retry attempt must open a completely fresh database transaction (BEGIN … COMMIT):

// CORRECT ORDER
@Retryable(retryFor = TransientDataAccessException.class)
@Transactional(propagation = Propagation.REQUIRES_NEW)
public void processOrder() { ... }

3. How must an enterprise HTTP client handle an `HTTP 429 Too Many Requests` response during retries?

Answer: A client must not apply naive exponential backoff to HTTP 429. The upstream server or API Gateway returns a specific instruction indicating when it will accept new traffic via the Retry-After HTTP header:

Retry-After: 120 (delay in seconds).
Retry-After: Wed, 21 Oct 2026 07:28:00 GMT (specific HTTP-date timestamp).

The client must parse the Retry-After header and suspend the retrying worker thread until that exact timestamp has elapsed. If the client retries earlier using local backoff timers, upstream security firewalls (Cloudflare, AWS WAF) will flag the IP address as malicious and impose an extended IP ban.

4. What is the difference between Full Jitter and Equal Jitter (based on AWS Architecture research)?

Answer: Given base delay $T$ and attempt number $k$, where $V = T \times 2^k$:

No Jitter: $\text{delay} = V$ (all retrying clients wake up simultaneously, generating massive synchronized traffic spikes).
Equal Jitter: $\text{delay} = \frac{V}{2} + \text{random}\left(0, \frac{V}{2}\right)$ (keeps half the interval deterministic while randomizing the remaining half; dampens spike amplitude but maintains rhythmic waves).
Full Jitter: $\text{delay} = \text{random}(0, V)$ (the delay is fully randomized between 0 and the exponential maximum). AWS Architecture research proved that Full Jitter minimizes total completion time and maximizes client dispersion, eliminating resonant spikes and allowing struggling servers to recover the fastest.

🎯 Interview Cheat Sheet

Must Know:

Retry: Automatically re-executes calls that failed due to temporary, transient faults (connection drops, 503, 429).
Mandatory Idempotency: Never retry mutating requests (POST) without an Idempotency-Key to prevent duplicate billing or record duplication.
Never Retry: Client errors (4xx), security errors (401/403), or domain business violations (insufficient balance).
Retry Storms: Unconstrained retries multiply failure traffic. Prevent them using Exponential Backoff, Full Jitter, and Retry Budgets (max 10% of total traffic).
Spring AOP Rule: @Retryable must wrap outside @Transactional, never inside, to ensure clean database transactions per attempt.
Deadline Propagation: Abort pending retries if remaining gateway deadline is shorter than backoff duration.

Common Follow-up Questions:

“What is a Retry Budget?” → A threshold (typically 10%) enforced by client proxies like Envoy that blocks retries if total retries exceed 10% of normal traffic, preventing self-inflicted DDoS.
“How many retry attempts are standard?” → 3 attempts maximum (1 initial + 2 retries). Anything beyond 3 attempts rarely succeeds and drastically inflates latency.

Red Flags (DO NOT Say):

❌ “Retry every failed request 10 times immediately without any backoff.” (Triggers instant server collapse via Retry Storms).
❌ “Retry HTTP 404 Not Found because the server might find the resource on the second try.” (404 is a deterministic client error).
❌ “Place @Transactional outside @Retryable so all retries share the same transaction.” (Corrupts the transaction with Rollback-Only errors).

What is exponential backoff — Backoff algorithms and jitter math
How to ensure fault tolerance of microservices — Resilience defense pipeline
What is Circuit Breaker Pattern — Fast-failure protection
What are Compensating Transactions — Saga rollback mechanisms