Remigiusz ZalewskiRemigiusz Zalewski

Fault handling in .NET with Polly

fault-handling-dotnet-polly
Sponsor this newsletter →

Introduction

Calling another service over a network means accepting that the call sometimes fails for reasons that have nothing to do with your code - a dropped packet, a downstream service mid-deploy, a momentary DNS hiccup. Polly gives .NET a vocabulary for handling exactly this class of transient fault: retry, circuit breaker, and timeout policies that wrap any call and decide how to react when it fails.

🎬 Watch the full video here:

Retry: the obvious first move, done carefully

var retryPolicy = Policy
    .Handle<HttpRequestException>()
    .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

WaitAndRetryAsync with an exponential backoff (1s, 2s, 4s) is deliberate - retrying instantly, three times in a row, against a service that's already struggling under load doesn't help it recover, it adds more load right when it can least afford it. The backoff gives the downstream service breathing room between attempts.

Circuit breaker: stop calling a service that's clearly down

Retrying is the wrong response once a service has been failing for a while - at that point, every retry is wasted latency and additional load on something that isn't coming back in the next few seconds. A circuit breaker tracks failures and, past a threshold, stops calling the downstream service entirely for a cooldown period:

var circuitBreakerPolicy = Policy
    .Handle<HttpRequestException>()
    .CircuitBreakerAsync(exceptionsAllowedBeforeBreaking: 5, durationOfBreak: TimeSpan.FromSeconds(30));

After 5 consecutive failures, the circuit "opens" and every call for the next 30 seconds fails immediately without even attempting the network call - fast, predictable failure instead of every caller waiting out a slow timeout against a service that's clearly not responding.

Timeout: don't wait forever for a hung call

var timeoutPolicy = Policy.TimeoutAsync(TimeSpan.FromSeconds(10));

Without an explicit timeout, a hung downstream call can block a request thread far longer than any reasonable UX tolerates - a timeout policy guarantees an upper bound.

Combining policies with PolicyWrap

Real usage rarely means picking just one - retry, circuit breaker, and timeout typically wrap each other, applied in a deliberate order:

var resiliencePolicy = Policy.WrapAsync(retryPolicy, circuitBreakerPolicy, timeoutPolicy);

The order matters: timeout should be the innermost policy (bounding each individual attempt), circuit breaker next (stopping retries once the service is clearly down), and retry outermost (deciding whether to try again at all).

Wiring it into HttpClient via IHttpClientFactory

Rather than applying policies manually at every call site, IHttpClientFactory's AddPolicyHandler attaches them to a named or typed client once, and every call through it is covered automatically:

builder.Services.AddHttpClient<OrdersApiClient>()
    .AddPolicyHandler(retryPolicy)
    .AddPolicyHandler(circuitBreakerPolicy);

Common pitfalls

  • Retrying without backoff, hammering an already-struggling service harder.
  • Retrying non-idempotent operations (a POST that creates a resource) without considering that a retry after a timeout might duplicate the effect if the original call actually succeeded server-side.
  • No circuit breaker at all, so every caller keeps paying full retry latency against a service that's been down for minutes.
  • Setting the circuit breaker's failure threshold too low, tripping it on ordinary transient blips instead of genuine outages.
  • Applying policies in the wrong order in a PolicyWrap, so timeout doesn't actually bound what you think it bounds.

Key Takeaways

  • Retry with exponential backoff handles genuinely transient faults without adding load to an already-struggling service.
  • A circuit breaker stops calling a service that's clearly down, failing fast instead of making every caller wait out a timeout.
  • An explicit timeout policy bounds how long any single attempt can hang.
  • PolicyWrap combines policies in a deliberate order - timeout innermost, circuit breaker next, retry outermost.
  • IHttpClientFactory.AddPolicyHandler applies resilience policies to every call through a client automatically.

Get the Full Source Code

The complete runnable solution - the combined retry/circuit-breaker/timeout policy wrap and its HttpClientFactory registration - is available to Patreon supporters. If you want the resilience setup ready to apply to your own HttpClient instead of rebuilding it from the walkthrough above, you can find it on Patreon.

Resources