C# – Circuit breaker with Polly

In an electrical system, a circuit breaker detects electrical problems and opens the circuit, which blocks electricity from flowing. To get electricity flowing again, you have to close the circuit. The same approach can be implemented in software when you’re sending requests to an external service.

This is especially important when you’re sending lots of concurrent requests. Without the circuit breaker, you can quickly run into big problems (such as port exhaustion if you’re using HttpClient).

To implement the circuit breaker pattern, you have to detect error conditions that indicate the service is temporarily down and then trip the circuit. You have to keep the circuit open for a short period of time to block request attempts. Then you have to carefully determine when it’s safe to close the circuit to let requests go through again.

This is similar to the retry pattern. The difference is the circuit breaker pattern applies to all requests while retries apply to individual requests.

Just like with retries, you can use the Polly .NET library to implement the circuit breaker pattern. It abstracts away the details so you can focus on your own code. This simplifies things quite a bit. When you use Polly, you configure a policy object, and then use it to execute your code.

Here’s a simple example of configuring a Polly circuit breaker policy and executing code with it:

var circuitBreakerPolicy = Policy.Handle<TransientException>()
	.CircuitBreaker(exceptionsAllowedBeforeBreaking: 3, durationOfBreak: TimeSpan.FromSeconds(10));

while (true)
{
	try
	{
		circuitBreakerPolicy.Execute(() =>
		{
			SendRequest();
			Log("Successfully sent request");
		});
		return;
	}
	catch(BrokenCircuitException)
	{
		Log("The circuit breaker tripped and is temporarily disallowing requests. Will wait before trying again");
		await Task.Delay(TimeSpan.FromSeconds(15));
	}
	catch (TransientException)
	{
		Log("Transient exception while sending request. Will try again.");
	}
}
Code language: C# (cs)

This tells Polly to trip the circuit for 10 seconds when it sees three TransientExceptions in a row.

Running this code outputs the following:

11:52:36.66007  Transient exception while sending request. Will try again.
11:52:36.67443  Transient exception while sending request. Will try again.
11:52:36.67645  Transient exception while sending request. Will try again.
11:52:36.67716  The circuit breaker tripped and is temporarily disallowing requests. Will wait before trying again
11:52:51.70436  Successfully sent requestCode language: plaintext (plaintext)

The TransientException was thrown three times in a row, so it tripped the circuit and kept it open for 10 seconds. The fourth request was allowed through (because it was sent after the circuit was no longer open) and succeeded.

In this article, I’ll go into more details about how the Polly circuit breaker policy works. At the end, I’ll show a full example of using the Polly circuit breaker with HttpClient.

Note: For more advanced error detection that uses sampling, use the AdvancedCircuitBreaker policy.

Install Polly

If you haven’t already, install the Polly nuget package by executing this command (this is using View > Other Windows > Package Manager Console):

Install-Package Polly
Code language: PowerShell (powershell)

Circuit states

There are three main circuit states: Closed, Open, and Half-Open. These can be summarized in the following table:

StateWhat it means
ClosedThe circuit is allowing requests through.

Just like a closed circuit allows electricity to flow through.
OpenThe circuit tripped and isn’t allowing requests through right now.

Just like an open circuit prevents electricity from flowing through.
HalfOpenThe next request that comes through will be used to test the service, while all other requests will be rejected.

If the test request succeeds, the circuit will close. Otherwise it will open again for the configured duration.

Note: There’s another state called “Isolated”. It’s only used when you manually trip the circuit.

Log circuit state changes

You can log circuit state changes by using the onBreak, onReset, and onHalfOpen callbacks, like this:

var circuitBreakerPolicy = Policy.Handle<TransientException>()
	.CircuitBreaker(exceptionsAllowedBeforeBreaking: 3, durationOfBreak: TimeSpan.FromSeconds(10),
		onBreak: (_, duration) => Log($"Circuit open for duration {duration}"),
		onReset: () => Log("Circuit closed and is allowing requests through"),
		onHalfOpen: () => Log("Circuit is half-opened and will test the service with the next request"));
Code language: C# (cs)

Note: You can do anything in these callbacks, not just logging. I’m showing a logging example because this is a good way to learn about when these callbacks are fired.

Run the request in a loop, logging the circuit state before the request is attempted.

Log("Sending request");
Log($"CircuitState: {circuitBreakerPolicy.CircuitState}");
circuitBreakerPolicy.Execute(() =>
{

	SendRequest();
	Log("Successfully sent request");
});
Code language: C# (cs)

Note: For brevity, the error handling, additional logging, and delaying logic aren’t shown here.

The circuit is closed for the first three requests. The third request causes it to reach the error threshold and it trips the circuit. When this happens, the onBreak callback is executed:

01:48:00.74850  Sending request
01:48:00.76498  CircuitState: Closed
01:48:00.77115  Transient exception while sending request. Will try again.
01:48:00.77133  Sending request
01:48:00.77150  CircuitState: Closed
01:48:00.77171  Transient exception while sending request. Will try again.
01:48:00.77190  Sending request
01:48:00.77202  CircuitState: Closed
01:48:00.77463  onBreak: Circuit open for duration 00:00:10
01:48:00.77487  Transient exception while sending request. Will try again.
Code language: plaintext (plaintext)

The circuit is now open, and when the fourth request is executed, it throws a BrokenCircuitException:

01:48:00.77498  Sending request
01:48:00.77543  CircuitState: Open
01:48:00.77614  The circuit breaker tripped and is temporarily disallowing requests. Will wait before trying againCode language: plaintext (plaintext)

The circuit breaker was configured to be open for 10 seconds. The request loop is waiting 15 seconds. After that, the fifth request is sent:

01:48:15.79555  Sending request
01:48:15.79615  onHalfOpen: Circuit is half-opened and will test the service with the next request
01:48:15.79633  CircuitState: HalfOpen
01:48:15.79676  Successfully sent request
01:48:15.79770  onReset: Circuit closed and is allowing requests throughCode language: plaintext (plaintext)

Notice the onHalfOpen callback wasn’t executed until the circuitBreakerPolicy object was interacted with. Logically, it was in the open state for 10 seconds and then in the half-open state. The onHalfOpen callback should’ve fired after 10 seconds, but it didn’t. This reveals that you shouldn’t rely on these callbacks for detecting state changes in real-time.

In the half-open state, it tests the service with the first request and blocks all other requests. Since the request was successful, it closed the circuit, resulting it in firing the onReset callback.

An open circuit doesn’t automatically close after the duration

Let’s say you have the following circuit breaker policy:

var circuitBreakerPolicy = Policy.Handle<HttpRequestException>()
	.CircuitBreaker(exceptionsAllowedBeforeBreaking: 3, durationOfBreak: TimeSpan.FromSeconds(10));
Code language: C# (cs)

After it runs into three HttpRequestExceptions in a row, the circuit breaker will trip, opening the circuit for 10 seconds and blocking all requests that come in during that time.

After 10 seconds, it transitions to the half-open state. The first request that comes in during this state is used to test if it’s ok to close the circuit. If it succeeds, the circuit transitions to the closed state. If it fails, the circuit will be opened again for the configured duration. Meanwhile, any other requests that come in while it’s in the half-open state will run into the BrokenCircuitException.

This behavior makes sense. You don’t want to send tons of requests to an endpoint that’s potentially still down. This is especially true if you have no other throttling mechanism in place.

The exception count resets when there is a successful request

Let’s say you have the following circuit breaker policy that trips if it runs into three TransientExceptions in a row:

var circuitBreakerPolicy = Policy.Handle<TransientException>()
	.CircuitBreaker(exceptionsAllowedBeforeBreaking: 3, durationOfBreak: TimeSpan.FromSeconds(10));
Code language: C# (cs)

What happens if a TransientException happens and then a successful request is sent? It resets the error count.

For example, let’s say you send six requests and it’s successful every other time:

12:46:20.92701  Transient exception while sending request. Will try again.
12:46:20.92723  Successfully sent request
12:46:21.93395  Transient exception while sending request. Will try again.
12:46:21.93409  Successfully sent request
12:46:22.94494  Transient exception while sending request. Will try again.
12:46:22.94534  Successfully sent requestCode language: plaintext (plaintext)

If it wasn’t resetting the error count, then the third TransientException would’ve tripped the circuit, and the request right after it would’ve failed.

It’s a good thing it resets the error count. Imagine if it didn’t do this. It would result in tripping the circuit when the service was in a known good state (and potentially several hours later from the time the first exception happened).

Manually change the circuit state

You can manually control the circuit state, closing or opening it as desired. There are many reasons why you might want to do this. Perhaps you know the endpoint is repaired and you want to immediately close the circuit to allow requests through again. Or maybe you’ve built-in an admin kill switch that’ll trip the circuit on demand.

Close the circuit with policy.Reset()

To manually close the circuit, call policy.Reset().

For example, let’s say you don’t like the half-open state functionality so you want to bypass it. Here’s how you’d do that:

if (circuitBreakerPolicy.CircuitState == CircuitState.HalfOpen)
{
	circuitBreakerPolicy.Reset();
}
Code language: C# (cs)

Note: This also resets the error count.

Open the circuit with policy.Isolate()

To manually open the circuit to block requests, call policy.Isolate(). When you do this, it won’t close automatically. You have to call policy.Reset() to take it out of this isolated state. For example, let’s say you’ve built an admin control panel with pause / resume functionality:

Log("Admin is pausing requests");
circuitBreakerPolicy.Isolate();

Log("Admin is resuming requests");
circuitBreakerPolicy.Reset();
Code language: C# (cs)

Isolate() puts the circuit in the isolated state, which means it’s open and can only be closed again by calling Reset().

You can check if it’s in the isolated state by checking the CircuitState property:

catch(BrokenCircuitException)
{
	if(circuitBreakerPolicy.CircuitState == CircuitState.Isolated)
	{
		Log("Circuit was intentionally tripped by the admin. Will try again after requests are resumed.");
	}
}
Code language: C# (cs)

Note: You may want to handle BrokenCircuitException differently if you’re in isolated mode, since you know the circuit was intentionally opened.

Full example – Using circuit breaker with HttpClient

In this section, I’ll show a full example of using the Polly circuit breaker by using it with HttpClient to send requests to a service.

To simulate the service being temporarily unavailable, I’ve implemented a service stub that returns HTTP Status Code 404 (NotFound) when you tell it to. The client sends requests to this service and has configured the circuit breaker policy to look for this specific error code.

RandomNumberClient – Sends requests with HttpClient

First, here’s the client. This uses HttpClient to send requests to the service stub.

It configures the circuit breaker policy to look for three 404’s in a row and then trip for 1 minute. It’s wired up all the callback parameters (onBreak, onReset, and onHalfOpen) to log when they happen.

using Polly;
using Polly.CircuitBreaker;

public class RandomNumberClient
{
	private readonly HttpClient HttpClient;
	private readonly string GetRandomNumberUrl;
	private readonly AsyncCircuitBreakerPolicy CircuitBreakerPolicy;

	public RandomNumberClient(string url)
	{
		GetRandomNumberUrl = $"{url}/RandomNumber/";
		HttpClient = new HttpClient();

		CircuitBreakerPolicy = Policy.Handle<HttpRequestException>(httpEx => httpEx.StatusCode == HttpStatusCode.NotFound)
			 .CircuitBreakerAsync(
			exceptionsAllowedBeforeBreaking: 3,
			durationOfBreak: TimeSpan.FromMinutes(1),
			onBreak: (_, duration) => Log($"Circuit tripped. Circuit is open and requests won't be allowed through for duration={duration}"),
			onReset: () => Log("Circuit closed. Requests are now allowed through"),
			onHalfOpen: () => Log("Circuit is now half-opened and will test the service with the next request"));

	}
	public async Task<string> GetRandomNumber()
	{
		try
		{
			return await CircuitBreakerPolicy.ExecuteAsync(async () =>
			{
				var response = await HttpClient.GetAsync(GetRandomNumberUrl);
				response.EnsureSuccessStatusCode();
				return await response.Content.ReadAsStringAsync();
			});
			
		}
		catch(HttpRequestException httpEx)
		{
			Log($"Request failed. StatusCode={httpEx.StatusCode} Message={httpEx.Message}");
			return "Failed";
		}
		catch(BrokenCircuitException ex)
		{
			Log($"Request failed due to opened circuit: {ex.Message}");
			return "CircuitBroke";
		}
	}

	private void Log(string message)
	{
		Console.WriteLine($"{DateTime.Now:hh:mm:ss.fffff}\t{message}");
	}
}
Code language: C# (cs)

RandomNumberService – Returns errors when you tell it to

Here’s a snippet of the service stub. This is an alternative approach to using a tool like toxiproxy to simulate service problems.

[ApiController]
[Route("[controller]")]
public class RandomNumberController : ControllerBase
{
	public static Mode Mode { get; set; } = Mode.Return200Ok;
	[HttpGet()]
	public ActionResult<string> Get()
	{
		Console.WriteLine($"Request received: GET /RandomNumber. Mode={Mode}");

		if (Mode == Mode.Return200Ok)
			return Ok(new Random().Next());

		return NotFound();
	}
}

public enum Mode
{
	Return200Ok,
	Return404NotFound
}
Code language: C# (cs)

Results

Start the RandomNumberService.

Stubbed endpoint: GET https://localhost:12345/RandomNumber
Commands:
        set-mode Return200Ok
        set-mode Return404NotFound
Current mode: Return200OkCode language: plaintext (plaintext)

Start the RandomNumberClient console app and send a request.

Press any key to send request
01:03:43.74248  Requesting random number
01:03:44.00662  Response: 1970545597Code language: plaintext (plaintext)

Change the service mode to return errors.

set-mode Return404NotFound
Current mode: Return404NotFoundCode language: plaintext (plaintext)

Send multiple requests until the circuit trips and the circuit opens.

01:07:10.88731  Request failed. StatusCode=NotFound Message=Response status code does not indicate success: 404 (Not Found).
01:07:10.88760  Response: Failed
01:07:17.24384  Requesting random number
01:07:17.25567  Request failed. StatusCode=NotFound Message=Response status code does not indicate success: 404 (Not Found).
01:07:17.25588  Response: Failed
01:07:18.10956  Requesting random number
01:07:18.11535  Circuit tripped. Circuit is open and requests won't be allowed through for duration=00:01:00
01:07:18.11568  Request failed. StatusCode=NotFound Message=Response status code does not indicate success: 404 (Not Found).
01:07:18.11587  Response: Failed
Code language: plaintext (plaintext)

Send another request while the circuit is still open.

01:08:14.91007  Requesting random number
01:08:14.91141  Request failed due to opened circuit: The circuit is now open and is not allowing calls.
01:08:14.91155  Response: CircuitBrokeCode language: plaintext (plaintext)

The request is blocked because the circuit is open. It immediately throws a BrokenCircuitException.

After 1 minute, send another request. This time the circuit will be in the half-open state. It’ll use the request to test the service to determine if it should be fully closed or opened:

01:10:12.55587  Requesting random number
01:10:12.55633  Circuit is now half-opened and will test the service with the next request
01:10:12.56626  Circuit tripped. Circuit is open and requests won't be allowed through for duration=00:01:00
01:10:12.56657  Request failed. StatusCode=NotFound Message=Response status code does not indicate success: 404 (Not Found).
01:10:12.56671  Response: FailedCode language: plaintext (plaintext)

This request failed because the service is still in error mode. Because the request failed in half-opened mode, the circuit will be opened again and we’ll have to wait another minute.

Change the service mode to stop returning errors:

set-mode Return200Ok
Current mode: Return200OkCode language: plaintext (plaintext)

After 1 minute, send another request.

01:15:47.46013  Requesting random number
01:15:47.46052  Circuit is now half-opened and will test the service with the next request
01:15:47.47420  Circuit closed. Requests are now allowed through
01:15:47.47439  Response: 723841862Code language: plaintext (plaintext)

It was in a half-opened state, so it used the request to test the service. The request was successful, so it fully closed the circuit, allowing future requests through.

Send a few more requests to see that they are allowed through.

01:18:12.82052  Requesting random number
01:18:12.83118  Response: 961049677
01:18:13.34879  Requesting random number
01:18:13.35227  Response: 280453765Code language: plaintext (plaintext)

Leave a Comment