2016-10-19 17:04:43 +03:00
|
|
|
# Circuit Breaker Pattern
|
|
|
|
|
|
|
|
Similar to electrical fuses that prevent fires when a circuit that is connected
|
|
|
|
to the electrical grid starts drawing a high amount of power which causes the
|
|
|
|
wires to heat up and combust, the circuit breaker design pattern is a fail-first
|
|
|
|
mechanism that shuts down the circuit, request/response relationship or a
|
|
|
|
service in the case of software development, to prevent bigger failures.
|
|
|
|
|
2020-05-08 11:16:04 +03:00
|
|
|
**Note:** The words "circuit" and "service" are used synonymously through this
|
2016-10-19 17:04:43 +03:00
|
|
|
document.
|
|
|
|
|
|
|
|
## Implementation
|
|
|
|
|
|
|
|
Below is the implementation of a very simple circuit breaker to illustrate the purpose
|
2020-08-01 02:44:26 +03:00
|
|
|
of the circuit breaker design pattern, only considering Open/Closed
|
2016-10-19 17:04:43 +03:00
|
|
|
|
|
|
|
### Operation Counter
|
|
|
|
|
|
|
|
`circuit.Counter` is a simple counter that records success and failure states of
|
|
|
|
a circuit along with a timestamp and calculates the consecutive number of
|
|
|
|
failures.
|
|
|
|
|
|
|
|
```go
|
|
|
|
package circuit
|
|
|
|
|
|
|
|
import (
|
2020-05-08 11:16:04 +03:00
|
|
|
"time"
|
2016-10-19 17:04:43 +03:00
|
|
|
)
|
|
|
|
|
|
|
|
type State int
|
|
|
|
|
|
|
|
const (
|
2020-05-08 11:16:04 +03:00
|
|
|
UnknownState State = iota
|
|
|
|
FailureState
|
|
|
|
SuccessState
|
2016-10-19 17:04:43 +03:00
|
|
|
)
|
|
|
|
|
|
|
|
type Counter interface {
|
2020-05-08 11:16:04 +03:00
|
|
|
Count(State)
|
|
|
|
ConsecutiveFailures() uint32
|
|
|
|
LastActivity() time.Time
|
|
|
|
Reset()
|
2016-10-19 17:04:43 +03:00
|
|
|
}
|
2020-05-22 09:18:17 +03:00
|
|
|
|
|
|
|
type counters struct {
|
|
|
|
ConsecutiveFailures uint32
|
|
|
|
ConsecutiveSuccesses uint32
|
|
|
|
}
|
|
|
|
|
2016-10-19 17:04:43 +03:00
|
|
|
```
|
|
|
|
|
|
|
|
### Circuit Breaker
|
|
|
|
|
|
|
|
Circuit is wrapped using the `circuit.Breaker` closure that keeps an internal operation counter.
|
|
|
|
It returns a fast error if the circuit has failed consecutively more than the specified threshold.
|
|
|
|
After a while it retries the request and records it.
|
|
|
|
|
2020-05-08 11:16:04 +03:00
|
|
|
**Note:** Context type is used here to carry deadlines, cancellation signals, and
|
2016-10-19 17:04:43 +03:00
|
|
|
other request-scoped values across API boundaries and between processes.
|
|
|
|
|
|
|
|
```go
|
|
|
|
package circuit
|
|
|
|
|
|
|
|
import (
|
2020-05-08 11:16:04 +03:00
|
|
|
"context"
|
|
|
|
"time"
|
2016-10-19 17:04:43 +03:00
|
|
|
)
|
|
|
|
|
|
|
|
type Circuit func(context.Context) error
|
|
|
|
|
2020-05-22 09:16:24 +03:00
|
|
|
var canRetry = func(cnt counters, failureThreshold uint32) bool {
|
|
|
|
backoffLevel := cnt.ConsecutiveFailures - failureThreshold
|
|
|
|
// Calculates when should the circuit breaker resume propagating requests
|
|
|
|
// to the service
|
|
|
|
shouldRetryAt := cnt.LastActivity().Add(time.Second << backoffLevel)
|
|
|
|
return time.Now().After(shouldRetryAt)
|
|
|
|
}
|
|
|
|
|
2016-10-19 17:04:43 +03:00
|
|
|
func Breaker(c Circuit, failureThreshold uint32) Circuit {
|
2020-05-22 09:16:24 +03:00
|
|
|
cnt := counters{}
|
2020-05-22 08:36:37 +03:00
|
|
|
|
2020-05-22 09:16:24 +03:00
|
|
|
//ctx can be used hold parameters
|
|
|
|
return func(ctx context.Context) error {
|
2020-05-22 08:36:37 +03:00
|
|
|
|
2020-05-22 09:16:24 +03:00
|
|
|
if cnt.ConsecutiveFailures >= failureThreshold {
|
|
|
|
if !canRetry(cnt, failureThreshold) {
|
|
|
|
// Fails fast instead of propagating requests to the circuit since
|
|
|
|
// not enough time has passed since the last failure to retry
|
|
|
|
return ErrServiceUnavailable
|
2020-05-22 08:36:37 +03:00
|
|
|
}
|
|
|
|
}
|
2020-05-22 09:16:24 +03:00
|
|
|
// Unless the failure threshold is exceeded the wrapped service mimics the
|
|
|
|
// old behavior and the difference in behavior is seen after consecutive failures
|
|
|
|
if err := c(ctx); err != nil {
|
|
|
|
cnt.Count(FailureState)
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
|
|
|
|
cnt.Count(SuccessState)
|
2020-05-22 08:36:37 +03:00
|
|
|
return nil
|
|
|
|
}
|
2016-10-19 17:04:43 +03:00
|
|
|
}
|
|
|
|
```
|
|
|
|
|
2020-05-22 08:38:36 +03:00
|
|
|
## Related Contents
|
|
|
|
|
|
|
|
- [Circuit Breaker Pattern](https://docs.microsoft.com/en-us/previous-versions/msp-n-p/dn589784(v=pandp.10)?redirectedfrom=MSDN)
|
2016-10-19 17:04:43 +03:00
|
|
|
|
2016-10-20 06:43:10 +03:00
|
|
|
- [sony/gobreaker](https://github.com/sony/gobreaker) is a well-tested and intuitive circuit breaker implementation for real-world use cases.
|