SystemReliability and Resilience

and stuff

Some things need to be cleared up first

http://en.wikipedia.org/wiki/Vedette_(cabaret)

tuple

//Initialize customer and invoiceInitialize(customer, invoice);

public void Initialize(Customer customer, Invoice

invoice){

customer.Name = “asdf”;invoice.Date = DateTime.Now;

}

Initialize(customer, invoice);//did something happen to customer// and/or invoice?

customer.Name =InitNameFrom(customer,

invoice);invoice.Date =

InitDateFrom(customer, invoice);

customer.Name =GetNameFrom(customer,

invoice);invoice.Date =

GetDateFrom(customer, invoice);

var results = Initialize(customer, invoice);

customer.Name = results.Item1;invoice.Date = results.Item2;

public tuple<string, DateTime>Initialize(customer,

invoice){

return new Tuple<string, DateTime>(“asdf”, DateTime.Now);

}

public static bool TryParse(string s, out DateTime result)

or

public static tuple<bool, DateTime?> TryParse(string s)

tuple• Avoid side effects

• Avoid out parameters

• multiple values without a specific type

null object

private ILogger _logger;public MyClass(ILogger logger) {

_logger = logger;}

…

if (_logger != null) {_logger.Debug(

“it worked on my machine!”);}

null checks for everyone!

forget one and…

public class NullLogger : ILogger {public void Debug(string text) {

//do sweet nothing}

}

private ILogger _logger = new NullLogger();

public MyClass(ILogger logger) {_logger = logger;

}

…

_logger.Debug(“it worked on my machine!”);

null object• Can eliminate null checks

• Simple to implement

Circuit Breaker

Retry

Your

Applicati

on O

ut o

f Pro

cess

Dependency

N times

Out o

f Pro

cess

Dependency

N times

*

Y clients

= Denial of

Service Attack

Limit the # of retries

N * Ybecomes

5 * Y

Y isstill a

problem

Circuit Breaker

State Machine

On :: Off

On Offwhen not healthy

Off Onmanually

Get to softwarebefore we ask you

to dance

Healthyor

Unhealthy

Out o

f Pro

cess

Dependency

State is independent of

requestor

Out o

f Pro

cess

Dependency

Your

Applicati

on

Has many independent external dependencies

Your

Applicati

on

Can throttle itself

Your

Applicati

on

Has a wait threshold

Your ApplicationExternal

DependencyCircuit Breaker

Threshold = 2Pause = 10msTimeout = 30sState = ClosedRequest

Request

Failure (i.e. HTTP 500)Failure Count = 1Pause 10ms

Request

Failure (i.e. HTTP 500)Failure Count = 2State = Open

OperationFailedException

Threshold = 2Pause = 10msTimeout = 30sState = OpenRequest

30s has not passed

CircuitBreakerOpenException

Request

30s has not passed

CircuitBreakerOpenException

System can try

to

become

healthy

for 30s

Your ApplicationExternal

DependencyCircuit Breaker

Threshold = 2Pause = 10msTimeout = 30sState = ½ OpenRequest

Request

Failure (i.e. HTTP 500)

Failure Count = 2State = Open

OperationFailedException

30s has passed

Your ApplicationExternal

DependencyCircuit Breaker

Threshold = 2Pause = 10msTimeout = 30sState = ½ OpenRequest

Request

Failure Count = 0State = Closed

Response

30s has passed

Response

Your ApplicationExternal

DependencyCircuit Breaker

ClosedOpen

½ Open

½ Open is like a

manual reset

PauseTimeout

Pausebetween calls

in the loop

Timeoutbefore you

can call again

Exceptions

OperationFailed:

AggregateException

CircuitBreakerOpen:

ApplicationException

Don’t Loose Exception Info

Always use InnerException(s)

Threshold = 3State = ClosedRequest

Request

Failure (i.e. HTTP 500)

Request

Failure (i.e. HTTP 500)Failure Count = 2

Failure Count = 0State = Closed

Response

Response

Request?

Your ApplicationExternal

DependencyCircuit Breaker

Failure Count = 1

SegregateDependencies

circuitBreaker(“database”)

circuitBreaker(“weatherservice”)

Dependency type, endpoint svc,

endpoint

Where?

Your

Applicati

on O

ut o

f Pro

cess

Dependency

Cir

cuit

Bre

aker

Pro

xy

Watch forInception

Your

Applicati

on W

eb S

erv

ice

Cir

cuit

Bre

aker

Cir

cuit

Bre

aker

Pro

xy

Data

baseR

eposi

tory

circuit breaker• retry looping

• slow down attempts

• good neighbour

¡Muchas gracias!

gracias

Donald Belcham@dbelcham

donald.belcham@igloocoder.com