Control the service restart after blown

[posilva]

Hi,

I am using fuse to control the access to a backend service (DAL API). It would be nice that we have some good way of passing from blown to ok gradually. Otherwise, if the backend is under load (502/503), for example, the requests will be back on charge all at the same time, after the "heal" interval and can cause problems again.

Thank you,

(I will be able to implement a solution if you think that may be useful)

Pedro

[jlouis]

Yes, this is a good idea, which I've also considered implementing.

The problem with its implementation is "how are you going to build a quickcheck model for it?". You need to come up with a good way of describing what gradually become ok means, and hopefully in a "deterministic" way. One of doing so is to control the RNG from the model so you can decide what the outcome of RNG lookups are.

The other problem is how you are going to let a few through. The fuse is an ETS table lookup, so if you flip that to 'ok' then the system will almost surely let a few through. So you would need some kind of "{gradual, Pct}" for some percentage, with the RNG controlled by the model.

This, and also its cousin of manually being able to disable/reenable fuses, are probably two of the most needed features.

If you come up with a better scheme, I can try to figure out if I can build a QC model for that.

[jlouis]

Ok, this is doable if we just control the RNG in the test cases, which is fairly easy.

What do you think the configuration should look like? I think there are a number of things here:

• We need to say when we need a gradual ramp-up.
• We need to decide what to return to the caller when we are in gradual mode.
• We need to say what period we use for the gradual ramp up, and how much to add over time. It lets a user say to gradually enable a service over 3 minutes, in 10 steps, say of 0, 10, 20, 30, 40 ... requests let through. In practice the system would then ramp every 180 div 10 = 18 seconds until it is fully working again. Or you could simply give it a ramping scheme: [{6000, 5}, {15*1000, 15}, 80*1000, 80}, {300*1000, 100}] would ramp up the system at 6, 15, 80 and 300 seconds with the amount given as the percentage.

I'm pretty sure I could build a quickcheck model for this kind of system, since I can mock the RNG and control its outcome, so I can say what the system should do in the different cases. I could also improve the timing mocking for this.

[jlouis]

More thoughts:

• The percentages are floating point numbers between 0.0 and 1.0. This allows one to supply fractions easily: 1/512. And so on.
• Having a way to put the system into a permanent gradual mode where a fuse is blown sometimes is a very good way of testing that your system supports the circuit breaker in the first place. This would be a really nice configuration option to have on a fuse when defining its melting point. And the patch you propose would make that support way easier.

[jlouis]

Some implementation plan for a QC model:

• Go to a component based model for handling randomness and timing better.
• Mock randomness
• Mock timing, transplant jlouis/dht's timing component to this model.
• Cluster

A first implementation should probably support a new type of fuse {fault_rate, Rate, Intensity, Period} which fault-injects every 1/Rate requests on average. This can verify the above model is in place and works.

Once you have support for this, it should be easy to add gradual ramping to the system.

The price to pay are parallel invocation models for this change, as they cannot be handled by such a system. So we would have to keep a parallel model around separately for this.

[posilva]

Hi,

I am not familiar with QuickCheck but now I have a good chance to learn about it, as soon as I have a model designed and something to show I will let you know.

[jlouis]

We already have most of the model in test/fuse_eqc.erl, so that is a starting point. It needs to use component based models however, to handle what I'm suggesting above.

[zeeshanlakhani]

Yeah, we've discussed something similarly w/ our use of Fuse to handle Solr (and other third-party systems) issues (w/ solr_cores) under load. Being able to gradually pass from blown->ok would be a better model of how we expect our fuse-wrapped operations to eventually resolve. I'd be down for reviewing and/or helping w/ QC if there are questions too when I'm back around next week.

[jlouis]

One important observation is that a standard fuse with a reset of 60*1000 would be a gradual fuse with [{60*1000, 1.0}] as in it goes to maximal rate in one step. This means we can handle the standard fuses as a special case of gradual fuses, which collapses a lot of the code base.

[zeeshanlakhani]

@jlouis yep... that observation makes 100% sense to me :).

[jlouis]

Ok, #10 has a new fuse_eqc based on an eqc_component model. This model can handle a fault_injection type fuse and will verify the RNG components needed to support this issue as well. Things needed:

• fuse:install/2 must be taught fault_injection fuses are valid. In the model and in the code.
• A fault_injection fuse does not push ok to the ETS table, but {gradual, Rate}. The fuse:ask/1 command has already been taught to handle this.
• The code must pass the model. There is no other task to do and we have fault_injection type fuses implemented.

[jlouis]

The model has been taught about installing and handling fuses of fault_injection type. This completes the model. We just need to handle the code itself.

Looking forward:

• Use eqc_component to directly run the timing component. Create a cluster with timing.
• When a timer trigger keep track of the current gradual scheme in the model. Make sure the correct timer is applied and verify that the system sets the right gradual timing component.
• Check for 1.0 and replace that with the default "ok" state of the fuse (ok or {gradual, Rate}).
• There are probably validity rules for gradual schemes which we better define and check.

[posilva]

We can have a also another approach, instead of adding delay to the "ok" state we can fail even faster if we are in a "gradual interval":

1. The fuse enters in the blown state
2. After reset interval it will pass to ok
3. If it fails in the short interval of time we will go again to blown otherwise we keep the ok
4. After gradual interval without melt we are 100% operational.

With this approach in the case of the backend service recover well, we do not loose requests, but if the service starts to fail again we will have the chance to fail fast/sooner and back off for a some short period of time (depending on the fail rate). If the period between fails is small in the gradual interval we will have the chance to back off more time.

This fuse could be a fail_fast type.

I hope this idea is clear enough :)

[jlouis]

I think it would make sense that in a "gradual" setting, we immediately fall back to error if it fails. I also think we can implement this with an update_counter/3 to the ETS table without accidentally ending up letting too many through. Of course, given the async context, we can't necessarily be totally void of races, but that is okay.

Perhaps with a bit more thinking, it is possible to figure out how this fuse type can be added to the system.

[jlouis]

The reset policy is a command language. You give commands [C1, C2, C3, ...]. The possible commands are:

• {delay, Ms} - Delay command processing for Ms milli-seconds. After that, proceed to the next command in the sequence.
• {test, N} - Let N requests through. If they all complete without error, go on in the command sequence. Otherwise, start the command sequence over. This should be implementable with an ETS update_count/3,4 style message sequence.
• {gradual, Rate} - If Rate = 0.05 we are letting 5% of all requests through to the service from here on in the command sequence. If they fail, they are subject to the standard Period/Intensity calculations.
• heal - Heal the fuse completely.

The standard {reset, Ms} is encodable as [{delay, Ms}, heal] in this scheme. Gradual ramping is supported, and @posilva's ideas are supported as well. You can get any mix possible: e.g., [{delay, 60*1000}, {test, 3}, {gradual, 0.25}, {delay, 10*1000}, {gradual, 0.5}, {delay, 10*1000}, heal] would:

• Wait 60 seconds upon failure
• Test 3 requests, if all pass:
• Ramp to 25% of all requests accepted for 10 seconds
• Ramp to 50% of all requests accepted for 10 seconds
• fully heal the fuse and go back to operational.