Review the Housekeeper recovery task

[lucasoares]

While #21 will deal with a few improvements for the housekeeper tasks, the recovery task is a special task activated when the entire cache data is flushed to recover from catastrophic failures in the cache engine.

This task can also run manually by deleting the following cache key:

recovery_storage_breakpoint

For Redis, it would be the equivalent of deleting the key deckard.recovery_storage_breakpoint

This task will iterate through the entire storage recovering all items and reconstructing the cache. There are a few issues with this process that could be improved:

• if the cache engine is flushed, locked messages will return to the queue immediately. Messages locked without a defined score (possibility being added in Address ambiguity for priority/score field names and additional priority score features #27), will return to the queue with the minimum score (highest priority).
  • if you run the recovery process manually without flushing the entire cache, locked messages and processing messages will not return to the queue immediately, only if the locked/processing pool cache keys were deleted.
• the recovery process doesn't take in mind the performance. It contains a few bulking algorithms but it was not created to be the fastest process in the world, since it is only used for catastrophic failures
• In the Redis cache engine, if the instance is configured with any key eviction elements may be removed from the cache and will only be recovered if inserted again manually or with the recovery process. Maybe we should think something about this inside the recovery task or any other solution inside Deckard.