Queue lifecycle thread may exit upon Runtime exception

[sbrossie]

Our queue lifecycle thread catches Throwable and exit upon RuntimeException (i.e CallbackFailedException , but the Executor restart thread with no task so nothing gets dispatched.

A normal stack trace would be:

java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.killbill.queue.DefaultQueueLifecycle$DispatcherRunnable.sleepSporadically(DefaultQueueLifecycle.java:285)
        at org.killbill.queue.DefaultQueueLifecycle$DispatcherRunnable.run(DefaultQueueLifecycle.java:253)
        at org.killbill.commons.concurrent.WrappedRunnable.run(WrappedRunnable.java:48)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Instead, we see:

java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005c9872b00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)