If the strand can't finish its job in the allowed time, the respirate
exits, and all threads will be terminated.
Currently, we run 3 respirate dynos on Heroku. When they crash, Heroku
restarts them automatically. However, if they crash multiple times in a
very short period, it delays the following restarts.
When a strand consistently causes apoptosis, it affects all respirate
dynos and crashes them.
If a strand causes apoptosis, we should delay its consecutive iterations
to avoid crashing all dynos. Since it crashes due to apoptosis, its try
count is currently not increasing.
As a first option, we can schedule it for 30 minutes later to allow time
for recovery. However, if the strand is performing a critical job, such
as failover, we may not want to delay it that much if apoptosis is
intermittent.
Instead, we can increase its try count, so our scheduler delays it if it
consistently causes apoptosis.
beeb7a0d32