Decide how to handle HSM related failures

[ximon18]

In the proof-of-concept if a key cannot be generated or the internal key mapping cannot be updated and persisted then Krill will exit. Is this okay, or should Krill retry and if so how often and how fast? If all retries fail should Krill exit or only produce errors in the logs? If failures are retried should we count how often this happens and expose it as a metric? Should failure to sign result in a warning but Krill keeps running, or? What about failure to delete a HSM key or failure to generate random values via the HSM?

[ximon18]

For the walking skeleton PR #679 @timbru and I decided that Krill should not fail to start nor should it exit if there is a problem communicating with the HSM, instead we should warn in the logs and a retry with backoff approach has been implemented for connections/requests that fail due to a network timeout or I/O error.

[ximon18]

Delivered with the Krill v0.10.0 release.