[WIP] Add timeout for container/sandbox recover.

[Random-Liu]

Although we don't want, containerd-shim sometimes hangs, e.g. containerd/containerd#2438. We should definitely root cause and fix it.

However, on the other hand, we should make sure that the system still functional when some containerd-shims hang. An example is that ctr task ls will hang if one single containerd-shim hangs. --> We should probably fix it. @crosbymichael

Luckily, the CRI plugin in most cases doesn't handle multiple containers at a time, this makes sure that a single container failure won't block other containers.

However, the only case that CRI plugin may handle multiple containers at a time is the recovery logic. This means that if a containerd-shim hangs, CRI plugin won't be able to restart. This is super bad.

This PR adds a timeout to per container/sandbox recovery logic, so that even one container hangs, we just don't load it, and continue dealing with other containers.

Signed-off-by: Lantao Liu [email protected]

[Random-Liu]

We should cherry-pick this into all supported branches.

[crosbymichael]

LGTM

[mikebrow]

/LGTM

[yujuhong]

LGTM

One question: if the load request timed out, how is the container going to be represented in the CRI plugin -- does CRI show the existence of the container and what state will it be in?

[Random-Liu]

@yujuhong Currently, won't show the container, but I think it would be better to show the container in Unknown state. Let me see whether showing a bad container will cause any issues.

BTW, how does Kubelet deal with unknown state? I remember it triggers a sync pod, and kubelet will try to start a new one.

[yujuhong]

Yes, kubelet will try to kill the container in the unknown state and start a new one.

[Random-Liu]

I found this PR doesn't actually help, because:

1. There is no grpc between CRI plugin and containerd, so no one handles the timeout;
2. The context will be eventually passed to containerd-shim, and ideally the rpc client will handle the timeout. However, ttrpc doesn't support timeout yet ttrpc: end to end timeout support ttrpc#3.

Given so, I'll leave this PR here now... We need a better timeout solution to make the system more reliable. I filed an issue containerd/containerd#2578.