When reloading/restarting old request may hang out for a long time

[mstoykov]

When reloading the config all the handlers are replaced. Old ones will continue to be around until all requests that have been started with them are finished.

This might be a problem when the log file has been rotated and the reload was issued in order to change the log file. In that case old handlers will keep the log file open and write to it for as long as a request lasts. A request can last for hours. A file which is kept open in a unixy OS will be preserved even after it has been deleted from the file system. The problem is when this starts to happen often: the storage will be full of already deleted log files.

We could set a deadline for every request's context and so restrict how long a request can last. But maybe we do not want to stop requests that take long time all the time. Maybe only doing it when we are reloading or restarting. In that case canceling the whole application context after some time will be better. This would require that a new application context is made on reload which would not be hard.

Both of the solutions require that handlers would actually stop handling requests when the context is canceled or the deadline is reached. This is probably not the case for the majority of them.

[ironsmile]

I think that two configuration options will solve this specific issue and also give nedomi users much more control on how long-lived requests are handled. The two options may be:

• A request_deadline: the maximum time duration which a request will be served by nedomi. When the deadline comes the request is dropped no matter what. This should support empty value which would mean "no deadline". A configuration like that may prevent resource drain, malicious or otherwise, because of many long-lived requests.
• reload_keep_requests_for: time duration. For how long a request will continue to be handled after its handler has been scheduled for removal. Again, a empty value would mean "do not drop requests". This configuration would be useful in environments where reloading is done often and there are occasional long-lived requests.

Combinations between the two should be possible. If a request qualifies for both deadlines the shorter should be honoured.