Fix: use max_workers and added worker timeout #18
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch forces the module to take into account the
max_workers
parameters.The previous implementation spawned as much workers as the machine's cores using the
multiprocessing
library. The main issue is that the first operation done bymultiprocessing
when spawning aProcess
is afork()
system call. The module being internal to the scheduler,it's the scheduler itself that is forked. With big configurations, the memory consumption is large enough to crash the whole machine.
The number of workers is now controlled by the
max_workers
parameter which was not used before.The other fix is the inclusion of a
worker_timeout
parameter.The previous implementation was waiting for the spawned workers to join 30 seconds, and was simply exiting afterwards if the job wasn't finished. Still with big configurations, if the MongoDB server does not
finish to handle queries quick enough, stalled scheduler processes (spawned workers) can remain.
With this patch, the scheduler waits for
worker_timeout
seconds, and kills the remaining jobs and raises an error in the logs.Setting
worker_timeout
to0
forces the scheduler to wait for all children to finish before going further.