Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change how the ansible-doc thread pooling works #89

Open
abadger opened this issue Jun 9, 2020 · 0 comments
Open

Change how the ansible-doc thread pooling works #89

abadger opened this issue Jun 9, 2020 · 0 comments
Labels
documentation Improvements or additions to documentation enhancement New feature or request Partial Some of the work for this has been merged

Comments

@abadger
Copy link
Contributor

abadger commented Jun 9, 2020

The configuration for threads currently allow 128 concurrent workers but with the ansible-doc parser we have significantly less than that. The reason is that we need to keep the worker count at or below 128 but in ansible-doc we're currently allocating workers twice. First, we allocate workers to process each of the 13 documentable plugin types. Then, inside of each of those 13, we spawn more workers to invoke ansible-doc and get the plugin documentation.

Because of that structure, we don't know how many workers we can allocate to each plugin type so we just divide them evenly between them. 128/13 is roughly 10 threads for each plugin type. Since modules vastly outnumber the other plugins, this means we spend the majority of the time with 10 threads waiting on ansible-doc IO after the other plugins have finished.

The right way to fix this is to restructure the work so that all the workers are allocated from the same place. That way they'll all just end up being allocated as one of the 128 threads becomes free. As a stopgap, we could simply give modules 70% or so of the threads and divide the remaining threads evenly amongst the other plugin types.

abadger added a commit to abadger/antsibull that referenced this issue Jun 9, 2020
There are vastly more modules than other types of plugins so we want to
allocate the bulk of our threads to processing documentation from them.

This is an intermediate fix for ansible-community#89

We may want to fix this in a better way later on.
abadger added a commit to abadger/antsibull that referenced this issue Jun 9, 2020
There are vastly more modules than other types of plugins so we want to
allocate the bulk of our threads to processing documentation from them.

This is an intermediate fix for ansible-community#89

We may want to fix this in a better way later on.
abadger added a commit that referenced this issue Jun 9, 2020
There are vastly more modules than other types of plugins so we want to
allocate the bulk of our threads to processing documentation from them.

This is an intermediate fix for #89

We may want to fix this in a better way later on.
@abadger abadger added enhancement New feature or request Partial Some of the work for this has been merged labels Jun 12, 2020
@acozine acozine added the documentation Improvements or additions to documentation label Jul 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request Partial Some of the work for this has been merged
Projects
None yet
Development

No branches or pull requests

2 participants