Avoid parallel creation of virtual environment and pip upgrade (fix i…

[stweil]

…ssue #165)

Signed-off-by: Stefan Weil [email protected]

[stweil]

PR #164 and PR #166 seem to fix the parallel build issues. I still see an exception, but it looks like all processors are built nevertheless in less than 5 minutes:

pip._vendor.pkg_resources.ContextualVersionConflict: (ocrd-utils 2.13.0 (/root/ocrd_all/venv/lib/python3.6/site-packages), Requirement.parse('ocrd-utils==2.13.1'), {'ocrd-models'})

Still nagging are the repeated tesseract builds (see PR #121), but they now take only 35 seconds with the parallel build.

[bertsky]

Good idea to use sem for this as well. But we should not mix the semaphore for git submodule action with that for venv modifications. Also, sub-venvs should be different again. And we should not only protect the initial creation, but any pip install usage as much as we can.

Suggested change

	$(SEM) $(PYTHON) -m venv $(VIRTUAL_ENV)
	. $(ACTIVATE_VENV) && $(SEM) $(PIP) install --upgrade $(PIP_OPTIONS_E) pip setuptools
	$(SEMPIP) $(PYTHON) -m venv $(VIRTUAL_ENV)
	. $(ACTIVATE_VENV) && $(SEMPIP) $(PIP) install --upgrade $(PIP_OPTIONS_E) pip setuptools

with SEMPIP defined in similar fashion to the SEM for git, only with --id ocrd_all_$(VIRTUAL_ENV).

[stweil]

I considered that, but think that using new semaphores is overkill because the protected actions are rather short. Typically the module updates will be finished when sub-venvs are created.

[stweil]

And we should not only protect the initial creation, but any pip install usage as much as we can.

I don't understand that part. Are there problems with parallel instances of pip install? I would have expected that pip can handle that. Parallel installation is desired, because it can save much time. Upgrading pip is a very special case.

[bertsky]

I considered that, but think that using new semaphores is overkill because the protected actions are rather short. Typically the module updates will be finished when sub-venvs are created.

Don't be so sure. Module checkouts can take a while for large/recursive repos. That's time wasted for others with large Python dependencies (Tensorflow etc). Especially since sub-venvs are completely independent of each other.

The cost of distinguishing SEMGIT and SEMPIP is minimal, and it maximizes parallel time available.

[bertsky]

And we should not only protect the initial creation, but any pip install usage as much as we can.

I don't understand that part. Are there problems with parallel instances of pip install? I would have expected that pip can handle that. Parallel installation is desired, because it can save much time. Upgrading pip is a very special case.

Yes, exactly. I don't think pip does file locking. You're right that pip updating itself is more sensitive, but I guess there are potential problems elsewhere, too (esp. after #142 makes pip intolerant of temporary inconsistencies). Have not tried/analysed it though. (And no, never saw make all -j fail like that either. But perhaps I have just not tried hard enough.)

[stweil]

Typically the module updates will be finished when sub-venvs are created.

Don't be so sure. Module checkouts can take a while for large/recursive repos. That's time wasted for others with large Python dependencies (Tensorflow etc).

See the new issue #169 for my answer. Yes, we can optimize. But currently introducing new semaphores does not result in faster builds. I just tested that. That might change when #169 is solved. I guess that new semaphores will reduce the build time by less than 5 seconds.

[bertsky]

Okay. So let's keep this in mind and check again. Yes, probably very unlikely that multiple modules want to install the same dependency at the same time. But even if so, as long as they picked the same version, it's probably going to work...

[kba]

LGTM. I don't see the harm in using different semaphores for different purposes but if the gains are inconsequential, it doesn't matter.

[stweil]

@bertsky, can we merge this (and think about optimization later)?

[bertsky]

can we merge this (and think about optimization later)?

agreed.