Architecture: Unit tests are non-hermetic

[audiodude]

Many (most?) of the unit tests actually make network requests to various MediaWiki projects like English Wikipedia. This leads them to be inherently flaky, because changes to data (not the API, just data) in these wikis can cause the tests to fail. See here for a simple and direct overview of the problem and the value of hermetic tests.

It is understood that mwoffliner needs to be tested against live wikis in order to prevent regressions. However, tests should be segregated so that any test which requires a live network connection is moved into e2e tests.

We should be able to test specific behaviors like "This utility method returns normalized titles" without downloading entire wiki pages.

Fixing this would also likely lead to architectural improvements, because we would be forced to decouple various modules more and use dependency injection.

It would also make the unit tests more speedy, because the network requests would not be necessary.

E2E tests could still be run in CI and on demand, when necessary.

[kelson42]

I can only agree with you, but not that easy to mockup MediaWiki with different configurations. That said we have an easy way to deploy MediaWiki with https://github.com/offspot/mediawiki-docker

[audiodude]

Yes, one of the ways to make the unit tests hermetic would be to have our own MediaWiki installations.

But I was thinking more along the lines of https://github.com/openzim/mwoffliner/pull/2043/files

Where the unit test just tests one aspect of the code and does not connect to any MediaWiki installation at all. I believe unit tests should test individual aspects and functions, instead of "Download this whole list of articles and then test if this one portion of it worked", which seems to be the current architecture.