Add option to only crawl website and not run warc2zim conversion

[benoit74]

For debugging purpose, it might be useful to only run the crawling and not run warc2zim conversion (which might be known to fail, or even hang forever in a dead loop).

We should add a --crawl-only CLI argument to support this scenario (and integrate this in the Zimfarm obviously).

[rgaudin]

PR looks good but I'm not sure about the operational value of this

which might be known to fail

How is that a problem if we can keep the WARCs (and even upload them via Zimfarm)? warc2zim is a speedy process

or even hang forever in a dead loop

Is this an existing problem? Where's the ticket about this? openzim/warc2zim#132 is zimit1 AFAIK

[benoit74]

How is that a problem if we can keep the WARCs (and even upload them via Zimfarm)? warc2zim is a speedy process

Speedy compared to the crawl, yes. But still time consuming for probably nothing.

Is this an existing problem?

It happened to me during development on my machine, I don't see why it would not happen during production. And I most probably have a case this morning but still did not had time to collect material to open corresponding issue.

[benoit74]

openzim/warc2zim#246

[benoit74]

But you're right that from an operational point of view, it would make more sense to adapt Zimfarm worker so that logs and artifacts are uploaded even when a cancellation is requested. It would probably help in multiple cases.

[rgaudin]

Yes, it's a huge frustration point for most scrapers

[benoit74]

Then, let's close this in favor of openzim/zimfarm#965