Optionally add a time.sleep after each backscraper request

[grossir]

Some backscrapes make a lot of requests, and this may cause the source server to temporarily block us or to be overloaded.

We could handle this by introducing a time.sleep(backscrape_wait) after each backscrape tick, where backscrape_wait has a default of 1 (?) and is configurable via command line argument.

For example, the following command would override the wait to be of 60 seconds after each backscrape tick

manage.py cl_back_scrape_opinions --courts juriscraper.opinions.united_states.state.ortc --backscrape-start 2011/12/14 --verbosity 3 --backscrape-wait=60

courtlistener/cl/scrapers/management/commands/cl_back_scrape_opinions.py

Lines 43 to 52 in 9f2dcd7

	for site in site_yielder(
	mod.Site(
	backscrape_start=options.get("backscrape_start"),
	backscrape_end=options.get("backscrape_end"),
	).back_scrape_iterable,
	mod,
	):
	site.parse()
	self.scrape_court(site, full_crawl=True)

[sentry-io]

Sentry Issue: COURTLISTENER-8CP

An instance of this server failure after a couple of backscrape ticks...