-
-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Puppeteer errors causing program termination #376
Comments
Puppeteer is the software driving the browser used to crawl the website. The error you get is a puppeteer error which seems to indicate that the browser is gone. Probably crashed. Using |
Thanks for your reply, I will set |
After changing the configuration the crawler still gets the same error, and after going to check the issue for browsertrix-crawler I found a similar error, but still don't know the solution at the moment. |
If you don't mind, yes please ; provide a link to this issue so that we can track them |
@MCSeekeri can you provide a more complete log of when this happened? After how many pages? We haven't been able to repro this type of error consistently, it could be browser running out of memory, or something else. Would also recommend trying with even less workers than 8, maybe 2 or 4. |
Currently there is only the full log for |
Apparently setting |
@MCSeekeri Unfortunately, no, this is very hard to reproduce because it takes 20K-30K pages before the issues pops up, and the stack trace is not helpful at all. |
Is there any way to make the program “more verbose” to find the possible root cause of the problem? |
Possibly, you can try adding this to the environment: |
Well... |
You can try it with browsertrix-crawler image directly with the same command line: |
@MCSeekeri I think it's a bug in Puppeteer that happens when it's cleaning up some internal objects, I opened an issue there: puppeteer/puppeteer#13056 |
Thank you @ikreymer for your support! |
- add additional catch() block - wrap page.title() in timedRun() to catch/log exception if this fails - log error in getting cookies - hopefully fixes hard-to-repro edge case crash in openzim/zimit#376
This might be fixed in 1.3.1, see if you can repro this again in this version - again very tricky to repro, puppeteer folks think its something that's not being caught in the crawler, so added a bunch of extra exception handling improvements, which might address this. |
Thanks. I'll try again. |
A new error reporting ............ is kind of progress? |
Yeah, I think these errors are most certainly way too many workers! |
I was using Zimit to archive the SCP-CN Wikidot site and encountered an interruption of the program due to a puppeteer error.
Attached here is the log output before the program exits.
docker-compose.yml:
I've modified the configuration with reference to the existing recipes, but I still encounter this problem.
The text was updated successfully, but these errors were encountered: