Week 30 routine

[kiwixbot]

Infra

• Check nodes free space

df -h / && df -h /data

• Nodes and worker system upgrades

apt update && apt upgrade

• Ensure all borg repositories are being updated
• Check Pod errors

k get pods -A -o wide|grep Error

• Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER | jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions | jq ".versions[].name"

• Upgrade k8s if applicable

Stats

• Ensure download.kiwix.org stats are being recorded
• Check whether matomo should be upgraded

Grafana

• Alert list is normal
• Zimfarm dashboard is normal
• Jobs durations dashboard is normal
• There is no abnormal behaviors on cluster resources consumption

Projects

• UptimeRobot has no alert
• zimit backlog is reasonable
• Analyze zimit failed tasks
• PRs awaiting your review

Security

• Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

Storage

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	/	37G	12G	24G	33%	-
stats	/	233G	108G	113G	49%	+2G
services	/	456G	286G	147G	67%	+1G
storage	/	33T	22T	9.4T	70%	+0.1T
demo	/	916G	205G	665G	24%	don't care
imager-worker	/	1.9T	831G	958G	47%	don't care
sisyphus	/	233G	24G	198G	11%	don't care
ondemand	/	25G	11G	14G	43%	don't care
ondemand	/data	216G	6G	199G	3%	don't care

[benoit74]

Misc

• k8s still needs to be upgrade
• setup of "ZIM_DIRENTLOOKUPCACHE" on library generation jobs seems to allow avoiding extreme durations of the job (multiple hours), but variability and duration are still too high (up to 45 minutes for prod library) ; clearly these jobs do not seem to be the issue but suffer from another issue

Zimit

NEW: openzim/warc2zim#356

• Zimit backlog is growing (19 items ATM).
• (2) NameResolutionError or similar
• (1) page response missing
• (1) ERR_CONNECTION_REFUSED
• (1) ERR_ABORTED
• (0) ERR_CONNECTION_CLOSED
• (0) INTERNAL ERROR (puppeteer) => opened Puppeteer INTERNAL ERROR  openzim/zimit#343
• (1) Pupeeter Execution context was destroyed (crash?)
• (2) Navigation timeout of 90000 ms exceeded at startup
• (0) ValueError: Title is too long.
• (0) Invalid seed specified
• (23) Unable to find WARC record for main page openzim/warc2zim#203 `Unable to find WARC record for main page
• (2) LookupError: unknown encoding: unicode openzim/warc2zim#331 LookupError: unknown encoding
• (0) Some websites are raising HTTP2 errors on sisyphus worker openzim/zimit#329 Is HTTP2 an issue
• (6) ValueError: Incorrect HttpUrl scheme in value openzim/warc2zim#332 ValueError: Incorrect HttpUrl scheme in value
• (1) Bad redirect location should be ignored instead of stopping crawler openzim/warc2zim#356 Unsupported upper-case chars in hostname
• (0) Can we do something when server return bad content-type openzim/warc2zim#346 Can we do something when server return bad content-type