Replacing existing stats.jenkins.io code with https://github.com/jenkins-infra/stats.jenkins.io

[krisstern]

Service(s)

Helpdesk, stats.jenkins.io

Summary

As the Implementing UI for Jenkins Infra Statistics project has officially completed, we would like to request for the existing code for the website https://stats.jenkins.io/ with that in https://github.com/jenkins-infra/stats.jenkins.io

c.c. @shlomomdahan @gounthar @Vandit1604

Reproduction steps

No response

[dduportal]

That would be the (almost) last mile of #4132 🥳

[krisstern]

Good to know @dduportal 👍🏼

[hervelemeur]

I'll prepare a plan for that today hopefully.

[lemeurherveCB]

From #4132 (comment):

It means planning the effective switch to the new webservice.

We would also need to define what we do with the existing service:

• How much time do we keep it ?
• Archiving ?
• etc.

As the existing website is running from Github Pages, I propose to just change its subdomain from stats.jenkins.io to old.stats.jenkins.io and keep it as there is no related infra cost, WDYT?

If we don't want to keep it, we can remove these lines: https://github.com/jenkins-infra/azure-net/blob/a51c2d937433c51eb289b319123beeb8e9477364/dns-records.tf#L287-L290

To switch to the new service we just have to exchange the DNS records.

Note: there is nothing to archive per se as https://github.com/jenkins-infra/infra-statistics is still used by the new website.

[lemeurherveCB]

To switch to the new service we just have to exchange the DNS records.

Opened jenkins-infra/azure-net#291

[dduportal]

From #4132 (comment):

It means planning the effective switch to the new webservice.
We would also need to define what we do with the existing service:

• How much time do we keep it ?
• Archiving ?
• etc.

As the existing website is running from Github Pages, I propose to just change its subdomain from stats.jenkins.io to old.stats.jenkins.io and keep it as there is no related infra cost, WDYT?

If we don't want to keep it, we can remove these lines: https://github.com/jenkins-infra/azure-net/blob/a51c2d937433c51eb289b319123beeb8e9477364/dns-records.tf#L287-L290

To switch to the new service we just have to exchange the DNS records.

Note: there is nothing to archive per se as https://github.com/jenkins-infra/infra-statistics is still used by the new website.

What about the "process" which updates the branch gh-pages?

[lemeurherveCB]

@dduportal the pipeline https://github.com/jenkins-infra/infra-statistics/blob/main/Jenkinsfile must be kept running.

It generates and pushes data to the gh-pages:

• https://github.com/jenkins-infra/infra-statistics/blob/8209b583ee5be409d2dc8cdf780bd4358579b8de/Jenkinsfile#L92-L94
• https://github.com/jenkins-infra/infra-statistics/blob/8209b583ee5be409d2dc8cdf780bd4358579b8de/publish-svgs.sh

The data generated is then used by the new stats website, which is only a frontend retrieving the data by downloading them from the content of gh-pages branch: https://github.com/jenkins-infra/stats.jenkins.io/blob/46f61c7264d72ae0b6d79551652ee4036c50a4ee/retrieve-infra-statistics-data.sh

[lemeurherveCB]

We may want to add some explanations on https://github.com/jenkins-infra/infra-statistics README to clarify this.

[dduportal]

• DNS change: LGTM. Let's plan it for 11 September if no objection (to be confirmed during team meeting)
• Old site on GH Page: it must be decommissioned. We don't want to have a resource published (it uses common resources: domain names, etc.) and it can be abused (like a DNS not renewed).
  • How much time should we keep it "just in case" once the DNS is switched? 1, 2 weeks? More ? (will discuss it during team meeting and I'll report here)
  • WDYT about changing the branch to data (instead of gh-pages) at this moment? It will allow us to disable the GH Page website once done

[lemeurherveCB]

data looks fine to me 🙂

[lemeurherveCB]

It will allow us to disable the GH Page website once done

Note that it can also be disabled by setting "none" as branch in GitHub Pages settings of the repository:

[lemeurherveCB]

As the job run once per month there will be plenty of time switching the branch name from gh-pages to data in both repositories without interrupting the service: https://github.com/jenkins-infra/infra-statistics/blob/8209b583ee5be409d2dc8cdf780bd4358579b8de/Jenkinsfile#L18-L19

// Make sure we run this job once a month (at 3am on the 2nd of the month) when the raw stats are available.
properties([pipelineTriggers([cron('0 3 2 * *')])])

[lemeurherveCB]

@MarkEWaite @shlomomdahan I'd also suggest that when jenkins-infra/azure-net#291 is merged and released, one of you could announce the new website on https://groups.google.com/g/jenkinsci-users and https://groups.google.com/g/jenkinsci-dev, WDYT?

Maybe a blog post too?

[krisstern]

Hi @lemeurherveCB Thanks! Or I could make an announcement for the revamped project too.

[dduportal]

@krisstern @lemeurherveCB We propose to switch the production to the new site tomorrow, Wednesday 11 September at 08:00am UTC (10h00am Paris time).

The rest can wait (by default 1 week).

No objection?

[hervelemeur]

@dduportal looks fine to me, but my opinion is not that important, @krisstern's one is.

[krisstern]

No objection from me

[dduportal]

We did change the DNS:

• The PR in Azure failed (TTL not up to date and non atomic change). We fixed it manually
• The stats website is now broken as the production in AKS does not have the required ingress setup. We are operating it right now to hotfix but the website is in outaghe until it's done

[dduportal]

Update: the new site is live (https://stats.jenkins.io/) and the old site is now available again (https://old.stats.jenkins.io/)

• Needed to fix the ingress on the AKS production to handle requests with the host stats.jenkins.io
  • See hotfix(publick8s/stats.jenkins.io) add the new stats.jenkins.io hostname to ingress kubernetes-management#5675
• Needed to change the expected CNAME on the GH Page setup
• See Update CNAME as part of the migration to AKS with new website infra-statistics#54

[krisstern]

I will make an announcement about the new revamped stats.jenkins.io site within the next couple of hours

[krisstern]

My apologies, I have been waiting for jenkins-infra/plugin-site#1890 to be merged first before making an announcement...

[dduportal]

My apologies, I have been waiting for jenkins-infra/plugin-site#1890 to be merged first before making an announcement...

many thanks @krisstern !

[dduportal]

I'm closing this issue as the scope "put the GSoC new website online as default" is ok. See #4132 (comment) for the next steps.

[lemeurherveCB]

EDIT: copied at #4132 (comment) to continue the discussion there (still open)

As noted in badges/shields#10522 (comment), there are far more consumers than we (I) initially though: https://github.com/search?q=stats.jenkins.io&type=code 😱

Links to (old.)stats.jenkins.io were pointing to files (still) stored on GitHub (currently on the gh-pages, generated from the pipeline of https://github.com/jenkins-infra/infra-statistics), while the new frontend doesn't serve them.

To avoid breaking existing consumers and without serving these files from our service, I propose to put in place redirections from the three folders of https://github.com/jenkins-infra/infra-statistics/tree/gh-pages (then from the data branch) to https://raw.githubusercontent.com/jenkins-infra/infra-statistics/gh-pages/ with the help of jenkins-infra/helm-charts#1332, WDYT?

Ex:
https://stats.jenkins.io/plugin-installation-trend/view-job-filters.stats.json would redirect to https://raw.githubusercontent.com/jenkins-infra/infra-statistics/gh-pages/plugin-installation-trend/view-job-filters.stats.json