Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change distros schedule to run each 4 days instead of 3 #309

Closed
wants to merge 2 commits into from

Conversation

Crola1702
Copy link

Description

Humble jobs have not run in the last 12 days (see Hci view and sort by last success). Humble has the lower priority of build.ros2 CI jobs, I think they might be getting down the queue because long buildtimes and a long job queue.

This PR changes the time between distro runs from 3 to 4 (except for Rolling), so that there is one day to run Rolling and clean the job queue.

See investigation

An alternative could be adding a third CI Agent, given the fact that the current 2 agents are busy all the time.

CC: @clalancette

Copy link

@clalancette clalancette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, I'm on board with reducing the frequency here. I'd like to get input from the rest of the infra team as well.

@claraberendsen
Copy link

claraberendsen commented Jul 10, 2024

I was doing a quick check on the agent pool for ci agents and we only have two ci agents when we should have 3. It seems at some point either due to build.ros2.org crashing or capacity problems one of the provisioned instances did not get connected to the jenkins master and that's the root of this problem.

I did a quick sum of the time it takes to run all Rci and it is around 68.21 hours which would still allow for the other jobs to run if there are three agents provisioned instead of two.

However, we should check how long do Iron and Jazzy add up to and see if the 72 hours per day provisioned is enough to run all the distros at a three day frequency.
For now I restored the missing ci agent @Crola1702 .

@nuclearsandwich
Copy link
Member

I was doing a quick check on the agent pool for ci agents and we only have two ci agents when we should have 3. It seems at some point either due to build.ros2.org crashing or capacity problems one of the provisioned instances did not get connected to the jenkins master and that's the root of this problem.

Oof. I wonder how long we've been paying for a crashed agent. I'd like to maintain the current cadence at 3x ci-agents if we can.

@claraberendsen
Copy link

@Crola1702 I think we can close this PR for now since it's no longer necessary.

@Crola1702 Crola1702 closed this Jul 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants