Ranking server sometimes looses some of subchanges

[kunyavskiy]

Description:

During IOI 2024 contest, we had a live stream, using scoreboard as a source of data for inforgrafics.
In particular, we are using /subchanges endpoint to receiver score updates. While not being optimal, for simplicity,
we just reload the whole endpoint every 5 seconds.

The problem is on the second day by some reason some of day1 submissions sumtimes disappeared from /subchanges endpoint and appeared back.

As an estimation here is log of 4-digit submission ids available requested several times.

/tmp/d1/message$ curl https://ranking.ioi2024.eg/subchanges/ | grep -oe '"submission": "...."' | wc -l
5383
/tmp/d1/message$ curl https://ranking.ioi2024.eg/subchanges/ | grep -oe '"submission": "...."' | wc -l
5383
/tmp/d1/message$ curl https://ranking.ioi2024.eg/subchanges/ | grep -oe '"submission": "...."' | wc -l
5373
/tmp/d1/message$ curl https://ranking.ioi2024.eg/subchanges/ | grep -oe '"submission": "...."' | wc -l
5373
/tmp/d1/message$ curl https://ranking.ioi2024.eg/subchanges/ | grep -oe '"submission": "...."' | wc -l
5383

There is normally 5383 submissions with 4-digit id, while 10 of them disappeared for some time, and then appeared back.

I'm not sure if it is cms bug, or setup problem, as I don't have any access to setup.

Expected: All submissions of the first day are always available during the second

Actual: They eventually disappears and appears back

Logs

System Information

Unfortunately, I don't have access to setup, so I have no logs.