Fix desync after errCatchupTooManyRetries

[MauriceVanVeen]

Looked into the issue reported in Antithesis of non-monotonic sequence / pub ack sequence moving back due to stream desync.

When looking at these logs:

[        48.508] [      service_nats-1] [inf] [1] 2024/09/27 09:29:29.624201 [WRN] Catchup for stream '$G > test-stream' stalled
[        48.508] [      service_nats-1] [inf] [1] 2024/09/27 09:29:29.624634 [DBG] RAFT [yrzKKRBu - S-R3F-gMW2rGjb] Resuming our apply channel
[        48.508] [      service_nats-1] [inf] [1] 2024/09/27 09:29:29.624796 [WRN] Error applying entries to '$G > test-stream': catchup failed, too many retries
[        48.510] [      service_nats-1] [inf] [1] 2024/09/27 09:29:29.626681 [DBG] SYSTEM - System connection closed: Internal Client
[        48.511] [      service_nats-1] [inf] [1] 2024/09/27 09:29:29.627700 [DBG] RAFT [yrzKKRBu - S-R3F-gMW2rGjb] Deleted
[        48.511] [      service_nats-1] [inf] [1] 2024/09/27 09:29:29.627853 [DBG] Exiting stream monitor for '$G > test-stream' [S-R3F-gMW2rGjb]

In monitorStream we call into applyStreamEntries which calls into processSnapshot when about a snapshot. If after reaching maxRetries catchup remains stalled, the RAFT data is deleted, which means n.pindex=0.

Then when a leader election comes around this server with missing RAFT data would allow an outdated server that misses data to become leader. This is reproduced in the test.

1. Start 3 servers, add a R3 stream.
2. Publish 10 messages into the stream.
3. One server will be shut down, so it misses data and will need to be caught up.
4. Publish 10 more messages, the 2 remaining servers will get this.
5. Stop the stream leader and don't start it anymore for this test.
6. Now simulate us processing a snapshot and failing, by calling mset.resetClusteredState(errCatchupTooManyRetries)
7. We now start the server from point 3 (which was missing data and should be caught up with the missing data)
8. Because of n.pindex=0 on the reset server, it grants leader to the outdated server. Resulting in desync.

This PR proposes to not fully delete the RAFT state when we are not able to reach the leader during the processing of a snapshot. Which ensures the outdated server does NOT get selected as a leader and it gets correctly caught up to contain the data it missed.

Signed-off-by: Maurice van Veen [email protected]

[MauriceVanVeen]

This was an off-by-one. maxRetries=3, but there would only be max 2 retries.

[derekcollison]

Story of my life ;)

[derekcollison]

LGTM

[derekcollison]

Story of my life ;)