Locking mechanism for index sets during datanode remote reindex migration

[todvora]

This PR adds proper locking mechanism for locking of index sets during remote reindex migration. These locks skip index rotation and retention if the migration is running. In the same time they force the migration to wait for finish of any currently running index set maintenance tasks before the actual data migration can start.

Description

The newly introduced DatanodeMigrationLockService, building on top of the existing distributed LockService is creating exclusive locks for index sets. The service allows two types of work with the lock - wait for a lock and block the thread or try to run a code block if the lock is free to take. The blocking mechanism is used for the migration itself. It will wait till the background maintenance is done. The nonblocking try approach is applied in the maintenance tasks. Their execution will be skipped for an index set if this is occupied by the migration.

Motivation and Context

Fixes https://github.com/Graylog2/graylog-plugin-enterprise/issues/8281

How Has This Been Tested?

Added unit tests, existing integration tests

Screenshots

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactoring (non-breaking change)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.

[luk-kaminski]

I do not fully understand the code, so please forgive me this silly question, but how do we handle multi-node setup?
Can each GL node have its own migration lock service?
If yes, is it important that activeLocks collection contains all the locks?
If yes, how do we get locks created by another GL nodes appear in that collection?

[luk-kaminski]

It seems that activeLocks collection is used just for extending the locks?
So maybe me previous questions are not important at all...

[todvora]

Each GL node will have its own migration lock service. The locks are persisted in the mongodb, so they are distributed and block parallel usage of the index set across the whole cluster. The activeLocks collection is there only to keep the locally requested locks from expiring. Otherwise they'll automatically disappear from the mongo collection, as they have defined a TTL. The local activeLocks collection doesn't need to be distributed, each node is refreshing its own locks. In the moment the node disappears, these lock disappear as well, which is a good thing.

[bernd]

Please check if you can use the RefreshingLockService to refresh the locks instead of implementing your own refresh system. 🙂

[todvora]

The problem with RefreshingLockService is that it can hold one lock only. I'd need one instance of the service for each lock. Instead of implementing own refresh system I'd need to implement a system for managing service instances, which seems even worse :-)

[luk-kaminski]

Looks good!

Looks complex as well. I have no idea how to simplify it, however.
I'd only suggest considering a few javadoc comments here and there, taken from the PR description and our discussions, so that no one has to ask himself the same questions we have been asking.

Thank you for a lot of hard work you have put into this one!