Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multiple replicas with a single leader #90

Open
stigok opened this issue Feb 9, 2023 · 1 comment
Open

Support multiple replicas with a single leader #90

stigok opened this issue Feb 9, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@stigok
Copy link
Contributor

stigok commented Feb 9, 2023

Problem

  • The GitHub API has rate limiting implemented that must be carefully respected to avoid lock-out. When multiple replicas are all reading from the API the chances of this happening increases linearily. If the replicas are aware of each other, a leader can be elected to take care of API reads before replicating the results to the other replicas.
  • When multiple replicas are behind a load balancer, if webhooks are implemented as in Update cache using Github webhooks #71, only a single member will receive the update. The webhook payload or the resulting side-effects of the event must be replicated to the other replicas.

Proposed solutions

  • Implement redis client code to connect to a redis cluster for both storage and leader election.
    • Introduces an extra moving part/system
  • Embedded database and/or clustering mechanism

Goal

  • Limit the amount of complexity of the selected solution as much as possible
@stigok stigok added the enhancement New feature or request label Feb 9, 2023
@stigok
Copy link
Contributor Author

stigok commented Jun 23, 2023

Raft seems very good, but also seems very complicated. Would probably suffice to implement some poor mans leader-election through memberlist instead. We don't need to be 100% sure we're the leader -- we just need to reduce the chance of more than one client querying the GitHub API at a time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant