Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store and expose harvest state #71

Open
tmaiaroto opened this issue Jun 17, 2015 · 0 comments
Open

Store and expose harvest state #71

tmaiaroto opened this issue Jun 17, 2015 · 0 comments

Comments

@tmaiaroto
Copy link
Member

Each harvester microservice should store its state and make it available to other harvesters. This way multiple harvesters can communicate with each other and work on different ranges of data. It prevents processing duplicate messages more than necessary. Such stateful information would include the current page of search results, etc.

This should be easily done using boltdb and exposed as a RPC via Kite. Should a service go offline and the same data be processed by another service (duplicate work), it should still be OK. While slightly inefficient it is "safer" and more consistent. The architecture is also setup in such a way that duplicate messages can not exist in the database.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant