Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Specify maximum annotations per user #5610

Open
lgienapp opened this issue Oct 18, 2024 · 0 comments
Open

[FEATURE] Specify maximum annotations per user #5610

lgienapp opened this issue Oct 18, 2024 · 0 comments

Comments

@lgienapp
Copy link

lgienapp commented Oct 18, 2024

Is your feature request related to a problem? Please describe.

When using argilla for crowdsourced annotation workflows on large datasets, it would be necessary to enforce per-user limits on how many annotations can be submitted.

Describe the solution you'd like

I would like a way to enforce an upper limit on how many instances each user can annotate. This could be added as part of a DistributionStrategy, featuring a max_annotations_per_user parameter.

As a bonus, if an upper limit is set, the progress tracker could then reflect the personal progress of the user instead of the dataset progress.

Describe alternatives you've considered

A workaround is sharding the dataset into slices of max annotations size and selectively adding users to each; this creates huge overhead.
A different approach is having secondary app running that monitors user stats and removes them from the workspace if their upper limit is reached. Again, overhead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant