Reject Duplicate Submissions #5047

rajpatel24 · 2024-08-05T12:43:21Z

Checklist

If you've added code that should be tested, add tests
If you've changed APIs, update (or create!) the documentation
Ensure the tests pass
Make sure that your code lints and that you've followed our coding style
Write a title and, if necessary, a description of your work suitable for publishing in our release notes
Mention any related issues in this repository (as #ISSUE) and in other repositories (as kobotoolbox/other#ISSUE)
Open an issue in the docs if there are UI/UX changes

Description

We have identified a race condition in the submission processing that causes duplicate submissions with identical UUIDs and XML hashes. This issue is particularly problematic under conditions with multiple remote devices submitting forms simultaneously over unreliable networks.

To address this issue, a PR has been raised with the following proposed changes:

Race Condition Resolution: A locking mechanism has been added to prevent the race condition when checking for existing instances and creating new ones. This aims to eliminate duplicate submissions.
UUID Enforcement: Submissions without a UUID are now explicitly disallowed. This ensures that every submission is uniquely identifiable and further mitigates the risk of duplicate entries.

These changes should improve the robustness of the submission process and prevent both race conditions and invalid submissions.

Notes

Implemented a fix to address the race condition that leads to duplicate submissions with the same UUID and XML hash.
Incorporated improvements from existing work, ensuring consistency and robustness in handling concurrent submissions.
The fix aims to prevent duplicate submissions, even under high load and unreliable network conditions.

Related issues

Supersedes kobotoolbox/kobocat#876

…licate_submissions

# Conflicts: # kobo/apps/openrosa/apps/logger/exceptions.py # kobo/apps/openrosa/apps/logger/models/instance.py # kobo/apps/openrosa/apps/logger/models/xform.py # kobo/apps/openrosa/apps/logger/xform_instance_parser.py # kobo/apps/openrosa/libs/utils/logger_tools.py

kobo/apps/openrosa/apps/logger/xform_instance_parser.py

kobo/apps/openrosa/apps/api/tests/viewsets/test_xform_submission_api.py

kobo/apps/openrosa/apps/logger/exceptions.py

kobo/apps/openrosa/libs/utils/logger_tools.py

kobo/apps/openrosa/apps/api/tests/viewsets/test_xform_submission_api.py

kobo/apps/openrosa/libs/utils/logger_tools.py

…-reject_duplicate_submissions

- Add test case for duplicate submission with an attachment - Improve logic to extract UUID from xml - Add logic to reject submission without UUID

notion-workspace · 2024-09-03T18:48:01Z

Reject Duplicate Submissions

kobo/apps/openrosa/apps/logger/tests/test_parsing.py

kobo/apps/openrosa/apps/logger/xform_instance_parser.py

kpi/tests/api/v2/test_api_submissions.py

kobo/apps/openrosa/libs/utils/logger_tools.py

kobo/apps/openrosa/apps/logger/tests/test_form_submission.py

…-reject_duplicate_submissions

…lbox/kpi into 862-reject_duplicate_submissions

kobo/apps/openrosa/apps/logger/models/instance.py

kobo/apps/openrosa/libs/utils/logger_tools.py

kobo/apps/openrosa/apps/logger/tests/test_form_submission.py

kobo/apps/openrosa/apps/logger/tests/data_from_sdcard/bulk_submission.zip

kobo/apps/openrosa/libs/utils/logger_tools.py

kpi/tests/api/v2/test_api_submissions.py

kobo/apps/openrosa/libs/utils/logger_tools.py

kobo/apps/subsequences/tests/test_submission_stream.py

…different content

…lbox/kpi into 862-reject_duplicate_submissions

…es but different content

kobo/apps/openrosa/apps/logger/management/commands/clean_duplicated_submissions.py

kobo/apps/openrosa/apps/logger/models/instance.py

…together constraint

# Conflicts: # kobo/apps/openrosa/apps/api/tests/viewsets/test_xform_submission_api.py # kobo/apps/openrosa/apps/logger/tests/test_simple_submission.py # kobo/apps/openrosa/apps/logger/xform_instance_parser.py # kobo/apps/openrosa/libs/utils/logger_tools.py

rajpatel24 changed the base branch from main to beta August 5, 2024 12:43

rajpatel24 added the Back end label Aug 5, 2024

noliveleger self-requested a review August 5, 2024 13:43

rajpatel24 force-pushed the 862-reject_duplicate_submissions branch from 28eefe5 to 039ed95 Compare August 5, 2024 16:20

rajpatel24 changed the title ~~862 Reject Duplicate Submissions~~ Reject Duplicate Submissions Aug 5, 2024

862 Avoid duplicate submissions to be saved

6a177da

rajpatel24 force-pushed the 862-reject_duplicate_submissions branch from 039ed95 to 6a177da Compare August 7, 2024 13:51

rajpatel24 added 3 commits August 15, 2024 18:34

Add root_uuid field to instance model with unique constraint

c9aee65

Merge branch 'beta' of github.com:kobotoolbox/kpi into 862-reject_dup…

98a7236

…licate_submissions

Merge branch 'beta' of github.com:kobotoolbox/kpi into 862-reject_dup…

71de437

…licate_submissions

noliveleger self-assigned this Aug 21, 2024

noliveleger changed the base branch from beta to kobocat-django-app-part-2-refactor-mock-deployment-backend August 21, 2024 19:47

noliveleger changed the base branch from kobocat-django-app-part-2-refactor-mock-deployment-backend to beta-refactored August 26, 2024 15:05

This was referenced Aug 27, 2024

Ensure xml_hash is not null and stop doing full-text search kobotoolbox/kobocat#843

Closed

Remove formatting restrictions on <instanceID> (submission UUID) strings kobotoolbox/kobocat#861

Closed

noliveleger requested changes Aug 27, 2024

View reviewed changes

noliveleger reviewed Aug 27, 2024

View reviewed changes

kobo/apps/openrosa/libs/utils/logger_tools.py Outdated Show resolved Hide resolved

Merge branch 'beta-refactored' of github.com:kobotoolbox/kpi into 862…

7630294

…-reject_duplicate_submissions

rajpatel24 force-pushed the 862-reject_duplicate_submissions branch 2 times, most recently from bf597d5 to 2958052 Compare September 3, 2024 10:51

Reject duplicate submissions and improve UUID extraction

f3c89f6

- Add test case for duplicate submission with an attachment - Improve logic to extract UUID from xml - Add logic to reject submission without UUID

rajpatel24 force-pushed the 862-reject_duplicate_submissions branch from 2958052 to f3c89f6 Compare September 3, 2024 17:29

rajpatel24 requested a review from noliveleger September 3, 2024 18:32

noliveleger requested changes Sep 3, 2024

View reviewed changes

noliveleger added 2 commits September 3, 2024 16:58

Merge branch 'beta-refactored' into 862-reject_duplicate_submissions

74dab52

Merge branch 'beta-refactored' into 862-reject_duplicate_submissions

37ed595

noliveleger requested changes Sep 3, 2024

View reviewed changes

kobo/apps/openrosa/apps/logger/tests/test_form_submission.py Outdated Show resolved Hide resolved

rajpatel24 requested a review from noliveleger September 5, 2024 09:55

rajpatel24 added 3 commits September 13, 2024 00:32

Improve submission flow based on the PR feedback

a01bf8f

Merge branch 'beta-refactored' of github.com:kobotoolbox/kpi into 862…

c2fc93e

…-reject_duplicate_submissions

Fix migration conflict: rename and update migration for root_uuid field

53c422b

rajpatel24 requested a review from noliveleger September 13, 2024 19:21

noliveleger and others added 3 commits September 17, 2024 11:57

Merge branch 'beta-refactored' into 862-reject_duplicate_submissions

aa01a6c

Merge branch '862-reject_duplicate_submissions' of github.com:kobotoo…

498ae91

…lbox/kpi into 862-reject_duplicate_submissions

Fix code linter errors

40b0d03

noliveleger requested changes Sep 17, 2024

View reviewed changes

noliveleger reviewed Sep 17, 2024

View reviewed changes

kobo/apps/subsequences/tests/test_submission_stream.py Outdated Show resolved Hide resolved

Address PR feedback and improve submission flow

24a51bf

rajpatel24 force-pushed the 862-reject_duplicate_submissions branch from 648aca5 to 24a51bf Compare September 18, 2024 18:03

noliveleger and others added 3 commits September 18, 2024 16:26

Merge branch 'beta-refactored' into 862-reject_duplicate_submissions

33e2c46

Add test for duplicate submission with identical attachment name but …

81d0b9d

…different content

Merge branch '862-reject_duplicate_submissions' of github.com:kobotoo…

728d0d1

…lbox/kpi into 862-reject_duplicate_submissions

rajpatel24 requested a review from noliveleger September 18, 2024 20:59

rajpatel24 force-pushed the 862-reject_duplicate_submissions branch from 1d6e6ea to e7f73d5 Compare September 19, 2024 14:43

Improve code linting compliance

cc37b6c

rajpatel24 force-pushed the 862-reject_duplicate_submissions branch from e7f73d5 to cc37b6c Compare September 19, 2024 14:44

Update logic to handle edit submissions with identical attachment nam…

cdc71db

…es but different content

rajpatel24 force-pushed the 862-reject_duplicate_submissions branch from b9bf71b to cdc71db Compare September 23, 2024 11:27

noliveleger requested changes Sep 23, 2024

View reviewed changes

kobo/apps/openrosa/apps/logger/models/instance.py Outdated Show resolved Hide resolved

rajpatel24 force-pushed the 862-reject_duplicate_submissions branch from c87f862 to 86e9313 Compare September 27, 2024 20:15

Refactor the command to clean duplicate submissions and add a unique …

306888b

…together constraint

rajpatel24 force-pushed the 862-reject_duplicate_submissions branch from 86e9313 to 306888b Compare September 30, 2024 09:00

rajpatel24 requested a review from noliveleger September 30, 2024 09:47

Merge branch 'beta-refactored' into 862-reject_duplicate_submissions

b3b9b35

noliveleger changed the base branch from beta-refactored to main October 7, 2024 18:31

noliveleger added 2 commits October 7, 2024 15:03

Fix bad merge

c416e42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reject Duplicate Submissions #5047

Reject Duplicate Submissions #5047

rajpatel24 commented Aug 5, 2024 •

edited

Loading

notion-workspace bot commented Sep 3, 2024

Reject Duplicate Submissions #5047

Are you sure you want to change the base?

Reject Duplicate Submissions #5047

Conversation

rajpatel24 commented Aug 5, 2024 • edited Loading

Checklist

Description

Notes

Related issues

notion-workspace bot commented Sep 3, 2024

rajpatel24 commented Aug 5, 2024 •

edited

Loading