Reduce number of DB queries while transferring a project to another user #5142
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
when moving files from one user's storage to another
rgraber
requested changes
Oct 4, 2024
|
|
||
| if media_files_to_update: | ||
| AssetFile.objects.bulk_update( | ||
| media_files_to_update, update_fields=['content', 'metadata'] |
There was a problem hiding this comment.
this should be fields not update_fields
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Try to avoid running one update query per attachment while maintaining the atomicity of the transaction.
Notes
The former choice to run
Attachment.save()right after.move()was done in purpose. The idea was to be sure that the new path of the file was saved in the DB before processing another file. Unfortunately, this created as many DB queries as there were existing attachments (or media files), which was slowing down the process even more. Usingbulk_updateis (way) more efficient. Using thetry/finallyblock should ensure the atomicity if an error occurs while looping on attachments (or media files).resume_failed_transfers_2_024_25_fixhas been revisited. The call of the commandupdate_attachment_storage_byteswas useless because counters were not affected by the bug and were still accurate.Moving files are now wrapped in try/except to avoid the command to crash if one transfer cannot be fixed