Import/Export backups of collections in the database #3252
Conversation
| async def export_backups(self, collection_name: str): | ||
| coll = self.db[collection_name] | ||
| documents = [] | ||
| async for document in coll.find(): | ||
| documents.append(document) | ||
| with open(f"{collection_name}.json", "w") as f: | ||
| json.dump(documents, f, cls=CustomJSONEncoder) | ||
| with open(f"{collection_name}.json", "rb") as f: | ||
| file = discord.File(f, f"{collection_name}.json") | ||
| success_message = f"Exported {len(documents)} documents from {collection_name} to JSON. Check your DMs for the file." | ||
| return success_message, file | ||
|
|
||
| async def import_backups(self, collection_name: str, file: discord.File): | ||
| contents = await self.bot.loop.run_in_executor(None, file.fp.read) | ||
| documents = json.loads(contents.decode("utf-8")) | ||
| coll = self.db[collection_name] | ||
| await coll.delete_many({}) | ||
| result = await coll.insert_many(documents) | ||
| success_message = ( | ||
| f"Imported {len(result.inserted_ids)} documents from {file.filename} into {collection_name}." | ||
| ) | ||
| return success_message |
There was a problem hiding this comment.
Instead of JSON, MongoDB stores data using BSON. Perhaps we can do something similar to this gist?
This way, some other native MongoDB types can be saved, and you wouldn't need the custom JSON encoder.
| """ | ||
| Export a backup of a collection in the form of a json file. | ||
|
|
||
| {prefix}export <collection_name> |
There was a problem hiding this comment.
The two usage examples, on L2196 and L2210, are unnecessary, since discord.py auto generates the usage spec. However, it might be useful to note in the import command's help doc that you should attach the backup file to the message.
| @checks.has_permissions(PermissionLevel.ADMINISTRATOR) | ||
| async def export_backup(self, ctx, collection_name): | ||
| """ | ||
| Export a backup of a collection in the form of a json file. |
There was a problem hiding this comment.
As noted in my other comment, we should store the backup as BSON file instead of JSON. Please change the references to BSON files instead.
| @@ -2186,6 +2187,56 @@ async def isenable(self, ctx): | |||
|
|
|||
| return await ctx.send(embed=embed) | |||
|
|
|||
| @commands.command(name="export") | |||
| @checks.has_permissions(PermissionLevel.ADMINISTRATOR) | |||
| async def export_backup(self, ctx, collection_name): | |||
There was a problem hiding this comment.
Instead of exporting a collection, most users doesn't know what even are collections. Could you change it so that it exports everything (into one file) and imports them as well?
There was a problem hiding this comment.
You mean export config/logs and any plugins into 1 bson file? The issue with that would be the bot wont be able to send the large files.
| documents = [] | ||
| async for document in coll.find(): | ||
| documents.append(document) | ||
| with open(f"{collection_name}.json", "w") as f: |
There was a problem hiding this comment.
Don't write to disk, that may lead to potential issues. Use a BytesIO instead.
|
Unable to resolve memory spike issues. If someone else is able to figure out a solution please make a new PR. |
ghost
closed this
Closes #3248
I dont feel the need to have import exports to a git based provider so this PR only works with JSON files sent via discord.