Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add pgzip support #163

Closed
Tracked by #149
wwoytenko opened this issue Aug 16, 2024 · 2 comments · Fixed by #164
Closed
Tracked by #149

feat: Add pgzip support #163

wwoytenko opened this issue Aug 16, 2024 · 2 comments · Fixed by #164
Assignees
@wwoytenko
Labels
enhancement New feature or request

Comments

@wwoytenko
Copy link
Contributor

wwoytenko commented Aug 16, 2024 •
edited
Loading

It was noted that gzip writes are quite slow, and it would be beneficial to have an option to configure parallelism for compression.

github.com/klauspost/pgzip

DOD:

  • Conduct tests to evaluate the performance gains from compressions parallelism.
  • Implement compressions parallelism if the performance improvement is significant.
@wwoytenko wwoytenko added the enhancement New feature or request label Aug 16, 2024
@wwoytenko wwoytenko self-assigned this Aug 16, 2024
@wwoytenko wwoytenko mentioned this issue Aug 16, 2024
Closed
11 tasks
wwoytenko added a commit that referenced this issue Aug 16, 2024
@wwoytenko
feat: Add pgzip support #163
d44b0dc 
* Added support for pgzip in restorers and dumpers
* Introduced --pgzip parameters in dump and restore command
* Refactored GzipWriter
* Added gzip support for data dumpers
* renamed package from countwriter to ioutils
@wwoytenko wwoytenko mentioned this issue Aug 16, 2024
Merged
@wwoytenko
Copy link
Contributor Author

According to issue #164, it significantly decreases dump time. However, note that CPU utilization increases on both the host running the grenmask utility and the PostgreSQL host where the dump is being taken.

@wwoytenko
Copy link
Contributor Author

The table with 20Gb was dumped 4-5 times faster

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wwoytenko wwoytenko

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Feat: dump pgzip support GreenmaskIO/greenmask
1 participant
@wwoytenko