Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

K8s job for using rclone to copy buckets instead of thanos-replicate #582

Merged
merged 2 commits into from
Aug 30, 2023
Merged

K8s job for using rclone to copy buckets instead of thanos-replicate #582

merged 2 commits into from
Aug 30, 2023

Conversation

moadz
Copy link
Contributor

@moadz moadz commented Aug 27, 2023

saswatamcode
saswatamcode approved these changes Aug 28, 2023
View reviewed changes
Copy link
Member

@saswatamcode saswatamcode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! What limitations did you notice with replicate?

philipgough
philipgough reviewed Aug 29, 2023
View reviewed changes
resources/operations/rclone-bucket-replicate/job-template.yaml
- name: NAMESPACE
description: The namespace where the Job should run.
value: 'observatorium-operations'
- name: IMAGE_TAG
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this still points at latest Thanos so we should fix that and pass it in down below instead of latest

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@moadz
* Adding rclone config template for logging/concurrency
455b413 
* Fixed default image tag

Signed-off-by: mzardab <[email protected]>
@moadz
Copy link
Contributor Author

moadz commented Aug 30, 2023

@saswatamcode did you notice with replicate? Usually not an issue for block storage syncing, but for en-masse migration it's quite slow.

We also don't support parallel uploaders (which would be a good feature if threadsafe):
This: observatorium/thanos-replicate#22

@moadz moadz merged commit 24d37fb into rhobs:main Aug 30, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@philipgough philipgough philipgough approved these changes

@saswatamcode saswatamcode saswatamcode approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@moadz @philipgough @saswatamcode