Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP]nydus-image: support chunk level deduplication between local images #956

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

mofishzz
Copy link
Collaborator

Add the dedup option to nydus-image to support block level deduplication between local images. A local database cas.db was introduced, which records all chunks saved locally. Then We can redirect the chunks to those that already exist locally to avoid duplicate downloads.

There is still some work needs to be done, including:

  1. Insert the on-demand downloaded chunk into the database to ensure that the database stores locally existing chunks.
  2. Snapshotter's GC mechanism with the reference relationship of bootstrap after deduplication.

Signed-off-by: Huang Jianan [email protected]

Add the dedup option to nydus-image to support block level deduplication
between local images. A local database cas.db was introduced, which records
all chunks saved locally. Then We can redirect the chunks to those that already
exist locally to avoid duplicate downloads.

There is still some work needs to be done, including:
1. Insert the on-demand downloaded chunk into the database to ensure that
the database stores locally existing chunks.
2. Snapshotter's GC mechanism with the reference relationship of bootstrap
after deduplication.

Signed-off-by: Huang Jianan <[email protected]>
@anolis-bot
Copy link
Collaborator

@mofishzz , the title of this PR starts with "WIP" or "[WIP]", so skip testing for now.

xwb1136021767 added a commit to xwb1136021767/image-service that referenced this pull request Aug 8, 2023
Original version from dragonflyoss#956.

Signed-off-by: xwb1136021767 <[email protected]>
Signed-off-by: mofishzz <[email protected]>
xwb1136021767 added a commit to xwb1136021767/image-service that referenced this pull request Aug 9, 2023
Original version from dragonflyoss#956.

Signed-off-by: xwb1136021767 <[email protected]>
Signed-off-by: mofishzz <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants