-
Notifications
You must be signed in to change notification settings - Fork 60
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bucket file layout #189
Comments
Looks mostly good. A few comments:
|
Yeah fair. Maybe just
This is mostly dependent on coreos/coreos-assembler#463; though I guess how the various archful builds are laid out are related but somewhat independent of how they get there, so we could strawman that part now too. |
Yeah - a concrete example (like the one in your description) is the best way I can think to actually hash it out. My proposal is we simply add arch directories under the buildid:
|
forgot to respond to this: 👍 |
OK, updated initial comment with those modifications! |
Couple of questions I need answered for the
|
Note that if the |
Few things need to understand to work on Stream Metadata Generator:
|
Yes, I think that's correct.
Whoops, I had missed the |
Makes sense, thanks @jlebon |
The current plan is not to use |
Ahh yeah, right. The
Hmm, seems like this should be a per-stream thing, right? So maybe at |
We don't have another name. "Release index" works for me. And yes, per-stream. |
@jlebon For the individual build release metadata right now the diagram seems to indicate one per architecture, is this correct? Or should there be one per build version (e.x.: |
By looking at json layout for release index at #98 (comment) , how does one know rwhich release metadata: is the latest one? Not sure if having OSTree version is enough to find that information. |
Yeah good point, I think that's indeed what we want so that it's a single file representing the whole build. Amended first comment! So thinking more on this, in #98 (comment) regarding
I think I would agree with this. Let |
From discussions w/ @jlebon & @bgilbert we're going to move forward with a single bucket and just modify the object permissions to be publicly readable as part of the release process. |
Everything except the stream metadata builder should be getting that info from the stream metadata. The metadata builder could look at:
3 seems easiest, and I don't think it has substantial downsides? |
I like option 3 as well, don't see any downside if we keep convention that latest release metadata info gets always appended in the index file of release metadata. |
As discussed in coreos#189.
As discussed in coreos#189.
Officially proposed in #208. (Not that we can't make tweaks later on of course). |
As discussed in coreos#189.
As discussed in coreos#189.
Add a new mode which allows cosa to manipulate multi-arch build layouts: ``` $ find builds builds builds/builds.json builds/30.1 builds/30.1/x86_64 builds/30.1/x86_64/coreos-assembler-config.tar.gz builds/30.1/x86_64/coreos-assembler-config-git.json builds/30.1/x86_64/fedora-coreos-30.1-qemu.qcow2 ... ``` A pipeline could e.g. dispatch builds for each architecture on different nodes, then group them back into a single workdir and have it manipulated by e.g. `buildupload` seamlessly. This new layout also matches the bucket layout for FCOS (see coreos/fedora-coreos-tracker#189). The basic idea is to add a `schema-version` to `builds.json` and denote the legacy behaviour as "pre-1.0.0", while `1.0.0` contains a different schema: each element in the `builds` array is now an object, which has an `id`, and a list of `archs` for which that build has been built: ``` $ cat builds/builds.json { "schema-version": "1.0.0", "builds": [ { "id": "30.1", "archs": [ "x86_64" ] } ], "timestamp": "2019-06-28T20:50:54Z" } ``` We retain backwards-compatibility by simply checking the schema version. Right now, only new workdirs will have this layout. Pipelines which use `buildprep` will fetch `builds.json` as is and key off of its contents to determine the bucket layout as well. We can write new code in the future to convert previously single-arch buckets into the new layout to then enable multi-arch.
Add a new mode which allows cosa to manipulate multi-arch build layouts: ``` $ find builds builds builds/builds.json builds/30.1 builds/30.1/x86_64 builds/30.1/x86_64/coreos-assembler-config.tar.gz builds/30.1/x86_64/coreos-assembler-config-git.json builds/30.1/x86_64/fedora-coreos-30.1-qemu.qcow2 ... ``` A pipeline could e.g. dispatch builds for each architecture on different nodes, then group them back into a single workdir and have it manipulated by e.g. `buildupload` seamlessly. This new layout also matches the bucket layout for FCOS (see coreos/fedora-coreos-tracker#189). The basic idea is to add a `schema-version` to `builds.json` and denote the legacy behaviour as "pre-1.0.0", while `1.0.0` contains a different schema: each element in the `builds` array is now an object, which has an `id`, and a list of `archs` for which that build has been built: ``` $ cat builds/builds.json { "schema-version": "1.0.0", "builds": [ { "id": "30.1", "archs": [ "x86_64" ] } ], "timestamp": "2019-06-28T20:50:54Z" } ``` We retain backwards-compatibility by simply checking the schema version. Right now, only new workdirs will have this layout. Pipelines which use `buildprep` will fetch `builds.json` as is and key off of its contents to determine the bucket layout as well. We can write new code in the future to convert previously single-arch buckets into the new layout to then enable multi-arch.
Add a new mode which allows cosa to manipulate multi-arch build layouts: ``` $ find builds builds builds/builds.json builds/30.1 builds/30.1/x86_64 builds/30.1/x86_64/coreos-assembler-config.tar.gz builds/30.1/x86_64/coreos-assembler-config-git.json builds/30.1/x86_64/fedora-coreos-30.1-qemu.qcow2 ... ``` A pipeline could e.g. dispatch builds for each architecture on different nodes, then group them back into a single workdir and have it manipulated by e.g. `buildupload` seamlessly. This new layout also matches the bucket layout for FCOS (see coreos/fedora-coreos-tracker#189). The basic idea is to add a `schema-version` to `builds.json` and denote the legacy behaviour as "pre-1.0.0", while `1.0.0` contains a different schema: each element in the `builds` array is now an object, which has an `id`, and a list of `archs` for which that build has been built: ``` $ cat builds/builds.json { "schema-version": "1.0.0", "builds": [ { "id": "30.1", "archs": [ "x86_64" ] } ], "timestamp": "2019-06-28T20:50:54Z" } ``` We retain backwards-compatibility by simply checking the schema version. Right now, only new workdirs will have this layout. Pipelines which use `buildprep` will fetch `builds.json` as is and key off of its contents to determine the bucket layout as well. We can write new code in the future to convert previously single-arch buckets into the new layout to then enable multi-arch.
Add a new mode which allows cosa to manipulate multi-arch build layouts: ``` $ find builds builds builds/builds.json builds/30.1 builds/30.1/x86_64 builds/30.1/x86_64/coreos-assembler-config.tar.gz builds/30.1/x86_64/coreos-assembler-config-git.json builds/30.1/x86_64/fedora-coreos-30.1-qemu.qcow2 ... ``` A pipeline could e.g. dispatch builds for each architecture on different nodes, then group them back into a single workdir and have it manipulated by e.g. `buildupload` seamlessly. This new layout also matches the bucket layout for FCOS (see coreos/fedora-coreos-tracker#189). The basic idea is to add a `schema-version` to `builds.json` and denote the legacy behaviour as "pre-1.0.0", while `1.0.0` contains a different schema: each element in the `builds` array is now an object, which has an `id`, and a list of `archs` for which that build has been built: ``` $ cat builds/builds.json { "schema-version": "1.0.0", "builds": [ { "id": "30.1", "archs": [ "x86_64" ] } ], "timestamp": "2019-06-28T20:50:54Z" } ``` We retain backwards-compatibility by simply checking the schema version. Right now, only new workdirs will have this layout. Pipelines which use `buildprep` will fetch `builds.json` as is and key off of its contents to determine the bucket layout as well. We can write new code in the future to convert previously single-arch buckets into the new layout to then enable multi-arch.
Add a new mode which allows cosa to manipulate multi-arch build layouts: ``` $ find builds builds builds/builds.json builds/30.1 builds/30.1/x86_64 builds/30.1/x86_64/coreos-assembler-config.tar.gz builds/30.1/x86_64/coreos-assembler-config-git.json builds/30.1/x86_64/fedora-coreos-30.1-qemu.qcow2 ... ``` A pipeline could e.g. dispatch builds for each architecture on different nodes, then group them back into a single workdir and have it manipulated by e.g. `buildupload` seamlessly. This new layout also matches the bucket layout for FCOS (see coreos/fedora-coreos-tracker#189). The basic idea is to add a `schema-version` to `builds.json` and denote the legacy behaviour as "pre-1.0.0", while `1.0.0` contains a different schema: each element in the `builds` array is now an object, which has an `id`, and a list of `archs` for which that build has been built: ``` $ cat builds/builds.json { "schema-version": "1.0.0", "builds": [ { "id": "30.1", "archs": [ "x86_64" ] } ], "timestamp": "2019-06-28T20:50:54Z" } ``` We retain backwards-compatibility by simply checking the schema version. Right now, only new workdirs will have this layout. Pipelines which use `buildprep` will fetch `builds.json` as is and key off of its contents to determine the bucket layout as well. We can write new code in the future to convert previously single-arch buckets into the new layout to then enable multi-arch.
Add a new mode which allows cosa to manipulate multi-arch build layouts: ``` $ find builds builds builds/builds.json builds/30.1 builds/30.1/x86_64 builds/30.1/x86_64/coreos-assembler-config.tar.gz builds/30.1/x86_64/coreos-assembler-config-git.json builds/30.1/x86_64/fedora-coreos-30.1-qemu.qcow2 ... ``` A pipeline could e.g. dispatch builds for each architecture on different nodes, then group them back into a single workdir and have it manipulated by e.g. `buildupload` seamlessly. This new layout also matches the bucket layout for FCOS (see coreos/fedora-coreos-tracker#189). The basic idea is to add a `schema-version` to `builds.json` and denote the legacy behaviour as "pre-1.0.0", while `1.0.0` contains a different schema: each element in the `builds` array is now an object, which has an `id`, and a list of `archs` for which that build has been built: ``` $ cat builds/builds.json { "schema-version": "1.0.0", "builds": [ { "id": "30.1", "archs": [ "x86_64" ] } ], "timestamp": "2019-06-28T20:50:54Z" } ``` We retain backwards-compatibility by simply checking the schema version. Right now, only new workdirs will have this layout. Pipelines which use `buildprep` will fetch `builds.json` as is and key off of its contents to determine the bucket layout as well. We can write new code in the future to convert previously single-arch buckets into the new layout to then enable multi-arch.
Discussed OOB: we should move the stream metadata to a separate directory (e.g. |
As discussed in coreos#189.
Resolved in #208. |
Early in FCOS development, we were still debating whether the S3 objects should be public or not. There was some initial agreement on uploading first as private and then make specific builds public during the release process, but in the end we've just always published in public from the start. It's not likely we'll change this anytime soon unless a good reason comes up. So let's just delete the code here that makes the build objects public since they already are. It'll live on in git at least if we ever want to restore it. Related: coreos/fedora-coreos-tracker#189
Early in FCOS development, we were still debating whether the S3 objects should be public or not. There was some initial agreement on uploading first as private and then make specific builds public during the release process, but in the end we've just always published in public from the start. It's not likely we'll change this anytime soon unless a good reason comes up. So let's just delete the code here that makes the build objects public since they already are. It'll live on in git at least if we ever want to restore it. Related: coreos/fedora-coreos-tracker#189
Early in FCOS development, we were still debating whether the S3 objects should be public or not. There was some initial agreement on uploading first as private and then make specific builds public during the release process, but in the end we've just always published in public from the start. It's not likely we'll change this anytime soon unless a good reason comes up. So let's just delete the code here that makes the build objects public since they already are. It'll live on in git at least if we ever want to restore it. Related: coreos/fedora-coreos-tracker#189
Early in FCOS development, we were still debating whether the S3 objects should be public or not. There was some initial agreement on uploading first as private and then make specific builds public during the release process, but in the end we've just always published in public from the start. It's not likely we'll change this anytime soon unless a good reason comes up. So let's just delete the code here that makes the build objects public since they already are. It'll live on in git at least if we ever want to restore it. Related: coreos/fedora-coreos-tracker#189
Early in FCOS development, we were still debating whether the S3 objects should be public or not. There was some initial agreement on uploading first as private and then make specific builds public during the release process, but in the end we've just always published in public from the start. It's not likely we'll change this anytime soon unless a good reason comes up. So let's just delete the code here that makes the build objects public since they already are. It'll live on in git at least if we ever want to restore it. Related: coreos/fedora-coreos-tracker#189 (cherry picked from commit ef8faa0)
Early in FCOS development, we were still debating whether the S3 objects should be public or not. There was some initial agreement on uploading first as private and then make specific builds public during the release process, but in the end we've just always published in public from the start. It's not likely we'll change this anytime soon unless a good reason comes up. So let's just delete the code here that makes the build objects public since they already are. It'll live on in git at least if we ever want to restore it. Related: coreos/fedora-coreos-tracker#189 (cherry picked from commit ef8faa0)
Early in FCOS development, we were still debating whether the S3 objects should be public or not. There was some initial agreement on uploading first as private and then make specific builds public during the release process, but in the end we've just always published in public from the start. It's not likely we'll change this anytime soon unless a good reason comes up. So let's just delete the code here that makes the build objects public since they already are. It'll live on in git at least if we ever want to restore it. Related: coreos/fedora-coreos-tracker#189 (cherry picked from commit ef8faa0)
Early in FCOS development, we were still debating whether the S3 objects should be public or not. There was some initial agreement on uploading first as private and then make specific builds public during the release process, but in the end we've just always published in public from the start. It's not likely we'll change this anytime soon unless a good reason comes up. So let's just delete the code here that makes the build objects public since they already are. It'll live on in git at least if we ever want to restore it. Related: coreos/fedora-coreos-tracker#189 (cherry picked from commit ef8faa0)
Early in FCOS development, we were still debating whether the S3 objects should be public or not. There was some initial agreement on uploading first as private and then make specific builds public during the release process, but in the end we've just always published in public from the start. It's not likely we'll change this anytime soon unless a good reason comes up. So let's just delete the code here that makes the build objects public since they already are. It'll live on in git at least if we ever want to restore it. Related: coreos/fedora-coreos-tracker#189 (cherry picked from commit ef8faa0)
Early in FCOS development, we were still debating whether the S3 objects should be public or not. There was some initial agreement on uploading first as private and then make specific builds public during the release process, but in the end we've just always published in public from the start. It's not likely we'll change this anytime soon unless a good reason comes up. So let's just delete the code here that makes the build objects public since they already are. It'll live on in git at least if we ever want to restore it. Related: coreos/fedora-coreos-tracker#189 (cherry picked from commit ef8faa0)
Early in FCOS development, we were still debating whether the S3 objects should be public or not. There was some initial agreement on uploading first as private and then make specific builds public during the release process, but in the end we've just always published in public from the start. It's not likely we'll change this anytime soon unless a good reason comes up. So let's just delete the code here that makes the build objects public since they already are. It'll live on in git at least if we ever want to restore it. Related: coreos/fedora-coreos-tracker#189 (cherry picked from commit ef8faa0)
Early in FCOS development, we were still debating whether the S3 objects should be public or not. There was some initial agreement on uploading first as private and then make specific builds public during the release process, but in the end we've just always published in public from the start. It's not likely we'll change this anytime soon unless a good reason comes up. So let's just delete the code here that makes the build objects public since they already are. It'll live on in git at least if we ever want to restore it. Related: coreos/fedora-coreos-tracker#189 (cherry picked from commit ef8faa0)
Early in FCOS development, we were still debating whether the S3 objects should be public or not. There was some initial agreement on uploading first as private and then make specific builds public during the release process, but in the end we've just always published in public from the start. It's not likely we'll change this anytime soon unless a good reason comes up. So let's just delete the code here that makes the build objects public since they already are. It'll live on in git at least if we ever want to restore it. Related: coreos/fedora-coreos-tracker#189 (cherry picked from commit ef8faa0)
How should our various files (metadata & build artifacts) be laid out in the bucket? Note this isn't something that we expect users to care about. Some pieces of published metadata will hold links to files into the bucket, but we should be able to change the bucket layout itself without impacting users. As such, we should also strongly discourage perusing the bucket (which without indexing it would be cumbersome anyway) or predicting locations for new builds.
Here's a strawman based on some initial thoughts in coreos/coreos-assembler#159:
The text was updated successfully, but these errors were encountered: