Stream metadata generator

[bgilbert]

Write a tool which consumes release metadata and an override file, produces stream metadata, and perhaps uploads it. This could be in cosa, or mantle, or elsewhere.

We'll need to decide where the override file should be stored. It should probably be in version control, and we should probably record the history of the stream metadata somewhere.

[jlebon]

and we should probably record the history of the stream metadata somewhere.

Just some random thoughts on this from discussions we've had on the subject. Accessible history is important for Cincinnati to build its graph. That's a separate use case from "archival" purposes though. E.g. I can imagine only keeping the subset relevant to Cincinnati somewhere in the bucket (basically just OSTree commit & version, right?), whereas we keep the full history of all changes to stream.json in git for posterity.

[bgilbert]

My understanding is that Cincinnati will build its graph from the release metadata, and will only use the stream metadata to determine the current target version for upgrades. So stream metadata history would just be for posterity.

[sinnykumari]

My understanding is that Cincinnati will build its graph from the release metadata, and will only use the stream metadata to determine the current target version for upgrades. So stream metadata history would just be for posterity.

For curiosity, what are the use-cases of having stream metadata history?

[bgilbert]

@sinnykumari People might sometimes need to find out why a machine ended up running a particular version of the OS. But mostly there isn't a concrete use case; it just seems wrong to have an important, mutable user-facing data source without any recorded history.

[sinnykumari]

Summary from community meeting discussion:
We leaned towards keeping both override file and stream history in same repo. Repo name can be coreos/fedora-coreos-streams
We came up with two approaches to update override file and stream history:

1. Have two branches - override and historical. This way we maintain git history independently
   OR,
2. Keep both override and historical content in same branch. In this approach, we open a PR against repo for a change in the override file. bot detects PR and updates it with processed output (override content from PR + updated stream.json), then merge (manually/through bot). During merge, bot will also upload stream output where FCOS artifacts are available. As a result stream-metadata-generator won't have to be responsible for stream metadata upload and auto-generated updates will also go through PR.

We mostly leaned towards option 2

[bgilbert]

Created coreos/fedora-coreos-streams.

[sinnykumari]

If we go through option 2, I thought of following directory/file layout for https://github.com/coreos/fedora-coreos-streams repo to keep override and historical stream file for respective streams.

├── stable
│   ├── override.json
│   └── stream.json
├── testing
│   ├── override.json
│   └── stream.json
└── testing-devel
    ├── override.json
    └── stream.json
...

[sinnykumari]

@bgilbert @jlebon @lucab @arithx does file layout for fedora-coreos-streams repo in #193 (comment) looks reasonable?

[jlebon]

@sinnykumari Yup, SGTM!

[bgilbert]

@sinnykumari SGTM!

[sinnykumari]

With the bucket layout change in #208 , We might want to rename corresponding stream.json proposed in #193 (comment) to below:

├── stable
│   ├── override.json
│   └── stable.json
├── testing
│   ├── override.json
│   └── testing.json
└── testing-devel
    ├── override.json
    └── testing-devel.json
...

[jlebon]

Hmm, we could even take it a step further and make it:

/streams
  /stable.json
  /testing.json
  ...
/overrides
  /stable.json
  /testing.json
  ...

?

Then the upload step would just be aws s3 sync streams/ s3://fcos-builds/streams/. Which is harder to screw up (and since we'll be running it by hand at first, that's a nice property to have :) ).

[sinnykumari]

SGTM

[bgilbert]

This is long since done in https://github.com/coreos/fedora-coreos-stream-generator/.