write script to send fedmsg when we want a release artifact signed #198

dustymabe · 2019-06-07T17:29:35Z

As part of the Fedora Signing Discussion it was identified that we need to send a message when we want to get our artifacts signed in preparation for a release. The payload of the message needs to include the artifacts as well as the checksums (so the signer can check before creating signature). We believe the meta.json should suffice as the payload.

This script would need to be executed somewhere in our release process, or perhaps any time a release candidate becomes available so that all signatures are in place well in time before we do a release.

The text was updated successfully, but these errors were encountered:

puiterwijk · 2019-06-07T19:04:26Z

Note that fedmsg is being replaced with fedora-messaging.
I think you should use the latter for any new code.

dustymabe · 2019-06-07T19:06:43Z

Note that fedmsg is being replaced with fedora-messaging.
I think you should use the latter for any new code.

Indeed that is defnitely the plan. The problem I have when describing fedora messaging messages is that there isn't a succinct way to do it like there was with fedmsg. I've just still been calling them fedmsgs because people know what you mean when you say it.

dustymabe · 2019-06-25T17:04:03Z

After some recent discussions there are actually a few different messages we probably want to send. Here is a proposal for now:

org.fedoraproject.prod.coreos.ostree
- gives information about an ostree commit that was just created
- can be used to trigger signing an ostree commit
- similar to pungi-compose-ostree message today.
org.fedoraproject.prod.coreos.build.status.change
- shares information about status of the current coreos build in jenkins (moving through stages of pipeline, etc)
- similar to pungi-compose-status-change
- should deliver the information in the meta.json from a fedora coreos pipeline run
- can be used to trigger artifact signing

Any other ones I'm missing?

jlebon · 2019-06-25T18:23:35Z

org.fedoraproject.prod.coreos.ostree

Maybe let's split this out into two, e.g. org.fedoraproject.prod.coreos.ostree.signing-request and org.fedoraproject.prod.coreos.ostree? The former is for #200; signing an OSTree commit during the build process so we can include it in the image artifacts. The latter would be when the OSTree commit is available in the bucket proper, and signalling to releng to pull it into the unirepo?

Actually, we could also fold the signing-request one into org.fedoraproject.prod.coreos.build.status.change messages. Not sure if that's harder for releng though.

dustymabe · 2019-06-25T18:43:25Z

org.fedoraproject.prod.coreos.ostree

Maybe let's split this out into two, e.g. org.fedoraproject.prod.coreos.ostree.signing-request and org.fedoraproject.prod.coreos.ostree? The former is for #200; signing an OSTree commit during the build process so we can include it in the image artifacts. The latter would be when the OSTree commit is available in the bucket proper, and signalling to releng to pull it into the unirepo?

Actually, we could also fold the signing-request one into org.fedoraproject.prod.coreos.build.status.change messages. Not sure if that's harder for releng though.

yeah maybe we should just do that for all of these and re-use the state change messages. Actually I just looked a bit closer and the pungi messages have a status change and a phase change. I guess a new proposal would be:

org.fedoraproject.prod.coreos.build.stage.start
- shares information about the COSA build moving through the pipeline
- similar to pungi-compose-phase-start
- should deliver the information in the meta.json from a fedora coreos pipeline run
- can be used to trigger ostree commit signing
- can be used to trigger artifact signing
- can be used to trigger transfer ostree to unified repo
org.fedoraproject.prod.coreos.build.status.change
- shares information about status of the current coreos COSA build in jenkins (i.e. whether it failed or not)
- similar to pungi-compose-status-change
- should deliver the information in the meta.json from a fedora coreos pipeline run

jlebon · 2019-06-25T19:17:33Z

So org.fedoraproject.prod.coreos.build.status.change would only actually be emitted at the end of builds with the final success/failure result? Minor bikeshed, though ISTM like those should be called e.g. org.fedoraproject.prod.coreos.build.complete or something if that's the case.

Anyway, SGTM!

dustymabe · 2019-06-25T19:19:38Z

So org.fedoraproject.prod.coreos.build.status.change would only actually be emitted at the end of builds with the final success/failure result?

Yeah when it starts and when it finishes (or fails).

I'll take this to fedora infra and see if they have any input.

dustymabe · 2019-06-25T20:49:31Z

I'll take this to fedora infra and see if they have any input.

https://pagure.io/fedora-infrastructure/issue/7940

jlebon · 2019-07-22T20:37:05Z

I have a suggestion: I think we should make more explicit the difference between informational messages ("if anyone cares, our build has reached state $x") from messages requesting direct action from releng ("please do $x for me; I'll be waiting").

For informational events we can start with:

build started
build ended

And we can add more messages in the future about intermediate states if useful.

For requests, there are three main ones:

OSTree commit signing
Build artifact signing
- These two requests would ideally happen on every build (see signing ostree commits #200 (comment)). In theory, build artifact signing could happen as part of a release only, but signing every build I think would help with e2e testing.
OSTree commit import into unified repo
- This request will only happen during a release.

Strawman for informational events:

topic: org.fedoraproject.prod.coreos.build.status.change
message:
    build_id: ${build_id}
    stream: ${stream}
    basearch: ${basearch}
    status: (STARTED|FINISHED|...)
    build_dir: s3://.../${build_id}/${basearch}/
    # if FINISHED
    result: (SUCCESS|FAILURE)

Strawman for OSTree commit signing request:

topic: org.fedoraproject.prod.coreos.build.request.ostree-sign
message:
    build_id: ${build_id}
    stream: ${stream}
    basearch: ${basearch}
    commit_object: s3://.../${build_id}/${basearch}/ostree-commit-object
    checksum: sha256:<hash>

Strawman for build artifact signing request:

topic: org.fedoraproject.prod.coreos.build.request.artifacts-sign
message:
    build_id: ${build_id}
    stream: ${stream}
    basearch: ${basearch}
    artifacts:
        - file: s3://.../${build_id}/${basearch}/file1
          checksum: sha256:<hash>
        - file: s3://.../${build_id}/${basearch}/file2
          checksum: sha256:<hash>
        ...

Strawman for OSTree commit import requests:

topic: org.fedoraproject.prod.coreos.build.request.ostree-import
message:
    build_id: ${build_id}
    stream: ${stream}
    basearch: ${basearch}
    commit: s3://.../ostree-commit.tar
    # checksum for integrity checking the tarball; but note the commit inside will already be signed
    checksum: sha256:<hash>
    ostree_ref: ${ref}
    ostree_checksum: <SHA256>
    target_repo: <compose|prod>

And we'll need matching messages from Fedora infra to signal to us when the request was fulfilled, maybe just reflecting back the same message but on topic $topic.finished with header status: (SUCCESS|FAILURE)?

Does that look sane?
/cc @bgilbert @sinnykumari @puiterwijk

bgilbert · 2019-07-22T20:48:01Z

For org.fedoraproject.prod.coreos.build.request.artifacts-sign, do we want release.json rather than meta.json? Or is that not physically possible?

LGTM otherwise.

jlebon · 2019-07-23T13:00:54Z

For org.fedoraproject.prod.coreos.build.request.artifacts-sign, do we want release.json rather than meta.json? Or is that not physically possible?

Actually, I think it might be cleaner to just go the other and give the explicit list of artifacts we want signatures for. The advantage is that code on the infra side doesn't need to learn how to read meta.json or release.json and doesn't need to be changed when we add a new artifact type.

I amended my proposal above!

abompard · 2019-07-23T13:35:47Z

I'm not sure you actually need to use the message headers, I think those values would be more suited for the message body. Consumers of fedora-messaging messages don't usually look at message headers, unless they're doing something advanced or meta (like comparing messages, generating notifications, etc).

jlebon · 2019-07-23T14:09:28Z

I'm not sure you actually need to use the message headers, I think those values would be more suited for the message body.

My understanding was that header fields can be selected on, so that consumers can be more specific in the event they're interested in. Is that correct? For example, when waiting for org.fedoraproject.prod.coreos.build.request.artifacts-sign.finished from releng, we can also select for the right build_id, rather than having to process the body itself to find out.

(My experience with the message bus mostly comes from fedmsg + https://github.com/jenkinsci/jms-messaging-plugin, but the internal broker that proxied fedmsgs had some magic which pulled out fields from the body and put them into headers so Jenkins jobs could trigger on them more accurately.)

Consumers of fedora-messaging messages don't usually look at message headers, unless they're doing something advanced or meta

Gotcha. Don't want to go against the convention if that's what most fedora-messaging msgs do.

sinnykumari · 2019-07-23T14:15:48Z

Agree with what @abompard said, consumers generally uses content from msg section. So, all needed information which consumers may need should be in message section.

Few additional comment on proposal at #198 (comment) :

To keep consistent with other messaging topic, let's use only dot(.) in topic name. Like rename topic org.fedoraproject.prod.coreos.build.request.ostree-import to org.fedoraproject.prod.coreos.build.request.ostree.import
I think we don't need to send topic org.fedoraproject.prod.coreos.build.request.ostree-sign. Consumer can listen to topic org.fedoraproject.prod.coreos.build.status.change and sign ostree commit when they are in FINISHED state, otherwise do nothing.

jlebon · 2019-07-23T14:28:55Z

Agree with what @abompard said, consumers generally uses content from msg section. So, all needed information which consumers may need should be in message section.

I can change this, I just want more confirmation about the implications of doing this as mentioned in #198 (comment).

To keep consistent with other messaging topic, let's use only dot(.) in topic name

Sure, done!

I think we don't need to send topic org.fedoraproject.prod.coreos.build.request.ostree-sign. Consumer can listen to topic org.fedoraproject.prod.coreos.build.status.change and sign ostree commit when they are in FINISHED state, otherwise do nothing.

OSTree commits need to be signed during the build, not at the end only, because they need to be included into the image artifacts (see discussions in #200). So we need to explicitly wait for the request to be fulfilled before we can go ahead with the rest of the build.

sinnykumari · 2019-07-23T14:37:33Z

I can change this, I just want more confirmation about the implications of doing this as mentioned in #198 (comment).

+1

OSTree commits need to be signed during the build, not at the end only, because they need to be included into the image artifacts (see discussions in #200). So we need to explicitly wait for the request to be fulfilled before we can go ahead with the rest of the build.

sgtm

abompard · 2019-07-23T15:05:17Z

My understanding was that header fields can be selected on, so that consumers can be more specific in the event they're interested in. Is that correct? For example, when waiting for org.fedoraproject.prod.coreos.build.request.artifacts-sign.finished from releng, we can also select for the right build_id, rather than having to process the body itself to find out.

The broker we use has that feature but we generally don't make use of it in our wrapper library, because it would make the binding phase much more complex. So I would recommend using the topic as the main filter and filtering further in the code itself.

jlebon · 2019-07-23T15:10:19Z

OK, updated to just fold headers into the message body!
(Also note that the actual bodies will be in JSON, not YAML as shown here).

abompard · 2019-07-23T15:41:37Z

Thanks!

jlebon · 2019-07-24T13:05:21Z

This is now blocked on https://pagure.io/fedora-infrastructure/issue/7940#comment-584414.

dustymabe · 2019-09-17T20:34:35Z

Strawman for OSTree commit import requests:

topic: org.fedoraproject.prod.coreos.build.request.ostree-import
message:
   build_id: ${build_id}
   stream: ${stream}
   basearch: ${basearch}
   commit: s3://.../ostree-commit.tar
   # checksum for integrity checking the tarball; but note the commit inside will already be signed
   checksum: <SHA256>

If we end up using this for both the annex repo and the prod repo we should maybe add a field that specifies which repo we want to import into.

jlebon · 2019-09-18T13:26:46Z

If we end up using this for both the annex repo and the prod repo we should maybe add a field that specifies which repo we want to import into.

So if we use the annex for storing all the OSTree streams, then at release time, should we instead have a different message for "pull-local this OSTree ref from the compose repo into prod" ?

dustymabe · 2019-09-19T14:24:46Z

So if we use the annex for storing all the OSTree streams, then at release time, should we instead have a different message for "pull-local this OSTree ref from the compose repo into prod" ?

Yeah. Except I think we can just use the same message and just make the code:

pull commit hash from fedora message
check the compose repo for the commit
if exists in compose repo then pull-local from there
if not then pull from s3

WDYT?

Add a new `cosa sign` top-level command for general signing operations. For now, it only supports one subcommand, `robosignatory`, which does signing by RoboSignatory via fedora-messaging (see [1][2] for details). Essentially with this, the pipeline flow will be: ``` $ cosa build ostree $ cosa sign robosignatory --ostree ... $ cosa buildextend-* $ cosa buildupload $ cosa sign robosignatory --images ... ``` [1] coreos/fedora-coreos-tracker#198 [2] coreos/fedora-coreos-tracker#200

jlebon · 2019-09-19T19:10:26Z

OK, I added the following field to the ostree-import message:

    target_repo: <compose|prod>

Does that work?

I opted for symbolic "compose" and "prod" names rather than hurling internal Fedora infra URLs at itself. (Of course, this is all still conditional on whether we can use the compose repo for this, right? Or did you get an answer about this?)

dustymabe · 2019-09-19T20:40:49Z

 target_repo: <compose|prod>
Does that work?

yep. can we also add the commit hash ?

I opted for symbolic "compose" and "prod" names rather than hurling internal Fedora infra URLs at itself. (Of course, this is all still conditional on whether we can use the compose repo for this, right? Or did you get an answer about this?)

I think that sounds good. I believe we can use the compose repo, but I'll get confirmation.

dustymabe · 2019-09-20T19:04:47Z

I opted for symbolic "compose" and "prod" names rather than hurling internal Fedora infra URLs at itself. (Of course, this is all still conditional on whether we can use the compose repo for this, right? Or did you get an answer about this?)

I think that sounds good. I believe we can use the compose repo, but I'll get confirmation.

I talked with @mohanboddu in IRC and he confirmed this should be OK:

14:46:03        dustymabe | nirik: mboddu quick question for you
14:46:24          +mboddu | dustymabe: shoot
14:46:35        dustymabe | we want to import our ostree builds from our composes into the compose ostree repo
                          | (the one pungi uses)
14:47:00        dustymabe | the idea is that we have two ostree repos today, and we'd like to just keep it at two
14:47:39        dustymabe | I just wanted to get explicit acknowledgement from you all that it was OK to do that
14:47:59        dustymabe | there will still be the prod repo where releases get synced to
14:48:51          +mboddu | two ostrees repo = prod repo + compose repo, right?
14:49:02        dustymabe | correct, no change from today
14:49:03          +mboddu | dustymabe: And we thought that we will keep both of them
14:49:25        dustymabe | yes we need to keep both of them because silverblue is still composing into the
                          | compose repo
14:49:38          +mboddu | Yes
14:50:04        dustymabe | mboddu: so you are OK with us importing the FCOS commits into the compose repo?
14:50:52          +mboddu | dustymabe: I guess so, but I am wondering if it at all affects silverblue somehow...?
14:51:17        dustymabe | mboddu: it should not, it will use different branch names, just like atomic host did
14:51:42          +mboddu | Right
14:53:52          +mboddu | But isn't fcos will mix and match different builds that might not be in stable?
14:54:24          +mboddu | Doesn't it affect the the repos?
14:54:44          +mboddu | s/repos/silverblue/
14:59:26        dustymabe | mboddu: the ostree repo is the place we store the already composed ostree commits. The
                          | inputs to composes come from yum repos. Silverblue will continue to pull from the
                          | locations it's currently pulling from
14:59:49        dustymabe | so this won't affect silverblue at all
15:00:03          +mboddu | dustymabe: Okay, then I am good with it
15:00:07        dustymabe | +1 thanks

dustymabe · 2019-09-20T19:35:43Z

in places where we have checksums let's qualify it by including the checksum algorithm:

checksum: <SHA256>

becomes

checksum: 'sha256:aabbccddee....'

WDYT?

jlebon · 2019-09-20T19:38:43Z

in places where we have checksums let's qualify it by including the checksum algorithm:

Yeah, that makes sense. We'll have to update RoboSignatory too.

Add a new `cosa sign` top-level command for general signing operations. For now, it only supports one subcommand, `robosignatory`, which does signing by RoboSignatory via fedora-messaging (see [1][2] for details). Essentially with this, the pipeline flow will be: ``` $ cosa build ostree $ cosa sign robosignatory --ostree ... $ cosa buildextend-* $ cosa buildupload $ cosa sign robosignatory --images ... ``` [1] coreos/fedora-coreos-tracker#198 [2] coreos/fedora-coreos-tracker#200

jlebon · 2019-10-16T16:04:59Z

Fixed by coreos/coreos-assembler#763.

dustymabe added the infra Related to Fedora Infrastructure team work/input label Jun 7, 2019

dustymabe mentioned this issue Jun 7, 2019

Signing for release artifacts #187

Closed

dustymabe mentioned this issue Jun 24, 2019

Create tool for push lockfiles around #205

Closed

jlebon mentioned this issue Sep 5, 2019

develop method for copying ostree content to the unified ostree repo #199

Closed

dustymabe assigned jlebon Sep 5, 2019

jlebon mentioned this issue Sep 19, 2019

Add cosa sign robosignatory for signing OSTrees and images coreos/coreos-assembler#763

Merged

jlebon mentioned this issue Sep 25, 2019

Add coreos-ostree-importer coreos/fedora-coreos-releng-automation#45

Merged

jlebon closed this as completed Oct 16, 2019

jlebon mentioned this issue Dec 17, 2019

Add support for Slack notifications coreos/fedora-coreos-pipeline#166

Merged

jlebon mentioned this issue Aug 20, 2020

Consider sending fedmsg when stream metadata changes #225

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

write script to send fedmsg when we want a release artifact signed #198

write script to send fedmsg when we want a release artifact signed #198

dustymabe commented Jun 7, 2019

puiterwijk commented Jun 7, 2019

dustymabe commented Jun 7, 2019

dustymabe commented Jun 25, 2019 •

edited

Loading

jlebon commented Jun 25, 2019

dustymabe commented Jun 25, 2019

jlebon commented Jun 25, 2019

dustymabe commented Jun 25, 2019

dustymabe commented Jun 25, 2019

jlebon commented Jul 22, 2019 •

edited

Loading

bgilbert commented Jul 22, 2019

jlebon commented Jul 23, 2019

abompard commented Jul 23, 2019

jlebon commented Jul 23, 2019

sinnykumari commented Jul 23, 2019

jlebon commented Jul 23, 2019

sinnykumari commented Jul 23, 2019

abompard commented Jul 23, 2019 •

edited

Loading

jlebon commented Jul 23, 2019

abompard commented Jul 23, 2019

jlebon commented Jul 24, 2019

dustymabe commented Sep 17, 2019

jlebon commented Sep 18, 2019

dustymabe commented Sep 19, 2019

jlebon commented Sep 19, 2019

dustymabe commented Sep 19, 2019

dustymabe commented Sep 20, 2019

dustymabe commented Sep 20, 2019 •

edited

Loading

jlebon commented Sep 20, 2019

jlebon commented Oct 16, 2019

write script to send fedmsg when we want a release artifact signed #198

write script to send fedmsg when we want a release artifact signed #198

Comments

dustymabe commented Jun 7, 2019

puiterwijk commented Jun 7, 2019

dustymabe commented Jun 7, 2019

dustymabe commented Jun 25, 2019 • edited Loading

jlebon commented Jun 25, 2019

dustymabe commented Jun 25, 2019

jlebon commented Jun 25, 2019

dustymabe commented Jun 25, 2019

dustymabe commented Jun 25, 2019

jlebon commented Jul 22, 2019 • edited Loading

bgilbert commented Jul 22, 2019

jlebon commented Jul 23, 2019

abompard commented Jul 23, 2019

jlebon commented Jul 23, 2019

sinnykumari commented Jul 23, 2019

jlebon commented Jul 23, 2019

sinnykumari commented Jul 23, 2019

abompard commented Jul 23, 2019 • edited Loading

jlebon commented Jul 23, 2019

abompard commented Jul 23, 2019

jlebon commented Jul 24, 2019

dustymabe commented Sep 17, 2019

jlebon commented Sep 18, 2019

dustymabe commented Sep 19, 2019

jlebon commented Sep 19, 2019

dustymabe commented Sep 19, 2019

dustymabe commented Sep 20, 2019

dustymabe commented Sep 20, 2019 • edited Loading

jlebon commented Sep 20, 2019

jlebon commented Oct 16, 2019

dustymabe commented Jun 25, 2019 •

edited

Loading

jlebon commented Jul 22, 2019 •

edited

Loading

abompard commented Jul 23, 2019 •

edited

Loading

dustymabe commented Sep 20, 2019 •

edited

Loading