Access to kubeflow run_id

[simon-streamotion]

Hi,

When an elyra pipeline is executed on kubeflow is there a way to retrieve the kfp.dsl.RUN_ID_PLACEHOLDER within a stage of a pipeline. I'd like to get the run_id so I can use it to store data on s3 with a similar uuid.

Thanks -- Simon

[ptitzler]

Hi Simon,

this information is currently not exposed but that could be easily done through an environment variable:

The problem is though that the returned value appears to be an internal id and not what the user is normally exposed to:

[ptitzler]

I've been thinking about this some more and would like to suggest the following. Correct me if I'm wrong, but I believe the main ask here is for Elyra to provide an identifier that is unique across multiple pipeline runs, identical for all nodes in the pipeline, and yet meaningful to the user. (The latter is important because a user needs to know what to look for on S3 after a KFP run has completed.)

Since neither the KFP provided ID nor the Elyra submission ID meets those requirements, we could, for the time being, combine the two IDs, e.g. <elyra_submission_id> + "-" + <kfp_run_id> and expose that as ELYRA_RUN_ID:

• Clarification: A user submits a pipeline for processing using the Elyra GUI or CLI. Elyra explicitly creates a KFP run to execute the pipeline. A submission produces an id that comprises of the user provided pipeline name combined with the submission date/time.
• A run id (per the findings above) is an internal id that uniquely identifies a pipeline run.
• Between a submission id and a run id a 1:n relationship is defined, because a user can run the same pipeline many times using the KFP UI/CLI.

[kevin-bates]

I like this idea. Is the kfp-run-id available before the initial launch when we need to upload dependencies to COS since the COS directory is a function of the submission ID (that we now want to combine with the kfp-run-id)?

[ptitzler]

Unfortunately not, because it is generated by KFP when the workflow is executed. It's therefore nothing we have access to during pipeline preparation. Whether or not this could also work as is for KFP-native components (2.3+) that produce file output, I haven't looked into yet. If it doesn't we might want to consider supporting a placeholder value in the component properties that is replaced with the generated value at runtime to assure the creation of run-unique output artifact ids. (This is something we probably need anyways to support loop constructs to prevent overriding.)

[simon-streamotion]

Hi @ptitzler - you've hit the nail on the head with your latest description :-)
There are essentially 3 groups of things I want to tie together:

1. The pipeline run on Kubeflow (identified by <kfp_run_id>)
2. The Elyra tar balls (containing code etc) & log files written to s3 during execution of the pipeline on KF. Currently this is written to s3 under a user specified bucket in folder '<pipeline_name-timestamp>'
3. Any output the stages of a pipeline produces - such as data sets ML models etc... This is specified in the user code, but instead of it being a random UUID which I use at the moment it would be great to store in the same place as [Imported PR] Initial changes for ai-workspace build #2.

What you're suggesting above should allow me to achieve this (I think).

An additional tweak would be grateful accepted :-) Currently files for #2 are stored under an s3 prefix ('<pipeline_name-timestamp>') directly under an s3 bucket (specified by the 'Cloud Object Storage Bucket Name' parameter in Runtime config). So, something like:
s3://my-bucket/mypipeline_0524024006

IMHO it would be cleaner to allow users to specify an additional s3 prefix so the '<pipeline_name-timestamp>' prefix is not created directly under the bucket. For example:
s3://my-bucket/my_project/mypipeline_0524024006

This would allow our org to have a single bucket for elyra pipelines, but better organise the output.

Cheers -- Simon

[ptitzler]

@simon-streamotion I opened #1702 for the run id support. As for your add-on request, please create a separate issue.