Update pipeline version to 6 and pin versions

[ajbozarth]

Fixes #2262

Sister PR to elyra-ai/pipeline-editor#172

Update the pipeline version and migration code to handle the changes
in Elyra 3.3

Pin package versions to prevent future backwards compatibility errors

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

[elyra-bot]

Thanks for making a pull request to Elyra!

To try out this branch on binder, follow this link:

[ajbozarth]

Leaving in draft until pipeline-editor is released and I update the version. I've updated it to point at the code sandbox for easier testing

[ptitzler]

Tested migration using pipelines from the examples repository as follows:

• git clone examples repository
• open legacy pipeline file (and perform migration)
• save pipeline
• verify properties view for each node
• review .pipeline file content
• run pipeline (* below indicates that the pipeline was not run, due to incomplete setup)
• close pipeline
• re-open pipeline

Without example components installed

• For pipelines containing custom components migration "fails" as expected -> might want to tweak title wording and use "aborted", if possible
• For pipelines containing only generic components migration succeeds (and yield the expected run results)

Without example components enabled

• Migration "fails" as expected -> might want to tweak title wording and use "aborted", if possible

With example components installed and enabled

• examples/pipelines/kubeflow_pipelines_component_examples (all pipelines)
• examples/pipelines/airflow_component_examples/bash_operator
• examples/pipelines/airflow_component_examples/email_operator
• examples/pipelines/airflow_component_examples/http_operator
• examples/pipelines/airflow_component_examples/slack_api_post_operator*
• examples/pipelines/airflow_component_examples/spark_sql_operator*
• examples/pipelines/airflow_component_examples/spark_submit_operator*

[ptitzler]

Migrate and run https://github.com/CODAIT/flight-delay-notebooks

• on KFP
• on AA

[ptitzler]

Migrate and run https://github.com/CODAIT/covid-notebooks

• on KFP
• on AA (us_data pipeline execution fails with "AttributeError: module 'text_extensions_for_pandas' has no attribute 'TensorDtype'" CODAIT/covid-notebooks#35) (Upstream Issue)

[ajbozarth]

I just pushed an update to the code sandbox version with the latest fix from the pipeline editor PR. This includes bug fixes for all reported issues. I'll be working on addressing the tests next

[akchinSTC]

• examples/pipelines/airflow_component_examples/spark_sql_operator*
• examples/pipelines/airflow_component_examples/spark_submit_operator*

I wouldnt be opposed to removing these two examples from the repo given the amount of setup these require.
Currently fixing the slack integration connection on the test cluster to test.

Successfully migrated my AA and kfp 3.1.x and 3.2.x pipelines with these latest changes

[marthacryan]

About to take a look at the code now but I just tried this out with a couple of older pipelines and it seemed to be working well.

[ptitzler]

LGTM

[kiersten-stokes]

LGTM!