Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: I want the image tag for my Metadata and Pipeline Invocation charts managed by the baseline chart. #307

Open
3 of 5 tasks
ewilkins-csi opened this issue Aug 29, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request slacktask Small ticket done in down time
Milestone

Comments

@ewilkins-csi
Copy link
Contributor

ewilkins-csi commented Aug 29, 2024

Description

Currently, even though metadata and pipeline invocation are v2 charts, the image tag (i.e. the aiSSEMBLE version) is set in the downstream project chart. This doesn't match our usual pattern, and isn't a great user experience. We should set the image tag in the base chart for these images.

DOD

  • Change the Metadata chart to extend from the base Quarkus chart
  • The MDA templates for Metadata and Pipeline Invocation values files should not set image tag
  • Metadata and Pipeline Invocation should pull the correct image tag without being set explicitly in the values file
  • The v1 chart migration in foundation-upgrade should no longer target Metadata or Pipeline Invocation for updates.
  • Write a baton migration to remove the tag if it is set to the current aissemble version

Test Script

  1. Create a project from the archetype
    mvn archetype:generate
  2. Add a data delivery pipeline with provenance enabled to issue-307-pipeline-models/src/main/resources/pipelines (e.g. SparkPipeline.json)
  3. Generate the project and resolve manual actions
    mvn clean install
  4. OTS ONLY: Deactivate the upgrade-v2-chart-files-aissemble-version-migration migration and change the Chart.yaml file to point to the local copy of the aiSSEMBLE baseline chart for the following apps:
    • pipeline-invocation
    • metadata
    • policy-decision-point
  5. Verify the image tag is not set in the following files:
    • issue-307/deploy/src/main/resources/apps/metadata/values.yaml
    • issue-307/deploy/src/main/resources/apps/pipeline-invocation/values.yaml files
  6. Build the project without the build cache and follow the final manual action
    mvn clean install -Dmaven.build.cache.skipCache
  7. Deploy the project
    tilt up; tilt down
  8. Ensure the policy-decision-point comes up successfully using the local project image
  9. Execute the pipeline via the invocation service
    curl -X POST http://localhost:8085/invoke-pipeline/start-spark-operator-job -d '{
        "applicationName": "spark-pipeline",
        "profile": "dev"
    }' -H "Content-Type: application/json"
  10. Watch the pipeline logs and wait for successful completion. It may take 15-20s for the driver pod to appear. Kill the log follow once the pipeline is finished.
    kubectl logs -f spark-pipeline-driver
  11. Query the metadata service and verify that metadata was created during pipeline execution. The metadata make take a couple tries to appear as the first creation of metadata takes some time.
    curl localhost:8082/metadata
@ewilkins-csi ewilkins-csi added the enhancement New feature or request label Aug 29, 2024
@ewilkins-csi ewilkins-csi added this to the 1.9.0 milestone Aug 29, 2024
@ewilkins-csi ewilkins-csi self-assigned this Sep 9, 2024
@ewilkins-csi ewilkins-csi added the slacktask Small ticket done in down time label Sep 12, 2024
@ewilkins-csi ewilkins-csi modified the milestones: 1.9.0, 1.10.0 Sep 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request slacktask Small ticket done in down time
Projects
None yet
Development

No branches or pull requests

1 participant