Add DBT Deploy support #74

neel-astro · 2024-09-03T09:34:38Z

Changes:

Add support for DBT Deploy for normal and preview deployments. This change includes adding deploy-type input to help distinguish between different forms of deploys, i.e. dags-only, dags-and-image, dbt and infer (to avoid introducing breaking change).
Add 'mount-path' as optional input to provide a mount path if the user wants to for dbt deploy
Add dev-ex team as the project code owners

Minor Fixes:

Fix the logic to install the custom CLI version, it was currently broken and mostly unnoticed because no one might be using it 🙂
Fix Github Action logging by grouping logs based on the steps, so that it is easy to navigate the logs for the users

Closes: astronomer/astro#23047

Testing:

Updated E2E tests to include sample DBT deploys

GHA Template I have been using for the tests:

name: Deploy (checkdeployaction)
on:
  push:
    branches:
      - main

jobs:
  prod-astronomer:
    env:
      ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
    runs-on: ubuntu-latest
    steps:
      - name: Print GITHUB_ACTIONS env var
        run: echo "GITHUB_ACTIONS=${GITHUB_ACTIONS}"
      - name: DBT Deploy
        uses: astronomer/deploy-action@dbt-deploy
        with:
          action: dbt-deploy
          deployment-name: neel-test-1
          root-folder: dbt/
          mount-path: /dbt
          workspace: ${{ secrets.WORKSPACE_ID }}
          cli-version: 1.28.1
      - name: Deploy DAGs/Image
        uses: astronomer/deploy-action@dbt-deploy
        with:
          root-folder: astro-project/
          deployment-name: neel-test-1
          workspace: ${{ secrets.WORKSPACE_ID }}

Image Deploy to Astro

DBT Deploy to Astro

DAGs Deploy to Astro

DBT Deploy to Preview Deployment in Astro

Case when someone uses CLI version < 1.28.1 for DBT Deploy

neel-astro · 2024-09-03T09:41:19Z

README.md

@@ -96,7 +97,7 @@ jobs:
    runs-on: ubuntu-latest
    steps:
    - name: Deploy to Astro
-      uses: astronomer/deploy-action@v0.4
+      uses: astronomer/deploy-action@v0.6


I will update these to 0.7 once this PR is merged and we cut a new release

neel-astro · 2024-09-03T09:42:21Z

action.yaml

      shell: bash
      id: deployment-preview
    - name: Determine if DAG Deploy is enabled
      run: |
+
+        echo ::group::Determine if DAG Deploy is enabled


all the group & endgroup allows GHA to group these logs

neel-astro · 2024-09-03T09:45:06Z

action.yaml

+        skip_deploy=1
+
+        # This for loop checks for following cases:
+        # 1. If no file is part of the input root folder, then it skips deploy


In order to accommodate for the case when dbt project and astro project are part of the same repo, we would need to ensure that we only trigger image deploy when any file is updated inside the astro project's root folder, this is slightly different from existing behaviour where if any file outside the dags folder is updated we trigger an image deploy

this makes sense! since these deploy-action releases are versioned, I'm comfortable with that and I wouldn't expect any breaking changes from it... in fact this might be better for users

iancmoritz · 2024-09-03T14:36:36Z

action.yaml

+          exit 1
+        fi
+
+        # validate that mount path is provided


Do we want the mount path to be required here? Can we instead rely on the default mount path of the dbt Deploy (i.e. the directory name)?

Also noticed the mount-path input above not required (

deploy-action/action.yaml

Line 85 in c7900c8

mount-path:

) so if we do make it required I assume we should change

cc @jeremybeard

The mount path is not required by the feature and ideally we should be encouraging users not to set it because the CLI already provides a sensible default.

sunkickr · 2024-09-03T15:28:50Z

README.md

@@ -48,11 +48,11 @@ The following table lists the configuration options for the Deploy to Astro acti

 | Name | Default | Description |
 | ---|---|--- |
-| `action` | `deploy` | Specify what action you would like to take. Use this option to create or delete deployment previews. Specify either `create-deployment-preview`, `delete-deployment-preview` or `deploy-deployment-preview`. Don't sepcify anything if you are deploying to a regular deployment. |
+| `action` | `deploy` | Specify what action you would like to take. Use this option to create or delete deployment previews. Specify either `create-deployment-preview`, `delete-deployment-preview`, `deploy-deployment-preview` or `dbt-deploy`. |


dbt-deploy-deployment-preview is also being added right? Instead of adding two new actions would it make sense to just add an option dbt-deploy that can be true/false?

sunkickr · 2024-09-03T15:30:28Z

README.md

+  with:
+    deployment-id: <deployment id>
+    root-folder: dbt/
+    mount-path: /dbt


What's the difference between mount-path and root-folder? would they always be the same?

These are both features of dbt deploys.

mount-path -> the directory where the dbt code is actually mounted in your container
root-folder -> the directory where the dbt code is retrieved from

action.yaml

README.md

jaketf · 2024-09-09T17:25:54Z

@neel-astro just dropping by to say kudos on a very well scoped PR with high quality description and leaving the codebase a little cleaner than you found it without distracting from the PR. I learned more about this feature by lurking here :)

jeremybeard

Thanks @neel-astro, this is much simpler with the deploy type restructure. I left some comments, a few around all the output variables that I hope we can simplify even further.

jeremybeard · 2024-09-18T14:59:47Z

.github/workflows/tests.yaml

+  dbt-deploy-test:
+    name: DBT Deploy Test
+    runs-on: ubuntu-latest
+    needs: [custom-docker-image-test, create-test-deployments]


Does this test depend on custom-docker-image-test?

it does not depend on the custom-docker-image-test, but because these tests are running against a single deployment, to avoid any flakiness in test results, we are running a single test at a time, and hence we are required to use the needs clause to create the dependencies to avoid concurrent test scenario runs

action.yaml

jeremybeard · 2024-09-18T15:08:06Z

action.yaml

+        if [[ "${{ inputs.deploy-type }}" == "dbt" && $CLI_VERSION != "" ]]; then
+          REQUIRED_VERSION="1.28.1"
+          CURRENT_VERSION=$(astro version | awk '{print $4}')
+          if [[ $CURRENT_VERSION =~ ([0-9]+)\.([0-9]+)\.([0-9]+) ]]; then


This looks robust. Another option might be to run npm install -g semver and use that but you've already done it this way so we don't have to change it.

good callout 👍 , I moved it to semver which reduces the logic

jeremybeard · 2024-09-18T15:25:56Z

action.yaml

@@ -257,62 +313,139 @@ runs:
          echo "IMAGE_DEPLOY_ONLY=false" >> $GITHUB_OUTPUT

          # set action
-          ACTION=deploy
+          ACTION=${{ inputs.action }}


Is this variable needed anymore? The only place I can see it used is at the end of the step to check the user provided a correct input, but we should have that at the start of the workflow.

dropped the ACTION variable, instead moved the corresponding validation check to the top of the function

jeremybeard · 2024-09-18T15:26:12Z

action.yaml

@@ -257,62 +313,139 @@ runs:
          echo "IMAGE_DEPLOY_ONLY=false" >> $GITHUB_OUTPUT


Do we also still need this variable?

And SKIP_DEPLOY? That's only true if ${{ inputs.action }} == delete-deployment-preview but we could just look at that?

jeremybeard · 2024-09-18T15:47:04Z

action.yaml

          dags_only=0
        fi

+        if [[ ${{ steps.deployment-preview.outputs.SKIP_DEPLOY }} == true ]]; then
+          # skip all deploy steps
+          dags_only=2


This pre-existing use of "2" for a boolean variable is... not ideal. I know this is creeping the scope but can we change these to use "true"/"false" and if we need to express other things like skipping the deploy then can we just output a second variable?

make sense, I had an itch as well, so replaced the numbering with boolean for both dags_only and dbt_only

jeremybeard · 2024-09-18T15:49:16Z

action.yaml

      shell: bash
      id: deployment-type
    # If only DAGs changed and dag deploys is enabled, do a DAG-only deploy
-    - name: setup deploy options
+    - name: Setup image or dags deploy options


Does this step need to be separate to the one before it?

We could merge them, but both are decently sized bash scripts, so we should avoid merging them unless we see a value addition by doing so otherwise it would be another very long script to manage. This function is well-scoped, and the other one isn't, and I agree with that.

action.yaml

.github/workflows/tests.yaml

jeremybeard

Thanks again, this is excellent. I've left one nitpick but approving ahead of that.

.github/workflows/tests.yaml

iancmoritz

Great work

neelavarrao · 2024-09-25T17:12:30Z

@neel-astro Do you know when you would be publishing a release with this feature? I was looking forward to using this.

neel-astro · 2024-09-26T09:24:19Z

@neelavarrao trying to get it out in the next couple of days, thanks for the interest 🙂

neel-astro force-pushed the dbt-deploy branch from 1b28ccb to 649765e Compare September 3, 2024 09:35

neel-astro requested review from sunkickr, iancmoritz and a team September 3, 2024 09:36

neel-astro force-pushed the dbt-deploy branch from 6c52997 to c7900c8 Compare September 3, 2024 09:40

neel-astro commented Sep 3, 2024

View reviewed changes

iancmoritz reviewed Sep 3, 2024

View reviewed changes

sunkickr reviewed Sep 3, 2024

View reviewed changes

jeremybeard reviewed Sep 3, 2024

View reviewed changes

iancmoritz mentioned this pull request Sep 4, 2024

add dbt deploy support #71

Closed

neel-astro changed the base branch from main to e2e-tests September 12, 2024 08:16

neel-astro force-pushed the dbt-deploy branch from 65e9967 to 203064b Compare September 13, 2024 12:02

neel-astro force-pushed the e2e-tests branch from 405cfbc to 495df2a Compare September 17, 2024 10:06

Base automatically changed from e2e-tests to main September 18, 2024 03:43

address code review changes

a7ad4dd

neel-astro force-pushed the dbt-deploy branch from 4ecbd17 to a7ad4dd Compare September 18, 2024 08:36

merge branch main into dbt-deploy

7f3b65d

neel-astro force-pushed the dbt-deploy branch from f902443 to 7f3b65d Compare September 18, 2024 08:39

merge branch main into dbt-deploy

81b3c21

neel-astro force-pushed the dbt-deploy branch 3 times, most recently from 3fb9bd2 to ff570e4 Compare September 18, 2024 12:38

update README and add test for infer case

954b90f

neel-astro force-pushed the dbt-deploy branch from b7d8981 to 954b90f Compare September 18, 2024 13:21

drop dbt-deploy branch from on push condition

3b00849

jeremybeard reviewed Sep 18, 2024

View reviewed changes

neel-astro force-pushed the dbt-deploy branch from dad9bf6 to cdb51e1 Compare September 19, 2024 12:53

code review changes: cleanup variables and booleans

959cdf3

neel-astro force-pushed the dbt-deploy branch from 9f73bcf to 959cdf3 Compare September 19, 2024 13:37

jeremybeard approved these changes Sep 19, 2024

View reviewed changes

.github/workflows/tests.yaml Outdated Show resolved Hide resolved

code review changes: s/infer-image/infer in e2e test file

43eed38

iancmoritz approved these changes Sep 23, 2024

View reviewed changes

neel-astro merged commit 09e2a1a into main Sep 24, 2024
1 check passed

This was referenced Sep 26, 2024

add dag-only-deploy parameter #75

Closed

Proposal: Add a dag_only parameter #64

Closed

neel-astro deleted the dbt-deploy branch September 26, 2024 09:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add DBT Deploy support #74

Add DBT Deploy support #74

neel-astro commented Sep 3, 2024 •

edited

Loading

neel-astro Sep 3, 2024

neel-astro Sep 3, 2024

neel-astro Sep 3, 2024

iancmoritz Sep 3, 2024 •

edited

Loading

iancmoritz Sep 3, 2024 •

edited

Loading

iancmoritz Sep 3, 2024

jeremybeard Sep 3, 2024

sunkickr Sep 3, 2024 •

edited

Loading

sunkickr Sep 3, 2024

iancmoritz Sep 3, 2024

jaketf commented Sep 9, 2024

jeremybeard left a comment

jeremybeard Sep 18, 2024

neel-astro Sep 19, 2024

jeremybeard Sep 18, 2024

neel-astro Sep 19, 2024

jeremybeard Sep 18, 2024

neel-astro Sep 19, 2024 •

edited

Loading

jeremybeard Sep 18, 2024

jeremybeard Sep 18, 2024

jeremybeard Sep 18, 2024

neel-astro Sep 19, 2024 •

edited

Loading

jeremybeard Sep 18, 2024

neel-astro Sep 19, 2024

jeremybeard left a comment

iancmoritz left a comment

neelavarrao commented Sep 25, 2024

neel-astro commented Sep 26, 2024

		@@ -257,62 +313,139 @@ runs:
		echo "IMAGE_DEPLOY_ONLY=false" >> $GITHUB_OUTPUT

Add DBT Deploy support #74

Add DBT Deploy support #74

Conversation

neel-astro commented Sep 3, 2024 • edited Loading

Changes:

Minor Fixes:

Testing:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iancmoritz Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

iancmoritz Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunkickr Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaketf commented Sep 9, 2024

jeremybeard left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neel-astro Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neel-astro Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeremybeard left a comment

Choose a reason for hiding this comment

iancmoritz left a comment

Choose a reason for hiding this comment

neelavarrao commented Sep 25, 2024

neel-astro commented Sep 26, 2024

neel-astro commented Sep 3, 2024 •

edited

Loading

iancmoritz Sep 3, 2024 •

edited

Loading

iancmoritz Sep 3, 2024 •

edited

Loading

sunkickr Sep 3, 2024 •

edited

Loading

neel-astro Sep 19, 2024 •

edited

Loading

neel-astro Sep 19, 2024 •

edited

Loading