Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update SBG e2e workflow to work across venues #86

Open
LucaCinquini opened this issue Apr 30, 2024 · 3 comments
Open

Update SBG e2e workflow to work across venues #86

LucaCinquini opened this issue Apr 30, 2024 · 3 comments
Assignees
Labels

Comments

@LucaCinquini
Copy link
Collaborator

Retrieve all dependent parameters from SSM or Airflow environment so that the DAG can run across all venues. Includes adding tests for the MCP test and prod venues.

@LucaCinquini LucaCinquini self-assigned this Apr 30, 2024
@LucaCinquini
Copy link
Collaborator Author

This should involve replacing the SSM key for the primary data bucket:

UNITY_OUTPUT_DATA_BUCKET = "/unity/ds/data/bucket/primary-data-bucket"

with a new expression that includes the venue and deployment - for which the values are retrieved from the AIRFLOW env variables.

This task include updating the nightly test so they can work for unity-venue-dev, unity-venue-test and unity-venue-prod.

@jpl-btlunsfo
Copy link
Collaborator

Chatted with Galen and decided to go with this format, for now:
UNITY_OUTPUT_DATA_BUCKET = "/unity/ds/data/bucket/{deployment_name}/{venue}-{counter}/primary-data-bucket"

As an example, "/unity/ds/data/bucket/luca/test-1/primary-data-bucket"

Does that make sense based on previous conversations y'all have had?

@LucaCinquini
Copy link
Collaborator Author

Sure, let's do that ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Todo
Development

No branches or pull requests

2 participants