Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial implementation of simplifying config interfaces #389

Merged
merged 25 commits into from
Jul 25, 2023
Merged

Conversation

jlaneve
Copy link
Collaborator

@jlaneve jlaneve commented Jul 25, 2023

Description

Over time, we've continued adding parameters everywhere without giving much thought to the interface. This PR aims to simplify the interface for working with Cosmos DAGs/Task Groups and operators by collapsing the arguments into four config objects:

  • ProjectConfig: parameters that Cosmos needs to understand where your dbt project is
  • RenderConfig: parameters to tell Cosmos how to render your dbt project as a DAG
  • ExecutionConfig: parameters to tell Cosmos how to execute your dbt project
  • ProfileConfig: parameters to tell Cosmos what to run your dbt against

You can see the exact parameters each config takes here: https://github.com/astronomer/astronomer-cosmos/blob/simplify-interface/cosmos/config.py

It also:

  • simplifies the dbt_project_name and dbt_root_path arguments into a single arg
  • removes dbt_args, which was a duplicate of operator_args

Related Issue(s)

closes #372

Breaking Change?

Yes, the parameters to the DAG/Task Group and operators change. This is an intentional breaking change before the 1.0 release.

Checklist

  • I have made corresponding changes to the documentation (if required)
  • I have added tests that prove my fix is effective or that my feature works

@netlify
Copy link

netlify bot commented Jul 25, 2023

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name Link
🔨 Latest commit d2f2a50
🔍 Latest deploy log https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/64c02462f6026800089d2c58

@codecov
Copy link

codecov bot commented Jul 25, 2023

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.16% 🎉

Comparison is base (3e018fa) 91.24% compared to head (d2f2a50) 91.40%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #389      +/-   ##
==========================================
+ Coverage   91.24%   91.40%   +0.16%     
==========================================
  Files          46       48       +2     
  Lines        1610     1664      +54     
==========================================
+ Hits         1469     1521      +52     
- Misses        141      143       +2     
Files Changed Coverage Δ
cosmos/__init__.py 100.00% <100.00%> (ø)
cosmos/config.py 100.00% <100.00%> (ø)
cosmos/converter.py 96.96% <100.00%> (-1.54%) ⬇️
cosmos/exceptions.py 100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jlaneve jlaneve marked this pull request as ready for review July 25, 2023 01:02
@jlaneve jlaneve requested a review from a team as a code owner July 25, 2023 01:02
@jlaneve jlaneve requested a review from a team July 25, 2023 01:02
dev/Dockerfile Show resolved Hide resolved
Copy link
Contributor

@harels harels left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this interface looks good to me, pushed some minor fixes

@harels
Copy link
Contributor

harels commented Jul 25, 2023

@tatiana wdyt?

cosmos/config.py Outdated Show resolved Hide resolved
cosmos/config.py Outdated Show resolved Hide resolved
cosmos/config.py Outdated Show resolved Hide resolved
cosmos/config.py Outdated Show resolved Hide resolved
cosmos/config.py Outdated Show resolved Hide resolved
@harels harels merged commit e1ff962 into main Jul 25, 2023
39 checks passed
@harels harels deleted the config-interface branch July 25, 2023 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unify dbt_project_name and dbt_root_path
3 participants