Skip to content

Commit

Permalink
Document dependency conflicts between dbt and Airflow (astronomer#347)
Browse files Browse the repository at this point in the history
Add a matrix of Apache Airflow versions versus dbt versions,
highlighting which combinations lead to conflict errors.
Document how to re-run the analysis.
  • Loading branch information
tatiana authored Jun 28, 2023
1 parent 1ebee49 commit ebc70e3
Show file tree
Hide file tree
Showing 2 changed files with 71 additions and 1 deletion.
70 changes: 70 additions & 0 deletions docs/dbt/execution-mode-local-conflicts.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
.. _execution-mode-local-conflicts:

Airflow and DBT dependencies conflicts
======================================

When using the `Local Execution Mode <execution-modes.html#local>`__, users may face dependency conflicts between
Apache Airflow and DBT. The amount of conflicts may increase depending on the Airflow providers and DBT plugins being used.

If you find errors, we recommend users look into using `alternative execution modes <execution-modes.html>`__.

In the following table, ``x`` represents combinations that lead to conflicts (vanilla ``apache-airflow`` and ``dbt-core`` packages):

+---------------+-----+-----+-----+-----+-----+-----+---------+
| Airflow \ DBT | 1.0 | 1.1 | 1.2 | 1.3 | 1.4 | 1.5 | 1.6.0b6 |
+===============+=====+=====+=====+=====+=====+=====+=========+
| 2.2 | | | | x | x | x | x |
+---------------+-----+-----+-----+-----+-----+-----+---------+
| 2.3 | x | x | | x | x | x | x |
+---------------+-----+-----+-----+-----+-----+-----+---------+
| 2.4 | x | x | x | | | | |
+---------------+-----+-----+-----+-----+-----+-----+---------+
| 2.5 | x | x | x | | | | |
+---------------+-----+-----+-----+-----+-----+-----+---------+
| 2.6 | x | x | x | x | x | | x |
+---------------+-----+-----+-----+-----+-----+-----+---------+

Examples of errors
-----------------------------------

.. code-block:: bash
ERROR: Cannot install apache-airflow==2.2.4 and dbt-core==1.5.0 because these package versions have conflicting dependencies.
The conflict is caused by:
apache-airflow 2.2.4 depends on jinja2<3.1 and >=2.10.1
dbt-core 1.5.0 depends on Jinja2==3.1.2
.. code-block:: bash
ERROR: Cannot install apache-airflow==2.6.0 and dbt-core because these package versions have conflicting dependencies.
The conflict is caused by:
apache-airflow 2.6.0 depends on importlib-metadata<5.0.0 and >=1.7; python_version < "3.9"
dbt-semantic-interfaces 0.1.0.dev7 depends on importlib-metadata==6.6.0
How to reproduce
----------------

The table was created by running `nox <https://nox.thea.codes/en/stable/>`__ with the following ``noxfile.py``:

.. code-block:: python
import nox
nox.options.sessions = ["compatibility"]
nox.options.reuse_existing_virtualenvs = True
@nox.session(python=["3.10"])
@nox.parametrize("dbt_version", ["1.0", "1.1", "1.2", "1.3", "1.4", "1.5", "1.6.0b6"])
@nox.parametrize("airflow_version", ["2.2.4", "2.3", "2.4", "2.5", "2.6"])
def compatibility(session: nox.Session, airflow_version, dbt_version) -> None:
"""Run both unit and integration tests."""
session.run(
"pip3",
"install",
"--pre",
f"apache-airflow=={airflow_version}",
f"dbt-core=={dbt_version}",
)
2 changes: 1 addition & 1 deletion docs/dbt/execution-modes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Local
By default, Cosmos uses the ``local`` execution mode.

The ``local`` execution mode is the fastest way to run Cosmos operators since they don't install ``dbt`` nor build docker containers. However, it may not be an option for users using managed Airflow services such as
Google Cloud Composer, since Airflow and ``dbt`` dependencies can conflict, the user may not be able to install ``dbt`` in a custom path.
Google Cloud Composer, since Airflow and ``dbt`` dependencies can `conflict <execution-mode-local-conflicts.html>`__, the user may not be able to install ``dbt`` in a custom path.

The ``local`` execution mode assumes a ``dbt`` binary is reachable within the Airflow worker node.

Expand Down

0 comments on commit ebc70e3

Please sign in to comment.