Dagster dbt

dbt dagster

The dagster-dbt library offers @dagster_dbt.dbt_assets to define Dagster assets for dbt models. It requires a dbt manifest, or manifest.json, to be created from your dbt project to parse your dbt project’s representation.

The manifest can be created in two ways:

1. At run time: A dbt manifest is generated when your Dagster definitions are loaded, or
2. At build time: A dbt manifest is generated before loading your Dagster definitions and is included as part of your Python package.

The easiest way to handle the creation of your manifest file is to use DbtProject.

For example:

from pathlib import Path  
  
from dagster_dbt import DbtProject  
  
my_dbt_project = DbtProject(  
project_dir=Path(__file__).joinpath("..", "..", "..").resolve(),  
	packaged_project_dir=Path(__file__)  
	.joinpath("..", "..", "dbt-project")  
	.resolve(),  
)  
my_dbt_project.prepare_if_dev()

For development, my_dbt_project.prepare_if_dev() will generate the manifest file at run time for your dbt and Dagster project when run dagster dev

In production, a precompiled manifest should be used. Using DbtProject, the manifest can be created at build time by running the dagster-dbt project prepare-and-package command in your CI/CD workflow.

Deploying a dbt project from a separate git repository​

If you are managing your Dagster project in a separate git repository from your dbt project, you should include the following steps in your CI/CD workflows.

In your CI/CD workflows for your Dagster project:

1. Include any secrets that are required by your dbt project in your CI/CD environment.
2. Clone the dbt project repository as a subdirectory of your Dagster project.
3. Run dagster-dbt project prepare-and-package --file path/to/project.py to
   • Build your dbt project’s dependencies,
   • Create a dbt manifest for your Dagster project, and
   • Package your dbt project

In the CI/CD workflows for your dbt project, set up a dispatch action to trigger a deployment of your Dagster project when your dbt project changes.