Dagster

Managing data pipelines is a crucial task. To address this need, Elementl introduced Dagster in 2018, an innovative tool designed to orchestrate and monitor data pipelines efficiently. Dagster is an open-source Python library for managing and orchestrating data workflows, designed to simplify the development, deployment, and monitoring of pipelines. Like Airflow, its competitor launched four years earlier, Dagster models data workflows as a directed acyclic graph (DAG). However, while Airflow constructs a graph where nodes represent operations, Dagster builds a graph where nodes represent the state of the data, and the edges represent operations. These nodes are called Assets.  

Dagster allows triggering these operations based on multiple criteria: a time interval, an event trigger (for example, the arrival of a new file in a storage bucket), or an explicit user request (via CLI or the web interface). It also provides an intuitive web interface for visualizing data flows, their state, executions, and other essential information.  

Regarding ecosystem and integration, Dagster now offers connectors for most common libraries (Snowflake, PostgreSQL, dbt, etc.) and can be deployed on Docker, Kubernetes, AWS, GCP, among others. Dagster addresses several issues that Airflow does not. First, it allows seamless data transfer between operations. Second, it provides an easy-to-set-up testing environment. Finally, Dagster includes runtime type-checking, ensuring data consistency.  

Airflow benefits from a larger community, with more resources, tutorials, and examples available, which can make transitioning to Dagster more challenging for users accustomed to traditional orchestrators. Dagster’s data-centric approach can also make it harder to visualize tasks that do not produce a data state in the user interface.  

Theodo’s point of view  

Today, we recommend preferring Airflow over Dagster for complex task orchestration. However, for simpler orchestrations where a data-centric (rather than task-centric) approach makes sense, Dagster offers a pleasant UI, quick implementation, and can be a good option to explore.  

MDN’s point of view  

Dagster shifts the paradigm of orchestrators by placing Assets at the center rather than operations and DAG transformations. The modern interface includes all the essential elements for managing a data platform. I particularly appreciate Dagster for its highly intuitive UI and its top-tier integration with dbt.