Dataform

Dataform was created in 2018 as an open-source framework to simplify the creation, execution, and orchestration of SQL workflows on BigQuery, Snowflake, Redshift, and Synapse. Since its acquisition by Google Cloud in 2020, it has been optimized and integrated into BigQuery. Like dbt, Dataform allows users to declare sources, define transformations, set up data quality tests, and document everything using a JSON-like syntax. Additionally, Dataform provides an integrated IDE for lineage visualization, compilation, testing, and live deployment.

Dataform stands out due to its native integration with Google Cloud, making interactions with other services seamless. The tool is included for free with BigQuery, with only compute costs being charged. Its setup and environment management are extremely simple. Its templating system in JavaScript reduces code duplication with great flexibility, although JavaScript is less commonly used by data engineers.

These features make Dataform a serious competitor to dbt, but the tool still lacks maturity. The developer experience is less refined due to the inability to test locally and the limited Git integration in the IDE. Although documentation is available, it remains difficult to grasp, and community support is limited compared to dbt. Furthermore, while dbt integrates well with many open-source technologies like Elementary, Airflow, or Airbyte, Dataform lacks similar tools within the Google Cloud ecosystem.

Theodo’s point of view

Although Dataform offers strong integration within the Google Cloud ecosystem, it has limitations compared to more mature alternatives like dbt. The weaker developer experience and smaller ecosystem make Dataform less appealing for large-scale or scalable projects. Apart from small teams working exclusively with BigQuery, we recommend favoring dbt for its completeness and scalability.