Airbyte

Created in 2020, Airbyte is a solution for ingesting and loading data into a data platform, simplifying data movement by standardizing sources and destinations. Instead of providing end-to-end connectors (e.g., GCS → BigQuery), it offers a list of source connectors and a list of destination connectors, allowing any source to be paired with any destination. This flexibility is enabled by the Airbyte protocol, which strictly defines the contract that sources and destinations must comply with.

The main difference between Airbyte and Fivetran, a historical player in modular connector-based ingestion, is Airbyte’s open-source approach: it provides over 350 connectors, most of which are developed by the community. Creating custom connectors is greatly simplified thanks to this protocol and the availability of SDKs, making it easy to develop a source connector for a specific API, which then becomes compatible with all existing destination connectors.

Airbyte is available in different forms, suited to various use cases:

• Airbyte OSS (open-source): No license costs, offering great flexibility in how you install and configure your instance, but requiring self-managed deployment and hosting. These can be complex, as they involve managing both the Airbyte instance and its associated database separately.
• Airbyte Cloud: A managed version with similar functionalities, plus technical support, without the operational burden of managing an on-premise instance. Pricing is based on data volume moved and number of rows processed.
• PyAirbyte: A Python package providing access to most Airbyte source connectors directly within scripts, without the orchestration functionalities of the other solutions. PyAirbyte is used as an entry point for data processing, rather than a full end-to-end solution like Airbyte OSS or Airbyte Cloud. It can be used alongside an orchestrator such as Airflow.

Despite its strengths, Airbyte has some limitations: while its generalized protocol is valuable, it enforces sequential data transfer, which can be inefficient for large-scale ingestions. Additionally, while Airbyte can be deployed and configured automatically, this feature remains experimental, particularly for custom connectors.

Theodo’s point of view

We recommend trying Airbyte, whether in cloud or managed mode, if you are not ingesting large data volumes. While the technology has some weaknesses, it is robust, well-designed, and benefits from a dynamic community that ensures rapid product evolution and valuable support in case of difficulties.