Kestra

Automating and orchestrating data pipelines can quickly become complex, especially with traditional tools. Kestra is a modern open-source solution that simplifies this process while providing flexibility and scalability. Workflow configuration in Kestra is done using YAML files, making it easy to define tasks, dependencies, and conditions. The absence of complex code makes orchestration accessible to everyone, allowing both development and data teams to share a single tool.  

Kestra stands out with its system of over 500 plugins, which extends the platform's capabilities. It can easily integrate with databases, interact with cloud services such as AWS S3, or even execute bash commands. The tool allows workflows to be triggered in response to events, such as a file being uploaded, a change in a database, or an API notification, making it ideal for real-time processing. Thanks to its distributed architecture, Kestra can handle thousands of tasks simultaneously. If your data volume scales up, Kestra keeps up without compromising performance.  

The tool also offers a clear and modern interface for monitoring workflow execution in real time. Users can immediately identify bottlenecks and restart individual tasks without having to relaunch an entire pipeline. It also provides an API and can be deployed using a simple docker-compose setup. However, the tool is still relatively young, and its ecosystem is not yet fully mature, though frequent releases continue to improve it.  

MDN’s point of view  

A must-try. Kestra is an excellent tool for sharing usage across multiple teams within a company without depending on a specific programming language. It is easier to use than Airflow and allows execution of any type of code—including SQL—within its workflows.