DuckDB

Created in 2019 in Amsterdam, DuckDB can be used as a single-node analytical engine, capable of replacing Spark, as well as a mutable column-oriented database. Compatible with major languages (Python, Java, R, Node, ODBC) and usable both in backend and frontend via WASM, it is an open-source technology that serves not only as a database but also as a computational tool.

DuckDB has matured with its version 1.0, introducing innovations that are reshaping the data ecosystem. The technology aligns with the "big data is dead" movement: most datasets are not large enough to justify distributed computing technologies. DuckDB removes the need for client-server communication, which, according to its creators, is one of the main causes of latency in traditional databases. Its performance is impressive, and when combined with the power of modern personal computers, it allows many data processing tasks to be performed locally rather than in the cloud.

It is important to note DuckDB’s main limitation: it is a single-node technology (runs on a single machine) and supports only one connection at a time (only one user can connect simultaneously).

MDN’S POINT OF VIEW

DuckDB is my favorite technology of the past two years: while the "modern data stack" philosophy pushes for executing SQL on a cloud connection, DuckDB runs everything locally in a simple and efficient manner. DuckDB has its limitations, but it can help reduce your processing costs significantly.