What are you guys using it for?
I built a DuckDB extension so you can connect via postgres protocol, or Arrow flight. DuckDB also has its own Quack protocol, but client support for Quack is very limited right now, whereas many data tools support Postgres protocol or ADBC.
Sorry, does this mean that their own duckdb format already supports async or doesn't?
> Other formats, such as JSON files and DuckDB's native database format, use synchronous I/O.
For the future:
> Most of its benefit comes from accessing remote data, but cold local reads can also benefit, albeit less. Next, we plan to add async reads for JSON and DuckDB-native files, as these are the two other formats most relevant to DuckDB core. Formats that live in out-of-tree extensions are not on the roadmap yet.
Still don't usually see a speedup past about 192vCPU, but prior to 2.0, speed would top out at more like 64vCPU, so this is a big win.
That has enabled us to scale Splink (FOSS record linakge software) to process 1bn records with 10bn record comparisons in 10 mins, which as far as I'm aware makes it the fastest linkage engine of this kind. Most of this performance comes from simply delegating the big calculations to the DuckDB engine! https://moj-analytical-services.github.io/splink/topic_guide...
Thanks to the team for all their work!
Do you find that with the performance of duckdb, you see large deployments of spark based workflows less often? It seems hard to justify the complexity of spark if you can use duckdb , but I’m wondering if you see end users still reach for spark for any reasons?
There are a variety of people still use Spark, but this is usually due to constraints on their platform/infrastructure, not because it performs better. For instance people on Databricks can sometimes find it hard to use DuckDB on large machines, or they're just more familiar with Spark.
For a new user who has flexibility over where to run their workloads, we'd always recommend DuckDB. This is particularly the case now that: - Splink 5 (released in September) allows you to run your workload in n chunks, meaning you can distribute it across multiple machines even on DuckDB - DuckDB 2.0 parallelises better on large machines, so throwing a single massive machine at problems is now more viable
In general, after using Spark for a long time, I think for most workloads, it's just not really needed any more. Too complex, too difficult to tune. Perhaps if you have genuinely massive data. I've written a bit more about why I advise against Spark in most instances here: https://www.robinlinacre.com/recommend_duckdb/