For example, I see they do this for Postgres:
`let max_wal_gb = (shared_buffers_gb).clamp(2, 16);`
2-16GB of WAL is not a lot, but I have no idea how large is the data set.
On the parameters, the relational tests use 5 million records per test. The exceptions are the key-value category, which uses 15 million records, and the embedded category, which uses 1 million records. The same dataset shape, workload, harness, and hardware are used across the engines being compared.
For WAL, the 2 to 16 GB range is not intended to be a limit based on the dataset size. For the published runs, the dataset is small enough that this should not be a bottleneck. The persistent runs are also full-durability runs, with Postgres using fsync and synchronous_commit.
We will update the benchmarks page so the versions, dataset sizes, and tuning details are easier to find without digging through the Rust source.
I greatly appreciate when a vendor is willing to run the test and publish unfavorable information, even if it's only in one benchmark category.
I think it is not so clear cut. I mean, the multi-model nature it is pretty neat. Yes, you can use pgvector on PostgreSQL, but here you also have native graph support. If you want to have both you need to also add something like apache AGE, but arguably that is also a small ecosystem (at least IMHO as I never heard it until I actually started looking for Neo4J alternatives). Also, pgvector has a hard limit on embedding size, while surrealdb does not. For instances in which you have less than 1M elements and retrieval performance matters surreal already has an advantage.
In my personal opinion is a great overall product. Probably not the best at anything, but close enough without having to fiddle with PostgreSQL extensions or adding another piece of machinery to support graph workloads.
The only thing I don't like is that they didn't use either pure SQL nor Cipher for the query(ies) language(s). They roll their own blend, meaning that you will likely need more work to move in the ecosystem and you can't fully use the muscle memory of users that worked with other DBs before.
> ...add something like apache AGE, but arguably that is also a small ecosystem (at least IMHO as I never heard it until I actually started looking for Neo4J alternatives)
I appreciate having the option, but for most use cases on Pg, you are better off just using JOINs or switch to Neo4j for your graph workloads. I switched some workloads back to using different approaches of approximating "connectedness" in Pg (e.g. using Jaccard similarity)
If you do go down this route, the easiest way to get coding agents to figure out AGE is actually their regressions SQL tests: https://github.com/apache/age/tree/master/regress/sql
This has a lot of examples for the agent to know what will/won't work with AGE versus Neo4j Cypher.
It is worth a try for startups if you won't mind. Try to vibe code around it and give the data model a new look. I have a prototype project that combines both tree-sitter AST and converted it to JSON, then since SurrealDB accepts JSON as native input I now get free graph lookup on the control flow and easily did ancestry analysis and finding what functions potentially calls to this segement. All of it is in SurrealDB nested graph queries and the performance is alright, but is abysmal in Postgres JSONB since JSONB does not linearize the JSON data structure.
ps: I'm building a K8S operator for deploying SurrealDB with TiKV operator integration too.
The innovation points you spend on this should generally be spent in other areas, not seeing if someone's unproven db is your breadwinner.
Oh, that's the reason the SurrealDB operator was here in the first place because I need the full K8S lifecycle to maintain the database state such as backing up, that is not really doable with Helm.
No one should pick us because we're the new hot thing (at least I'd hope not). But at SurrealDB, we've got real enterprises in production at scale. For a lot of startups building today, having LLM/vector features, graph, auth, and the database in one place can really help you ship faster without stitching a bunch of tools together.
As a former DBA I got to see the general purpose databases bolt on a lot of shitty addons, and a lot of upstarts build just enough to get the sale done (or targeting bigger customers than I) - I hope y'all can get enough polish and reliability done and grow into something I want to use in five years :)
Though to be honest most people won't scale enough that DB performance is important in the first place. For most people they don't even need a database, your language has built in containers that will do everything you need.
On the ecosystem side, we have also grown a lot over the last few years across the community, integrations, cloud offering, and customers. Still work to do, but we are not as far off as people might assume.
Regarding the benchmarks themselves, they would've been more interesting with reproducible steps and more hardware variety.