Dogma is more effective than reliability concerns in my experience. It just happens that the dogma prefers Postgres in this case.
It is just difficult to make these decisions in a more informed way because it requires actual manual testing with the concrete use case and also a decent understanding of databases in question. Chances are you won't have people that understand these things or those people won't have much of a say in decisions.
Project managers and Human Resources rolling out overengineered projects will outnumber current software engineers 10 to 1
And different people will be doing product management and HR
all because today’s software engineers don’t want to babysit AI agents and choose antiquated libraries so that their resume said they'd been using a popular framework for a couple years
Nobody else is playing that outdated game, its just a rotation
just use the faster library? leverage compute resources more effectively and justify value to an organization better than the next person
otherwise, what needs to be done? I don’t see a problem with any of this aside from organizations risking less experienced people doing less efficient things in other parts of the stack
We use Postgres because it’s fantastic at the scale and problems it solve, if you have an extremely critical system where raw speed is at the core of everything and you’re dealing with petascale then maybe you’re bringing a solution you like to a problem it doesn’t fit?
PostgreSQL has historically been bad at managing the noisy neighbor problem, but with thread pools, and io priorities, it can be solved.
Has this been tackled here ?
[0] https://15721.courses.cs.cmu.edu/spring2016/papers/p743-leis...
[1] https://db.in.tum.de/~kohn/papers/query-scheduling-sigmod21....
You need a server with enough ram to fit it all.
But it kind of make the database fly.
2.
Commit #1's message is "hey claude, do a breakthrough" from a week ago and is 1.5M lines. Commit #2 is "blog post" from 4 days ago.
My head is spinning. I don't mind AI stuff or AI enabled stuff but there's gotta be some bar for ending up on HN, and also personal accountability: the lack of humility and honesty sets a new low for me. There is no "we" who "released pgrust 0.2". It's one person cosplaying a serious engineering team doing a mountain of work. The bus factor is 1, and its one you can't trust on the basics.
ex. the first 1/3 of the blog post is bloviating about how a rust for loop is faster at summing 500M numbers on the heap than loading the numbers from a table and summing them.
It leaves me in quite some anguish. This site kept me well-informed and growing for 16 years. It is no longer reliable for that as long as things like this can be the #1 post with 60 comments, with the author here, and no one mentioned any of this.
I think it's a weird way to handle git history versus squashing feature branches into single commits, but it's not just one Claude session slapped up on GitHub. The OP also has a post above about their exhaustive testing which has uncovered a goodly number of bugs in Postgres itself, too. I think it's fair to say they're putting in a good amount of work on this.
5,940 commits, one author, 29 days. 5,067 of them, 85%, have a `Co-Authored-By: Claude` trailer. Busiest day is 1,393 commits, running 60-105/hour for ~20 hours straight. Commit messages reference .claude/skills/fleet/ and agent worktree "lanes". That's an unattended agent fleet committing once a minute around the clock. Commit count used to mean review time. Here it means GPU time.
Also: the v0.2 tag shares no common ancestor with main. git merge-base fails. "The actual git history" is an orphan branch grafted in after the fact.
And the testing rigor we're crediting them for should be weighed against the headline claim: 300x faster than Postgres, ahead of ClickHouse: fastest analytical DB on earth, one guy, one month.
Most damning: the Postgres baseline ran with *max_parallel_workers_per_gather = 0*. Parallel engine vs. deliberately-hobbled single-core Postgres, headline says 300x.
Using AI is fine. But "look how many commits" is precisely the signal this workflow is built to fake.
Everyone who knows a bit about databases knows the difference between execution models and what kind of optimization it brings.
What makes you say this is a useless demo? I can't count the number of people who've struggled to do analytics inside of Postgres. Almost always they end up setting up a separate system such as Clickhouse and replicating the data between the two systems. Now they can have one system that's Postgres-compatible, and it's faster than either of the original systems.
> Everyone who knows a bit about databases knows the difference between execution models and what kind of optimization it brings.
In our last post[0], when we mentioned we were getting close to Clickhouse level performance (now faster than Clickhouse), we were met with disbelief. This post is meant to explain part of how we closed the 300x gap between Postgres and Clickhouse. The execution model being 10x of it.
I stand corrected, there is. Wishing you the best of luck here.
Now, I don't know how MIT -> AGPL re-licensing specifics work, but still.
For example, if I compiled PG into x86-64 assembly, and then decompiled it into C (via, say, IDA), and then polished that decompiled C code into very readable C code, it is still a derived work. For some reason, people think that if you include an LLM or Agent, copyright can be ignored, and plagiarism is now no longer possible.
It is similar to the crypto-folks thinking that if you use crypto, you no longer have to pay taxes, because the internet/computers make all inconvenient realities go away.
Honestly, such flagrant and arrogant copyright violations make it hard for me to take the project seriously, because it seems like a desperate stunt for attention (which itself may be a solid business move, but that is besides the point).
Put differently, if one were to fork pgrust, strip away the new license and copyright, and restore the original PG license and copyright (while also adding malisper+team to that copyright), they would face no legal consequences at all. In fact, they would probably be a less legal risk than the pgrust team.
This is correct
> and thus a fork, and should thus have the original license and copyright preserved)
This is not correct. The Postgres license is permissive. We need to include a copy of the license (which we do in the NOTICE file[0]) but we CAN relicense the Postgres code however we want as long as we meet the requirements of the license. pgrust is a derived work of Postgres, but Postgres allows derived works to be under a different license.
And your sales folks would call and say: "No need to change anything, we still run PostgreSQL, and ours is just called pgrust, but it's N times as fast".
I’ve done some experiments replacing spatial SQL queries with custom vectorised batch mode code in C# and the speed up was astonishing.
The people dubious about these claims have no idea what their computers are really capable of.
Have pgrust folks reconsidered this? Else, IMO we can have an independant rust port of pgrust, which can be MIT, which will garner more attention.
For instance, Mongo, Cockroach, and Materialize have all gone source available. We picked AGPL because it's the best balance between open source and prevents Amazon from just repackaging it and selling it.
If AGPL is an issue for anyone, we would be happy to dual-license under a commercial license.
I consider AGPL a poison pill in my work. That is not true with a suitable commercial license, although I expect a lot more commercial product (support/features/etc). As you note, your objective is to prevent commercialization of your software, but radically speeding up analytics is primarily a concern of large organizations so it seems like a mismatch in purpose.
That's up to GCP then. I'm sure OP would be willing to license pgrust to them for an appropriate price.
You should be setting up a CLA process then ASAP if you are looking to take outside contributions.
Will await a MIT based fork myself.
AGPL is untested in courts. There is no definitive definition of what could be considered within the blast radius such that it would require AGPL licensing.
There's a reason AGPL is banned at Google and most sane companies. It's simply too dangerous.
You can't simply say "clients aren't bound" because it depends.
I'd rather see the BSL used here to be perfectly honest. At least it's simple.
It’s weird because those who actually care about optimized pg gains are most likely large corporate customers. Why make a product targeting them and license it in such a way they’ll never use it?
This also hard blocks upstreaming any beneficial features into core Postgres.
Wait, sorry, you're asking why make something enterprise customers might pay for, and then not give it away to them for free?
PS Learn how DBs do joins for more information. Specifically the differences between hash joins, merge joins and nested loop joins. They are basically fancy ways to page part of your working set to disk at huge performance penalties.
PPS As memory gets more expensive, these techniques get more valuable. When it gets cheap, they lose value.
[0]: With Exhibit B, which prevents relicensing to GPL. It is also analogous to CDDL.
I doubt even 30000x faster would prompt a policy change.
> light-years away from what happened here
There is also zero evidence a human has reviewed a meaningful amount of the code generated for the port, let alone the original C code from Postgres, given the size/scale of the project and the how quickly it was produced.
Also, you can successfully sell a systems project that is only 10% faster. 30000x faster and they are throwing illegal and debauched things through your window to get access to your improvements.
What does it being system code have to do with anything?
Most systems code has to be "canonical". That means it can't be improved. Imagine the most difficult PR review you have ever had. Now give the reviewers OCD and an extra 30 IQ points. And every single code change has to live up to this standard. That's what it has to do with this.
You didn't express anything unique to systems engineering.
Yes, at [tech corp dayjob], any dependency is likely to be banned for arbitrary reasons if you bring it to the attention of the wrong people. It doesn't have to go against any of our policies e.g. don't mention anything with GPL in the name around the "risk" people. In fact, do not ever talk to the "risk" people and hope they don't talk to you.
Latest news: Apparently, devtools are a legal risk. Basic reverse-engineering of client-side JS is now banned.
The delusions really seem to scale with headcount.
What we don’t know for sure is whether there is any copyright for LLM-generated code. The license might be irrelevant!
hey claude, do a breakthrough
You can find the actual git history at the v0.2 github tag.
Co-Authored-By: Fable <noreply@anthropic.com>
It's easy to claim AI slop nowadays, but you should still mistrust-but-verify.
[0] https://benchmark.clickhouse.com/#system=+liH|pgrs|gQ&type=-...
SELECT COUNT(*) FROM large_text_db WHERE X
Where X is something that must be matched exactly. X can be FTS query on FTS-indexed table, but the way COUNT() works in PG is that it's impossible to make it fast. Over large tables, lets say 1B+ rows, it can be very very slow.
Example use case is: searching through a hospital DB of reports that have "pancreatic cancer" in them. This is trivial in SQLite, but in PG it's hard.
Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust and postgres (see the proofs directory if you're curious). For cases where formal verification is not easy, we've taken the c implementation of a function and the rust implementation of a function and ran millions of inputs through each of them and confirmed they gave the same results every time.
We've only covered about 15% of the surface area so far, but in the process, we've discovered ~100 bugs in pgrust and ~20 bugs in Postgres itself. My favorite postgres bug we found is this one[0]. Postgres has a quadtree implementation. Due to floating point rounding, it was possible for a point to be neither above, nor below, nor even with the center point of the quadtree.
We've also entered engagements with Antithesis[1] to do Jepsen style fault testing and Aretta[2] to do more serious formal verification.
If you want to support the project, the easiest way is to give us a star on GitHub[3]
[0] https://www.postgresql.org/message-id/19597-39c532e61d78dff6...
[1] https://antithesis.com/
[2] https://aretta.ai/
[3] https://github.com/malisper/pgrust
I struggle with this a lot with Marginalia's index. Where I identify a hot method in a prod profiler run, try to replicate it on a test machine where I can never get the same cache characteristics because everything in this space is like an onion of caching layers that you affect the real performance of the system. I may get it to run significantly faster, but that only sometimes makes the production profiler sample move its needle.
e.g. I've recently been experimenting with using a cursed hybrid model in Marginalia's index, where based on a mincore probe, I switch between mmap and io_uring for reading a cluster of pseudoadjacent data. There are real tanglible benefits both in the test machine and in prod with this, but the numbers do not agree at all about how the needle moves :P
I had an issue like this for optimizing pgrust. I had an optimization that showed no impact on my test machine (c8g.4xl) and showed a 20% improvement when ran on my mac. It turns out the issue was the instruction cache on the c8g.4xl was being saturated on the test machine but not on my laptop, moving the bottleneck to a different place
If you can consistently reproduce the performance difference, you're already half way there
The hard part about optimizing this type of code, IMO, is that there are so many cache layers, both in the CPU and the OS and sometimes in the storage medium. You can warm all of those caches up, but then you're testing a nonsense scenario that will basically never happen in a realistic scenario, where wall clock time is what matters, and not much the CPU is working or how many IOPS you're pushing.
A question on 20s postgresql time - It does not look like you are accounting for reading data from disk? Wouldn't the aggregation query have to load data from disk first? Or is it somewhat guaranteed that the table is already in memory? The Rust version is clearly in memory (I am no rust expert, so that may not even be actually in memory, if its a generator).
I choose the data size so that it would fit in memory on the machine I was testing on. fwiw, there's still a ton of overhead Postgres has that the toy example does not. For example Postgres will serialize the numbers into tuples and need to deserialize them to execute the query. That's why it's not an apples-to-apples comparison
AlloyDB from Google Cloud uses columnar storage like a secondary index, while the relations are still stored in TOAST.
Its unlikely an AI written project is one which people will want to maintain even if it "works", which means AI has to maintain it. Maintaining a code base of a production database is going to continue costing 100s of thousands, if its even possible at all with AI. Its also unlikely to find users even if its almost perfect, because a bug in a database is not something users can accept.
Postgres is postgres because its like 30 years old and battle tested. Sure its missing some modern features, but its incredibly stable, which is part of its popularity.