My understanding is that modern databases will take advantage of that if you create an index of those fields - ie it'll read from the index, not the records.
Theoretically one can have a custom index extension which stores the data in a column-oriented manner, yet then one has major duplication, which is a problem for petabyte-scale databases, and is an even bigger problem in the Cloud.
KDB+ is the best implementation of a columnar DB in my opinion. Simple C, and one can understand how the data is stored down to the DRAM cells. Durability is just MMAP, which is fine as OLAP DBMSs tend not to be ACID.
In my personal opinion one shall learn from the best. The best in the world in massively parallel vectorized computation are the people at OpenAI and Anthropic, and the people who built PyTorch. Much of their work can be applied to analytical DBMSs.
(Draft, of course the topic is much more nuanced)
For example, if I cluster on my sorted primary key, that just means that the index can jump to the tuple with the correct primary key quickly (log time), but you will still end up reading* the entire tuple in the same way I explain in the blog. And if you only store a record ID that points to the other fields instead of the entire tuple at the leaves of the index, then that wouldn't be considered a clustered index.
I will note, however, that DBMSs should be smart enough to know that an aggregation on only an indexed column does not need to fetch the rest of the columns, only the data at the leaves of the index.
What they recently (and some of these were added in late 2000s) added is columnstore indexes... with compression and also in memory (Sybase Hana, SQL Server In Memory aka Hekaton, Oracle IM column store)
You can also combine these two: if you have a numeric id that increases by 1 every row, you can delta-encode that to just get a massive list of 1's, which you can then run-length encode to just be "the delta for the next million rows is 1", saving huge amounts of space.
You generally don't do this with AOS/SOA, because you do want to be able to random-access your entries ("index" them, in database terms) and I/O performance is much less important. Databases have this issue too, which is why indexes tend to either not be available or much slower in column-oriented DBs.
EDIT: ah, read the original commenter a bit too quickly, missed that they mentioned run-length encoding. Apologies.
I believe Vortex is unique in its ability to retain random access while recursively compressing data. And turns out that this is helpful for both in-memory _and_ on-disk (we might write about that in the future)!
I unfortunately didn't have much space to write about that in this blog, but in the compression section I did link to other blogs that we have written that go into much more detail, if you are interested :D
Array of Structures vs Structure of Arrays.
People who might be technical and might even intuitively understand what is meant by "Array of Structures vs Structures of Arrays" might not be familiar with the acronym AOS vs SOA, especially given SOA is already a technical acronym their brain might shortcut to, "Service Oriented Architecture", even if instantly dismissed as clearly incorrect.
By expanding the algorithm you also help let people better decide if they want to spend the time reading up more about it.
Maybe you agree with that and just disagree with where the line should be drawn?
> [...]
> [24] People have tried to build one system that does both in the past (HTAP, or hybrid transactional/analytical processing), but those attempts generally didn't work out. Some newer systems seem to be bridging the gap, however. CedarDB, the commercial version of Umbra (a research system from TU Munich), runs transactions and analytics on just one engine and one copy of the data. Databricks' Lakebase keeps Postgres (a row store) as the engine for transactional queries, but its storage layer rewrites the same data into columnar formats in the background, allowing an analytical engine to query it.
The footnote says "those attempts generally didn't work out" but then mentions Lakebase and CedarDB as newer efforts. Can anyone speak to whether or not CedarDB, Lakebase (and any others?) are achieving the fabled transactions + analytics in one database? Should we expect, at some point in the near future, for transactions + analytics to be standard in all databases, or is combining them a niche / esoteric thing with many tradeoffs?
HTAP has historically been niche because the cost of analytics on fresh data wasn't worth the squeeze (sooner or later the MBAs step in). Separating the OLTP and OLAP layers gives you better workload isolation and the ability to optimize each independently.
Databricks Lakebase/LTAP uses Postgres for transactions and lakehouse engines for analytics, with a shared storage layer. They also use Spark to build Postgres pages in parallel and write them directly to storage, avoiding the single-writer bottleneck when loading large volumes of data.
SingleStore takes a different approach: a distributed database based on MySQL with an in memory rowstore for transactions tightly coupled with a merger to columnstore for analytics.
I don't think HTAP DBs will become the default for every use case. But I do think the space will grow as we move towards agents and automated decision making and optimizations.
A simple case is a sorted list, you can sort it after every update, you can ignore the ordering and sort it as needed, or you can use a datastructure that is typically O(log(N)) at both updates and retrieval. Better datastructurs help, but they cannot remove the tradeoff entirely (in fact O(log(N)) inserts are kind of theoretically minimal, though a trie can do it in O(1) for fixed length datatypes).