TurboKV: Insanely fast Rust key-value store
23 points by rgbimbochamp 2 hours ago | 5 comments
nine_k 30 minutes ago
I suppose the insane speed is due to this:
reply> TurboKV's persisted Bloom-filter format uses hardware AES.
Also, built-in LZ4 compression.
I would expect SIMD to be used for scans.
haberman 6 minutes ago
I assume this is for hashing. I've seen several hashing algorithms turn to hardware AES instructions before, but I haven't seen any evidence that this technique outperforms state-of-the-art hashes like RapidHash (https://github.com/Nicoshev/rapidhash) in either quality or speed.
replyrgbimbochamp 28 minutes ago
Those help but the main write speed gain is the WAL, that uses preallocated mmap segments to avoid a write(2) per durable mutation while preserving crash recovery.
AES hashing mainly helps Bloom filter point lookups and LZ4 mainly helps SSTable I/O. Scans benefit indirectly, but don’t yet use a custom SIMD merge loop.
reply
> Appended to the WAL without a per-write sync
So… it’s not durable? Durable doesn’t mean “survives a process restart”, it means “durably saved to persistent storage”. For example, this “durable” mode wouldn’t survive power loss.