Author here: I wrote mizu (R) and pymizu (Python), packages that offer lock-free async and parallel computing over shared memory, built on the same C core, libmizu. The link above is for the Python package.
A task round trip drops to under one microsecond, from around 100 µs over sockets with Python's ProcessPoolExecutor or mirai, my other R framework. These are numbers from my laptop (an M4 Pro), and benchmark scripts are in the repos for you to run.
Workers claim tasks off lock-free queues, there is no task dispatcher, and system calls are avoided on the hot path.
`PoolExecutor` is a drop-in `concurrent.futures.Executor`, tasks are awaitable in asyncio, NumPy arrays and polars, pyarrow and pandas frames cross as zero-copy views. As mizu and pymizu share the same wire format, data frames and Arrow types also cross zero-copy between worker pools of different languages.
Of course, this is single-machine only as it uses shared memory. The API is still experimental. Usability feedback and bug reports are most welcome at the repos. I'd love to hear about places you'd like to see this used.
I have no use for this but I want to commend you on defending your claims and taking objective numeric measurements. The lack of these lately on performance improvement based projects has been disturbing
Impressive work! This looks like it could be a game changer for some of our tasks... I don't mean to be one of those pushy people but... does it, perhaps support Polars?
A task round trip drops to under one microsecond, from around 100 µs over sockets with Python's ProcessPoolExecutor or mirai, my other R framework. These are numbers from my laptop (an M4 Pro), and benchmark scripts are in the repos for you to run.
Workers claim tasks off lock-free queues, there is no task dispatcher, and system calls are avoided on the hot path.
`PoolExecutor` is a drop-in `concurrent.futures.Executor`, tasks are awaitable in asyncio, NumPy arrays and polars, pyarrow and pandas frames cross as zero-copy views. As mizu and pymizu share the same wire format, data frames and Arrow types also cross zero-copy between worker pools of different languages.
Of course, this is single-machine only as it uses shared memory. The API is still experimental. Usability feedback and bug reports are most welcome at the repos. I'd love to hear about places you'd like to see this used.
Write-up with full benchmarks and the R/Python code examples: https://opensource.posit.co/blog/2026-11-05_mizu-0-1-0/