Show HN: Flide: Make Jev-like decision models up to 9.5x faster
17 points by unikcc 5 hours ago | 2 comments
Decision models need their own inference engine.
So we developed Flide, a high-performance and exact inference engine for Jev-like decision models.
Flide achieves up to a 9.5x speedup over vLLM and SGLang.
It now supports 30+ popular local Jev-like decision models.
Start to make your decision immediately:
pip install flide
We built Flide from scratch to make Jev-like decision model inference up to 9.5x faster than vLLM and SGLang, while keeping exactness.
Here is the source code: https://github.com/OmniJev/Flide
Look forward to hearing your feedback and ideas!
Are there any circumstances where it is just about as fast as something like vLLM? For instance, single user, single token/decision inference?
How did you achieve the biggest speed up?
Are there any tradeoffs? “While keeping exactness” implies not.
Are there any specifics about how one must structure their context to take full advantage of it?