Buoyant Launches Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM

Buoyant, the company behind Linkerd, has released Kinference, an open source project for building inference platforms on Kubernetes. Kinference is an inference control plane for Kubernetes and vLLM. It sits in front of a fleet of vLLM workers and decides where each request goes, applying routing policies, token rate limits, KV-cache-aware load balancing, and per-tenant metrics in a single layer.

For development teams serving many open-weight models to many internal clients, Kinference targets the layer above the model server. The project is Apache 2.0 licensed on GitHub, and the current release is v0.1.0-alpha.7.

TL;DR

• Size: v0.1.0-alpha.7, written in Rust, Apache 2.0, configured through 5 Kubernetes CRDs.
• Runs on: any GPU-enabled Kubernetes cluster running vLLM. Buoyant's benchmark used 3 L40S GPUs per pool.
• Performance: large gains on prefix-heavy traffic versus plain Kubernetes Service routing (self-reported).
• Best: median TTFT fell fro...

via MarkTechPost