Buoyant, the company behind Linkerd, has released Kinference, an open-source project for building inference platforms on Kubernetes. Placed in front of a set of vLLM servers, it decides, for each request, which server it is sent to. The project is distributed under the Apache 2.0 license on GitHub, in its v0.1.0-alpha.7 version.
- Kinference routes each request to the right vLLM server on Kubernetes.
- It applies routing policies, token rate limits, KV cache-aware load balancing, and per-tenant metrics.
- Buoyant reports a 95% lower time to first token and 7.4 times more throughput on shared-prefix traffic, compared with random routing.
- The project is still in alpha, and rate limiting, classifiers, and Anthropic API support are not yet available.
A Control Layer on Top of vLLM
Kinference combines Kubernetes, the container orchestration platform, and vLLM, the engine that runs language models. According to Buoyant’s FAQ, vLLM constitutes the data plane, while Kinference acts as the control plane. In a single layer, it applies routing policies, token rate limits, load balancing that takes the KV cache into account—that is, the memory storing already performed computations—as well as per-tenant metrics.
The system comprises two elements: a traffic gateway and an inference manager. The latter exposes a gRPC interface, receives a request, and returns the selected endpoint, along with the potentially modified request body. It works with a proxy, such as the Buoyant Gateway. Kinference monitors KV cache events and engine metrics, and relies on NVIDIA’s dynamo-kv-router crate to process these events. Its configuration is based on five Kubernetes resources: GatewayBackend, InferencePolicy, InferencePool, InferenceProvider, and InferenceWorkerSet.
Inference policies are declarative rules that specify how each request is routed or rejected. They can be based on the tenant, the requested model, the state of GPU pools, or semantic labels. The project’s website provides several examples, such as sending simple requests to a smaller model, or falling back to a cloud provider like Bedrock when a server has more than 2 pending requests. Policies are stored in git and can be validated offline using the kinfctl command-line tool.
Results Measured by Buoyant
Buoyant compared Kinference to simple Kubernetes load balancing. Each configuration used three vLLM 0.31.0 pods, each on an L40S GPU, with the Qwen3.8-27B-FP8 model. On traffic sharing a common prefix, the median time to first token (TTFT) latency reportedly drops from 6.88 s to 0.31 s, a decrease of about 95%. Without a shared prefix, the figures are 6.92 s versus 6.83 s, with no significant difference according to the publisher.
In a capacity test, the useful throughput, measured in requests per minute meeting a TTFT target, reached 103.0 versus 14.0 req/min on the shared-prefix traffic, or about 7.4 times higher. These figures are self-reported. The baseline is random routing, not another cache-aware router like llm-d or Dynamo. Buoyant also notes that its thresholds of 8 s and 4 s are more suitable for offline agents than for chatbots. The comparison table published with the article presents Kinference’s distinctive angle as governance—namely policies, model aliases, fallbacks to providers, and per-tenant accounting—while llm-d and Dynamo are described as more performant on raw serving.
The project remains in alpha phase. Rate limiting, classifiers, and Anthropic API support are not yet delivered.
