Tokens straight off the GPU to the client you already use, and the GPU stops the moment they stop mattering.
Point your OpenAI-compatible client at the line and go. Nothing in your app has to change.
A hang-up, a missed deadline or a newer turn reaches the model and stops the generation. You stop paying for tokens nobody will hear.
ClutchKV saves the attention state of a conversation and loads it back on the next turn. In our vLLM benchmark the follow-up turn took 129 ms instead of 353 ms, with identical output.
We also serve open speech models on our own engine. In our runs of Coval's open method (not leaderboard entries), Qwen3-TTS 1.7B starts to speak in 53 ms (p50) on one RTX A5000, and Parakeet v3 gives the final transcript 42 ms after the caller stops on one H100 (49 ms on an A5000). One A5000 holds 64 live Qwen3-ASR streams with partial transcripts.
The router reads cache and load metrics from Triton and TensorRT-LLM and sends each request to the GPU that suits it.
Inference shares the same transport as voice, video and robotics. One line to operate, not five.
Anything llama.cpp loads (Qwen 2.5, Llama 3 family, Mistral, Gemma, Phi, the lot), plus first-class ONNX Runtime for the embedding/classifier side. GGUF in, OpenAI-compatible SSE out.
Open speech and robot models, on our own C++ engine. On one RTX A5000 ($0.27/hr): Qwen3-TTS 1.7B starts to speak in 53 ms p50 (64 concurrent calls per card), and Qwen3-ASR holds 64 live streams with partial text every half second. On one H100, Parakeet v3 gives the final transcript 42 ms (p50) after speech ends, and Whisper large-v3 64 ms. These latencies are our own runs of Coval's open benchmark method, not leaderboard entries. Against vLLM-Omni on the same card, at 64 calls, Voxtral-4B-TTS starts to speak in about 60 ms on our engine and in more than 6 s on vLLM-Omni. For robots, π0.5 (DROID) takes about 27 ms of model time per action chunk on an RTX PRO 6000 with 8-bit weights, on our C++ runtime inside our serving stack. Some of these weights (Fish Audio, Voxtral) carry non-commercial licences; you bring the licence for production use.
For the same reason voice does: a single dropped packet stalls every concurrent stream on HTTP/2 (TCP head-of-line blocking). With 16 concurrent users on one connection, that means everyone's tokens pause on one loss. QUIC streams are independent: only the affected user feels it.
GGUF weights are mmap'd at process start, so a warm worker answers without a load step. A cold model load (changing model mid-flight) takes as long as your disk needs to read the weights. We measure both on your hardware during the pilot.
Yes. Drop a GGUF into the tenant's weights bucket and point the agent at it. We don't re-quantize; what you upload is what runs.
Both. The default surface is OpenAI-compatible SSE: drop our endpoint into any client that already speaks `openai.chat.completions.stream(...)` and it works. Tool-calling, JSON mode, and streaming function call deltas all pass through. See OpenAI's streaming docs for the wire format we mirror.
FirstByte ships every modality on the same transport.