We’ve optimized 30+ deployments for companies that own their GPUs but have no performance team. We find out why your inference is slow, fix it, and give you a report you can sign off on — with before/after numbers your own people can re-run. You don’t need to hire a CUDA specialist, and you don’t need to take our word for it.
We work with companies that bought GPUs through their budget process — and whose IT team runs the infrastructure but doesn’t do kernel- or bandwidth-level tuning. That is not a weakness: this skill is needed once or twice a year, not every day.
You own more of your own hardware than almost anyone — and have the least room to hire for it.
Your data cannot leave the building. So “we work inside your environment” isn’t a bonus — it’s the condition.
Every request reads dozens of pages. The pain sits in the first token, not in throughput.
Compliance signs off on the purchase, not on the performance. Nobody checks whether the cards are actually working.
The GPUs are in the rack, the model is serving. But under real load it doesn’t hold up — and nobody here can tell whether that’s the hardware, the model, or a setting.
That was already the answer last year. We would rather know whether the money we have already spent is doing its job.
A machine-learning performance engineer costs $200,000–400,000 a year — and even if we approved that, our pay structure tops out far below it. This is a service we can buy. It is not a position we can fill.
Someone tried something once. It felt faster. That is not a result you can take to management.
If you already have a dedicated inference performance engineer — full time, with time to spare, and authorised to sign the result off — you probably don’t need us. We will say so on the first call.
The diagnostic is valuable on its own — it tells you how much headroom is actually left, and whether it is worth acting on.
We measure your current state as it actually runs — hardware, stack, model, real workload. You get a reproducible baseline and a reachable target.
Fix the bottleneck, verify each change individually. Nothing ships on a hunch, and every gain is reported next to its cost.
Load changes, models change, the next card purchase comes around. We keep the numbers current and answer the question that actually costs money.
Doubling your GPU count is not a small line item: an 8×A100 node runs around $48,000 and an 8×H100 node around $240,000 — before procurement time and downtime.
We charge as one project against one outcome. If we don’t reach the agreed target, we don’t charge for that stage.
Generating a token at batch size 1 reads every weight exactly once. So the ceiling on a single stream is pure physics — not a framework trick.
Memory-bandwidth utilization at batch size 1, RTX 4090 (1008 GB/s peak). Row 5 is the point: FP8 lowers utilization while raising throughput — because each token moves fewer bytes. That is the only lever left.
Every row was produced on a single RTX 4090 at batch size 1. Published so you can try to disprove them.
| Model | Stack | Precision | tok/s @ batch 1 | Bandwidth | Verdict |
|---|---|---|---|---|---|
| Qwen2.5-0.5B | transformers | FP16 | 55.1 | 5.1% | overhead-bound |
| Qwen2.5-7B | transformers | FP16 | 40.2 | 56.6% | headroom |
| Qwen2.5-7B | vLLM | FP16 | 63.0 | 88.7% | at the wall |
| Qwen2.5-7B | vLLM | FP8 W | 90.5 | 73.4% | 1.44× |
| Qwen3-32B | vLLM | INT4 AWQ | 47.0 | 83.8% | KV-limited |
All rows: RTX 4090, batch size 1, identical script for before/after. Measuring scripts ship with every report.
Same flag. Same card. Same vLLM version. Two completely different outcomes:
On the 7B model it destroyed the model — GSM8K output became random digits, and it ran faster because the output was garbage. On the 32B model it was near-lossless. vLLM’s own FP8 KV-cache validation covers Hopper and Blackwell; Ada (sm_89) is outside that scope.
So the right question is not “is FP8 KV-cache good?” but “will it break the model on this card, this model and this engine build?” — and that question does not have a search answer.
A tuned vLLM baseline reaches 88.7% of peak memory bandwidth. Once you are there, only 1.13× remains — and the way forward is smaller weights, not scheduler tuning.
why is my vLLM inference slow · 8 minEveryone says FP8 is “effectively lossless.” Our token-level comparison found 5 of 6 prompts diverging, with an average prefix agreement of 55.5%.
FP8 quantization accuracy · 11 minOne flag destroyed a model and left another untouched. Why the official validation probably doesn’t cover your GPU — and how to check it yourself in 20 minutes.
KV cache quantization · 9 minNo. On Qwen2.5-7B with vLLM on an RTX 4090 we measured GSM8K falling from 89.23% to 88.17% and wikiText perplexity rising from 7.15 to 7.20. Comparing token by token across 6 prompts, 5 of 6 diverged from the FP16 baseline, with an average prefix agreement of 55.5%. A small but real cost — and a larger one than “lossless” suggests.
It depends on the GPU, model and engine build — and the outcome can be catastrophic. The same flag, same card, same vLLM version: on Qwen2.5-7B, GSM8K collapsed from 89.23% to 1.14%. On Qwen3-32B, perplexity was essentially unchanged (+0.33%) with 21–29% more throughput at concurrency 64. vLLM’s own validation covers Hopper and Blackwell; Ada (sm_89) is outside that scope.
We measured 88.7% memory-bandwidth utilization on an already-tuned vLLM FP16 baseline — 894 GB/s out of a 1008 GB/s peak. Once you are at the bandwidth wall, the only remaining lever is making the weights smaller; scheduler tuning buys you nothing.
No. All work runs in a test environment. If you prefer, we work entirely inside your network so no data leaves it. We sign NDAs, and the access method — SSH, VPN, jump host, remote desktop — is your call.
The target is agreed during the diagnostic stage and attached to the contract. If the optimization stage does not reach it, we do not charge for that stage.
Already running vLLM, and you do have a performance engineer? We still find headroom others miss — our measurements show where.
A short call is usually enough: model, card, current performance. If we don’t see room to improve, we say so — even though that means no project for us.