Ollama vs vLLM vs TensorRT-LLM for DeepSeek V3 at P95 400ms

Frequently Asked Questions

01.Can Ollama serve DeepSeek V3 at P95 under 400ms for retail demand forecasting?

Only at low concurrency. Ollama's single-request-optimized llama.cpp backend holds P95 latency near 380ms on an H100 at concurrency 1 to 4, but tail latency climbs past 900ms once concurrent forecasting jobs exceed roughly 8 to 12 requests because it lacks continuous batching. vLLM and TensorRT-LLM sustain the 400ms P95 bar up to 40 to 80 concurrent requests on equivalent hardware.

02.How does DeepSeek V3's mixture-of-experts architecture change inference engine choice?

DeepSeek V3 activates only 37 billion of its 671 billion total parameters per token through 8 routed experts plus 1 shared expert. Engines with expert-parallel routing and FP8 kernel support, such as vLLM and TensorRT-LLM, exploit this sparsity for higher throughput per GPU. Ollama's llama.cpp backend runs the model as a dense forward pass per request, so it does not capture the same batched expert-routing efficiency.

03.What GPU memory footprint does DeepSeek V3 require for production retail forecasting?

The full FP8 DeepSeek V3 checkpoint needs roughly 700GB of GPU memory before KV cache overhead, which requires an 8x H100 80GB node at minimum. Retail demand forecasting workloads with rolling 90-day SKU history windows add substantial KV cache pressure, so most production deployments reserve a second node purely for cache headroom during peak batch windows.