Senior AI Engineer | Building & Telling Stories about AI/ML Systems | Software Engineer
Your GPU is sitting idle while your LLM inference requests queue up, one by one, painfully slow.
You know there's a better way. You've heard about continuous batching, PagedAttention, and throughput numbers that seem too good to be true.
Welcome to vLLM—the inference engine that's quietly becoming the backbone of production LLM deployments everywhere.
Running large language models in production isn't just about downloading weights and calling a predict function.
It's about squeezing every drop of performance from expensive GPU hardware. It's about handling hundreds of concurrent requests without melting your infrastructure. It's about exposing a clean API that your applications can actually use.
vLLM solves all of these problems, and Docker makes deploying it almost embarrassingly simple. In this guide, you'll learn exactly how to deploy vLLM using Docker, configure it for your specific needs, and scale it to production with Kubernetes. Let's dive in.
vLLM is a high-throughput inference engine designed specifically for large language models. Developed by researchers at UC Berkeley, it introduces PagedAttention—a memory management technique that treats the KV cache like virtual memory pages. This seemingly simple innovation unlocks massive efficiency gains.
Traditional LLM serving wastes enormous amounts of GPU memory on the key-value cache. Each request reserves a contiguous block of memory, even when it doesn't need it all. vLLM's PagedAttention allocates memory in small blocks on demand, dramatically reducing waste. The result? You can serve 2-4x more concurrent requests on the same hardware. docs.vllm
But memory efficiency is only part of the story. vLLM implements continuous batching, which means new requests can join an existing batch mid-inference. Traditional batching waits for an entire batch to complete before starting the next one. Continuous batching keeps your GPU busy almost constantly, maximizing throughput.
The cherry on top? vLLM exposes an OpenAI-compatible API out of the box. Your existing applications that call OpenAI's /v1/chat/completions endpoint can switch to vLLM with a single URL change. No code rewrites. No SDK changes. Just point to your new endpoint and go.
Understanding how vLLM works under the hood helps you make better deployment decisions. Let's walk through the request lifecycle.
When vLLM starts, it loads model weights from Hugging Face Hub (or a local path) into GPU memory. It initializes the tokenizer, configures the KV cache based on available memory, and prepares the inference engine. docs.vllm
The KV cache configuration is crucial. vLLM calculates how many tokens it can cache based on your --gpu-memory-utilization setting. Higher utilization means more concurrent requests but less headroom for spikes.
Your application sends an HTTP POST to /v1/chat/completions or /v1/completions. The request includes your prompt, generation parameters (temperature, max_tokens, etc.), and optional streaming flags. docs.premai
vLLM's server validates the request, tokenizes the input, and adds it to the scheduling queue. If you've configured an API key, authentication happens at this stage.
Here's where vLLM's magic happens. The scheduler doesn't wait for a full batch before starting inference. Instead, it uses continuous batching to add new requests to running batches whenever there's capacity. ideas2it
PagedAttention manages memory allocation. Each request gets blocks of KV cache as needed, not a pre-allocated contiguous chunk. When a request finishes, its blocks return to the pool immediately.
The inference loop runs the transformer forward pass on the current batch. For each request, vLLM updates the KV cache, samples the next token based on your temperature/top_p settings, and checks stopping conditions. x
This loop repeats until all requests in the batch hit their max_tokens limit or generate a stop token. New requests can join mid-loop thanks to continuous batching.
If streaming is enabled, vLLM sends tokens back as they're generated using server-sent events (SSE). Otherwise, it accumulates the full response and returns it as a JSON object—matching OpenAI's response format exactly. docs.vllm
Now let's get practical.
The fastest way to run vLLM is with the official vllm/vllm-openai Docker image. This image packages everything you need: vLLM, its dependencies, and the OpenAI-compatible server.
Before you start, ensure you have:
Docker with NVIDIA GPU support: Install the NVIDIA Container Toolkit (nvidia-docker2 or the newer nvidia-container-toolkit).
A capable GPU: vLLM works best with NVIDIA GPUs that have sufficient VRAM for your target model.
Hugging Face token (optional): Required only for gated models like Llama 3.
Verify your GPU is visible to Docker:
docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi
You should see your GPU listed. If not, troubleshoot your NVIDIA Container Toolkit installation before proceeding.
Here's the simplest command to run vLLM:
docker run --rm \
--gpus all \
-p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e HUGGING_FACE_HUB_TOKEN=<your_hf_token> \
vllm/vllm-openai:latest \
--model mistralai/Mistral-7B-v0.1
Let's break down each flag:
--rm: Remove the container when it stops. Clean and simple for testing.
--gpus all: Expose all host NVIDIA GPUs to the container. You can also specify --gpus '"device=0"' for a single GPU.
-p 8000:8000: Map container port 8000 to host port 8000. vLLM's server listens here by default.
-v ~/.cache/huggingface:/root/.cache/huggingface: Mount your local HF cache to avoid re-downloading models on every run.
-e HUGGING_FACE_HUB_TOKEN: Your HF token for accessing gated models.
--model: The model to load. Can be a HF model ID or a local path.
The server starts, downloads the model (if not cached), and begins listening for requests.
Once the server is running, test it with a simple curl command:
curl http://localhost:8000/v1/models
You should see a JSON response listing your loaded model. Now try a chat completion:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "mistralai/Mistral-7B-v0.1",
"messages": [
{"role": "user", "content": "Explain Docker in one sentence."}
],
"max_tokens": 100
}'
The response follows OpenAI's format exactly. Your existing OpenAI client libraries will work without modification—just change the base URL.
The minimal setup gets you running, but production deployments need tuning. vLLM offers extensive configuration options through command-line arguments.
Switch models by changing the --model argument:
docker run --rm --gpus all -p 8000:8000 \
vllm/vllm-openai:latest \
--model meta-llama/Meta-Llama-3-8B-Instruct
For local models, mount the model directory and point to it:
docker run --rm --gpus all -p 8000:8000 \
-v /path/to/models:/models \
vllm/vllm-openai:latest \
--model /models/my-fine-tuned-llama
In production, you don't want an open endpoint. vLLM supports API key authentication:
docker run --rm --gpus all -p 8000:8000 \
-e VLLM_API_KEY=your-secret-key \
vllm/vllm-openai:latest \
--model mistralai/Mistral-7B-v0.1
Clients must include the key in the Authorization header:
curl http://localhost:8000/v1/models \
-H "Authorization: Bearer your-secret-key"
By default, vLLM uses 90% of available GPU memory. You can adjust this with --gpu-memory-utilization:
docker run --rm --gpus all -p 8000:8000 \
vllm/vllm-openai:latest \
--model mistralai/Mistral-7B-v0.1 \
--gpu-memory-utilization 0.85
Lower values leave more headroom for memory spikes but reduce the maximum number of concurrent requests. Higher values maximize throughput but risk out-of-memory errors under heavy load. nm-vllm.readthedocs
For models too large for a single GPU, use tensor parallelism:
docker run --rm --gpus all -p 8000:8000 \
vllm/vllm-openai:latest \
--model meta-llama/Meta-Llama-3-70B-Instruct \
--tensor-parallel-size 4
This splits the model across 4 GPUs. Ensure your container has access to all required GPUs.
Some models support longer contexts than their default configuration. Override with --max-model-len:
docker run --rm --gpus all -p 8000:8000 \
vllm/vllm-openai:latest \
--model mistralai/Mistral-7B-v0.1 \
--max-model-len 16384
Be careful: longer contexts require more KV cache memory, reducing concurrent capacity.
For memory-constrained deployments, enable quantization:
docker run --rm --gpus all -p 8000:8000 \
vllm/vllm-openai:latest \
--model TheBloke/Mistral-7B-v0.1-AWQ \
--quantization awq
vLLM supports AWQ, GPTQ, and other quantization formats. Quantized models use less memory at the cost of some quality.
The official image works for most cases, but sometimes you need customizations. Maybe you want to pin a specific vLLM version, add custom dependencies, or bake in your model weights.
Clone the vLLM repository and build:
git clone https://github.com/vllm-project/vllm.git
cd vllm
DOCKER_BUILDKIT=1 docker build . \
--target vllm-openai \
--tag my-vllm:latest \
--file docker/Dockerfile
The build takes a while—it compiles CUDA kernels for your target architecture. Once complete, run your custom image:
docker run --rm --gpus all -p 8000:8000 \
my-vllm:latest \
--model mistralai/Mistral-7B-v0.1
For faster startup in production, bake model weights into the image:
FROM vllm/vllm-openai:latest
# Download model during build
RUN python -c "from huggingface_hub import snapshot_download; \
snapshot_download('mistralai/Mistral-7B-v0.1', local_dir='/models/mistral-7b')"
# Set default model
CMD ["--model", "/models/mistral-7b"]
Build and push to your registry. Now containers start instantly without downloading models.
PS:
If you like this article, share it with others ♻️
Would help a lot ❤️
And feel free to follow me for more content like this.
Docker is great for single-node deployments. For production clusters, Kubernetes provides orchestration, scaling, and resilience. vLLM deploys cleanly on K8s with GPU node pools. docs.vllm
Your Kubernetes cluster needs:
NVIDIA device plugin: Makes nvidia.com/gpu resources visible to the scheduler.
GPU node pool: At least one node with NVIDIA GPUs.
kubectl access configured.
Create a namespace for your vLLM deployment:
kubectl create namespace vllm
If you're using gated models, create a secret for your HF token:
kubectl create secret generic hf-token \
--from-literal=token=<your_hf_token> \
--namespace vllm
Here's a production-ready Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-server
namespace: vllm
spec:
replicas: 1
selector:
matchLabels:
app: vllm-server
template:
metadata:
labels:
app: vllm-server
spec:
containers:
- name: vllm
image: vllm/vllm-openai:latest
args:
- "--model"
- "mistralai/Mistral-7B-v0.1"
- "--host"
- "0.0.0.0"
- "--port"
- "8000"
- "--gpu-memory-utilization"
- "0.9"
env:
- name: HUGGING_FACE_HUB_TOKEN
valueFrom:
secretKeyRef:
name: hf-token
key: token
ports:
- containerPort: 8000
name: http
resources:
requests:
cpu: "2"
memory: "6Gi"
nvidia.com/gpu: "1"
limits:
cpu: "4"
memory: "16Gi"
nvidia.com/gpu: "1"
volumeMounts:
- mountPath: /root/.cache/huggingface
name: cache-volume
- name: shm
mountPath: /dev/shm
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 60
periodSeconds: 5
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 60
periodSeconds: 10
volumes:
- name: cache-volume
emptyDir: {}
- name: shm
emptyDir:
medium: Memory
Key configuration points:
GPU resources: Request and limit nvidia.com/gpu: "1" to ensure GPU scheduling.
Shared memory: Mount /dev/shm as a memory-backed emptyDir for PyTorch's multiprocessing.
Health probes: vLLM exposes /health for readiness and liveness checks.
Initial delay: 60 seconds gives the model time to load before health checks start.
HF cache volume: Persists downloaded models across pod restarts within the same node.
Apply the deployment:
kubectl apply -f vllm-deployment.yaml
Create a Service to make vLLM accessible within your cluster: docs.vllm
apiVersion: v1
kind: Service
metadata:
name: vllm-service
namespace: vllm
spec:
type: ClusterIP
selector:
app: vllm-server
ports:
- name: http
port: 8000
targetPort: 8000
Apply it:
kubectl apply -f vllm-service.yaml
Other pods in your cluster can now reach vLLM at:
http://vllm-service.vllm.svc.cluster.local:8000
For external access, add an Ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: vllm-ingress
namespace: vllm
annotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
spec:
ingressClassName: nginx
rules:
- host: vllm.your-domain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: vllm-service
port:
number: 8000
The timeout annotations are important—LLM requests can take 30+ seconds for long generations.
For more complex deployments, vLLM provides an official Helm chart: docs.vllm
helm repo add vllm https://vllm-project.github.io/helm-charts
helm repo update
helm upgrade --install vllm vllm/vllm \
--namespace vllm --create-namespace \
--set resources.requests."nvidia\.com/gpu"=1 \
--set resources.limits."nvidia\.com/gpu"=1 \
--set extraArgs="{--model,mistralai/Mistral-7B-v0.1}"
The Helm chart supports autoscaling, multiple replicas, and advanced configuration through values files. vllm.hyper
vLLM transforms LLM deployment from a research project into a production-ready operation.
PagedAttention and continuous batching deliver throughput that was previously impossible without custom infrastructure. The OpenAI-compatible API means you can adopt vLLM without rewriting your applications.
Docker makes deployment straightforward: pull the official image, expose your GPUs, and start serving. For production scale, Kubernetes provides the orchestration layer you need for resilience and scaling.
The key decisions you'll make are:
GPU memory utilization: Balance concurrent capacity against OOM risk.
Model selection: Match model size to your GPU capabilities.
Scaling strategy: Decide between horizontal replicas or tensor parallelism.
Caching and persistence: Avoid cold start delays with persistent model storage.
Start with the minimal Docker command, validate your setup with a few test requests, then iterate toward production. vLLM's extensive configuration options let you tune performance as you learn your workload patterns.
Your GPU no longer needs to sit idle. Deploy vLLM, and put that hardware to work.
PS:
If you like this article, share it with others ♻️
Would help a lot ❤️
And feel free to follow me for more content like this.