Serving an open-weight LLM with vLLM on an H100

This is the setup we use for the vLLM template on our H100 plans, written out step by step so you can reproduce it on a bare Ubuntu image or adapt it. The target is a production-shaped endpoint: Llama 3.1 70B Instruct in FP8 on a single 80 GB H100, an OpenAI-compatible API behind a reverse proxy with an API key, and enough measurement to know when it is time for a second card.

What fits on one H100

An 80 GB card holds a 70B model only if the weights are quantised. In FP8, Llama 3.1 70B weighs about 70 GB, which leaves roughly 8 GB for the KV cache after CUDA and vLLM overhead. That is enough for a 32k context at a handful of concurrent sequences, or an 8k context at around 40. If you need long contexts and high concurrency on 70B, you want two H100s with tensor parallelism; if you need neither, an 8B or 14B model on an RTX 5090 costs a fraction and answers faster.

The base system

Start from the Ubuntu 24.04 · CUDA 12.8 · driver 570 base. Confirm the card and driver, then install a Python environment. We use uv because it resolves the CUDA wheels in seconds.

nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv /opt/vllm && source /opt/vllm/bin/activate
uv pip install vllm==0.6.6 huggingface_hub[cli]

Log in to Hugging Face with a token that has accepted the Llama licence, and pre-download the weights to the persistent volume so a reinstall does not cost you 70 GB of transfer:

huggingface-cli login
huggingface-cli download neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 --local-dir /data/models/llama-70b-fp8

The flags that matter

vLLM has a hundred flags. Five of them decide whether the server is good.

vllm serve /data/models/llama-70b-fp8 \
  --served-model-name llama-3.1-70b \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 48 \
  --enable-prefix-caching \
  --host 127.0.0.1 --port 8000

• --max-model-len reserves KV cache per sequence. Set it to the longest prompt you will actually serve, not to the model's maximum; 16k halves the memory per slot compared with 32k.
• --gpu-memory-utilization 0.95 is safe on a card that runs nothing else. Leave it at the default 0.9 if you share the GPU with a second process.
• --max-num-seqs caps concurrent sequences. Too high and requests queue inside the scheduler with long tails; too low and the card idles. Start at 48 for 70B FP8 at 16k and adjust from the metrics.
• --enable-prefix-caching reuses KV blocks for identical prompt prefixes, which is most of a chat system prompt. It is close to free and often worth 20 to 30% throughput on chat traffic.
• --host 127.0.0.1: vLLM has no authentication. It must never listen on the public interface.

A systemd unit that survives reboots

cat > /etc/systemd/system/vllm.service <<'EOF'
[Unit]
Description=vLLM OpenAI-compatible server
After=network-online.target
[Service]
User=vllm
Environment=HF_HOME=/data/hf
ExecStart=/opt/vllm/bin/vllm serve /data/models/llama-70b-fp8 --served-model-name llama-3.1-70b --max-model-len 16384 --gpu-memory-utilization 0.95 --max-num-seqs 48 --enable-prefix-caching --host 127.0.0.1 --port 8000
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
useradd -r -d /data -s /usr/sbin/nologin vllm && chown -R vllm /data
systemctl daemon-reload && systemctl enable --now vllm
journalctl -fu vllm

Loading 70 GB of weights takes about 90 seconds from NVMe. The log line Uvicorn running on http://127.0.0.1:8000 means the server is ready.

A reverse proxy with an API key

Caddy gets a certificate automatically and can check a bearer token in three lines. Point a DNS record at the server first.

apt install -y caddy
cat > /etc/caddy/Caddyfile <<'EOF'
llm.example.com {
  @noauth not header Authorization "Bearer CHANGE-ME-32-RANDOM-CHARS"
  respond @noauth 401
  reverse_proxy 127.0.0.1:8000 {
    flush_interval -1
  }
}
EOF
systemctl reload caddy

flush_interval -1 is what makes streaming responses stream instead of buffering. Test with the OpenAI SDK by setting base_url to https://llm.example.com/v1 and api_key to the token; nothing else in your client changes.

What we measured

With the settings above, on one H100 SXM, using a 1,000-token prompt and 300-token completions:

The curve flattens at 40 to 48 concurrent sequences; beyond that, queueing only adds latency. If your traffic regularly sits above that, the next step is a second H100 with --tensor-parallel-size 2, which roughly doubles throughput and also unlocks BF16.

Monitoring

vLLM exposes Prometheus metrics at /metrics. The three worth alerting on are vllm:num_requests_waiting (anything above a handful for more than a minute means you are saturated), vllm:gpu_cache_usage_perc (above 90% means sequences are being preempted) and vllm:e2e_request_latency_seconds at p95. Add nvidia-smi dmon -s um or the netdata add-on for the card itself.

When to move on

Three signals say the setup has outgrown one card: the waiting-requests metric is rarely zero, your product needs 32k contexts at real concurrency, or you want to serve two models at once. All three are solved by the 2× and 4× H100 plans, where the same unit file works with the tensor-parallel flag added. Until then, this single card serves a surprising amount of traffic for a fixed monthly price.