How to Deploy Gemma 3 on a Dedicated GPU Server with vLLM and Docker

vLLM serves the model through an OpenAI-compatible HTTP API, and Docker keeps the Python and CUDA dependencies isolated from the host.

Choose a model size

Gemma 3 comes in 1B, 4B, 12B and 27B sizes. The 4B, 12B and 27B models accept images as well as text and support a 128K-token context window. The 1B model is text-only with a shorter context. This guide uses the 4B model; the others work with the same commands and a different --model value.

The figures below are for bfloat16 weights. Weights are only part of the picture: the KV cache, vision encoder and activations need additional VRAM, so treat the GPU column as a practical minimum rather than an exact requirement.

Prerequisites

• A dedicated GPU server running Ubuntu 22.04 or 24.04 LTS, with sudo access.
• A Hugging Face account that has accepted the Gemma license on the model page (the model is gated, and downloads fail with a 401 or 403 error otherwise).
• A Hugging Face access token with read permission, created under Settings → Access Tokens.

1. Install the NVIDIA driver

Only the driver is needed on the host. The CUDA runtime ships inside the vLLM container.

sudo apt-get update
sudo ubuntu-drivers devices     # shows the recommended driver
sudo ubuntu-drivers autoinstall
sudo reboot

After the reboot, confirm the driver is loaded:

nvidia-smi

The output lists your GPUs, the driver version, and the highest CUDA version that driver supports. The driver must support the CUDA version used by the vLLM image you run (shown in that release's notes). A current driver from the 550 series or newer covers recent vLLM images.

2. Install Docker and the NVIDIA Container Toolkit

The NVIDIA Container Toolkit lets containers access the host GPUs.

# Docker (Ubuntu's package; Docker's official apt repository also works)
sudo apt-get install -y docker.io curl gnupg
sudo systemctl enable --now docker

# NVIDIA Container Toolkit repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# Register the NVIDIA runtime with Docker and restart it
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Check that containers can see the GPU:

sudo docker run --rm --gpus all ubuntu nvidia-smi

Note: This guide runs Docker with sudo. Adding your user to the docker group also works, but the change only applies after you log out and back in.

3. Store the Hugging Face token and model cache

Keep the token in a root-only environment file instead of typing it into commands or unit files, where it would end up in shell history or process listings.

# Persistent directory for downloaded weights
sudo mkdir -p /var/lib/gemma-vllm/hf-cache

# Create the env file with restrictive permissions first, then write the token
sudo install -m 600 /dev/null /etc/gemma-vllm.env
read -rsp "Hugging Face token: " HF_INPUT; echo
echo "HF_TOKEN=${HF_INPUT}" | sudo tee /etc/gemma-vllm.env > /dev/null
unset HF_INPUT

The cache directory is created once and reused by every run, so weights are downloaded only the first time.

4. Run vLLM in a test container

Start with a manual container to confirm the hardware and model work before adding systemd.

sudo docker run -d \
  --name gemma-server \
  --gpus all \
  --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v /var/lib/gemma-vllm/hf-cache:/root/.cache/huggingface \
  --env-file /etc/gemma-vllm.env \
  vllm/vllm-openai:v0.8.4 \
  --model google/gemma-3-4b-it \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

What the options do:

• vllm/vllm-openai:v0.8.4: A pinned image tag, so an upstream release cannot change behavior underneath you. Check the vLLM releases page for a newer version that supports Gemma 3, and test it before moving production traffic.
• -p 127.0.0.1:8000:8000: Publishes the API on the loopback interface only. Docker bypasses common firewall rules for published ports, so binding to localhost is the safest default.
• --dtype bfloat16: Matches the precision Gemma 3 was trained for.
• --max-model-len 8192: Limits the context length. Without a limit, vLLM sizes the KV cache for the model's full 128K window, which fails or leaves little room for concurrent requests on most GPUs. Raise it as far as your VRAM allows.
• --gpu-memory-utilization 0.90: The fraction of GPU memory vLLM may claim. Lower it if the GPU is shared with other processes.
• --ipc=host: Recommended by the vLLM Docker documentation so worker processes can use shared memory.

Watch startup. The first run downloads the weights, which can take several minutes:

sudo docker logs -f gemma-server

The server is ready when the log shows that application startup is complete.

5. Test the API

List the loaded model:

curl http://127.0.0.1:8000/v1/models

Send a text request:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-3-4b-it",
    "messages": [
      {"role": "user", "content": "Explain in two sentences what GPU acceleration does for AI inference."}
    ],
    "temperature": 0.7,
    "max_tokens": 256
  }'

Send an image request. Replace the URL with any publicly reachable JPEG or PNG:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-3-4b-it",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "Describe this image."},
          {"type": "image_url", "image_url": {"url": "https://example.com/your-image.jpg"}}
        ]
      }
    ],
    "max_tokens": 256
  }'

The server downloads the image itself, so it needs outbound internet access, and some hosts reject automated downloads. If the request fails, send the image as a base64 data URI instead (data:image/jpeg;base64,...).

6. Run it as a systemd service

Remove the test container first so the names do not collide:

sudo docker rm -f gemma-server

Create /etc/systemd/system/gemma-vllm.service:

[Unit]
Description=Gemma 3 vLLM inference server
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=simple
Restart=always
RestartSec=10
TimeoutStopSec=60

# Remove any leftover container with the same name (the leading "-" ignores errors)
ExecStartPre=-/usr/bin/docker rm -f gemma-server

ExecStart=/usr/bin/docker run --rm \
  --name gemma-server \
  --gpus all \
  --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v /var/lib/gemma-vllm/hf-cache:/root/.cache/huggingface \
  --env-file /etc/gemma-vllm.env \
  vllm/vllm-openai:v0.8.4 \
  --model google/gemma-3-4b-it \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

ExecStop=/usr/bin/docker stop gemma-server

[Install]
WantedBy=multi-user.target

Optionally, enable the NVIDIA persistence daemon so the driver stays initialized between runs, and add nvidia-persistenced.service to the After= line:

sudo systemctl enable --now nvidia-persistenced

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now gemma-vllm
sudo systemctl status gemma-vllm
sudo journalctl -u gemma-vllm -f

7. Scaling and hardening

• Larger models: Change the --model value to google/gemma-3-12b-it or google/gemma-3-27b-it, and use a GPU setup from the sizing table.
• Multiple GPUs: Add --tensor-parallel-size N, where N is the number of GPUs, to split the weights across cards. Use matching GPUs, and keep --ipc=host.
• Quantized models: To fit the 12B model on a single 24 GB GPU you need a 4-bit checkpoint. Find an AWQ or GPTQ build of gemma-3-12b-it on Hugging Face from a source you trust, use it as the --model value, and add the matching --quantization flag. Quantized runs usually need a different --dtype than bfloat16, so check the model card, and compare output quality against the full-precision model before relying on it.
• Authentication and TLS: The vLLM server accepts an --api-key argument, but it does not provide TLS. Keep the localhost binding and put a reverse proxy such as Nginx or Caddy in front for HTTPS, API key checks and rate limiting.
• Monitoring: vLLM exposes Prometheus metrics at /metrics on the same port by default. No extra flag is needed.

Troubleshooting