LifeBytes Journal

Overview: This step-by-step guide shows how to run local large language models with GPU acceleration using Ollama and Open WebUI. You will install Ollama to serve models like Llama 3 or Mistral, and deploy Open WebUI as an easy web interface. The tutorial covers Ubuntu 22.04/24.04 and Windows 11 via WSL2, with security, auto-start, and troubleshooting tips.

What you need: A 16 GB RAM system (32 GB recommended), an NVIDIA GPU with recent drivers (8 GB+ VRAM recommended), admin/root access, a stable internet connection, and around 20–30 GB of free disk space for models and containers.

Why this stack? Ollama provides a simple runtime and model manager for local LLMs, while Open WebUI gives a modern, browser-based chat interface, prompt management, RAG integrations, and multi-model switching. Both are lightweight and work on a single machine.

Step 1 — Prepare GPU drivers

Ubuntu (bare metal/VM with GPU passthrough):

sudo apt update && sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, verify:

nvidia-smi

If you see your GPU, the driver is good. Ollama downloads the CUDA user-space libs it needs automatically; you only need a working NVIDIA driver.

Windows 11 with WSL2:

Install the latest NVIDIA Game Ready/Studio driver (535+), then update WSL:

wsl --update
wsl --shutdown

Open your Ubuntu WSL distro and confirm:

echo $WSL_DISTRO_NAME

Ollama will use the GPU via WSL automatically if the Windows driver supports it.

Step 2 — Install Docker (for Open WebUI)

Ubuntu:

sudo apt update && sudo apt install -y docker.io
sudo usermod -aG docker $USER
newgrp docker
sudo systemctl enable --now docker

Windows WSL2:

Install Docker Desktop for Windows and enable the WSL2 integration for your Ubuntu distro. Start Docker Desktop before running containers.

Step 3 — Install Ollama

Ubuntu/WSL2:

curl -fsSL https://ollama.com/install.sh | sh

Enable the service on Ubuntu (bare metal):

sudo systemctl enable --now ollama

Test by pulling a small model:

ollama pull llama3.1:8b

Run a quick prompt:

ollama run llama3.1:8b "Write a haiku about GPUs."

If you see a note like “using CUDA,” GPU is active. If not, ensure drivers are correct.

Step 4 — Deploy Open WebUI

Ubuntu (recommended: host networking):

docker run -d --name open-webui --restart unless-stopped --network host \
-e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
-e WEBUI_AUTH=true \
-e [email protected] \
-e DEFAULT_USER_PASSWORD=ChangeMeStrong! \
ghcr.io/open-webui/open-webui:main

Open your browser at http://127.0.0.1:8080 (host network uses the container’s internal port). Log in with the credentials you set.

WSL2 with Docker Desktop (no host networking):

docker run -d --name open-webui --restart unless-stopped -p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-e WEBUI_AUTH=true \
-e [email protected] \
-e DEFAULT_USER_PASSWORD=ChangeMeStrong! \
ghcr.io/open-webui/open-webui:main

Browse to http://localhost:3000. The variable OLLAMA_BASE_URL points Open WebUI to the local Ollama API.

Step 5 — Optimize and manage models

Popular models: llama3.1:8b, mistral-nemo:12b, qwen2.5:7b. You can also choose quantized variants like llama3.1:8b-instruct-q4_K_M for lower VRAM usage.

ollama list
ollama pull mistral-nemo:12b
ollama rm modelname:tag

To limit VRAM, use smaller models or quantized builds. In Open WebUI, select the model per chat. For longer contexts, try a 70B model on a bigger GPU or use smaller context windows to conserve memory.

Step 6 — Secure and persist

Keep Open WebUI bound to localhost if the machine is shared. If you must expose it, use a reverse proxy with TLS (Nginx, Caddy, or Traefik) and keep WEBUI_AUTH=true. On Ubuntu, ensure the firewall blocks unwanted access:

sudo ufw allow 8080/tcp comment 'Open WebUI (local)'
sudo ufw status

Data for Open WebUI is stored in the container’s volume by default. For manual control, mount a volume: -v open-webui:/app/backend/data. Ollama stores models under ~/.ollama; back that up regularly.

Step 7 — Auto-start on boot

Ollama installs a systemd service on Ubuntu. Ensure Docker is enabled (already done) and containers use --restart unless-stopped so Open WebUI comes up after reboots. On Windows, set Docker Desktop to start on login.

Troubleshooting

GPU not used: Update NVIDIA driver, reboot, and confirm with nvidia-smi (Ubuntu) or update WSL and drivers (Windows). Reinstall Ollama if needed. Check that ollama run logs mention CUDA.

Out-of-VRAM: Use a smaller or more aggressively quantized model. Reduce max tokens or context length in Open WebUI settings. Close other GPU apps.

Port conflicts: If 11434 or 8080/3000 are used, change them. For example, -p 3333:8080 for Open WebUI and OLLAMA_HOST=127.0.0.1:11500 ollama serve for Ollama.

Slow downloads: Use a reliable network, or pre-fetch models with ollama pull. You can also host a local model library if bandwidth is limited.

Clean up disk: Remove unused models and images:
ollama list && ollama rm model:tag
docker image prune -f

What’s next

Explore RAG in Open WebUI by attaching local documents, add embeddings, and test function calling or tool integrations. With this setup, you have a fast, private, and GPU-accelerated local AI workstation ready for coding, content, and research.

If you want a fast, private, and low-cost AI chat system that stays on your server, Ollama plus Open WebUI is a rock-solid choice. In this tutorial, you will deploy a local large language model stack on Ubuntu using Docker, enable NVIDIA GPU acceleration for speed, put it behind HTTPS with Caddy, and learn how to back it up, update it, and troubleshoot common issues.

What you will build: a three-container stack (Ollama + Open WebUI + Caddy) running on Ubuntu 22.04/24.04. Ollama hosts models (like Llama 3.1), Open WebUI provides a friendly chat interface with built-in auth, and Caddy terminates TLS with a free certificate. All traffic to Ollama is kept internal so only the web UI is exposed.

Prerequisites

- An Ubuntu 22.04 or 24.04 server with sudo access and outbound internet. - A domain name pointing to your server’s public IP (A/AAAA record). - Ports 80 and 443 reachable from the internet. - Optional but recommended: an NVIDIA GPU (T4, A10, RTX 30/40, etc.). CPU-only mode also works, just slower. - 16 GB RAM minimum recommended for 7–8B models; more for larger models.

Step 1 — Install Docker and Compose

Install Docker Engine and the Compose plugin.

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release; echo $VERSION_CODENAME) stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker $USER
newgrp docker
docker compose version

Step 2 — Enable NVIDIA GPU (optional but recommended)

If your server has an NVIDIA GPU, install the NVIDIA drivers on the host and the NVIDIA Container Toolkit so Docker can use the GPU. If you plan to run CPU-only, skip this step.

# Install the NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Sanity check: should show GPU info
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If the sanity check fails, verify drivers with nvidia-smi on the host and ensure Secure Boot isn’t blocking the kernel modules.

Step 3 — Create docker-compose.yml

This Compose file deploys Ollama, Open WebUI, and Caddy. The Ollama API is bound to localhost for safety; only Caddy listens publicly with HTTPS. Replace your.domain.com and the email in the Caddyfile later.

mkdir -p ~/private-ai && cd ~/private-ai
cat > docker-compose.yml <<'YAML'
version: "3.9"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    environment:
      - OLLAMA_KEEP_ALIVE=24h
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama
    gpus: all
    networks: [ai]

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    depends_on: [ollama]
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_AUTH=True
    volumes:
      - open-webui:/app/backend/data
    ports:
      - "127.0.0.1:3000:8080"
    restart: unless-stopped
    networks: [ai]

  caddy:
    image: caddy:2
    container_name: caddy
    depends_on: [open-webui]
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    ports:
      - "80:80"
      - "443:443"
    restart: unless-stopped
    networks: [ai]

networks:
  ai:
    driver: bridge

volumes:
  ollama:
  open-webui:
  caddy_data:
  caddy_config:
YAML

Note: If you do not have an NVIDIA GPU or Docker Compose errors on the gpus: all line, remove that line and run CPU-only. Performance will be slower.

Step 4 — Add a Caddyfile for HTTPS

Create a simple Caddyfile that reverse-proxies your domain to Open WebUI and provisions a free TLS certificate automatically.

cat > Caddyfile <<'CADDY'
your.domain.com {
  encode gzip
  reverse_proxy open-webui:8080
  tls [email protected]
  header {
    Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
    X-Content-Type-Options "nosniff"
    X-Frame-Options "DENY"
    Referrer-Policy "no-referrer-when-downgrade"
  }
}
CADDY

Ensure your DNS A/AAAA record for your.domain.com points to this server’s public IP and that ports 80 and 443 are open in any firewall or cloud security group.

Step 5 — Launch the stack

Start everything with one command:

docker compose up -d
docker compose ps

Visit https://your.domain.com in a browser. The first user who signs up becomes admin in Open WebUI. Keep your credentials safe. By default, the Ollama API is not exposed publicly; it is only reachable by Open WebUI inside the Docker network.

Step 6 — Pull a model and test

Use Ollama to download a model. Smaller models start faster and fit more GPUs; larger models are smarter but need more VRAM.

# Example: Llama 3.1 8B
docker exec -it ollama ollama pull llama3.1:8b

# Quick API test (CPU/GPU both work)
curl -s http://127.0.0.1:11434/api/generate \
  -d '{"model":"llama3.1:8b","prompt":"Say hello in one sentence."}' | jq .

Open WebUI will list the model automatically. Start chatting at https://your.domain.com and choose the model in the UI. If VRAM is limited, try 7B/8B variants or quantized builds (e.g., Q4_K_M).

Step 7 — Tuning for performance and memory

- GPU VRAM: 8B models typically need 6–10 GB VRAM depending on quantization; 13B often needs 10–16 GB. If you run out of memory, pick a smaller or more heavily quantized model. - Context length: use smaller context (e.g., 4096) for speed; larger contexts consume more RAM/VRAM. - Batching: in Open WebUI, keep concurrent chats lower on small GPUs. - Keep-alive: OLLAMA_KEEP_ALIVE=24h keeps models warm and reduces first-token latency at the cost of memory.

Step 8 — Back up and update safely

Your data lives in Docker volumes. Back them up regularly (especially Open WebUI data if you store conversations). These commands create tar archives in the current directory.

# Stop the stack before a consistent backup (optional but recommended)
docker compose down

# Back up Ollama models and cache
docker run --rm -v ollama:/data -v "$(pwd)":/backup busybox \
  tar czf /backup/ollama-$(date +%F).tgz -C /data .

# Back up Open WebUI data (users, settings, history)
docker run --rm -v open-webui:/data -v "$(pwd)":/backup busybox \
  tar czf /backup/openwebui-$(date +%F).tgz -C /data .

# Bring the stack back up
docker compose up -d

To update to the latest versions, pull new images and recreate containers without losing data:

cd ~/private-ai
docker compose pull
docker compose up -d

Step 9 — Troubleshooting

- GPU not detected in containers: run docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi. If it fails, reinstall the NVIDIA driver and the container toolkit, and restart Docker. - Permission denied with Docker: add your user to the docker group and re-login (usermod -aG docker $USER + newgrp docker). - HTTPS not provisioning: make sure the domain’s DNS points to the server, and ports 80/443 are open and not used by another service (stop Apache/NGINX if present). - 502/Bad Gateway from Caddy: check docker compose logs open-webui to ensure it started; it can take 10–30 seconds on the first run. - Out-of-memory or crashes when chatting: choose a smaller model (e.g., 7B/8B), use a more aggressive quantization, or reduce context length. - Compose complains about gpus: remove the gpus: all line and start CPU-only, or run Ollama via docker run --gpus all instead of Compose.

What’s next?

You now have a production-ready private AI chat server with HTTPS and optional GPU acceleration. Explore model variants (instruction-tuned, coding, reasoning), enable role-based access in Open WebUI, add nightly backups, and monitor GPU/CPU usage with tools like nvtop and cAdvisor. For advanced setups, place the stack behind a VPN or zero-trust proxy and add per-user rate limits.