LifeBytes Journal

Overview: This step-by-step guide shows how to run local large language models with GPU acceleration using Ollama and Open WebUI. You will install Ollama to serve models like Llama 3 or Mistral, and deploy Open WebUI as an easy web interface. The tutorial covers Ubuntu 22.04/24.04 and Windows 11 via WSL2, with security, auto-start, and troubleshooting tips.

What you need: A 16 GB RAM system (32 GB recommended), an NVIDIA GPU with recent drivers (8 GB+ VRAM recommended), admin/root access, a stable internet connection, and around 20–30 GB of free disk space for models and containers.

Why this stack? Ollama provides a simple runtime and model manager for local LLMs, while Open WebUI gives a modern, browser-based chat interface, prompt management, RAG integrations, and multi-model switching. Both are lightweight and work on a single machine.

Step 1 — Prepare GPU drivers

Ubuntu (bare metal/VM with GPU passthrough):

sudo apt update && sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, verify:

nvidia-smi

If you see your GPU, the driver is good. Ollama downloads the CUDA user-space libs it needs automatically; you only need a working NVIDIA driver.

Windows 11 with WSL2:

Install the latest NVIDIA Game Ready/Studio driver (535+), then update WSL:

wsl --update
wsl --shutdown

Open your Ubuntu WSL distro and confirm:

echo $WSL_DISTRO_NAME

Ollama will use the GPU via WSL automatically if the Windows driver supports it.

Step 2 — Install Docker (for Open WebUI)

Ubuntu:

sudo apt update && sudo apt install -y docker.io
sudo usermod -aG docker $USER
newgrp docker
sudo systemctl enable --now docker

Windows WSL2:

Install Docker Desktop for Windows and enable the WSL2 integration for your Ubuntu distro. Start Docker Desktop before running containers.

Step 3 — Install Ollama

Ubuntu/WSL2:

curl -fsSL https://ollama.com/install.sh | sh

Enable the service on Ubuntu (bare metal):

sudo systemctl enable --now ollama

Test by pulling a small model:

ollama pull llama3.1:8b

Run a quick prompt:

ollama run llama3.1:8b "Write a haiku about GPUs."

If you see a note like “using CUDA,” GPU is active. If not, ensure drivers are correct.

Step 4 — Deploy Open WebUI

Ubuntu (recommended: host networking):

docker run -d --name open-webui --restart unless-stopped --network host \
-e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
-e WEBUI_AUTH=true \
-e [email protected] \
-e DEFAULT_USER_PASSWORD=ChangeMeStrong! \
ghcr.io/open-webui/open-webui:main

Open your browser at http://127.0.0.1:8080 (host network uses the container’s internal port). Log in with the credentials you set.

WSL2 with Docker Desktop (no host networking):

docker run -d --name open-webui --restart unless-stopped -p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-e WEBUI_AUTH=true \
-e [email protected] \
-e DEFAULT_USER_PASSWORD=ChangeMeStrong! \
ghcr.io/open-webui/open-webui:main

Browse to http://localhost:3000. The variable OLLAMA_BASE_URL points Open WebUI to the local Ollama API.

Step 5 — Optimize and manage models

Popular models: llama3.1:8b, mistral-nemo:12b, qwen2.5:7b. You can also choose quantized variants like llama3.1:8b-instruct-q4_K_M for lower VRAM usage.

ollama list
ollama pull mistral-nemo:12b
ollama rm modelname:tag

To limit VRAM, use smaller models or quantized builds. In Open WebUI, select the model per chat. For longer contexts, try a 70B model on a bigger GPU or use smaller context windows to conserve memory.

Step 6 — Secure and persist

Keep Open WebUI bound to localhost if the machine is shared. If you must expose it, use a reverse proxy with TLS (Nginx, Caddy, or Traefik) and keep WEBUI_AUTH=true. On Ubuntu, ensure the firewall blocks unwanted access:

sudo ufw allow 8080/tcp comment 'Open WebUI (local)'
sudo ufw status

Data for Open WebUI is stored in the container’s volume by default. For manual control, mount a volume: -v open-webui:/app/backend/data. Ollama stores models under ~/.ollama; back that up regularly.

Step 7 — Auto-start on boot

Ollama installs a systemd service on Ubuntu. Ensure Docker is enabled (already done) and containers use --restart unless-stopped so Open WebUI comes up after reboots. On Windows, set Docker Desktop to start on login.

Troubleshooting

GPU not used: Update NVIDIA driver, reboot, and confirm with nvidia-smi (Ubuntu) or update WSL and drivers (Windows). Reinstall Ollama if needed. Check that ollama run logs mention CUDA.

Out-of-VRAM: Use a smaller or more aggressively quantized model. Reduce max tokens or context length in Open WebUI settings. Close other GPU apps.

Port conflicts: If 11434 or 8080/3000 are used, change them. For example, -p 3333:8080 for Open WebUI and OLLAMA_HOST=127.0.0.1:11500 ollama serve for Ollama.

Slow downloads: Use a reliable network, or pre-fetch models with ollama pull. You can also host a local model library if bandwidth is limited.

Clean up disk: Remove unused models and images:
ollama list && ollama rm model:tag
docker image prune -f

What’s next

Explore RAG in Open WebUI by attaching local documents, add embeddings, and test function calling or tool integrations. With this setup, you have a fast, private, and GPU-accelerated local AI workstation ready for coding, content, and research.

Overview

Running local large language models (LLMs) is now practical with modern GPUs. In this tutorial, you will learn how to deploy Ollama and Open WebUI on Ubuntu using Docker and NVIDIA GPU acceleration. Ollama handles model downloads and inference, while Open WebUI provides a clean, browser-based chat interface. By the end, you will have a secure, upgradable stack for self-hosted AI on your own server.

Prerequisites

You will need Ubuntu 22.04 or 24.04 (server or desktop), an NVIDIA GPU with recent drivers, and sudo access. Ensure your GPU is visible by the OS with nvidia-smi. If you are starting from a clean install, use sudo ubuntu-drivers autoinstall, reboot, and verify nvidia-smi shows your card and driver version.

Step 1: Install Docker Engine

Install Docker from the official repository to get current features and security patches. First, add Docker’s GPG key and repository, then install Docker Engine and the Compose plugin.

sudo apt update && sudo apt install -y ca-certificates curl gnupg lsb-release
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release; echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER && newgrp docker

Step 2: Enable NVIDIA GPU in Containers

Install the NVIDIA Container Toolkit so Docker can access your GPU. This allows GPU passthrough to Ollama in a container.

sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

If the last command prints your GPU details, Docker sees your GPU. If not, verify your NVIDIA driver installation and rerun the steps above.

Step 3: Create a Docker Compose file

Use Docker Compose to run Ollama and Open WebUI together. Create a folder such as ~/ai-stack and add a compose.yml file with the following contents.

version: "3.8"
services:
  ollama:
    image: ollama/ollama:latest
    runtime: nvidia
    environment:
     - NVIDIA_VISIBLE_DEVICES=all
     - OLLAMA_KEEP_ALIVE=48h
    volumes:
     - ollama:/root/.ollama
    ports:
     - "11434:11434"
  openwebui:
    image: ghcr.io/open-webui/open-webui:latest
    depends_on:
     - ollama
    environment:
     - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
     - openwebui:/app/backend/data
    ports:
     - "3000:8080"
volumes:
  ollama:
  openwebui:

If you prefer AMD GPUs, replace ollama/ollama:latest with ollama/ollama:rocm and ensure your host has ROCm drivers configured. You will also need to pass /dev/kfd and /dev/dri devices; consult Ollama’s ROCm documentation for the exact mappings.

Step 4: Launch the Stack and Pull a Model

Start the services and tail the logs to confirm that everything is healthy.

docker compose up -d
docker compose logs -f ollama

Pull a model with good balance between quality and VRAM needs, such as Llama 3.1 8B in a quantized format. You can do this via the terminal or from Open WebUI’s Model Manager.

docker exec -it $(docker compose ps -q ollama) ollama pull llama3.1:8b

Open your browser to http://SERVER_IP:3000. On first launch, create your admin account. In Settings, select the default model and start chatting. The first generation may be slower while the model warms the cache.

Step 5: Security, Updates, and Backups

If you plan to expose the service over the internet, place Open WebUI behind a reverse proxy with TLS (Caddy, Traefik, or Nginx) and restrict access by IP or set up SSO. For private networks, at minimum change the default ports and disable self-registration from the Admin panel to prevent unauthorized accounts.

To update to the latest versions safely, pull new images and recreate the containers. Volumes preserve your models and settings.

docker compose pull && docker compose up -d

Back up your volumes regularly, especially before upgrades. A simple snapshot approach with tar works for low-downtime maintenance.

docker run --rm -v ollama:/data -v $(pwd):/backup busybox sh -c "cd /data && tar czf /backup/ollama-vol-$(date +%F).tgz ."
docker run --rm -v openwebui:/data -v $(pwd):/backup busybox sh -c "cd /data && tar czf /backup/openwebui-vol-$(date +%F).tgz ."

Performance Tips

Choose quantized models that fit your VRAM; for 8–12 GB GPUs, Q4_K_M variants are often the best starting point. Monitor VRAM usage with watch -n1 nvidia-smi. In Open WebUI, tune the context length and batch size to match your GPU memory. Keep the driver, CUDA runtime, and container toolkit reasonably current for stability and speed. If you run multiple simultaneous chats, consider increasing concurrency carefully and watch for swapping or GPU OOM errors.

Troubleshooting

Container cannot see the GPU: Ensure nvidia-smi works on the host, then verify nvidia-container-toolkit is installed and that you ran nvidia-ctk runtime configure followed by systemctl restart docker. Test with docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi.

Model fails to load or out-of-memory: Pick a smaller quantized model or reduce context length. Close other GPU applications. Ensure the Ollama service has GPU access and that the model variant matches your hardware constraints.

Ports already in use: Change the published ports in your Compose file (e.g., map 3001:8080) and rerun docker compose up -d.

With this setup, you can run private, high-performance LLMs on your own hardware with a friendly web interface, straightforward updates, and simple backups. Scale up by adding larger models, enabling GPU persistence mode, or deploying behind a production reverse proxy with TLS and access controls.