LifeBytes Journal

Run Local AI with Ollama and Open WebUI: GPU-Accelerated Setup on Windows and Linux with Docker

Local large language models (LLMs) have matured to the point where you can run fast, private, and cost-effective AI on your own computer or server. In this step-by-step guide, you will deploy Ollama (the LLM backend) and Open WebUI (a sleek web interface) using Docker, with optional GPU acceleration on both Windows and Linux. This stack lets you chat with models like Llama 3, Phi-4, or Mistral, completely on your hardware.

By the end, you will have a browser-based interface, persistent model storage, and a clean way to update or back up your local AI environment. The instructions are written in simple, SEO-friendly language and focus on practical steps.

What You Will Build

You will run two containers on the same Docker network: Ollama exposes an API on port 11434 and performs all model work, while Open WebUI listens on port 3000 and connects to Ollama. You will enable GPU acceleration (NVIDIA or AMD) when available to dramatically improve performance.

Prerequisites

- A 64-bit Windows 11/10 (with WSL2) or a modern Linux distribution (Ubuntu/Debian/CentOS/RHEL).
- Docker installed (Docker Desktop on Windows, Docker Engine on Linux).
- Optional GPU: NVIDIA (CUDA) or AMD (ROCm) with up-to-date drivers. CPU-only also works, but is slower.
- 16 GB RAM recommended; disk space 10–40+ GB depending on model size.

Step 1 – Install Docker

Windows: Install Docker Desktop and enable WSL 2 integration. In Settings, ensure “Use the WSL 2 based engine” is on. Update your GPU driver from NVIDIA/AMD. For NVIDIA, CUDA is not required on Windows for Docker Desktop; the latest Game Ready/Studio drivers are enough.

Linux: Install Docker from your distribution’s repository or Docker’s official repo. Add your user to the docker group and log out/in. Example (Ubuntu):

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER

Step 2 – Enable GPU Acceleration (Optional but Recommended)

NVIDIA on Linux: Install the NVIDIA Container Toolkit to pass your GPU into containers.

# Add the NVIDIA container toolkit repo (Ubuntu example)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

AMD on Linux (ROCm): Install the latest AMDGPU/ROCm stack. To give containers access, pass /dev/kfd and /dev/dri and add the video group. Example device flags are shown in the Ollama run step below.

Windows: Docker Desktop exposes the GPU automatically when the host has a compatible driver. Ensure your GPU driver is up to date and “Use the WSL 2 based engine” is enabled.

Step 3 – Start Ollama (LLM Backend)

Create a Docker network and a persistent volume for models. Then start the Ollama container. Use the NVIDIA command if you have an NVIDIA GPU; use the AMD/CPU command otherwise.

# Common network and volumes
docker network create llmnet
docker volume create ollama

# NVIDIA GPU (Linux or Windows with Docker Desktop)
docker run -d --name ollama \
  --network llmnet \
  --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama:latest

# AMD ROCm or CPU-only (Linux)
# Remove the two --device flags if you want CPU-only
docker run -d --name ollama \
  --network llmnet \
  --device=/dev/kfd --device=/dev/dri --group-add video \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama:latest

Verify Ollama is live:

curl http://localhost:11434/api/tags
# or
docker logs -f ollama

Step 4 – Start Open WebUI (Front-End)

Open WebUI connects to the Ollama API and gives you a beautiful chat interface. Map port 3000 for access and point it to the Ollama container over the private network.

docker volume create open-webui

docker run -d --name open-webui \
  --network llmnet \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Open your browser at http://localhost:3000 and follow the first-run prompts. If you are on a server, replace localhost with the server’s IP or hostname.

Step 5 – Pull and Test a Model

Use Ollama to download a model. Smaller 7–8B models are a good starting point. You can pull directly from the container or from the WebUI Models page.

# Examples (choose one)
docker exec -it ollama ollama pull llama3.1:8b
docker exec -it ollama ollama pull phi3:mini
docker exec -it ollama ollama pull mistral:7b

After the download, open Open WebUI and start a new chat. Pick the model you pulled and send a test prompt. If you see fast tokens and low latency, your GPU is active. If generation is slow, you may be on CPU.

Step 6 – Secure, Persist, and Back Up

Enable authentication in Open WebUI and control who can sign up. You can preconfigure basic auth behavior with environment variables. Example: disable new signups and set an admin email.

# Stop and re-create Open WebUI with tighter auth (example)
docker rm -f open-webui
docker run -d --name open-webui \
  --network llmnet \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -e ENABLE_SIGNUP=false \
  -e [email protected] \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

To back up models and chat history, archive the Docker volumes. This keeps your setup portable.

# Backup Ollama models
docker run --rm -v ollama:/data -v "$PWD":/backup alpine \
  tar czf /backup/ollama-volume-backup.tgz -C /data .

# Backup Open WebUI data
docker run --rm -v open-webui:/data -v "$PWD":/backup alpine \
  tar czf /backup/open-webui-volume-backup.tgz -C /data .

To update, pull the latest images and recreate:

docker pull ollama/ollama:latest
docker pull ghcr.io/open-webui/open-webui:latest
docker rm -f open-webui ollama
# Re-run the "docker run" commands from Steps 3 and 4

Performance Tips

- Prefer smaller, quantized models (e.g., 7–8B) if you have limited VRAM. Many Ollama models include quantized tags that fit 8–12 GB GPUs.
- Close other GPU-heavy apps to free VRAM.
- Keep GPU drivers and Docker updated for the best kernel-accelerated performance.

Troubleshooting

Open WebUI cannot reach Ollama: Make sure both containers share the same network and the URL is correct: http://ollama:11434. Run docker logs open-webui for connection errors.

“no gpus found” or slow generation: On Linux with NVIDIA, confirm nvidia-smi works on the host and that nvidia-container-toolkit is installed. Run the container with --gpus=all. On AMD, pass --device=/dev/kfd --device=/dev/dri --group-add video. On Windows, ensure Docker Desktop is using WSL2 and that your GPU driver is current.

Port already in use: Adjust published ports, e.g., use -p 3001:8080 or -p 11435:11434, and update the URLs accordingly.

Out of memory (VRAM): Pick a smaller or more heavily quantized model. Close other GPU apps and try again.

What’s Next

With Ollama and Open WebUI running, you can add multiple models, enable embeddings and RAG, or connect tools and function calling. This setup gives you a private, fast local AI workspace that you can back up and upgrade in minutes—all without sending your data to the cloud.

Local large language models (LLMs) are now practical for developers, researchers, and privacy-focused teams. In this step-by-step guide, you will install and run Ollama (LLM runtime) and Open WebUI (a modern chat interface) on Ubuntu 22.04/24.04, with optional NVIDIA GPU acceleration. The setup uses Docker for easy updates, isolation, and backups.

Why this stack?

Ollama makes downloading and running models simple, offering a fast API on your machine. Open WebUI provides a sleek, extensible web app for chatting with multiple models, managing prompts, and moderating access. Together, they create a private, cost-effective alternative to cloud AI services.

Prerequisites

You need an Ubuntu 22.04/24.04 system with internet access and a user with sudo rights. If you have an NVIDIA GPU (recommended), you can enable GPU acceleration for much faster inference. CPU-only works too—skip the GPU steps if you do not have a supported GPU.

Step 1: Update the system

Update packages to ensure you have the latest dependencies and security fixes.

sudo apt update && sudo apt -y upgrade
sudo reboot

Step 2 (Optional): Enable NVIDIA GPU support

Install the latest proprietary NVIDIA driver and the container toolkit so Docker can use your GPU. Reboot when asked.

# Install recommended NVIDIA driver
sudo ubuntu-drivers install
sudo reboot

# Install NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list > /dev/null

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Verify GPU visibility
nvidia-smi

Step 3: Install Docker Engine and Docker Compose plugin

Install Docker from the official repository to get the latest stable version. Add your user to the docker group to run Docker without sudo.

sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker $USER
newgrp docker

# Quick test
docker run --rm hello-world

Step 4: Create a Docker Compose file for Ollama + Open WebUI

Create a working directory and define services. The configuration below enables GPU when present; for CPU-only, remove the gpus: all line under the Ollama service.

mkdir -p ~/local-llm && cd ~/local-llm
nano docker-compose.yml

services:
  ollama:
    container_name: ollama
    image: ollama/ollama:latest
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    # Comment the next line if you are CPU-only
    gpus: all
    environment:
      - OLLAMA_KEEP_ALIVE=24h

  open-webui:
    container_name: open-webui
    image: ghcr.io/open-webui/open-webui:latest
    depends_on:
      - ollama
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=change_this_long_random_secret
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:

Step 5: Start the stack

Bring the services up in the background, then check their status. The Open WebUI will be available at http://SERVER_IP:3000 and the Ollama API at http://SERVER_IP:11434.

docker compose up -d
docker compose ps

Step 6: Pull and run a model

Use Ollama to pull an LLM. You can pick models like llama3.2, mistral, or a coding model. The first pull downloads model weights, which can be several GB.

# Pull a general-purpose model
docker exec -it ollama ollama pull llama3.2

# Test it via CLI
docker exec -it ollama ollama run llama3.2 "Write a two-sentence summary of Ubuntu."

# Or use the API
curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Hello!"}'

Open your browser to http://SERVER_IP:3000, select the model from the dropdown, and start chatting. In Settings, you can change default models, system prompts, and appearance.

Step 7: Secure access

If exposing the WebUI beyond your LAN, add authentication. In Open WebUI, create an admin user at first login, then disable new signups in Settings. For internet exposure, place a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS (Let’s Encrypt) in front of port 3000.

Step 8: Update and backup

To update, pull the latest images and recreate containers without losing data stored in volumes. To back up, save the volumes before upgrades.

# Update images
docker compose pull
docker compose up -d

# Backup volumes (example)
docker run --rm -v local-llm_ollama:/data -v "$PWD":/backup \
  busybox tar czf /backup/ollama-vol.tgz /data

docker run --rm -v local-llm_openwebui:/data -v "$PWD":/backup \
  busybox tar czf /backup/openwebui-vol.tgz /data

Troubleshooting

GPU not used: Run docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi. If it fails, recheck the driver and NVIDIA Container Toolkit steps. Ensure the gpus: all line is present and Docker was restarted.

Permission denied with Docker: You may need to log out and back in after adding your user to the docker group, or run newgrp docker.

Port conflicts: Change the left side of port mappings in docker-compose.yml (e.g., use "8081:8080" for WebUI) and restart.

Slow or failed model pull: Verify disk space and retry. Large models require several GB of free space in the ollama volume.

Uninstall (optional)

To stop and remove everything, run:

cd ~/local-llm
docker compose down
docker volume rm local-llm_ollama local-llm_openwebui

Wrap-up

You have a modern local AI stack: Ollama for fast model serving and Open WebUI for a friendly, multi-model chat interface. With Docker, updates are quick and backups are simple. Add your favorite models, tune system prompts, and integrate the Ollama API into your apps—all without sending data to the cloud.

Overview

This step-by-step guide shows how to run open-source large language models (LLMs) locally on Ubuntu 24.04 using Ollama for model serving and Open WebUI for a friendly chat interface. You will install Ollama, enable optional GPU acceleration (NVIDIA or CPU fallback), and deploy Open WebUI with Docker. The result is a private, fast, and controllable AI setup suitable for home labs and small teams.

Prerequisites

You need an Ubuntu 24.04 LTS host with internet access, a user with sudo rights, and at least 8 GB of RAM. A modern NVIDIA GPU is optional but recommended for faster inference. Make sure the system is up to date: sudo apt update && sudo apt -y upgrade

Step 1 — Install Ollama

Ollama is a lightweight server that downloads and runs models locally. Install it with the official script:
curl -fsSL https://ollama.com/install.sh | sh

Enable and start the service so it runs at boot:
sudo systemctl enable ollama
sudo systemctl start ollama
sudo systemctl status ollama

Verify the API is listening on port 11434:
curl http://127.0.0.1:11434/api/tags

Step 2 — Optional: Enable GPU Acceleration (NVIDIA)

If you have an NVIDIA GPU, install the recommended driver. Ubuntu makes this easy:
sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, confirm the driver is active:
nvidia-smi

Ollama detects GPUs automatically when drivers are present. No extra flags are required. If you need to force CPU or GPU behavior, you can set:
export OLLAMA_NO_GPU=1 (CPU only) or export OLLAMA_NO_GPU=0 (GPU allowed). For a persistent setting, add the variable to your shell profile and restart Ollama:
sudo systemctl restart ollama

AMD GPUs can work with ROCm on supported cards and drivers. If you are using AMD, install the ROCm runtime from AMD’s repository for Ubuntu 24.04, confirm with rocminfo, and ensure your user is in the video and render groups. If ROCm is not available for your hardware, Ollama will fall back to CPU.

Step 3 — Pull a Model and Test Locally

Pull a well-supported model. Llama 3 is a popular choice:
ollama pull llama3

Run a quick test:
ollama run llama3 "Write one sentence about Ubuntu 24.04."

Tip: For smaller footprints, choose tiny models like llama3:8b or phi3. VRAM needs vary; an 8B model typically benefits from 8–12 GB of GPU VRAM, while CPU-only runs need more system RAM and patience.

Step 4 — Install Docker and Open WebUI

Open WebUI gives you a clean browser interface for Ollama. Install Docker from Ubuntu repos for a quick start:
sudo apt install -y docker.io docker-compose-plugin

Allow your user to manage Docker without sudo, then re-login:
sudo usermod -aG docker $USER

Create a persistent volume for Open WebUI data and start the container. It will connect to Ollama on the host:
docker volume create openwebui
docker run -d --name open-webui -p 3000:8080 --restart unless-stopped -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v openwebui:/app/backend/data ghcr.io/open-webui/open-webui:latest

Open your browser to http://SERVER_IP:3000 and complete the initial admin setup. Add a model in Settings if it does not appear automatically, for example llama3.

Step 5 — Optional TLS with Caddy (Automatic HTTPS)

If you have a domain pointing to your server (A record), Caddy can auto-provision HTTPS certificates. Install it and configure a simple reverse proxy:
sudo apt install -y caddy

Edit /etc/caddy/Caddyfile (replace ai.example.com with your domain):
ai.example.com {
  reverse_proxy 127.0.0.1:3000
}

Reload Caddy:
sudo systemctl reload caddy. Visit https://ai.example.com. Ensure ports 80 and 443 are open on your firewall and router.

Step 6 — Backups and Updates

Ollama models are stored under ~/.ollama for non-root users or /usr/share/ollama when installed system-wide. Back up this directory to avoid re-downloading models. Example:
tar czf ollama-backup.tgz ~/.ollama

Open WebUI data is in the Docker volume openwebui. Back it up with:
docker run --rm -v openwebui:/data -v $(pwd):/backup alpine sh -c "cd /data && tar czf /backup/openwebui-backup.tgz ."

To update Ollama:
curl -fsSL https://ollama.com/install.sh | sh && sudo systemctl restart ollama. To update Open WebUI:
docker pull ghcr.io/open-webui/open-webui:latest && docker stop open-webui && docker rm open-webui && docker run ... (re-run the previous docker run command).

Troubleshooting

If port 11434 or 3000 is in use, change the port in the docker run command or stop the conflicting service. For slow responses, try a smaller model or ensure your GPU driver is working. If Open WebUI cannot reach Ollama, verify curl http://127.0.0.1:11434/api/tags succeeds on the host and confirm the OLLAMA_BASE_URL is correct.

Wrap-up

You now have a private AI stack on Ubuntu 24.04 with Ollama handling model inference and Open WebUI offering a clean chat interface. With optional GPU acceleration, HTTPS, and simple backups, this setup is fast, secure, and maintainable—perfect for learning, prototyping, or running an internal assistant.

Overview

This tutorial shows how to self-host a fast, private AI chatbot using Ollama and Open WebUI on Ubuntu 24.04 with an NVIDIA GPU. Ollama runs large language models locally and uses your GPU for acceleration; Open WebUI provides a sleek, browser-based interface. By the end, you will have a secure, production-ready setup using Docker for the UI and a systemd service for Ollama.

Prerequisites

- Ubuntu 24.04 LTS (server or desktop) with sudo access.

- An NVIDIA GPU (Turing or newer recommended) and a reliable internet connection.

- Optional: A domain name if you plan to expose the UI over HTTPS.

Step 1: Install NVIDIA Driver and Verify CUDA

Install the recommended NVIDIA driver:

sudo apt update && sudo ubuntu-drivers install

Reboot and verify:

sudo reboot

nvidia-smi

You should see your GPU details and driver version. If Secure Boot is enabled and blocks the driver, either enroll a MOK when prompted or temporarily disable Secure Boot in firmware settings.

Step 2: Install Ollama and Enable the Service

Ollama provides a one-line installer that sets up the binary and a systemd service listening on localhost:11434.

curl -fsSL https://ollama.com/install.sh | sh

Check the service:

systemctl status ollama

If needed, start and enable it:

sudo systemctl enable --now ollama

Step 3: Pull a Model and Test GPU Acceleration

Pull a modern, efficient model like Llama 3 (choose the size that fits your VRAM; 8B works on many GPUs):

ollama pull llama3

Run a quick test and watch GPU utilization in another terminal with nvidia-smi:

ollama run llama3 "Write a two-line poem about Ubuntu and GPUs."

If the model runs and nvidia-smi shows activity, GPU acceleration is working. If not, confirm the driver is loaded and retry.

Step 4: Install Docker and Compose Plugin

Open WebUI will run in Docker to simplify updates and isolation.

sudo apt update && sudo apt install -y docker.io docker-compose-plugin

Add your user to the Docker group to avoid using sudo:

sudo usermod -aG docker $USER && newgrp docker

Step 5: Deploy Open WebUI with Docker Compose

Create a project folder and a compose file that points Open WebUI to the host’s Ollama endpoint.

mkdir -p ~/open-webui && cd ~/open-webui

cat > compose.yaml << 'YAML'
services:
open-webui:
image: ghcr.io/open-webui/open-webui:main
ports:
- "3000:8080"
environment:
- OLLAMA_BASE_URL=http://host.docker.internal:11434
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- open-webui:/app/backend/data
restart: unless-stopped
volumes:
open-webui: {}
YAML

Start the container:

docker compose up -d

Open the firewall (if UFW is enabled):

sudo ufw allow 3000/tcp

Step 6: First-Time Access and Configuration

Visit http://<server-ip>:3000, create your admin account, and confirm the Ollama connection URL is set to http://host.docker.internal:11434. You can now select or download models from the UI or use the ones you already pulled via the CLI. Start chatting to verify responses are fast and local.

Optional: Security and HTTPS

If you plan to expose Open WebUI to the internet, do not publish port 3000 directly. Instead, set the container to bind locally and put a reverse proxy in front with TLS.

- Bind UI to localhost only: edit the ports line to "127.0.0.1:3000:8080".

- Use a reverse proxy like Caddy or Nginx, obtain a TLS certificate (e.g., Let’s Encrypt), and enable basic auth or OAuth. Keep your system updated and restrict access with a firewall or VPN.

Troubleshooting

No GPU usage: Run nvidia-smi. If it shows “No devices were found,” reinstall the driver with ubuntu-drivers install, ensure Secure Boot is handled, and reboot. Confirm you are not in a VM without GPU passthrough.

Ollama not responding: Check systemctl status ollama and logs with journalctl -u ollama -e. Ensure it listens on localhost:11434 and no other service conflicts.

Open WebUI cannot reach Ollama: Confirm the extra_hosts entry for host-gateway is present. Try setting OLLAMA_BASE_URL=http://<host-ip>:11434 instead of host.docker.internal. Restart with docker compose up -d.

Docker permission denied: Re-run sudo usermod -aG docker $USER, then newgrp docker or log out/in.

Out-of-memory on big models: Choose a smaller model (e.g., llama3:8b), reduce context, or set OLLAMA_NUM_GPU=1 to limit sharding. Monitor with nvidia-smi.

Updates, Backups, and Removal

Update Ollama: Re-run the installer to fetch the latest version: curl -fsSL https://ollama.com/install.sh | sh. Update models with ollama pull <model:tag>.

Update Open WebUI: cd ~/open-webui && docker compose pull && docker compose up -d.

Back up UI data: The named volume open-webui stores settings and chats. Snapshot it with docker run --rm -v open-webui:/data -v $(pwd):/backup busybox tar czf /backup/open-webui-backup.tgz -C / data.

Uninstall (optional): Stop UI with docker compose down. Remove volume with docker volume rm open-webui. Disable Ollama with sudo systemctl disable --now ollama. Remove the Ollama binary and files only if you no longer need them.

Conclusion

You now have a private, GPU-accelerated AI chatbot running locally with Ollama and Open WebUI on Ubuntu 24.04. This setup is fast, secure, and easy to maintain. You can switch models on demand, keep everything offline, and scale performance by upgrading your GPU or choosing optimized models.

Overview

This guide shows you how to install Ollama and Open WebUI on Ubuntu 24.04 with NVIDIA GPU acceleration. You will get a fast local AI stack that can run modern large language models (LLMs) like Llama 3 on your own hardware, secured and accessible in a web browser. We will cover GPU driver setup, Docker with the NVIDIA Container Toolkit, Ollama installation, Open WebUI deployment, basic security, and troubleshooting.

Prerequisites

- A machine running Ubuntu 24.04 (fresh or updated).
- An NVIDIA GPU with at least 8 GB VRAM recommended for 8B models (more is better).
- Internet access and sudo privileges.

Step 1: Install the NVIDIA GPU Driver

Install the recommended proprietary driver so CUDA becomes available to both Ollama and containers.

sudo apt update && sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, verify the GPU is recognized:

nvidia-smi

You should see a driver version and your GPU model. If not, check “Troubleshooting.”

Step 2: Install Docker and the NVIDIA Container Toolkit

We will run Open WebUI in Docker and connect it to the host’s Ollama API. First, install Docker Engine from the official repository:

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Now install the NVIDIA Container Toolkit so containers can access your GPU:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Quick GPU-in-container test (optional):

docker run --rm --gpus all nvidia/cuda:12.6.2-base-ubuntu24.04 nvidia-smi

Step 3: Install Ollama (GPU-Accelerated LLM Runtime)

Ollama downloads, runs, and serves models locally. The installer sets up a system service by default.

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama

Verify the API is listening on port 11434:

curl http://127.0.0.1:11434/api/tags

Check GPU visibility and the model directory:

ollama info

Note: On Ubuntu, the Ollama service runs as the “ollama” user. Models are typically stored under /usr/share/ollama (service) or ~/.ollama (when run as your user). The exact location appears in ollama info.

Step 4: Deploy Open WebUI (Docker)

Open WebUI provides a friendly browser UI that connects to Ollama’s API. We’ll run it in host networking mode so it can reach 127.0.0.1:11434 directly.

docker run -d --name open-webui --restart unless-stopped --network=host -e OLLAMA_BASE_URL=http://127.0.0.1:11434 ghcr.io/open-webui/open-webui:latest

Open your browser and visit http://<server-ip>:8080. On first launch, create the admin account. In Settings, confirm the Ollama endpoint is http://127.0.0.1:11434.

Step 5: Pull a Model and Test

Download a model with good quality/speed balance. The 8B variants work well on many consumer GPUs.

ollama pull llama3.1:8b

Quick API test:

curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.1:8b","prompt":"Say hello in one short sentence."}'

Or try the CLI:

ollama run llama3.1:8b "Explain the difference between VRAM and system RAM in one paragraph."

You should see GPU utilization in nvidia-smi while the model is generating.

Step 6: Basic Security and Management

If your server is exposed to a network, restrict Open WebUI to trusted IPs with UFW (replace the subnet with your LAN):

sudo apt install -y ufw
sudo ufw allow OpenSSH
sudo ufw allow from 192.168.0.0/16 to any port 8080 proto tcp
sudo ufw enable

Keep your stack up to date:

sudo systemctl stop ollama && curl -fsSL https://ollama.com/install.sh | sh && sudo systemctl start ollama
docker pull ghcr.io/open-webui/open-webui:latest && docker restart open-webui

To persist Open WebUI data, mount a volume (recommended for production):

docker run -d --name open-webui --restart unless-stopped --network=host -v /opt/open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 ghcr.io/open-webui/open-webui:latest

Troubleshooting

nvidia-smi: command not found or no devices were found
- Reinstall drivers: sudo ubuntu-drivers autoinstall and reboot.
- If Secure Boot is enabled, you may need to enroll the NVIDIA kernel module (MOK) or temporarily disable Secure Boot in firmware.
- Use a supported driver (typically 535+ on newer GPUs).

Docker can’t see the GPU
- Test: docker run --rm --gpus all nvidia/cuda:12.6.2-base-ubuntu24.04 nvidia-smi.
- Reconfigure toolkit: sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker.
- Ensure you’re using the NVIDIA proprietary driver, not Nouveau.

Open WebUI cannot connect to Ollama
- Confirm Ollama API is running: curl http://127.0.0.1:11434/api/tags.
- Check logs: docker logs -f open-webui.
- Ensure --network=host is used and OLLAMA_BASE_URL is set to http://127.0.0.1:11434.

Models consume too much VRAM
- Try a smaller quantization (e.g., llama3.1:8b default Q4_K_M) or use a 7B/8B model instead of 13B/70B.
- Limit parallel requests in Open WebUI settings and Ollama.

What You Built

You now have a local, GPU-accelerated AI environment with Ollama serving LLMs and Open WebUI providing a clean chat interface. This setup is fast, private, and easy to maintain, giving you full control over your models and data. You can add more models with ollama pull, create prompt presets in Open WebUI, and secure access with firewalls or a reverse proxy if you plan to expose it beyond your LAN.

Local large language models (LLMs) have matured rapidly, and running them with GPU acceleration on your own server is now simple. In this step-by-step tutorial, you will deploy Ollama (model runtime) and Open WebUI (a friendly chat interface) on Ubuntu 22.04/24.04 using Docker Compose and the NVIDIA Container Toolkit.

Prerequisites

- An Ubuntu 22.04 or 24.04 machine with an NVIDIA GPU (Turing or newer recommended) and internet access.

- Administrative (sudo) access.

- Basic familiarity with the terminal and Docker.

1) Install NVIDIA Driver

First, make sure your system is up to date, then install the recommended NVIDIA driver. If you are already on the correct proprietary driver, you can skip this step.

sudo apt update && sudo apt upgrade -y
sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, confirm the GPU is available:

nvidia-smi

You should see a table with your GPU model and driver version.

2) Install Docker Engine

Use the convenience script from Docker to install the latest Docker Engine quickly. Alternatively, follow the official repository instructions if you prefer a locked version.

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker

Verify Docker is working:

docker run --rm hello-world

3) Install NVIDIA Container Toolkit

The NVIDIA Container Toolkit lets containers access your GPU. Install it and configure Docker to use the NVIDIA runtime.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test GPU access inside a container:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If you see your GPU, you are ready for Ollama.

4) Create a Docker Compose file

Make a project directory and create a docker-compose.yml that runs both services with persistent volumes and GPU support.

mkdir -p ~/ollama-webui && cd ~/ollama-webui
nano docker-compose.yml

version: "3.9"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    environment:
      - OLLAMA_KEEP_ALIVE=24h
    volumes:
      - ollama:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    environment:
      - OLLAMA_API_BASE_URL=http://ollama:11434
      - WEBUI_AUTH=true
    ports:
      - "3000:8080"
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:

Note: The deploy.resources.devices block requests GPU access using the NVIDIA driver. Docker Compose v2+ supports this on modern hosts. If your Compose version ignores it, see the troubleshooting section for alternatives.

5) Start the stack

Bring up both containers in the background:

docker compose up -d
docker compose ps

Open your browser to http://localhost:3000 (or the server’s IP on port 3000). The web interface will prompt for account setup if authentication is enabled.

6) Pull and run a model

You can pull models from the UI (Models section), or via CLI inside the Ollama container. Example with Llama 3.1 (8B):

docker exec -it ollama ollama pull llama3.1:8b

Once the model is downloaded, select it in Open WebUI and start chatting. GPU memory matters: 8B models typically need ~6–8 GB VRAM; 70B needs much more. If you are low on VRAM, try smaller or quantized variants (e.g., Q4_K_M builds).

7) Persistence, updates, and backups

- Your models and settings live in Docker volumes named “ollama” and “openwebui.” To back them up, stop the stack and archive /var/lib/docker/volumes/ollama and /var/lib/docker/volumes/openwebui.

- To update images: docker compose pull && docker compose up -d.

- To move the setup to another host, copy the Compose file and restore the volumes.

8) Secure access (optional)

For internet exposure, place Open WebUI behind a reverse proxy with TLS (e.g., Caddy or Nginx) and keep WEBUI_AUTH=true. Consider network ACLs or a VPN like Tailscale/WireGuard for private, zero-trust access.

Troubleshooting

- GPU not detected in containers: ensure nvidia-smi works on the host; re-run sudo nvidia-ctk runtime configure --runtime=docker; restart Docker; try docker compose down && up -d.

- If your Compose version ignores deploy.devices, try adding a profile or running with CLI flags. For example, launch Ollama separately:

docker run -d --name ollama --gpus all -p 11434:11434 \
  -v ollama:/root/.ollama --restart unless-stopped ollama/ollama:latest

- Performance tips: set model context length lower in WebUI, avoid running multiple models at once, and monitor VRAM usage with nvidia-smi.

With this setup, you get a modern, GPU-accelerated local LLM stack that is fast, private, and easy to maintain using Docker Compose. Enjoy building AI workflows without sending your data to the cloud.