LifeBytes Journal

Overview

This tutorial shows you how to deploy a fast, private, local AI stack on Ubuntu using Ollama and Open WebUI with NVIDIA GPU acceleration. You will install the NVIDIA driver, Docker, and the NVIDIA Container Toolkit, then run Ollama on the host and Open WebUI in a container. By the end, you will have a browser-based interface to run powerful large language models (LLMs) like Llama 3 with CUDA acceleration on your own machine.

Prerequisites

- Ubuntu 22.04 or 24.04 (freshly updated).
- An NVIDIA GPU with at least 6 GB VRAM (more is better).
- sudo privileges and Internet access.
- Optional: a domain or reverse proxy if you plan to expose the UI externally.

Step 1 — Install NVIDIA Driver

Use Ubuntu’s built-in tools to install a compatible proprietary driver. Reboot afterward and confirm the GPU is detected.

sudo apt update
sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

nvidia-smi

If you see a table with your GPU and driver version (e.g., 535+), you are ready for CUDA-enabled workloads.

Step 2 — Install Docker Engine

If Docker is not installed, use the official convenience script. Add your user to the docker group so you can run containers without sudo.

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
docker version

Step 3 — Enable GPU Access in Containers

Install the NVIDIA Container Toolkit so Docker can pass the GPU into containers.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU visibility inside a container:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Step 4 — Install Ollama (runs on the host)

Ollama simplifies downloading and running LLMs locally. It automatically uses CUDA if your NVIDIA driver is installed.

curl -fsSL https://ollama.com/install.sh | sh

Confirm the service is active and the API is reachable on port 11434:

systemctl --user status ollama || systemctl status ollama
curl http://127.0.0.1:11434/api/tags

Pull and test a model (replace with your preferred model/quantization):

ollama pull llama3
ollama run llama3 "Write a two-line poem about GPUs."

Tip: Use smaller quantizations if VRAM is limited, for example llama3:8b-instruct-q4_0.

Step 5 — Deploy Open WebUI in Docker

Open WebUI provides a clean, modern interface for chatting with models served by Ollama. We will run it in Docker and point it to the host’s Ollama API. On Linux, add a host-gateway entry so the container can reach the host at host.docker.internal.

docker run -d --name open-webui \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  --gpus all \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Open your browser at http://<server-ip>:3000. On first login, create a user; that account becomes admin. If you need authentication enabled from the start, add -e WEBUI_AUTH=True to the run command.

Alternative: If --add-host=host-gateway is not supported on your Docker version, use host networking and point to 127.0.0.1:

docker run -d --name open-webui \
  --network host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  --gpus all \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

With host networking, Open WebUI listens on http://0.0.0.0:8080 (no -p flag needed).

Step 6 — Use and Tune Your Local AI

From Open WebUI, select a model (e.g., Llama 3) and start chatting. You can pull additional models with Ollama CLI and they will appear in the UI. To speed up responses and reduce VRAM, try smaller or more aggressive quantizations; to maximize quality, try larger quantizations if your GPU can handle them.

Common environment variables for Open WebUI include:
- WEBUI_AUTH=True to require sign-in.
- OLLAMA_BASE_URL to point to the Ollama server URL.
- PORT to customize the UI port if you use host networking.

Troubleshooting

Open WebUI cannot reach Ollama: Ensure you used --add-host=host.docker.internal:host-gateway and OLLAMA_BASE_URL=http://host.docker.internal:11434, or use host networking. Test connectivity with docker exec -it open-webui curl -s http://host.docker.internal:11434/api/tags.

No GPU in containers: Re-check the container toolkit setup and driver. Run docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi. If it fails, reboot and ensure nvidia-smi works on the host first.

Out-of-memory errors: Use a smaller model or more compressed quantization. Close other GPU-heavy apps. You can also run with a larger system swap to reduce crashes when VRAM is exhausted, but performance will be slower.

Docker permissions: If you see “permission denied,” ensure your user is in the docker group (id to verify) and run newgrp docker or re-log in.

Optional: Reverse Proxy and TLS

If exposing Open WebUI on the Internet, put it behind a reverse proxy (Caddy, Nginx, or Traefik) for HTTPS and access control. At minimum, enforce authentication and limit access to trusted IPs. Never expose Ollama’s port 11434 directly without protection.

Maintenance

- Update Ollama periodically by re-running the install script or checking the project release notes, then systemctl restart ollama.
- Update Open WebUI with docker pull ghcr.io/open-webui/open-webui:main and docker restart open-webui.
- Prune old images and volumes with docker system prune (review carefully before confirming).
- Back up /var/lib/ollama (models) and the Open WebUI volume for settings and chats.

You now have a modern, GPU-accelerated, private AI chat environment running locally on Ubuntu. This setup is fast, secure, and fully under your control—and you can expand it with additional models, prompt libraries, and integrations as your needs grow.

Overview

This guide shows how to deploy Ollama and Open WebUI on Ubuntu so you can run large language models (LLMs) locally with NVIDIA GPU acceleration. You will install Docker and the NVIDIA Container Toolkit, run the Ollama API, connect Open WebUI as a front end, and pull a model like Llama 3. This setup is fast, private, and easy to maintain.

Prerequisites

Before you start, make sure you have: Ubuntu 22.04 or later, an NVIDIA GPU with a recent driver (525+), sudo access, internet connectivity, and open ports 11434 (Ollama) and 3000 (Open WebUI). If you have an existing Docker installation, ensure it is up to date.

1) Install NVIDIA driver and verify GPU

Install a stable NVIDIA driver from Ubuntu’s repository, reboot, and verify the GPU is visible:

sudo apt update
sudo apt install -y nvidia-driver-535
sudo reboot
# After reboot:
nvidia-smi

If nvidia-smi prints your GPU details, the driver is working. If not, check Secure Boot, which can block kernel modules; disable it or sign the modules accordingly.

2) Install Docker and enable GPU in containers

Install Docker using the official convenience script, add your user to the docker group, then install the NVIDIA Container Toolkit so containers can access the GPU.

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker

# NVIDIA Container Toolkit
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Verify GPU works inside Docker:
docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

If the last command shows GPU output, you are ready to run GPU-enabled containers.

3) Deploy Ollama (LLM runtime)

Ollama serves models locally via an HTTP API. Create a volume for persistent model storage and run the container with GPU support:

docker volume create ollama
docker run -d --name ollama --gpus all \
  -p 11434:11434 \
  -e OLLAMA_HOST=0.0.0.0:11434 \
  -v ollama:/root/.ollama \
  ollama/ollama:latest

Pull a model to test. Quantized models use less VRAM; llama3.1:8b is a good starting point on 8–12 GB GPUs.

docker exec -it ollama ollama pull llama3.1:8b
# Quick test (CLI in the container):
docker exec -it ollama ollama run llama3.1:8b

If the model loads and you can send a prompt, Ollama is ready.

4) Deploy Open WebUI (front end)

Open WebUI provides a user-friendly chat interface and features like prompt sets and file uploads. Create an isolated network, connect Ollama, and run Open WebUI:

docker network create ai
docker network connect ai ollama

docker volume create openwebui
docker run -d --name open-webui --network ai \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -v openwebui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Open a browser and go to http://<server-ip>:3000. The first user to sign up becomes the administrator. After creating the admin account, open Settings and disable public signups if you want to restrict access.

5) Use your local AI

In Open WebUI, pick the model you pulled (e.g., llama3.1:8b) and start chatting. You can pull more models from the “Models” area or via:

docker exec -it ollama ollama pull mistral:7b
docker exec -it ollama ollama pull neural-chat:7b

Tip: If a model fails to load due to VRAM limits, choose a smaller or more aggressively quantized variant (e.g., Q4 or 4-bit builds).

6) Update and maintenance

To update to the latest images while keeping your data, pull and recreate the containers with the same volumes:

# Update Ollama
docker pull ollama/ollama:latest
docker stop ollama && docker rm ollama
docker run -d --name ollama --gpus all \
  -p 11434:11434 -e OLLAMA_HOST=0.0.0.0:11434 \
  -v ollama:/root/.ollama --network ai \
  ollama/ollama:latest

# Update Open WebUI
docker pull ghcr.io/open-webui/open-webui:latest
docker stop open-webui && docker rm open-webui
docker run -d --name open-webui --network ai \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -v openwebui:/app/backend/data \
  ghcr.io/open-webui/open-webui:latest

Models and settings persist in the Docker volumes. Back up these volumes regularly with your usual server backup process.

7) Troubleshooting

If Open WebUI cannot talk to Ollama, confirm both containers share the same network and that OLLAMA_BASE_URL points to http://ollama:11434. Use docker logs open-webui to check errors.

If the GPU is not used, verify nvidia-smi inside a container works and the Docker daemon has the NVIDIA runtime configured. Also confirm you started Ollama with --gpus all. For small VRAM, prefer smaller models (e.g., 7–8B) and quantized builds.

If you see slow generation, check CPU/GPU utilization with top and nvidia-smi. Running models from SSD storage and avoiding swap helps latency. Restart long-running containers after driver updates.

What you get

With Ollama and Open WebUI on Ubuntu, you have a private, GPU-accelerated local AI stack. You can chat, summarize, and prototype apps against the Ollama API at http://<server-ip>:11434, while Open WebUI provides a polished interface for everyday use.

Running large language models locally is now practical and secure for many teams. In this guide, you will deploy a production-ready stack on Ubuntu using Ollama (for model runtime) and Open WebUI (for a clean, chat-style interface). The tutorial covers both CPU-only and NVIDIA GPU acceleration with the NVIDIA Container Toolkit, plus tips for updates, security, and backups.

Prerequisites

You need an Ubuntu 22.04 or 24.04 machine, at least 16 GB RAM for smooth performance, and optional NVIDIA GPU (Turing or newer recommended). You also need root or sudo access and a public DNS name if you plan to expose the UI securely.

Step 1: Update Ubuntu

Start by updating your system packages to ensure compatibility with recent Docker and NVIDIA components.

sudo apt update && sudo apt -y upgrade
sudo reboot

Step 2: Install Docker Engine and Compose Plugin

Install Docker from the official repository and enable the Compose plugin. This method ensures you receive timely security fixes and new features.

sudo apt -y install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Step 3 (Optional but Recommended): NVIDIA GPU Acceleration

If you have an NVIDIA GPU, install the NVIDIA Container Toolkit to enable GPU pass-through for containers. Make sure you already have the proprietary NVIDIA driver installed (check with nvidia-smi).

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt -y install nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify that Docker can see your GPU:

docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

Step 4: Create a Docker Compose File

We will run two services: ollama (the LLM runtime and model manager) and open-webui (a modern web UI that connects to Ollama). Save the file as docker-compose.yml in an empty directory.

version: "3.8"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    # Uncomment the next line if you have an NVIDIA GPU:
    # gpus: all

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_API_BASE_URL=http://ollama:11434
    depends_on:
      - ollama
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:

If you are on CPU-only, keep the file as is. If you have a GPU, uncomment the gpus: all line under the ollama service.

Step 5: Launch the Stack

Start both containers in detached mode:

docker compose up -d

Open WebUI should now be available at http://<your-server-ip>:3000. The first visitor will be asked to create an admin account. Leave the browser open; we will add a model next.

Step 6: Pull and Test a Model

Pull a model using Ollama. You can choose from many OSS models; Llama 3.1 8B is a balanced starter option:

docker exec -it ollama ollama pull llama3.1:8b

Confirm that the model is available:

curl http://localhost:11434/api/tags | jq

Back in Open WebUI, select this model in the top bar and start chatting. If GPU is enabled, generation should be significantly faster.

Optional: Secure Public Access with Caddy

If you want to access Open WebUI over HTTPS on a domain (for example, ai.example.com), a simple approach is to put Caddy in front. Caddy obtains and renews certificates automatically via Let’s Encrypt.

sudo apt -y install debian-keyring debian-archive-keyring apt-transport-https
curl -fsSL https://dl.cloudsmith.io/public/caddy/stable/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/caddy-stable-archive-keyring.gpg] \
https://dl.cloudsmith.io/public/caddy/stable/deb/ubuntu all main" | \
sudo tee /etc/apt/sources.list.d/caddy-stable.list

sudo apt update && sudo apt -y install caddy

Create a simple Caddyfile that proxies traffic to Open WebUI on port 3000:

sudo bash -c 'cat >/etc/caddy/Caddyfile' << "EOF"
ai.example.com {
  reverse_proxy 127.0.0.1:3000
}
EOF
sudo systemctl reload caddy

Replace ai.example.com with your real domain and make sure DNS A/AAAA records point to your server’s public IP.

Operations: Updates, Backups, and Cleanup

Update containers regularly for new features and security patches:

docker compose pull
docker compose up -d

Backup volumes to keep models and chat history safe. Stop containers briefly, archive volumes, then restart:

docker compose down
docker run --rm -v ollama:/data -v $PWD:/backup alpine tar czf /backup/ollama-vol.tar.gz -C / data
docker run --rm -v openwebui:/data -v $PWD:/backup alpine tar czf /backup/openwebui-vol.tar.gz -C / data
docker compose up -d

Remove everything if you want to reclaim space later (this deletes models and chat data):

docker compose down
docker volume rm $(docker volume ls -q | grep -E "(ollama|openwebui)")

Troubleshooting

GPU not detected: Ensure the NVIDIA driver is installed on the host, the toolkit is configured, and your Compose service includes gpus: all. Validate with docker run --rm --gpus all nvidia/cuda:... nvidia-smi.

Permission denied: If you cannot run Docker without sudo, confirm your user is in the docker group (use id) and re-log in or run newgrp docker.

Port conflicts: If ports 3000 or 11434 are in use, change them in the Compose file and update your reverse proxy accordingly.

Low VRAM or OOM: Prefer 4–8B parameter models or quantized variants (e.g., llama3.1:8b in Q4_K_M). Ollama will automatically pick quantized builds when available.

Logs: Review service logs for errors and performance clues:

docker logs -f ollama
docker logs -f open-webui

Why This Stack?

Ollama offers a consistent way to pull and run many open-source models locally, while Open WebUI gives you a friendly chat experience, prompt presets, file uploads, and team features. Everything stays on your hardware, which improves privacy and often reduces cost. With Docker and a reverse proxy, this setup scales from a single developer laptop to a small team server with SSL and authentication.

You now have a modern, local LLM environment with a clear upgrade path. Add more models with ollama pull, automate backups on a cron schedule, and secure public access with Caddy or another reverse proxy. For most use cases, this stack is fast, reliable, and easy to maintain.

Overview

Running a private AI assistant on your own machine is now practical, fast, and secure. In this step-by-step guide, you will deploy Ollama (to run large language models locally) and Open WebUI (a clean chat interface) using Docker. The tutorial covers CPU-only mode and GPU acceleration with both NVIDIA and AMD ROCm, so you get the best performance out of your hardware. By the end, you will have a persistent setup that auto-starts on boot, supports one-click model management, and is easy to update.

Prerequisites

- A 64-bit Linux host (Ubuntu 22.04+ recommended) with internet access.
- Docker Engine installed and running.
- For NVIDIA GPUs: proprietary drivers and the NVIDIA Container Toolkit.
- For AMD GPUs: ROCm-capable GPU with ROCm drivers (5.7+).
- Open ports 11434 (Ollama API) and 3000 (Open WebUI). You can change ports if they collide with other services.

Step 1 — Install Docker (Ubuntu quick method)

If Docker is not installed, run:

sudo apt-get update && sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Step 2 — Enable GPU in Docker (optional but recommended)

NVIDIA: Install the NVIDIA driver and container toolkit, then restart Docker:

sudo apt-get install -y nvidia-driver-535 (or newer)
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker

AMD ROCm: Install ROCm drivers appropriate for your GPU. Ensure devices /dev/kfd and /dev/dri exist. No extra Docker runtime is needed; you will pass devices to the container.

Step 3 — Create a dedicated Docker network

Create a private bridge network so containers can discover each other by name:

docker network create ai

Step 4 — Start Ollama (choose one)

NVIDIA GPU:

docker run -d --name ollama --gpus=all --restart unless-stopped --network ai -p 11434:11434 -v ollama:/root/.ollama ollama/ollama:latest

AMD ROCm GPU:

docker run -d --name ollama --device /dev/kfd --device /dev/dri --group-add video --ipc=host --restart unless-stopped --network ai -p 11434:11434 -v ollama:/root/.ollama ollama/ollama:rocm

CPU-only:

docker run -d --name ollama --restart unless-stopped --network ai -p 11434:11434 -v ollama:/root/.ollama ollama/ollama:latest

Step 5 — Start Open WebUI and connect it to Ollama

Run Open WebUI on port 3000 and point it at the Ollama API using the container name:

docker run -d --name open-webui --restart unless-stopped --network ai -p 3000:8080 -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://ollama:11434 open-webui/open-webui:latest

Open a browser and visit http://YOUR_SERVER_IP:3000. On first launch, create your admin account. In Settings, disable public sign-ups if this instance is exposed to untrusted networks.

Step 6 — Download a model and chat

You can pull models from Open WebUI’s UI. Or pull via CLI:

docker exec -it ollama ollama pull llama3.1:8b

For lower VRAM, try quantized variants (for example: llama3.1:8b-instruct-q4_0). After the download completes, select the model in Open WebUI and start chatting locally.

Security and access tips

- Keep ports private if possible; bind to localhost and use an authenticated reverse proxy (Nginx, Caddy, or Traefik) if exposing to the internet.
- Regularly update images and disable open registration in Open WebUI.
- Use Docker volumes (already configured) to persist models and settings across updates.

Updating to the latest versions

docker pull ollama/ollama:latest (or ollama/ollama:rocm for AMD)
docker pull open-webui/open-webui:latest
docker stop open-webui ollama && docker rm open-webui ollama
Re-run the docker run commands from Steps 4–5. Your data persists in volumes.

Troubleshooting

- GPU not detected (NVIDIA): test with docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi.
- GPU not detected (AMD): ensure /dev/kfd and /dev/dri exist and your user is in the video group (the container already adds it).
- Slow or failed model pulls: check DNS/proxy and rerun the pull. Try a smaller or more quantized model.
- Port conflicts: change -p 11434:11434 or -p 3000:8080 to other free host ports.

Uninstall and cleanup

docker rm -f open-webui ollama
docker volume rm open-webui ollama
docker network rm ai

You now have a flexible, private AI stack that runs entirely on your machine. Swap models as needed, tune quantization for your hardware, and keep your data under your control.

Overview

This tutorial shows how to deploy a private, local AI stack with Ollama (model runtime) and Open WebUI (chat interface) using Docker on Ubuntu 22.04/24.04. You will learn how to run it on CPU, enable NVIDIA or AMD/ROCm GPU acceleration, secure the web interface, and keep everything up to date. The result is a fast, reliable, and low-maintenance setup suitable for labs, developers, and small teams.

Prerequisites

You need an Ubuntu 22.04 or 24.04 system with sudo access, 16 GB+ RAM (more is better), 20 GB+ free disk space, and a stable internet connection. For GPU acceleration, use a recent NVIDIA GPU with official drivers or a compatible AMD GPU with ROCm-capable kernel and hardware. Ensure ports 11434 (Ollama) and 3000 (Open WebUI) are free. If you plan to expose the service on the internet, prepare a domain name and DNS A/AAAA record pointing to the server.

Step 1: Install Docker Engine and Compose

sudo apt update && sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Step 2: GPU Preparation (optional but recommended)

NVIDIA: Install the proprietary driver and the NVIDIA Container Toolkit so Docker can access your GPU.

sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers install
sudo reboot

distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

AMD (ROCm): Ensure your GPU is ROCm-capable and the kfd and dri devices are present. Give your user access to the required groups.

sudo usermod -aG render,video $USER
sudo reboot

Step 3: Create a Docker Compose file

Create a working directory like ~/ai-stack, then create docker-compose.yml. The following example starts Ollama and Open WebUI with volumes for persistence. It includes variants for CPU, NVIDIA, and AMD. Only keep one GPU option at a time.

docker-compose.yml (CPU-only by default):

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
     - "11434:11434"
    volumes:
     - ollama:/root/.ollama
    environment:
     - OLLAMA_KEEP_ALIVE=24h
     - OLLAMA_NUM_THREADS=8
  # For NVIDIA GPU (uncomment the next 4 lines and comment the AMD lines below):
  #   runtime: nvidia
  #   environment:
  #    - NVIDIA_VISIBLE_DEVICES=all
  #    - NVIDIA_DRIVER_CAPABILITIES=compute,utility
  # For AMD ROCm GPU (use the ROCm image and device mappings):
  #   image: ollama/ollama:rocm
  #   devices:
  #    - /dev/kfd
  #    - /dev/dri
  #   group_add:
  #    - "video"
  #    - "render"
  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    depends_on:
     - ollama
    restart: unless-stopped
    environment:
     - OLLAMA_API_BASE=http://ollama:11434
    ports:
     - "3000:8080"
    volumes:
     - openwebui:/app/backend/data
volumes:
  ollama:
  openwebui:

Step 4: Start the stack

docker compose up -d

Check containers and logs to confirm both services are healthy.

docker ps
docker logs -f ollama
docker logs -f open-webui

Step 5: Pull a model and test

Use Ollama to download a model. Popular choices are llama3.1:8b, llama3.1:70b (needs more VRAM), mistral, or qwen2. Start with an 8B or 7B model to validate your setup.

docker exec -it ollama ollama pull llama3.1:8b
curl http://localhost:11434/api/tags

Open a browser to http://<server-ip>:3000. The first user that signs up in Open WebUI becomes the admin. In Settings, point the Ollama endpoint to http://ollama:11434 (it is already set via OLLAMA_API_BASE). Create a new chat and pick your model from the dropdown.

Step 6: Optional security and HTTPS

By default, Open WebUI is accessible on port 3000 and provides its own user system. For internet exposure, put it behind an HTTPS reverse proxy and disable public signups after creating the admin. If you use UFW, allow only necessary ports:

sudo ufw allow 22/tcp
sudo ufw allow 80,443/tcp
sudo ufw enable

A simple approach is to add a Caddy or Nginx reverse proxy in front of Open WebUI for automatic TLS. Map your domain (e.g., ai.example.com) to the server, then proxy requests to open-webui:8080. Limit administrative access using firewall rules, strong passwords, and, if available, SSO/OIDC in Open WebUI.

Step 7: Updating and backing up

To update images to the latest versions and apply them with minimal downtime:

cd ~/ai-stack
docker compose pull
docker compose up -d

Your models and chat data live in Docker volumes. Back them up regularly:

docker run --rm -v ollama:/data -v $(pwd):/backup alpine tar czf /backup/ollama-vol-$(date +%F).tgz -C /data .
docker run --rm -v openwebui:/data -v $(pwd):/backup alpine tar czf /backup/openwebui-vol-$(date +%F).tgz -C /data .

Troubleshooting tips

If GPU is not used on NVIDIA, confirm nvidia-smi works on the host and the container runtime is configured. For AMD, ensure /dev/kfd and /dev/dri exist and the container uses the ollama/ollama:rocm image with the proper device mappings. Model loading failures typically indicate insufficient RAM/VRAM; try a smaller quantization or a smaller model. If the UI cannot see Ollama, verify OLLAMA_API_BASE and that containers can resolve each other by service name.

You are done

You now have a modern, private AI chat stack running on Docker with optional GPU acceleration. Ollama keeps model management simple, and Open WebUI provides a clean, multi-user interface. This setup is easy to maintain, portable across servers, and ready for experimentation with different open-source models and embeddings.

Local large language models have matured fast. With Ollama and Open WebUI, you can run modern models like Llama 3.1 or Phi-3 locally, enjoy a clean chat interface, and use your NVIDIA GPU for real speed. This guide walks through a clean setup on Ubuntu 22.04 or 24.04 using Docker and the NVIDIA Container Toolkit.

What you will build: Ollama as your model runtime, Open WebUI as the web interface, both running on Docker, and NVIDIA GPU acceleration for high throughput.

Prerequisites: An Ubuntu 22.04/24.04 machine, an NVIDIA GPU with at least 8 GB VRAM (more is better), 16+ GB system RAM recommended, sudo access, and an internet connection.

1) Install NVIDIA Driver and Validate GPU

First, ensure your system sees the GPU and the driver is installed. If you already have a working driver and nvidia-smi reports correctly, you can skip to Docker installation.

Detect the GPU:
lspci | grep -i nvidia

Install the recommended driver:
sudo apt update
sudo ubuntu-drivers autoinstall
sudo reboot

Verify after reboot:
nvidia-smi
You should see driver, CUDA version, and GPU utilization stats.

2) Install Docker and NVIDIA Container Toolkit

We will run both Ollama and Open WebUI via containers. Install Docker first, then enable GPU support inside containers.

Install Docker Engine:
sudo apt update && sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER && newgrp docker

Install NVIDIA Container Toolkit:
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) && \
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test GPU in containers:
docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi
If you see the same GPU readout, the toolkit works.

3) Install Ollama

Ollama brings one-line model setup and fast local inference. Use the official installer:

curl -fsSL https://ollama.com/install.sh | sh

The install registers a systemd service and puts the binary at /usr/local/bin/ollama. Start or check status:

sudo systemctl enable --now ollama
systemctl status ollama

Quick test with CPU or GPU:
ollama run llama3.1:8b
Type a prompt; use Ctrl+C to exit. If you have a supported NVIDIA setup, Ollama will prefer GPU automatically.

4) Deploy Open WebUI with Docker

Open WebUI gives you a polished chat interface, prompt templates, and conversation history. We will connect it to the local Ollama API at http://host.docker.internal:11434 (or your host IP) from inside the container.

Run Open WebUI:
docker run -d --name openwebui --restart unless-stopped -p 3000:8080 \
-e OLLAMA_API_BASE=http://host.docker.internal:11434 \
-e WEBUI_AUTH=True \
-v openwebui-data:/app/backend/data \
ghcr.io/open-webui/open-webui:latest

Open a browser to http://SERVER_IP:3000. Create the first admin user and configure the default model (for example, llama3.1:8b).

5) Using the GPU with Open WebUI + Ollama

If your driver and toolkit are correct, Ollama will use the GPU. To confirm, watch GPU usage while making a request:

watch -n 1 nvidia-smi

To control GPU usage and offloading, set environment variables for the Ollama service. For example:

sudo systemctl edit ollama
Add lines under [Service]:
Environment="OLLAMA_NUM_GPU=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Save and reload:
sudo systemctl daemon-reload && sudo systemctl restart ollama

You can also pick quantized models to fit your VRAM. Examples: llama3.1:8b-q4_K_M (balanced), llama3.1:8b-q5_K_M (higher quality), or phi3:mini-4k-instruct-q4_K_M for small GPUs.

6) Optional: Docker Compose Setup

To run everything together and persist data, use Docker Compose:

mkdir -p ~/llm-stack && cd ~/llm-stack
nano compose.yml

Paste:

services:
  ollama:
   image: ollama/ollama:latest
   container_name: ollama
   restart: unless-stopped
   ports:
    - "11434:11434"
   volumes:
    - ollama:/root/.ollama
   deploy:
    resources:
     reservations:
      devices:
       - capabilities: ["gpu"]

  openwebui:
   image: ghcr.io/open-webui/open-webui:latest
   container_name: openwebui
   restart: unless-stopped
   ports:
    - "3000:8080"
   environment:
    - OLLAMA_API_BASE=http://ollama:11434
    - WEBUI_AUTH=True
   depends_on:
    - ollama
   volumes:
    - openwebui-data:/app/backend/data

volumes:
  ollama:
  openwebui-data:

docker compose up -d

7) Security, Backups, and Troubleshooting

Secure access: Keep Open WebUI behind a reverse proxy (Caddy, Nginx) with HTTPS. At minimum, enable built-in auth (already set with WEBUI_AUTH=True). For remote access, consider Tailscale or WireGuard rather than exposing port 3000 to the internet.

Persist and back up data: Ollama stores models in ~/.ollama/models (or the volume you mapped). Open WebUI keeps data in the openwebui-data volume. Back up with:

docker run --rm -v ollama:/src -v $PWD:/dst alpine tar czf /dst/ollama-backup.tgz -C /src .
docker run --rm -v openwebui-data:/src -v $PWD:/dst alpine tar czf /dst/openwebui-backup.tgz -C /src .

Common issues:

- Ollama says it cannot find a GPU: check driver and run nvidia-smi; ensure nvidia-container-toolkit is installed and Docker restarted.
- Model fails to load due to VRAM limits: pick a smaller or more aggressively quantized variant (q4, q3), or set num_ctx lower in the model settings.
- High CPU usage: disable background indexing features in the UI and avoid running multiple heavy models simultaneously.

8) Next Steps

Try function-calling and RAG with Open WebUI extensions, schedule model updates with ollama pull, and benchmark different quantizations for your GPU. With this stack, you own your data, enjoy low latency, and can iterate quickly without cloud costs.

Overview

Running large language models locally is easier than ever thanks to Ollama and Open WebUI. This tutorial shows you how to deploy both on Docker with optional NVIDIA GPU acceleration on Ubuntu 22.04/24.04. You will get a clean, repeatable setup, persistent storage, and a modern web interface to chat with models like Llama 3.1, Gemma, and Phi-3. If you do not have a GPU, you can still run smaller models on CPU.

What You Will Build

We will launch two containers: one for Ollama (the model runtime and API) and one for Open WebUI (the interface). They will be connected on a Docker network, with volumes for persistence. When done, you can open your browser, select a model, and start chatting locally—no cloud required.

Prerequisites

- Ubuntu 22.04 or 24.04, sudo access, and a stable internet connection.

- Docker installed (we will cover a quick install).

- Optional NVIDIA GPU with drivers for acceleration. CPU-only instructions are included.

1) Install Docker (if not installed)

Install the latest Docker CE from the official repo for better compatibility and security updates.

sudo apt update
sudo apt install -y ca-certificates curl gnupg lsb-release
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

2) Enable NVIDIA GPU Support (optional but recommended)

If you have an NVIDIA GPU, install the proprietary driver and NVIDIA Container Toolkit so Docker can access the GPU.

# Install recommended NVIDIA driver
sudo apt update
sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, install the NVIDIA Container Toolkit and integrate with Docker.

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPUs are visible to Docker:

docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

3) Create a Network and Volumes

We will keep Ollama models and WebUI data persistent across container restarts.

docker network create ai-net
docker volume create ollama-data
docker volume create openwebui-data

4) Run the Ollama Container

Start Ollama with GPU acceleration if available. The container exposes port 11434 for the API.

# GPU-enabled
docker run -d --name ollama --restart unless-stopped \
  --gpus all \
  -p 11434:11434 \
  -v ollama-data:/root/.ollama \
  --network ai-net \
  ollama/ollama:latest

# CPU-only (no GPU flag)
# docker run -d --name ollama --restart unless-stopped \
#   -p 11434:11434 \
#   -v ollama-data:/root/.ollama \
#   --network ai-net \
#   ollama/ollama:latest

5) Pull a Model

Use the Ollama CLI inside the container to download a model. Start with a balanced choice like Llama 3.1 8B or pick a smaller one if you are on CPU.

# Enter the container and pull a model
docker exec -it ollama bash -lc "ollama pull llama3.1:8b"

# Alternative smaller models:
# docker exec -it ollama bash -lc "ollama pull phi3:mini"
# docker exec -it ollama bash -lc "ollama pull mistral:7b"

6) Run Open WebUI

Open WebUI connects to the Ollama API. We will publish it on port 3000 and persist its configuration.

docker run -d --name openwebui --restart unless-stopped \
  -p 3000:8080 \
  -v openwebui-data:/app/backend/data \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  --network ai-net \
  ghcr.io/open-webui/open-webui:latest

Open your browser to http://localhost:3000, create an admin account, and pick your default model (e.g., llama3.1:8b). You can switch models anytime in the interface.

7) Test the Setup

Send a quick API test to confirm Ollama is serving responses:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "In one sentence, explain what a container is."
}'

If you get a streamed JSON response with text tokens, the backend is working. In Open WebUI, start a new chat and ask a question to validate the full stack.

8) Performance Tips

- Prefer GPU for 7B–14B models; CPU can be slow or memory-constrained. Smaller models like Phi-3 Mini run decently on modern CPUs.

- Add “q4_0” or “q5_1” quantized variants if available to reduce VRAM/RAM usage. Example: llama3.1:8b-instruct-q4_0.

- Limit GPU layers or context length if you see out-of-memory errors. In Open WebUI, lower max tokens and system prompt size.

9) Troubleshooting

Docker can’t see the GPU: Ensure the NVIDIA driver is installed, run nvidia-smi on the host, and confirm nvidia-ctk runtime configure was applied. Restart Docker and try the CUDA test container again.

Model download is slow or fails: Check DNS and firewall, or use a different mirror via environment variables if your network requires a proxy. You can also prefetch models on a faster connection and copy the volume.

Open WebUI cannot reach Ollama: Confirm both containers are on the ai-net network and OLLAMA_BASE_URL points to http://ollama:11434. Check logs with docker logs openwebui and docker logs ollama.

Out of memory: Choose a smaller model, use a stronger quantization, or on GPU, close other VRAM-heavy apps. For CPU, add swap if RAM is limited.

10) Security and Maintenance

Do not expose ports 11434 or 3000 publicly without authentication and TLS. If you must access remotely, restrict with a reverse proxy, HTTPS, and basic auth or OAuth. Keep images updated:

docker pull ollama/ollama:latest
docker pull ghcr.io/open-webui/open-webui:latest
docker stop openwebui ollama
docker rm openwebui ollama
# re-run the docker run commands from above (volumes preserve your data)

Wrap-Up

You now have a robust, GPU-capable local AI stack: Ollama serving models and Open WebUI providing a polished chat interface. Because everything runs in Docker with volumes, you can upgrade, backup, and migrate with minimal friction. Experiment with different models, tune prompts, and enjoy private, offline AI on your own hardware.

Running a private, fast large language model (LLM) on your own hardware is now practical thanks to Ollama and Open WebUI. This guide shows how to install Ollama on Ubuntu (with NVIDIA GPU acceleration) and connect it to Open WebUI for a clean, chat-style interface. You will get a secure, local setup that can run modern models such as Llama 3.1 and Mistral without sending data to the cloud.

What You’ll Build

You will install Ollama as a system service on Ubuntu 22.04/24.04, download a model, and run Open WebUI in Docker. The GPU will be used by Ollama to accelerate inference while Open WebUI provides a browser-based interface. The result is a local chat environment accessible at http://localhost:3000.

Prerequisites

• Ubuntu 22.04 LTS or 24.04 LTS on a machine with at least 16 GB RAM (more is better).
• An NVIDIA GPU (6 GB+ VRAM recommended) and the proprietary NVIDIA driver.
• Sudo privileges and internet access.

Step 1: Prepare Ubuntu and NVIDIA Drivers

Update the system first: sudo apt update && sudo apt upgrade -y. Install the latest recommended NVIDIA driver: ubuntu-drivers devices to inspect, then sudo ubuntu-drivers autoinstall. Reboot: sudo reboot. After reboot, confirm the GPU is visible with nvidia-smi. If you see your GPU and driver version, you’re ready.

Step 2: Install Ollama

Ollama is a lightweight runtime for local LLMs. Install it with: curl -fsSL https://ollama.com/install.sh | sh. The installer sets up the service and binary. Start or restart the service if needed: sudo systemctl restart ollama. By default, the Ollama API listens on http://127.0.0.1:11434.

Step 3: Pull a Model and Test

Pull a modern, efficient model. Examples: ollama pull llama3.1:8b or ollama pull mistral:7b. Test generation on the CLI: ollama run llama3.1:8b then type a prompt. Or test the API: curl http://localhost:11434/api/generate -d '{"model":"llama3.1:8b","prompt":"Hello"}'. If a model loads and responds, your base setup is working and it will use the GPU when possible.

Step 4: Install Docker (for Open WebUI)

If Docker is missing, install it quickly:
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io
Optional but recommended: sudo usermod -aG docker $USER then log out/in.

Step 5: Launch Open WebUI

Open WebUI connects to the local Ollama API and gives a feature-rich chat interface with history and prompt templates. On Linux, map the host gateway inside the container so it can reach Ollama on the host:

docker run -d --name openwebui --restart unless-stopped -p 3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 --add-host=host.docker.internal:host-gateway -v openwebui-data:/app/backend/data ghcr.io/open-webui/open-webui:latest

Open a browser and visit http://localhost:3000. In Settings → Connections, ensure the Ollama base URL is http://host.docker.internal:11434. You can now select the model (e.g., llama3.1:8b) and chat.

Step 6: Optimize Memory and Context

If VRAM is limited, choose a smaller quantization (e.g., llama3.1:8b often defaults to Q4_K_M). List quantizations with ollama show llama3.1:8b. To increase context, create a custom model:
printf "FROM llama3.1:8b\nPARAMETER num_ctx 8192\n" > Modelfile
ollama create llama3.1-8k -f Modelfile
Then select llama3.1-8k in Open WebUI.

Step 7: Secure and Expose (Optional)

Keep Ollama bound to localhost unless you need remote access. If you must expose it, put Open WebUI behind a reverse proxy with TLS (e.g., Nginx or Caddy) and enable authentication in Open WebUI (Settings → Auth). For UFW: sudo ufw allow 22/tcp, sudo ufw allow 3000/tcp (if local), then sudo ufw enable. Prefer a proper domain and HTTPS if accessing remotely.

Updating and Backups

Update Ollama with the installer again or your package manager, then restart: sudo systemctl restart ollama. Update Open WebUI by pulling a new image: docker pull ghcr.io/open-webui/open-webui:latest then docker stop openwebui && docker rm openwebui and re-run the docker run command. Back up your chat data from the Docker volume openwebui-data using docker run --rm -v openwebui-data:/data -v $PWD:/backup alpine tar czf /backup/openwebui-data.tgz -C / data.

Troubleshooting

• GPU not used: verify nvidia-smi shows a running process during inference. Ensure the proprietary driver is loaded. If you previously installed CUDA separately, avoid driver mismatches.
• Open WebUI can’t reach Ollama: confirm curl http://localhost:11434 on the host works and that the Docker container uses --add-host=host.docker.internal:host-gateway or switch to --network host if acceptable.
• Slow first response: the first prompt loads weights into memory; subsequent prompts are faster. Consider smaller models or different quantization if latency is too high.
• Out-of-memory: reduce context (num_ctx), switch to a lower-precision quantization, or choose a smaller model (e.g., llama3.1:8b-instruct with Q4).

You’re Done

You now have a private, GPU-accelerated LLM stack on Ubuntu. Ollama handles fast local inference, and Open WebUI gives you a slick, familiar chat interface. With careful model selection, quantization, and context tuning, this setup can power assistants, coding copilots, and knowledge bots entirely on your hardware.

Overview

This step-by-step guide shows you how to deploy an AI chatbot stack with Ollama (for running local LLMs) and Open WebUI (a clean, browser-based interface) on Ubuntu 22.04 or 24.04. We will run both apps in Docker, enable NVIDIA GPU acceleration, and put Nginx in front with a free TLS certificate from Let's Encrypt. You will get a production-friendly setup with persistent storage, HTTPS, and simple maintenance commands.

What you'll build

You will end up with two containers on a private Docker network: ollama (listening on 11434) and openwebui (listening on 8080, mapped to localhost:3000). Nginx will reverse proxy a public domain (for example, ai.example.com) to Open WebUI and handle SSL. Models and chat data will be stored on the host so updates don't wipe them.

Prerequisites

- Ubuntu 22.04/24.04 with sudo access
- An NVIDIA GPU (Turing or newer recommended) and a supported driver
- A DNS A record pointing your domain (e.g., ai.example.com) to your server's public IP
- Outbound internet access to pull images and models

1) Install NVIDIA driver and container toolkit

Update the system and install the proprietary driver. If you don't already have the correct driver, Ubuntu can choose one for you:
sudo apt update && sudo apt -y upgrade
sudo ubuntu-drivers autoinstall
Reboot:
sudo reboot

After reboot, confirm the GPU is visible:
nvidia-smi
Install the NVIDIA Container Toolkit so Docker can use the GPU:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt -y install nvidia-container-toolkit
Configure Docker to use it and restart Docker:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

2) Install Docker Engine and Docker Compose plugin

Install the official Docker packages:
sudo apt -y install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release; echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Verify:
docker --version
docker compose version

3) Create persistent folders and a Docker Compose file

Create directories for persistent data:
sudo mkdir -p /opt/ollama /opt/openwebui
sudo chown -R $USER:$USER /opt/ollama /opt/openwebui
Now create a compose.yml in a new project folder (for example, /opt/ai-stack/compose.yml) with the following content:

version: "3.8"
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
volumes:
- /opt/ollama:/root/.ollama
environment:
- OLLAMA_KEEP_ALIVE=24h
networks:
- ai
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]

openwebui:
image: ghcr.io/open-webui/open-webui:latest
container_name: openwebui
restart: unless-stopped
depends_on:
- ollama
environment:
- OLLAMA_BASE_URL=http://ollama:11434
ports:
- "127.0.0.1:3000:8080"
volumes:
- /opt/openwebui:/app/backend/data
networks:
- ai

networks:
ai:

Notes: We bind Open WebUI to localhost:3000 so it is not exposed directly. Nginx will handle public traffic. The NVIDIA device reservation passes the GPU into the Ollama container. If your Docker Compose version supports it, you may also use gpus: all under the ollama service instead of the deploy block.

4) Start the stack and test

From the folder with compose.yml, bring the stack up:
docker compose up -d
Watch logs until both services are healthy:
docker compose logs -f
Pull a model and perform a quick GPU test (you should see GPU usage spike in nvidia-smi):
docker exec -it ollama ollama pull llama3:8b
docker exec -it ollama ollama run llama3:8b
Locally, you can visit Open WebUI at http://127.0.0.1:3000. Next, we'll put it behind HTTPS.

5) Install Nginx and obtain a Let's Encrypt certificate

Install Nginx and Certbot:
sudo apt -y install nginx certbot python3-certbot-nginx
Create an Nginx server block (replace ai.example.com with your domain):
sudo nano /etc/nginx/sites-available/ai.conf
Paste:

server {
listen 80;
server_name ai.example.com;
location / {
proxy_pass http://127.0.0.1:3000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
client_max_body_size 20m;
}

Enable and test:
sudo ln -s /etc/nginx/sites-available/ai.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
Issue a certificate and force HTTPS:
sudo certbot --nginx -d ai.example.com --redirect --agree-tos -m [email protected]
Now browse to https://ai.example.com and you should see Open WebUI served over TLS.

6) Backups, updates, and security tips

Back up data by archiving the two directories we created:
sudo tar -czf /root/ollama-backup.tgz /opt/ollama
sudo tar -czf /root/openwebui-backup.tgz /opt/openwebui
Open WebUI stores conversations and settings in /opt/openwebui; Ollama stores models and blobs in /opt/ollama.

To update images with minimal downtime:
docker compose pull
docker compose up -d
Old images can be cleaned with docker image prune when you're done testing. For security, keep ports private (we only published 3000 to localhost) and use your firewall to allow 80/443 only. If you need extra protection, add HTTP Basic Auth to Nginx and restrict by IP when possible.

Troubleshooting

- GPU not used: watch nvidia-smi while running a model. If it stays idle, recheck the NVIDIA driver, container toolkit, and the GPU reservation in compose.yml.
- Models fail due to VRAM limits: try smaller variants (e.g., llama3:8b instead of 70B) or quantized builds (like q4_K_M).
- Port conflicts: change the host port mapping in compose.yml if 3000 is taken (e.g., use 127.0.0.1:3100:8080 and update the Nginx proxy_pass accordingly).
- Certbot issues: make sure your domain points to the server’s public IP and TCP/80 is reachable from the internet during certificate issuance.

What's next

From here, you can connect more tools to the Ollama API, add multiple models, fine-tune your Nginx headers, or place the stack behind Cloudflare. The setup is simple to maintain: pull updates, restart the stack, and your data persists. With GPU acceleration and HTTPS in place, you have a fast, private AI assistant ready for everyday use.

Running a private, local large language model (LLM) stack has become straightforward thanks to Ollama and Open WebUI. In this tutorial, you will set up Ollama on Ubuntu 24.04 for local inference and connect it to Open WebUI for a clean, feature-rich chat interface. Optional steps cover NVIDIA GPU acceleration and a one-container alternative. The end result is a fast, private, and flexible AI workstation or lab setup.

Prerequisites

You need an Ubuntu 24.04 machine with at least 16 GB RAM (more is better for larger models), 20+ GB free disk space, and a stable internet connection. For GPU acceleration, an NVIDIA GPU with recent drivers (CUDA 12+ capable) is recommended. Administrative shell access (sudo) is required.

Step 1 — Update the system

Start by updating the OS and rebooting to ensure a clean baseline:

sudo apt update && sudo apt -y upgrade
sudo reboot

Step 2 — (Optional) Enable NVIDIA GPU acceleration

If you have an NVIDIA GPU, install the recommended proprietary driver. This enables Ollama to offload model layers to the GPU for significant speedups.

sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, verify the driver is active:

nvidia-smi

If you see your GPU listed with a driver version, you are ready for GPU acceleration. If not, check Secure Boot status, ensure the driver matches your GPU, and review dmesg for driver signing or module load errors.

Step 3 — Install Ollama

Ollama is a lightweight runtime for local LLMs. Install it with the official script and enable the systemd service:

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama
systemctl status ollama

By default, Ollama listens on localhost:11434 and will auto-detect NVIDIA GPUs if drivers are present. To test the API, run:

curl -s http://127.0.0.1:11434/api/tags

Pull a model (e.g., Llama 3 8B) and do a quick prompt test:

ollama pull llama3
ollama run llama3

Type a sample question to confirm token generation. Exit with Ctrl+C.

Step 4 — Install Docker (for Open WebUI)

Open WebUI is easiest to run in a container. Install the Docker Engine and Compose plugin:

sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Step 5 — Run Open WebUI connected to host Ollama

This keeps Ollama on the host and runs Open WebUI in Docker. The container will reach Ollama on 127.0.0.1:11434 via the special host-gateway address.

docker run -d --name open-webui \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -e OLLAMA_API_BASE=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  --restart unless-stopped \
  ghcr.io/open-webui/open-webui:latest

Open your browser to http://SERVER_IP:3000, create the initial admin account, and confirm that the Ollama connection shows as healthy. From Settings, you can select the default model, import additional models, and configure conversation settings such as system prompts and context length.

Step 6 — Verify GPU usage and performance

Start a chat in Open WebUI and watch GPU utilization:

watch -n 1 nvidia-smi

If GPU utilization stays at 0%, verify your NVIDIA driver is active. Ollama will automatically offload supported layers to the GPU when available. For very small models, GPU may not be heavily utilized.

Optional — All-in-one container (Open WebUI + Ollama)

If you prefer to containerize everything, use the combined image. This requires the NVIDIA Container Toolkit for GPU access from Docker.

sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

docker run -d --name openwebui-ollama \
  --gpus all \
  -p 3000:8080 \
  -v open-webui:/app/backend/data \
  -v ollama:/root/.ollama \
  --restart unless-stopped \
  ghcr.io/open-webui/open-webui:ollama

This image embeds Ollama and automatically wires it to Open WebUI. It will download models into the ollama volume. If you do not have a GPU, remove the --gpus all flag.

Security and hardening tips

By default, Ollama listens only on localhost. Keep it that way unless you are placing a reverse proxy in front (Nginx, Caddy, or Traefik) for TLS. Restrict access to port 3000 using a firewall (UFW or security group) and enable Open WebUI authentication during initial setup. If you must expose the interface, ensure HTTPS termination and strong credentials, and consider IP allowlists.

Troubleshooting

If Open WebUI cannot connect to Ollama, confirm the API is reachable: curl -s http://127.0.0.1:11434/api/tags. If it works, recheck the container run command and the host-gateway mapping. For slow inference, try a smaller model (e.g., mistral, phi3, or qwen2:0.5b/1.5b) and ensure swap is available. If downloads fail, verify DNS and outbound firewall rules.

You now have a modern, local AI stack on Ubuntu 24.04 that is private, fast, and extensible. Explore models like llama3:instruct, qwen2, mistral, or codellama for coding, and tailor prompts, templates, and context settings in Open WebUI for your specific workflows.