Deploy OpenWebUI and Ollama with NVIDIA GPU on Ubuntu using Docker Compose

on

Local AI is now practical: with Ollama you can run large language models (LLMs) on your machine, and OpenWebUI gives you a clean, chat-style interface. In this tutorial, you will deploy both on Ubuntu 22.04/24.04 using Docker Compose, with optional NVIDIA GPU acceleration for much faster inference.

Why OpenWebUI + Ollama?

Ollama manages model downloads and provides an OpenAI-compatible API at /v1. OpenWebUI is a lightweight, self-hosted web frontend that connects to Ollama and adds chat history, prompt templates, and simple administration. Together, they create a private, zero-cost alternative to cloud AI for development, prototyping, and offline use.

Prerequisites

- Ubuntu 22.04 LTS or 24.04 LTS with sudo access.
- Stable internet connection and at least 16 GB of RAM recommended for medium models.
- Optional but recommended: an NVIDIA GPU (Turing or newer) with recent drivers for CUDA acceleration.

Step 1 — Install Docker and Docker Compose

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER

Log out and back in (or reboot) to apply the new group membership so you can run Docker without sudo.

Step 2 — Enable GPU support (NVIDIA Container Toolkit)

If you do not have an NVIDIA GPU, skip to Step 3. If you do, install the proprietary driver first:

sudo ubuntu-drivers install
sudo reboot

After reboot, verify the driver:

nvidia-smi

Install the NVIDIA Container Toolkit so Docker can access the GPU:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test Docker GPU access:

docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

Step 3 — Create the Docker Compose file

Create a project directory and the Compose file:

mkdir -p ~/ai-stack && cd ~/ai-stack
nano compose.yml

Paste the following contents and save:

version: "3.9"
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
ports:
- "11434:11434"
volumes:
- ollama:/root/.ollama
environment:
- OLLAMA_KEEP_ALIVE=5m
- OLLAMA_MAX_LOADED_MODELS=2
gpus: all # Remove this line if you do not have an NVIDIA GPU

openwebui:
image: ghcr.io/open-webui/open-webui:main
container_name: openwebui
restart: unless-stopped
ports:
- "8080:8080"
environment:
- OLLAMA_BASE_URL=http://ollama:11434
- WEBUI_AUTH=True # Require login
volumes:
- openwebui:/app/backend/data
depends_on:
- ollama

volumes:
ollama:
openwebui:

Step 4 — Launch the stack

docker compose up -d

Open OpenWebUI in your browser: http://<your-server-ip>:8080. On first run, create an admin account when prompted. The backend (Ollama) will be reachable at http://ollama:11434 inside the Docker network and at http://<your-server-ip>:11434 from your LAN.

Step 5 — Download a model and test

Pull a model with the Ollama CLI (inside the container) or use OpenWebUI’s “Models” tab:

docker exec -it ollama ollama pull llama3.1:8b

Try a quick prompt:

docker exec -it ollama ollama run llama3.1:8b "Explain what a vector database is in one paragraph."

Return to OpenWebUI and start chatting with the downloaded model. If you have a GPU configured, latency will drop significantly compared to CPU-only mode.

Optional — Secure and expose the UI

- Keep OpenWebUI private on your LAN and enable authentication (WEBUI_AUTH=True) as shown.
- For public access, place a reverse proxy like Caddy, Nginx Proxy Manager, or Traefik in front, and obtain Let’s Encrypt certificates. Bind OpenWebUI to 127.0.0.1:8080 and publish the proxy instead.

Troubleshooting

- GPU not detected: verify nvidia-smi works on the host. Re-run sudo nvidia-ctk runtime configure --runtime=docker and sudo systemctl restart docker. Ensure the gpus: all line is present for the ollama service and restart with docker compose up -d.
- Slow or out-of-memory errors: try a smaller model (for example, llama3.2:3b), or add --num-ctx 2048 in OpenWebUI’s model settings to reduce memory use.
- Port conflicts: change the host ports in the Compose file (e.g., "8081:8080").
- Persistence: models are stored in the ollama volume; UI data (prompts, chats) in the openwebui volume. Back them up with docker run --rm -v ollama:/data -v $(pwd):/backup alpine tar czf /backup/ollama.tar.gz -C / data.

Maintenance tips

- Update images: docker compose pull && docker compose up -d.
- View logs: docker compose logs -f ollama and docker compose logs -f openwebui.
- Use the OpenAI-compatible API: your apps can point to http://<server-ip>:11434/v1 with the model name you downloaded. Most SDKs accept a custom base URL and a dummy API key.

You now have a modern, private AI stack running locally. Iterate on prompts, fine-tune your workflow, and scale up to larger models as your hardware allows—all while keeping your data on your own machine.

Read more
Post a Comment
Subscribe to: Posts (Atom)

Popular Posts

Install Ollama and Open WebUI on Ubuntu 24.04 with NVIDIA GPU Acceleration (Step-by-Step)

Overview Running a local large language model (LLM) is easier than ever thanks to Ollama and Open WebUI. Ollama handles model downloads and inference, while Open WebUI gives you a clean, chat-style interface in your browser. In this tutorial, you'll install both on Ubuntu 24.04 (works on 22.04 too), enable NVIDIA GPU acceleration, and deploy them with Docker Compose. The result is a fast, private AI stack you control. What You'll Need - Ubuntu 24.04 or 22.04 (fresh or existing server/desktop). - An NVIDIA GPU with recent drivers (Turing/RTX or newer recommended). - Root or sudo access. - Open ports 3000 (Open WebUI) and 11434 (Ollama) on your firewall if you access remotely. Step 1: Install NVIDIA Drivers and Verify CUDA First, update your system and install the recommended NVIDIA driver. On Ubuntu Desktop you can use Additional Drivers, but the CLI route is reliable: sudo apt update && sudo apt -y upgrade sudo ubuntu-drivers autoinstall sudo reboot After reboo...

Install Ollama + Open WebUI on Ubuntu 24.04 with NVIDIA GPU Acceleration (Step-by-Step)

Overview This guide shows you how to install Ollama and Open WebUI on Ubuntu 24.04 with NVIDIA GPU acceleration. You will get a fast local AI stack that can run modern large language models (LLMs) like Llama 3 on your own hardware, secured and accessible in a web browser. We will cover GPU driver setup, Docker with the NVIDIA Container Toolkit, Ollama installation, Open WebUI deployment, basic security, and troubleshooting. Prerequisites - A machine running Ubuntu 24.04 (fresh or updated). - An NVIDIA GPU with at least 8 GB VRAM recommended for 8B models (more is better). - Internet access and sudo privileges. Step 1: Install the NVIDIA GPU Driver Install the recommended proprietary driver so CUDA becomes available to both Ollama and containers. sudo apt update && sudo apt install -y ubuntu-drivers-common sudo ubuntu-drivers autoinstall sudo reboot After reboot, verify the GPU is recognized: nvidia-smi You should see a driver version and your GPU model. If not, check...

Install a Local AI Chatbot on Ubuntu 24.04 with Ollama and Open WebUI (Step-by-Step)

Running a fast, private AI chatbot on your own computer or server is easier than ever. In this guide, you will install Ollama (a lightweight local LLM runtime) and Open WebUI (a modern web interface) on Ubuntu 24.04 . You will be able to chat with models like Llama 3 or Mistral without sending data to the cloud, and with optional GPU acceleration if you have an NVIDIA card. What you will need : an Ubuntu 22.04/24.04 machine (VM, bare metal, or WSL), at least 8 GB RAM (16 GB recommended for larger models), 15–30 GB free disk space for models, Internet access, and optional NVIDIA GPU drivers for acceleration. Why Ollama + Open WebUI? Ollama manages local large language models (LLMs) with simple commands and sensible defaults. Open WebUI gives you a clean, chat-style interface with features like prompt history, file uploads (for some models), and model switching. Together, they are a simple, reliable stack for a self-hosted AI experience. 1) Update Ubuntu and install basics First, ...

Trending Now

Recovering from Btrfs Boot Failures Using GUI Tools on Fedora

on
By the end of this guide the reader will be able to identify a Btrfs‑based Fedora installation, boot from a live USB, list and restore snapshots using the graphical utilities btrfs‑assistant and snapper, and verify that the system returns to a functional state without resorting to the command line. Understanding the Btrfs Layout Used by Fedora Fedora Workstation and Fedora KDE install the root filesystem as a single Btrfs partition that contains two default sub‑volumes. One sub‑volume holds the traditional “/” hierarchy, while the second is dedicated to /var/lib/machines . The latter exists to keep container images out of snapshot operations; it remains empty on systems that do not run virtual machines. Because Btrfs stores data in sub‑volumes rather than separate partitions, a snapshot captures the state of an entire sub‑volume at a point in time. The installer (Anaconda) automatically registers these sub‑volumes with the snapper service. Snapper maintains a series of read‑only ...