How to Deploy Ollama and Open WebUI with NVIDIA GPU on Ubuntu using Docker Compose

on

Overview

This tutorial shows how to deploy Ollama (a lightweight local LLM runner) and Open WebUI (a clean chat interface) on Ubuntu using Docker Compose, with optional NVIDIA GPU acceleration. You will get a private, browser-based interface to run models like Llama 3 locally without sending data to the cloud. The steps work on Ubuntu 22.04/24.04. If you do not have a compatible NVIDIA GPU, you can still run everything on CPU.

Prerequisites

- Ubuntu 22.04 or newer (server or desktop)

- Sudo privileges

- For GPU acceleration: an NVIDIA GPU with recent drivers and CUDA support

1) Install Docker and Docker Compose

Install the Docker Engine and the Compose plugin in a few commands. You can also use official packages from Docker to stay current.

curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker $USER
newgrp docker
sudo apt-get install -y docker-compose-plugin
docker compose version

2) Install NVIDIA Container Toolkit (GPU only)

If you have an NVIDIA GPU, install drivers (if you have not already), then add the NVIDIA Container Toolkit so Docker can access the GPU.

# Install recommended NVIDIA drivers (reboot may be required)
sudo apt-get update
sudo apt-get install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
# After this finishes, reboot if drivers were installed/updated
# sudo reboot

# Add NVIDIA Container Toolkit repository and install
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# Configure Docker runtime and restart Docker
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU access from Docker:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

You should see your GPU listed. If not, ensure drivers are installed correctly and Docker has restarted.

3) Create the Docker Compose project

Create a working directory and a Compose file that runs two services: Ollama (backend API) and Open WebUI (frontend). The configuration below binds Ollama to localhost for safety and exposes Open WebUI on port 3000.

mkdir -p ~/ollama-openwebui
cd ~/ollama-openwebui
nano docker-compose.yml

Paste the following content. If you do not have a GPU, remove the runtime line and the NVIDIA environment variable in the ollama service.

version: "3.8"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=24h
      - NVIDIA_VISIBLE_DEVICES=all
    runtime: nvidia

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_API_BASE=http://ollama:11434
    volumes:
      - open-webui:/app/backend/data

volumes:
  ollama:
  open-webui:

4) Start the stack

Bring everything up in the background and follow the logs to ensure both services are healthy.

docker compose up -d
docker compose ps
docker compose logs -f ollama

5) Pull and run a model

Use Ollama to pull a model such as llama3 (Meta’s Llama 3 8B). This downloads the model into the ollama volume. The first pull may take a while.

docker exec -it ollama ollama pull llama3
docker exec -it ollama ollama run llama3

You can also open the browser interface at http://<your-server-ip>:3000, choose Ollama as the provider (it should auto-detect), select or download a model, and start chatting. When a supported GPU is available, Ollama will automatically use it.

6) Secure and tune

Network access: By default, Ollama is bound to localhost and Open WebUI is exposed on port 3000. If this is a public server, restrict access with a firewall (UFW, security groups) or place Open WebUI behind a reverse proxy (Caddy, Nginx) with HTTPS.

Authentication: Open WebUI supports user accounts. Open the UI, create an admin, and disable open signups in Settings if you do not want others to register. For single-user setups, keep the service bound to a private network.

Model storage: Models are stored in the ollama named volume. To reclaim space, remove unused models with docker exec -it ollama ollama rm <model>.

Updates: Update to the latest images and restart:

docker compose pull
docker compose up -d

Troubleshooting

Docker cannot see the GPU: If --gpus all fails, confirm the NVIDIA driver is installed and loaded (nvidia-smi on host). Re-run sudo nvidia-ctk runtime configure --runtime=docker and sudo systemctl restart docker. Some older setups require a reboot after driver installation.

Address already in use: Change ports in the Compose file (for example, map Open WebUI to "127.0.0.1:3001:8080") if port 3000 is occupied.

CPU-only mode: If you have no GPU, remove the runtime: nvidia line and NVIDIA_VISIBLE_DEVICES environment variable from the ollama service. Performance will be lower but still usable for smaller models.

Slow inference: Use quantized models (e.g., llama3:8b-instruct-q4_0) and increase RAM/swap. On GPU, ensure you have sufficient VRAM; otherwise, use a smaller or more aggressively quantized model.

What you built

You now have a maintainable, containerized local AI stack with Ollama and Open WebUI on Ubuntu. This setup is easy to update, safe to run offline, and flexible enough to add more services later (embeddings, vector databases, or reverse proxies). Enjoy private, low-latency LLM chat on your own hardware.

Comments

Post a Comment

Subscribe to: Post Comments (Atom)

Popular Posts

Install Ollama and Open WebUI on Ubuntu 24.04 with NVIDIA GPU Acceleration (Step-by-Step)

Overview Running a local large language model (LLM) is easier than ever thanks to Ollama and Open WebUI. Ollama handles model downloads and inference, while Open WebUI gives you a clean, chat-style interface in your browser. In this tutorial, you'll install both on Ubuntu 24.04 (works on 22.04 too), enable NVIDIA GPU acceleration, and deploy them with Docker Compose. The result is a fast, private AI stack you control. What You'll Need - Ubuntu 24.04 or 22.04 (fresh or existing server/desktop). - An NVIDIA GPU with recent drivers (Turing/RTX or newer recommended). - Root or sudo access. - Open ports 3000 (Open WebUI) and 11434 (Ollama) on your firewall if you access remotely. Step 1: Install NVIDIA Drivers and Verify CUDA First, update your system and install the recommended NVIDIA driver. On Ubuntu Desktop you can use Additional Drivers, but the CLI route is reliable: sudo apt update && sudo apt -y upgrade sudo ubuntu-drivers autoinstall sudo reboot After reboo...

Install a Local AI Chatbot on Ubuntu 24.04 with Ollama and Open WebUI (Step-by-Step)

Running a fast, private AI chatbot on your own computer or server is easier than ever. In this guide, you will install Ollama (a lightweight local LLM runtime) and Open WebUI (a modern web interface) on Ubuntu 24.04 . You will be able to chat with models like Llama 3 or Mistral without sending data to the cloud, and with optional GPU acceleration if you have an NVIDIA card. What you will need : an Ubuntu 22.04/24.04 machine (VM, bare metal, or WSL), at least 8 GB RAM (16 GB recommended for larger models), 15–30 GB free disk space for models, Internet access, and optional NVIDIA GPU drivers for acceleration. Why Ollama + Open WebUI? Ollama manages local large language models (LLMs) with simple commands and sensible defaults. Open WebUI gives you a clean, chat-style interface with features like prompt history, file uploads (for some models), and model switching. Together, they are a simple, reliable stack for a self-hosted AI experience. 1) Update Ubuntu and install basics First, ...

Install Ollama + Open WebUI on Ubuntu 24.04 with NVIDIA GPU Acceleration (Step-by-Step)

Overview This guide shows you how to install Ollama and Open WebUI on Ubuntu 24.04 with NVIDIA GPU acceleration. You will get a fast local AI stack that can run modern large language models (LLMs) like Llama 3 on your own hardware, secured and accessible in a web browser. We will cover GPU driver setup, Docker with the NVIDIA Container Toolkit, Ollama installation, Open WebUI deployment, basic security, and troubleshooting. Prerequisites - A machine running Ubuntu 24.04 (fresh or updated). - An NVIDIA GPU with at least 8 GB VRAM recommended for 8B models (more is better). - Internet access and sudo privileges. Step 1: Install the NVIDIA GPU Driver Install the recommended proprietary driver so CUDA becomes available to both Ollama and containers. sudo apt update && sudo apt install -y ubuntu-drivers-common sudo ubuntu-drivers autoinstall sudo reboot After reboot, verify the GPU is recognized: nvidia-smi You should see a driver version and your GPU model. If not, check...

Trending Now

Homogeneous Nucleation on Mars: A Textbook Process Finally Observed in Nature

on
By the end of this piece, you will understand what the Arsia Mons Elongated Cloud is, why it resisted explanation for several years after its first observation, and what the newly published model reveals about a physical process that has never been directly confirmed on any planet's atmosphere. The cloud that broke the model Every morning during Martian spring and summer, a thin, extremely long cloud forms near the Arsia Mons volcano and gradually elongates until it stretches to more than 1,000 miles before dissipating. The cycle then repeats for several months. The formation was first recorded by a probe in 2018, and from that point onward it became a persistent puzzle for atmospheric scientists working with Mars data. The cloud's geometry, its timing, and its seasonal recurrence all pointed to a real and reproducible physical process, yet no existing model could reproduce it. For a technical audience familiar with simulation work, the situation is analogous to a product...