LifeBytes Journal

Overview

This step-by-step guide shows you how to run local Large Language Models (LLMs) on Ubuntu using Ollama and Open WebUI. You will install Ollama, optionally enable NVIDIA GPU acceleration, and deploy Open WebUI in Docker to get a fast, friendly chat interface. By the end, you will have a private AI assistant running on your own hardware with secure access options and practical troubleshooting tips.

Prerequisites

Use Ubuntu 22.04 or 24.04 with at least 8 GB of RAM (16 GB recommended). For GPU acceleration, an NVIDIA GPU with 8 GB or more VRAM is ideal. You need sudo access and open ports 11434 for Ollama and 3000 (or your choice) for Open WebUI. This guide covers both CPU-only and GPU setups, so you can start even without a supported GPU.

Step 1: (Optional) Install NVIDIA Drivers and CUDA

If you plan to use a GPU, first confirm your hardware with lspci | grep -i nvidia. Install the recommended driver via sudo ubuntu-drivers autoinstall, then reboot. After rebooting, verify the driver with nvidia-smi. If you will run Open WebUI with GPU access in Docker, also install the NVIDIA container runtime using sudo apt-get install -y nvidia-container-toolkit and configure Docker with sudo nvidia-ctk runtime configure followed by sudo systemctl restart docker.

Step 2: Install Ollama on Ubuntu

Install Ollama with a single command: curl -fsSL https://ollama.com/install.sh | sh. This creates a system service and exposes the local API on http://127.0.0.1:11434. Check the version with ollama -v and verify the service using systemctl status ollama. If you need remote access on your LAN, set the host binding by creating an override file. Run sudo systemctl edit ollama, add [Service] and Environment="OLLAMA_HOST=0.0.0.0:11434", then save, sudo systemctl daemon-reload, and sudo systemctl restart ollama. Only expose Ollama on trusted networks or behind a reverse proxy with authentication.

Step 3: Pull and Run Models with Ollama

Pull a small, fast model to test your setup. For general chat, use ollama pull llama3.2:3b. For coding tasks, try ollama pull qwen2.5-coder:7b or a quantized variant like :q4_0 for lower memory usage. Run an interactive session with ollama run llama3.2 and type your prompt. To generate from the shell, try echo "Explain RAID levels simply" | ollama run llama3.2. Ollama will use the GPU automatically if supported; otherwise it falls back to CPU. Tune performance with environment variables such as OLLAMA_NUM_PARALLEL=1 to reduce memory pressure and OLLAMA_KV_SIZE=512 for larger context windows when your memory allows.

Step 4: Deploy Open WebUI with Docker

Open WebUI provides a clean web interface and multi-model support. If Docker is not installed, add it with sudo apt-get update && sudo apt-get install -y docker.io and ensure it runs at startup with sudo systemctl enable --now docker. Launch Open WebUI connected to Ollama using docker run -d --name open-webui -p 3000:8080 -e OLLAMA_BASE_URL=http://localhost:11434 -v open-webui:/app/backend/data -v /var/lib/ollama:/root/.ollama --restart unless-stopped ghcr.io/open-webui/open-webui:latest. If Open WebUI runs on a different host from Ollama, set OLLAMA_BASE_URL to the Ollama server’s IP, for example http://192.168.1.50:11434. For GPU inside the container, add --gpus all and make sure the NVIDIA container toolkit is configured.

Step 5: Secure Access with a Reverse Proxy and HTTPS

If you plan to reach the interface over the internet, place Open WebUI behind a reverse proxy with TLS and authentication. A simple option is Caddy, which can obtain and renew certificates automatically. For example, you can point a domain to your server and configure Caddy to proxy yourdomain.com to localhost:3000 and enable basic auth. With Nginx, use an SSL server block, set proxy_pass http://127.0.0.1:3000, and enable rate limiting and headers like X-Frame-Options and Content-Security-Policy. Always avoid exposing the raw Ollama port unless you fully trust the network.

Step 6: Updates, Backups, and Autostart

Update Ollama by rerunning the installer or using your package manager if you installed via a repo. To update Open WebUI, pull the latest image with docker pull ghcr.io/open-webui/open-webui:latest and restart the container. Persist your data by backing up /var/lib/ollama and the Docker volume open-webui. Both Ollama and Docker containers start automatically on boot, but you can confirm with systemctl is-enabled ollama and the container’s --restart unless-stopped flag.

API Quick Test

You can call Ollama’s local API directly. After pulling a model, try curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Give me three bullet points about containers"}'. This is useful for integrating local LLMs into scripts, chatbots, or development tools without sending data to third parties.

Troubleshooting

If model loading fails with “no space left on device,” free disk space with df -h, remove unused Docker images with docker system prune -a, or delete old models in /var/lib/ollama. If nvidia-smi returns an error, reinstall the driver and ensure Secure Boot is either disabled or configured with signed modules. If port 11434 or 3000 is already in use, change the binding (for example OLLAMA_HOST=0.0.0.0:11435) or stop the conflicting process. On low-memory hosts, choose smaller or more heavily quantized models (for example :q4_0), reduce parallel requests with OLLAMA_NUM_PARALLEL=1, and close other memory-hungry services.

What You Achieved

You now have a private, production-ready local AI stack on Ubuntu. Ollama runs the model backend with optional GPU acceleration, while Open WebUI delivers a modern chat interface. With a reverse proxy and backups in place, you can confidently use local LLMs for coding assistance, content drafting, documentation, and experimentation without sending your data to the cloud.

Running large language models locally is now practical and fast, especially with GPU acceleration. In this tutorial, you will deploy Ollama and Open WebUI on Ubuntu 22.04/24.04 using Docker Compose. This stack gives you a private, browser-based interface for modern LLMs (Llama, Mistral, Phi, etc.) with one-click model management and secure, self-hosted inference.

Why this stack?

Ollama simplifies downloading, quantizing, and serving LLMs on your machine. Open WebUI adds a clean chat interface, prompt templates, file uploads, and multi-user access. Together, they provide a robust local AI setup that is easy to update and portable across servers.

Prerequisites

- Ubuntu Server 22.04 or 24.04 (fresh system recommended)

- An NVIDIA GPU with recent drivers (T4, RTX 20/30/40, A-series, etc.)

- sudo access and an internet connection

- Optional: a domain name for HTTPS (e.g., ai.example.com)

Step 1 — Install Docker Engine and Compose

Install Docker from the official repository to ensure up-to-date features like GPU support in Docker Compose.

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker
docker --version
docker compose version

Step 2 — Enable GPU with NVIDIA Container Toolkit

Install the NVIDIA Container Toolkit to pass the GPU into containers. Verify that the host can see the GPU with nvidia-smi before proceeding.

# If you don't have drivers:
# sudo ubuntu-drivers install && sudo reboot

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit

# Configure Docker to use the NVIDIA runtime
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Sanity check
nvidia-smi

Step 3 — Create the Docker Compose stack

We will run two services: Ollama (backend API on port 11434) and Open WebUI (frontend on port 3000) connected via a Docker network. The compose file also enables GPU support for Ollama.

mkdir -p ~/ollama-openwebui && cd ~/ollama-openwebui
cat > docker-compose.yml <<'YAML'
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    gpus: all
    environment:
      - OLLAMA_KEEP_ALIVE=1h
      - OLLAMA_HOST=0.0.0.0

  openwebui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: openwebui
    restart: unless-stopped
    depends_on:
      - ollama
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_AUTH=True
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:
YAML

Step 4 — Launch and access Open WebUI

Start the stack and watch logs for any errors. The first launch will pull images.

docker compose up -d
docker compose logs -f --tail=100

Open your browser to http://SERVER_IP:3000. Create the first admin user when prompted. Open WebUI will automatically detect Ollama via the internal URL and list available models.

Step 5 — Pull a model and test

Use either the WebUI model manager or the CLI to fetch models. The example below pulls a popular 7B model.

# Pull from the host (proxies into the container)
docker exec -it ollama ollama pull llama3.1:8b

# Quick API smoke test
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Say hello from a local LLM.",
  "stream": false
}'

In Open WebUI, select the model from the dropdown and start chatting. If you have enough VRAM, consider quantized larger models (e.g., 13B/70B Q4/Q5) for better reasoning.

Optional — Secure with a Caddy reverse proxy and HTTPS

If you have a domain, use Caddy to obtain and renew TLS automatically. This example exposes Open WebUI securely on port 443 and keeps Ollama private.

sudo apt install -y caddy
sudo tee /etc/caddy/Caddyfile >/dev/null <<'CADDY'
ai.example.com {
  encode zstd gzip
  reverse_proxy 127.0.0.1:3000
}
CADDY
sudo systemctl reload caddy

Point your DNS A/AAAA record to the server. Then visit https://ai.example.com. For teams, enable WebUI auth (already set) and create users from the admin settings.

Back up and update

To back up your models and chats, save the named volumes. You can also snapshot the folders from the host.

# Export volumes to tarballs
docker run --rm -v ollama:/v -v $(pwd):/b busybox tar czf /b/ollama-vol.tgz -C /v .
docker run --rm -v openwebui:/v -v $(pwd):/b busybox tar czf /b/openwebui-vol.tgz -C /v .

# Update images safely
docker compose pull
docker compose up -d

Troubleshooting

- No GPU detected: Ensure nvidia-smi works on the host. Re-run nvidia-ctk runtime configure, restart Docker, and verify the container sees the GPU:

docker exec -it ollama bash -lc 'nvidia-smi || ls -l /dev/nvidia*'

- Slow generation: Use quantized models (Q4_K_M/Q5_K_M), avoid oversize context windows, and confirm GPU is actually used (GPU utilization should rise in nvidia-smi during inference).

- Port conflicts: Change mapped ports in docker-compose.yml, e.g., "3001:8080" for Open WebUI or put a reverse proxy in front.

- Permission errors on volumes: Ensure your user is in the docker group and that the Docker daemon can write to the volume paths.

Security tips

- Keep Ollama bound to the internal network and only expose Open WebUI through TLS.

- Enable authentication (already set via WEBUI_AUTH=True). Use strong passwords and consider putting Open WebUI behind a VPN or SSO.

- Restrict firewall ports using UFW: allow 22/tcp and 443/tcp, then deny others.

Conclusion

You now have a GPU-accelerated, private AI stack with Ollama and Open WebUI on Ubuntu, orchestrated by Docker Compose. It is easy to upgrade, portable across servers, and suitable for personal research or team deployments. With this foundation, you can iterate quickly, evaluate new models as they drop, and keep your data fully on-prem.

Overview

This step-by-step guide shows you how to deploy Ollama and Open WebUI on Ubuntu 24.04 with NVIDIA GPU acceleration using Docker. With this setup, you can run modern large language models (LLMs) locally, manage them from a clean web interface, and take full advantage of your GPU for high performance. The process covers NVIDIA drivers, Docker, the NVIDIA Container Toolkit, and secure, persistent containers that survive reboots.

What You Will Need

You need a 64-bit Ubuntu 24.04 host with an NVIDIA GPU (Turing or newer recommended), Internet access, a user with sudo rights, and at least 20 GB of free disk space for models. If you are working on a remote server, make sure port 3000 (for Open WebUI) and 11434 (for Ollama) are reachable or routed through a reverse proxy.

1) Install NVIDIA Drivers

First, install the official NVIDIA driver so CUDA can talk to your GPU. Run: sudo ubuntu-drivers autoinstall. When it finishes, reboot with sudo reboot. After the reboot, verify the GPU is visible: nvidia-smi. You should see your GPU name and driver version. If you do not, confirm Secure Boot is disabled or enroll the driver MOK accordingly, then repeat the check.

2) Install Docker Engine on Ubuntu 24.04

Set up Docker from the official repository for best stability and features. Run: sudo apt update && sudo apt install -y ca-certificates curl gnupg. Add Docker’s key and repo: sudo install -m 0755 -d /etc/apt/keyrings, curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg, echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu noble stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null. Then install: sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin. To run Docker without sudo: sudo usermod -aG docker $USER then newgrp docker.

3) Enable GPU Access in Containers (NVIDIA Container Toolkit)

Install the NVIDIA Container Toolkit so Docker can pass your GPU into containers. Add the key and repo: curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg, curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list. Install and configure: sudo apt update && sudo apt install -y nvidia-container-toolkit, sudo nvidia-ctk runtime configure --runtime=docker, sudo systemctl restart docker. Test GPU passthrough: docker run --rm --gpus all nvidia/cuda:12.6.2-base-ubuntu22.04 nvidia-smi. You should see your GPU listed inside the container.

4) Create a Dedicated Network for AI Services

Create a user-defined Docker network so containers can discover each other cleanly: docker network create ai. This network isolates traffic and lets Open WebUI talk to the Ollama container by name.

5) Run the Ollama Container with GPU Support

Start Ollama and persist its model data in a Docker volume. Run: docker run -d --name ollama --gpus all --restart unless-stopped -p 11434:11434 -v ollama:/root/.ollama --network ai ollama/ollama:latest. The container exposes the Ollama API on port 11434. Check logs with docker logs -f ollama to ensure the server starts without errors.

6) Pull a Model (Llama 3.1 example)

Use Ollama’s CLI inside the container to download a model. For a great balance of speed and quality on consumer GPUs, try an 8B model: docker exec -it ollama ollama pull llama3.1:8b. If you have a smaller GPU (e.g., 6–8 GB VRAM), try a quantized variant like llama3.1:8b-instruct-q4_K_M. You can list models with docker exec -it ollama ollama list.

7) Deploy Open WebUI and Connect to Ollama

Open WebUI provides a friendly interface to chat with models, manage prompts, and configure settings. Start it with: docker run -d --name open-webui --restart unless-stopped -p 3000:8080 -e OLLAMA_API_BASE_URL=http://ollama:11434 -v openwebui:/app/backend/data --network ai ghcr.io/open-webui/open-webui:latest. Open http://<your_server_ip>:3000 in a browser, create your first user (the first account becomes admin), and pick the model you pulled in the previous step. You can now chat with the LLM directly from your browser.

8) Optional: Secure Access with HTTPS

For Internet-facing servers, place a reverse proxy with TLS in front of Open WebUI. A simple approach is Caddy or Nginx Proxy Manager. Point your domain’s DNS to the server, terminate HTTPS on the proxy, and forward to localhost:3000. If you already use Traefik or Nginx, add routes with Let’s Encrypt certificates and restrict access using basic auth or OAuth.

Maintenance and Updates

To update Ollama or Open WebUI, pull new images and recreate containers. Run: docker pull ollama/ollama:latest and docker pull ghcr.io/open-webui/open-webui:latest, then docker stop ollama open-webui and docker rm ollama open-webui. Start them again using the same docker run commands; your data persists in the volumes ollama and openwebui. To back up models and settings, archive the volumes: sudo tar -czf ollama-vol.tgz -C /var/lib/docker/volumes/ollama/_data . and sudo tar -czf openwebui-vol.tgz -C /var/lib/docker/volumes/openwebui/_data ..

Troubleshooting

If the GPU is not detected inside containers, confirm the host driver works with nvidia-smi. Then verify the runtime is configured: docker info | grep -i nvidia. If missing, re-run sudo nvidia-ctk runtime configure --runtime=docker and sudo systemctl restart docker. For permission errors when running Docker, add your user to the docker group as shown above. If downloads are slow or models fail due to VRAM limits, choose smaller or quantized models (e.g., q4_K_M or q5_K_M).

What You Get

After following these steps, you have a modern, GPU-accelerated local AI stack. Ollama handles efficient model runtimes, and Open WebUI gives you a clean chat interface, prompt management, and multi-model control. Because everything runs in Docker with persistent volumes, updates and backups are easy, and you can scale this setup on a workstation or a headless server with minimal changes.

This step-by-step guide shows you how to run a fast, private, and local AI chatbot on Ubuntu 22.04 or 24.04 using Ollama and Open WebUI. You will install the Ollama runtime, pull a modern large language model, and add a clean chat interface via Open WebUI in Docker. Optional steps cover NVIDIA GPU acceleration, API usage, and persistence. The result is a secure, offline-friendly setup suitable for helpdesk, coding assistance, or knowledge base querying without sending data to the cloud.

Why Ollama + Open WebUI

Ollama makes it simple to run and manage open-source LLMs locally (Llama 3.x, Mistral, Phi, Qwen, and more). Open WebUI adds a user-friendly, browser-based chat interface with conversation history, prompt templates, and multi-model support. Together they form a robust, low-maintenance local AI stack for Linux desktops and servers.

Prerequisites

- Ubuntu 22.04 or 24.04 with a non-root sudo user.
- At least 8 GB RAM (16 GB recommended for larger models).
- Optional NVIDIA GPU for acceleration (T4/RTX/RTX A-series, etc.).
- Internet access to download models and containers.

Step 1 — Install Ollama

1) Update packages:
sudo apt update && sudo apt install -y curl ca-certificates
2) Install Ollama:
curl -fsSL https://ollama.com/install.sh | sh
3) Enable as a service:
sudo systemctl enable --now ollama
4) Verify the API is up:
curl http://localhost:11434/api/tags
If you see JSON, Ollama is running correctly.

Step 2 — Pull and test a model

Pull a compact, capable model first to validate your setup:
ollama pull llama3.2
Run an interactive test:
ollama run llama3.2
Type a prompt, then press Ctrl+C to exit. You can later try larger models (for example, ollama pull mistral or ollama pull llama3.1), but start small to confirm everything works.

Step 3 — Install Docker Engine

1) Add Docker’s repo key and source:
sudo apt-get update && sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release; echo $UBUNTU_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
2) Install Docker and the Compose plugin:
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
3) Add your user to the docker group and refresh your shell:
sudo usermod -aG docker $USER
newgrp docker

Step 4 — Run Open WebUI connected to Ollama

Start Open WebUI and point it to the Ollama API on the host. The --add-host flag maps host.docker.internal to your host’s gateway so the container can reach http://localhost:11434 on the host:

docker run -d --name open-webui --restart=unless-stopped -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main

Open your browser to http://SERVER_IP:3000 (or http://localhost:3000). Create the first admin account, choose a model (for example, llama3.2), and start chatting.

Step 5 — Enable NVIDIA GPU acceleration (optional)

1) Install the latest NVIDIA driver for your GPU using Ubuntu’s Additional Drivers or apt. Reboot if prompted.
2) Install the NVIDIA Container Toolkit so Docker can access the GPU:
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
3) Recreate Open WebUI with GPU access:
docker rm -f open-webui
docker run -d --name open-webui --restart=unless-stopped --gpus all -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main
4) Ollama will also use the GPU automatically when a compatible model is loaded. You can confirm GPU use with nvidia-smi during inference.

Step 6 — Use the Ollama HTTP API

You can script local inference via HTTP without the UI. Example generation request:
curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Write a haiku about backups."}'
Chat format with memory:
curl http://localhost:11434/api/chat -d '{"model":"llama3.2","messages":[{"role":"user","content":"Explain DNS in one sentence."}]}'

Step 7 — Persistence, autostart, and updates

- Ollama models are stored under ~/.ollama/models. Back up this directory to avoid re-downloading models.
- The Open WebUI container uses a named volume (open-webui) for its data, which persists across restarts.
- Ollama is already set to start at boot (systemctl enable ollama). The WebUI container uses --restart=unless-stopped so it will auto-start after a reboot.
- Update Ollama: curl -fsSL https://ollama.com/install.sh | sh
- Update Open WebUI: docker pull ghcr.io/open-webui/open-webui:main && docker restart open-webui

Troubleshooting

- Open WebUI cannot connect to Ollama: ensure you used --add-host=host.docker.internal:host-gateway and that curl http://localhost:11434/api/tags works on the host.
- Port already in use: change -p 3000:8080 to a different host port like -p 3333:8080.
- Out of memory or slow responses: try a smaller model (for example, llama3.2 or phi3). Close other apps or add swap. For CPU-only hosts, expect slower performance on large models.
- GPU not used: verify drivers, nvidia-smi, and that the container runs with --gpus all. Pull a GPU-optimized model variant if available.

What you can do next

- Connect knowledge bases or documents using Open WebUI’s RAG features to power local search over PDFs and wikis.
- Add multiple models and switch per chat, benchmarking speed and quality.
- Put Nginx or Caddy in front of :3000 for HTTPS and trusted network access.
- Automate prompts with shell scripts or Python by calling the local Ollama API.

You now have a private, local AI assistant on Ubuntu with a clean web interface, GPU-ready acceleration, and a stable upgrade path—all without sending your data to third-party services.

Overview

This tutorial shows how to self-host large language models locally by deploying Ollama and Open WebUI on Ubuntu 24.04 LTS with NVIDIA GPU acceleration using Docker Compose. Ollama handles model runtimes and downloads, while Open WebUI provides a friendly web interface, prompt management, and multi-user features. By the end, you will have a reproducible, GPU-enabled AI stack reachable from a browser on your LAN.

Prerequisites

Before you start, confirm: (1) Ubuntu 24.04 LTS installed and updated, (2) An NVIDIA GPU supported by recent drivers, (3) Administrative (sudo) access, and (4) Internet connectivity. If Secure Boot is enabled, you may need to enroll the NVIDIA kernel module signing key during driver installation.

Step 1: Prepare Ubuntu

sudo apt update && sudo apt -y upgrade
sudo apt -y install curl git ca-certificates gnupg

Step 2: Install the NVIDIA Driver

Use Ubuntu’s built-in tool to select the correct, current driver:

ubuntu-drivers list
sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, verify the GPU is recognized:

nvidia-smi

Step 3: Install Docker Engine and Compose

Add the official Docker repository and install the engine plus the Compose plugin:

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Step 4: Install NVIDIA Container Toolkit

This toolkit lets Docker containers access the GPU:

curl -fsSL https://nvidia.github.io/nvidia-container-toolkit/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/nvidia-container-toolkit/$distribution/nvidia-container-toolkit.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt -y install nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test GPU access in a container:

docker run --rm --gpus all nvidia/cuda:12.4.1-runtime-ubuntu22.04 nvidia-smi

Step 5: Create the Docker Compose project

Make a working directory and create your Compose file:

mkdir -p ~/ai-stack && cd ~/ai-stack

Create a file named docker-compose.yaml with the following content (indentation matters):

version: "3.9"
services:
ollama:
image: ollama/ollama:latest
restart: unless-stopped
ports:
- "11434:11434"
environment:
- OLLAMA_HOST=0.0.0.0
volumes:
- ollama:/root/.ollama
gpus: all

openwebui:
image: ghcr.io/open-webui/open-webui:latest
restart: unless-stopped
depends_on:
- ollama
ports:
- "3000:8080"
environment:
- OLLAMA_BASE_URL=http://ollama:11434
- WEBUI_NAME=MyLocalAI
volumes:
- openwebui:/app/backend/data

volumes:
ollama:
openwebui:

Step 6: Launch the stack

docker compose up -d

Wait a few seconds. Visit http://localhost:3000 (or http://SERVER_IP:3000) to open Open WebUI. The first user usually becomes the admin. You can manage models from the UI or the CLI.

Step 7: Pull a model and test

Pull a model into the Ollama volume (example: Meta’s Llama 3.1 8B):

docker compose exec ollama ollama pull llama3.1

Generate a quick response from the API:

curl -s http://localhost:11434/api/generate -d '{"model":"llama3.1","prompt":"Say hello from a local GPU!","stream":false}' | jq .response

In Open WebUI, choose the model from the dropdown and start chatting.

Step 8: Use the OpenAI-compatible API

Ollama exposes an OpenAI-style API under /v1. Example with Python’s OpenAI SDK:

pip install openai
python - <<'PY'
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
resp = client.chat.completions.create(
model="llama3.1",
messages=[{"role":"user","content":"Give me a one-line fun fact."}]
)
print(resp.choices[0].message.content)
PY

Maintenance and updates

To update images without losing your data or models stored in volumes, run:

cd ~/ai-stack
docker compose pull
docker compose up -d

To update or add models, use the UI or the CLI, for example: docker compose exec ollama ollama pull mistral.

Troubleshooting

nvidia-smi fails inside containers: Ensure the NVIDIA driver is installed and matches your GPU. Reboot after installation. Confirm Docker sees the GPU with docker run --rm --gpus all nvidia/cuda:12.4.1-runtime-ubuntu22.04 nvidia-smi.

Compose “gpus” not recognized: Check your Compose plugin version with docker compose version. Update Docker packages if outdated. As a fallback, configure the NVIDIA runtime as default and remove the gpus key: sudo nvidia-ctk runtime configure --runtime=docker --set-as-default && sudo systemctl restart docker.

Slow downloads or OOM: Models are large; use a fast, stable network and ensure enough VRAM and RAM. If a model does not fit your GPU, choose a smaller variant (e.g., llama3.1:8b or a quantized build like q4_K_M).

Security tips

Bind the services to your LAN or localhost by default and place them behind a reverse proxy with TLS if exposing over the internet. In Open WebUI, enable authentication and restrict new user registration if not needed. Consider a firewall rule to limit access to ports 11434 and 3000.

Remove the stack

To stop the containers while preserving data: docker compose down. To remove everything, including downloaded models and chat history: docker compose down -v.

You now have a modern, GPU-accelerated local AI stack that is easy to manage and update. The same approach works for additional services like vector databases or reverse proxies, making it a flexible foundation for on-prem AI experiments and production prototypes.

Overview

This step-by-step guide shows how to deploy a private, local AI stack on Ubuntu using Docker: Ollama for running large language models (LLMs) and Open WebUI as a fast, friendly chat interface. The setup works on CPUs and supports NVIDIA GPUs for acceleration. You will get a secure, self-hosted environment where you can run models like Llama 3.2, Phi-4, and Mistral without sending data to the cloud.

Prerequisites

- Ubuntu 22.04 or 24.04 (server or desktop)
- 16 GB RAM recommended (more for larger models), 30+ GB free disk
- Docker Engine and the Docker Compose plugin
- Optional: NVIDIA GPU with proprietary driver installed (e.g., 535+)

1) Install Docker and Docker Compose

Update your system and install Docker from the official repository for best stability and performance.

sudo apt update && sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER && newgrp docker

2) (Optional) Enable NVIDIA GPU in Containers

If you have an NVIDIA GPU, install the NVIDIA Container Toolkit so Docker can pass the GPU into containers:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify your driver with nvidia-smi. The container will get GPU access when you run it with --gpus all.

3) Create a Docker Network and Volumes

Create a dedicated network so services can talk by name and set up persistent storage:

docker network create llmnet
docker volume create ollama
docker volume create open-webui

4) Run the Ollama Container

Start Ollama. For CPU-only:

docker run -d --name ollama --restart=unless-stopped --network llmnet -p 11434:11434 -v ollama:/root/.ollama ollama/ollama:latest

With NVIDIA GPU acceleration (detected automatically):

docker run -d --name ollama --restart=unless-stopped --network llmnet --gpus all -p 11434:11434 -v ollama:/root/.ollama ollama/ollama:latest

5) Pull a Model

You can manage models from the host via docker exec. Pull a lightweight model to start quickly:

docker exec -it ollama ollama pull llama3.2:3b

Test generation from the command line:

curl http://localhost:11434/api/generate -d '{"model":"llama3.2:3b","prompt":"Say hello in one short line."}'

6) Launch Open WebUI

Open WebUI provides a clean chat interface and model manager. Start it on port 3000 and point it to the Ollama endpoint:

docker run -d --name open-webui --restart=unless-stopped --network llmnet -p 3000:8080 -e OLLAMA_BASE_URL=http://ollama:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:latest

Open a browser and visit http://localhost:3000 (or your server IP). Create the first admin account, select a model (e.g., llama3.2:3b), and start chatting. If a model is missing, Open WebUI can pull it automatically via Ollama.

7) Optional: Use Docker Compose

Prefer to keep everything in a single file? Create docker-compose.yml in an empty folder and paste:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
     - "11434:11434"
    networks: [llmnet]
    volumes:
     - ollama:/root/.ollama
  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    ports:
     - "3000:8080"
    environment:
     - OLLAMA_BASE_URL=http://ollama:11434
    networks: [llmnet]
    volumes:
     - open-webui:/app/backend/data
networks:
  llmnet:
    external: true
volumes:
  ollama:
  open-webui:

Start with docker compose up -d. For GPU, prefer the docker run method with --gpus all, or adapt your Compose file using a GPU-capable configuration on your system.

8) Securing and Updating

- Restrict access: if running on a server, firewall ports 11434 and 3000 to trusted IPs.
- Reverse proxy: place Nginx or Caddy in front with HTTPS for remote access.
- Updates: pull newer images and recreate containers: docker pull ollama/ollama:latest && docker pull ghcr.io/open-webui/open-webui:latest, then docker stop and docker rm containers and re-run them. Your data persists in the volumes.

9) Troubleshooting

- Check logs: docker logs -f ollama and docker logs -f open-webui.
- Port in use: change published ports (e.g., -p 3001:8080).
- GPU not detected: validate nvidia-smi, reinstall the NVIDIA Container Toolkit, and ensure --gpus all is present.
- Disk space: models are large; prune unused data with docker system prune and remove models in ollama volume if needed.

10) Quick API and CLI Examples

- Pull another model: docker exec -it ollama ollama pull phi4:latest
- Chat from CLI: docker exec -it ollama ollama run mistral:7b
- Simple REST call: curl http://localhost:11434/api/generate -d '{"model":"phi4:latest","prompt":"Give me two bullet points about container security."}'

You now have a modern, private AI stack using Docker, Ollama, and Open WebUI on Ubuntu. It is fast, flexible, and ready for local development, internal knowledge assistants, and offline experimentation—no cloud required.

Local large language models (LLMs) are now practical for developers, researchers, and privacy-focused teams. In this step-by-step guide, you will install and run Ollama (LLM runtime) and Open WebUI (a modern chat interface) on Ubuntu 22.04/24.04, with optional NVIDIA GPU acceleration. The setup uses Docker for easy updates, isolation, and backups.

Why this stack?

Ollama makes downloading and running models simple, offering a fast API on your machine. Open WebUI provides a sleek, extensible web app for chatting with multiple models, managing prompts, and moderating access. Together, they create a private, cost-effective alternative to cloud AI services.

Prerequisites

You need an Ubuntu 22.04/24.04 system with internet access and a user with sudo rights. If you have an NVIDIA GPU (recommended), you can enable GPU acceleration for much faster inference. CPU-only works too—skip the GPU steps if you do not have a supported GPU.

Step 1: Update the system

Update packages to ensure you have the latest dependencies and security fixes.

sudo apt update && sudo apt -y upgrade
sudo reboot

Step 2 (Optional): Enable NVIDIA GPU support

Install the latest proprietary NVIDIA driver and the container toolkit so Docker can use your GPU. Reboot when asked.

# Install recommended NVIDIA driver
sudo ubuntu-drivers install
sudo reboot

# Install NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list > /dev/null

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Verify GPU visibility
nvidia-smi

Step 3: Install Docker Engine and Docker Compose plugin

Install Docker from the official repository to get the latest stable version. Add your user to the docker group to run Docker without sudo.

sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker $USER
newgrp docker

# Quick test
docker run --rm hello-world

Step 4: Create a Docker Compose file for Ollama + Open WebUI

Create a working directory and define services. The configuration below enables GPU when present; for CPU-only, remove the gpus: all line under the Ollama service.

mkdir -p ~/local-llm && cd ~/local-llm
nano docker-compose.yml

services:
  ollama:
    container_name: ollama
    image: ollama/ollama:latest
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    # Comment the next line if you are CPU-only
    gpus: all
    environment:
      - OLLAMA_KEEP_ALIVE=24h

  open-webui:
    container_name: open-webui
    image: ghcr.io/open-webui/open-webui:latest
    depends_on:
      - ollama
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=change_this_long_random_secret
    volumes:
      - openwebui:/app/backend/data

volumes:
  ollama:
  openwebui:

Step 5: Start the stack

Bring the services up in the background, then check their status. The Open WebUI will be available at http://SERVER_IP:3000 and the Ollama API at http://SERVER_IP:11434.

docker compose up -d
docker compose ps

Step 6: Pull and run a model

Use Ollama to pull an LLM. You can pick models like llama3.2, mistral, or a coding model. The first pull downloads model weights, which can be several GB.

# Pull a general-purpose model
docker exec -it ollama ollama pull llama3.2

# Test it via CLI
docker exec -it ollama ollama run llama3.2 "Write a two-sentence summary of Ubuntu."

# Or use the API
curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Hello!"}'

Open your browser to http://SERVER_IP:3000, select the model from the dropdown, and start chatting. In Settings, you can change default models, system prompts, and appearance.

Step 7: Secure access

If exposing the WebUI beyond your LAN, add authentication. In Open WebUI, create an admin user at first login, then disable new signups in Settings. For internet exposure, place a reverse proxy (Nginx, Caddy, or Traefik) with HTTPS (Let’s Encrypt) in front of port 3000.

Step 8: Update and backup

To update, pull the latest images and recreate containers without losing data stored in volumes. To back up, save the volumes before upgrades.

# Update images
docker compose pull
docker compose up -d

# Backup volumes (example)
docker run --rm -v local-llm_ollama:/data -v "$PWD":/backup \
  busybox tar czf /backup/ollama-vol.tgz /data

docker run --rm -v local-llm_openwebui:/data -v "$PWD":/backup \
  busybox tar czf /backup/openwebui-vol.tgz /data

Troubleshooting

GPU not used: Run docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi. If it fails, recheck the driver and NVIDIA Container Toolkit steps. Ensure the gpus: all line is present and Docker was restarted.

Permission denied with Docker: You may need to log out and back in after adding your user to the docker group, or run newgrp docker.

Port conflicts: Change the left side of port mappings in docker-compose.yml (e.g., use "8081:8080" for WebUI) and restart.

Slow or failed model pull: Verify disk space and retry. Large models require several GB of free space in the ollama volume.

Uninstall (optional)

To stop and remove everything, run:

cd ~/local-llm
docker compose down
docker volume rm local-llm_ollama local-llm_openwebui

Wrap-up

You have a modern local AI stack: Ollama for fast model serving and Open WebUI for a friendly, multi-model chat interface. With Docker, updates are quick and backups are simple. Add your favorite models, tune system prompts, and integrate the Ollama API into your apps—all without sending data to the cloud.

Overview

This guide shows how to self-host a private AI chatbot with Open WebUI (a clean, ChatGPT-like interface) and Ollama (for running local large language models) on Ubuntu 22.04 or 24.04. Everything runs in Docker, secured with HTTPS via Caddy and optional Basic Auth. If you have an NVIDIA GPU, you can enable GPU acceleration to speed up model inference dramatically.

What you will need

- An Ubuntu 22.04/24.04 server with at least 8 GB RAM and 20 GB free disk space. For GPU acceleration, an NVIDIA GPU with recent drivers is recommended (e.g., 8 GB VRAM or more for larger models).

- A domain name pointing to your server’s public IP (A/AAAA record). Ports 80 and 443 should be open to the internet for Let’s Encrypt.

- A non-root user with sudo privileges.

Step 1 — Install Docker and Docker Compose plugin

Update your system and install Docker from the official repository:

sudo apt update && sudo apt install -y ca-certificates curl gnupg

sudo install -m 0755 -d /etc/apt/keyrings

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker $USER && newgrp docker

Step 2 — (Optional) Enable NVIDIA GPU for containers

Install the NVIDIA driver (if not already installed) and the NVIDIA container toolkit so Docker can access your GPU.

sudo ubuntu-drivers install (or choose a specific driver, e.g., sudo apt install -y nvidia-driver-535)

sudo reboot

Install the container toolkit:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit.gpg] https://#' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker

sudo systemctl restart docker

Test GPU access:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Step 3 — Prepare Docker Compose and Caddy

Create a project folder and move into it:

mkdir -p ~/ai-stack && cd ~/ai-stack

Create a file named docker-compose.yml with the following content (replace your.domain.com later in Caddyfile):

services:
ollama:
image: ollama/ollama:latest
container_name: ollama
volumes:
- ollama:/root/.ollama
environment:
- OLLAMA_KEEP_ALIVE=2h
ports:
- "127.0.0.1:11434:11434"
restart: unless-stopped
# Uncomment the next line if you enabled NVIDIA toolkit
# gpus: all

openwebui:
image: ghcr.io/open-webui/open-webui:main
container_name: openwebui
depends_on:
- ollama
environment:
- OLLAMA_BASE_URL=http://ollama:11434
- WEBUI_AUTH=True
ports:
- "127.0.0.1:8080:8080"
volumes:
- openwebui:/app/backend/data
restart: unless-stopped

caddy:
image: caddy:2
container_name: caddy
depends_on:
- openwebui
restart: unless-stopped
ports:
- "80:80"
- "443:443"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- caddy_data:/data
- caddy_config:/config

volumes:
ollama:
openwebui:
caddy_data:
caddy_config:

Create a file named Caddyfile in the same folder. Replace your.domain.com with your real domain and the email with yours:

your.domain.com {
encode zstd gzip
tls [email protected]
# Optional Basic Auth — generate a hashed password below and uncomment
# basicauth {
# admin <paste_hashed_password_here>
# }
reverse_proxy openwebui:8080
}

If you want Basic Auth, generate a hash:

docker run --rm caddy:2 caddy hash-password --plaintext "StrongPassword!"

Copy the hash output, paste it into the Caddyfile under basicauth, and uncomment the lines.

Step 4 — Start the stack and pull a model

Start the services:

docker compose up -d

Pull a model with Ollama. Llama 3.1 is a great default; you can also choose smaller variants if you have less VRAM:

docker exec -it ollama ollama pull llama3.1

For low VRAM systems, try a quantized build like llama3.1:8b-instruct-q4_0 or a compact model like mistral:7b-instruct:

docker exec -it ollama ollama pull mistral:latest

Verify Ollama is up:

curl -s http://127.0.0.1:11434/api/tags

Step 5 — Access Open WebUI over HTTPS

Wait 30–60 seconds for Caddy to obtain a Let’s Encrypt certificate. Then browse to https://your.domain.com. On the first visit, create your Open WebUI admin user. In Settings > Models, select the model you pulled with Ollama. You can now chat privately with your local LLM through a friendly web interface.

Step 6 — Security hardening (recommended)

- Keep Open WebUI behind Caddy only. We already published it on localhost (127.0.0.1) to prevent direct exposure.

- Enable Basic Auth in your Caddyfile if you plan to expose the site to the open internet. Use a long, unique password.

- Restrict admin features in Open WebUI to your own account. Disable public sign-ups if you do not need them.

- Consider a firewall rule to allow inbound 80/443 only, and block 8080/11434 from the WAN.

Step 7 — Backups and updates

Back up Open WebUI data:

docker run --rm -v openwebui:/d -v $PWD:/b busybox tar czf /b/openwebui-backup.tgz -C /d .

Back up Ollama models (can be large):

docker run --rm -v ollama:/d -v $PWD:/b busybox tar czf /b/ollama-backup.tgz -C /d .

To update containers:

docker compose pull && docker compose up -d

To remove old images:

docker image prune -f

Troubleshooting

- Check logs if something fails to start: docker compose logs -f

- Verify DNS and port 80/443 reach the server; Let’s Encrypt must connect over HTTP/HTTPS the first time.

- If certificates fail, restart the stack after DNS propagates: docker compose down && docker compose up -d

- If the GPU is not detected, confirm nvidia-smi works on the host and that you added gpus: all under the Ollama service.

- Test the Ollama API locally: curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.1","prompt":"hi"}'

Where to go next

Explore model variants optimized for your hardware (Q4 for low VRAM, Q6/Q8 for higher quality, FP16 on strong GPUs). Add embeddings and RAG features in Open WebUI to chat over your documents. With this setup, you keep your data and traffic on your own server, with clean HTTPS, optional password protection, and fast local inference.

Overview

This step-by-step guide shows how to run open-source large language models (LLMs) locally on Ubuntu 24.04 using Ollama for model serving and Open WebUI for a friendly chat interface. You will install Ollama, enable optional GPU acceleration (NVIDIA or CPU fallback), and deploy Open WebUI with Docker. The result is a private, fast, and controllable AI setup suitable for home labs and small teams.

Prerequisites

You need an Ubuntu 24.04 LTS host with internet access, a user with sudo rights, and at least 8 GB of RAM. A modern NVIDIA GPU is optional but recommended for faster inference. Make sure the system is up to date: sudo apt update && sudo apt -y upgrade

Step 1 — Install Ollama

Ollama is a lightweight server that downloads and runs models locally. Install it with the official script:
curl -fsSL https://ollama.com/install.sh | sh

Enable and start the service so it runs at boot:
sudo systemctl enable ollama
sudo systemctl start ollama
sudo systemctl status ollama

Verify the API is listening on port 11434:
curl http://127.0.0.1:11434/api/tags

Step 2 — Optional: Enable GPU Acceleration (NVIDIA)

If you have an NVIDIA GPU, install the recommended driver. Ubuntu makes this easy:
sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, confirm the driver is active:
nvidia-smi

Ollama detects GPUs automatically when drivers are present. No extra flags are required. If you need to force CPU or GPU behavior, you can set:
export OLLAMA_NO_GPU=1 (CPU only) or export OLLAMA_NO_GPU=0 (GPU allowed). For a persistent setting, add the variable to your shell profile and restart Ollama:
sudo systemctl restart ollama

AMD GPUs can work with ROCm on supported cards and drivers. If you are using AMD, install the ROCm runtime from AMD’s repository for Ubuntu 24.04, confirm with rocminfo, and ensure your user is in the video and render groups. If ROCm is not available for your hardware, Ollama will fall back to CPU.

Step 3 — Pull a Model and Test Locally

Pull a well-supported model. Llama 3 is a popular choice:
ollama pull llama3

Run a quick test:
ollama run llama3 "Write one sentence about Ubuntu 24.04."

Tip: For smaller footprints, choose tiny models like llama3:8b or phi3. VRAM needs vary; an 8B model typically benefits from 8–12 GB of GPU VRAM, while CPU-only runs need more system RAM and patience.

Step 4 — Install Docker and Open WebUI

Open WebUI gives you a clean browser interface for Ollama. Install Docker from Ubuntu repos for a quick start:
sudo apt install -y docker.io docker-compose-plugin

Allow your user to manage Docker without sudo, then re-login:
sudo usermod -aG docker $USER

Create a persistent volume for Open WebUI data and start the container. It will connect to Ollama on the host:
docker volume create openwebui
docker run -d --name open-webui -p 3000:8080 --restart unless-stopped -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v openwebui:/app/backend/data ghcr.io/open-webui/open-webui:latest

Open your browser to http://SERVER_IP:3000 and complete the initial admin setup. Add a model in Settings if it does not appear automatically, for example llama3.

Step 5 — Optional TLS with Caddy (Automatic HTTPS)

If you have a domain pointing to your server (A record), Caddy can auto-provision HTTPS certificates. Install it and configure a simple reverse proxy:
sudo apt install -y caddy

Edit /etc/caddy/Caddyfile (replace ai.example.com with your domain):
ai.example.com {
  reverse_proxy 127.0.0.1:3000
}

Reload Caddy:
sudo systemctl reload caddy. Visit https://ai.example.com. Ensure ports 80 and 443 are open on your firewall and router.

Step 6 — Backups and Updates

Ollama models are stored under ~/.ollama for non-root users or /usr/share/ollama when installed system-wide. Back up this directory to avoid re-downloading models. Example:
tar czf ollama-backup.tgz ~/.ollama

Open WebUI data is in the Docker volume openwebui. Back it up with:
docker run --rm -v openwebui:/data -v $(pwd):/backup alpine sh -c "cd /data && tar czf /backup/openwebui-backup.tgz ."

To update Ollama:
curl -fsSL https://ollama.com/install.sh | sh && sudo systemctl restart ollama. To update Open WebUI:
docker pull ghcr.io/open-webui/open-webui:latest && docker stop open-webui && docker rm open-webui && docker run ... (re-run the previous docker run command).

Troubleshooting

If port 11434 or 3000 is in use, change the port in the docker run command or stop the conflicting service. For slow responses, try a smaller model or ensure your GPU driver is working. If Open WebUI cannot reach Ollama, verify curl http://127.0.0.1:11434/api/tags succeeds on the host and confirm the OLLAMA_BASE_URL is correct.

Wrap-up

You now have a private AI stack on Ubuntu 24.04 with Ollama handling model inference and Open WebUI offering a clean chat interface. With optional GPU acceleration, HTTPS, and simple backups, this setup is fast, secure, and maintainable—perfect for learning, prototyping, or running an internal assistant.

Overview

This step-by-step guide shows how to install Ollama and connect it to Open WebUI on Ubuntu 24.04. With this setup, you can run modern large language models like Llama 3 locally, use your NVIDIA GPU for acceleration, and chat through a clean web interface—no cloud required. The process includes installing system dependencies, enabling GPU support, running Open WebUI in Docker, pulling models, and basic troubleshooting. The language is simple, and every command is tested on Ubuntu 24.04.

Prerequisites

Before you start, make sure you have: (1) Ubuntu 24.04 with sudo access, (2) a modern NVIDIA GPU and driver support (optional but recommended), (3) at least 16 GB of RAM for medium models, and (4) stable internet access to download models and containers.

1) Update Ubuntu and install essentials

Begin by updating your packages and installing the tools we will use. If prompted, confirm with Y:

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl ca-certificates gnupg ufw git

2) Install NVIDIA drivers (for GPU acceleration)

Ollama uses your GPU automatically when the correct NVIDIA driver is present. If you do not have a GPU, you can still run models on the CPU (slower). To enable GPU acceleration on NVIDIA cards:

sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, verify the driver:

nvidia-smi

You should see your GPU listed. If you prefer manual control, install a specific driver from “Additional Drivers” in Ubuntu.

3) Install Ollama

Ollama is a lightweight server that manages models locally and exposes an HTTP API on port 11434. Install it with the official script:

curl -fsSL https://ollama.com/install.sh | sh

Enable and verify the system service:

sudo systemctl enable ollama
sudo systemctl start ollama
systemctl status ollama

If you see it active and running, Ollama is ready at http://localhost:11434.

4) Pull a model and test locally

Pull a modern, efficient model. Llama 3.1 8B is a good starting point (adjust model to your hardware):

ollama pull llama3.1:8b

Run a quick chat in the terminal to verify GPU usage:

ollama run llama3.1:8b

If your GPU is recognized, the first generation will warm up, and subsequent responses should be fast. You can also try other models like mistral, phi-3, or neural-chat.

5) Install Docker and run Open WebUI

Open WebUI provides a clean browser interface for chatting with local models. Install Docker from Ubuntu’s repo for simplicity:

sudo apt install -y docker.io docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
# Log out and back in to apply docker group membership (or run a new shell).

Start Open WebUI and point it to the host Ollama API:

docker run -d --name open-webui \
  -p 3000:8080 \
  -v open-webui:/app/backend/data \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_API_BASE_URL=http://host.docker.internal:11434 \
  ghcr.io/open-webui/open-webui:latest

Open a browser and go to http://YOUR_SERVER_IP:3000 to access Open WebUI. On first run, create an admin user. In Settings > Connections, confirm the Ollama endpoint is http://host.docker.internal:11434.

6) Secure basic network access

If UFW is enabled, allow the Open WebUI port:

sudo ufw allow 3000/tcp
sudo ufw status

For internet exposure, place Open WebUI behind a reverse proxy (Nginx/Caddy) with HTTPS. If you only use it on your LAN, keep it on port 3000 and block external access at your router or firewall.

7) Daily use tips

- To list models: ollama list. To remove one: ollama rm MODEL.
- To update Ollama when a new version is released: rerun the install script, then sudo systemctl restart ollama.
- For faster chat, choose 7B–8B models or quantized variants (like Q4_K_M). Larger models need more VRAM and RAM.

Troubleshooting

No compatible GPU found: Check nvidia-smi. If it fails, reinstall drivers with ubuntu-drivers autoinstall and reboot. Ensure Secure Boot is disabled or properly configured for NVIDIA modules.

Open WebUI cannot reach Ollama: Confirm the container can resolve the host gateway. We used --add-host=host.docker.internal:host-gateway. Also verify the env OLLAMA_API_BASE_URL and that the Ollama service is active: systemctl status ollama.

Slow generations on CPU: Use smaller models (e.g., 3–8B) or quantized versions. GPU acceleration is the biggest speed boost; ensure drivers are correct.

Ports already in use: If 3000 or 11434 is used, change the exposed port for Open WebUI (-p 4000:8080 for example) and update firewall rules.

Check logs: Ollama logs: journalctl -u ollama -f. Open WebUI logs: docker logs -f open-webui.

Optional: Reverse proxy with Nginx (HTTPS)

For public access with TLS, install Nginx and Certbot, then map a domain to your server and issue a Let’s Encrypt certificate. Point Nginx to the Open WebUI container on 3000. Keep strong passwords and consider IP allowlists or SSO for security.

What you get

You now have a private, GPU-accelerated local AI stack: Ollama runs models efficiently on your Ubuntu host, and Open WebUI gives you a modern chat interface. This setup is ideal for development, research, and privacy-focused workflows without sending your data to external clouds.

Overview

Running large language models locally is easier than ever. In this guide, you will deploy a private, GPU-accelerated AI stack with Ollama and Open WebUI on Ubuntu 24.04 using Docker. Ollama handles model downloads and inference, while Open WebUI provides a friendly chat interface, prompt library, RAG features, and multi-user management. By the end, you will have a fast, on-prem AI assistant accessible via a browser, without sending data to third parties.

Prerequisites

System: Ubuntu 24.04 (Noble), an NVIDIA GPU (e.g., RTX 3060+), NVIDIA driver 535+ (or latest), at least 16 GB RAM, and reliable internet. You should have sudo access. This tutorial uses Docker; no prior Kubernetes skills required.

Step 1 — Install NVIDIA Driver and Reboot

If you have not installed the proprietary driver, do it now and reboot:

sudo ubuntu-drivers autoinstall
sudo reboot

Step 2 — Install Docker Engine

Add Docker’s official repository and install the engine:

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu noble stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Optional: allow your user to run Docker without sudo and start a new shell:

sudo usermod -aG docker $USER
newgrp docker

Step 3 — Enable GPU inside Containers

Install NVIDIA Container Toolkit so Docker can use your GPU:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU access from Docker:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Step 4 — Create Network and Volumes

Create an isolated Docker network and persistent volumes for data:

docker network create ai
docker volume create ollama
docker volume create openwebui

Step 5 — Run Ollama (GPU-accelerated)

Start the Ollama server and keep it local-only on port 11434. The volume stores models and caches:

docker run -d --name ollama --gpus all --restart unless-stopped --network ai -p 127.0.0.1:11434:11434 -v ollama:/root/.ollama ollama/ollama:latest

Pull a starter model (choose one that fits your GPU memory):

docker exec -it ollama ollama pull llama3.1:8b
docker exec -it ollama ollama pull qwen2.5:7b-instruct

Step 6 — Run Open WebUI

Open WebUI will use Ollama via the internal Docker network. Expose the web interface on port 3000:

docker run -d --name open-webui --restart unless-stopped --network ai -e OLLAMA_BASE_URL=http://ollama:11434 -p 3000:8080 -v openwebui:/app/backend/data ghcr.io/open-webui/open-webui:latest

Open a browser and visit http://SERVER_IP:3000/. Create the first admin user. In the model selector, choose the model you pulled (e.g., llama3.1:8b) and start chatting.

Step 7 — Test the API

You can also use the local API directly. From the host:

curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.1:8b","prompt":"Say hello in one sentence."}'

Developers can point tools like LangChain or LlamaIndex at http://127.0.0.1:11434 for private inference.

Optimization Tips

If you see out-of-memory errors, switch to smaller or more aggressively quantized models (e.g., q4_k_m) when pulling: ollama pull llama3.1:8b-instruct-q4_K_M. Keep prompts concise and lower context length in Open WebUI settings. On multi-GPU systems, Ollama auto-detects devices; you can fine-tune behavior with environment variables like OLLAMA_NUM_GPU and OLLAMA_NUM_GPU_LAYERS if needed. Always use persistent volumes to avoid re-downloading models after updates.

Security Hardening

By binding Ollama to 127.0.0.1, the API is not exposed externally. Expose Open WebUI only to trusted networks. If you must publish it on the internet, use a reverse proxy with TLS and authentication. For example, with UFW, allow only your LAN:

sudo ufw allow from 192.168.0.0/16 to any port 3000 proto tcp

For Nginx or Caddy, enable HTTPS and basic auth or OIDC. Never expose port 11434 directly without protection.

Troubleshooting

GPU not detected in containers: Re-run sudo nvidia-ctk runtime configure --runtime=docker, then sudo systemctl restart docker. Confirm host drivers with nvidia-smi and container access with the CUDA test image.

Permission errors with Docker: Add your user to the docker group (sudo usermod -aG docker $USER) and re-login.

Slow responses or model crashes: Try a smaller/quantized model, reduce context window, and verify VRAM usage. Ensure swap is enabled for stability when RAM is tight.

Update containers: docker pull ollama/ollama:latest and docker pull ghcr.io/open-webui/open-webui:latest, then docker restart your containers.

Cleanup (Optional)

To stop and remove everything:

docker rm -f open-webui ollama
docker volume rm openwebui ollama
docker network rm ai

What You Achieved

You now have a modern, private AI stack with GPU acceleration on Ubuntu 24.04. Ollama simplifies model management and inference, while Open WebUI offers a polished interface ready for daily use, prototyping, and team collaboration. With careful model choice, proper security, and regular updates, this setup can replace many cloud-based assistants—keeping your data on your hardware.

Local large language models are now practical on a single server. In this step-by-step guide, you will deploy a private AI chatbot by running Ollama (for models) and Open WebUI (for the user interface) on Ubuntu using Docker. We will enable GPU acceleration with NVIDIA so responses are fast and efficient. By the end, you will have a persistent setup that survives reboots and is easy to update.

Overview

Ollama is a lightweight runtime that downloads and serves popular open-source models like Llama 3. Open WebUI is a web app that connects to Ollama and provides a clean chat interface, prompt templates, conversation history, and model management. We will run both components in Docker containers on the same Docker network and map persistent volumes for data. Optional GPU acceleration uses the NVIDIA Container Toolkit.

Prerequisites

- Ubuntu Server or Desktop (22.04 or 24.04 recommended)
- An NVIDIA GPU with proprietary drivers installed (verify with nvidia-smi) if you want GPU acceleration; CPU-only also works
- Sudo access and outbound internet connectivity
- Basic command line familiarity

1) Install Docker Engine and Compose

Install the official Docker packages and add your user to the docker group for passwordless usage.

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

2) Enable NVIDIA GPUs in Docker (optional but recommended)

If you have an NVIDIA GPU and drivers are installed, add the NVIDIA Container Toolkit so Docker can access the GPU. Verify drivers first with nvidia-smi. Then install the toolkit and restart Docker.

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test compute visibility by running a CUDA-enabled container (optional):

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

3) Create a Docker network and persistent volumes

We will create an isolated network for both containers and define persistent volumes so model files and WebUI data survive restarts.

docker network create ollama-net
docker volume create ollama
docker volume create open-webui

4) Run Ollama (model server)

Start the Ollama container. If you have a GPU, include --gpus all. The port 11434 is the Ollama API.

# GPU-enabled
docker run -d --name ollama --gpus all --restart unless-stopped \
  -p 11434:11434 \
  -v ollama:/root/.ollama \
  --network ollama-net \
  ollama/ollama:latest

# CPU-only (if you do not have an NVIDIA GPU)
# docker run -d --name ollama --restart unless-stopped \
#   -p 11434:11434 \
#   -v ollama:/root/.ollama \
#   --network ollama-net \
#   ollama/ollama:latest

Pull a model and do a quick test inside the container. Llama 3 and Qwen are great starting options.

docker exec -it ollama ollama pull llama3.1:8b
docker exec -it ollama ollama run llama3.1:8b "Write a two-line poem about local AI."

5) Run Open WebUI (front-end)

Open WebUI connects to the Ollama API. On first launch it creates an admin account when you sign in. We will point it at Ollama via the internal Docker network name.

docker run -d --name open-webui --restart unless-stopped \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -v open-webui:/app/backend/data \
  --network ollama-net \
  ghcr.io/open-webui/open-webui:latest

Open a browser to http://<server-ip>:3000. Create your account, select the model you pulled (for example, llama3.1:8b), and start chatting. You can pull additional models anytime using docker exec -it ollama ollama pull qwen2.5:7b and select them in Open WebUI.

6) Optional: Use Docker Compose instead of docker run

If you prefer a single file, create docker-compose.yml in an empty folder. The gpus: all key enables GPU acceleration when the NVIDIA toolkit is installed.

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    networks:
      - ollama-net
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: ["gpu"]
    # Alternatively for Compose v2+:
    # gpus: all

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open-webui:/app/backend/data
    networks:
      - ollama-net

volumes:
  ollama:
  open-webui:

networks:
  ollama-net:
    external: true

Then run:

docker network create ollama-net
docker compose up -d

7) Updating and maintenance

To update images, pull the latest versions and recreate the containers. Your data and models remain in volumes.

docker pull ollama/ollama:latest
docker pull ghcr.io/open-webui/open-webui:latest
docker stop open-webui ollama && docker rm open-webui ollama
# re-run the docker run commands (or docker compose up -d)

To see logs for troubleshooting, run docker logs -f ollama and docker logs -f open-webui. If the WebUI cannot see models, ensure the environment variable OLLAMA_BASE_URL points to http://ollama:11434 and both containers share the same Docker network.

Troubleshooting tips

GPU not detected: Confirm the NVIDIA driver works on the host (nvidia-smi), the NVIDIA Container Toolkit is installed, and the container uses --gpus all. If using Compose, ensure gpus: all or the device reservation is defined.

Ports already in use: Change host ports in the run commands (for example, map Open WebUI to -p 8080:8080 instead of 3000).

Slow downloads or storage limits: Models are large. Consider attaching a larger Docker volume or moving /var/lib/docker to a disk with more space. You can also choose smaller models (7B) or quantized variants.

HTTPS and access control: Put Open WebUI behind a reverse proxy such as Nginx or Caddy with HTTPS and firewall rules. For internet exposure, add authentication, rate limits, and consider a VPN or zero-trust tunnel.

What you built

You now have a local, private AI chatbot with GPU acceleration on Ubuntu using Docker. Ollama handles model serving, while Open WebUI gives you a friendly interface with history, prompts, and multi-model management. This setup is repeatable, easy to update, and keeps your data on your own hardware.

Overview

If you want a fast, private, and low-cost way to chat with large language models on your own machine, pairing Ollama with Open WebUI inside Docker is a great setup. Ollama handles model downloads and inference (CPU or NVIDIA GPU), while Open WebUI provides a clean, modern chat interface in your browser. This guide shows how to deploy both with Docker Compose on Ubuntu 22.04/24.04 and enable GPU acceleration for significant speedups.

By the end, you will have a persistent, self-hosted AI chat running at http://localhost:3000, with models managed by Ollama at http://localhost:11434. The instructions also include CPU-only notes, backup tips, and troubleshooting for common pitfalls.

Prerequisites

- Ubuntu 22.04 or 24.04 with sudo access. Windows and macOS work with Docker too, but this tutorial focuses on Ubuntu.
- For GPU acceleration: an NVIDIA GPU with a recent driver (typically 525+). CPU-only also works, just slower.
- Docker Engine and Docker Compose plugin (we will install them below).
- At least 16 GB RAM for 7B–8B models; more is better for larger models.
- Open ports: 11434 (Ollama API) and 3000 (Open WebUI).

Step 1 — Install Docker Engine and Compose

Commands:
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER && newgrp docker

Verify Docker works: docker run --rm hello-world. Verify Compose works: docker compose version.

Step 2 — Install NVIDIA Driver (GPU users)

If you already have a recent NVIDIA driver, you can skip this step. Otherwise, install the recommended driver and reboot:

sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, confirm the GPU is visible: nvidia-smi. You should see your GPU model and driver version.

Step 3 — Enable GPU inside Docker

Install the NVIDIA Container Toolkit so Docker containers can access your GPU:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

Configure Docker to use the NVIDIA runtime by default:

sudo mkdir -p /etc/docker
cat <<'EOF' | sudo tee /etc/docker/daemon.json
{
"default-runtime": "nvidia",
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
}
}
EOF
sudo systemctl restart docker

Test GPU access in containers: docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi. If you see the usual output, you are set.

CPU-only? Skip Step 3 and the GPU test. The rest works the same, just remove the NVIDIA-specific line from the compose file noted below.

Step 4 — Create a Docker Compose file

Create a new folder and the compose file:

mkdir -p ~/ai-stack && cd ~/ai-stack
nano docker-compose.yml

Paste the following content, then save:

version: "3.9"
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
ports:
- "11434:11434"
environment:
- OLLAMA_KEEP_ALIVE=30m
volumes:
- ollama:/root/.ollama
runtime: nvidia

open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
restart: unless-stopped
depends_on:
- ollama
environment:
- OLLAMA_API_BASE_URL=http://ollama:11434
ports:
- "3000:8080"
volumes:
- openwebui:/app/backend/data

volumes:
ollama:
openwebui:

Note: If you are running CPU-only, delete the line runtime: nvidia and keep everything else.

Step 5 — Launch the stack and pull a model

docker compose up -d

Wait a few seconds and confirm both containers are healthy: docker ps. Next, pull a model into Ollama. Good starters are llama3.1:8b, mistral:7b, or a small qwen2:7b.

docker exec -it ollama ollama pull llama3.1:8b

List installed models with: docker exec -it ollama ollama list.

Step 6 — Chat in Open WebUI

Open http://localhost:3000 in your browser. Open WebUI should auto-detect Ollama via the environment variable, but you can also set the API in Settings > Connections to http://ollama:11434 (inside Docker) or http://localhost:11434 (host access). Create a new chat, choose your model (for example, llama3.1:8b), and start chatting locally.

Backups, Updates, and Performance Tips

Persistence: Your models and chats are stored in the named volumes ollama and openwebui. Back them up with docker run --rm -v ollama:/data -v $(pwd):/backup alpine tar czf /backup/ollama.tgz -C / data (and similarly for openwebui).

Updates: Pull fresh images and recreate: docker compose pull && docker compose up -d. Ollama keeps your models; no need to re-download.

Performance: Prefer GPU for best speed. If RAM/VRAM is tight, choose smaller or quantized models (e.g., llama3.1:8b-q4_K_M). Set OLLAMA_KEEP_ALIVE to keep models warm between requests.

Remote access: If exposing over the internet, place Open WebUI behind a reverse proxy (Nginx, Caddy, or Traefik) and enable authentication and TLS. Never expose Ollama directly without controls.

Troubleshooting

GPU not detected: Check nvidia-smi works on the host. Then run docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi. If that fails, revisit Step 3 and confirm /etc/docker/daemon.json is correct and Docker was restarted.

Open WebUI cannot reach Ollama: Ensure both containers are up. Verify docker logs open-webui and confirm OLLAMA_API_BASE_URL is http://ollama:11434. From the host, curl http://localhost:11434/api/tags should list installed models.

Downloads are slow: Models can be several GB. Use a wired connection or pre-fetch models off-peak. You can also copy existing models into the ollama volume if you have them from another machine.

Port conflicts: If ports 11434 or 3000 are in use, change them in the compose file (left side of the colon) and recreate the stack.

What You Achieved

You now have a self-hosted AI chat stack running locally with Docker. Ollama manages lightweight, high-quality models, and Open WebUI provides a comfortable chat experience. With GPU acceleration, responses are significantly faster, and your data never leaves your machine. Extend this setup with a reverse proxy, add more models, or integrate the Ollama API into your own apps for a powerful private AI workstation.