LifeBytes Journal

Overview

This tutorial shows how to self-host large language models locally by deploying Ollama and Open WebUI on Ubuntu 24.04 LTS with NVIDIA GPU acceleration using Docker Compose. Ollama handles model runtimes and downloads, while Open WebUI provides a friendly web interface, prompt management, and multi-user features. By the end, you will have a reproducible, GPU-enabled AI stack reachable from a browser on your LAN.

Prerequisites

Before you start, confirm: (1) Ubuntu 24.04 LTS installed and updated, (2) An NVIDIA GPU supported by recent drivers, (3) Administrative (sudo) access, and (4) Internet connectivity. If Secure Boot is enabled, you may need to enroll the NVIDIA kernel module signing key during driver installation.

Step 1: Prepare Ubuntu

sudo apt update && sudo apt -y upgrade
sudo apt -y install curl git ca-certificates gnupg

Step 2: Install the NVIDIA Driver

Use Ubuntu’s built-in tool to select the correct, current driver:

ubuntu-drivers list
sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, verify the GPU is recognized:

nvidia-smi

Step 3: Install Docker Engine and Compose

Add the official Docker repository and install the engine plus the Compose plugin:

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Step 4: Install NVIDIA Container Toolkit

This toolkit lets Docker containers access the GPU:

curl -fsSL https://nvidia.github.io/nvidia-container-toolkit/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/nvidia-container-toolkit/$distribution/nvidia-container-toolkit.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt -y install nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test GPU access in a container:

docker run --rm --gpus all nvidia/cuda:12.4.1-runtime-ubuntu22.04 nvidia-smi

Step 5: Create the Docker Compose project

Make a working directory and create your Compose file:

mkdir -p ~/ai-stack && cd ~/ai-stack

Create a file named docker-compose.yaml with the following content (indentation matters):

version: "3.9"
services:
ollama:
image: ollama/ollama:latest
restart: unless-stopped
ports:
- "11434:11434"
environment:
- OLLAMA_HOST=0.0.0.0
volumes:
- ollama:/root/.ollama
gpus: all

openwebui:
image: ghcr.io/open-webui/open-webui:latest
restart: unless-stopped
depends_on:
- ollama
ports:
- "3000:8080"
environment:
- OLLAMA_BASE_URL=http://ollama:11434
- WEBUI_NAME=MyLocalAI
volumes:
- openwebui:/app/backend/data

volumes:
ollama:
openwebui:

Step 6: Launch the stack

docker compose up -d

Wait a few seconds. Visit http://localhost:3000 (or http://SERVER_IP:3000) to open Open WebUI. The first user usually becomes the admin. You can manage models from the UI or the CLI.

Step 7: Pull a model and test

Pull a model into the Ollama volume (example: Meta’s Llama 3.1 8B):

docker compose exec ollama ollama pull llama3.1

Generate a quick response from the API:

curl -s http://localhost:11434/api/generate -d '{"model":"llama3.1","prompt":"Say hello from a local GPU!","stream":false}' | jq .response

In Open WebUI, choose the model from the dropdown and start chatting.

Step 8: Use the OpenAI-compatible API

Ollama exposes an OpenAI-style API under /v1. Example with Python’s OpenAI SDK:

pip install openai
python - <<'PY'
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
resp = client.chat.completions.create(
model="llama3.1",
messages=[{"role":"user","content":"Give me a one-line fun fact."}]
)
print(resp.choices[0].message.content)
PY

Maintenance and updates

To update images without losing your data or models stored in volumes, run:

cd ~/ai-stack
docker compose pull
docker compose up -d

To update or add models, use the UI or the CLI, for example: docker compose exec ollama ollama pull mistral.

Troubleshooting

nvidia-smi fails inside containers: Ensure the NVIDIA driver is installed and matches your GPU. Reboot after installation. Confirm Docker sees the GPU with docker run --rm --gpus all nvidia/cuda:12.4.1-runtime-ubuntu22.04 nvidia-smi.

Compose “gpus” not recognized: Check your Compose plugin version with docker compose version. Update Docker packages if outdated. As a fallback, configure the NVIDIA runtime as default and remove the gpus key: sudo nvidia-ctk runtime configure --runtime=docker --set-as-default && sudo systemctl restart docker.

Slow downloads or OOM: Models are large; use a fast, stable network and ensure enough VRAM and RAM. If a model does not fit your GPU, choose a smaller variant (e.g., llama3.1:8b or a quantized build like q4_K_M).

Security tips

Bind the services to your LAN or localhost by default and place them behind a reverse proxy with TLS if exposing over the internet. In Open WebUI, enable authentication and restrict new user registration if not needed. Consider a firewall rule to limit access to ports 11434 and 3000.

Remove the stack

To stop the containers while preserving data: docker compose down. To remove everything, including downloaded models and chat history: docker compose down -v.

You now have a modern, GPU-accelerated local AI stack that is easy to manage and update. The same approach works for additional services like vector databases or reverse proxies, making it a flexible foundation for on-prem AI experiments and production prototypes.

Overview

This step-by-step guide shows you how to run a local AI stack on Ubuntu 22.04/24.04 using Docker Compose, Ollama, and Open WebUI with NVIDIA GPU acceleration. Ollama provides a lightweight local API for popular large language models (LLMs) like Llama 3, Mistral, and Qwen, while Open WebUI delivers a clean, user-friendly chat interface. By the end, you will have a secure, updatable setup that serves a local LLM with GPU support for fast responses and offline privacy.

Prerequisites

System: Ubuntu 22.04 or 24.04 with a recent NVIDIA GPU driver installed. Aim for at least 16 GB RAM and sufficient disk space (20–40 GB or more, depending on models). This tutorial uses Docker Engine, Docker Compose plugin, and the NVIDIA Container Toolkit.

Network/Ports: Ollama exposes port 11434 (local only in this guide). Open WebUI will use port 3000. Adjust firewall rules if the server is internet-facing.

1) Install Docker Engine and Compose

Run the following commands to install Docker and the Compose plugin:

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

2) Enable NVIDIA GPU for Containers

Install the NVIDIA Container Toolkit so Docker can pass your GPU to containers. Make sure the host driver is already installed and nvidia-smi works on the host.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Quick GPU test inside a container (optional):

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

3) Create Docker Compose for Ollama + Open WebUI

Create a working folder and a docker-compose.yml file:

mkdir -p ~/local-llm && cd ~/local-llm
nano docker-compose.yml

Paste the following content, then save:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=2h
    gpus: all
    ports:
      - "127.0.0.1:11434:11434"

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    environment:
      - OLLAMA_API_BASE=http://ollama:11434
      - WEBUI_SECRET_KEY=change_me_long_random
      - ENABLE_SIGNUP=false
      - [email protected]
    volumes:
      - open-webui:/app/backend/data
    ports:
      - "3000:8080"

volumes:
  ollama:
  open-webui:

Binding Ollama to 127.0.0.1 keeps the model API private. Open WebUI is exposed on port 3000. For a remote VPS, secure it with a firewall or reverse proxy before exposing it.

4) Start the Stack and Pull a Model

Bring everything up:

docker compose up -d

Pull a model into Ollama (example: Llama 3.1 8B):

docker exec -it ollama ollama pull llama3.1:8b

Test the API locally:

curl http://127.0.0.1:11434/api/tags
curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.1:8b","prompt":"Say hello in one sentence."}'

Open your browser to http://SERVER_IP:3000 (or http://localhost:3000) to access Open WebUI and start chatting with the model you pulled.

5) Secure Access

If this runs on a server, restrict port 3000 to trusted IPs or place it behind HTTPS using a reverse proxy (Caddy, Nginx, or Traefik). For quick private access through SSH tunneling, use:

ssh -L 3000:localhost:3000 user@SERVER_IP

Set ENABLE_SIGNUP=false to prevent public registrations and choose a strong WEBUI_SECRET_KEY. You can also bind Open WebUI to localhost only by changing the port mapping to 127.0.0.1:3000:8080 and serving it via your reverse proxy.

6) Update, Backup, and Maintenance

Update images: keep the stack current with:

docker compose pull && docker compose up -d

Backup models and data: the named volumes hold your models and UI data. You can archive them like this:

mkdir -p ~/local-llm/backups && cd ~/local-llm
docker run --rm -v ollama:/data -v "$PWD/backups":/backup alpine sh -c 'tar czf /backup/ollama-vol.tar.gz -C /data .'
docker run --rm -v open-webui:/data -v "$PWD/backups":/backup alpine sh -c 'tar czf /backup/openwebui-vol.tar.gz -C /data .'

Stop/Start: docker compose down stops containers but keeps volumes. Use docker compose down -v to remove volumes as well (this deletes downloaded models and chat history).

7) Troubleshooting

GPU not detected: run nvidia-smi on the host; if it fails, reinstall the NVIDIA driver. Verify the container sees your GPU with docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi. Ensure you ran sudo nvidia-ctk runtime configure --runtime=docker and restarted Docker.

Slow or out-of-memory: choose a smaller or quantized model (for example, llama3.1:8b or mistral:7b). Large models need more VRAM. You can also run CPU-only by removing GPU options, but performance will drop.

Port conflicts: change the host ports in docker-compose.yml (e.g., "127.0.0.1:11435:11434" and "3001:8080"), then docker compose up -d.

Cannot access WebUI: confirm the container is healthy with docker ps and check logs via docker logs open-webui. If remote, verify firewall rules allow your IP to reach port 3000 or use SSH tunneling.

What You Achieved

You now have a modern, GPU-accelerated local LLM platform on Ubuntu using Docker Compose. Ollama handles model management and API requests, while Open WebUI provides a polished chat experience. This stack is easy to update, simple to back up, and private by default—ideal for development, helpdesk knowledge assistants, and secure, offline AI workflows.

Overview

This step-by-step guide shows how to self-host Ollama with Open WebUI using Docker on Ubuntu, with optional NVIDIA GPU acceleration. You will get a modern local AI stack that can run LLMs such as Llama 3.1 or Mistral privately, with a clean web interface, persistent storage, and an easy update path. The tutorial targets Ubuntu 22.04/24.04 and works on servers, workstations, and homelabs.

Prerequisites

- Ubuntu 22.04/24.04, sudo access, and basic command-line knowledge.
- For GPU acceleration: an NVIDIA GPU with up-to-date drivers (CUDA-compatible). CPU-only mode also works; you can skip the GPU steps.

1) Install Docker and Compose

Run the following commands to install Docker Engine and the Compose plugin (official repository):

sudo apt update && sudo apt install -y ca-certificates curl gnupg lsb-release
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER && newgrp docker

2) Enable NVIDIA GPU in Containers (optional)

If your system has a supported NVIDIA GPU, install the driver from Ubuntu’s repo or NVIDIA’s site (e.g., sudo apt install nvidia-driver-535), reboot, and verify nvidia-smi works. Then install the NVIDIA Container Toolkit so Docker can pass GPUs into containers:

curl -fsSL https://nvidia.github.io/nvidia-container-toolkit/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
distribution=$(. /etc/os-release; echo $ID$VERSION_ID) && \
echo "deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit.gpg] https://nvidia.github.io/libnvidia-container/$distribution/$(uname -m) /" | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test GPU access in Docker:

docker run --rm --gpus all nvidia/cuda:12.3.1-base-ubuntu22.04 nvidia-smi

3) Create a Docker Compose file

We will run two services: Ollama (the model server) and Open WebUI (the web interface). Create a working folder such as ~/ollama-stack and inside it create docker-compose.yml with the following content:

version: "3.9"

services:
ollama:
image: ollama/ollama:latest
container_name: ollama
ports:
- "127.0.0.1:11434:11434"
volumes:
- ollama:/root/.ollama
environment:
- OLLAMA_NUM_PARALLEL=2
- OLLAMA_MAX_LOADED_MODELS=2
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
restart: unless-stopped

open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
depends_on:
- ollama
ports:
- "127.0.0.1:3000:8080"
environment:
- OLLAMA_BASE_URL=http://ollama:11434
volumes:
- openwebui:/app/backend/data
restart: unless-stopped

volumes:
ollama:
openwebui:

Notes: The loopback bindings (127.0.0.1) keep services off the public network; put a reverse proxy in front if you need remote access. If your Docker Compose version complains about the deploy GPU section, remove it and start Ollama with: docker run --gpus all ... or add --gpus all via docker compose overrides.

4) Start the stack and pull a model

Start the services:

docker compose up -d

Pull a model into Ollama (examples for CPU/GPU-capable LLMs):

docker exec -it ollama ollama pull llama3.1:8b
docker exec -it ollama ollama pull mistral:7b

Open your browser to http://localhost:3000, create the first admin user, and select your Ollama model in Open WebUI. You can now chat, run prompts, and manage models from the interface.

5) Optional: Reverse proxy and HTTPS

For secure remote access, put Nginx or Caddy in front of Open WebUI with HTTPS and basic auth or OAuth. Example: expose Open WebUI only on 127.0.0.1:3000 and publish a domain via the proxy to terminate TLS with Let’s Encrypt. Always restrict access; these services should not be open on the public internet without authentication.

6) Performance tips

- Use GPU if available: it accelerates inference dramatically, especially for 13B+ models.
- Tune concurrency with OLLAMA_NUM_PARALLEL and limit memory pressure using OLLAMA_MAX_LOADED_MODELS.
- Choose model sizes that match your VRAM/RAM. For 8 GB VRAM, 7B/8B models work well; for 12–24 GB, 13B–30B is more comfortable.
- Keep images updated: docker compose pull && docker compose up -d.

7) Backup and restore

Your data lives in Docker volumes. To back up:

docker run --rm -v ollama:/data -v $(pwd):/backup alpine tar czf /backup/ollama-vol.tar.gz -C /data .
docker run --rm -v openwebui:/data -v $(pwd):/backup alpine tar czf /backup/openwebui-vol.tar.gz -C /data .

Restore by creating empty volumes and extracting the archives back into them using the same pattern.

8) Troubleshooting

- No GPU inside container: verify nvidia-smi works on the host, ensure NVIDIA Container Toolkit is installed, and try docker run --gpus all to validate. On WSL2, enable GPU support and CUDA toolkit for WSL.
- High RAM usage: reduce parallel requests and use smaller quantizations (e.g., q4_K_M variants).
- Slow downloads: Ollama model downloads depend on upstream mirrors; retry or prefetch models during off-peak hours.
- Port already in use: change 11434 or 3000 in the compose file.

Conclusion

You now have a modern, private AI stack running locally with Docker: Ollama for efficient model serving and Open WebUI for a friendly interface. This setup is easy to update, secure behind a proxy, and flexible for both CPU-only and GPU-accelerated systems. Add or swap models as your needs grow, and keep your data under your control.

Overview

This tutorial shows how to run large language models locally using Ollama and Open WebUI with Docker on Ubuntu 22.04 or 24.04. You will get a private, fast AI chat interface in your browser with optional NVIDIA GPU acceleration. We will cover prerequisites, Docker setup, GPU configuration, a ready-to-use docker-compose.yml, updates, backups, and troubleshooting. The steps also work for CPU-only machines.

What You Will Need

Before you begin, make sure you have the following:

• Ubuntu 22.04 or 24.04 (freshly updated)
• Docker Engine and Docker Compose plugin
• Optional: NVIDIA GPU with recent drivers (e.g., 535+), CUDA-capable
• At least 16 GB RAM recommended; more VRAM helps with larger models
• 1 open TCP port for the web UI (default 3000)

Step 1: Install Docker and Compose

Install Docker from the official repository and enable it on boot. If you already have Docker, ensure it is up to date.

sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Step 2 (Optional): Enable NVIDIA GPU for Containers

If you have an NVIDIA GPU, install the NVIDIA Container Toolkit so Docker can access the GPU. First verify the GPU is detected:

nvidia-smi

Then install the container toolkit:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit.gpg] https://#' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

If you are on a CPU-only system, skip this step. The stack will still work, just slower.

Step 3: Create the Docker Compose File

Create a working directory and a docker-compose.yml. This configuration runs two services: Ollama (model runtime) and Open WebUI (browser UI). It includes a GPU-enabled section that you can remove if you are running on CPU.

mkdir -p ~/ai-stack && cd ~/ai-stack
nano docker-compose.yml

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=12h
      - OLLAMA_NUM_PARALLEL=1
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: ["gpu"]   # Remove this block on CPU-only hosts

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - openwebui-data:/app/backend/data

volumes:
  ollama-data:
  openwebui-data:

For CPU-only systems, delete the deploy.resources.reservations.devices block under the ollama service to avoid GPU scheduling errors.

Step 4: Start the Stack and Pull a Model

Launch the containers in the background:

docker compose up -d

Pull a model with Ollama. The following example downloads a compact, general-purpose model:

docker exec -it ollama ollama pull llama3.1:8b

You can list available models or search the model library at the official Ollama registry. Popular options include llama3.1:8b, mistral:7b, and neural-chat. Larger models require more RAM/VRAM.

Step 5: Access the Web Interface

Open your browser and visit http://SERVER_IP:3000 to access Open WebUI. On first launch, create an admin user. In Settings, confirm the Ollama base URL is http://ollama:11434. Choose your default model and start chatting locally.

Useful Tips

Switch or Add Models: Use the Models section in Open WebUI or run docker exec -it ollama ollama pull MODEL:TAG. You can host multiple models and select them per chat.

Performance Tuning: On GPU hosts, keep drivers current. In low-VRAM scenarios, choose quantized models (e.g., Q4_K_M variants). Adjust OLLAMA_NUM_PARALLEL and context window settings to balance speed and quality.

Storage Paths: Models are stored in the ollama-data volume; Open WebUI data lives in openwebui-data. Back up both volumes regularly.

Updating and Maintenance

To update to the latest images without losing data, pull and recreate:

docker compose pull
docker compose up -d

To back up volumes, stop the stack and export them or bind-mount to a backup path. Example quick export:

docker run --rm -v ollama-data:/data -v $(pwd):/backup alpine tar czf /backup/ollama-data.tar.gz -C /data .
docker run --rm -v openwebui-data:/data -v $(pwd):/backup alpine tar czf /backup/openwebui-data.tar.gz -C /data .

Security Considerations

Do not expose port 3000 or 11434 directly to the internet. If remote access is required, use a reverse proxy (Caddy, Nginx, or Traefik) with HTTPS and authentication, or place the service behind a VPN like WireGuard or Tailscale. Limit container memory/CPU if sharing the host.

Troubleshooting

Permission denied on Docker: Run newgrp docker or log out/in after adding your user to the docker group.

GPU not detected in container: Ensure nvidia-smi works on the host, the NVIDIA Container Toolkit is installed, and you did not remove the GPU reservation block in compose. Restart Docker after changes.

Port already in use: Change ports in docker-compose.yml (e.g., 3001:8080) and recreate the stack.

Models fail to load due to memory: Choose smaller or quantized models, reduce context length, or add swap on the host.

Uninstall or Remove

To stop and remove the stack while keeping volumes:

docker compose down

To remove everything including data volumes:

docker compose down -v

Conclusion

With Docker, Ollama, and Open WebUI, you can run private AI models on your own hardware in minutes. This setup scales from a simple laptop to a GPU workstation and is easy to update and back up. Start with a lightweight model, then experiment with larger options as your resources allow.