LifeBytes Journal

Overview

Running a private AI assistant on your own machine is now practical, fast, and secure. In this step-by-step guide, you will deploy Ollama (to run large language models locally) and Open WebUI (a clean chat interface) using Docker. The tutorial covers CPU-only mode and GPU acceleration with both NVIDIA and AMD ROCm, so you get the best performance out of your hardware. By the end, you will have a persistent setup that auto-starts on boot, supports one-click model management, and is easy to update.

Prerequisites

- A 64-bit Linux host (Ubuntu 22.04+ recommended) with internet access.
- Docker Engine installed and running.
- For NVIDIA GPUs: proprietary drivers and the NVIDIA Container Toolkit.
- For AMD GPUs: ROCm-capable GPU with ROCm drivers (5.7+).
- Open ports 11434 (Ollama API) and 3000 (Open WebUI). You can change ports if they collide with other services.

Step 1 — Install Docker (Ubuntu quick method)

If Docker is not installed, run:

sudo apt-get update && sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Step 2 — Enable GPU in Docker (optional but recommended)

NVIDIA: Install the NVIDIA driver and container toolkit, then restart Docker:

sudo apt-get install -y nvidia-driver-535 (or newer)
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker

AMD ROCm: Install ROCm drivers appropriate for your GPU. Ensure devices /dev/kfd and /dev/dri exist. No extra Docker runtime is needed; you will pass devices to the container.

Step 3 — Create a dedicated Docker network

Create a private bridge network so containers can discover each other by name:

docker network create ai

Step 4 — Start Ollama (choose one)

NVIDIA GPU:

docker run -d --name ollama --gpus=all --restart unless-stopped --network ai -p 11434:11434 -v ollama:/root/.ollama ollama/ollama:latest

AMD ROCm GPU:

docker run -d --name ollama --device /dev/kfd --device /dev/dri --group-add video --ipc=host --restart unless-stopped --network ai -p 11434:11434 -v ollama:/root/.ollama ollama/ollama:rocm

CPU-only:

docker run -d --name ollama --restart unless-stopped --network ai -p 11434:11434 -v ollama:/root/.ollama ollama/ollama:latest

Step 5 — Start Open WebUI and connect it to Ollama

Run Open WebUI on port 3000 and point it at the Ollama API using the container name:

docker run -d --name open-webui --restart unless-stopped --network ai -p 3000:8080 -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://ollama:11434 open-webui/open-webui:latest

Open a browser and visit http://YOUR_SERVER_IP:3000. On first launch, create your admin account. In Settings, disable public sign-ups if this instance is exposed to untrusted networks.

Step 6 — Download a model and chat

You can pull models from Open WebUI’s UI. Or pull via CLI:

docker exec -it ollama ollama pull llama3.1:8b

For lower VRAM, try quantized variants (for example: llama3.1:8b-instruct-q4_0). After the download completes, select the model in Open WebUI and start chatting locally.

Security and access tips

- Keep ports private if possible; bind to localhost and use an authenticated reverse proxy (Nginx, Caddy, or Traefik) if exposing to the internet.
- Regularly update images and disable open registration in Open WebUI.
- Use Docker volumes (already configured) to persist models and settings across updates.

Updating to the latest versions

docker pull ollama/ollama:latest (or ollama/ollama:rocm for AMD)
docker pull open-webui/open-webui:latest
docker stop open-webui ollama && docker rm open-webui ollama
Re-run the docker run commands from Steps 4–5. Your data persists in volumes.

Troubleshooting

- GPU not detected (NVIDIA): test with docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi.
- GPU not detected (AMD): ensure /dev/kfd and /dev/dri exist and your user is in the video group (the container already adds it).
- Slow or failed model pulls: check DNS/proxy and rerun the pull. Try a smaller or more quantized model.
- Port conflicts: change -p 11434:11434 or -p 3000:8080 to other free host ports.

Uninstall and cleanup

docker rm -f open-webui ollama
docker volume rm open-webui ollama
docker network rm ai

You now have a flexible, private AI stack that runs entirely on your machine. Swap models as needed, tune quantization for your hardware, and keep your data under your control.

Overview

This tutorial shows how to deploy a private, local AI stack with Ollama (model runtime) and Open WebUI (chat interface) using Docker on Ubuntu 22.04/24.04. You will learn how to run it on CPU, enable NVIDIA or AMD/ROCm GPU acceleration, secure the web interface, and keep everything up to date. The result is a fast, reliable, and low-maintenance setup suitable for labs, developers, and small teams.

Prerequisites

You need an Ubuntu 22.04 or 24.04 system with sudo access, 16 GB+ RAM (more is better), 20 GB+ free disk space, and a stable internet connection. For GPU acceleration, use a recent NVIDIA GPU with official drivers or a compatible AMD GPU with ROCm-capable kernel and hardware. Ensure ports 11434 (Ollama) and 3000 (Open WebUI) are free. If you plan to expose the service on the internet, prepare a domain name and DNS A/AAAA record pointing to the server.

Step 1: Install Docker Engine and Compose

sudo apt update && sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Step 2: GPU Preparation (optional but recommended)

NVIDIA: Install the proprietary driver and the NVIDIA Container Toolkit so Docker can access your GPU.

sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers install
sudo reboot

distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

AMD (ROCm): Ensure your GPU is ROCm-capable and the kfd and dri devices are present. Give your user access to the required groups.

sudo usermod -aG render,video $USER
sudo reboot

Step 3: Create a Docker Compose file

Create a working directory like ~/ai-stack, then create docker-compose.yml. The following example starts Ollama and Open WebUI with volumes for persistence. It includes variants for CPU, NVIDIA, and AMD. Only keep one GPU option at a time.

docker-compose.yml (CPU-only by default):

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
     - "11434:11434"
    volumes:
     - ollama:/root/.ollama
    environment:
     - OLLAMA_KEEP_ALIVE=24h
     - OLLAMA_NUM_THREADS=8
  # For NVIDIA GPU (uncomment the next 4 lines and comment the AMD lines below):
  #   runtime: nvidia
  #   environment:
  #    - NVIDIA_VISIBLE_DEVICES=all
  #    - NVIDIA_DRIVER_CAPABILITIES=compute,utility
  # For AMD ROCm GPU (use the ROCm image and device mappings):
  #   image: ollama/ollama:rocm
  #   devices:
  #    - /dev/kfd
  #    - /dev/dri
  #   group_add:
  #    - "video"
  #    - "render"
  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    depends_on:
     - ollama
    restart: unless-stopped
    environment:
     - OLLAMA_API_BASE=http://ollama:11434
    ports:
     - "3000:8080"
    volumes:
     - openwebui:/app/backend/data
volumes:
  ollama:
  openwebui:

Step 4: Start the stack

docker compose up -d

Check containers and logs to confirm both services are healthy.

docker ps
docker logs -f ollama
docker logs -f open-webui

Step 5: Pull a model and test

Use Ollama to download a model. Popular choices are llama3.1:8b, llama3.1:70b (needs more VRAM), mistral, or qwen2. Start with an 8B or 7B model to validate your setup.

docker exec -it ollama ollama pull llama3.1:8b
curl http://localhost:11434/api/tags

Open a browser to http://<server-ip>:3000. The first user that signs up in Open WebUI becomes the admin. In Settings, point the Ollama endpoint to http://ollama:11434 (it is already set via OLLAMA_API_BASE). Create a new chat and pick your model from the dropdown.

Step 6: Optional security and HTTPS

By default, Open WebUI is accessible on port 3000 and provides its own user system. For internet exposure, put it behind an HTTPS reverse proxy and disable public signups after creating the admin. If you use UFW, allow only necessary ports:

sudo ufw allow 22/tcp
sudo ufw allow 80,443/tcp
sudo ufw enable

A simple approach is to add a Caddy or Nginx reverse proxy in front of Open WebUI for automatic TLS. Map your domain (e.g., ai.example.com) to the server, then proxy requests to open-webui:8080. Limit administrative access using firewall rules, strong passwords, and, if available, SSO/OIDC in Open WebUI.

Step 7: Updating and backing up

To update images to the latest versions and apply them with minimal downtime:

cd ~/ai-stack
docker compose pull
docker compose up -d

Your models and chat data live in Docker volumes. Back them up regularly:

docker run --rm -v ollama:/data -v $(pwd):/backup alpine tar czf /backup/ollama-vol-$(date +%F).tgz -C /data .
docker run --rm -v openwebui:/data -v $(pwd):/backup alpine tar czf /backup/openwebui-vol-$(date +%F).tgz -C /data .

Troubleshooting tips

If GPU is not used on NVIDIA, confirm nvidia-smi works on the host and the container runtime is configured. For AMD, ensure /dev/kfd and /dev/dri exist and the container uses the ollama/ollama:rocm image with the proper device mappings. Model loading failures typically indicate insufficient RAM/VRAM; try a smaller quantization or a smaller model. If the UI cannot see Ollama, verify OLLAMA_API_BASE and that containers can resolve each other by service name.

You are done

You now have a modern, private AI chat stack running on Docker with optional GPU acceleration. Ollama keeps model management simple, and Open WebUI provides a clean, multi-user interface. This setup is easy to maintain, portable across servers, and ready for experimentation with different open-source models and embeddings.